SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 2 summarized the results of all compared methods on DuConv, Persona-Chat and CMU-DoG datasets. Firstly, we can see that our method achieves competitive performance over all datasets, especially in cross-lingual scenario where our method outperforms the baselines by large margins no matter fine-tuning or freezing the language model during the CSRL training stage. Although CSAGN exceeds our method on DuConv test set, it fails to work well in cross-lingual scenario. We think this is because it heavily relies on the rich features from the Chinese pre-trained language model and it is overfitting on the predicate-aware information. Superior to CSAGN, our model with the
$^{2}$ https://wit3.fbk.eu/
Method	DuConv	Persona-Chat	CMU-DoG
F1all	F1cross	F1intra	F1all	F1cross	F1intra	F1all	F1cross	F1intra
All objectives	88.60	83.72	89.27	66.75	24.13	69.44	58.45	58.92	58.82
w/o TLM & HPSI	88.07	81.90	89.06	65.07	23.91	68.34	58.23	53.15	59.24
w/o SPI & UOR	87.75	81.56	88.81	68.35	22.86	71.29	58.08	47.93	60.22
w/o SAI	88.00	83.16	89.06	64.74	23.33	67.99	59.94	54.68	61.87
only w/ TLM & HPSI	87.82	83.21	88.95	65.56	24.12	68.60	57.32	52.74	59.11
only w/ SPI & UOR	88.45	83.70	89.10	64.09	24.09	67.50	59.71	57.23	60.80
only w/ SAI	88.49	82.97	89.24	65.82	23.30	69.18	57.20	50.54	57.63
w/ end2end pre-training	87.28	81.02	88.73	64.37	21.17	67.77	57.86	50.40	58.20
OursXLM-R	88.30	83.38	89.17	65.57	24.11	68.51	59.60	56.16	60.78
w/o SC-Encoder	88.02	79.11	89.05	63.12	17.55	66.70	57.72	50.42	58.03
w/o PA-Encoder	88.10	81.32	88.78	64.05	22.38	64.82	58.24	54.00	59.23
w/o SC-Encoder and PA-Encoder	86.14	73.63	87.12	62.87	12.38	63.02	52.44	41.02	56.23
w/o MTRANS	88.25	83.01	89.08	65.27	23.10	68.38	58.58	55.41	59.98
multilingual backbone achieves outstanding performance on both language in-domain and cross-lingual datasets. This observation is expected because (1) our model is language-agnostic which makes the cross-lingual transfer easier; (2) our model captures high-level conversational features in SC-Encoder, thus enhancing the capacities of the model to recognize cross-arguments; (3) rich semantic features are modeled by PA-Encoder, which would improve the capacities of the model to recognize intra-arguments.
Secondly, although our model has achieved good performance over all datasets, further improvements can be observed after incorporating the proposed pre-training objectives, especially when freezing the parameters of the language model. Exceptionally, we find that the performance on the CMU-DoG dataset heavily drops after introducing the pre-training objectives, especially in terms of  $\mathrm{F1}_{\text {intra }}$ . We think this is because the semantic argument spans in CoNLL-2012 are relatively different from those in CMU-DoG, thus leading to the vague boundary detection and performance drop. To verify this assumption, we conduct an ablation study by removing SAI from the pre-training stage. Interestingly, we observe substantial improvements over  $\mathrm{F1}_{\text {all }}$  and  $\mathrm{F1}_{\text {intra }}$ , suggesting that pre-training on CoNLL-2012 does hurt the performance on CMU-DoG. Furthermore, we also find that fine-tuning all parameters leads to slightly better performance than freezing the language model during the CSRL training stage. This finding is consistent with the previous work (Conia et al., 2021).
Table 3 presents the results of ablation studies on pre-training objectives and different modules.
Table 3: Ablation studies on pre-training objectives and different modules.
U1	how many games did the colts win?
U2	the ColtsARG0 finished with a 12-2 record.
Question	who did they playpredicate in the playoffs?
Question'	who did the Colts play in the playoffs?
Table 4: An example of question-in-context rewriting.
For the pre-training objectives, we found that (1) removing TLM & HPSI objective hurts the performance consistently but slightly; (2) SPI & UOR objectives help the model to better identify the cross-arguments; (3) SAI objective helps to find intra-arguments on DuConv and Persona-Chat, but might hurt the F1_intra score on CMU-DoG; (4) hierarchical pre-training is superior to end-to-end pre-training which simultaneously optimizes all auxiliary objectives. We think this is because the end2end pre-training is extremely unstable and confuses the optimization process of the model.
For model components, we found that only removing one of the SC-Encoder, PA-Encoder or MTRANS slightly affect the performance. However, the performance heavily decreases when SC-Encoder and PA-Encoder are both removed. We think the reason is that at least one module is needed to capture the high-level features on the top of the language model. We preserve these two modules in our model since they essentially learn different abilities, i.e., the ability of dialogue modeling and semantics modeling, which also makes our model more explainable.
4.4 Applications
Xu et al. (2021) has confirmed the usefulness of CSRL by applying CSRL parsing results to two Chinese dialogue tasks, including dialogue context rewriting and dialogue response generation. In the
Method	Persona-Chat (en)	BConTrast (de)	BSD (ja)
B1/2	D1/2	Human [1-5]	B1/2	D1/2	B1/2	D1/2
Seq2Seq	0.138/0.069	0.051/0.094	2.72	0.089/0.042	0.041/0.089	0.125/0.051	0.123/0.248
mUniLMw/CSRL	0.188/0.113	0.114/0.217	3.02	0.107/0.061	0.079/0.187	0.162/0.080	0.175/0.320
mUniLMw/CSRL	0.195/0.122	0.116/0.223	3.16	0.112/0.065	0.082/0.191	0.178/0.088	0.177/0.326
mBARTwo/CSRL	0.198/0.125	0.120/0.228	3.20	0.115/0.072	0.086/0.206	0.193/0.097	0.182/0.340
mBARTw/CSRL	0.217/0.136	0.124/0.233	3.25	0.118/0.077	0.090/0.212	0.205/0.110	0.185/0.346
Table 5: Evaluations on response generation tasks in English, German and Japanese.
Method	B1	B2	B4
Seq2Seq	-	-	49.67
SARG(Huang et al., 2020)	-	-	54.80
RUN(Liu et al., 2020a)	70.50	61.20	49.10
Human evaluation	-	-	59.92
Oursw/ CSRL	69.24	62.93	52.78
Oursw/ CSRL	70.26	64.19	54.23
Table 6: Evaluation results on the dataset of CANARD. same vein, we also explore whether CSRL could benefit to the same non-Chinese dialogue tasks.
Question-in-context Rewriting Question-in-context rewriting (Elgohary et al., 2019) is a challenging task which requires the model to resolve the conversational dependencies between the question and the context, and then rewrite the original question into independent one. This is an example in Table 4. The question "who did they play in the playoffs?" cannot be independently understood without knowing "they" refer to, but it can be resolved with the given context.
Since the CSRL models can identify the predicate-argument structures from the entire conversation, we believe that it can help this rewriting task by searching the dropped or referred components from the context. For example, in Table 4, our CSRL parser can find that the ARG0 of the predicate "play" is "the Colts". Motivated by this observation, we attempt to borrow CSRL to help the question rewriting with the context. We first employ the pre-trained cross-lingual CSRL parser  $(\text{Ours}_{\text{XLM-R}} + \text{pre-train})$  to extract predicate-argument pairs from conversations. We adopt the model proposed in (Xu et al., 2020) to achieve the rewriting. More details about the model are in Appendix E.
Since the rewriting datasets are only available in Chinese and English, we hereby only evaluate on CANARD (Elgohary et al., 2019) which is a widely used English question rewriting dataset, and report the BLEU scores. Table 6 lists the evaluation results on CANARD. We can see that our implementation with CSRL achieves competitive performance against the state-of-the-art rewriting models,
i.e., SARG (Huang et al., 2020) and RUN (Liu et al., 2020a), and significantly outperforms the baseline method (Bahdanau et al., 2014). Note that, in this part, we are more focused on the improvements after introducing CSRL information. We find that the scores across all metrics are improved with the aid of CSRL. To figure out the reasons of these improvements, we investigate which type of questions could benefit from CSRL information most. By comparing the rewritten questions of different methods, we find that the questions that require information completion, especially those containing referred components (around  $15\%$  cases), benefit from CSRL most. This observation is in line with our expectation that our CSRL parser could consistently offer essential guidance by recovering the dropped or referred text components.
Multi-turn Dialogue Response Generation Besides the rewriting task that is heavily affected by omitted components, we also explore the usefulness of CSRL to multi-turn dialogue response generation, one of the main challenges in dialogue community. In contrast to single-turn dialogue response generation, multi-turn dialogues suffer more frequently occurred ellipsis and anaphora, which leads to vague context representations. To this end, we attempt to employ CSRL to build better context representations. In specific, we highlight the words picked up by the CSRL parser, and then teach the model to pay more attention on those words which would hold more semantic features.
We evaluate on three dialogue datasets in different languages, including Persona-Chat (Zhang et al., 2018) in English, BConTrast (Farajian et al., 2020) in German and BSD (Rikters et al., 2019) in Japanese. We report BLEU-1/2 and Distinct-1/2 scores for the comparison. We employ the pre-trained cross-lingual CSRL parser (OursXLM-R) to analyze the latest utterance, and obtain the predicate-argument pairs. Then the concatenated sequence of the extracted pairs and the context is fed into our model for response generation. We
adopt the UniLM (Dong et al., 2019) or mBART (Liu et al., 2020b) as our generation model. More implementation details are in Appendix E.
Table 5 summarizes the results on three datasets. We can see that the models with different backbones can consistently benefit from the additional introduced CSRL information. While substantial gains from CSRL information are obtained on English and Japanese dialogues, smaller improvements are observed on the German dialogue task. We think this is because English is well-represented in pre-trained multilingual models and Japanese is more similar to Chinese while German accounts for none of both. Apart from automatic evaluation criteria, we also conduct human evaluation on the English dataset. Specifically, we randomly select 200 generated responses for each method, and then recruit three annotators to evaluate the coherence and informativeness of the response against the conversation context by giving a score ranging from 1 (worst) to 5 (best). We find that the method with CSRL wins in  $35\%$  cases, and ties with the vanilla model in around  $55\%$  cases. With more careful analysis, we find that the responses that contains entities mentioned in histories benefit from CSRL information most. We think this is because nonephrases are more likely to be recognized as semantic arguments by CSRL parser, and then receive more attentions during encoding.
In this work, we propose a simple but effective model with five pre-training objectives to perform zero-shot cross-lingual CSRL, and also confirm the usefulness of CSRL to non-Chinese dialogue tasks.
This work was supported in part by the Hong Kong RGC grant ECS 21212419, the Technological Breakthrough Project of Science, Technology and Innovation Commission of Shenzhen Municipality under Grants JSGG20201102162000001, InnoHK initiative, the Government of the HKSAR, Laboratory for AI-Powered Financial Technologies, the Hong Kong UGC Special Virtual Teaching and Learning (VTL) Grant 6430300, Shenzhen Science and Technology Program and Guangdong Basic and Applied Basic Research Foundation 2021A1515110717, and the Tencent AI Lab Rhino-Bird Gift Fund.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.
Xavier Carreras and Lluis Marquez. 2005. Introduction to the conll-2005 shared task: Semantic role labeling. In Proceedings of the ninth conference on computational natural language learning (CoNLL-2005), pages 152-164.
Guanhua Chen, Shuming Ma, Yun Chen, Li Dong, Dongdong Zhang, Jia Pan, Wenping Wang, and Furu Wei. 2021. Zero-shot cross-lingual transfer of neural machine translation with multilingual pretrained encoders. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 15–26, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Simone Conia, Andrea Bacciu, and Roberto Navigli. 2021. Unifying cross-lingual semantic role labeling with heterogeneous linguistic resources. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 338-351.
Simone Conia and Roberto Navigli. 2020. Bridging the gap in multilingual semantic role labeling: a language-agnostic approach. In Proceedings of the 28th International Conference on Computational Linguistics, pages 1396–1410.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale. In ACL.
Alexis Conneau and Guillaume Lample. 2019. Crosslingual language model pretraining. Advances in Neural Information Processing Systems, 32:7059-7069.
Angel Daza and Anette Frank. 2019. Translate and label! an encoder-decoder approach for cross-lingual semantic role labeling. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 603-615.
Angel Daza and Anette Frank. 2020. X-srl: A parallel cross-lingual semantic role labeling dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3904-3914.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Li Dong, Nan Yang, Wenhui Wang, Furu Wei, Xiaodong Liu, Yu Wang, Jianfeng Gao, Ming Zhou, and Hsiao-Wuen Hon. 2019. Unified language model pre-training for natural language understanding and generation. In Proceedings of the 33rd International Conference on Neural Information Processing Systems, pages 13063-13075.
Zi-Yi Dou and Graham Neubig. 2021. Word alignment by fine-tuning embeddings on parallel corpora. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2112-2128.
Ahmed Elgohary, Denis Peskov, and Jordan Boyd-Graber. 2019. Can you unpack that? learning to rewrite questions-in-context. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5918-5924, Hong Kong, China. Association for Computational Linguistics.
M Amin Farajian, António V Lopes, André FT Martins, Sameen Maruf, and Gholamreza Haffari. 2020. Findings of the wmt 2020 shared task on chat translation. In Proceedings of the Fifth Conference on Machine Translation, pages 65-75.
Hao Fei, Meishan Zhang, and Donghong Ji. 2020a. Cross-lingual semantic role labeling with high-quality translated training corpus. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7014-7026.
Hao Fei, Meishan Zhang, Fei Li, and Donghong Ji. 2020b. Cross-lingual semantic role labeling with model transfer. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 28:2427-2437.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural computation*, 9(8):1735-1780.
Mengzuo Huang, Feng Li, Wuhe Zou, Weidong Zhang, and Weidong Zhang. 2020. Sarg: A novel semi autoregressive generator for multi-turn incomplete utterance restoration. arXiv preprint arXiv:2008.01474.
Anne Lauscher, Vinit Ravishankar, Ivan Vulic, and Goran Glavaš. 2020. From zero to hero: On the limitations of zero-shot language transfer with multilingual transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4483-4499.
Zehui Lin, Xiao Pan, Mingxuan Wang, Xipeng Qiu, Jiangtao Feng, Hao Zhou, and Lei Li. 2020. Pretraining multilingual neural machine translation by leveraging alignment information. In Proceedings of the 2020 Conference on Empirical Methods in
Natural Language Processing (EMNLP), pages 2649-2663.
Qian Liu, Bei Chen, Jian-Guang Lou, Bin Zhou, and Dongmei Zhang. 2020a. Incomplete utterance rewriting as semantic segmentation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2846-2857.
Yinhan Liu, Jiatao Gu, Naman Goyal, Xian Li, Sergey Edunov, Marjan Ghazvininejad, Mike Lewis, and Luke Zettlemoyer. 2020b. Multilingual denoising pre-training for neural machine translation. Transactions of the Association for Computational Linguistics, 8:726-742.
Zihan Liu, Jamin Shin, Yan Xu, Genta Indra Winata, Peng Xu, Andrea Madotto, and Pascale Fung. 2019. Zero-shot cross-lingual dialogue systems with transferable latent variables. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1297-1303.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization. In International Conference on Learning Representations.
Shikib Mehri, Evgeniia Razumovskaia, Tiancheng Zhao, and Maxine Eskenazi. 2019. Pretraining methods for dialog context representation learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3836-3845, Florence, Italy. Association for Computational Linguistics.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32:8026-8037.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNLP and CoNLL-Shared Task, pages 1-40.
Shruti Rijhwani, Jiateng Xie, Graham Neubig, and Jaime Carbonell. 2019. Zero-shot neural transfer for cross-lingual entity linking. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 6924-6931.
Matiss Rikters, Ryokan Ri, Tong Li, and Toshiaki Nakazawa. 2019. Designing the business conversation corpus. In Proceedings of the 6th Workshop on Asian Translation, pages 54-61.
Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2020. Contrastive learning with hard negative samples. In International Conference on Learning Representations.
Shi-qi Shen, Yun Chen, Cheng Yang, Zhi-yuan Liu, Mao-song Sun, et al. 2018. Zero-shot cross-lingual neural headline generation. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 26(12):2319-2327.
Tom Sherborne and Mirella Lapata. 2021. Zero-shot cross-lingual semantic parsing. arXiv preprint arXiv:2104.07554.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255.
Hui Su, Xiaoyu Shen, Rongzhi Zhang, Fei Sun, Pengwei Hu, Cheng Niu, and Jie Zhou. 2019. Improving multi-turn dialogue modelling with utterance ReWriter. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 22-31, Florence, Italy. Association for Computational Linguistics.
Xiangpeng Wei, Rongxiang Weng, Yue Hu, Luxi Xing, Heng Yu, and Weihua Luo. 2020. On learning universal representations across languages. In International Conference on Learning Representations.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2020. Transformers: State-of-the-art natural language processing. In EMNLP (Demos).
Han Wu, Kun Xu, Linfeng Song, Lifeng Jin, Haisong Zhang, and Linqi Song. 2021a. Domain-adaptive pretraining methods for dialogue understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 665-669, Online. Association for Computational Linguistics.
Han Wu, Kun Xu, and Linqi Song. 2021b. CSAGN: Conversational structure aware graph network for conversational semantic role labeling. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2312-2317, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Wenquan Wu, Zhen Guo, Xiangyang Zhou, Hua Wu, Xiyuan Zhang, Rongzhong Lian, and Haifeng Wang. 2019. Proactive human-machine conversation with explicit conversation goal. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3794-3804, Florence, Italy. Association for Computational Linguistics.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
Kun Xu, Haochen Tan, Linfeng Song, Han Wu, Haisong Zhang, Linqi Song, and Dong Yu. 2020. Semantic role labeling guided multi-turn dialogue rewriter. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6632-6639.
Kun Xu, Han Wu, Linfeng Song, Haisong Zhang, Linqi Song, and Dong Yu. 2021. Conversational semantic role labeling. IEEE/ACM Transactions on Audio, Speech, and Language Processing.
Saizheng Zhang, Emily Dinan, Jack Urbanek, Arthur Szlam, Douwe Kiela, and Jason Weston. 2018. Personalizing dialogue agents: I have a dog, do you have pets too? In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2204-2213.
Zhuosheng Zhang and Hai Zhao. 2021. Structural pretraining for dialogue comprehension. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5134-5145, Online. Association for Computational Linguistics.
Kangyan Zhou, Shrimai Prabhumoye, and Alan W Black. 2018. A dataset for document grounded conversations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 708-713.
A Hard Parallel Sentence Identification Sampling
Following previous work (Robinson et al., 2020; Wei et al., 2020) which suggests that contrastive learning of representations benefits from hard negative samples, we also try to select hard negative samples for PSI task based on n-gram similarity and text perturbation. Specifically, for each sentence, we calculate its n-gram similarity scores to other sentences, where  $n = 1,2,3,4$ , and then we select the sentence with the highest score at each gram as the candidate sentence; additionally, we construct the corrupted sentence as the candidate by token deletion, token replacement and token order permutation. Finally, we sample from the candidate set created by n-gram similarity at  $40\%$  time and from the candidate set created by text perturbation at  $60\%$  time.
B Modified Transformer Encoder Layer
To overcome the information forgetting of hierarchical models, we attempt to modify the standard Transformer to better reserve the information from the previous layers. In specific, we try following variants: