SlowGuess/NAACL_2022_OCR huggingface dataset:
Han Wu $^{1,2}$ , Haochen Tan $^{1,2}$ , Kun Xu $^{3\dagger}$ , Shuqi Liu $^{1,2}$ , Lianwei Wu $^{4}$ , Linqi Song $^{1,2*}$
<sup>1</sup>City University of Hong Kong Shenzhen Research Institute
$^{2}$ Department of Computer Science, City University of Hong Kong,  $^{3}$ Huawei
$^{4}$ Laboratory for Integrated Aero-Space-Ground-Ocean Big Data Application Technology,
School of Computer Science, Northwestern Polytechnical University
{hanwu32-c, haochetan-2, shuqiliu-4}@my.cityu.edu.hk
syxu828@gmail.com, wlw@nwpu.edu.cn
linqi song@cityu.edu.hk
While conversational semantic role labeling (CSRL) has shown its usefulness on Chinese conversational tasks, it is still under-explored in non-Chinese languages due to the lack of multilingual CSRL annotations for the parser training. To avoid expensive data collection and error-propagation of translation-based methods, we present a simple but effective approach to perform zero-shot cross-lingual CSRL. Our model implicitly learns language-agnostic, conversational structure-aware and semantically rich representations with the hierarchical encoders and elaborately designed pre-training objectives. Experimental results show that our model outperforms all baselines by large margins on two newly collected English CSRL test sets. More importantly, we confirm the usefulness of CSRL to non-Chinese conversational tasks such as the question-in-context rewriting task in English and the multi-turn dialogue response generation tasks in English, German and Japanese by incorporating the CSRL information into the downstream conversation-based models. We believe this finding is significant and will facilitate the research of non-Chinese dialogue tasks which suffer the problems of ellipsis and anaphora.
Conversational Semantic Role Labeling (CSRL) (Xu et al., 2021) is a recently proposed dialogue understanding task, which aims to extract predicate-argument pairs from the entire conversation. Figure 1 illustrates a CSRL example where a CSRL parser is required to identify "《泰坦尼克号》(Titanic)" as the ARG1 argument of the predicate "看过 (watched)" and the ARG0 argument of the predicate "是 (is)". We can see that in the original conversation, "《泰坦尼克号》(Titanic)" is omitted in the second turn and referred as "这 (this)" in the
Figure 1: An example of CSRL parsing.
last turn. By recovering the dropped and referred components in conversation, CSRL has shown its usefulness to a set of Chinese dialogue tasks, including multi-turn dialogue rewriting (Su et al., 2019) and response generation (Wu et al., 2019). However, there remains a paucity of evidence on its effectiveness towards non-Chinese languages owing to the lack of multilingual CSRL models. To adapt a model into new languages, previous solutions can be divided into three categories: 1) manually annotating a new dataset in the target language (Daza and Frank, 2020) 2) borrowing machine translation and word alignment techniques to transfer the dataset from the source language into the target language (Daza and Frank, 2019; Fei et al., 2020a) 3) zero-shot transfer learning with multilingual pre-trained language model (Rijhwani et al., 2019; Sherborne and Lapata, 2021). Due to the fact that manually collecting annotations is costly and translation-based methods might introduce translation or word alignment errors, zero-shot cross-lingual transfer learning is more practical to the NLP community.
Recent works have witnessed prominent performances of multilingual pre-trained language models (PrLMs) (Devlin et al., 2019; Conneau et al., 2020) on cross-lingual tasks, including machine translation (Lin et al., 2020; Chen et al., 2021), semantic role labeling (SRL) (Conia and Navigli, 2020; Conia et al., 2021) and semantic parsing (Fei et al., 2020b; Sherborne and Lapata, 2021). How
† Work done when Kun Xu was in Tencent AI Lab.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 269 - 281
ever, cross-lingual CSRL, as a combination of three challenging tasks (i.e., cross-lingual task, dialogue task and SRL task), suffers three outstanding difficulties: 1) latent space alignment - how to map word representations of different languages into an overlapping space; 2) conversation structure encoding - how to capture high-level dialogue features such as speaker dependency and temporal dependency; and 3) semantic arguments identification - how to highlight the relations between the predicate and its arguments, wherein PrLMs can only partially encode multilingual inputs to an overlapping vector space. Although there are some success that can separately achieve structural conversation encoding (Mehri et al., 2019; Zhang and Zhao, 2021) and semantic arguments identification (Wu et al., 2021a), a unified method for jointly solving these problems is still under-explored, especially in a cross-lingual scenario.
In this work, we summarize our contributions as follows: (1) We propose a simple but effective model which consists of three modules, namely cross-lingual language model (CLM), structure-aware conversation encoder (SA-Encoder) and predicate-argument encoder (PA-Encoder), and five well-designed pre-training objectives. Our model implicitly learns language-agnostic, conversational structure-aware and semantically rich representations to perform zero-shot cross-lingual CSRL. (2) Experiments show that our method achieves impressive cross-lingual performance on the language pair  $(\mathrm{Zh}\rightarrow \mathrm{En})$ , and outperforms all baselines on the two newly collected English CSRL test sets. (3) We confirm the usefulness of CSRL to the question-in-context rewriting task in English and multi-turn response generation tasks in English, German and Japanese. We believe this finding is important and will facilitate the research of non-Chinese dialogue tasks that suffer from ellipsis and anaphora. (4) We release our code, the new annotated English CSRL test sets and checkpoints of our best models to facilitate the further research at https://github.com/hahahawu/Zero-Shot-XCSRL.
Zero-shot cross-lingual transfer learning. Recently, thanks to the rapid development of multilingual pre-trained language models such as multilingual BERT (Devlin et al., 2019) and XLM-R (Connseau et al., 2020), a number of approaches have been proposed for zero-shot cross-lingual transfer
learning on various downstream tasks, including semantic parsing (Sherborne and Lapata, 2021), natural language generation (Shen et al., 2018) and understanding (Liu et al., 2019; Lauscher et al., 2020). In this work, we claim our method is zero-shot because no non-Chinese CSRL annotations are seen during the CSRL training stage. For decoding, we directly use the cross-lingual CSRL model trained on Chinese CSRL data to analyze conversations in other languages. To our best knowledge, our work is the first step to cross-lingual CSRL.
Conversational semantic role labeling. While ellipsis and anaphora frequently occur in dialogues, Xu et al. (2021) observed that most of the dropped or referred components can be found in dialogue histories. Following this observation, they proposed conversational semantic role labeling (CSRL) which required the model to find predicate-argument structures over the entire conversation instead of a single sentence. In this way, when analyzing a predicate in the latest utterance, a CSRL model needs to consider both the current turn and previous turns to search potential arguments, and thus might recover the omitted components. Furthermore, Xu et al. (2020, 2021) also confirmed the usefulness of CSRL to Chinese dialogue tasks by applying CSRL information into downstream dialogue tasks. However, there are still two main problems to be solved for CSRL task: (1) the performance of current state-of-the-art CSRL model (Xu et al., 2021) is still far from satisfactory due to the lack of high-level conversational and semantic features modeling; (2) the usefulness of CSRL to conversational tasks in non-Chinese languages has not been confirmed yet due to the lack of cross-lingual CSRL models. In this work, we primarily focus on the latter problem and propose a simple but effective model to perform cross-lingual CSRL. We would like to distinguish our work from the work (Wu et al., 2021b) which purely focuses on improving the monolingual CSRL performance where they try to model predicate-aware representations. This solution could benefit to monolingual CSRL task, but hurt the cross-lingual performance, because the relative positions of the predicates may differ from language to language.
Following Xu et al. (2021), we solve the CSRL task as a sequence labeling problem. Formally, given a dialogue  $C = \{u_1, u_2, \dots, u_N\}$  of  $N$  utter
Figure 2: Overall model architecture.
ances, where  $u_{i} = \{w_{1}^{i}, w_{2}^{i}, \dots, w_{|u_{i}|}^{i}\}$  consisting of a sequence of words, and a predicate indicator  $p = (p_{1}^{1}, \dots, p_{k}^{i}, \dots, p_{|u_{N}|}^{N})$  used to identify whether a word is the predicate or not, our goal is to assign each word with a semantic role label  $l \in L$  where  $L$  is the label set. We also incorporate speaker role indicator  $r$  to distinguish speakers, and dialogue turn indicator  $t$  to distinguish dialogue turns.
Cross-lingual Language Model (CLM) We concatenate all utterances into a sequence and then use a pre-trained cross-lingual language model such as XLM-R (Conneau et al., 2020) or mBERT (Devlin et al., 2019) to capture the syntactic and semantic characteristics. Following Conia et al. (2021), we obtain word representations  $\mathbf{e} \in \mathbb{R}^{|S| \times d}$  by concatenating the hidden states of the four top-most layers of the language model, where  $|S|$  is the sequence length and  $d$  is the dimension of the hidden state.
Structure-aware Conversation Encoder (SCEncoder) Different from standard SRL(Carreras and Marquez, 2005), CSRL requires the models to find arguments from not only the current turn, but also previous turns, thus bringing more challenges of dialogue modeling. To address this problem, we propose a universal structure-aware conversation encoder which comprises of two parts, i.e., word-level encoder and utterance-level encoder. Formally, with the speaker role embedding  $\pmb{r} \in \mathbb{R}^{|S| \times d}$  and dialogue turn embedding  $\pmb{t} \in \mathbb{R}^{|S| \times d}$ , the word-level encoder computes a sequence of timestep en
codings  $s\in \mathbb{R}^{|S|\times d}$  as follows:
$$
\boldsymbol {s} _ {(i, k)} ^ {j} = \left\{ \begin{array}{l l} \boldsymbol {e} _ {k} ^ {i} \oplus \boldsymbol {t} _ {k} ^ {i} \oplus \boldsymbol {r} _ {k} ^ {i} & \text {i f} j = 0 \\ \boldsymbol {s} _ {(i, k)} ^ {j - 1} \oplus \mathrm {M T R A N S} ^ {j} \left(\boldsymbol {s} _ {(i, k)} ^ {j - 1}\right) & \text {o t h e r w i s e} \end{array} \right. \tag {1}
$$
where  $s_{(i,k)}^j$  is the timestep encoding of  $k$ -th token in  $i$ -th utterance from  $j$ -th word-level encoder layer while  $j \in (0, \dots, N_1)$ ,  $\oplus$  represents vector concatenation, and MTRANS is the Modified Transformer encoder layer. Concretely, we replace the [Add] operation in the first residual connection layer with [Concat] because we argue that concatenation is a superior approach to preserve the information from previous layers<sup>1</sup>.
We obtain utterance representations  $\pmb{u} \in \mathbb{R}^{N \times d}$  by max-pooling over words in the same utterance. Then we pass the resulting utterance representations  $\pmb{u}$  through a stack of Bi-LSTM (Hochreiter and Schmidhuber, 1997) layers to obtain the sequentially encoded utterance representations  $\pmb{u}' \in \mathbb{R}^{N \times d}$ . Finally, we combine the utterance-level feature  $\pmb{u}'$  with the word-level feature  $\pmb{s}$  to obtain structure-aware dialogue context representations  $\pmb{g} \in \mathbb{R}^{|S| \times d}$  as follows:
$$
\boldsymbol {g} _ {k} ^ {i} = \operatorname {S w i s h} \left(\mathbf {W} ^ {g} \left[ \boldsymbol {s} _ {(i, k)} ^ {N _ {1}} \oplus \boldsymbol {u} _ {i} ^ {\prime} \right] + \mathbf {b} ^ {g}\right) \tag {2}
$$
where  $\operatorname{Swish}(x) = x \cdot \operatorname{sigmoid}(x)$  is a non-linear activation function,  $s_{i,k}^{N_1}$  is the encoding of  $k$ -th token in  $i$ -th utterance from the last layer of the word-level encoder.  $\mathbf{W}^g$  and  $\mathbf{b}^g$  are trainable parameters.
Predicate-Argument Encoder (PA-Encoder) We introduce the third module (i.e., predicateargument encoder) whose goal is to capture the
1 More details about MTRANS in Appendix B.
Figure 3: Illustration of pre-training objectives for latent space alignment.
Figure 5: Illustration of pre-training objectives for semantic argument identification.
(a) An example of SPI pre-training.
Figure 4: Illustration of pre-training objectives for conversation structure encoding.
Fine-tune the module.
relations between each predicate-argument couple that appears in the conversation. Similar with the word-level encoder, we use a stack of MTRANS layers to implement this encoder. Formally, with the predicate embedding  $\pmb{p} \in \mathbb{R}^{|S| \times d}$ , the model calculates the predicate-specific argument encodings  $\pmb{a} \in \mathbb{R}^{|S| \times d}$  as follows:
$$
\boldsymbol {a} _ {(i, k)} ^ {j} = \left\{ \begin{array}{l l} \boldsymbol {g} _ {k} ^ {i} \oplus \boldsymbol {p} _ {k} ^ {i} & \text {i f} j = 0 \\ \boldsymbol {a} _ {(i, k)} ^ {j - 1} \oplus \mathrm {M T R A N S} ^ {j} (\boldsymbol {a} _ {(i, k)} ^ {j - 1}) & \text {o t h e r w i s e} \end{array} \right. \tag {3}
$$
where  $\pmb{a}_{(i,k)}^{j}$  is the argument encoding of  $k$ -th token in  $i$ -th utterance from  $j$ -th encoder layer while  $j \in (0, \dots, N_2)$ . Finally, we obtain the semantic role encoding  $l$  using the resulting argument encodings from the last layer of the predicate-argument encoder:
$$
\boldsymbol {l} _ {k} ^ {i} = \operatorname {S w i s h} \left(\mathbf {W} ^ {l} \boldsymbol {a} _ {(i, k)} ^ {N _ {2}} + \mathbf {b} ^ {l}\right) \tag {4}
$$
In particular, our proposed model is mostly language-agnostic since we do not explicitly introduce any language-specific knowledge such as word order, part-of-speech tags or dependent relations, and only introduce the predicate indicator that might contain some language-specific information in the semantic module, which would not affect latent space alignment and dialogue modeling.
3.2 Pre-training Objectives
Besides the universal model, we also elaborately design five pre-training objectives to model task-specific but language-agnostic features for better cross-lingual performance. In this section, we divide our pre-training objectives into three groups according to the challenges to be solved.
Latent space alignment In cross-lingual language module, we use mBERT or XLM-R to align the latent space of different languages. Although mBERT and XLM-R have exhibited good alignment ability, even both of them are trained with unpaired data, we may further improve it when we have access to parallel data.
We first use translation language model (TLM) (Conneau and Lample, 2019) to learn word-level alignment ability. Concretely, we concatenate parallel sentences as a single consecutive token sequence with special tokens separating them and then perform masked language modeling (MLM) (Devlin et al., 2019) on the concatenated sequence. Besides, we also attempt to improve sentence-level alignment ability using hard parallel sentence identification (HPSI). Specifically, we select a pair of parallel or non-parallel sentences from the training set with equal probability. Then the model is required to predict whether the sampled sentence pair is parallel or not. Different from the standard PSI (Dou and Neubig, 2021), we sample the non-parallel sentence upon the n-gram similarity or construct it by text perturbation (details in Appendix A) instead of in a random manner. Figure 3 illustrates the workflows of TLM and HPSI. We pre-train the CLM using the combination of TLM and HPSI, finally achieving latent space alignment.
Conversation structure encoding Although there are a number of pre-training objectives proposed to learn dialogue context representations (Mehri et al., 2019) and structural representations (Zhang and Zhao, 2021), we tend to explicitly model speaker dependency and temporal depen
Dataset	language	#dialogue	#utterance	#predicate	#tokens per utterance	cross ratio
DuConv	ZH	3,000	27,198	33,673	10.56	21.89%
Persona-Chat	EN	50	2,669	477	17.96	17.74%
CMU-DoG	EN	50	3,217	450	12.57	7.41%
Table 1: Statistics of the annotations on DuConv, NewsDialog and PersonalDialog.
dency in the conversation, both of which have been proven to be critical to CSRL task (Xu et al., 2021).
We first propose speaker role identification (SPI) to learn speaker dependency in the conversation. Specifically, we randomly sample  $K_{1}$  % utterances and replace their speaker indicators with special mask tags. To make the task harder and effective, we split the utterances into clauses if only two interlocutors utter in turn in a conversation. The goal of SPI is to predict the masked speaker roles according to the corrupted speaker indicators and context. Secondly, we borrow utterance order permutation (UOR) to encourage the model to be aware of temporal connections among utterances in the context. Concretely, given a set of utterances, we randomly shuffle the last  $K_{2}$  % utterances and require the model to organize them into a coherent context. Figure 4 illustrates the workflows of SPI and UOR. We pre-train the SC-Encoder using the combination of SPI and UOR.
Semantic arguments identification The core of all SRL-related tasks is to recognize the predicate-argument pairs from the input. Therefore, we propose semantic arguments identification (SAI) objective to strengthen the correlations between the predicate and its arguments with the help of external standard SRL corpus, i.e., CoNLL-2012. Specifically, for each SRL sample, we only focus on those arguments, including ARG0-4, ARG-LOC, ARG-TMP and ARG-PRP, all of which are defined in both SRL and CSRL tasks. The model is encouraged to find the textual spans of these arguments with the given predicate. We believe this objective would benefit to boundary detection, especially for location and temporal arguments. Figure 5 illustrates the workflow of SAI. We drop the SCEncoder to fit in standard SRL samples which do not have any conversational characteristics.
Hierarchical Pre-training The pre-training is hierarchically conducted according to different modules, and the pre-training of the upper module is based on the pre-trained lower modules. Specifi
cally, we first train CLM module with TLM and HPSI; then we train SC-Encoder with SPI and UOR while keeping the weights of pre-trained CLM module unchanged; finally we train PA-Encoder with SAI while freezing the weights of pre-trained CLM and SC-Encoder modules. Hopefully, we expect that each module could acquire different knowledge with specific pre-training objectives.
CSRL training We initialize the specific modules, including CLM, SC-Encoder and PA-Encoder, from the pre-trained checkpoints. The CSRL model is trained only using Chinese CSRL annotations and no additional data is introduced during the CSRL training stage. We train our model to minimize the cross-entropy error for a training sample with label  $y$  based on the semantic role encoding  $l$ ,
$$
p = \operatorname {s o f t m a x} \left(\boldsymbol {l} _ {t}\right) \quad \mathcal {L} _ {C S R L} = - \sum_ {l = 1} ^ {L} y \log p \tag {5}
$$
CSRL data We use the same split as Xu et al. (2021) where DuConv annotations are splitted into  $80\% / 10\% / 10\%$  as train/dev/in-domain test set. Furthermore, we manually collect two CSRL test sets for cross-lingual evaluation based on PersonaChat(Zhang et al., 2018) and CMU-DoG(Zhou et al., 2018), both of which are English conversation datasets. The CSRL data annotation is difficult because it needs great expertise in SRL and dialogue. So we only explore cross-lingual CSRL on Chinese  $\rightarrow$  English  $(\mathrm{Zh} \rightarrow \mathrm{En})$  here, and we leave other languages for future work.
Following the instructions in Xu et al. (2021), we manually collect two out-of-domain CSRL test sets based on English dialogue datasets Persona-Chat and CMU-DoG. Specifically, we also annotate the arguments ARG0-4, ARG-TMP, ARG-LOC and ARG-PRP and require that the labeled arguments can only appear in the current turn or the previous turns. We employ three annotators who have studied Chinese CSRL annotations for a period time before this annotation. The first two annotators are
Method	Trainable parameters	DuConv	Persona-Chat	CMU-DoG
F1all	F1cross	F1intra	F1all	F1cross	F1intra	F1all	F1cross	F1intra
SimpleBERT	117 M	86.54	81.62	87.02	-	-	-	-	-	-
CSRL-BERT	147 M	88.46	81.94	89.46	-	-	-	-	-	-
CSAGN	163 M	89.47	84.57	90.15	-	-	-	-	-	-
SimpleXLMR	292 M	84.75	63.44	85.12	62.96	14.29	63.03	50.54	14.29	58.50
CSRL-XLMR	320 M	88.03	78.12	89.33	63.18	18.71	65.05	53.84	34.20	59.78
CSAGN-XLMR	338 M	88.52	82.45	89.98	63.02	17.82	64.97	52.73	30.11	58.91
Translation-test	-	-	-	-	63.49	13.90	66.67	47.91	27.44	50.92
Translation-train	-	-	-	-	60.12	9.67	62.50	44.27	25.40	47.87
Fine-tune all parameters
OursmBERT	272 M	87.20	81.14	88.11	58.38	9.39	61.77	48.13	20.92	52.91
OursXLM-R	372 M	88.35	83.39	89.21	67.29	24.29	70.61	61.74	60.32	62.67
OursXLM-R + pre-train	372 M	88.60	84.10	89.24	67.23	25.43	69.89	59.24	58.94	60.89
Freeze parameters of the language model
OursmBERT	180 M	87.08	81.46	87.98	59.04	11.23	62.13	48.87	21.78	53.54
OursXLM-R	180 M	88.30	83.38	89.17	65.57	24.11	68.51	59.60	56.16	60.78
OursXLM-R + pre-train	180 M	88.60	83.72	89.27	66.75	24.13	69.44	58.45	58.92	58.82
Table 2: Evaluation results on the DuConv, Persona-Chat and CMU-DoG datasets.
required to label all cases and any disagreements between them are solved by the third annotator. The statistics of the datasets are listed in Table 1.
Pre-training data For TLM and HPSI objectives which requires parallel data to enhance alignment ability, we choose IWSLT'14 English  $\leftrightarrow$  Chinese (En  $\leftrightarrow$  Zh) translations. For SPI and UOR objectives whose goal is to model high-level conversational features, we select samples from Chinese conversation dataset (i.e., DuConv) and English conversation datasets (i.e., Persona-Chat and CMU-DoG) with equal probability. For SAI, we borrow the Chinese and English SRL annotations from CoNLL-2012(Pradhan et al., 2012).
We stress that by keeping the sampling balance of Chinese and English data for every pre-training objective and sharing all parameters across the languages, our model would capture task-specific but language-agnostic features.
Following previous work(Xu et al., 2021), we evaluate our system on micro-average  $\mathrm{F1}_{all}$ ,  $\mathrm{F1}_{cross}$  and  $\mathrm{F1}_{intra}$  over the (predicate, argument, label) tuples, wherein we calculate  $\mathrm{F1}_{cross}$  and  $\mathrm{F1}_{intra}$  over the arguments in the different, or same turn as the predicate. We refer these two types of arguments as cross-arguments and intra-arguments. For language in-domain evaluation, we compare to SimpleBERT (Shi and Lin, 2019), CSRL-BERT (Xu et al., 2021) and CSAGN (Wu et al., 2021b), all
of which employ the Chinese pre-trained language model as the backbone. For cross-lingual evaluation, we compare to SimpleXLMR, CSRL-XLMR and CSAGN-XLMR by simply replacing the BERT backbones of those models with XLM-R. Additionally, we also compare to the back-translation baselines, i.e., Translate-test and Translate-train. Specifically, Translate-test means that the English test data is translated and projected to Chinese annotations using Google Translate (Wu et al., 2016) and the state-of-the-art word alignment toolkit Awesome-align(Dou and Neubig, 2021). Similarly, Translate-train means the Chinese training data is translated and projected to English annotations for training. We feed the translated samples into CSAGN/CSAGN-XLMR to obtain the back-translation results.