SlowGuess/NAACL_2022_OCR huggingface dataset:
Based on our newly annotated multi-domain Chinese SRL data, we conduct preliminary experiments, aiming to provide benchmark results. Specifically, we present a simple basic SRL model and enhance the model with the contextualized word representations from BERT for further improvements. Besides, we also present a MTL framework to improve the SRL performance by learning from multiple heterogeneous datasets simultaneously (Conia et al., 2021).
In this work, we focus on the predicate-given setting, which means we do argument identification and classification according to the given predicates in one sentence.
Following previous works (Cai et al., 2018; Zhang et al., 2019), we treat the predicate-given SRL task as a word pair classification problem and try to find the predicate-argument structure  $\hat{\pmb{y}}$  with the highest score:
$$
\hat {\boldsymbol {y}} = \underset {\boldsymbol {y} \in \mathcal {Y} (\boldsymbol {x})} {\operatorname {a r g m a x}} \operatorname {s c o r e} (\boldsymbol {x}, \boldsymbol {y}) \tag {1}
$$
where  $\mathcal{Y}(\pmb{x}) = \{(i,j,r)|i\in \mathcal{P},1\leq j\leq n,r\in$ $\mathcal{R}\}$  represents the set of all possible predicateargument pairs.  $\mathcal{P}$  is the set of given predicates,  $n$  is the number of sentence, and  $\mathcal{L}$  is the semantic role label set, which contains 24 semantic role labels and an extra "None" label to indicate there is no semantic relationship between the given predicate and the  $j$  -th word.
5.1 Basic SRL Model
Inspired by previous works (Cai et al., 2018; Zhang et al., 2019), we build a basic SRL model that utilizes the biaffine attention mechanism (Dozat and Manning, 2017) to score each candidate predicateargument pair. Figure 4 shows the architecture of the basic model. During both training and evaluation, multiple predicates in the same sentence are handled simultaneously. First, the input sentence is encoded; then, scores between predicates and all other words are computed; finally, the roles of each predicate are determined via local classification.
Figure 4: The architecture of our basic SRL model.
The input vector is the concatenation of the pre-trained word embedding  $\mathbf{e}_i^{pre}$ , the randomly initialized word embedding  $\mathbf{e}_i^r$ , the character-based word representation  $\mathbf{r}_i^c$ , and the predicate indicator embedding  $\mathbf{e}_i^p$ .
$$
\mathbf {x} _ {i} = \mathbf {e} _ {i} ^ {p r e} \oplus \mathbf {e} _ {i} ^ {r} \oplus \mathbf {r} _ {i} ^ {c} \oplus \mathbf {e} _ {i} ^ {p}
$$
where  $\mathbf{r}_i^c$  is produced by CNN, and the Boolean predicate indicator is true only for words that are given predicates.
A three-layer BiLSTM is applied to obtain context-aware representation of each word, i.e.,  $\mathbf{h}_i$ .
Two separate MLPs are applied over  $\mathbf{h}_i$  to get two lower-dimensional representation  $\mathbf{h}_i^{pred}$  (as predicate) and  $\mathbf{h}_i^{arg}$  (as candidate argument).
Biaffines are used to compute scores of labels between a predicate and a word.
During training, we adopt the local cross-entropy loss. To obtain cross-domain results on the basic SRL model, we train the model on source domain data and make predictions on target domain data.
5.2 Enhancing with BERT
Recently proposed PLMs, such as BERT (Devlin et al., 2019), have shown the great power in learning and capturing contextualized representations and have proven to be beneficial in a variety of NLP tasks, such as information retrieval (Yang et al., 2019b), question answering (Yang et al., 2019a), and word segmentation (Huang et al., 2020). In this work, we extract the fixed contextualized representations from BERT for words and treat them as additional features to augment the input representation, i.e.,  $\mathbf{x}_i = \mathbf{e}_i^{pre} \oplus \mathbf{e}_i^r \oplus \mathbf{r}_i^c \oplus \mathbf{e}_i^p \oplus \mathbf{e}_i^{BERT}$ .
(#Pred / #Sent)	Source	PB	PC	ZX	LAW	MED	CPB2.0
Train	32,790 / 13,022	-	-	-	-	-	72,616 / 13,170
Dev	4,098 / 1,875	3,796 / 1,255	5,658 / 1,295	1,784 / 492	3,718 / 778	1,874 / 478	-
Test	4,101 / 2,077	7,521 / 2,498	11,416 / 2,595	4,107 / 1,083	7,438 / 2,035	3,750 / 1,414	-
Table 4: Statistics of MuCPAD and CPB2.0. "Pred" and "Sent" represent the number of predicates and sentences.
Figure 5: The framework of MTL.
5.3 Utilizing Heterogeneous Data with MTL
MTL is a commonly used method to improve the model performance by learning the underlying knowledge from multiple related tasks or datasets (Collobert and Weston, 2008; Guo et al., 2016a; Li et al., 2019a). In this work, we design a MTL framework to utilize heterogeneous SRL datasets to boost the SRL model performance.
As shown in Figure 5, we extend the basic SRL model to the MTL framework. Specifically, the SRL parsing on MuCPAD data and CPB2.0 data are considered as two separate tasks. They share the same word/predicate embeddings and BiLSTM parameters. Over the shared BiLSTMs, two separate MLPs and bioaffines are employed for MuC-PAD and CPB2.0 SRL parsing respectively.
Data. Our experiments mainly focus on zero-shot single-source domain adaptation, that is, we have labeled training data for the source domain, and do not have labeled training data for the target domain. Specifically, we use the News domain of MuCPAD as the source domain, and the other five domains as target domains. The data statistics for source and target domains are shown in Table 4. For the auxiliary data used in the MTL framework, we randomly select 13,170 sentences with 72,616 predicates from CPB2.0 (Xue, 2006b), which be
longs to the same newswire genre with the source domain data.
Evaluation metric. We adopt the standard precision  $\left(\frac{\# \text{Arg}_{\text{correct}}}{\# \text{Arg}_{\text{pred}}}\right)$ , recall  $\left(\frac{\# \text{Arg}_{\text{correct}}}{\# \text{Arg}_{\text{gold}}}\right)$ , and F1 score  $\left(\frac{2PR}{P + R}\right)$  for SRL evaluation.
Settings. We implement the basic SRL model and MTL framework with PyTorch $^6$  and mainly follow the hyperparameters of Cai et al. (2018), such as the dimensions of embeddings, learning rate, and dropout ratios. We use bert-base-chinese $^7$  to obtain contextualized representations for words, and the dimension of the BERT representations is 768. During training, early stopping is triggered if the peak performance in dev data does not increase in 50 consecutive iterations.
Results of the basic model. The first row of Table 5 presents the results in the source/target domain dev/test data using the basic SRL model trained on the source data.
First, it is obvious that the performance in all the five target domains drops dramatically compared with the results on source data, with the gap of more than  $18\%$  in F1. This indicates that the model trained on source data has a challenge in making reliable predictions on target domain data due to the distributional mismatch between different domains. Second, we find that the basic SRL model performs better on ZX and LAW compared with the other three target domains data, i.e., PB, PC, and MED. The probable reason is that ZX and LAW are novel and legal case, respectively, which are more canonical in text. Third, PB has the lowest F1 score in both dev and test. This can be explained by the fact that PB is non-canonical data from Taobao headline website. The dissimilarity between the source training data and PB target data causes the low performance.
Results with BERT. The second row of Table 5 shows the results of the baseline with BERT rep
$^{6}$ https://pytorch.org/
<sup>7</sup>https://huggingface.co/bert-base-chinese
Source	PB	PC	ZX	LAW	MED	AVG
dev	test	dev	test	dev	test	dev	test	dev	test	dev	test
Baseline	69.55	68.40	44.24	44.59	46.12	47.37	50.98	49.26	47.05	49.77	44.88	46.70	50.74
Baseline+BERT	80.25	79.46	64.28	66.23	66.77	67.14	71.65	71.77	70.47	73.28	66.06	67.97	70.44
Baseline+MTL	74.55	73.43	46.95	47.36	48.61	48.95	54.38	53.54	48.55	51.25	54.53	55.26	54.78
Baseline+MTL+BERT	81.05	80.85	64.35	65.27	67.75	68.13	72.44	71.72	70.53	73.79	66.38	69.07	70.94
Table 5: F1 scores of different models on MuCPAD. "AVG" is obtained by averaging the values of both dev and test in all domains.
resentations. We can see that the results of "Baseline+BERT" consistently increase by large margins compared with the corresponding baseline models without BERT (as shown in the first major row of table 5), demonstrating the great power of BERT in contextualized representation.
Results with heterogeneous CPB2.0. As shown in the third row of Table 5, benefiting from the additional semantic information provided by the auxiliary CPB2.0 data using the MTL framework, the SRL performance in all domains are improved compared with the baseline model. This indicates that the MTL framework is effective in capturing and learning the underlying common knowledge from heterogeneous data.
On the one hand, comparing the improvements brought by MTL in all domains, we find that MED data obtains the largest gains of  $9.65\% / 8.56\%$  F1 in dev/test, respectively. The main reason is that the MED data belongs to the same newswire domain as the auxiliary CPB2.0 data. On the other hand, the improvement in LAW is the smallest. This can be explained by the difference in label distribution between LAW and CPB2.0. For example, as mentioned in Section 4, the labels "time" and "loc" in LAW account for the largest proportion (13.95% and 6.14% respectively) compared with other domains. However, the proportions of "time" and "loc" in CPB2.0 data are only 6.10% and 3.40% respectively (about half of that in LAW). Therefore, CPB2.0 cannot provide much more valid information to increase the performance of these labels.
Results with BERT and heterogeneous CPB2.0. Finally, when utilizing both BERT representations and the heterogeneous CPB2.0 data on our baseline, the enhanced model gives the best or comparable results in 5 of the 6 domains, with an average increase of  $0.5\%$  F1, showing that the MTL framework is effective in utilizing heterogeneous data and can complement the information obtained
from BERT representations.
This paper presents a multi-domain Chinese predicate-argument dataset, named MuCPAD, which consists of 30,897 sentences with 92,051 predicates and covers 6 different domains. In particular, we adopt a frame-free annotation methodology, which does not require high-level linguistic background for defining frames for large amounts of new predicates or new senses in multi-domain data. Besides, considering that omission of content words is ubiquitous in Chinese, we explicitly annotate omitted core arguments with two special designed labels "hidden-subject" and "hidden-object" for better semantic understanding. To ensure annotation quality, we adopt strict double annotation and ask a third expert to handle annotation inconsistency. We also perform analysis on MuCPAD from different perspectives. Finally, we conduct preliminary cross-domain experiments and analysis on MuCPAD.
The authors would like to thank the anonymous reviewers for the helpful comments. We are greatly grateful to all our annotators for their hard work in data annotation. This work was supported by the National Natural Science Foundation of China (Grant No. 62176173, 61876116), and a project funded by the Priority Academic Program Development of Jiangsu Higher Education Institutions.
Collin F Baker, Charles J Fillmore, and John B Lowe. 1998. The berkeley framenet project. In Proceedings of COLING-ACL, pages 86-90.
Emanuele Bastianelli, Giuseppe Castellucci, Danilo Croce, and Roberto Basili. 2013. Textual inference
and meaning representation in human robot interaction. In Proceedings of JSSP, pages 65-69.
Jiaxun Cai, Shexia He, Zuchao Li, and Hai Zhao. 2018. A full end-to-end semantic role labeler, syntactic-agnostic over syntactic-aware? In Proceedings of COLING, pages 2753-2765.
Kevin Clark, Minh-Thang Luong, Christopher D. Manning, and Quoc V. Le. 2018. Semi-supervised sequence modeling with cross-view training. In Proceedings of EMNLP, pages 1914-1925.
Ronan Collobert and Jason Weston. 2008. A unified architecture for natural language processing: deep neural networks with multitask learning. In Proceedings of ICML, pages 160-167.
Simone Conia, Andrea Bacciu, and Roberto Navigli. 2021. Unifying cross-lingual semantic role labeling with heterogeneous linguistic resources. In Proceedings of NAACL-HLT, pages 338-351.
Simone Conia and Roberto Navigli. 2020. Bridging the gap in multilingual semantic role labeling: a language-agnostic approach. In Proceedings of COLING, pages 1396-1410.
Hal Daumé III. 2007. Frustratingly easy domain adaptation. In Proceedings of ACL, pages 256-263.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT, pages 4171-4186.
Timothy Dozat and Christopher D. Manning. 2017. Deep bioaffine attention for neural dependency parsing. In Proceedings of ICLR.
Magali Sanches Duran and Sandra Maria Aluísio. 2011. Propbank-br: a brazilian portuguese corpus annotated with semantic role labels. In Proceedings of STIL.
Magali Sanches Duran and Sandra Maria Aluisio. 2012. "Propbank-br: a Brazilian treebank annotated with semantic role labels". In Proceedings of LREC, pages 1862-1867.
Hagen Fürstenau and Mirella Lapata. 2009. Semi-supervised semantic role labeling. In Proceedings of EACL, pages 220-228.
Yaroslav Ganin and Victor Lempitsky. 2015. Unsupervised domain adaptation by backpropagation. In Proceedings of ICML, pages 1180-1189.
Jiang Guo, Wanxiang Che, Haifeng Wang, and Ting Liu. 2016a. A universal framework for inductive transfer parsing across multi-typed treebanks. In Proceedings of COLING, pages 12-22.
Jiang Guo, Wanxiang Che, Haifeng Wang, Ting Liu, and Jun Xu. 2016b. A unified architecture for semantic role labeling and relation classification. In Proceedings of COLING, pages 1264-1274.
L. Guo, L. I. Zhenghua, X. Peng, and M. Zhang. 2018. Annotation guideline of chinese dependency treebank from multi-domain and multi-source texts. Journal of Chinese Information Processing, 32(10):28-35.
Jan Hajic, Massimiliano Ciaramita, Richard Johansson, Daisuke Kawahara, M Antônia Martí, Lluis Marquez, Adam Meyers, Joakim Nivre, Sebastian Padó, Jan Štepánek, et al. 2009. The CoNLL-2009 shared task: Syntactic and semantic dependencies in multiple languages. In Proceedings of CoNLL, pages 1-18.
Katri Haverinen, Jenna Kanerva, Samuel Kohonen, Anna Missilä, Stina Ojala, Timo Viljanen, Veronika Laippala, and Filip Ginter. 2015. The finnish proposition bank. Language Resources and Evaluation, 49:907-926.
Shexia He, Zuchao Li, Hai Zhao, and Hongxiao Bai. 2018. Syntax for semantic role labeling, to be, or not to be. In Proceedings of ACL, pages 2061-2071.
Weipeng Huang, Xingyi Cheng, Kunlong Chen, Taifeng Wang, and Wei Chu. 2020. Towards fast and accurate neural Chinese word segmentation with multi-criteria learning. In Proceedings of COLING, pages 2062-2072.
Chen Jia, Liang Xiao, and Yue Zhang. 2019. Cross-domain NER using cross-domain language modeling. In Proceedings of ACL, pages 2464-2474.
Jing Jiang and Chengxiang Zhai. 2007. Instance weighting for domain adaptation in nlp. In Proceedings of ACL, pages 264-271.
Young-Bum Kim, Karl Stratos, and Dongchan Kim. 2017. Adversarial adaptation of synthetic or stale data. In Proceedings of ACL, pages 1297-1307.
Young-Bum Kim, Karl Stratos, and Ruhi Sarikaya. 2016. Frustratingly easy neural domain adaptation. In Proceedings of COLING, pages 387-396.
Paul Kingsbury and Martha Palmer. 2002. From treebank to propbank. pages 1989-1993.
Zhenghua Li, Xue Peng, Min Zhang, Rui Wang, and Luo Si. 2019a. Semi-supervised domain adaptation for dependency parsing. In Proceedings of ACL, pages 2386-2395.
Zuchao Li, Junru Zhou, Hai Zhao, and Rui Wang. 2019b. Cross-domain transfer learning for dependency parsing. In Proceedings of NLPCC, pages 835-844.
Diego Marcheggiani, Anton Frolov, and Ivan Titov. 2017. A simple and accurate syntax-agnostic neural model for dependency-based semantic role labeling. In Proceedings of CoNLL, pages 411-420.
Mitch Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The Penn Treebank: Annotating predicate argument structure. In Proceedings of HLT, pages 114-119.
Mitchell Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of english: The penn treebank. Computational Linguistics, 19:313-330.
Lluis Marquez, Xavier Carreras, Kenneth C Litkowski, and Suzanne Stevenson. 2008. Semantic role labeling: an introduction to the special issue. Computational linguistics, 34(2):145-159.
Adam Meyers, Ruth Reeves, Catherine Macleod, Rachel Szekely, Veronika Zielinska, Brian Young, and Ralph Grishman. 2004. Annotating noun argument structure for nombank. In Proceedings of LREC, volume 4, pages 803-806.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, Rishita Anubhai, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. arXiv preprint arXiv:2101.05779.
Merin Paul and Sangeetha Jamal. 2015. An improved srl based plagiarism detection technique using sentence ranking. Procedia Computer Science, 46:223-230.
Xue Peng, Zhenghua Li, Min Zhang, Rui Wang, Yue Zhang, and Luo Si. 2019. Overview of the nlpcc 2019 shared task: Cross-domain dependency parsing. In Proceedings of NLPCC, pages 760-771.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL-2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Proceedings of EMNLP-CoNLL, pages 1-40.
Alan Ramponi and Barbara Plank. 2020. Neural unsupervised domain adaptation in nlp-a survey. In Proceedings of COLING, pages 6838-6855.
Gözde Gül Sahin and Esref Adali. 2018. Annotation of semantic roles for the turkish proposition bank. Language Resources and Evaluation, 52:673-706.
Chen Shi, Shujie Liu, Shuo Ren, Shi Feng, Mu Li, Ming Zhou, Xu Sun, and Houfeng Wang. 2016. Knowledge-based semantic embedding for machine translation. In Proceedings of ACL, pages 2245-2254.
Peng Shi and Jimmy Lin. 2019. Simple bert models for relation extraction and semantic role labeling. arXiv preprint arXiv:1904.05255.
Zhen Wang, Tingsong Jiang, Baobao Chang, and Zhi-fang Sui. 2015. Chinese semantic role labeling with bidirectional recurrent neural networks. In Proceedings of EMNLP, pages 1626-1631.
Qiaolin Xia, Lei Sha, Baobao Chang, and Zhifang Sui. 2017. A progressive learning approach to Chinese srl using heterogeneous data. In Proceedings of ACL, pages 2069-2077.
Qingrong Xia, Rui Wang, Zhenghua Li, Yue Zhang, and Min Zhang. 2020. Semantic role labeling with heterogeneous syntactic knowledge. In Proceedings of COLING, pages 2979-2990.
Nianwen Xue. 2006a. Annotating the predicateargument structure of Chinese nominalizations. In Proceedings of LREC, pages 1382-1387.
Nianwen Xue. 2006b. Semantic role labeling of nominalized predicates in Chinese. In Proceedings of NAACL-HLT, pages 431-438.
Nianwen Xue and Martha Palmer. 2005. Automatic semantic role labeling for chinese verbs. In Proceedings of IJCAI, pages 1160-1165.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019a. End-to-end open-domain question answering with bertserini. In Proceedings of NAACL-HLT, pages 72-77.
Wei Yang, Haotian Zhang, and Jimmy Lin. 2019b. Simple applications of bert for ad hoc document retrieval. volume abs/1903.10972.
Yue Zhang, Rui Wang, and Luo Si. 2019. Syntax-enhanced self-attention-based semantic role labeling. In Proceedings of EMNLP-IJCNLP, pages 616-626.
Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuaijiang Zhang, Xi Zhou, and Xiang Zhou. 2020. Semantics-aware BERT for language understanding. In Proceedings of AAAI, pages 9628-9635.
Han Zhao, Shanghang Zhang, Guanhang Wu, José MF Moura, Joao P Costeira, and Geoffrey J Gordon. 2018. Adversarial multiple source domain adaptation. In Proceedings of ICLR, pages 8559-8570.