SlowGuess/NAACL_2022_OCR huggingface dataset:
Yahui Liu, Haoping Yang, Chen Gong*, Qingrong Xia, Zhenghua Li, Min Zhang Institute of Artificial Intelligence, School of Computer Science and Technology, Soochow University, China
{yahuiliu.nlp, hpyang3, gongchen.nlp}@foxmail.com,  
kirosummer.nlp@gmail.com,  
{zhli13, minzhang}@suda.edu.cn
During the past decade, neural network models have made tremendous progress on in-domain semantic role labeling (SRL). However, performance drops dramatically under the out-of-domain setting. In order to facilitate research on cross-domain SRL, this paper presents MuCPAD, a multi-domain Chinese predicate-argument dataset, which consists of 30,897 sentences and 92,051 predicates from six different domains. MuCPAD exhibits three important features. 1) Based on a frame-free annotation methodology, we avoid writing complex frames for new predicates. 2) We explicitly annotate omitted core arguments to recover more complete semantic structure, considering that omission of content words is ubiquitous in multi-domain Chinese texts. 3) We compile 53 pages of annotation guidelines and adopt strict double annotation for improving data quality. This paper describes in detail the annotation methodology and annotation process of MuCPAD, and presents in-depth data analysis. We also give benchmark results on cross-domain SRL based on MuCPAD.
As a fundamental NLP task, semantic role labeling (SRL), also known as shallow semantic parsing, aims to capture the major semantic information of a sentence based on predicate-argument structure. Basically, SRL tries to answer "who did what to whom where and when" (Márquez et al., 2008). Previous works have shown that SRL can help various downstream tasks, including information extraction (Bastianelli et al., 2013), plagiarism detection (Paul and Jamal, 2015), machine translation (Shi et al., 2016), reading comprehension (Zhang et al., 2020), etc.
Figure 1 gives two examples of SRL structure. According to the definition of semantic roles, there exist two typical representation forms, i.e., the
(a) Word-based SRL representation adopted in MuCPAD
(b) Span-based SRL representation adopted in CPB and CNB
Figure 1: Examples of two SRL formulations.
word-based and the span-based. This work adopts the word-based form, in which an argument corresponds to a single word. In contrast, span-based SRL, adopted by most previous datasets, takes a word span as an argument. The direction of arcs is from predicates to arguments, and the labels indicate the types of semantic roles. For example, the arc from "买(bought)" to "裙子(dress)" with a label "patient" means that the semantic role between the predicate "买(bought)" and the argument "裙子(dress)" is "patient".
Recently, Chinese SRL research has achieved tremendous progress, thanks to the rise of deep learning methods (Marcheggiani et al., 2017; He et al., 2018; Cai et al., 2018), especially of powerful pre-trained language models (PLMs) (Shi and Lin, 2019; Conia and Navigli, 2020; Paolini et al., 2021). However, existing studies on Chinese SRL mainly focus on the in-domain setting, where training and test data are from the same domain (Wang et al., 2015; Guo et al., 2016b; Xia et al., 2017). SRL performance drops dramatically when the domain of test data is different from that of the training data, known as the domain adaptation problem.
Meanwhile, with the rapid growth of user-generated web data, cross-domain SRL has be
Human Language Technologies, pages 1707 - 1717
Figure 2: An example sentence with an omitted core argument in MuCPAD.
come an important and challenging task in realistic NLP systems (Jiang and Zhai, 2007; Ramponi and Plank, 2020). However, due to the scarcity of multi-domain labeled data, recent research on SRL makes very limited progress in the domain adaptation scenario.
As far as we know, there are three publicly available Chinese SRL datasets, i.e., Chinese Proposition Bank (CPB) (Xue and Palmer, 2005), Chinese NomBank (CNB) (Xue, 2006a), and Chinese SemBank (CSB) (Xia et al., 2017). All these datasets mainly contain canonical texts from newspapers or magazine/textbook articles.
In order to facilitate research on cross-domain SRL, this paper presents MuCPAD, a multi-domain Chinese predicate-argument dataset, consisting of 30,897 sentences and 92,051 predicates, from 6 different domains. Overall, MuCPAD has the following important features.
(1) Following CSB instead of CPB and CNB, we adopt a frame-free annotation methodology, considering that it requires a very high level of linguistic background to define new frames for new predicates or new senses, and a lot of new predicates or new senses may appear in multi-domain texts.
(2) As shown in Figure 2, we explicitly annotate omitted core arguments with two special labels, i.e., "hidden-subject" and "hidden-object", in order to capture richer semantics expressed by predicates. It is ubiquitous that people try to avoid repetition by omitting previous content in context, especially in non-canonical Chinese texts.
(3) We adopt strict double annotation for all sentences in order to improve quality. If two annotators submit inconsistent results, a senior annotator determines the final answer. We also compile 53 pages of annotation guidelines to be studied and referred to by the annotators.
Based on our newly annotated MuCPAD, we conduct preliminary cross-domain SRL experiments and analysis. We enhance the basic SRL model by exploiting CPB2.0 as a heterogeneous
dataset under the multi-task learning (MTL) framework, and by utilizing powerful contextualized word representations from pretrained language models (PLMs).
We release MuCPAD along with our annotation guidelines for research usage at https://github.com/SUDA-LA/MuCPAD.
English SRL Data. Large-scale annotated data is a prerequisite to develop high-performance SRL systems (Fürstenau and Lapata, 2009; Xia et al., 2020). The most representative ones in English are FrameNet (Baker et al., 1998), PropBank (Kingsbury and Palmer, 2002), and NomBank (Meyers et al., 2004). FrameNet is a large-scale manually annotated semantic lexicon resource and uses semantic frames to represent meanings of words. A frame corresponds to a sense of a word, and defines the specific meanings of its core roles (i.e., "A0-A5"). In other words, labels for core semantic roles have predicate-sense-specific meanings.
PropBank and NomBank are built by adding predicate-argument structures to the constituents of syntactic parser trees in Penn Treebank (Marcus et al., 1993). Their semantic roles are naturally span-based, instead of word-based. PropBank considers verbal predicates, while NomBank supplements nominal predicates. Following FrameNet, PropBank and NomBank use frames to represent semantic meanings of predicates. However, the development of frames is both time-consuming and labor-intensive, and requires annotators to be equipped with strong linguistic background.
The texts of PropBank and NomBank are mainly from the news domain, i.e., Wall Street Journal, except 426 sentences from the Brown corpus, which is usually used as an out-of-domain section of PropBank.
It is also noteworthy that there are PropBank-style SRL data for other languages, such as Portuguese (Duran and Aluisio, 2011, 2012), Arabic (Pradhan et al., 2012), Finnish (Haverinen et al., 2015), and Turkish (Sahin and Adali, 2018).
Chinese SRL Data. CPB (Xue and Palmer, 2005), CNB (Xue, 2006a), and CSB (Xia et al., 2017) are the three publicly available SRL datasets in Chinese. CPB and CNB, corresponding to PropBank and NomBank in English respectively, add predicate-argument structure of verbal predicates and nominal predicates into Penn Chinese Tree
Label	Example	Argument
Core roles	agent	我[打]他(I [hit] him)	我(I)
co-agent	我和他[讨论] (I [discuss] with him)	我(I)、他(him)
expe (experiencer)	天气真[好] (The weather [is] really good)	天气(weather)
hidden-subject	[吃]饭了吗? ([Ate] ?)	隐藏主体(hidden-subject)
patient	他被[打]了(He was [hit])	他(he)
pred-patient (predicate-patient)	他[喜欢]打篮球(He [likes] playing basketball)	打(play)
dative	[给]他书([Give] him a book)	他(him)
relative	这[是]他的书(This [is] his book)	书(book)
hidden-object	你[吃]了吗? (Have you [eaten]?)	隐藏客体(hidden-object)
subj-obj (subject-object)	温度计[伸入]水中(The thermometer is dipped into the water)	温度计(thermometer)
Non-core roles	tool (instrument)	用钢笔[写]字([Write] with pen)	钢笔(pen)
material	用颜料[刷]墙([Brush] the wall with pigment)	颜料(pigment)
manner	按计划[执行] ([Perform] according to plan)	计划(plan)
loc (location)	在学校[学习] ([Study] at school)	学校(school)
beg-loc (begin location)	从学校[出发] ([Start] from school)	学校(school)
end-loc (end location)	[流]入大海([Flow] to the ocean)	大海(ocean)
dir (direction)	向西[流] ([Flow] to the west)	西(west)
time	星期天去[打篮球] ([Play] basketball on Sunday)	星期天(Sunday)
beg-tm (begin time)	比赛七点开始[进行] (The game [starts] at seven o 'clock)	七点(seven o'clock)
end-tm (end time)	会议[开]到三点(The meeting [runs] until three o'clock)	三点(three o'clock)
range	在数学上[有]天赋([Have] an aptitude for mathematics)	数学(mathematics)
cause	我因为爱你才[撒谎] (I [lied] because I love you)	爱(love)
quantity	我[跑]了一圈(I [ran] a lap)	一圈(a lap)
separated	我们[见]过面(We have [met])	面(met)
Table 1: Semantic role labels adopted in our guidelines. Predicates in the example sentences are marked by "["]
bank (Marcus et al., 1993). The semantic roles are based on pre-defined frames as well. Moreover, sentences in CPB and CNB mainly come from canonical texts, such Xinhua newswire, Hong Kong news, and Sinorama Magazine (Hajic et al., 2009).
In contrast, CSB uses general-purpose role labels, such as "agent" and "patient", and the sentences are mainly from canonical texts such as online articles and news as well.
Domain adaptation. Domain adaptation has been an important and challenging research topic in NLP (Daumé III, 2007; Ganin and Lempitsky, 2015; Guo et al., 2016a; Kim et al., 2017; Clark et al., 2018; Zhao et al., 2018).
Kim et al. (2016) proposed a neural shared-private model for the cross-domain slot sequence tagging task, which utilizes separate BiLSTM encoders to obtain domain-invariant and domain-specific representations, achieving significant improvements on all domains. Jia et al. (2019) proposed parameter generation networks for cross-domain NER. They idea is dynamically generate parameters of network modules (such as BiLSTMs) according to predicted domain distribution.
To facilitate cross-domain Chinese dependency parsing research, Li et al. (2019a) proposed a large-
scale multi-domain dataset for Chinese dependency parsing. They organized the NLPCC-2019 shared task on cross-domain dependency parsing (Peng et al., 2019). Li et al. (2019b) rank the first place in the shared task, based on a tri-training approach.
However, possibly due to the lack of multi-domain data, research on cross-domain SRL is scarce so far. We hope our newly annotated MuC-PAD can promote future research in this direction.
3 Data Annotation
This section describes the annotation methodology and annotation process of MuCPAD in detail.
Annotation guidelines. After an extensive survey of previous works on SRL data annotation, we compile 53 pages of annotation guidelines. We adopt 24 fine-grained general-purpose role labels to capture the semantic relationships between predicates and arguments, as shown in Table 1, most of which are borrowed from the guidelines of CSB. In particular, we introduce two special labels, i.e., "hidden-subject" and "hidden-object", to explicitly annotate omitted core arguments. Our guidelines illustrate each label in detail using concrete examples, and are gradually improved according to feedback of annotators during the course of the annotation project.
News	PB	PC	ZX	LAW	MED
#Sent	16,974	3,753	3,890	1,575	2,813	1,892
#Pred	40,989	11,317	17,074	5,891	11,156	5,624
Table 2: Statistics of annotated data. "Pred" and "Sent" represent the number of predicates and sentences.
Data selection. We select the data to annotate from six domains, i.e., news, product blogs (PB), product comments (PC), web fictions (ZX), justices (LAW), and medical (MED) domains. Table 2 shows the data statistics.
News consists of the sentences in Chinese SemBank (Xia et al., 2017) and CoNLL-2009 Chinese dataset (Hajic et al., 2009). Specifically, we choose all 10.3K sentences with 16.5K predicates from Chinese SemBank (Xia et al., 2017) and randomly select 6.7K sentences with 24.5K predicates from CoNLL-2009 Chinese dataset (Hajic et al., 2009). Both PB and PC are non-canonical data from Taobao<sup>1</sup>, where PB is from Taobao headline website, and PC is comments on products written by users. ZX is selected from a popular Chinese fantasy novel called "Zhuxian" (ZX, known as "Jade Dynasty"). LAW is extracted from the China artificial intelligence law challenge 2018.<sup>2</sup> MED is crawled from the medical section of People's Daily Online<sup>3</sup> and Sina.com<sup>4</sup>.
After selecting the sentences, we also need to select the concerned predicates in the sentences for annotators to annotate their corresponding arguments. For news domain, we directly choose the predicates in Chinese SemBank and CoNLL-2009 Chinese dataset. For other 5 domains, we choose the predicates according to several predefined rules which are carefully designed by considering both the dependency tree structures of the sentences and a frame dictionary extracted from the Chinese frames<sup>5</sup>. For example, the root words of dependency trees are considered as predicates; the head words with "subject" or "object" dependency labels are considered as predicates; all the words that can be matched in the frame dictionary are considered as predicates.
Quality Control. We employ 20 undergraduate students as annotators, and select 5 experienced annotators as expert annotators to handle annotation inconsistency issues. All the annotators are paid for their work, and the salary is determined by their annotation quantity and quality. The average salary is 28 RMB per hour.
Before real annotation, each annotator is trained for several hours to be familiar with our guidelines and our annotation tool. During the annotation process, we adopt a strict double annotation workflow to guarantee the annotation quality. Specifically, each task is randomly assigned to two annotators to annotate independently. If the submissions from the pair of two annotators are the same, the consistent answer is taken as the final answer. Otherwise, the task is assigned to a third expert annotator to decide the final answer by comparing and analyzing the inconsistent submissions.
Annotation tool. We build a browser-based annotation tool to support the double annotation workflow. For each annotation sentence, the annotation tool highlights the predicate in the sentence for the annotators to annotate all the arguments of the highlighted predicate. We also design a "not-predicate" checkbox in the annotation tool and ask annotators to click this checkbox to inform us when the highlighted word is out of the range of the predicate types in our guidelines.
4 Analysis on MuCPAD
In this section, we analyze the MuCPAD dataset from different perspectives to gain more insights.
Annotation consistency. As aforementioned, each task is assigned to two annotators. If the two submissions are inconsistent, a third expert annotator is asked to handle the inconsistency and decide the final results. The first major row in Table 3 shows the predicate- and argument-wise annotation consistency ratios (Marcus et al., 1994; Guo et al., 2018) in all domains.
The predicate-wise consistency ratio is defined as  $\frac{\#Pred_{annoA∩annoB}}{\#Pred_{annoAU∩annoB}}$ , where the denominator is the total number of predicates submitted by all annotators, and the numerator is the number of predicates with consistent arguments from all annotator pairs. We can see that the predicate-wise annotation consistency ratios in most domains are lower than  $60\%$ . Even the highest predicate-wise consistency ratio, which is achieved in PC domain, is only  $71.23\%$ .
<sup>1</sup>http://www.taobao.com
<sup>2</sup> http://cail.cipsc.org.cn:2018/
<sup>3</sup>http://paper.people.com.cn
<sup>4</sup>https://news.sina.com.cn
<sup>5</sup>https://verbs.colorado.edu/chinese/cpb/html_frames
News	PB	PC	ZX	LAW	MED	AVG
Consistency ratio	predicate-wise	48.86	57.58	71.23	48.86	47.18	50.57	54.05
argument-wise	74.48	74.67	83.63	74.65	71.49	73.24	75.36
overall	85.86	82.08	89.40	83.80	84.57	85.78	85.25
agent	93.50	82.19	88.16	91.45	85.54	86.09	87.82
time	91.17	88.98	85.02	83.69	85.65	86.87	86.90
Argument-wise	hidden-subject	88.79	90.61	94.45	79.65	85.54	90.45	88.25
annotation accuracy	patient	87.02	88.36	90.20	86.32	85.66	89.26	87.80
loc	85.96	79.12	83.71	84.32	84.25	79.30	82.78
pred-patient	85.12	84.28	84.08	83.97	84.11	84.58	84.36
expe	81.58	85.27	90.89	86.39	84.20	87.67	86.00
Table 3: Analysis on consistency ratio and accuracy. "AVG" is obtained by averaging the values of the six domains. For the first major row, "AVG" represents the average predicate/argument-wise consistency ratios in six domains. For the second major row, "AVG" represents the average accuracy of overall/each label in six domains. Boldface indicates the maximum value of each row, underline represents the minimum value of each row.
This means that more than a quarter of the annotation tasks need to be further checked by a third expert annotator, demonstrating the difficulty of the SRL annotation task and the importance of performing strict double annotation to guarantee data quality.
In addition, it is worth noting that the predicate-wise consistency ratio in PC domain is much higher than that in the other five domains. We believe this is related to the average number of arguments per predicate. For further investigation, we calculate the average number of arguments and find the number of average arguments per predicate is the lowest in PC domain. Therefore, it is relatively easier for the annotators to recognize the arguments in PC domain.
The argument-wise consistency ratio is defined as  $\frac{\# \text{Arg}_\text{annoA}\text{annoB}}{\# \text{Arg}_\text{annoA}\text{annoB}}$ , where the denominator is the total number of arguments submitted by all annotator pairs, and the numerator is the number of arguments that receive the same arcs and labels from the annotator pairs. As shown in Table 3, the argument-wise consistency ratios in most of the domains are lower than  $75\%$ , except that PC achieves the highest argument-wise consistency ratio of  $83.63\%$ .
Annotation accuracy. In the second major row of Table 3, we present the argument-wise annotation accuracy. The overall argument-wise annotation accuracy is defined as  $\frac{\sum_{i=1}^{n} \# \mathsf{Arg}_{\text {correct } i}}{2 \times \# \mathsf{Arg}_{\text {gold}}}$ , where the numerator is the sum of the number of correctly annotated arguments submitted by all annotators;
the denominator is the total number of all gold arguments;  $n$  is the number of annotators. The reason for “ $2 \times$ ” in the denominator is that each task is annotated twice since it is assigned to two annotators for double annotation. The annotation accuracies in all the domains are more than  $80\%$ , indicating that our guidelines are reasonable, which ensures the quality of annotation data.
To gain more insights into the accuracy regarding different labels, we calculate the accuracy of 5 core labels and 2 non-core labels with high proportions for further analysis, which is shown in the third major row of Table 3. The argument-wise annotation accuracy for each label is calculated by  $\frac{\sum_{i=1}^{n} \# \mathrm{Arg}_{\mathrm{correct}_i}^l}{2 \times \# \mathrm{Arg}_{\mathrm{gold}}^l}$ , where the numerator is the sum of the number of correctly annotated arguments with the concerned label  $l$  submitted by all annotators, the denominator is the total number of all gold arguments with the concerned label  $l$ ;  $n$  is the number of annotators. As we can see, "hidden-subject" achieves the highest average accuracy, demonstrating the omitted subject is easy to recognize. The lowest average accuracy is  $82.78\%$  on "loc", probably because it is a non-core label with the lowest proportion of all labels and is prone to be ignored by the annotators.
Label distribution. Figure 3 illustrates the label distribution in the 6 domains. The labels in Figure 3 are sorted in descending order by their proportion in News data. We choose 2 core labels and 2 non-core labels with the highest proportions in News data. Besides, we also analyze "hidden-subject"
Figure 3: Label distribution in different domains.
and “hidden-object” since their distributions in different domains are specific. As shown in Figure 3, the label distributions are vary across different domains.
For PC, both "hidden-subject" and "hidden-object" account for the largest proportion compared with that for all the other 5 domains, which means PC contains the most omitted core arguments. The reason is that PC is user-generated comments on a concerned product, and people tend to directly write the comment of the concerned product and omit the product name and the personal pronoun. For example, in the sentence "很 (very) 喜欢 (like),买 (buy) 了好 (very) 多 (much)", both the hidden-subjects and the hidden-objects of the predicates "喜欢 (like)" and "买 (buy)" are omitted. The omitted core arguments in PB also take relatively large proportion, similar to the reason for PC.
For ZX, it has the most "agent" roles and the least "hidden-subject" roles compared with other domains. This is owing to the genre of ZX texts, which are extracted from a popular Chinese fantasy novel with a lot of fictional characters. In order to make the story more understandable by readers, the names of the fictional characters are often explicitly written in the sentences, leading to more "agent" roles and fewer "hidden-subject" roles.
For LAW, it has more "time" roles and "loc" roles than other domains, since the elements (i.e., time and location) of the cases are usually frequently occurred to provide more accurate information.
For MED, the number of "hidden-subject" label accounts for a large proportion among all the labels in MED, only fewer than that of "agent" label, mainly because the descriptions of symptoms in
MED usually omit the subjects. For example, in the sentence “酒精(alcohol)中毒(poisoning): 发生(occur)昏迷(coma)不能(cannot)催吐(induce vomiting), the subjects of the predicates “中毒(poisoning)”, “发生(poisoning)”, “昏迷(coma)”, “不能(cannot)” are all omitted.
Looking into the distribution of "hidden-subject" and "hidden-object" labels in all the domains, we find that hidden labels exist in all the domains, especially in non-canonical texts like PC and PB, demonstrating the necessity of annotating hidden labels. In addition, "hidden-subject" takes a higher proportion than "hidden-object" in all the 6 domains, reflecting that the subject of the predicate in Chinese sentences is often omitted.
Annotation difficulties. To understand difficulties during annotation, we calculate the proportion of the arguments with the same arcs but different labels from two annotators among all the arguments with the same arcs. We find that the confusion pattern "agent, expe" accounts for the largest proportion of  $22.23\%$ , which means the label "agent" is prone to be confused with "expe". This is possibly because the POS for some predicates is subtle and vague in Chinese, causing the confusion of the argument labels. Taking the sentence "纽扣 Buttons) 一天(a day) 坏(geting broken) 一个(one)" as an example, "坏(geting broken)" may be misunderstood as an adjective and thus the argument "纽扣 buttons)" is incorrectly annotated as "exe". Actually, "坏(geting broken)" acts as a verb in this sentence and the correct label of "纽扣 buttons") is "agent". The second confusion pattern is "patient, pred-patient", with a proportion of  $12.6\%$ , due to the misunderstanding of the POS of the argument. It is also difficult for annotators to dis
tinguish "agent" and "patient". For example, in the sentence "新(new) 衣服(cloths) 被(was) 弄脏了(soiled)", the preposition "被(was)" is omitted. As a result, the label of "新(new) 衣服(cloths)", which is "patient", may be confused with "agent" due to the omission.