## 3 Hierarchical Clause Annotation
### HCA Corpus
#### 3.3.3 Quality Assurance  
There were mainly two steps taken jointly to ensure the quality of the final HCA corpus, i.e., multi-round annotation and consistency measurement.  
The total annotation work consisted of 39,260 sentences, and three rounds of annotation were arranged by 5%, 5%, and 90% of the total sentences and conducted with a progressive and negotiable strategy. Before the first round, every annotator thoroughly understood the HCA framework and used the tool ClausAnn proficiently after adequate hands-on training. During the first two rounds, the lead annotator, who majors in English grammar, organized a discussion on complex or abnormal cases with other annotators.  
For consistency measurement, we tracked inter-annotator agreement (IAA) after each round of the annotation work. As discussed in Section 2.1, the HCA and RST parsing tasks aim to extract the clause/EDU hierarchy from texts. Thus, the evaluation metrics of the discourse segmentation [17] and discourse parsing [18] subtasks in RST parsing were adopted as the IAA metrics in evaluating the annotation quality of the HCA corpus:  
* \(\mathrm{P/R/F_{1}}\) on clauses: precision, recall, and \(\mathrm{F_{1}}\)-score on the segmented clauses, where a positive match means that both segmented clauses from two annotators have the same start and end boundaries.  
Figure 6: Operating steps of an annotation trial in the browser-based tool, ClausAnn. (**a**) Switch annotator labels and review the corresponding annotation. (**b**) Segment a text span into two and choose a coordination or subordination between them. (**c**) When choosing coordination, select the exact coordinate relation, i.e., _and_, _or_, or _but_. (**d**) When choosing subordination, select the superordinate clause and the exact subordinate relation, i.e., _Subjective_, _Objective_, and such.  
RST-Parseval [32] on interrelations: consisting of _span_, _nuclearity_, _relation_, and _full_ used to evaluate unlabeled, nuclearity-, relation-, and fully labeled interrelations, respectively, between the matched clauses from two annotators.  
In the first two rounds of annotation, a total of 10% sentences were double-annotated, and the ratio was 16% in the last round, higher than 13.8% in the RST-DT corpus [7]. According to the statistics, the IAA measured by the above two metrics grew as the annotation rounds increased, indicating that the two steps of multi-round annotation and consistency measurement played a significant role in ensuring annotation quality.  
As shown in Table 5, the final IAA achieved high consistencies, where:  
* F\({}_{1}\)-scores on clauses ranged from 98.4 to 100;
* RST-Parseval scores on interrelations ranged from 97.3, 97.0, 93.6, and 93.4 to 98.1, 97.8, 94.2, and 94.1 for _span_, _nuclearity_, _relation_, and _full_, respectively.  
Compared with the RST-DT corpus, whose IAA score on EDUs ranged from 95.1 to 100 and IAA scores on rhetorical relations with three metrics, _spans_, _nuclearity_, _relation_, ranged from 77.8, 69.5, and 59.7 to 92.9, 88.2, and 79.2, our HCA corpus reached better consistencies, as the HCA framework has more-restricted definitions on the elementary unit (i.e., clauses) and fewer types of interrelation.