SlowGuess/NAACL_2022_OCR huggingface dataset:
Xiangci Li Biswadip Mandal Jessica Ouyang
Richardson, TX 75080
lixiangci8@gmail.com,
{Biswadip.Mandal, Jessica.Ouyang}@UTDallas.edu
Academic research is an exploratory activity to discover new solutions to problems. By this nature, academic research works perform literature reviews to distinguish their novelties from prior work. In natural language processing, this literature review is usually conducted under the "Related Work" section. The task of related work generation aims to automatically generate the related work section given the rest of the research paper and a list of papers to cite. Prior work on this task has focused on the sentence as the basic unit of generation, neglecting the fact that related work sections consist of variable length text fragments derived from different information sources. As a first step toward a linguistically-motivated related work generation framework, we present a Citation Oriented Related Work Annotation (CORWA) dataset that labels different types of citation text fragments from different information sources. We train a strong baseline model that automatically tags the CORWA labels on massive unlabeled related work section texts. We further suggest a novel framework for human-in-the-loop, iterative, abstractive related work generation.
Academic research is an exploratory activity to solve problems that have never been solved before. By this nature, each academic research work must sit at the frontier of its field and present novel contributions that have not been addressed by prior work; in order to convince readers of the novelty of the current work, the authors must compare against the prior work. While the format may vary among different fields, in natural language processing (NLP), this literature review is usually conducted under the "Related Work" section. Since each paper must review the relevant prior work in its field, which is shared among papers on the same topic or task, many related work sections in a given field can be similar in both content and format. Therefore,
Figure 1: An example of CORWA labels displayed using the BRAT interface (Stenetorp et al., 2012).
it is a natural motivation to develop a system for generating related work sections automatically.
The task of automatic related work generation is that of generating the related work section of a target paper given the rest of the target paper and a set of papers to cite. Prior works (Hoang and Kan, 2010; Hu and Wan, 2014; Chen and Zhuge, 2019; Wang et al., 2019; Xing et al., 2020; Ge et al., 2021; Luu et al., 2021; Chen et al., 2021) mostly simplify related work generation as a general summarization task, generating related work sections using sentence-level models. This approach ignores the nature of the related work section, which consists of variable-length text fragments derived from different information sources. These text fragments refer to different cited papers, and they range in length from a few words to multiple sentences. There are also non-citation, supporting sentences that serve various discursive roles, such as introducing new topics, transitioning between topics, or reflecting on the current work. We argue it is necessary to dis
Human Language Technologies, pages 5426 - 5440
tinguish these heterogeneous text fragments, rather than treating related work sections as concatenations of homogeneous sentences.
In addition to the heterogeneous information sources for related work section sentences, the writing styles of these sentences also vary. Khoo et al. (2011) classify literature reviews to be integrative or descriptive, depending on whether they focus on high-level ideas or provide more detailed information on specific studies. However, this document-level classification scheme was intended as a descriptive, information science study of related work sections, and it has not been previously used in automatic related work generation.
Inspired by these observations, as a first step towards linguistically-motivated related work generation, we present a Citation Oriented Related Work Annotation (CORWA) dataset of related work sections from NLP papers. We distinguish text fragments from different information sources by tagging each sentence with discourse labels and identifying the spans of tokens belonging to each citation. We further distinguish citations that give detailed explanations of cited papers and those that illustrate high-level concepts.
Our main contributions are as follows: (1) We collect a CORWA dataset that decomposes the related work section with three inter-related annotation tasks — discourse tagging, citation span detection, and citation type recognition — and demonstrate the significance of CORWA with analyses from multiple perspectives ( $\S 3$ ). (2) We propose a strong baseline model that automatically tags the CORWA annotation scheme on massive unlabeled related work section texts ( $\S 4$ ). (3) We show that citation spans are a better target than citation sentences with two example tasks ( $\S 5$ ). (4) We discuss a novel framework for human-in-the-loop, iterative, abstractive related work generation ( $\S 6$ ).
Extractive Related Work Generation. Early related work generation systems employed the extractive summarization approach. Hoang and Kan (2010) pioneered the task, developing rules to select sentences following a topic hierarchy tree that was assumed to be given as input. Hu and Wan (2014) grouped sentences into topic-biased clusters with PLSA, modeled sentence importance with SVR, and applied a global optimization framework to select sentences. Chen and Zhuge (2019) se
lected sentences from papers that co-cited the same cited papers as the target paper in order to cover a minimum Steiner tree constructed from the paper's keywords. Wang et al. (2019) extracted Cited Text Spans (CTS), the matched text spans in the cited paper that are most related to a given citation. However, these extractive approaches aim to maximally cover the citation texts with the extracted sentences, thus mostly ignoring the reference type citations that are concise and abstractive ( $\S 3.1.3$ ).
Abstractive Related Work Generation. Recently, Xing et al. (2020) extend the pointergenerator (See et al., 2017) to take two text inputs, allowing them to recover a masked citation sentence given its neighboring context sentences. Ge et al. (2021) encode the citation context, cited paper's abstract, and citation network and train their model with multiple objectives: sentence salience score regression of the cited paper's abstract, functional role classification of the citation sentence, and citation sentence generation. Chen et al. (2021) propose a relation-aware, multi-document encoder to generate a related work paragraph given a set of cited papers. Luu et al. (2021) fine-tune GPT2 (Radford et al., 2019) on scientific texts and explore several techniques for representing documents, such as using extracted named entities.
All of the works described above focus on the generation aspect, while neglecting dataset collection; their datasets are mostly extracted automatically. Moreover, the datasets are not reused, though they are publicly available, because these works all use slightly different problem definitions, and thus the models are not directly comparable (Li and Ouyang, 2022). In this work, we focus on collecting a dataset that is widely applicable to various related work generation settings, rather than proposing another incomparable approach.
3 CORWA Dataset
In this work, we limit our scope to publications from the NLP domain for ease of automatically extracting the related work section; existing work on related work generation has also focused on NLP in the past. We build our dataset on top of the NLP partition of the S2ORC dataset (Lo et al., 2020), a large-scale corpus of scientific papers derived from LATEX source code and PDF files. We extract the related work section by matching the section titles. Because not all papers cited in the extracted related work sections are available in S2ORC, we
prioritize annotating related work sections where the majority of their cited papers are available.
3.1 Annotation Scheme
Our CORWA dataset decomposes the related work section with three inter-related annotation tasks: discourse tagging, citation span detection, and citation type recognition.
3.1.1 Discourse Tagging
Each sentence in a related work section has a specific role and information source. Some may be general topic or transition sentences; some summarize one or multiple prior works in detail, while others describe the general relationship among prior works at a high level. Our discourse tagging task tags the role of each related work sentence with one of six labels: {single_summ, multi_summ, narrative_cite, reflection, transition, other}.
Single Document Summarization. Single_summ refers to sentences that summarize one single cited work in detail. Most typically, this includes sentences with explicit citation marks, as when a work is mentioned for the first time. We also include the following cases: (1) follow-up sentences without explicit citation marks that describe the same paper as a preceding single_summ sentence, and (2) sentences containing multiple citations that heavily focus on one of those works.
Multi-Document Summarization. Multi_summ refers to sentences that summarize multiple prior works of equal importance. As with single_summ, we include the case of follow-up sentences without explicit citation marks that continue describing the same group of prior works discussed in a preceding multi_summ sentence.
Narrative Citation. In contrast to single_summ and multi_summ, narrative citation (narrative_cite) refers to citation sentences that do not summarize specific cited works in detail, but rather convey high-level observations from the authors of the current work. Narrative_cite sentences may contain general statements about the field or task, or the authors' comments on or comparisons of prior works.
Reflection. In addition to describing prior works, authors discuss how they relate to the current work, highlighting the authors' novel contributions. These reflection sentences focus on the current work, instead of prior works.
Transition. Non-citation sentences in related work sections serve as topic introductions or transitions from one topic to another. We label these supplemental sentences that do not belong to any of the above cases as transition sentences.
Other. The related work sections in our dataset are extracted automatically using heuristics based on section titles, and there are occasionally some errors in section boundary detection; we label those sentences that are not actually part of the related work section as other.
3.1.2 Citation Span Detection
In order to understand sentences that describe prior work, it is crucial to recognize the token-level mapping between the citation text and the cited paper(s). Our citation span detection task identifies the span of text whose information is directly derived from a specific cited paper. For example, if a cited paper is explained with a summary, its citation span covers the entire summary, which may range from part of a sentence to a few consecutive sentences; if a cited paper is mentioned with an explicit citation, but is not described or discussed at all, then the citation span is just the citation mark.
In constructing the dataset, we find that a single citation rarely spans across paragraph boundaries without a new explicit citation mark, so we require our spans to be bounded by paragraph boundaries.
3.1.3 Citation Type Recognition
Our citation type recognition task indicates whether a cited work is discussed in detail or used to illustrate a high-level concept. We label these types of citations as dominant and reference, respectively.
Dominant. These citations are discussed in detail, usually via summarization of their content, and are often longer than reference citations.
Reference. These citations are not discussed in detail. They frequently appear in narrative_cite sentences, but may also appear in single_summ and multi_summ sentences when they are not the main focus of the sentence, and thus it is not sufficient to depend on the sentence-level discourse tags to distinguish them. For example, in Figure 1, line 5, the pointer-generator network (See et al., 2017) is cited for reference as part of a longer dominant citation span. Reference citations tend to be more abstractive than dominant citations.
Disc. Label (d)	n(d)	p(d)	p(d | D)	p(d | R)	p(D | d)	p(R | d)	p(D, d)	p(R, d)
single_summ	4255	30.8%	80.8%	1.1%	98.5%	1.5%	36.9%	0.6%
transition	3371	24.4%	0	0.2%	12.5%	87.5%	0	0.1%
narrative_cite	2540	18.4%	0.4%	90.2%	0.4%	99.6%	0.2%	48.9%
reflection	2489	18.0%	0.1%	6.1%	1.5%	98.5%	0.1%	3.3%
multi_summ	671	4.8%	18.7%	2.5%	86.4%	13.6%	8.5%	1.3%
other	510	3.7%	0	0	0	100.0%	0	0
Table 1: Distributions of discourse labels and citation spans in CORWA dataset.  $d$ : Discourse labels.  $D / R$ : Dominant/reference type citation span.  $n(D) = 3565$ ,  $n(R) = 4228$ . 2927 paragraphs in total.
3.2 Annotation Process and Agreement
Two graduate students from our university's Computer Science Department<sup>1</sup>, manually annotated 927 related work sections. They first annotated 23 related work sections from scratch, after which we incrementally trained a transformer-based tagging model (Vaswani et al., 2017) ( $\S 4$ ) to assist the annotation process, asking the annotators to correct the model's predictions, rather than performing manual annotation from scratch. We split the 362 annotated related work sections from papers published in 2019 and later as our test set and all 565 earlier papers as the training set.
Since each related work section is labeled by a single annotator, we calculate agreement by sampling 50 related work sections from the test set and asking the other annotator to re-announce them from scratch<sup>2</sup>. We obtain strong agreement on all tasks (Cohen's  $\kappa$  of 0.824, 0.965 and 0.878 for discourse tagging, citation type recognition, and citation span detection, respectively; citation type recognition and citation span detection are converted to token-level labels for agreement calculation).
The automated, correction-based annotation process is much faster than annotating from scratch and allows us to collect a much larger annotated dataset. As a trade-off, the annotations may be biased by the model's predictions if the annotators fail to notice any incorrect predictions. This may explain why our model performance reported in §4.2 is higher than the inter-annotator agreement.
3.3 Analysis of CORWA
The tasks of discourse tagging, citation span detection, and citation type recognition, capture distinct but overlapping perspectives of information.
3.3.1 Relations among CORWA Subtasks
We investigate the relationships among the CORWA subtasks by calculating the co-occurrence
Figure 2: Histogram of the length of dominant and reference-type citation spans, excluding citation marks. The dashed vertical lines are the means of dominant and reference span lengths, 34.5 and 8.2, respectively.
distributions of discourse labels and citation span types. A citation span is considered dominant if it contains any dominant citations, and reference otherwise. Figure 2 shows that dominant-type spans (average of 34.5 tokens) are significantly longer than reference-type spans (average of 8.2 tokens).
Table 1 shows the count of each discourse label, the conditional probability and the joint probability of discourse labels and citation span types. Single_summ with dominant span, multi_summ with dominant span, and narrative_cite with reference span are the most frequent combinations. These observations make intuitive sense, since dominant-type spans describe cited papers in detail, often taking the form of a summary, while reference-type spans are highly abstracted, making them more likely to be mixed into narrative-type sentences that discuss high-level ideas, often encompassing multiple cited papers. This difference is analogous to informative versus indicative summaries, where the former serves as a surrogate for the document, and the latter characterizes what the document is about (Kan et al., 2001).
3.3.2 Related Work Writing Styles
Integrative or Descriptive? As Khoo et al. (2011) note, authors may describe the same cited paper in two different styles: descriptive, which ex
One of them later became the second author of this paper.
2The disagreements are adjudicated by the first author.
Figure 3: Parallel plot of the proportion of summarization and narrative sentences in each paragraph. Paragraphs with neither type of sentences are excluded.
plicitly summarizes the cited paper, or integrative, which describes and comments on the cited paper in a narrative form. We examine the ratio of summarization (both single_summ and multi_summ) and narrative sentences (narrative_cite) in related work paragraphs (Figure 3). The CORWA discourse labels capture writing style differences among papers:  $34.6\%$  of related work section paragraphs only contain summarization sentences, resembling Khoo et al.'s descriptive literature review, while  $32.1\%$  of paragraphs contain only narrative sentences, resembling an integrative literature review. Interestingly,  $33.3\%$  of paragraphs mix both styles and are neither purely descriptive nor purely integrative.
Frequent Discourse Label Subsequences. Scientific discourse is used by paper authors to promote their ideas (Li et al., 2021a). We analyze the patterns of CORWA discourse labels to uncover how authors promote their ideas using a mix of sentence types. We apply the rule-based PrefixSpan (Han et al., 2001) and Gap-Bide (Li and Wang, 2008) algorithms to extract frequent discourse label subsequences. We identify six typical subsequences, shown in Supplementary Tables 7 and 8. For example, the pattern of single_summ followed by reflection compares the cited paper to the current work, usually without directly criticizing the cited paper, while single_summ followed by transition is the more impersonal pattern for criticism of a cited paper, where authors tend to avoid direct comparison with the current work.
4 Joint Related Work Tagger
To help propagate our CORWA annotations to massive unlabeled related work sections, we build a
Figure 4: The architecture of our joint related work tagger, which performs discourse tagging (Disc), citation type recognition (CT), and citation span detection (CS).
joint related work tagger baseline<sup>3</sup> that is trained on the three annotation tasks, discourse tagging, citation span detection, and citation type recognition, via multi-task learning (Caruana, 1997).
4.1 Model Design
Figure 4 shows the model architecture of our joint related work tagger. We encode related work sections using a transformer-encoder (Vaswani et al., 2017) paragraph by paragraph, as we enforce the independence of paragraphs in CORWA citation span annotations. We decode citation span labels and citation type labels token by token, while our discourse tagging task uses the paragraph-level sentence tagging mechanism proposed by Li et al. (2021b). Because the three sub-tasks of CORWA are inter-related, we use multi-task learning to jointly train the tagger by sharing the encoder across tasks.
4.1.1 Paragraph Encoder
We experiment with several pre-trained transformer-encoders (Devlin et al., 2019; Beltagy et al., 2019; Liu et al., 2019; Beltagy et al., 2020), and eventually focus on SciBERT (Beltagy et al., 2019), which is a variant of the BERT model (Devlin et al., 2019) that is trained on a scientific corpus with domain-specific tokenization schemes, including NLP papers.
4.1.2 Task-specific Decoders
Citation Span Detection & Citation Type Recognition. We use the BIO2 tagging scheme (Sang and Veenstra, 1999) for the citation span detection and citation type recognition tasks; we use  $B$ ,  $I$ ,  $O$  for citation span detection and five labels —  $B$ -Dominant,  $I$ -Dominant,  $B$ -Reference,  $I$ -Reference,
<sup>3</sup>https://github.com/jacklxc/CORWA
Model	Disc	CT	CS
SciBERT	0.898	0.959	0.930
+ Distant Dataset	0.908	0.963	0.933
Table 2: Test set micro-F1 scores of the SciBERT-based joint related work tagger, with and without training on distantly labeled data, on the discourse tagging (Disc), citation type recognition (CT), and citation span detection (CS) tasks.
Parameter Name	Value
Encoder Learning Rate	10-5
Decoder Learning Rate	5 × 10-6
Dropout	0
Epoch	15
Batch Size	1
Steps per Update	10
γd	1
γt	3
γs	1.75
Table 3: Hyper-parameters of our best joint related work tagger (SciBERT + Distant Dataset).
and  $O$  — for citation type recognition. We use a two-layer feed-forward network to decode the encoded paragraph-level token embeddings to the output sequence of BIO2 tags.
Discourse Tagging. We apply Li et al. (2021b)'s paragraph-level sentence tagging approach for the discourse labels: a simple attention mechanism is used to aggregate token embeddings, sentence by sentence, into sentence encodings, before decoding the sentence encodings into discourse labels using a two-layer multi-layer feed-forward network.
4.1.3 Multi-task Learning
We use cross-entropy loss on all three CORWA subtasks. We balance the relative importance of the sub-tasks by taking a weighted sum of the sub-task losses of discourse tagging, citation span detection, and citation type recognition  $\{L_d,L_s,L_t\}$ :
$$
L = \gamma_ {d} L _ {d} + \gamma_ {s} L _ {s} + \gamma_ {t} L _ {t} \tag {1}
$$
where  $\{\gamma_d,\gamma_s,\gamma_t\}$  are tuned hyper-parameters; their values are given in Table 3.
We perform five-fold cross-validation to tune the model hyper-parameters. Table 2 shows the strong performance of the model<sup>4</sup>. We use the joint related work tagger to automatically label the unannotated 11,465 related work sections remaining in the
Figure 5: Histogram of top-1 ROUGE recall scores of retrieved sentences from cited papers using different queries. The dashed vertical lines are the means of reference sentence (0.220), dominant sentence (0.293), dominant span (0.316), and reference spans (0.449).
S2ORC NLP partition and then use this distantly-supervised data to further boost the model's performance. For the citation span detection and citation type recognition tasks, we use a token-level F1 score. Our final, distantly-supervised joint related work tagger achieves more than 0.9 test F1 on all three tasks, indicating the high quality of the model's predictions. This model can be used to propagate our labels on the unannotated related work sections to create a very large training set for future work.
5 Spans as an Alternative to Sentences
We argue that the citation spans annotated in CORWA are a better alternative to the citation sentences that have previously been used for the tasks of ROUGE-based retrieval and citation text generation.
5.1 Queries for Relevant Sentence Retrieval
Citations focus on a small portion of the content in cited papers, and this focus is not explicitly recorded in the citation network. A popular approach for determining relevant sentences retrieves sentences from the cited papers by comparing the similarity between the gold citation sentence and candidate sentences in the cited paper (Cao et al., 2015; Yasunaga et al., 2017, 2019; Ge et al., 2021). Figure 5 compares the distribution of the top-1 average of ROUGE-1 and ROUGE-2 recall scores (Lin, 2004) of retrieved sentences from cited papers using citation spans with those using citation sentences<sup>5</sup>. There is no significant difference between the average ROUGE scores of dominant spans and sentences containing dominant citations, which is
5Only papers included in S2ORC dataset are considered.
Supplementary Table 6 shows the full cross-validation and test performances.
Figure 6: Histogram of the ratio of between the lengths of dominant and reference type citation spans and the corresponding citation sentences. None of the reference spans are longer than one sentence.  $27.7\%$ ,  $46.6\%$ , and  $25.7\%$  of dominant spans are shorter than, equal to, or longer than one sentence, respectively.