Venue: NAACL 2022

# CORWA: A Citation-Oriented Related Work Annotation Dataset

Xiangci Li Biswadip Mandal Jessica Ouyang Department of Computer Science University of Texas at Dallas Richardson, TX 75080 lixiangc18@gmail.com,{Biswidip.Mandal, Jessica.Ouyang}@UTDallas.edu

###### Abstract

Academic research is an exploratory activity to discover new solutions to problems. By this nature, academic research works perform literature reviews to distinguish their novelties from prior work. In natural language processing, this literature review is usually conducted under the "Related Work" section. The task of related work generation aims to automatically generate the related work section given the rest of the research paper and a list of papers to cite. Prior work on this task has focused on the sentence as the basic unit of generation, neglecting the fact that related work sections consist of variable length text fragments derived from different information sources. As a first step toward a linguistically-motivated related work generation framework, we present a Citation Oriented Related Work Annotation (CORWA) dataset that labels different types of citation text fragments from different information sources. We train a strong baseline model that automatically tags the CORWA labels on massive unlabeled related work section texts. We further suggest a novel framework for human-in-the-loop, iterative, abstractive related work generation.

+
Footnote †: Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5426 - 5440

## 1 Introduction

Academic research is an exploratory activity to solve problems that have never been solved before. By this nature, each academic research work must sit at the frontier of its field and present novel contributions that have not been addressed by prior work; in order to convince readers of the novelty of the current work, the authors must compare against the prior work. While the format may vary among different fields, in natural language processing (NLP), this literature review is usually conducted under the "Related Work" section. Since each paper must review the relevant prior work in its field, which is shared among papers on the same topic or task, many related work sections in a given field can be similar in both content and format. Therefore, it is a natural motivation to develop a system for generating related work sections automatically.

The task of automatic related work generation is that of generating the related work section of a target paper given the rest of the target paper and a set of papers to cite. Prior works Hoang and Kan (2010); Hu and Wan (2014); Chen and Zhuge (2019); Wang et al. (2019); Xing et al. (2020); Ge et al. (2021); Luu et al. (2021); Chen et al. (2021) mostly simplify related work generation as a general summarization task, generating related work sections using sentence-level models. This approach ignores the nature of the related work section, which consists of variable-length text fragments derived from different information sources. These text fragments refer to different cited papers, and they range in length from a few words to multiple sentences. There are also non-citation, supporting sentences that serve various discursive roles, such as introducing new topics, transitioning between topics, or reflecting on the current work. We argue it is necessary to dis

Figure 1: An example of CORWA labels displayed using the BRAT interface Stenetorp et al. (2012).



tinguish these heterogeneous text fragments, rather than treating related work sections as concatenations of homogeneous sentences.

In addition to the heterogeneous information sources for related work section sentences, the writing styles of these sentences also vary. Khoo et al. (2011) classify literature reviews to be integrative or descriptive, depending on whether they focus on high-level ideas or provide more detailed information on specific studies. However, this document-level classification scheme was intended as a descriptive, information science study of related work sections, and it has not been previously used in automatic related work generation.

Inspired by these observations, as a first step towards linguistically-motivated related work generation, we present a Citation Oriented Related Work Annotation (CORWA) dataset of related work sections from NLP papers. We distinguish text fragments from different information sources by tagging each sentence with discourse labels and identifying the spans of tokens belonging to each citation. We further distinguish citations that give detailed explanations of cited papers and those that illustrate high-level concepts.

Our main contributions are as follows: (1) We collect a CORWA dataset that decomposes the related work section with three inter-related annotation tasks -- discourse tagging, citation span detection, and citation type recognition -- and demonstrate the significance of CORWA with analyses from multiple perspectives (SS3). (2) We propose a strong baseline model that automatically tags the CORWA annotation scheme on massive unlabeled related work section texts (SS4). (3) We show that citation spans are a better target than citation sentences with two example tasks (SS5). (4) We discuss a novel framework for human-in-the-loop, iterative, abstractive related work generation (SS6).

## 2 Related Work

Extractive Related Work Generation.Early related work generation systems employed the extractive summarization approach. Hoang and Kan (2010) pioneered the task, developing rules to select sentences following a topic hierarchy tree that was assumed to be given as input. Hu and Wan (2014) grouped sentences into topic-biased clusters with PLSA, modeled sentence importance with SVR, and applied a global optimization framework to select sentences. Chen and Zhuge (2019) selected sentences from papers that co-cited the same cited papers as the target paper in order to cover a minimum Steiner tree constructed from the paper's keywords. Wang et al. (2019) extracted Cited Text Spans (CTS), the matched text spans in the cited paper that are most related to a given citation. However, these extractive approaches aim to maximally cover the citation texts with the extracted sentences, thus mostly ignoring the _reference_ type citations that are concise and abstractive (SS3.1.3).

Abstractive Related Work Generation.Recently, Xing et al. (2020) extend the pointer-generator (See et al., 2017) to take two text inputs, allowing them to recover a masked citation sentence given its neighboring context sentences. Ge et al. (2021) encode the citation context, cited paper's abstract, and citation network and train their model with multiple objectives: sentence salience score regression of the cited paper's abstract, functional role classification of the citation sentence, and citation sentence generation. Chen et al. (2021) propose a relation-aware, multi-document encoder to generate a related work paragraph given a set of cited papers. Luu et al. (2021) fine-tune GPT2 (Radford et al., 2019) on scientific texts and explore several techniques for representing documents, such as using extracted named entities.

All of the works described above focus on the generation aspect, while neglecting dataset collection; their datasets are mostly extracted automatically. Moreover, the datasets are not reused, though they are publicly available, because these works all use slightly different problem definitions, and thus the models are not directly comparable (Li and Ouyang, 2022). In this work, we focus on collecting a dataset that is widely applicable to various related work generation settings, rather than proposing another incomparable approach.

## 3 CORWA Dataset

In this work, we limit our scope to publications from the NLP domain for ease of automatically extracting the related work section; existing work on related work generation has also focused on NLP in the past. We build our dataset on top of the NLP partition of the S2ORC dataset (Lo et al., 2020), a large-scale corpus of scientific papers derived from LaTeX source code and PDF files. We extract the related work section by matching the section titles. Because not all papers cited in the extracted related work sections are available in S2ORC, we 

prioritize annotating related work sections where the majority of their cited papers are available.

### Annotation Scheme

Our CORWA dataset decomposes the related work section with three inter-related annotation tasks: discourse tagging, citation span detection, and citation type recognition.

#### 3.1.1 Discourse Tagging

Each sentence in a related work section has a specific role and information source. Some may be general topic or transition sentences; some summarize one or multiple prior works in detail, while others describe the general relationship among prior works at a high level. Our discourse tagging task tags the role of each related work sentence with one of six labels: {_single_summ_, _multi_sum_, _narrative_cite_, _reflection_, _transition_, _other_}.

Single Document Summarization._Single_summ refers to sentences that summarize one single cited work in detail. Most typically, this includes sentences with explicit citation marks, as when a work is mentioned for the first time. We also include the following cases: (1) follow-up sentences without explicit citation marks that describe the same paper as a preceding _single_summ_ sentence, and (2) sentences containing multiple citations that heavily focus on one of those works.

Multi-Document Summarization._Multi_summ refers to sentences that summarize multiple prior works of equal importance. As with _single_summ_, we include the case of follow-up sentences without explicit citation marks that continue describing the same group of prior works discussed in a preceding _multi_summ_ sentence.

Narrative Citation.In contrast to _single_summ_ and _multi_summ_, narrative citation (_narrative_cite_) refers to citation sentences that do not summarize specific cited works in detail, but rather convey high-level observations from the authors of the current work. _Narrative_cite_ sentences may contain general statements about the field or task, or the authors' comments on or comparisons of prior works.

Reflection.In addition to describing prior works, authors discuss how they relate to the current work, highlighting the authors' novel contributions. These _reflection_ sentences focus on the current work, instead of prior works.

Transition.Non-citation sentences in related work sections serve as topic introductions or transitions from one topic to another. We label these supplemental sentences that do not belong to any of the above cases as _transition_ sentences.

Other.The related work sections in our dataset are extracted automatically using heuristics based on section titles, and there are occasionally some errors in section boundary detection; we label those sentences that are not actually part of the related work section as _other_.

#### 3.1.2 Citation Span Detection

In order to understand sentences that describe prior work, it is crucial to recognize the token-level mapping between the citation text and the cited paper(s). Our citation span detection task identifies the span of text whose information is directly derived from a specific cited paper. For example, if a cited paper is explained with a summary, its citation span covers the entire summary, which may range from part of a sentence to a few consecutive sentences; if a cited paper is mentioned with an explicit citation, but is not described or discussed at all, then the citation span is just the citation mark.

In constructing the dataset, we find that a single citation rarely spans across paragraph boundaries without a new explicit citation mark, so we require our spans to be bounded by paragraph boundaries.

#### 3.1.3 Citation Type Recognition

Our citation type recognition task indicates whether a cited work is discussed in detail or used to illustrate a high-level concept. We label these types of citations as _dominant_ and _reference_, respectively.

Dominant.These citations are discussed in detail, usually via summarization of their content, and are often longer than _reference_ citations.

Reference.These citations are not discussed in detail. They frequently appear in _narrative_cite_ sentences, but may also appear in _single_summ_ and _multi_summ_ sentences when they are not the main focus of the sentence, and thus it is not sufficient to depend on the sentence-level discourse tags to distinguish them. For example, in Figure 1, line 5, the pointer-generator network See et al. (2017) is cited for reference as part of a longer _dominant_ citation span. _Reference_ citations tend to be more abstractive than _dominant_ citations 



### Annotation Process and Agreement

Two graduate students from our university's Computer Science Department1, manually annotated 927 related work sections. They first annotated 23 related work sections from scratch, after which we incrementally trained a transformer-based tagging model Vaswani et al. (2017) (SS4) to assist the annotation process, asking the annotators to correct the model's predictions, rather than performing manual annotation from scratch. We split the 362 annotated related work sections from papers published in 2019 and later as our test set and all 565 earlier papers as the training set.

Footnote 1: One of them later became the second author of this paper.

Since each related work section is labeled by a single annotator, we calculate agreement by sampling 50 related work sections from the test set and asking the other annotator to re-annotate them from scratch2. We obtain strong agreement on all tasks Cohen's \(\kappa\) of 0.824, 0.965 and 0.878 for discourse tagging, citation type recognition, and citation span detection, respectively; citation type recognition and citation span detection are converted to token-level labels for agreement calculation).

Footnote 2: The disagreements are adjudicated by the first author.

The automated, correction-based annotation process is much faster than annotating from scratch and allows us to collect a much larger annotated dataset. As a trade-off, the annotations may be biased by the model's predictions if the annotators fail to notice any incorrect predictions. This may explain why our model performance reported in SS4.2 is higher than the inter-annotator agreement.

### Analysis of CORWA

The tasks of discourse tagging, citation span detection, and citation type recognition, capture distinct but overlapping perspectives of information.

#### 3.3.1 Relations among CORWA Subtasks

We investigate the relationships among the CORWA subtasks by calculating the co-occurrence distributions of discourse labels and citation span types. A citation span is considered _dominant_ if it contains any _dominant_ citations, and _reference_ otherwise. Figure 2 shows that _dominant_-type spans (average of 34.5 tokens) are significantly longer than _reference_-type spans (average of 8.2 tokens).

Table 1 shows the count of each discourse label, the conditional probability and the joint probability of discourse labels and citation span types. _Single_summ_ with _dominant_ span, _multi_summ_ with _dominant_ span, and _narrative_cite_ with _reference_ span are the most frequent combinations . These observations make intuitive sense, since _dominant_-type spans describe cited papers in detail, often taking the form of a summary, while _reference_-type spans are highly abstracted, making them more likely to be mixed into _narrative_-type sentences that discuss high-level ideas, often encompassing multiple cited papers. This difference is analogous to _informative_ versus _indicative_ summaries, where the former serves as a surrogate for the document, and the latter characterizes what the document is about Kan et al. (2001).

#### 3.3.2 Related Work Writing Styles

Integrative or Descriptive?As Khoo et al. (2011) note, authors may describe the same cited paper in two different styles: descriptive, which ex

\begin{table}
\begin{tabular}{l c c c c c c c c} \hline
**Disc. Label \((d)\)** & \(n(d)\) & \(p(d|~{}D)\) & \(p(d|~{}I)\) & \(p(D|~{}d)\) & \(p(R|~{}d)\) & \(p(D,d)\) & \(p(D,d)\) \\ \hline _single\_summ_ & 4255 & 30.8\% & 80.8\% & 1.1\% & 98.5\% & 1.5\% & **36.9\%** & 0.6\% \\ \hline _transition_ & 3371 & 24.4\% & 0 & 0.2\% & 12.5\% & 87.5\% & 0 & 0.1\% \\ \hline _narrative\_cite_ & 2540 & 18.4\% & 0.4\% & 90.2\% & 0.4\% & 99.6\% & 0.2\% & **48.9\%** \\ \hline _reflection_ & 2489 & 18.0\% & 0.1\% & 6.1\% & 1.5\% & 98.5\% & 0.1\% & 3.3\% \\ \hline _multi\_summ_ & \(671\) & \(4.8\%\) & 18.7\% & 2.5\% & 86.4\% & 13.6\% & **8.5\%** & 1.3\% \\ \hline _other_ & \(510\) & \(3.7\%\) & 0 & 0 & 0 & 100.0\% & 0 & 0 \\ \hline \end{tabular}
\end{table}
Table 1: Distributions of discourse labels and citation spans in CORWA dataset. \(d\): Discourse labels. \(D/R\): _Dominant/reference_ type citation span. \(n(D)=3565,n(R)=4228\). 2927 paragraphs in total.

Figure 2: Histogram of the length of _dominant_ and _reference_-type citation spans, excluding citation marks. The dashed vertical lines are the means of _dominant_ and _reference_ span lengths, 34.5 and 8.2, respectively.



plicitly summarizes the cited paper, or integrative, which describes and comments on the cited paper in a narrative form. We examine the ratio of _summarization_ (both _single\(\_summ\)_ and _multi\(\_summ\)_) and _narrative_ sentences (_narrative\(\_cite\)_) in related work paragraphs (Figure 3). The CORWA discourse labels capture writing style differences among papers: 34.6% of related work section paragraphs only contain _summarization_ sentences, resembling Khoo et al.'s descriptive literature review, while 32.1% of paragraphs contain only _narrative_ sentences, resembling an integrative literature review. Interestingly, 33.3% of paragraphs mix both styles and are neither purely descriptive nor purely integrative.

Frequent Discourse Label Subsequences.Scientific discourse is used by paper authors to promote their ideas Li et al. (2021). We analyze the patterns of CORWA discourse labels to uncover how authors promote their ideas using a mix of sentence types. We apply the rule-based PrefixSpan Han et al. (2001) and Gap-Bide Li and Wang (2008) algorithms to extract frequent discourse label subsequences. We identify six typical subsequences, shown in Supplementary Tables 7 and 8. For example, the pattern of _single\(\_summ\)_ followed by _reflection_ compares the cited paper to the current work, usually without directly criticizing the cited paper, while _single\(\_summ\)_ followed by _transition_ is the more impersonal pattern for criticism of a cited paper, where authors tend to avoid direct comparison with the current work.

## 4 Joint Related Work Tagger

To help propagate our CORWA annotations to massive unlabeled related work sections, we build a joint related work tagger baseline3 that is trained on the three annotation tasks, discourse tagging, citation span detection, and citation type recognition, via multi-task learning Caruana (1997).

Footnote 3: [https://github.com/jacklxc/CORWA](https://github.com/jacklxc/CORWA)

### Model Design

Figure 4 shows the model architecture of our joint related work tagger. We encode related work sections using a transformer-encoder Vaswani et al. (2017) paragraph by paragraph, as we enforce the independence of paragraphs in CORWA citation span annotations. We decode citation span labels and citation type labels token by token, while our discourse tagging task uses the paragraph-level sentence tagging mechanism proposed by Li et al. (2021). Because the three sub-tasks of CORWA are inter-related, we use multi-task learning to jointly train the tagger by sharing the encoder across tasks.

#### 4.1.1 Paragraph Encoder

We experiment with several pre-trained transformer-encoders Devlin et al. (2019); Beltagy et al. (2019); Liu et al. (2019); Beltagay et al. (2020), and eventually focus on SciBERT Beltagy et al. (2019), which is a variant of the BERT model Devlin et al. (2019) that is trained on a scientific corpus with domain-specific tokenization schemes, including NLP papers.

#### 4.1.2 Task-specific Decoders

Citation Span Detection & Citation Type Recognition.We use the \(BIO2\) tagging scheme Sang and Veenstra (1999) for the citation span detection and citation type recognition tasks; we use \(B\), \(I\), \(O\) for citation span detection and five labels -- _B-Dominant_, _I-Dominant_, _B-Reference_, _I-Reference_,

Figure 4: The architecture of our joint related work tagger, which performs discourse tagging (Disc), citation type recognition (CT), and citation span detection (CS).

Figure 3: Parallel plot of the proportion of _summarization_ and _narrative_ sentences in each paragraph. Paradigms with neither type of sentences are excluded.

 

and \(O\) -- for citation type recognition. We use a two-layer feed-forward network to decode the encoded paragraph-level token embeddings to the output sequence of \(BIO2\) tags.

Discourse Tagging.We apply Li et al. (2021b)'s paragraph-level sentence tagging approach for the discourse labels: a simple attention mechanism is used to aggregate token embeddings, sentence by sentence, into sentence encodings, before decoding the sentence encodings into discourse labels using a two-layer multi-layer feed-forward network.

#### 4.1.3 Multi-task Learning

We use cross-entropy loss on all three CORWA subtasks. We balance the relative importance of the sub-tasks by taking a weighted sum of the sub-task losses of discourse tagging, citation span detection, and citation type recognition {\(L_{d},L_{s},L_{t}\)}:

\[L=\gamma_{d}L_{d}+\gamma_{s}L_{s}+\gamma_{t}L_{t} \tag{1}\]

where {\(\gamma_{d},\gamma_{s},\gamma_{t}\)} are tuned hyper-parameters; their values are given in Table 3.

### Experiments

We perform five-fold cross-validation to tune the model hyper-parameters. Table 2 shows the strong performance of the model5. We use the joint related work tagger to automatically label the unannotated 11,465 related work sections remaining in the S2ORC NLP partition and then use this distantly-supervised data to further boost the model's performance. For the citation span detection and citation type recognition tasks, we use a token-level F1 score. Our final, distantly-supervised joint related work tagger achieves more than 0.9 test F1 on all three tasks, indicating the high quality of the model's predictions. This model can be used to propagate our labels on the unannotated related work sections to create a very large training set for future work.

Footnote 5: Only papers included in S2ORC dataset are considered.

## 5 Spans as an Alternative to Sentences

We argue that the citation spans annotated in CORWA are a better alternative to the citation sentences that have previously been used for the tasks of ROUGE-based retrieval and citation text generation.

### Queries for Relevant Sentence Retrieval

Citations focus on a small portion of the content in cited papers, and this focus is not explicitly recorded in the citation network. A popular approach for determining relevant sentences retrieves sentences from the cited papers by comparing the similarity between the gold citation sentence and candidate sentences in the cited paper Cao et al. (2015); Yasunaga et al. (2017, 2019); Ge et al. (2021). Figure 5 compares the distribution of the top-1 average of ROUGE-1 and ROUGE-2 recall scores Lin (2004) of retrieved sentences from cited papers using citation spans with those using citation sentences6. There is no significant difference between the average ROUGE scores of _dominant_ spans and sentences containing _dominant_ citations, which is

Figure 5: Histogram of top-1 ROUGE recall scores of retrieved sentences from cited papers using different queries. The dashed vertical lines are the means of reference sentence (0.220), dominant sentence (0.293), dominant span (0.316), and reference spans (0.449).

\begin{table}
\begin{tabular}{l c c c} \hline
**Model** & **Disc** & **CT** & **CS** \\ \hline SciBERT & \(0.898\) & \(0.959\) & \(0.930\) \\  + Distant Dataset & **0.908** & **0.963** & **0.933** \\ \hline \end{tabular}
\end{table}
Table 2: Test set micro-F1 scores of the SciBERT-based joint related work tagger, with and without training on distantly labeled data, on the discourse tagging (Disc), citation type recognition (CT), and citation span detection (CS) tasks.

 

reasonable because _dominant_ spans are often full sentences anyway. In contrast, the average score of _reference_ spans is significantly higher than that of sentences containing _reference_-type citations; _reference_ spans are shorter and contain highly concentrated key information derived from their cited papers. Thus, using CORWA citation spans as queries for ROUGE-based cited sentence retrieval is superior for _reference_-type citations and comparable for _dominant_-type citations.

### Span-based Related Work Generation

Existing neural network-based, abstractive related work generation systems generate citation sentences given the surrounding context sentences Xing et al. (2020); Ge et al. (2021); Luu et al. (2021) or generate entire paragraphs containing multiple citations Chen et al. (2021). These task settings neglect the fact that the citation text corresponding to a cited paper is not necessarily in the form of a sentence, but could be a portion of a sentence or a block of multiple sentences. Our span-based annotation scheme identifies the citation tokens that are directly derived from the cited papers.

As Figure 6 shows, _reference_ spans are not full sentences, while _dominant_ spans can cover multiple sentences. For _reference_-type citations, using a full sentence as the generation target includes potentially unrelated tokens outside the citation span that do not refer to the cited paper. For _dominant_-type citations, using a single sentence as the generation target can result in 1) information loss when not all sentences describing the cited paper are included in the target, and the model never learns to generate them, or 2) information leak when sentences that actually describe the cited paper are used as context sentences instead of target sentences. Thus, we propose a span-level citation text generation task and present a pilot study using a Longformer-Encoder-Decoder (LED) Beltagy et al. (2020) baseline model.

#### 5.2.1 Experimental Setting

The common Transformer-based language models Devlin et al. (2019); Liu et al. (2019); Lewis et al. (2020); Raffel et al. (2020) have a limited input window size (typically 512 or 1024 tokens), which presents a major challenge for tasks like related work generation that use multiple long documents as inputs. LED Beltagy et al. (2020) addresses this challenge by using a local self-attention mechanism, rather than global self-attention, handling in input context windows of up to 16k tokens. We present an LED-based baseline model for the citation span generation task.

We first pretrain the LED-base model on the masked language modeling (MLM) task Devlin et al. (2019) using related work sections from S2ORC papers in the computer science domain, as well as on the cross-document language modeling (CDLM) task Caciularu et al. (2021), which aligns masked citation sentences with their context sentences and the full text of their cited papers. We further pretrain the LED encoder with the three CORWA sub-tasks (Supplementary Table 6). All pretraining strictly excludes the texts from test set.

For the citation span generation task, we input the concatenation of {the target paper's introduction following Luu et al. (2021)), the partial related work paragraph excluding the target citation span, and the concatenation of {explicit citation mark, title, and abstract} of each cited paper in the target span6); the generation target is the ground truth citation span from CORWA. We provide the explicit citation mark (e.g. Devlin et al. (2018) because it is simple to extract but cannot be inferred from the paper text alone. Just as a human reader may remember the content of the frequently cited papers or the research topics of frequently cited authors, so the citation mark tokens may carry information about the cited paper and its authors.

Footnote 6: We indicate whether the target span is _dominant_ or _refer_-

In addition to the CORWA training set, we use the distantly supervised labels predicted by our joint related work tagger (SS4.2) for training. We use the default hyper-parameters of the Huggingface LED implementation Wolf et al. (2020).

Figure 6: Histogram of the ratio of between the lengths of _dominant_ and _reference_ type citation spans and the corresponding citation sentences. None of the reference spans are longer than one sentence. 27.7%, 46.6%, and 25.7% of _dominant_ spans are shorter than, equal to, or longer than one sentence, respectively.

 



#### 5.2.2 Experimental Results

As Table 4 shows, the ROUGE scores of our LED-base models for citation span/sentence generation are similar to previous sentence-level citation text generation models Xing et al. (2020); Ge et al. (2021), and our pretraining improves the citation span generation performance. Compared to sentence-level generation, span-level generation has lower scores for _dominant_ citations, but higher scores for _reference_ citations. However, because the span- and sentence-level tasks have different generation targets, their scores cannot be directly compared.

We perform a human evaluation following the setting of Xing et al. (2020); Ge et al. (2021). We sample 15 instances each for _dominant_ and _reference_ citations and compare their corresponding span- and sentence-based generation outputs, as well as the gold spans from the original related work sections. Each citation text is rated by three NLP graduate students who are fluent in English on a 1 (very poor) to 5 (excellent) point scale, with respect to four aspects: _fluency_ (whether a citation span/sentence is fluent), _relevance_ (whether a citation span/sentence is relevant to the cited paper(s)), _coherence_ (whether a citation span/sentence is coherent within its context), and _overall quality_.

Table 5 shows human evaluation results, with moderate inter-annotator agreement Kendall's \(\tau\) of 0.298, 0.205, and 0.172 among three annotators). All citation texts are judged to be highly fluent.

Interestingly, in previous studies Xing et al. (2020); Ge et al. (2021) the scores of gold sentences are higher than those of generated texts, but our gold spans have a significantly lower relevance scores than the generated spans. This is likely because the gold spans contain information derived from the body sections of the cited papers, which are not provided to either the models or to the human judges. As a result, some gold spans appear to be irrelevant to the human judges, echoing our earlier finding in SS5.1 that citation spans contain more focused information. This observation also suggests that gold citation spans are not necessarily the best target for all task settings.

We also see that, while _dominant_ sentences and spans receive similar scores, the _reference_ sentences have lower relevance scores than the spans. This result makes sense because _reference_ citation spans are short and focused, so the full sentences include tokens unrelated to the cited paper(s). Overall, the generated spans are rated slightly higher than the generated sentences by the human judges, confirming that span-level citation text generation is preferable to sentence-level generation.

## 6 Toward Full Related Work Generation

Existing extractive related work generation systems Hoang and Kan (2010); Hu and Wan (2014); Chen and Zhuge (2019); Wang et al. (2019) select sentences from the target paper and/or the cited papers, which can be concatenated to form a full related work section; neural network-based, abstractive related work generation systems generate individual citation sentences Xing et al. (2020); Ge et al. (2021); Luu et al. (2021) or paragraphs Chen et al. (2021). However, none of these prior works address the ordering of the extracted/generated sentences or the grouping of sentences into paragraphs, nor are they able to produce rhetorical sentences to smooth the transitions between citations. No prior work bridges the gap from generating individual citation texts to generating a full related work section.

We suggest a bottom-up, iterative approach to generate full related work sections. The process would begin with generating citation spans under

\begin{table}
\begin{tabular}{l|c c c|c c c} \hline  & \multicolumn{3}{c|}{_Dominant_} & \multicolumn{3}{c}{_Reference_} \\
**Models** & **R-I** & **R-Z** & **R-L** & **R-I** & **R-Z** & **R-L** \\ \hline LED-base w/o pretrain & \(0.220\) & \(0.060\) & \(0.183\) & \(0.228\) & \(0.091\) & \(0.223\) \\ LED-base Span & \(0.230\) & \(0.062\) & \(0.186\) & \(0.244\) & \(0.107\) & \(0.240\) \\ \cline{2-7} LED-base Sentence & \(0.244\) & \(0.075\) & \(0.202\) & \(0.193\) & \(0.050\) & \(0.151\) \\ \hline \end{tabular}
\end{table}
Table 4: Performance of citation span/sentence generation using LED-base Beltagy et al. (2020). Citation marks are excluded from the scores since they are trivial to generate and bring up the scores unintentionally. Note that the performance of span/sentence generations are NOT directly comparable due to different generation targets.

\begin{table}
\begin{tabular}{l l l l l} \hline  & Flu. & Rel. & Coh. & Overall \\ \hline _Dominant_ & & & & \\ Gold Span & \(4.61\) & \(3.53\) & \(4.17\) & \(3.64\) \\ Span & **4.92** & **4.07** & **4.20** & \(3.99\) \\ Sentence & \(4.83\) & \(4.03\) & \(4.17\) & **4.02** \\ \hline _Reference_ & & & & \\ Gold Span & **4.87** & \(4.04\) & \(4.18\) & **4.00** \\ Span & \(4.68\) & **4.24** & **4.26** & \(3.96\) \\ Sentence & \(4.48\) & \(3.64\) & \(4.09\) & \(3.70\) \\ \hline \end{tabular}
\end{table}
Table 5: Average fluency, relevance, coherence and overall scores, rated by human judges.



the settings proposed in SS5.2. Then, multiple generated citation spans would be aggregated and rewritten into citation text blocks in either the _summarization_ or _narrative_ style. These blocks would be further aggregated and rewritten into paragraphs by generating _transition_ and _reflection_ sentences.

Generating and rewriting in this pipeline fashion has the following benefits: (1) It mitigates the practical issue of computational resource limitations, given that state-of-the-art models do not perform well on long text generation. (2) The auxiliary inputs, such as citation functions or discourse tags, may vary for each stage of generation. (3) As a practical system to assist researchers, it is crucial to allow user involvement in the iterative generation process. Due to the large search space, consisting of multiple valid related work section candidates with different writing styles, it is extremely challenging to precisely generate a satisfying text with a one-shot, end-to-end system. A human-in-the-loop approach allows the user to significantly prune the search space and simultaneously reduces the error-propagation issue caused by the pipeline design.

## 7 Other Related Tasks

### Scientific Document Understanding

Besides summarization, scientific document understanding also plays an important role in related work generation.

Citation Analysis.Citations are the core of related work sections. There has been a line of research on citation analysis, including citation function [46, 80, 9], citation intent [10, 11], citation sentiment [13, 14, 15, 16], etc. These studies annotate citations with different labeling schemes to study the various usages and purposes of citations.

Discourse Analysis.Scientific discourse analysis studies the rhetorical components of clauses, sentences, or text spans that are not limited to citations, uncovering how authors persuade expert readers with their claims. There is a significant amount of prior work proposing discourse schemes and developing models for discourse tagging for scientific articles [46, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21].

Our CORWA discourse tagging task focuses on distinguishing the source of the information in each related work sentence, which is complementary to the discourse tagging work listed above.

### Cited Text Span

AbuRa'ed et al. (2020) extend Hoang and Kan (2010)'s RWSData dataset by annotating the Cited Text Span (CTS) [23]. They annotate the specific sentences in cited papers that each citation in the target paper is based on. For each cited paper, they further collect a set of papers that co-cite this cited paper. jaidka2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018citingjorke2018d 



## References

* A. AbuRa'ed, H. Saggion, and L. Chiruzzo (2020)A multi-level annotated corpus of scientific papers for scientific document summarization and cross-document relation discovery. In Proceedings of the 12th Language Resources and Evaluation Conference, Marseille, France, pp. 6672-6679. External Links: Link Cited by: SS3.4.
* A. Athar (2011)Sentiment analysis of citations using sentence structure-based features. In Proceedings of the ACL 2011 student session, pp. 81-87. Cited by: SS5.
* A. Athar and S. Teufel (2012)Context-enhanced citation sentiment detection. In Proceedings of the 2012 conference of the North American chapter of the Association for Computational Linguistics: Human language technologies, pp. 597-601. Cited by: SS1.
* I. Beltagy, K. Lo, and A. Cohan (2020)SciBERT: a pretrained language model for scientific text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), Hong Kong, China, pp. 3615-3620. External Links: Link Cited by: SS1.
* G. Ape Burns, P. Dasigi, A. de Waard, and E. Hovy (2016)Automated detection of discourse segment and experimental types from the text of cancer pathway results sections. Database. Cited by: SS1.
* A. Caciularu, A. Cohan, I. Beltagy, M. E. Peters, A. Cattan, and I. Dagan (2021)CdIm: cross-document language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2648-2662. Cited by: SS1.
* Z. Cao, F. Wei, L. Dong, S. Li, and M. Zhou (2015)Ranking with recursive neural networks and its application to multi-document summarization. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 29. Cited by: SS1.
* R. Caruana (1997)Multitask learning. Machine learning28 (1), pp. 41-75. Cited by: SS1.
* J. Chen and H. Zhuge (2019)Automatic generation of related work through summarizing citations. Concurrency and Computation: Practice and Experience31 (3), pp. e4261. Cited by: SS1.
* X. Chen, H. Alamro, M. Li, S. Gao, X. Zhang, D. Zhao, and R. Yan (2021)Capturing relations between scientific papers: an abstractive model for related work section generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 6068-6077. External Links: Link Cited by: SS1.
* A. De Waard and H. Paader Maat (2012)Epistemic modality and knowledge attribution in scientific discourse: a taxonomy of types and overview of features. In Proceedings of the Workshop on Detecting Structure in Scholarly Discourse, pp. 47-55. Cited by: SS1.
* F. Dernoncourt and J. Yee Lee (2017)Pubmed 200k RCT: a dataset for sequential sentence classification in medical abstracts. CoRRabs/1710.06071. Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS1.
* C. Dong and U. Schafer (2011)Ensemble-style self-training on citation classification. In Proceedings of 5th international joint conference on natural language processing, pp. 623-631. Cited by: SS1.
* Y. Ge, L. Dinh, X. Liu, J. Su, Z. Lu, A. Wang, and J. Diesner (2021)BACO: a background knowledge- and content-based framework for citing sentence generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 1466-1478. External Links: Link Cited by: SS1.
* Y. Guo, A. Karbonen, M. Liakata, I. Silins Karolinska, L. Sun, and U. Steinius (2010)Identifying the information structure of scientific abstracts: an investigation of three different schemes. In Proceedings of the 2010 Workshop on Biomedical Natural Language Processing, pp. 99-107. Cited by: SS1.
* J. Han, J. Pei, B. Mortazavi-Asl, H. Pinto, Q. Chen, U. Dayal, and M. Hsu (2001)Prefixspan: mining sequential patterns efficiently by prefix-projected pattern growth. In proceedings of the 17th international conference on data engineering, Citeseer, pp. 215-224. Cited by: SS1.



Kenji Hirohata, Naoaki Okazaki, Sophia Ananiadou, and Mitsuru Ishizuka. 2008. Identifying sections in scientific abstracts using conditional random fields. In _Proceedings of the Third International Joint Conference on Natural Language Processing: Volume-I_.
* Vu Hoang and Kan (2010) Cong Duy Vu Hoang and Min-Yen Kan. 2010. Towards automated related work summarization. In _Coling 2010: Posters_, pages 427-435.
* Hu and Wan (2014) Yue Hu and Xiaojun Wan. 2014. Automatic generation of related work sections in scientific papers: an optimization approach. In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 1624-1633.
* Huang et al. (2020) Ting-Hao Kenneth Huang, Chieh-Yang Huang, Chien-Kuang Cornelia Ding, Yen-Chia Hsu, and C. Lee Giles. 2020. CODA-19: Using a non-expert crowd to annotate research aspects on 10,000+ abstracts in the COVID-19 open research dataset. In _Proceedings of the 1st Workshop on NLP for COVID-19 at ACL 2020_, Online. Association for Computational Linguistics.
* Jaidka et al. (2018) Kokil Jaidka, Muthu Kumar Chandrasekaran, Sajal Rustagi, and Min-Yen Kan. 2018. Insights from cl-scisumm 2016: the faceted scientific document summarization shared task. _International Journal on Digital Libraries_, 19(2):163-171.
* Jaidka et al. (2010) Kokil Jaidka, Christopher Khoo, and Jin-Cheon Na. 2010. Imitating human literature review writing: An approach to multi-document summarization. In _International Conference on Asian Digital Libraries_, pages 116-119. Springer.
* Jaidka et al. (2013) Kokil Jaidka, Christopher SG Khoo, and Jin-Cheon Na. 2013. Literature review writing: how information is selected and transformed. In _Aslib Proceedings_. Emerald Group Publishing Limited.
* Jaidka et al. (2019) Kokil Jaidka, Michihiro Yasunaga, Muthu Kumar Chandrasekaran, Dragomir Radev, and Min-Yen Kan. 2019. The cl-scisumm shared task 2018: Results and key insights. _arXiv preprint arXiv:1909.00764_.
* Jaidka et al. (2011) Kokil Jaidka Jaidka, Christopher Khoo Khoo, and Jin-Cheon Na Na. 2011. Literature review writing: a study of information selection from cited papers/koiki jaidka, christopher khoo and jun-cheon na.
* Jurgens et al. (2018) David Jurgens, Srijan Kumar, Raine Hoover, Dan McFarland, and Dan Jurafsky. 2018. Measuring the evolution of a scientific field through citation frames. _Transactions of the Association for Computational Linguistics_, 6:391-406.
* Kan et al. (2001) Min-Yen Kan, Kathleen R. McKeown, and Judith L. Klavans. 2001. Applying natural language generation to indicative summarization. In _Proceedings of the ACL 2001 Eighth European Workshop on Natural Language Generation (EWNLG)_, Toulouse, France. Association for Computational Linguistics.
* Khoo et al. (2011) Christopher SG Khoo, Jin-Cheon Na, and Kokil Jaidka. 2011. Analysis of the macro-level discourse structure of literature reviews. _Online Information Review_.
* Leuscher et al. (2021) Anne Leuscher, Brandon Ko, Bailey Kuhl, Sophie Johnson, David Jurgens, Arman Cohan, and Kyle Lo. 2021. Multicitic: Modeling realistic citations requires moving beyond the single-sentence single-label setting. _arXiv preprint arXiv:2107.00414_.
* Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7871-7880, Online. Association for Computational Linguistics.
* Li and Wang (2008) Chun Li and Jianyong Wang. 2008. Efficiently mining closed subsequences with gap constraints. In _proceedings of the 2008 SIAM International Conference on Data Mining_, pages 313-322. SIAM.
* Li et al. (2021a) Xiangci Li, Gully Burns, and Nanyun Peng. 2021a. Scientific discourse tagging for evidence extraction. In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_, pages 2550-2562.
* Li et al. (2021b) Xiangci Li, Gully A Burns, and Nanyun Peng. 2021b. A paragraph-level multi-task learning model for scientific fact-verification. In _SDU@ AAAI_.
* Li and Ouyang (2022) Xiangci Li and Jessica Ouyang. 2022. Automatic related work generation: A meta study. _arXiv preprint arXiv:2201.01880_.
* Liakata (2010) Maria Liakata. 2010. Zones of conceptualisation in scientific papers: a window to negative and speculative statements. In _Proceedings of the Workshop on Negation and Speculation in Natural Language Processing_, pages 1-4. Association for Computational Linguistics.
* Liakata et al. (2012) Maria Liakata, Shyamasree Saha, Simon Dobnik, Colin Batchelor, and Dietrich Rebholz-Schuhmann. 2012. Automatic recognition of conceptualization zones in scientific articles and two life science applications. _Bioinformatics_, 28(7):991-1000.
* Lin (2004) Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In _Text summarization branches out_, pages 74-81.
* Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Lo et al. (2020) Kyle Lo, Lucy Lu Wang, Mark Neumann, Rodney Kinney, and Daniel Weld. 2020. S2ORC: The semantic scholar open research corpus. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 4969-4983, Online. Association for Computational Linguistics.

 

Kelvin Luu, Xinyi Wu, Rik Koncel-Redziorski, Kyle Lo, Isabel Cachola, and Noah A. Smith. 2021. Explaining relationships between scientific documents. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 2130-2144, Online. Association for Computational Linguistics.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _Journal of Machine Learning Research_, 21(140):1-67.
* Ravi et al. (2018) Kumar Ravi, Srirangaraj Setlur, Vadlamani Ravi, and Venu Govindaraju. 2018. Article citation sentiment analysis using deep learning. In _2018 IEEE 17th International Conference on Cognitive Informatics & Cognitive Computing (ICCI* CC)_, pages 78-85. IEEE.
* Sang and Veenstra (1999) Erik F Sang and Jorn Veenstra. 1999. Representing text chunks. In _Proceedings of the ninth conference on European chapter of the Association for Computational Linguistics_, pages 173-179. Association for Computational Linguistics.
* See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017. Get to the point: Summarization with pointer-generator networks. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1073-1083, Vancouver, Canada. Association for Computational Linguistics.
* Stenetorp et al. (2012) Pontus Stenetorp, Sampo Pyysalo, Goran Topic, Tomoko Ohta, Sophia Ananiaidou, and Jun'ichi Tsujii. 2012. Brat: a web-based tool for nlp-assisted text annotation. In _Proceedings of the Demonstrations at the 13th Conference of the European Chapter of the Association for Computational Linguistics_, pages 102-107.
* Teufel and Moens (1999) Simone Teufel and Marc Moens. 1999. Discourse-level argumentation in scientific articles: human and automatic annotation. _Towards Standards and Tools for Discourse Tagging_.
* Teufel and Moens (2002) Simone Teufel and Marc Moens. 2002. Summarizing scientific articles: experiments with relevance and rhetorical status. _Computational linguistics_, 28(4):409-445.
* Teufel et al. (2006) Simone Teufel, Advaith Siddharthan, and Dan Tidhar. 2006. Automatic classification of citation function. In _Proceedings of the 2006 conference on empirical methods in natural language processing_, pages 103-110.
* Tuarob et al. (2019) Suppawong Tuarob, Sung Woo Kang, Poom Wettyakorban, Chanatip Pornprasit, Tanakiti Sachati, Saeed-Ul Hassan, and Peter Haddaway. 2019. Automatic classification of algorithm citation functions in scientific literature. _IEEE Transactions on Knowledge and Data Engineering_, 32(10):1881-1896.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreti, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in neural information processing systems_, pages 5998-6008.
* Vyas et al. (2020) Vishal Vyas, Kumar Ravi, Vadlamani Ravi, V Uma, Srirangaraj Setlur, and Venu Govindaraju. 2020. Article citation study: Context enhanced citation sentiment detection. _arXiv preprint arXiv:2005.04534_.
* Wang et al. (2019) Pancheng Wang, Shasha Li, Haifang Zhou, Jintao Tang, and Ting Wang. 2019. Toc-rwg: Explore the combination of topic model and citation information for automatic related work generation. _IEEE Access_, 8:13043-13055.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Mai, Pierric Cistac, Tim Rault, Remi Louf, Morgan Fantowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Phu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Xing et al. (2020) Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020. Automatic generation of citation texts in scholarly papers: A pilot study. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6181-6190.
* Yasunaga et al. (2019) Michihiro Yasunaga, Jungo Kasai, Rui Zhang, Alexander R Fabbri, Irene Li, Dan Friedman, and Dragomir R Radey. 2019. Scisummnet: A large annotated corpus and content-impact models for scientific paper summarization with citation networks. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 7386-7393.
* Yasunaga et al. (2017) Michihiro Yasunaga, Rui Zhang, Kshitiju Meelu, Ayush Pareek, Krishnan Srinivasan, and Dragomir Radey. 2017. Graph-based neural multi-document summarization. In _Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017)_, pages 452-462 

Appendix

### Training Configurations

For the joint related work tagger training, we use GeForce GTX 1080 11 GB GPUs. The training process lasts 2.5 hours on a single GPU using Huggingface's Wolf et al. (2020) SciBERT, BERT-base or Roberta-base as the paragraph encoders, and it lasts 6.5 hours using LED-base encoder. We train the models for 15 epochs. It takes approximately one week to run the hyper-parameter search using five-fold cross-validation for all language models, using 8 GPUs in total.

For training the citation span generation model, we use Tesla V100s-PCIE-32GB GPUs. The training process lasts for 2 days on a single GPU. We run the training for a maximum of 3 epochs with early stopping based on the validation loss.

### Ethical Considerations

We present a new dataset that is derived from the S2ORC dataset Lo et al. (2020), which is released under CC BY-NC 2.0 license. The Huggingface models Wolf et al. (2020) we develop upon are released under Apache License 2.0.

Our annotators were compensated for their work at a rate of double the minimum wage in our local area.

 



\begin{table}
\begin{tabular}{l|c c c|c c c} \hline  & \multicolumn{4}{c}{_Five-fold cross-validation scores_} & \multicolumn{2}{c}{_Test-set scores_} \\
**Models** & \multicolumn{3}{c}{**Disc**} & \multicolumn{1}{c}{**CT**} & \multicolumn{1}{c|}{**CS**} & \multicolumn{1}{c|}{**Disc**} & \multicolumn{1}{c}{**CT**} & \multicolumn{1}{c}{**CS**} \\ \hline SciBERT Beltagy et al. (2019) & **0.900** (0.0099) & **0.961** (0.0038) & **0.926** (0.0059) & **0.898** & **0.959** & **0.930** \\ \hline Roberta-base Liu et al. (2019) & 0.886 (0.0050) & 0.956 (0.0036) & 0.922 (0.0048) & 0.885 & 0.956 & 0.929 \\ \hline BERT-base Devlin et al. (2019) & 0.879 (0.0070) & 0.954 (0.0055) & 0.910 (0.0064) & 0.875 & 0.952 & 0.915 \\ \hline LED-base Pretrained & 0.872 (0.0253) & 0.948 (0.0117) & 0.905 (0.0088) & 0.869 & 0.910 & 0.907 \\ LED-base Beltagy et al. (2020) & 0.865 (0.0090) & 0.922 (0.0128) & 0.907 (0.0074) & 0.842 & 0.874 & 0.909 \\ \hline \end{tabular}
\end{table}
Table 6: Micro-F1 scores for the joint related work tagger using different language models as the encoder. The tasks are discourse tagging (Disc), citation type recognition (CT), and citation span detection (CS). Five-fold cross-validation scores are reported as the mean (standard deviation) across all folds. The pretraining of LED is explained in §5.2.1.

\begin{table}
\begin{tabular}{l l} \hline \multicolumn{3}{l}{**Discourse Subsequence**} \\ _single\text



**Discourse Subsequence**

_single\(\_\)summ, single\(\_\)summ, transition_

**Functionalities**

Commenting previous works summarized.

**Examples**

1. Walker et al. (2012) extract rules representing characters from their annotated movie subtitle corpora.

2. Miyazaki et al. (2015) propose a method of converting utterances using rewriting rules automatically derived from a Twitter corpus.

3. These approaches have a fundamental problem to need some manual annotations, which is a main issue to be solved in this work.

**Discourse Subsequence**

_narrative\(\_\)cite, transition_, single\(\_\)summ_

**Functionalities**

Criticizing the previously cited work and citing an improved work.

**Examples**

1. There have also been several classical studies based on nonneural approaches to headline generation Woodsend et al. (2010); Alfonseca et al. (2013); Colmenares et al. (2015) , but they basically addressed sentence compression after extracting important linguistic units such as phrases.

2. In other words, their methods can still yield erroneous output, although they would be more controllable than neural models.

3. One exception is the work of Alotaiby (2011) , where fixed-sized substrings were considered for headline generation.

**Discourse Subsequence**

_narrative\(\_\)cite, transition_, single\(\_\)summ_

**Functionalities**

Describing an idea following by a comment and then citations implementing the idea.

**Examples**

1. One of the classes of errors in the Helping Our Own (HOO) 2011 shared task Dale and Kilgarriff (2011) was punctuation.

2. Comma errors are the most frequent kind of punctuation error made by learners.

3. Israel et al. (2012) present a model for detecting these kinds of errors in learner texts.