CiteBench: A Benchmark for Scientific Citation Text Generation
Authors: Martin Funkquist
Ilia Kuznetsov
Yufang Hou
Iryna Gurevych
Venue: EMNLP 2023 Main (2023)
Keywords: related work generation
citation text generation
scientific document processing
multi-document summarization
summarization
benchmark
evaluation
TL;DR: We propose CiteBench, a unified framework for the study of the citation text generation task, that brings together divergent task definitions and enables systematic comparison of approaches to the task

Abstract: Science progresses by building upon the prior body of knowledge documented in scientific publications. The acceleration of research makes it hard to stay up-to-date with the recent developments and to summarize the ever-growing body of prior work. To address this, the task of citation text generation aims to produce accurate textual summaries given a set of papers-to-cite and the citing paper context. Due to otherwise rare explicit anchoring of cited documents in the citing paper, citation text generation provides an excellent opportunity to study how humans aggregate and synthesize textual knowledge from sources. Yet, existing studies are based upon widely diverging task definitions, which makes it hard to study this task systematically. To address this challenge, we propose CiteBench: a benchmark for citation text generation that unifies multiple diverse datasets and enables standardized evaluation of citation text generation models across task designs and domains. Using the new benchmark, we investigate the performance of multiple strong baselines, test their transferability between the datasets, and deliver new insights into the task definition and evaluation to guide future research in citation text generation. We make the code for CiteBench publicly available at https://github.com/UKPLab/citebench.

CiteBench: A Benchmark for Scientific Citation Text Generation
Martin Funkquist!; Ilia Kuznetsov?, Yufang Hou², Iryna Gurevych2
'Linköping University
PKK Lab, Department of Computer Science and Hessian Center for AI (hessian.Al)
Technical University of Darmstadt
IIBM Research Europe- Ireland
martin. funkquist@liu
Abstract
cited papers
generate:
[1]
btll
abstr
content
Prior work has shown effective
[2]
bitle
abstr
content
supervised
Science progresses by building upon the prior body of knowledge documented in scientific publications. The acceleration of research makes it hard to stay up-to-date with the re- cent developments and to summarize the growing body of prior work. To address this, the task of citation text generation aims to pro- duce accurate textual summaries given set of papers-to-cite and the citing paper context. Due otherwise rare explicit anchoring of cited documents in the citing paper, citation text gen- eration provides an excellent opportunity to study how humans aggregate and synthesize textual knowledge from sources. Yet, exist- ing studies are based upon widely diverging task definitions, which makes hard to study this task systematically. To address this chal- lenge, we propose ENCH: benchmark for citation text generation that unifies multi- ple diverse datasets and enables standardized evaluation of citation text generation models across task designs and domains. Using the new benchmark, we investigate the performance of multiple strong baselines, test their transfer- ability between the datasets, and deliver new insights into the task definition and evaluation guide future research in citation text gen- eration. We make the code for CITEBENCH publicly available at https://gi thub. com/ UKPLab/ci
citing paper
large datasets, such natural languege inference [1] and machine tranelation (2).
Bile
abstr
content
ctx-before
ctx-after
Figure 1: Citation text generation task, text in the grey box adopted from Devlin et al. (2019).
aggregate information from the ever-growing pool of pre-existing work, affecting both junior and ex- pert researchers alike. The task of related work generation aims facilitate these processes by providing topic-based multi-document summaries given pre-defined set of papers-to-cite (Hu and Wan, 2014: Hoang and Kan, 2010).
The applications of related work generation would greatly reduce the reading and writing effort that accompanies research. Yet, the importance of related work generation spans beyond practical applications. Unlike most other genres, scientific writing enforces the use of citation markers that point to the information source, which in most cases is itself highly structured scientific text backed by further citations. By circumventing the challenge of establishing information provenance. related work generation provides a unique opportu- nity to study knowledge aggregation and synthesis from textual sources, and contributes to our better understanding of text work in general.
1 Introduction
A key sub-task of related work generation is citation text generation - generating citation text for pre-selected cited papers given the context of the citing paper (Figure 1). This task is actively studied by the NLP community and plethora of approaches and datascts have been proposed in the past years: has been cast as extractive (Hoang and Kan, 2010; Hu and Wan, 2014) and abstractive summarization (AbuRa'ed
Citations are key characteristic of scientific com- munication. A paper is expected to substantiate its arguments and relate to prior work by means of exact bibliographic references embedded into the text, and the accumulated number of citations serves metric of publication importance. The acceleration of rescarch and publishing in the past decades makes it increasingly challenging both stay up-to-date with the recent publications and to Work done during an internship at UKP Lab.
al., 2020; Xing et al., 2020; Ge et al., 2021; Chen et al., 2021; Luu et al., 2021; Lu et al., 2020; Shah and Barzilay, 2021;

---

Arita al.. 2022); taking one (AbuRa'ed ed al., 2020; Xing al., 2020; Ge et al., 2021) or multi- ple papers as input (Hu and Wan, 2014; Lu et al., 2020; Chen et al., 2021; Shah and Barzilay, 2021), and aiming to generate a single citation sentence (AbuRa's ed et al., 2020; Xing et al., 2020; Ge et al., 2021; Luu et al., 2021) or paragraph (Lu et al.,
(Kiela et al., 2021) for dynamic benchmark data collection. CITEBENCH is the first benchmark for the citation text generation task.
2.2 Text generation for scientific documents
Scientific documents are characterized by academic vocabulary and writing style, wide use non- linguistic elements like formulae, tables and fig- ures, well as structural elements like abstracts and citation anchors. Recent years have seen a rise in natural language generation for scientific text, including text simplification (Luo al., 2022), summarization (Qazvinian and Radev, 2008; Erera et al., 2019; Cachola et al., 2020), slides gener- ation (Sun et al., 2021), table-to-text generation (Moosavi et al.. 2021), and citation text genera- tion (Li and Ouyang, 2022). Closely related to the task of citation text generation, Luu et al. (2021) study how scientific papers relate to each other, and how these relations can be expressed in text. Related our work, Mao et al. (2022) propose a benchmark for scientific extreme summariza- tion. Compared to extreme summarization, which amounts to generating short context-independent summaries of individual manuscripts, citation text generation focuses on context-dependee descrip- tions that relate the cited papers to the citing pa- per. In line with the recent efforts that address the lack of systematic automated evaluation of natural language generation in general (Gehrmann et al., 2021), our paper contributes the first unified bench- mark for citation text generation in the scientific domain.
2020; Chen al., 2021).
Yet, the divergence in the existing task defini- tions and evaluation setups (Table 1) prevents the systematic study of the task. A common task for- mulation and a unified evaluation setup that would enable fair comparison of citation text generation models are missing. To address this gap, we con- tribute (1) CITEBENCH: citation text generation benchmark that brings together four existing task designs by casting them into single, general task definition, and unifying the respective datasets. We couple our benchmark with (2) range of baseline implementations, and (3) a standardized evaluation kit complemented by additional diagnostic modules for qualitative intent- and discourse-basee analy- sis of citation texts. We use CITEBENCH to (4) systematically investigate the task of citation text generation, revealing qualitative and quantitative differences between the existing datasets.
CITEBENCH adds substantial value to the re- search in citation text generation. It enables sys- tematic comparison of existing approaches across a range of task architectures and domains, and pro- vides a scaffolding for future citation text genera- tion efforts. Our proposed qualitative evaluation methodology based on citation intent and discourse structure advances the state of the art in assessing the performance and properties of citation text gen- eration models beyond shallow evaluation metrics.
2.3 Citation text generation
The task of citation text generation was introduced in Hoang and Kan (2010), who generate a sum- mary of related work specific to the citing paper. Since then, several task definitions and setups have been proposed (Table 1). Lu et al. (2020) cast the task as generating a multi- mulli-paragraph related work section given the abstracts of the citing paper and of the cited papers. AbuRa' ed et al. (2020) use the cited paper's 's title and abstract to generate a citation sentence. Xing et al. (2020) use the abstract of the cited paper and include context before and after the citation sentence as the input, and produce the citation sentence the output. A recent work by Chen al. (2021) uses multiple cited abstracts input to generate related work paragraph. The great variability of the task definitions and setups in citation text generation prevents the study of ci-
2
Related work
2.1 Benchmarking
NLP benchmarks are unified dataset collections coupled with evaluation metrics and baselines that are used to systematically compare the perfor- mance of NLP systems for the targeted tasks in a standardized evaluation setup. Well-constructed benchmarks can boost progress in the correspond- ing research areas, such SQuAD (Rajpurkar al., 2016) for question answering, GLUE (Wang al., 2018) for natural language understanding, KILT (Petroni et al., 2021) for knowledge-intensive tasks, GEM (Gehrmann et al., 2021, 2022) for general-purpose text generation, and DynaBench

---

Input
Output
Dataset
Data sources
Cited document (1))
Citing context (C*)
Citation text (T)
Single Ahs
Multi Ahs
Titlc
Abs
Text
Sent
Para
ABURAED
Multiple
CHEN
S20RC and Delve
LU
arXiv.org and MAG
XING
AAN
Table I: Overview of datasets in CITEBENCH. Single Abs Single abstract, i.e., cited document per instance.
Multi Abs = Multiple abstracts, i.e. multiple cited documents per instance. Abs = Abstract, i.e., dataset contains the abstract of citing paper. Text a dataset contains additional context from citing paper. Sent generation target is single sentence. Para generation target is paragraph.
Dataset ABURAED
#Train
#Validation
#Test
Inputs 4,096 tok.
Outputs 1,024 tok.
15,000
1,384
0%
0%
30,369
5,066
5,093
<0.001%
0%
XING
77,086
8,566
400
<0.001% %
<0.001%
CHEN
Deelve
72,927
3,000
3,000
<0.0015
0.004%
S20RC
126,655
5,000
5,000
0.017%
<0.001
Total
322,037
23,016
13,712
0.007%
<0.001 %
Table 2: Datasets statistics. The validation set for XING has been created by us via randomly sampling 10% of the original training data. Across datasets, very few imputs contain more than 4,096 tokens, and few outputs are longer than 1,024 tokens. We exploit this property speed up the evaluation in Section 3.3.
tation text generation methods across datasets and evaluation setups. Unlike prior work that explores varying task settings, CITEBENCH brings the di- verging task definitions and datasets together in unified setup. This allows compare citation text generation models across different datasets in a standardized manner using an extensive set of quantitative metrics, as well as novel automated qualitative metrics.
mark, we select four datasets, focusing on the task design and domain variety: ABURAED (AbuRa' ed et al., 2020), CHEN (Chen et al., 2021), LU (Lu et al., 2020), and XING (Xing et al., 2020). Dataset trans- formation details are provided in Appendix A.1. Table 2 shows the quantitative statistics, and Fig- ure 2 provides data examples from each dataset. The CHEN dataset has two subsets - CHEN Delve and CHEN S20RC based on the data source: we use CHEN to denote the union of the two subsets. The datasets are distributed under varying licenses; we have obtained explicit permissions from the au- thors to use the data for research purposes in cases when licensing was underspecified (see Ethics state- ment).
3 Benchmark
3.1 Task definition and datasets
We formalize the task citation text generation as follows: Given set of n (cited) target documents (D,,..D',,, (citing) source document D and a set of m citing document contexts (Ci ...O%; ) e D generate citation text T' that is close as possible the original citation text T D This general definition allows wide variation in how the task is implemented. The cited document Dj can be represented by the abstract a the concatenation of the title and the abstract, or even the full text of the paper. The context set C* covers sentences before and after the citation text in D*, as well as the abstract a D
We note that the datasets included the bench- mark are not only structurally diverse, but also cover wide range of domains, from medicine to computer science. In particular, ABURAED and XING exemplify citation text generation in the computa- tional linguistics domain, CHEN Delve cover the computer science domain; LU and CHEN S20RC span wide range of domains represented on arxiv.org and the S2ORC corpus, respectively, including biology, medicine and physics.
3.2 Evaluation and analysis kit
Such general, open definition allows us ac- commodate diverse approaches to the task within one framework (Table 1). To populate the bench-
CITEBENCH uses two quantitative evaluation met- rics to estimate the performance of the models.

---

Cited document(s)
Citing context(s)
Citation text
ABURAED
extracting structural paraphrases from
[0] asked judges whether their
allgned monolingual corpora. present approach for automatically leamming paraphrases from...
paraphrases were roughly interchangeable given the genre.
[None]
This paper proposes vehicietovehicle communication protocol for cooperative callision wamning... emerging wireless
medium access control mac
issues have been addressed in
[O]... transportation safety issues
CHEN
technologies for...
[0]
[None]
have been addressed in [1].
where vehicles communicate with
in this paper conduct a feasibility study of delaycritical safety applications over vehicular ad hoc networks...
each other and with static
network nodes such traffic
lights bus sheiters...
Word groupings useful language processing tasks are increasingly
Abstract Interaction virtual reality
Resnik [0] combines the of
(VR) environments (e.g. grasping and manipulating virtual objects) is essential ensure pleasant and immersive
WordNet... Ng and
make
available...
[0]
of several sources
information inside training collection...
this paper, we present new approach for
experience. In this work, we
word disambiguation (WSD) using...
propose...
this paper. we propose novel neural
<abs-We We present joint language and translation model based recurrent
More recently. another encoder-decoder model using RNN was proposed in [0], where the decoder is conditioned on representation either a source sentence or a source context.
network model called RNN Encoder.
Decoder that consists of two recurrent neural networks (RNN), One RNN encodes a sequence of symbols into a fixedlength vector representation, the other decodes representation into another sequence of symbols...
neural aatoork..s/bbss..
XING
<bbx b- This is closely related to both the b> <cbx ane important difference between
proposed RNN Encoder-Decader...
proposed </ctx_
Encoder-
Data examples extracted from the datasets (left). <abs> citing paper's abstract, <ctx, b> context before target citation text, <ctx_a>- context after target citation text, rightmost column generated citation Title in ABURAED marked with orange.
Figure 2:
ROUGE (Lin, 2004) measures the n-gram over- lap between the output text and the reference text. Following the recently published GEM benchmark (Gehrmann et al., 2021), we use the Huggingface ROUGE calculation package', with the original citing text and the model outputs the input, without additional stemming and lemmatization. BERTScore (Zhang al., 2020) recent alter- native to ROUGE that uses contextual embeddings calculate similarity score between the model outputs and reference texts. We use the BERTScore implementation provided by Huggingface*.
ACL-ARC schema by Jurgens et al. (2018) to classify citation sentences into six intents: Background provides background information: CompareOrContrast explains similarities or dif- ferences between the cited and the citing paper; Extends builds upon the cited papers; Future suggests cited papers potential basis for future work; Motivation illustrates research need, and Uses indicates the of data methods from the cited papers. CORWA uses the sentence classifi- cation schema proposed by Li et al. (2022), cover- ing six citation sentence types: Single_s ___mmm_mmmmmm_ and Mult tultimsumm refer to citation sentences that are de- tailed summaries of a single and multiple cited pa- pers, respectively; Narrative_cts cireiettteeeeeeeeeeeeeeeee te are high-level statements related to the cited papers; Reflection sentences relate the cited paper the current work with focus on the citing paper; Transition are non-citation sentences that connect different parts of related work, and Other accommodates all other sentences.
Quantitative metrics offer coarse, high-level picture of comparative model performance, but de- liver little qualitative insights into the model be- havior. To alleviate this, we enrich our kit with two recently proposed citation analysis tools to study the discourse structure of citation texts citation intent labeling (Cohan et al., 2019) and CORWA tagging (Li et al., 2022), discussed below.
Citation intent labeling uses the fine-grained
The two schemata offer complementary views on the citation text analysis: while citation intent fo- cuses on why a paper is cited, the CORWA schema offers insights on how the citation text is composed
/nttps://hugginggace httpc/ee.....pppeeeeeeeeeeeeeeee co/spaces/ evaluate-metric/roug
"https://hugg" //hggg/nggfggeeee wttpo/aaaa.cc..a.eeeeeeeoeeeeeee ct/ co/spaces/

---

and what role plays in the surrounding text. Both schemata offer valuable insights on the composi- tion of citation texts, and our analysis toolkit uses the publicly available model and implementation by Cohan et al. (2019)² for ACL-ARR-style citation intent analysis, and the publicly available code and model from Li et al. (2022)4 for CORWA tagging. The authors report F1 scores of 67.9 for citation intent classification and 90.8 for CORWA tagging on ACL Anthology corpus (Bird et al., 2008) and S2ORC respectively, indicating that the CORWA tagger can be used with confidence for gaining in- sights into citation text generation, particularly on citation texts from the domains similar to ACL and S2ORC.
are fine-tuned on the mixture of all training splits of all datasets in the benchmark; the *led-base model trained for epochs with batch size 16 on 4 GPUs; the +led-large-arxix model trained for 3 epochs with batch size 8 on 4 GPUs. In trans- fer learning experiments, we train and evaluate the * red-base-[XX models, each fine-tuned on spe- cific dataset X, for example, *
Results
4.1 Baseline performance
Table 3 reports the baseline performance on the CITEBENCH datasets ROUGE-L and BERTScore. As a coarse reference point, and keeping in mind the differences in evaluation settings (Section 5.1), Lu et al. (2020) report ROUGE-L score of 30.63 for their best super- vised, task-specific model, and 18.82 for LEAD. We note that our extractive LEAD baseline outperforms the distantly supervised abstractive led-base and Led-large baselines on all datasets in terms of all different ROUGE metrics and BERTScore. The led-large-arxiv baseline systematically outper- forms led-base and led-large as well, which we attribute to the in-domain fine-tuning on the arXiv data that consists of scientific text simi- lar to CITEBENCH datasets. Directly supervised baselines (*led-base and *led-large-arxin achieve the best overall performance. We note that with few exceptions the rankings produced by the two evaluation metrics correlate: the best per- forming models on ROUGE also achieve the best performance on BERTScore, except * *led-base that performs best on the CHEN Delve subset in terms of BERTScore but slightly falls behind * #led-large-arxii in terms of ROUGE-L.
3.3 Baselines
We complement our evaluation setup with widely used unsupervised extractive baselines from pre- vious work. LEAD (Lu et al., 2020) selects the first three sentences from the input, represented as the concatenation of the cited abstracts, as citation text. TextRank (Mihalcea and Tarau, 2004) and LexRank (Erkan and Radev, 2004) are graph-based unsupervised models for extractive text summariza- tion. For TextRank, from the package summa'. For LexRank, we use the package lexrank with the default settings and summary size of three sentences to match LEAD.
use the default settings
In addition, we provide a range of new distantly supervised abstractive baselines based on the pre- trained Longformer Encoder Decoder (LED) (Belt- agy et al., 2020). While LED is capable of handling inputs up to 16,384 tokens, to reduce the compu- tational overhead, we truncate the inputs to 4,096 tokens, substamtially reducing the computation cost while only affecting negligible proportion of data instances (Table 2). We experiment with three different versions of the LED model: led-base, led-large and led-large-arxe 7 which is fine- tuned on the arXiv dataset for long document sum- marization (Cohan et al., 2018). 8).
4.2 Transfer learning results
The unified task formulation in CITEBENCH allows us to explore transfer learning between different domains and citation text generation setups. We ex- amine the transfer learning performance using the * bedd-base-[X] models fine-tuned on individual datasets, starting from the pre-trained led-base model. Table 4 presents the results in ROUGE- I. and BERTScore; Table 7 (Appendix) provides additional details. Expectedly, the models per- form best both on ROUGE-L and BERTScore when evaluated on the test portion of the same
Finally, CITEBENCH features range of directly supervised abstractive baselines marked with *: the +led-base and sled-large-arxtl models
bed-base-16384; https://huuginggaace co/ellooallllll allanni///
1ed-1arge-16384 https://huggingfaace co/ allenai/
*ROUGE-1 and ROUGE-2 sonres are provided in the Ap- pendix for the sake of completeness.
1ed-large-1638-arr

---

Delve
CHEN
S20RC
XING
Model
BertS
R-L
BertS
R-L
BertS
R-L
BertS
11.32
75.42
11.48
74.70
11.34
74.33
13.34
75.89
10.55
75.25
TextRank
9.35
64.59
14.04
76.41
12.82
74,91
12.75
72.61
6.61
45.97
LexRank
10.80
74.90
12.93
75.96
12.85
75.11
14.24
76.70
10.06
75.14
led-base
9.06
74.84
5.55
70.86
5.41
70.55
7.36
72.35
10.07
74.87
led-large
8.30
73.26
6.22
69.57
6.22
69.77
6.89
70.51
9.35
74.30
-aiilaye--arxiv
10.22
75.01
13.37
76.02
12.89
75,45
14.41
76.65
10.23
75.08
*led-base
(3,44(0.0z)
78.75
15.93
78,32
15,94
78.72
15.95
79.32
13,58
78.49
#led-large-arxiv
14.90
79.0
16.27
78.13
15.92
78.58
16.53
79.41
12.42
77.57
Table 3: Baseline performance per dataset, ROUGE-L and BERTScore, average three runs, best are bolded. For readability, only standard deviation equal or higher than 10- 2 is shown (in parentheses).
ABURAED
CHEN
Delve
CHEN
S20RC
LU
XING
Model
R-I.
BertS
R-I.
BertS
R-I.
BertS
R-1.
BertS
R-I.
BertS
LLed-base-aburaan
13,95
78.35
9.96
75.24
10.66
75.63
11.63
76.82
12.63*
78.00*
-hee
11.48
76.57
16.04
78.23
16.21
78.71
15.40
78.58*
10.14
75.38
+led-base-lu 12.19
76.31
14.58*
77.55*
14.59
77.11*
16.25
79.25
11.56
77.03
+led-base-xing
11.97
77.56*
9.54
75.06
10.01
74.57
9.88
75.66
14.06
78.70
Table 4: In- and cross-dataset fine-tuned model performance in ROUGE-L and BERTScore on one run (standard deviations multiple runs very small for all models). Best scores bolded, denotes the second best scores. We use the union of two subsets in CHEN for fine-tuning and evaluation.
dataset (in-domain). We note that cross-dataset (out-of-domain) transfer in several cases outper- forms the strong unsupervised baselines. For in- stance, *led-base-chen achieves better results than all unsupervised models and off-the-shelf neu- ral models on the ABURAED and LU datasets.
outputs and individual datasets.
The distribution of citation intents in Fig- ure 3 suggests discrepancy between the gen- erated and original citation texts: while the baselines tend to under-generate the Background and CompareOrContras sentences, they produce more Future, Uses and Extends sentences than the gold reference. The KL divergence of citation intent distributions (Figure 3) shows that all mod- els' outputs are more deviated from the original citation intentions on ABURAED compared to other datasets. Interestingly, the two fine-tuned models that perform well in terms of ROUGE scores tend also achieve lower KL divergence scores for all datasets, e.g., *led-base and Wled-large-ard arxiv... This suggests that the two models learn a dataset's 's citation intent distribution during fine-tuning.
We also observe that the models often achieve better lcarning performance on the test sets that have task definitions similar to the train- ing data. For instance, fled-base-aburee leed-base-chen reach the best out-of-domain scores the XING dataset and the LU dataset, re- spectively. As indicated by Table 1, both ABURAED are XING generate citation sentence for single cited document, while CHEN and generate cita- tion paragraph for multiple cited papers. We elabo- rate on this observation in the Appendix A.4.
and
Turning to the CORWA results, Figure sug- gests high discrepancy between the baseline model ouputs and the test set outputs. Most our baselines under-generate the Narrative and - surprisingly - Single_ summ class, while over- generating the Reflection, compared to the dis- tributions in the gold reference texts. The only two exceptions are the fine-tuned *led-base and * *led-14 aod-large-arxi models, which aligns with their high performance in terms of ROUGE and BERTScore. High KI. divergence values in Figure confirm that the predicted CORWA tags are more discriminative across the datasets compared to ci-
4.3 Discourse analysis
We now use the discourse analysis tools introduced in Section 3.2 to qualitatively compare the citation texts found in datasets and generated by the models. We apply the citation intent and CORWA taggers to the baseline test set outputs and macro-average the results across all datasets in CITEBENCH. We com- pare the resulting distributions the distributions in the citation text outputs found in the datasets, along with macro-averaged total over the datasets. To quantify the discrepancies, we calculate KL di- vergence between the label distributions in model

---

LEAD
TextRank
LexRank
led-base
larle
TexRankk
0.55
0.37
0.21
0.32
0.31
ind large arxiv
fled-base
Lexttank
0.33
0.35
0.2
Med-large-arxiv
ABURAED
led- Aod-aarg--
0.57
0:35
0.21
0.31
0.32
0.3
CHEN Delve
led led-arge-aree anxi-
0.06
0144
0.29
0.4
0.39
EU
Red base
0.57
0.26
0.072
0.19
0.15
*ed-large-ansie
Total
ABURAED
CHEN Delie
CHER S20RC
XNG
Background
Extends
Motivation
Compareorcontrast
Future
Uses
Figure Citation intent distribution (left) for model outputs (top) and datasets (bottom), and KL divergence between datasets and model outputs (right).
tation intent labels. The lowest CORWA tag distri- bution divergence is observed for the *led-base and *led-Large-arxix baselines, suggesting that the learned ability to capture the CORWA tag dis- tribution might play role in improved citation text generation performance.
such an analysis is often prohibitively expensive, limiting the number of instances that can be manu- ally assessed. The progress automatic discourse analysis of citation texts makes it possible to study the composition of reference texts and compare it to the aggregate model outputs. We proposed cita- tion intent classification and CORWA tagging inexpensive muddle-ground solution for qualitative evaluation of citation texts. Our results indicate that the composition of citation texts indeed differs. among datasets and system outputs.
5 Discussion
5.1 Replicability of ROUGE
While re-implementing the baselines, we found that replacing the Huggingface ROUGE implementa- tion with the files2rouge package used by Lu et al. (2020) results in a substantial ROUGE-IA 1/L score increase of approx. points. Our investigation revealed that this discrepancy is due to preprocess- ing: both packages upply transformations to the inputs, yet, while files2rouge includes stemming by default, the Huggingface implementation does not. We found additional effects due to tokeniza- tion and stopword removal. This implics that the differences in ROUGE scores across publications can be attributed to the particularities of the eval- uation library, and not to the merits of a particular model. This underlines the importance of clearly specifying the ROUGE package and configuration in future evaluations.
Driven by this, we used KL divergence between reference and generated texts to study the differ- ences between citation texts in aggregate. Fol- lowing up on our analysis in Section 4.3, ob- serve that while the two perfformin baseline models terms ROUGE and BERTScore also achieve the lowest KL divergence on both discourse schemata, the correlation is not perfect. We thereby suggest the use of KL divergence of citation intent and CORWA tag distributions as supplementary metric to track progress in citation text generation, while keeping in mind that the performance of the underlying discourse analysis models is itself sub- ject future improvement (see Limitations).
5.3 Human evaluation
To get better understanding of the model out- puts, three human annotators have manually in- spected 25 baseline outputs from each of the best-performing LexRank, led-large-arxiv and -led-large-ard arxiv models, in each model type
5.2 Qualitative evaluation
None of the existing automatic evaluation metrics for text generation can substitute in-depth quali- tative analysis of model outputs by humans. Yet,

---

LEAD
TextRank
LexRank
2.0
LEAD
13
0.59
0.6
0.85
1.8
large
TextRank
15
0.75
led large-ggaaaa anov
15
0.58
Red
14
Med bed-large-ancv
led-base
1.5
0.77
0.35
1.2
12
ABURAED
led largar
2
1.3
0.36
2.9
1.3
10
CHEN Delve
CHEN S2ORC
led large
17
0.93
0.65
1.2
0.8
0.6
XING
Hed largo andiv
0.53
0.27
0.28
0.36
0.0
0.2
0.4
0.6
0.8
1.0
ABURARO
CHEN Delve
CHEN szonc
w
XNG
Multi summ
Other
Singlessumm summ
Reflection
Figure 4: CORWA tag distribution (left) for model outputs (top) and datasets (bottom), and KL divergence between datasets and model outputs (right). Empty cells in XING (right) denote co due to missing labels system predictions. Note the scale differences between the KL divergence plots here and in Figure 3, kept for presentation clarity.
category, on each of the two architecturally distinct XING and CHEN DELVE datasets, 150 instances in total. Each output was rated on a 5-point scale in terms of readability and consistency. The estimated Quadratic Cohen's kappa agreement between anno- tators 50-instance subsample was approx. 0.50 for readability and ranging between 0.18 and 0.42 for consistency, depending on the annotator pair (Appendix A.5). Turning to the evaluation results, on CHEN DELVE, the fine-tuned * #led- lad-laage--rrxv receives the highest average readability score of 4.32 and consistency score of 2.36, while the base- line led-large-arxiv achieves the best scores both dimensions XING, with average readability score of 3.88 and consistency score of 2.20. Overall, we observe that while some models achieved satisfactory readability scores on some datasets, none scored higher than 3 on the con- sistency scale: while the generated citation texts can be topically related to the gold reference, in most cases they miss key factual details about the cited papers and the contextual information in the reference. Motivated by this, we conclude our dis- cussion by turning to the question on the optimal definition for the citation text generation task.
sions the citation text generation task, input and output structure and granularity are not the only pa- rameters that influence task complexity, and future work in citation text generation might explore other, qualitative aspects of the task. A manual review of the datasets included in CITE.ENCE reveals that in some cases cilation text can be generated based on the provided inputs due to missing information (see Appendix A.6 for examples). This raises the question of what information is in fact required to produce accurate citation texts.
In addition, we observed that prior data includes instances of self-refcrence where citation text in fact talks about the citing paper itself. To further substantiate this observation, we have searched ci- tation texts throughout CITEBENCH for keywords that might indicate self-reference ("our work", "in this paper", "we present", "we introduce")) and found that the datasets widely differ in terms of self-reference, from 1% the single-sentence citation text datasets (ABURAED and XING) to up to 14% in CHEN Delve (Appendix A.6). This sug- gests that task design can influence the qualitative composition of citation texts, and calls for further investigation of data selection procedures for cita- tion text generation.
5.4 Task refinement
Finally, note that citation text might vary de- pending on the author's intent. As our analysis in Figures and demonstrates, datasets do differ in terms of the distribution of citation intents and discourse tags. Lauscher et al. (2022) show that
Unifying a wide range of citation text generation setups within a common framework allows addi- tional insights into the definition the task. While Ci teBench accommodates different structural ver-

---

intent classification can benefit from citation con- texts, and we argue that modeling citation intent explicitly could lead to a more robust and realis- tic definition of the citation text generation task: a Compari isn citation would differ in content and style from Background. Thus, we deem it promis- ing to extend the input with the information about intent, guiding the model towards more plausible outputs.
Limitations
The datasets in CITEBENCH are in English, lim- iting our baseline models and results to English. While CiTEBENCH covers a wide variety of sci- entific domains, humanities and social sciences are under-represented. Since the citation patterns might differ across research communities, this might lead to performance degradation due to do- main shift. The over-focus on English data from natural and computer sciences is general feature of scholarly NLP. We hope that the future devel- opments in the field will enable cross-domaie and cross-lingual applications of NLP to citation text generation.
6
Conclusion
Citation text generation is a key task in scholarly document processing - yet prior work has been scattered across varying task definitions and evalu- ation setups. We introduced CITEBENCH: bench- mark for citation text generation that unifies four diverse task designs under a general definition, har- monizes the respective datasets, provides multiple basclines and a standard evaluation framework, and thereby enables systematic study of citation text generation.
The analysis results on citation intent and CORWA tagging are based on the output of the corresponding discourse tagging models. The per- formance of these models not perfect. For ci- tation intent classification, the best model by Co- han et al. (2019) on ACL-ARC has an Fl score of 67.9. In a further analysis of the classification rors, Cohan et al. (2019) note that the model tends to make false positive errors for the Backgr class and that it confuses Uses with Background. For CORWA tagging, Li et al. (2022) report an Fl score of 90.8 for their best-performing model. The limitations of automatically inferred labels need to be taken into consideration when interpreting the results. Given the data source overlap between the CORWA and ACL-ARC datasets and CITEB have attempted find overlapping instances to estimate the performance of the models on the subset of data used in CITEBENCH. Yet, the com- parison between CORWA and ACL-ARC test sets and CITE.BENCH test sets only yielded eight over- lapping instances in the LU dataset out of 14,459 total instances, preventing the further investigation. We leave the validation of our findings with human- annotated citation intents and discourse tags to fu- ture work. Furthermore, while we focus on sen- tences as an casy-to-obtain coarse-grained unit for discourse tagging, optimal granularity for citation analysis is another open research question to be investigated (Li et al., 2022).
Our analysis delivered new insights about the baseline performance and the discourse composi- tion of the datasets and model outputs. The base- line results show that simple extractive summa- rization models like LexRank perform surprisingly well. Furthermore, we observe non-trivial ability for transfer learning among the baselines. The dis- course analysis- based evaluation suggests that the models performing best in terms of ROUGE and BERTScores capture the natural citation intent dis- tribution in the generated texts.
Finally, our discussion outlines recommenda- tions and promising directions for future work, which include detailed reporting of evaluation pack- ages, the use of discourse-based qualitative metries for evaluation, and further refinements to the cita- tion text generation task definition. We argue that the current definitions for citation text generation overly simplify the complex process of composing a related work section, which hinders the devel- opment of successful real-world applications. In general, view our work represents crucial first step in addressing the extremely challenging prob- lem of Al-assisted writing related work. We invite the community to build upon CITEBENCH by ex- tending it with new datasets, trained models and metrics.
While our general task definition allows incorpo- rating any information from the target and source documents, we offer no standardized way to in-
"We calculate the normalized Levenshtein distance (using package: https: httss///mabachaaanm gi thub. io/ @evenshtiin/levessie html) with a threshold of 0.9 to identify the overlapped instances between two datascts.

---

clude structured information like citation graph context and paper metadata. Yet such information might be useful: for example, Wang et al. (2022) ex- plore the use of citation graphs for citation text gen- eration. We leave the standardization of additional information sources for citation text generation to the future work. Our baseline implementations limit the input sequences to 4,096 tokens, which only affects small portion of the data. This re- striction can be lifted long as the target language model can efficiently process long documents, and experimental time is not a concern even in a lim- ited setting, performing the full run of all fine-tuned citation text generation models in the benchmark is computationally expensive (Appendix A.2). Fi- nally, CITEBENCH inherits the structural limita- tions of the datasets it subsumes, e.g. not preserv- ing the full document information and document structure, and filtering out multimodal content. We leave the investigation of these extensions to the future.
corpus is released under the open non-commercial CC BY-NC 4.0 license" yet, we could not es- tablish the licensing status of the Delve corpus based on available sources. XING! based on the openly licensed ACL Anthology Network corpus but does not specify license for the annotations. We contacted the authors to obtain explicit permis- sion to use their annotations for research purposes. ABURAED do not attach license to the data, which is partly extracted from the ACL Anthology Net- work Corpus. We collected an explicit permission from the authors to use their dataset for research purposes. Our observations underline the urgent need for clear standartized data licensing practices in NLP. To work around the potential limitations on data reuse, CITEBENCH provides the instructions and script fetch and transform the data from the respective datasets. All datasets used in this work were created for the task of citation text generation and are employed according to their intended use.
Acknowledgements
Ethics Statement
This study is part of the InterText initiative at UKP Lab 16 It was funded by the German Federal Min- istry of Education and Research and the Hessian Ministry of Higher Education, Research, Science and the Arts within their joint support of the Na- tional Research Center for Applied Cybersecurity ATHENE; by the German Research Foundation (DFG) as part of the PEER project (grant GU 798/28-1) 1); and by the European Union as part of the InterText ERC project (101054961). Views and opinions expressed here are, however, those of the author(s) only, and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the grant- ing authority can be held responsible for them. The work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation.
Citation text generation is intended to support scholars in performing high-quality research, cop- ing with information overload, and improving writ- ing skills. Although do not foresee negative societal impact of the citation text generation task or the accompanying data per se, we point at the general challenges related to factuality and bias in machine-generated texts, and call the potential users and developers of citation text generation applications to exert caution and to follow the up- to-date ethical policies, incl. non- von-violation of intel- lectual property rights, maintaining integrity and accountability.
In addition, the raising ethical standards in the field put new demands on clear dataset licensing. While enforcing the license on third-party datasets presents organizational and legal challenge and lies beyond the scope of our work, we made the effort clarify the copyright status of the existing datasets to the best of our ability. Lu is distributed under the open MIT license"". "1 C openly dis- tribute their annotations for non-commercial use, while referring to the source datasets for license on the included research publications: the S2ORC
References
Ahmed AbuRa'ed ed, Horacio Saggion, Alexander Shvets, and Àlex Bravo. 2020. Automatic related work sec-
13 Ptt.e://rraaiiccmmmmoo.o/////eee
8/
" "nttps::/cpensooucee iicssesiss/// "https://github.com relatedworkgeneratioo
generation
15 "httts:///githbb https//ww...oommeededee/eeeeeeee 16 http://innerretttuuulaaa...ee ukp.

---

tion generation: experiments scientific document abstracting. Scientometrics, 125(3):3159-331 -1855
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre- tre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex: Humun Language Tech- nologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Akito Arita, Hiroaki Sugiyama, Kohji Dohsaka, Rikuto Tanaka, and Hirotoshi Taira. 2022. Citation sentence generation leveraging the content of cited papers. In Proceedings of the Third Workshop on Scholarly Doc- ument Processing, pages 170-174, Gyeongju, Repub- lic of Korea. Association for Computational Linguis- tics.
Shai Erera, Michal Shmueli-Scheuer, Guy Feigenblat, Ora Peled Nakash, Odellia Boni, Haggai Roitman, Doron Cohen, Bar Weiner, Yosi Mass, Or Rivlin, Guy Lev, Achiya Jerbi, Jonathan Herzig, Yufang Hou, Charles Jochim, Martin Gleize, Francesca Bonin, Francesca Bonin, and David Konopnicki. 2019. A summarization system for scientific documents. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNE tions, pages 211-216, Hong Kong, China. Associa- tion for Computational Linguistics.
Iz Beltagy, Matthew E. Peters, and Arman Cohan, 2020. Longformer: The long-document transformer. ArXiv, abs/2004.05150.
Steven Bird. Robert Dale, Bonnie Dorr. Bryan Gibson, Mark Joseph, Min- Yen Kan, Dongwon Lee, Brett Powley, Dragomir Radev, and Yee Fan Tan. 2008. The ACL Anthology reference corpus: A reference dataset for bibliographic research in computational linguistics. In Proceedings of the Sixth International Conference Language Resources and Evaluation (8))C'08), Marrakech. Morocco. European Lan- guage Resources Association (ELRA).
System Demonstra-
Günes Erkan and Dragomir R. Radev. 2004. Lexrank: Graph- based lexical centrality as salience text sum- marization. Journal of Artificial Intelligence Re- search, 22:457-479.
Isabel Cachola, Kyle Lo. Arman Cohan, and Daniel Weld. 2020. TLDR: Extreme summarization of sci- entific documents. In Findings of the Association for Computational Linguistics: Empirical Methods in Natural Language Processing (EMINLP) 2020, pages 4766-4777, Online. Association for Computa- tional Linguistics.
Yubin Ge, Ly Dinh, Xiaofeng Liu, Jinsong Su, Ziyao Lu, Ante Wang, and Jana Diesner. 2021. BACO: A background knowledge- and content-based frame- work for citing sentence generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 1466-1478, Online. Association for Computational Linguistics.
Xiuying Chen, Hind Alamro, Mingzhe Li, Shen Gao, Xi- angliang Zhang, Dongyan Zhao, and Rui Yan. 2021. Capturing relations between scientific papers: An abstractive model for related work section generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 1thh International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 6068-6077, Online. Association for Computational Linguistics.
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi. Anuoluwapo Aremu, Antoine Bosselut, Khy- athi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondfej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto, Yufang Hou, Yacine Jernite, Harsh Jham- tani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela. Khyati Mahajan, Saad Mahamood, Bod- hisattwa Prasad Majumder, Pedro Henrique Martins, Angelina McMillan-Major, Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Perez-Beltrachini, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Rodriguez. Sashank Santhanam. João Sedoc. Thibault Sellam, Samira Shaikh, Anastasia Shimo- rina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu. Diyi Yang. Akhila Yerukola, and Jiawei Zhou. 2021. The GEM benchmark: Natural language generation, its evaluation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation,
Arman Cohan, Waleed Ammar, Madeleine van Zuylen, and Field Cady. 2019. Structural scaffolds for ci- tation intent classification in scientific publications. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 3586-3596, Minncapolis, Minnesota. Association for Computa- tional Linguistics.
Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. A discourse-aware attention model for abstractive summarization of long documents. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 615-621, New Or- leans, Louisiana, Association for Computational Lin- guistics.

---

Evaluation, and Metrics (GEM 2021). pages 96-120, Online. Association for Computational Linguistics.
Xiangei Li and Jessica Ouyang. 2022. Automatic related work generation: A meta study. ArXiv, 0bs/2201.01885
Sehastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-man majoo, Anna Shvets, Ashish Upadhyay, and Bernd Bohnet. 2022. GEMv2: Multilingual NLG benchmarking in a single line of code. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing: System Demonstrations, pages 266-281, Abu Dhabi, UAE Association for Computational Linguis- tics.
Chin- Yew Lin. 2004, ROUGE: A package for auto- matic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguisties.
Yao Lu, Yue Dong, and Laurent Charlin. 2020. Multi- XScience: A large-scale dataset for extreme multi- document summarization of scientific articles. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). pages 8068-8074 Online. Association for Computa- tional Linguistics.
Cong Duy Vu Hoang and Min- Yen Kan. 2010. Towards automated related work summarization. In Inter- national Conference on Computational Linguistics (COLING) 2010: Posters, pages 427-435, Beijing, China. Coling 2010 Organizing Committee.
Junyu Luo, Junxian Lin, Chi Lin, Cao Xiao, Xinning Gui, and Fenglong Ma. 2022. Benchmarking au- tomated clinical language simplification: Dataset, algorithm, and evaluation. In Proceedings of the 29th International Conference on Computational Lin- guistics, pages 3550-3562, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
Yue Hu and Xiaojun Wan. 2014. Automatic gener- ation of related work sections in scientific papers: An optimization approach. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6624-1633. Doha, Qatar. Association for Computational Linguis- tics.
Kelvin Luu, Xinyi Wu, Rik
Kyle Lo, Isabel Cachola, and Noah A. Smith. 2021. Ex- plaining relationships between scientific documents. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Lan- guage Processing (Volume 1: Long Papers), pages 21 2130-2144, Online. Association for Computational Linguistics.
David Jurgens, Srijan Kumar, Raine Hoover, Dan Mc- Farland, and Dan Jurafsky. 2018. Measuring the evolution of scientific field through citation frames. Transactions of the Association for Computational Linguistics, 6:391-406.
Douwe Kiela, Max Bartolo, Yixin Nie Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vid- gen, Grusha Prasad, Amanpreet Singh, Pratik Ring- shia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams, 2021. Dynabench: Rethinking benchmarking in NLP. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies. pages 4110-4124, Online. Association for Computa- tional Linguistics.
Yuning Mao, Ming Zhong, and Jiawei Han. 2022. Cite- Sum: Citation text-guided scientitic extreme sum- marization and domain adaptation with limited su- pervision. In Proceedings of the 2022 Conference Empirical Methods in Natural Language Process- ing, pages 10922-10933 10935, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bring- ing order into text. In Proceedings of the 2004 Con- ference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Asso- ciation for Computational Linguistics.
Anne Lauscher, Brandon Ko, Bailey Kuehl, Sophie Johnson, Arman Cohan, David Jurgens, and Kyle Lo. 2022. MultiCite: Modeling realistic citations requires moving beyond the single-sentence single- label setting. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 1875-1889, Seattle, United States. Association for Computational Linguistics.
Nafise Moosavi, Andreas Rücklé, Dan Roth, and Iryna Gurevych. 2021. Scigen: dataset for reasoning- aware text generation from scientific tables. In Pro- ceedings of the Neural Information Processing Syz- tems Track on Datasets and Benchmarks, volume 1. Online.
Xiangci Li, Biswadip Mandal, and Jessica Ouyang. 2022. CORWA: A citation-oriented related work annotation dataset. In Proceedings of the 2022 Con- ference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies. pages 5426-5440, Seattle, United States. Association for Computational Lin- guistics.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorn,, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a benchmark for knowledge weoleege intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human

---

Language Technologies, pages 2523-2544, Online. Association for Computational Linguistics.
A
Appendix
A.1
Data transformation
Vahed Qazvinian and Dragomir R. Radev. 2008, Sci- entific paper summarization using citation summary networks. In Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008), pages 689 696, Manchester, UK. Coling 2008 Organizing Committee.
Prior work not only explores a wide range of task definitions for citation text genration, but also em- ploys multitude of formats to serialize the task information in the data. Thus, to construct unified benchmark, we also needed to unify the data. The data format used in CITEBENCH is a concatenation of different types of inputs embedded into tags special tokens that denote which input is stored within. The abstracts of the cited papers are always available and are not embedded into a tag. The final CITEBENCH input is assembled as follows, depending on the availability of the inputs in the source dataset's task definition:
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference Empirical Methods in Natu- ral Language Processing, pages 2383-2392 Austin, Texas. Association for Computational Linguistics.
Darsh J. Shah and Regina Barzilay. 2021. Generating related work. ArXiv, abs/2104.0866 086688.
Edward Sun, Yufang Hou, Dakuo Wang, Yunfeng Zhang, and Nancy X. R. Wang. 2021. D2S:
<abs></abs>> the abstract of the citing paper <<tx_ <ctx_b></ctx_b> b>> the sentence(s) before the
Document-to-slide generation via query-based text summarization. In Proceedings of the 2021 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, pages 1405-1418, Online. As- sociation for Computational Linguistics.
citation text in the citing paper
the abstract(s) of the cited paper(s) discussed in the citation text (without a surrounding tag) <etx_a><<<> a>> the sentence(s) after the ci- tation text in the citing paper
Simeng Sun, Ori Shapira, Ido Dagan, and Ani Nenkova. 2019. How compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature. In Proceedings of the Workshop on Methods for Optimizing and Eval- uating Neural Language Generation, pages 21-29, Minneapolis, Minnesota. Association for Computa- tional Linguisties.
The CITEBENCH data stores these input com- ponents as a list. To pass the data to the model, we simply concatenate the items on the list. addition, for each instance we store the gold refer- ence target text, which is used as output example at the training stage, and as an evaluation target at the testing stage. Table 5 provides example of an instance coded according to the CITEBENCH schema.
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A muulti-task benchmark and analysis platform for nat- ural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355 Brussels. Belgium. Association for Com- putational Linguistics.
Unification required us to process the original data. No instance-level filtering was applied, i.c. every instance from the source datasets is contained in the benchmark. All changes to the underlying datasets are documented in the automatic scripts supplied with this paper, which can be used to re- construct CITEBENCH from the sources. The para- graphs below briefly outline the main modifications that these scripts perform.
Yifan Wang. Yiping Song, Shuai Li, Chaoran Cheng, Wei Ju, Ming Zhang, and Sheng Wang. 2022. Dis- encite: Graph-based disentangled representation learning for context-spp -peeext citation generation. In Proceedings of the AAAI Conference on Artificial In- telligence, volume 36, pages 11449-11458, Online.
For all datasets, we unify the citation anchor format. In CITEBENCH, cited papers in the cita- tion text are represented by an integer placeholder (e.g. [O], [1]) which increments with each new en- countered paper in the input text, but is consistent throughout the text. For example, if paper "[O]" was encountered twice, this would result in a ci- tation text going as "As it has been shown in [0], [1], [2]... In particular, [0] demonstrate that.."" In XING the citations of papers that are not included in the input are marked in the original data using a
Xinyu Xing, Xiaosheng Fan, and Xiaojun Wan. 2020. Automatic generation of citation texts in scholarly papers: A pilot study. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6181-6190. Online. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoay Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In Inter- national Conference on Learning Representations, Online.

---

Input
Target
["<abs> The SRI Core Language Engine (CLR) a general-purpose natural anguage front end for interac- tive systems. It translates English <..> is described and evaluated. </abs>".
The idea is similar to Finkler and Neumann #OTHEREFR, though simplified for our purposes; [0] his VEX system also uses the method of giving sunple questions to the user (supposedly non- con-linguist) to learn about word is behaviour, but is for English and primarily intended for assigning syntax properties rather than morphological.
"ectx_bs be The system also facilitates the process of adding word forms to the user is own dictionary. <..> how to ac- complish this task with the user is assisstance. </ctx b>". pppelling-cheekee have hecome an integral part of most text processing software. <..> a special method has been developed for easy word classification.",
"<ctx_a> The huplementation the algoritlma <..> it should he included in the second version). </ctx_a>> _ctx_a>>]
Table 5: Example CITRBENCH instance taken from the XING dataset.
placeholder #OTHEREFR; these tokens are kept in the benchmark.
gold reference and it is copied over to the "tar- get" field in CITEBENCH, representing the gold reference. The sigt_abstactt (cited paper's ab- stract) is wrapped in "<abs></abs>" tags, the _eext_befor___e www.oooeeeloliiitteeeeeeeeeeeeee _ee_eeeee_eeitatiiiiee
ABURAED dataset operates with single cited docu- ments, providing their abstract and title, and aims to generate single citation sentence. We use the script from the original study!? to extract the input and the gold reference. The gold reference is used directly to populate the "target" field. The input is constructed by concatenating the title and the abstract of the cited paper.
(sentences before citation text) wrapped in "<etx_ bttp_b>//cx_bbbbee <_>>">>"""__ tags, "text_after_ eettereexpiiittt__ citation" (sentences after cita- explicit_citation"
tion text) is wrapped in <xct__a><<<> __>" tags, These items are added to the "input" list together with the "src_absti" "rr_abbtract" (cited paper's abstract), us- ing the tags and order described above.
Both CHEN datasets operate with multiple cited documents, storing their abstracts, and have ci- tation paragraph as the target granularity. CHEN uses field named "abs" to denote the citation text to generate, which is not to be confused with the "<abs><</bss" tag used in CITEBENCH denote the citing paper abstract. This field is used as gold reference. To construct the input, we extract the "multi _diiiddooo" field from CHEN data. The abstracts the individual documents in the source data be- gin with a special marker "[x]". We remove this marker and concatenate the inputs, maintaining the original order of the abstracts.
A.2 Model training
For the LED models, the following hyperparame- ters were used: Encoder length 4, 096, Decoder length 1, 024, Decoding type beam search, 2 beams, length penalty 2, with early stopping, no repeat n-gram size 3, learning rate 5e To make the study feasible given the number of experiments in this work, no hyperparameter search was per- formed. The * #led-base and -led-large-arxtr model were fine-tuned for 9 days 4 GPUs; each of the models fine-tuned on individual datasets *led-base- where trained for a maximum of 3 days on 4 GPUs (most of them took less than 3 days to train). Using the upper bounds that would result in (9d 9d 4 x 3d) x 4 GPUs x24hours 2880 GPU hours in total for model fine-tuning.
LU also follows multiple-abstrace citation text paragraph generation setting. We take the "re- lated_work" field from the original data and use it as our golden reference "target". From the abes_actt dictionary the citation markers and cited paper abstracts are extracted. The citation markers are converted to indexes e.g. "[O]" and the abstracts are ordered according to these indices. The "abstract" (citing paper's s abstract) and the cited paper's s abstracts from "ref abstract" are concate- nated into the model input according to the proce- dure described above.
A.3 ROUGE calculation
The ROUGE scores are calculated using the Hug- gingface evaluate library with package versions: r0uge-score==000.44 and evaluate==0.1.2. The scores that are reported are fmeasure for the mid scores (see the library documentation for de- tails). See Tables and 7 for detailed ROUGE-L, ROUGE-1 and ROUGE-2 results.
Lastly, for XING the "explieit_citation" is the
1 httttp:://ittuubcoom/// wwtptttuu.ccmmmeeeee http:///www.eeeeuueee///eeeeeae hlbb///msteerrprpprocssssttr

---

Medel
ABURAED
CHEN Delve
CHEN S20RC
LU
XING
M0157227711
2E.07447 TX
149872.22 10.55
15,90/2.00/
23.79/3.12122
22.35/3.10/ 12.85
lod-base
11.06/13 9.72//666
7.16/0.44
0,75 75.41
0.73/135/7.36
12.36/ 1.66/ 10.07
led large
10.39 /1 1.36/
7.85 /0.66 66.22
7.56/0.99/ 0.99/4.3.
8.847 1.25
11.6111 79/9
led-Larger e
15.55 1,89 10.23
/5.78 15.37
23,45/3,30/ 12.89
26.27 /4.44/ 14.41
14.57/2.04 / 10.23
#Led-base
15.94
3.03
18.88
(0.00)
(0.094)7-2 16.27
(0.055)
(0,007))
(0.09))
-lod-Largeaarsdy
26.70 6.07
17.90/
(0 (0000) (0085)) (0.011)
(0.007)) (0.002) (0/00))
(0.07) (0.00/00000)0000000 (0.000))
(0,010) (0400) (0.00))
(0.012)/ (0.009) (0.008)
Table 6: Detailed baseline performance, ROUGE-1/2/L, average over three runs, best scores are bolded. For readability, only standard deviation higher than 10 is shown (in parentheses).
A.4 Transfer Learning Results
the text readable but has some major grammatical or fluency issues;
A possible explanation for the better transfer per-
4: the text is readable and has only one or two
formance between ABURAED and XING (discussed in
minor grammatical or fluency issues; 5: the text is perfectly readable.
Section 4.2 and seen in Table and 7) might lie in
the behavior of the ROUGE metric: as observed by
Consistency scores were defined
Sun al. (2019), ROUGE tends to award higher scores to texts that are around 100 tokens long. The average lengths of the outputs for ABURAED and XING are 47.5 and 38.3 words respectively. For CHEN Delve, CHEN S20RC and LU the average out- put lengths are 265.8, 186.3 and 140.2 words re- spectively. For the inputs the average lengths for ABURAED and XING are 65.3 and 145.9 words re- spectively, while for CHEN Delve, CHEN S20RC and LU the lengths are 691.4, 1230.1 and 676.1 words respectively.
the text is related the gold reference; the text is topically related to the gold ref- erence but misses all key aspects and the cor- responding factual details about the cited pa- pers;
3: the text is topically related to the gold ref- erence but misses misrepresents a few key factual details about the cited papers;
4: the text topically related the gold ref- erence and captures all key aspects about the cited papers, but it misses some factual de- tails:
A.5 Human evaluation details
5: the text conveys the same message and detail as the original text.
For the human evaluation we focus on the datasets XING and CHEN DELVE as these are the datasets with most divergent task definitions, and the models LexRank, Led-large-arxiv and -led-large-arxiv these are the systems with highest performance their respective baseline cat- egories. For each dataset, 25 data instances were randomly selected and for each of these instances the output from each model was generated. This resulted in 2 3 25- = 150 instances to annotate. The instances were shuffled and distributed among three annotators (paper authors), fluent non-native English speakers with background in computer sci- ence and natural language processing, who were asked to rate each instance according to its read- ability and consistency with the gold reference. For both dimensions we have used a 5-point scale. Re- sults for the human evaluation are shown in Table 8. Readability scores were defined as:
To estimate the agreement between the anno- tators, we randomly selected 25 instances from each of the two annotators and B, that were re- labeled by the annotator C, resulting in a total 50 instances. The Quadratic Cohen's 's kappa for the annotator pair (A- was measured at 0.48 for read- ability and O. 18 for consistency; the annotator pair (B-C) yielded the agreement of 0.52 for readability and 0.42 for consistency. We note the discrepancy with regard the consistency score between an- notator pairs! which indicates that compared readability, consistency a less intuitive notion and might require a more strict definition or training in future human evaluations.
A.6 Discussion examples
In Table 9 two examples from the benchmark are shown where the input is not sufficient to generate the output. It is not reasonable to expect model
I: the text lacks basic grammatical coherency and not readable;
2: the text lacks basic grammatical coherency but is readable with some effort;
19 Upon close examination, annotator C. tends to he stricter in evaluating the consistency score compared to annotator A.

---

Model
ABURAED
CHEN Delve
CHEN S20RC
LU
XING
+led-base-abbu
19,65
15.96
2.544/2.54/11
1.46 12.63
-le--base-chen
17.66/ 2.41/ 11.48
30.82/ 5.58 16.04
27.28
5.89 16.21
28.92 4.80/ 15.40
14.51 1.89 10.14
+led-base-lu
72.78/2.32/ 12.19
26.97 3,97 14.58
3.790/3.72 14.59
28,94
5.00 16.25
17.50/
1,92/ 11.56
1led-base-xing
16.56 1.32/1///111 11.97
16.01 1.879955
16.09/ / 1.89 10.01
14.87/
/981/
19.77 2,80 14,06
Table 7: Detailed transfer learning performance, in- and cross-dataset, ROUGE-
Dataset
Model
Readability
Consistency
AlI
All
3.73 1.19
2.04 0.84
All
LexRank
3.42 a: 1.09
2.04 0.75
All
Led-Large-arxiv
3,94 1.00
2.18 = 0.85
All
*Led-Large-arxiv
3.90 1.39
1.94 0.91
XING
All
3.64 1.30
1.92 + 0.85
CHEN DELVE
All
3.87 1.06 1.06
2.19 0.82
KING
LexRank
3.56 1.12
2.04 0.79
KING
led led-large-rrxiv
3.88 1.01
2.20 0.87
XING
lldd-large-arxiv
3.48 1.69
1.52 0.77
CHEN DELVE
LexRank
3.28
2.04 0.73
CHEN DELVE
led-large-arxiv
4.00 1.00
2.16 0.85
CHEN DELVVE
+led-large-arxiv
4.32 0.85
2.36 0.86
Table 8: Human evaluation results; average + standard deviation.
be able generate these data instances. Table 10 provides examples of self-reference, and Table provides self-reference statistics.

---

Input
Target
Comment
We present the syntax-based string-totree statisti- cal machine translation systems built for the WMT 2013 shared translation task. Systems were devel- oped for four language pairs. We report on adapting parameters, largeted reduction of the tuning set, and post-evaluation experiments on rule binarization and preventing dropping of verbs.
is worth noting that the Ger- man parse trees [0] tend to he brouder and shallower than those for English.
From this input we can not de- termine whether German parse trees are hroader and shallower than English parse trees.
Data selection an effective approach to domain adaptation in statistical machine translation. The idea is to use language models trained on small in- domain text to select similar sentences from large general-domale
Analyses have shown that this augmented data can lead to bet- ter stutistical estimation or word coverage [0].
Here we do not know what "this" refers to in the target and the input does not mention any- thing about "better statistical es- timation" or "word coverage"
corpora, which are then incorpo- rated into the training data. Substantial gains have been demonstrated in previous works, which em- ploy standard ngram language models. Here, we explore the use of neural language models for data selection. We hypothesize that the continuous vec- for representation of words in neural language mod- els makes them more effective than n-grams for modeling unknown word contexts, which preva- lent in eeneral-domain text. In comprehensive evaluation of language pairs (English to German, French, Russian, Spanish), found that neural language models selection: while the improvements varied (i.e. 0.1 to 1.7 gains in BLEU), they fast train small in-domain data and sometimes substan- tially outperform conventional n-grams.
indeed viable tools for data
Table Examples of targets that contain information missing from input.
Samples self reference sentences in the reference citation text
(1) In our context, we call upon computer vision teclniques to study the cephaloocular behavior of drivers.
(2) The main objective of our work is the elaboration of new computer vision system for evaluating and improving
driving skills of older drivers (age between 65 and 80 years).
(3) The features used this work complex and difficult interpret and it isn't 't clear that this complexity is required.
Our approach enables easy praning of the RNN decoder equipped with visual ottention, whereby the hest number of weights to prane each layer automatically determined.
Table 10: Self-reference sentence examples; (1) and (2) from Chen et al. (2021); (3) and (4) from Lu et al. (2020).
Dataset
Data points
Ref to citing paper
Prevalence
ABURAED
16,603
143
0.86%
CHEN Delve
78,927
11,787
14.93%
CHEN S20RC
136,655
6,711
4.91%
LU
40,528
4,470
11.03%
XING
86,052
1,475
1.71%
Total
530,869
28,167
5.31 %
Table 11: Self-reference sentence statistics.


Reviews:
[{"review_id": "DeG1gwhhKG", "reviewer": "Reviewer_GP3Z", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank you for taking your time and are happy to hear that you appreciated our work! We will follow your suggestion to improve readability of the tables. \n\nRegarding the color scale in Figures 3 and 4, we opted for using different color scales since the magnitude of KL divergence differs between CORWA and citation intent classifiers.The values for citation intent are on average much lower than the values for CORWA classification. Our main goal here is to compare the differences between the models and not between CORWA and citation intents. Thus, if the same color scale were used for both plots, all the cells in Figure 3 would be close to white, making it difficult to visually compare the models on the citation intent dimension. We will think about how to highlight the scale difference in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper introduces a clever formalization of the task of Citation Text Generation that allows the unification of several existing resources and the standardization of the evaluation of the task.\n\n2. The paper is very well written: it explains the task clearly, the figures are well-made and are very useful to understand the paper and the overall presentation is very clear.\n\n3. The experiments in the paper are very solid: they described clearly the variability of their results, specified the training of the models completely, averaged their results over many runs, specified their computational budget and justified well the use of quantitative and qualitative tools.", "reasons_to_reject": "I don't see any reason to reject this paper.", "questions_to_authors": "", "paper_topic_and_main_contributions": "# Paper Summary\n\nThe paper introduces a novel dataset for the task of Citation Text Generation of scientific documents. Its main contribution is a formalization of the task which allows the unification of several existing datasets and the standardization of the evaluation of the task. They also test several models from the state of the art on the resulting dataset and perform a comprehensive analysis of the results via qualitative and quantitative results.\n\n# Contributions \n\n1. A clever formalization of the task of Citation Text Generation that allows the unification of several existing resources and the standardization of the evaluation of the task.\n2. A large-scale dataset for the task of Citation Text Generation of scientific documents.\n3. An standardized evaluation methodology for the task based on quantitaive and qualitative metrics and human analysis.\n4. A range of baseline implementations of models from the state of the art to foster research in this task.", "excitement": 5, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. To improve the readability of the tables, I suggest aligning the numbers to the right and text to the left.\n\n2. It would be useful to present the right part of Figures 3 and 4 in the same color scale to make it easier to compare the results."}}, {"review_id": "1MIqTt8ZzT", "reviewer": "Reviewer_cNNt", "summary": "", "questions": "A: What are the core distinctions that you perceive between your paper and previous works on citation generation datasets in this field?\n\nB: Is it feasible to apply the general text generation benchmark population methods to this specific area?\n\nC: What about exploring a broader range of state-of-the-art generation models for the citation text generation task?", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking your time and giving feedback on our work, we appreciate the detailed comments. Please see our responses to your comments below.\n\n**How the four datasets were transformed into the new task**\n- We are happy to clarify here and in the appendix. Each of the datasets was adapted to the new task definition by taking the different types of inputs e.g. *cited documents* and *abstract of citing paper* and concatenating these into a list that serves as input to the model along with special separators to distinguish the items, such as “<abs></abs>”, for the citing paper’s abstract. This transformation process will be included in the code that we will release along with the paper upon acceptance.\n\n**The problem formulation presented in this paper lacks novelty**\n- Our main goal in this paper is unification of the existing task definitions rather than designing a novel task definition. Our motivation is that there are a lot of papers that define the citation text generation task differently, and with all the different task definitions it becomes difficult to compare solutions and determine the SOTA. The unified task definition facilitates comparison between different approaches to citation text generation, which is important because in a real world scenario we want to know which solution would work best in a given setting.\nFurthermore, new datasets in this area can be easily integrated into our benchmark by extending the cited paper representation set D and the citing context set C (Line 173-176), our unified dataset structure allows to integrate new definitions that include additional textual information in the future, such as whole texts from the cited papers, or citation intents. \n\n**About the core distinctions of our paper and previous works on citation generation datasets in this field and more comprehensive discussion**\n- First, much of previous work has defined its own version of the task, created a dataset according to that task definition and designed a model to solve the task. This has led to a large number of highly divergent task setups and datasets, making it hard to compare approaches to citation text generation. Rather than focusing on creating a new task and dataset, here we examine and compare the current approaches. We also propose new evaluation metrics to look at the results from the perspectives of citation intent and discourse structure, which, as far as we know, has not been done in the field of citation text generation. \n- Second, writing a good related work section requires a very high level of expertise which requires a mental model of a big picture of the field. There is a wealth of literature on how to write a good literature review and on studying the elements/structures of different types of literature reviews [1] [2] [3]. In general, writing a literature review is closely related to writing a related work section. Unfortunately, the current task definitions in previous NLP studies on citation text generation overly simplify this very complex process, which hinders the development of successful real-world applications. In our work, the new knowledge we contribute to this field beyond a unified benchmark is discussed in Section 5: 1) we report the problematic​ ROUGE calculation practice in the previous studies on citation text generation; 2) more importantly, through our human evaluations, we identify a few problems of the existing datasets, and question the current definitions for the citation text generation task. In general, we view our work represents a crucial first step in approaching the extremely challenging problem of AI-assisted writing related work. We will make it more visible in the abstract and the introduction.\n\n**Is it feasible to apply the general text generation benchmark population methods to this specific area?**\n- In principle, general text generation benchmark population methods such as GEM v2 [4] can be applied to this specific area. The problem is that if each dataset owner contributes their dataset directly to the GEM benchmark using different task definitions, we still end up with a large number of highly divergent task setups and datasets. One of the core contributions of our work is to provide **a unified and standardized dataset structure** for citation text generation which allows researchers to compare different approaches easily. \n\n**What about exploring a broader range of state-of-the-art generation models for the citation text generation task?**\n- The current paper is a benchmarking paper which contributes the resources including dataset, evaluation metrics as well as solid baselines for other researchers to evaluate their models on. Similar to other benchmarking papers such as SQuAD [5], GLUE [6] and KILT [7], our benchmark and standardized evaluation enables systematic comparison of a wide range of generation models for citation text generation in the future.\n\n**About the missing references**\n- Thank you, we will add those to our related work overview.\n\nReferences:\n\n[1] JRandolph, Justus (2009) \"A Guide to Writing the Dissertation Literature Review,\" Practical Assessment, Research, and Evaluation: Vol. 14, Article 13. \n\n[2] Diana Ridley. 2012. The Literature Review: A Step-by-Step Guide for Students. SAGE Publications.\n\n[3] Maria J. Grant and Andrew Booth. 2009. A typology of reviews: An analysis of 14 review types and associated methodologies. Health Information & Libraries Journal, 26(2):91–108.\n\n[4] Sebastian Gehrmann, Abhik Bhattacharjee, Abinaya Mahendiran, Alex Wang, Alexandros Papangelis, Aman Madaan, Angelina Mcmillan-major, Anna Shvets, Ashish Upadhyay, Bernd Bohnet, Bingsheng Yao, Bryan Wilie, Chandra Bhagavatula, Chaobin You, Craig Thomson, Cristina Garbacea, Dakuo Wang, Daniel Deutsch, Deyi Xiong, et al.. 2022. GEMv2: Multilingual NLG Benchmarking in a Single Line of Code. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 266–281, Abu Dhabi, UAE. Association for Computational Linguistics.\n\n[5] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383–2392, Austin, Texas. Association for Computational Linguistics\n\n[6] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353–355, Brussels, Belgium. Association for Computational Linguistics.\n\n[7] Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. 2021. KILT: a Benchmark for Knowledge Intensive Language Tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2523–2544, Online. Association for Computational Linguistics.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) This paper is well-organized, and easy to follow.\n\n2) This paper is relevant to the topics of the conference.\n\n3) It is meaningful to bridge NLP and bibliometrics.", "reasons_to_reject": "The paper first redefines the task, and populates the four existing different existing benchmarks without introducing the detail of the task specific modification. The authors introduce a new task benchmark however the four origin benchmarks are from different tasks. How to populate the four different datasets to fit the new task formulation is not clearly introduced.\n\nThe problem formulation presented in this paper lacks novelty and does not show significant differences compared to other citation generation tasks, such as generating citations for related works.\n\nMany existing studies are merely listed without undergoing in-depth analysis.\n\nThe findings appear to be trivial, and the paper would benefit from a more comprehensive discussion.", "questions_to_authors": "A: What are the core distinctions that you perceive between your paper and previous works on citation generation datasets in this field?\n\nB: Is it feasible to apply the general text generation benchmark population methods to this specific area?\n\nC: What about exploring a broader range of state-of-the-art generation models for the citation text generation task?", "paper_topic_and_main_contributions": "In this research, CITEBENCH is introduced as a comprehensive benchmark for citation text generation, consolidating diverse datasets and facilitating consistent evaluation of citation text generation models across various task designs and domains. By employing this novel benchmark, the authors thoroughly examine the performance of several robust baselines, assessing their adaptability across different datasets. Furthermore, the study provides valuable insights into task definition and evaluation methods, offering guidance for future research in the field of citation text generation.\n\nThis paper presents a comprehensive set of experiments conducted to evaluate the newly integrated benchmark.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "N/A", "Justification_for_Ethical_Concerns": "N/A"}, "extra_text": {"Missing_References": "Mao, Y., Zhong, M., & Han, J. (2022, December). CiteSum: Citation Text-guided Scientific Extreme Summarization and Domain Adaptation with Limited Supervision. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (pp. 10922-10935).\n\nWang, Y., Song, Y., Li, S., Cheng, C., Ju, W., Zhang, M., & Wang, S. (2022, June). DisenCite: Graph-Based Disentangled Representation Learning for Context-Specific Citation Generation. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 36, No. 10, pp. 11449-11458)."}}, {"review_id": "0eHn1xn6ZD", "reviewer": "Reviewer_Z3w2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking your time and thoroughly reviewing our work, we appreciate your detailed feedback and respond to it below, starting with minor points.\n\nWe will mention the lack of humanities and the limited citation style diversity as a data limitation, and restate the insights on the baseline performance and discourse composition in the Conclusion. We will also add information on the number of epochs. We will fix the typographical error in Table 2, thanks for spotting it! Correct values are 0.006% for the total inputs with more than 4096 tokens; the reason that the total outputs over 1024 tokens is 0% is because we round any value below 0.001 to zero. We only report RL scores in the main tables to save space. We will explicitly redirect the reader to the appendix that also provides R1 and R2 ROUGE scores. Regarding the intents (lines 529-531), what we suggest is explicitly providing intent as part of the concatenated model input. This means that it would not be an auxiliary task but rather an extension to the input of the current task. Alternatives are of course possible! We will clarify this in the text.\n\n\n-- **Regarding the intent labels used in our work**\n\nThe original paper for SciCite provides two models. Here we only use the ACL-ARC model from that work, hence we only report the performance of 67.9. We chose the ACL-ARC label set due to its higher granularity, which we deemed desirable for our task. We will mention this in the main text of the paper to avoid confusion.\n\n\n-- **Regarding human validation for the citation intent and CORWA labelling**\n\nWe adopt citation intent and CORWA schemata from prior work and rely on the performance measurements from prior work to judge their validity for our task. A systematic, non-anecdotal human evaluation of citation and CORWA performance would require substantial resources and lies beyond our scope. We will highlight this point in the limitation section.\n\n\n-- **Regarding inter-rater reliability**\n\nOriginally 150 instances were annotated by three annotators (A, B, C) and each annotator annotated a distinct set of 50 instances. We conduct additional human annotations by randomly selecting 25 instances from each of the first two annotators' sets,  and then annotator C annotate these 50 instances. The Quadratic Cohen’s kappa is 0.48 (readability) score and 0.18 ( consistency) between the first annotator pair (A-C), as well as 0.52 (readability) and 0.42 (consistency) between the second annotator pair (B-C). The reason that the consistency agreement score is a bit lower for the first annotator pair A-C is that annotator C is a bit stricter with the consistency score than annotator A. We will report this inter-annotator reliability study in the final version. \n\n\n-- **Regarding the task definition**\n\nIndeed, we want to “generate a citation text” that is as close as possible to the original “$ T \\in D^s $” – but the generated text does not itself belong to the document. The citing document contexts are *distinct* from the citation text e.g. “$ T \\notin \\{C_1,...,C_m\\}  $”. We will clarify this in the camera-ready version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is an insightful, well-written, and well-designed benchmark. The work positions itself well and responds to other works investigating this task by solving a number of problems for the community: (1) how to unify structurally different citation text generation datasets? (2) how to evaluate citation text generation (3) what are some of the main directions we should be considering in this task? (Section 5.4) The resulting benchmark, evaluations, and baselines will be valuable to the community working on this task. The baselines are thorough including performance evaluations on indirect supervision and OOD settings and a human evaluation study. The evaluations also include insightful discussions about issues with ROUGE (5.1) as well as a novel KL divergence metric which already reveals in this study a number of interesting problems for future studies to follow up on.\n\nIn addition to proposing the above, the authors provide a number of interesting insights like Section 4.3, 382-385, Figure 3 + 4: Insights about differences between generative models and gold reference discursive roles. Insights about what information is required for generating citations (Appendix A.2) and the presence of self-reference in citation generation datasets (Appendix A.2). The limitations and ethics statements also include important information for people considering working on this task.\n", "reasons_to_reject": "This paper has very few weaknesses. One minor weakness I have identified is: (1) There was no validation done to check if the Scicite or CORWA labeling of either the benchmark or the generated texts actually reflect how humans would label them. SciCite and CORWA are S2ORC based, so I do think the authors could do some automatic validation to check the KL divergence between samples which have gold labels as reference points for the readers. ", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper contributes a large benchmark of citation text generation unifying four structurally different previous datasets (see Figure 2). The authors provide an evaluation and analysis kit for both quantitative and qualitative evaluation on this task including a novel measure on discursive behavior of citation text generation mechanisms based on KL divergence between source corpora and generated text intent and composition classification. The authors provide a comprehensive set of baselines for future community members to build on as well as a number of suggestions of future directions that can be explored using this benchmark.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "542: It would be nice if you reiterated those insights here.\n\n549: (Just a small comment) An additional limitation that might be worth mentioning is you are missing humanities and social sciences domains (for example in humanities Chicago style citations has page numbers extensively, which could be interesting for the citation generation task!).\n\n176: I think the general definition is good but the statement “generate a citation text $T \\in D^s$” needs work. To me, this is ambiguous: Are we trying to generate T such that T is as close to a citing document context $C_1^s … C_m^s$? Or a T such that T is faithful to what is said in the cited target documents? Or such that T has high utility for a user? Or what? In practice, I think the training objective is that you want to generate a T as close as possible to one of C (cited document context) so it might just be simplest to add this to the definition to clarify what you are after in the training case. In the inference case, you don’t have a C to compare with so not exactly sure what you can provide other than maybe to suggest its faithful to the D (target documents) that it is citing…\n\nTable 2: How is the total of S2ORC input tokens over 4k tokens is 0.018% and everything else is 0% but the total is  0.005%. Same questions for the outputs where there are 0.004% in the Delve outputs but 0% in the outputs? Perhaps you did some filtering that should be mentioned in Section 3.1 around 199? If my understanding is correct you only collected permission for ABURAED (623) and Xing(618)  so it can’t be filtering out by licenses?\n\n\n313: Is there a particular reason why R1 and R2 are not reported as well just RL ? I do see that there are large differences in scores for R-L so perhaps its suffecient as a metric? But i’d like to see explicit reasoning here. I do realize it reported in Table  6 and 7 where you could maybe direct curious readers to?\n\n254-258: Well justified!\n\n265: it's 67.9 on the ACL report but 84.0 on Scicite - Maybe you can clarify this? 67.9 sells the validity of your discourse evaluation method short since there is a higher score you could report. Ultimately, I think its best to report both to illustrate the model’s performance variance across datasets. Unless you only used the ACL model and not the scicite one which I think you should make clear here. I realize this is discussed 557-558 but I think could be clarified here.\n\nLike: 316, section 5.1, Section A. 5 insightful discussion on ROuge differences\n\n929: How many epochs was this?\n\n442-445,454-458: well argued\n\n529-531: I am having difficulty imagining what you have in mind here specifically? Like an auxiliary classification task or what?\n\nSection A.4: This appendix is clear and well written. I am curious about any statistical significance testing or inter-rater reliability you did to evaluate the scores? Can you provide any of those?\n"}}]

Author rebuttal:

Meta-review: This paper presents CITEBENCH, a novel benchmark for citation text generation. The authors propose an approach for merging and harmonizing diverse datasets, ensuring a consistent evaluation of models across different tasks and domains. The work also includes an exhaustive examination of several robust baselines, offering insights into their adaptability across different datasets. The paper is well-received with some minor concerns raised by the reviewers. The reviewers praised the novelty of the approach, the clarity of presentation, and the thoroughness of the experiments described. The reviewers also highlighted the potential of this work to become a reference benchmark in the field of Citation Text Generation.

The paper introduces a novel dataset for Citation Text Generation and provides a clever formalization of the task that allows the unification of several existing resources and the standardization of the evaluation of the task. The paper contributes significantly to the field by providing an exhaustive set of baselines for future work, a standardized evaluation methodology, and a comprehensive analysis of the results. The quality of the paper is high, with clear explanations, well-made figures, and solid experiments. On the other hand, one reviewer raised concerns about the clarity of task-specific modifications when populating the new benchmark from different existing benchmarks. There were questions about the novelty of the problem formulation and a perceived lack of depth in the analysis of existing studies, which was tried to answer in rebuttal. A minor weakness identified by a reviewer is the absence of validation done to check if the labeling of the benchmark or the generated texts actually reflects how humans would label them.

Given the novelty and potential impact of the work, as well as the overall positive feedback from the reviewers, the paper is recommended for acceptance. However, the authors are strongly encouraged to address the reviewers' concerns and suggestions in the final version of the paper.
Decision: Accept-Main