Cited Text Spans for Scientific Citation Text Generation (2024)

Abstract: An automatic citation generation system aims to concisely and accurately describe the relationship between two scientific articles. To do so, such a system must ground its outputs to the content of the cited paper to avoid nonfactual hallucinations. Due to the length of scientific documents, existing abstractive approaches have conditioned only on cited paper abstracts. We demonstrate empirically that the abstract is not always the most appropriate input for citation generation and that models trained in this way learn to hallucinate. We propose to condition instead on the cited text span (CTS) as an alternative to the abstract. Because manual CTS annotation is extremely timeand labor-intensive, we experiment with distant labeling of candidate CTS sentences, achieving sufficiently strong performance to substitute for expensive human annotations in model training, and we propose a human-in-theloop, keyword-based CTS retrieval approach that makes generating citation texts grounded in the full text of cited papers both promising and practical.

heading: 1 Introduction
text: An automatic citation generation system aims to concisely and accurately describe the relationship between two scientific articles, a citing paper and a cited paper. Due to the rigorous nature of academic research, even large language models (LLMs), such as the GPT family (Brown et al., 2020; OpenAI, 2023), still have to ground their outputs to the input cited papers to avoid non-factual hallucinations.
Given a set of papers to cite, prior extractive approaches (Hoang and Kan, 2010; Hu and Wan, 2014; Chen and Zhuge, 2019; Wang et al., 2019a; Deng et al., 2021) selected salient sentences from the full text of the cited papers. However, existing abstractive approaches (AbuRa’ed et al., 2020; Xing et al., 2020; Ge et al., 2021; Luu et al., 2021;
Chen et al., 2021; Li et al., 2022) generate citation texts conditioned on only cited paper abstracts, since the full texts of the papers often exceed the length limit of fine-tuned Transformer-based models (Li and Ouyang, 2022). Li et al. (2022) show that, when human judges are asked to rate the relevance of human-written gold citations with respect to the cited paper abstract (i.e. the generation system input), the scores are significantly lower than those of system-generated citations. We hypothesize that this discrepancy is caused by the gold citations referring to content from the cited paper body sections, not just the abstract, and these body sections are not available to either the generation model or the human judges.
Intuitively, the cited text span (CTS), the specific span of the cited paper that the gold citation refers to, should be a better alternative to the abstract for grounding a citation. Unfortunately, CTS labels are difficult to obtain. Existing manually-annotated sentence-level CTS datasets (Jaidka et al., 2018, 2019; AbuRa’ed et al., 2020) suffer from two key issues: annotation is extremely labor-intensive, even for domain experts, so the datasets are small; and inter-annotator agreement is relatively low (Cohen’s κ of 0.16∼0.52). Further, automatic CTS retrieval systems trained on these datasets give low performance (F1<0.2) and also use the gold citation text as input, making them inapplicable in a citation generation setting. Given these obstacles, CTS-based citation text generation is underexplored.
In this work, we study scalable CTS retrieval for citation text generation without expensive manual annotations. Our main contributions are as follows:
• We argue that CTS retrieval is an important and under-explored subtask of citation text generation. We demonstrate that citation texts generated using CTS are significantly more accurate and faithful than those using the existing approach of cited paper abstracts.
90
• Because existing human-annotated CTS datasets are too small to train an effective CTS retrieval system, we show that distant CTS labeling achieves sufficiently good performance to substitute for expensive human annotations. Using this method, we create a large-scale, distantly-labeled CTS dataset.
• We argue that existing CTS retrieval systems are inappropriate for the citation generation task because they use the gold citation text as the retrieval query. Instead, we propose a simple, human-in-the-loop, keyword-based CTS retrieval method that makes generating citations grounded in the full text of the cited paper both promising and practical1.

heading: 2 Background and Related Work
text: There are two main CTS datasets: Jaidka et al. (2018, 2019) proposed the CL-Scisumm shared task and manually annotated CTS, citation facets, and summaries for 364 citation texts that cite a set of 20 papers; AbuRa’ed et al. (2020) manually annotated sentence-level CTS for 20 related work sections. Unfortunately, both suffer from low interannotator agreement, and automatic CTS retrieval systems trained and evaluated on these datasets perform poorly (Jaidka et al., 2018, 2019; AbuRa’ed et al., 2020; Chandrasekaran et al., 2020).
Only a few studies have used CTS for scientific document summarization. ScisummNet (Yasunaga et al., 2019) leveraged a ROUGE-based
1https://github.com/jacklxc/ CTS4CitationTextGeneration
CTS retrieval approach (Cao et al., 2015) to collect community-based evidence for summarizing a paper. Wang et al. (2019a) trained a feature-based ensemble model for CTS retrieval using ScisummNet and used the retrieved CTS for extractive related work generation. Syed et al. (2023) used BM25 and SciBERT (Beltagy et al., 2019) to retrieve CTS to provide more contexts for scientific paper summarization. However, all of these works used the target citation text as the query for CTS retrieval, which cannot be used in the citation generation setting that we tackle in this paper.

heading: 3 A Large-Scale CTS Dataset: Distant vs. Human Labeling
text: When authors cite a paper, they locate salient passages and compose a citation text grounded in these cited text spans (CTS). Unfortunately, such finegrained CTS information is not recorded anywhere; even an author may not be able to precisely identify the CTS that they used. While prior CTS retrieval systems have trained on small, human-annotated CTS datasets (Jaidka et al., 2018, 2019; AbuRa’ed et al., 2020; Chandrasekaran et al., 2020), we argue that the relatively low inter-annotator agreement of these datasets suggests there can be multiple reasonable CTS for a given citation, just as there can be multiple valid extractive summaries of a document (see Section 3.3). Thus, distantlylabeled CTS is not necessarily of lower quality than human-annotated CTS, as long as the distant labels identify one of these multiple reasonable candidates. In this section, we use a simple distant approach, with the gold citation text as query,
to automatically label plausible CTS candidates to create a large-scale CTS dataset without expensive and time-consuming human annotations. We demonstrate that our distantly-labeled CTS are of comparable quality to existing human-annotated datasets.

heading: 3.1 Approach
text: With the observation that high keyword overlap with the citation text is a crucial characteristic of CTS, we follow Cao et al. (2015); Yasunaga et al. (2019) in using gold citation texts as queries to rank the top-k sentences in the corresponding cited papers as CTS candidates. The similarity is measured by the average of the ROUGE-1 and -2 recall scores (Lin, 2004).2 We use NLTK3 to remove stop words for both the query and cited paper sentences. Compared to dense retrievers (Zhang* et al., 2020), ROUGE-based retrieval does not use GPUs, and thus computationally more efficient. Moreover, since there is no good fine-tuning dataset for CTS retrieval due to the small size of human-labeled datasets Jaidka et al. (2018, 2019), an unsupervised approach is preferable.
We use the human-labeled datasets of Jaidka et al. (2018, 2019) (CL-SciSumm) and AbuRa’ed et al. (2020) as test sets to evaluate our distantlylabeled CTS4. We take the union of all annotators’ selected sentences as the human-labeled CTS.

heading: 3.2 Evaluation Metrics
text: Token overlap. We use ROUGE-recall to measure the coverage of the distantly-labeled tokens that support the gold citation text.
Faithfulness. We use QuestEval (Scialom et al., 2021) and ANLI (Nie et al., 2020) to compare the faithfulness of the gold citation text when grounded in human-labeled versus distantly-labeled CTS. For ANLI, we report the non-contradiction score (sum of entailment and neutral) given by a fine-tuned RoBERTa-large model (Liu et al., 2019).

heading: 3.3 Findings
text: Distant labeling has high coverage of human annotations. As Figure 2 shows, our top-40 re-
2We also experiment with BERTScore (Zhang* et al., 2020) as a dense retrieval approach, but the latter slightly underperforms ROUGE in all our experiments, including the comparison to human-labeled CTS (Figure 2).
3https://www.nltk.org/index.html 4We exclude 45 of AbuRa’ed et al. (2020)’s 239 cited papers due to an XML format error, but we do not expect this omission to affect the overall trend of our evaluation.
trieved CTS cover 80% and 95% of the humanlabeled CTS in CL-SciSumm and AbuRa’ed, respectively. Further, as Figure 3 shows, the papers in these two datasets average 200 sentences long, so we can see that our automatic ranking concentrates the human-annotated CTS into roughly the top 20% (∼40/200) of cited paper sentences.
Distant labeling is more similar to the gold citation text. As Figure 4 and Table 1 show, although both human-labeled and distantly-labeled CTS are comparably faithful, distantly-labeled CTS has higher token overlap with the target citation text. This is unsurprising, given that using ROUGE as our similarity metric explicitly prioritizes token overlap, while human annotators may not do so. Evaluating human CTS in the intersection, rather than the union, of different annotators’
labels (strict) has a noticeable impact on the token overlap, indicating that individual annotators can select different, but all reasonable, CTS sentences.
Distant labeling performs better on the downstream generation task. We also experiment with citation text generation using both distantlylabeled and human-labeled CTS, using the LEDoracle model described in Section 4. As Table 2 shows, distantly-labeled CTS performs comparably to human-labeled CTS for the AbuRa’ed dataset and outperforms human labels for CL-SciSumm, confirming that distant CTS labeling produces competitive performance on the downstream generation task. This experiment shows the upper bound of CTS performance using human and distant oracles, demonstrating that it is not necessary to acquire expensive, human-labeled CTS for training a retriever for downstream citation text generation.

heading: 4 Citation Text Generation Using CTS
text: In this section, we experiment with citation text generation conditioned on CTS, rather than the abstract-only approach used in prior work, and
demonstrate that conditioning on CTS produces more accurate and faithful citations.

heading: 4.1 Problem Formulation
text: Following Li et al. (2022), who note the information loss and leak issue of the citation sentence generation task, we aim to generate citation spans conditioned on (1) the citing paper context and (2) the full text of the cited paper(s) via retrieved CTS. We assume an auto-regressive generation setting, where citations are generated one at a time, in order of their appearance in the citing paper. Thus, we have access to up to two sentences before the target citation to use as context.
Data. We use the CORWA citation span5 dataset Li et al. (2022), which is derived from the ACL partition of S2ORC (Lo et al., 2020). CORWA consists of a human-annotated training set and test set, as well as an automatically-labeled distant training set, with 1654, 1206, and 19784 “dominant"type citation texts, respectively. We use only the dominant-type citations that are grounded to specific CTS, following Jaidka et al. (2018, 2019); AbuRa’ed et al. (2020), who filter out very short (“reference"-type) citations.

heading: 4.2 Approach

heading: 4.2.1 CTS Retrieval Strategies
text: Context-based retrieval. In this fully automatic approach, we retrieve CTS given only the target citation’s context sentences. We apply the distant labeling approach described in Section 3 on the CORWA training set to create a large-scale CTS dataset and use this data to train a Dense Passage Retrieval (DPR; Karpukhin et al., 2020) model. We initialize both the query and candidate document encoders with Aspire (Mysore et al., 2022), a scientific sentence encoder fine-tuned on top of SciBERT (Beltagy et al., 2019). For training, we
5To avoid confusion between cited text spans and citation spans, we will refer to the generation targets as citation texts.
use the top-40 distantly-labeled CTS sentences as positive documents and the rest of the sentences in the cited papers as hard-negative documents.
Unsupervised oracle retrieval. The distant labeling method described in Section 3 uses the gold citation text as a query; while this approach cannot be used at test time when the target citation text is not known, we use it as an Oracle setting to explore the upper-bound of CTS-based citation text generation.
Human-in-the-loop keyword retrieval. There may be multiple reasonable citation texts given the same input context, depending on the interests of the reader. Therefore, we also experiment with a practical, human-in-the-loop scenario where a researcher already has a general idea of what they want to know about a particular cited paper, so they input some keywords as a query to guide CTS retrieval. To simulate this scenario, we apply a keyword extractor to the target citation texts and use the extracted keywords as simulated user-input keywords to retrieve CTS using unsupervised retrieval (Keyword). See Appendix Table 10 for an example.
To train the keyword extractor, we manually annotate a small set (673 instances) of citation keywords on top of the CORWA dataset. Each instance consists of a short, “reference"-type citation text with manually highlighted keywords6. We pre-train a T5-small model (Raffel et al., 2020) on the named-entity recognition labels of the SciREX training set (Jain et al., 2020) and further fine-tune it on manually annotated CORWA keywords as a sequence-to-sequence keyword extractor7.
Table 3 shows the token overlap (ROUGE-L recall) between CTS retrieved using each of our three strategies and the target citation, as well as the faithfulness of the target citation grounded in the CTS; we compare against the cited paper abstract, which is used by existing citation generation systems instead of CTS. The token overlap of CTS is much higher than that of the abstract, as we expect, and more importantly, the faithfulness of the target citation grounded in CTS is higher than if it were grounded in the abstract, supporting our hypothesis
6We use reference-type citations to train the keyword extractor only, not in any citation text generation experiments.
7Alternatively, any reasonable keyword extractor, such as recent LLMs, can also be used to simulate the user-input keywords.
that the human-written target citations use information from the body sections of the cited papers. Thus, it is important to condition on CTS, not just the cited paper abstract, for citation generation.

heading: 4.2.2 Citation Text Generation Strategies
text: Retrieval-Augmented Generation. The Retrieval-Augmented Generation (RAG; Lewis et al., 2020b) bridges document retrieval and text generation by fine-tuning the query encoder and generator jointly. RAG uses DPR (Karpukhin et al., 2020) to retrieve k documents using the input query. We concatenate the target citation’s context sentences and the retrieved CTS sentences8 as the input to the citation text generator. We use Fusion-in-Decoder (FiD; Izacard and Grave, 2021) with BART-base (Lewis et al., 2020a) as the generator in our experiments.
Longformer-Encoder-Decoder. We also experiment with a simple Longformer-Encoder-Decoder (LED; Beltagy et al., 2020) model. We input the concatenation of the target citation context, the citation mark (eg. “Smith et al. (2024)"), and the retrieved CTS for each cited paper.

heading: 4.2.3 Experimental Settings
text: Training. We train on the large, automaticallylabeled CORWA distant training set using crossentropy loss and perform hyper-parameter tuning on the smaller, human-labeled training set; details and the hyper-parameter settings are reported in
8Unless explicitly specified, to balance performance and GPU memory usage, we retrieve the top k = 10 CTS in all experiments for the inference of citation text generation.
Appendix A. Each model takes about one day to train on one Nvidia Tesla V100S-32GB GPU.
Testing. For all experiments, we report performance on the human-labeled CORWA test set.

heading: 4.3 Human Evaluation
text: Following the setup used in prior works (Xing et al., 2020; Ge et al., 2021; Li et al., 2022), we recruit six NLP graduate students who are fluent in English as human evaluation judges. To increase their coverage of the test set, we split the judges into two groups of three and assign each group 25 instances sampled from the output of the RAG-FiD model. Each instance consists of a single target citation context and several candidate generations: one conditioned on the Abstract only and one for each of our three CTS retrieval strategies: Context, Oracle, and Keyword. The target citation context, cited paper abstracts, and retrieved CTS from all three strategies are also provided to the judges. Each generated citation is rated on a five-point Likert scale with respect to Fluency, Relevance (to the input; either abstract or CTS), Coherence (with respect to the context), and Overall Quality.
We perform two rounds of evaluation: first, the gold citation texts are mixed in with the candidate generations but are not identified to the judges; second, we reveal the gold citation texts and ask the judges to revise their ratings with this new information.

heading: 4.4 Results and Discussion
text: CTS-based generation outperforms abstractonly generation. Table 4 shows that the Oracle
and Keyword CTS settings massively outperform the Abstract approach. Moreover, Table 5 shows that Abstract receives the lowest human Overall rating. In contrast with prior works in the abstractonly setting (AbuRa’ed et al., 2020; Xing et al., 2020; Ge et al., 2021; Luu et al., 2021; Chen et al., 2021; Li et al., 2022), all settings now score well on Relevance (higher than 4). Previously, even the gold citation text scored poorly on Relevance because the judges were only shown the abstract, which did not contain sufficient information to support the citation texts; our judges are shown the retrieved CTS sentences, which are a much better match for the citation texts.
CTS-based generation extracts more short phrases. The ROUGE-recall scores in Table 6 show that, compared to Abstract, CTS-based generations are more similar to their inputs, suggesting that models trained on CTS are more extractive. To investigate further, we use Grusky et al. (2018)’s coverage and density metrics. Coverage measures how much of the generated citation text is extracted from the input (abstract or CTS), while density measures the size of the extractions (e.g. individual unigrams versus entire sentences). Additionally, we calculate the compression ratio between the lengths of the inputs and the generated citation texts.
As Figure 5 shows, although the Abstract setting has the lowest compression ratio, it has a significantly lower coverage score than the CTS settings, confirming that the latter is more extractive. From this finding, along with the lower faithfulness scores in Table 69, we can infer that Abstract produces more non-factual hallucinations that distort the ideas of the cited paper. This finding supports our hypothesis that, in the abstract-only setting, the citation generation input does not contain all of the information present in the target gold citation text, forcing the model to learn to hallucinate that missing information.
Moreover, Abstract has a significantly higher density score, indicating longer copied sequences from the input, while the CTS-based settings extract and reorganize shorter phrases from the input. Since copying long text spans from other documents results in plagiarism in the citation generation setting, CTS-based generation is again preferable. Nonetheless, we stress that our CTS-
9We exclude QuestEval from this table due to an issue arising from the relative compression ratios of the different input settings.
based generation approach is intended to increase accuracy and faithfulness and does not explicitly address the plagiarism issue; additional postprocessing, such as a rewriting module, should be used to further reduce the risk of plagiarism.
Predicting CTS fully automatically is challenging. Although the fully automatic Context setting has strong input retrieval performance (Table 3), its performance on the downstream citation text generation task is mixed. It underperforms on automated metrics (Table 4), but human judges rate it slightly higher than Oracle and Keyword (Table 5). Table 7 shows that Oracle and Keyword tend to retrieve CTS from more specific sections, such as “Model or Architecture” or “Results” (Appendix Table 10), suggesting that the human judges may have felt that the generated citation texts contained too much unnecessary detail.
Fully automatic CTS retrieval given only the target citation context sentences is challenging because the citation context and the CTS candidates come from different papers (citing versus cited), and the semantic mismatch between them lowers the performance of the DPR module. In the example in Appendix Table 10, the citation context only mentions “end-to-end generation models”, omitting the “content selection” and “surface realization” keywords that appear in the gold citation text, making CTS retrieval in context setting much harder than in oracle and keyword, which have explicit guidance with all important keywords from the gold citation text.
Interestingly, Table 5 shows that human evaluators cannot distinguish between gold and generated citation texts during the blind evaluation round. When the gold citation text is identified to the judges, they revise its scores higher, while
lowering the scores of the generated candidates. This finding suggests that even humans need some guiding information to help them process citation texts. Thus, as Li et al. (2022) argue, the citation text generation task should be human-in-the-loop, leveraging a researcher’s expertise to reduce both the search space size and cascading error propagation in the pipeline, and our Keyword-based CTS retrieval approach provides a practical workaround for the challenge of predicting CTS fully automatically.

heading: 5 Conclusion
text: In this work, we show that CTS-based citation text generation is more accurate and faithful than the abstract-based approaches used in prior work. We address the challenges that previously prevented CTS from being used effectively for citation text generation. Distant labeling, such as ROUGEbased labeling, achieves sufficiently strong performance to substitute for expensive manual CTS annotations, allowing us to build a dataset large enough to train a supervised CTS retriever. To avoid the pitfall of using the gold citation text as the retrieval query, which has been used in prior work but would not be possible in a real application, we propose a human-in-the-loop, keyword-based CTS retrieval approach that achieves strong performance in a practical and realistic setting. Finally, our experimental results show that, while retrieving CTS using only the citation context sentences is challenging and does not achieve the best performance using automated metrics, it is preferred by human judges and is a good direction for further investigation.
Limitations
Challenge of CTS retrieval. As discussed in Section 2, existing CTS retrieval approaches all use the gold citation text as the query. Our results also show that, while CTS retrieved using the gold citation text (or its keywords) yields strong generation performance, CTS retrieved with only the citation context sentences is not satisfactory. Therefore, we argue that citation text generation should be divided into two parts: support document retrieval and document-based citation text generation. Based on our results, we can achieve strong generation performance, but the automated retrieval part requires further investigation.
As a workaround solution without a strong,
context-based CTS retriever, we show that, when the user provides a few keywords to guide CTS retrieval, our models can perform quite well. This setting is practical under the human-in-the-loop configuration suggested by Li et al. (2022). However, we realize that this human-in-the-loop setting might be unfriendly to less-experienced researchers who are not sure what keyword to use. Future work may automatically predict the keywords of the gold citation text by performing paragraph- or document-level topic analysis using LLMs.
Choice of ROUGE-labeling approach & SOTA. We choose ROUGE-based labeling for CTS because it empirically shows the advantage of using even distantly labeled CTS for citation text generation, compared to using only cited paper abstracts. However, we do not claim that our RAG model using this retriever trained on these distant labels is SOTA. ROUGE-based labeling selects CTS with high lexical overlap with the target citation and may miss CTS that are semantically relevant but less lexically similar. In addition, there are many alternative ways to obtain better performance, such as using a more carefully crafted CTS retriever and training a larger generator with extensive hyperparameter tuning. Instead, we simply claim that using CTS can improve citation text generation performance over using cited paper abstracts, and human labeling of CTS is not required because distantly labeled CTS is of comparable quality.
Quality Assurance of CTS. Due to the high cost of manually collecting or evaluating the quality of the retrieved CTS, we are not able to perform quality assurance of the distantly retrieved CTS on a large scale to avoid error-propagation to generated citation texts. Future work can incorporate a factchecking module to further filter out CTS sentences that do not entail the citation text.
Ethics Statement
In this work, we focus on showing that retrieved CTS provides relevant support information to enable faithful citation text generation. Unsurprisingly, the trained CTS-based citation text generation models turn out to be more extractive (Section 4.4), directly copying parts of the input CTS to the output citation texts. This copying behavior has the potential to result in plagiarism if the output of our CTS-based citation text generator is directly without further modification. As we mention in Section
4.4, it may be necessary to apply a post-processing step, such as a rewriting module, to avoid plagiarism, which is out of the scope of this work and left for future investigation. We also point out that the existing, abstract-based approach to citation text generation extracts longer subsequences from its input, compared to our CTS-based approach, which extracts shorter subsequences and rearranges their order, so the plagiarism problem is not unique to CTS.

heading: A More Implementation Details
text: A.1 Opting Out of FAISS
The original RAG implementation was designed for Wikipedia-based dialogue response generation, where every Wikipedia article is a candidate for retrieval. Therefore, FAISS (Johnson et al., 2019) was used to speed up the search process. However, our search space is limited to cited paper sentences, and we need to retrieve CTS separately for each citation and its corresponding cited paper. Unfortunately, FAISS does not support partitioned search, so we replace FAISS with our simpler CTS retrieval strategies, as described in Section 4.2.1.
A.2 Freezing Query Encoder Weights
The RAG model retrieves candidate documents on the fly while jointly fine-tuning the pretrained DPR query encoder and the generator. In our preliminary experiments on retrieving CTS at the paragraph level, we did not find any significant improvement from fine-tuning the query encoder over simply freezing its weights. Further, due to the much larger candidate search space of the sentencelevel CTS setting, the on-the-fly retrieval needed to update the query encoder becomes intractable. Therefore, in our citation text generation experiments, we retrieve and cache CTS sentences for all target citation/cited paper pairs in advance and do not fine-tune the query encoder.
A.3 Citation Text Generation Hyper-parameter Settings
Table 8 shows the tuned hyper-parameter settings for our CTS-based citation text generation experiments.

heading: B More Related Work
text: B.1 Retrieval and Generation
Our proposed framework of retrieving useful information (CTS) to guide generation is partly inspired by work in knowledge-augmented, open-domain
dialogue generation, where the combination of Dense Passage Retrieval (DPR; Karpukhin et al., 2020), Retrieval-Augmented Generation (RAG; Lewis et al., 2020b) and Fusion-in-Decoder (FiD; Izacard and Grave, 2021) has been shown to yield strong performance (Shuster et al., 2021).
DPR (Karpukhin et al., 2020) separately applies two BERT-like models (Devlin et al., 2019) as query and document encoders. The DPR model is trained with contrastive loss and uses maximum inner product search (MIPS) to retrieve the documents with the highest cosine similarity between the [CLS] token representations of the query and the knowledge base documents.
Once informative documents have been retrieved, the RAG (Lewis et al., 2020b) paradigm can be applied using any sequence-to-sequence model. The conditional generation input is paired and encoded with each retrieved document in parallel, and the generation output is jointly decoded using all input/retrieved document pairs. RAGsequence decodes the entire output sequence based on a fixed distribution over the retrieved documents, while RAG-token decodes each token using a different distribution over the retrieved documents. Finally, Shuster et al. (2021) proposed FiD (Izacard and Grave, 2021) as the decoder in the RAG framework by leverageing the attention mechanism to jointly decode from the retrieved document encodings.

heading: C QuestEval and CTS-based Citation Text Generation
text: QuestEval (Scialom et al., 2021) is a questionanswering-based faithfulness metric composed of a precision score, where questions are generated from the summary (citation text) and answered using the document (generation input), and a recall score, where questions are generated from the input and answered using the citation text. Table 9 shows the full faithfulness evaluation results for our citation text generation task, including QuestEval.
We find that QuestEval encounters some diffi-
culty evaluating CTS-based citation texts because they are very short (Figure 5 shows that CTS-based citation texts have a much higher compression ratio than abstract-only citation texts); as a result, the QuestEval recall score is extremely low because many questions can be generated from the detailed information in the CTS, but the citation text is too short to contain all of the corresponding answers.
Table 11 shows an example of an Oracle CTS citation text that receives a low QuestEval score: Wang et al. (2019b) citing Swayamdipta et al. (2018). We can see that the retrieved CTS sentences include multiple entities and other detailed information that trigger QuestEval to generate many questions that the generated citation text cannot answer. However, a human reader can tell that the Oracle citation text is indeed factual; its claim about parse tree validity is supported by Swayamdipta et al. (2018)’s Introduction section: “Since the scaffold task is not an end in itself, we relax the syntactic parsing problem to a collection of independent span-level predictions, with no constraint that they form a valid parse tree."
We can also see that the Abstract-only citation text is longer and contains more entities than the Oracle citation text, while the Abstract input is much shorter and contains fewer entities than the Oracle input, so it does not suffer from the same low QuestEval recall score. We find many similar examples in our test set of longer Abstract citations being scored higher than shorter Oracle citations by QuestEval.

heading: Future Work
text: Future work may automatically predict the keywords of the gold citation text by performing paragraph- or document-level topic analysis using LLMs. Choice of ROUGE-labeling approach & SOTA. We choose ROUGE-based labeling for CTS because it empirically shows the advantage of using even distantly labeled CTS for citation text generation, compared to using only cited paper abstracts. However, we do not claim that our RAG model using this retriever trained on these distant labels is SOTA. ROUGE-based labeling selects CTS with high lexical overlap with the target citation and may miss CTS that are semantically relevant but less lexically similar. In addition, there are many alternative ways to obtain better performance, such as using a more carefully crafted CTS retriever and training a larger generator with extensive hyperparameter tuning. Instead, we simply claim that using CTS can improve citation text generation performance over using cited paper abstracts, and human labeling of CTS is not required because distantly labeled CTS is of comparable quality. Quality Assurance of CTS. Due to the high cost of manually collecting or evaluating the quality of the retrieved CTS, we are not able to perform quality assurance of the distantly retrieved CTS on a large scale to avoid error-propagation to generated citation texts. Future work can incorporate a factchecking module to further filter out CTS sentences that do not entail the citation text