Controllable Citation Sentence Generation with Language Models (2024)

Abstract: Citation generation aims to generate a citation sentence that refers to a chosen paper in the context of a manuscript. However, a rigid citation generation process is at odds with an author’s desire to control specific attributes, such as 1) the citation intent, e.g., either introducing background information or comparing results, and 2) keywords that should appear in the citation text. To provide these degrees of controllability during citation generation, we propose to integrate the manuscript context, the context of the referenced paper, and the desired control attributes into a structured template and use it to fine-tune a language model (LM) via next-token prediction. We then utilize Proximal Policy Optimization to directly optimize the LM in favor of a high score of our proposed controllability metric. The proposed workflow harmoniously combines citation attribute suggestion and conditional citation generation into one LM, allowing for better user control. 1

heading: 1 Introduction
text: A common practice in scientific writing is to cite and discuss relevant papers that support an argument, provide background information, or compare results (Penders, 2018). Recent studies aim to facilitate this citation process by using neural networks to generate a citation sentence based on the context of the manuscript and the paper to be cited. These efforts focused primarily on developing a sequence-to-sequence pipeline that works in a fully automated, uncontrolled manner, leaving little room for users to control the generation process. We believe control is desirable because authors often have a clear motivation before writing a citation sentence. For example, they may have a specific intent to cite, such as comparing results or presenting background information, or they may have keywords in mind to appear in the
1Our code and data are available at https://github. com/nianlonggu/LMCiteGen
citation sentence. When the generated citation does not match an author’s motivation, the author may wish to change the generation by specifying certain attributes, such as citation intent or keywords.
This study aims to develop a citation generation model that is controllable, such that users can alter the citation intent or topic by explicitly providing the citation intent and keywords. Our proposed method involves the following two phases: Supervised fine-tuning. We design a structured prompt template that systematically and consecutively incorporates contextual information, citation attributes (intent and keywords), and the citation sentence into a sequence of tokens, and fine-tune an LM via next-token prediction. Through supervised fine-tuning, the LM learns to generate citation sentences not only based on the manuscript/cited paper’s context but also conditioned on the citation attributes, thus allowing flexible control of generation by altering the citation attributes (Keskar et al., 2019). Controllability enhancement via reinforcement learning. We propose measuring the controllability of a citation generation system from multiple aspects with the following metrics: i. Intent Alignment Score (IAS), which measures whether the intent of the generated citation sentence matches the given control intent; ii. Keyword Recall (KR), which measures the recall of the control keywords in the generated citation; iii. Fluency Score (FS); and iv. ROUGE-F1 (Lin, 2004) score of the generated sentence compared with the groud-truth citation. These controllability evaluation metrics provide a further opportunity to guide the training of our system by using them to estimate a reward function for Proximal Policy Optimization (PPO) (Schulman et al., 2017). This allows us to explore the effect of using feedback from our chosen metrics to improve our system’s controllability.
Our contributions are summarized as follows:
• We present a novel strategy that unifies cita-
22
tion attributes and citation sentence generation within one language model, thereby enabling user control of the citation generation process.
• We evaluate the control exerted by the various attributes, employing a multi-metric system that includes an intent matching score, keyword recall, ROUGE score, and a referenceless fluency score, and we use these controllability metrics as a reward for Proximal Policy Optimization (PPO) to effectively improve the controllability of our model following its initial supervised training.
• We curate a comprehensive dataset, parsing both contextual text and citation attributes, to offer a valuable resource for future controllable citation generation research.

heading: 2 Related Work
text: Previous work in citation generation, including Xing et al. (2020); Ge et al. (2021); Wang et al. (2022), approached the task as a sequence-tosequence translation problem, utilizing recurrent networks and knowledge graph enhancements. Nikiforovskaya et al. (2020) proposed a BERTbased extractive summarizer (Liu, 2019) that produces a paper review by extracting one sentence from each of the related papers. Chen and Zhuge (2019) proposed automatically generating a related work section by extracting information on how papers in the reference list have been cited in previous articles. Xing et al. (2020) developed an RNN-based pointer generator network that can copy words from the manuscript and the abstract of the cited paper based on cross-attention. Ge et al. (2021) further extended this work by enhancing citation generation using information from the citation graph. Despite their advances, these approaches do not fully address the complexities of citation sentence generation. Our research highlights the importance of user-specified attributes and illustrates the limitations of large language models in reliably inferring key attributes such as topic keywords. We propose that this semantic gap necessitates a shift towards models that take explicit user control into account in citation generation.
Recent research Jung et al. (2022); Wu et al. (2021); Yu et al. (2022) has also explored controlled citation generation, though the focus is often limited to controlling a single attribute like citation intent. Our work extends this concept to a
broader range of attributes, introducing methods to suggest potential attributes and balance the automation and controllability of citation generation. We further differentiate our research by conducting extensive experiments with cutting-edge language models and investigating the augmentation of controllability via reinforcement learning, contributing to a more comprehensive understanding of controlled citation generation. In addition, we go beyond prompt-based approaches (Yang et al., 2022) by investigating enhancing the controllability of our citation generation system with reinforcement learning.

heading: 3 Method
text: We first describe our supervised method for finetuning an LM, and then we describe how we measure the control of the generated sentence by the attributes. Finally, we introduce strategies for enhancing controllability through reinforcement learning. Our method is illustrated in Figure 1.

heading: 3.1 LM Supervised Fine-tuning
text: We fine-tune a language model (LM) to generate a citation sentence S given the context C and citation (control) attributes A. The context, denoted as C, consists of information from three distinct sources: 1) the local manuscript context, including up to Ns sentences from the same section (we use Ns = 5) that precede the citation sentence to be generated; 2) the manuscript’s title and abstract, serving as the global context; and 3) the title and abstract of the paper to be cited, providing an external context.
The citation attributes denoted A, include the combined citation intent and keywords. Following the framework proposed in Cohan et al. (2019), we define three categories of citation intents: 1) background: The citation offers context or background information about a pertinent problem, concept, method, or topic. 2) method: The citation refers to a specific method, tool, approach, or dataset in the cited paper. 3) result: The citation contrasts or compares the results or findings of the manuscript with those in the referenced paper. In terms of keywords, we define keyword attributes as one or two noun phrases extracted from the target citation sentence that bears semantic similarity to the context of either the manuscript or the cited paper.
The training objective is to optimize the LM to maximize the log-likelihood of generating both the citation attributes A and the citation sentence S
given the context C. Let ai and si be the tokens in A and S , with total tokens |A| and |S| respectively, the training objective can be expressed as:
θ̂ = argmax θ
log p(S,A|C)
= argmax θ
[log p(A|C) + log p(S|A, C)]
= argmax θ
[ |A|∑
i=1
log p(ai|a1 . . . ai−1, C) +
|S|∑
i=1
log p(si|s1 . . . si−1,A, C) ]
(1) Such an objective allows us to fine-tune a single LM for two purposes: 1) inferring citation attributes A given the context C, and 2) crafting citation sentences S based on the context C and control attributes A. This dual-purpose training strategy enables the fine-tuned LM to switch dynamically between controlled and uncontrolled modes: In the controlled mode, the LM assimilates the context and the user-specified control attributes to generate the citation sentence. In the uncontrolled mode, the LM initiates by automatically inferring possible citation attributes based solely on the context, and uses these inferred attributes to guide itself when generating the citation sentence. This flexibility also allows us to compare the performance of our citation generation model between uncontrolled and controlled modes in a fair manner, since we can compare the performance on the same model just with different working modes.
Structured Input Template for Supervised Fine-tuning. We design an input template to help the LM differentiate between input sources. The
template arranges different input components into a unique prompt, including the cited paper’s global context, the manuscript’s local and global contexts, and the citation attributes, as in Raffel et al. (2020); Keskar et al. (2019); Taori et al. (2023). The final input text is structured as follows:
###Manuscript Title: XXX XXX ###Manuscript Abstract: XXX XXX ###Cited Paper Title: XXX XXX ###Cited Paper Abstract: XXX XXX ###Text Before Citation: XXX XXX (manuscript local context) ###Citation Intent: XXX (one word from ‘background’, ‘method’ and ‘result’) ###Keywords: XXX; XXX (keywords relevant to the citation, separated by ‘; ’) ###Citation: XXX XXX (target citation sentence)
For decoder-only LMs like GPT-NEO (Black et al., 2021), the fine-tuning phase uses the generated prompt as input for next-token prediction, masking context-related tokens (from “###Manuscript Title: XXX” to “###Text Before Citation: XXX”) to confine prediction loss within citation attributes A and citation sentence S, as per Equation (1). In the inference stage, the LM is fed with the template up to “###Citation Intent:”, “###Keywords:”, and “###Citation:” when the task is to decode the citation intent, keywords, and the citation sentence, respectively. In contrast, for encoder-decoder LMs like BART (Lewis et al., 2020), context-related tokens are the encoder input during both fine-tuning and inference. During fine-tuning, the LM learns to decode tokens within A and S . At inference, the
decoder generates citation attributes and sentences based on the provided prompt.

heading: 3.2 Controllability Evaluation Metrics
text: We deem good controllability by control attributes can be reflected in the following aspects:
1. The generated citation sentence matches the given intent. For example, given a control intent “background”, the model will generate a sentence that introduces the background of the cited paper.
2. The generated citation sentence contains the given control keywords.
3. The generated sentence is fluent so that the control keywords are embedded into the sentence in a logically coherent way.
4. The generated sentence is content-wise related to the cited paper and fits well with the context of the manuscript.
Along this vein, apart from the human evaluation (Section 6.4), we propose the following four automatic metrics, each corresponding to one aspect:
1. Intent Alignment Score (IAS) evaluates the alignment between the generated citation sentence and the specified citation intent. Suppose the control intent is i (one of three possible intents: ‘background’, ‘method’, and ‘result’) and LM generates a citation sentence S, we use SciBERT (Beltagy et al., 2019) to process the citation sentence (preceded by a “[CLS]” token) and compute the last hidden state of “[CLS]”, which we input to the intent scoring head, a fully connected two-layer network to yields the intent logits x(intent = i|S) for three possible intents. The intent alignment score IAS(S) is given by the probability of the intent i by applying softmax to the logits:
IAS(S) = exp (x(intent = i|S))∑ k∈all intents exp (x(intent = k|S))
(2) The intent scorer is trained using the SciCite dataset (Cohan et al., 2019) containing humanannotated intents. Details on training and evaluating are further described in Appendix A.
2. Keyword Recall (KR) is a metric that assesses the presence of provided keywords in the generated citation sentence. A higher value signifies that the generated sentence includes the given keywords, indicating good control over keyword
incorporation. Given a generated citation sentence S, the keyword recall KR(S) is calculated using ROUGE-L recall to compare the keyword attribute (or a concatenated string of multiple keywords, if applicable) with S.
3. Fluency Score (FS) evaluates the fluency of the generated citation sentence S. In addition to keyword recall, the fluency metric ensures that the citation generator incorporates the keyword attributes naturally and logically without compromising fluency. Drawing inspiration from Kann et al. (2018), we employ SLOR, a language model-based fluency metric, and we normalize this score using a sigmoid function. FS(S) is calculated as follows:
FS(S) = 1 1 + e−(SLOR(S)−η)
SLOR(S) = 1|S|(log(pLM (S))− log(pu(S))) (3) In this equation, |S| denotes the number of tokens in sentence S, pLM (S) represents the probability of generating the sentence with a pre-trained language model, and pu(S) is the product of the unconditional probabilities of all tokens in the sentence. The unconditional probability of a token refers to the probability of that token being generated as the first token in a sentence. As SLOR(S) does not naturally range between [0,1], we apply an offset η and a sigmoid function to normalize the score. The offset η is introduced to the SLOR scores before sigmoid normalization, allowing greater distinguishability between fluency scores for fluent and less fluent sentences. We set the offset η to 4 based on empirical observations. We utilize the 560m-Bloom (Scao et al., 2023) language model to calculate the SLOR because its expansive vocabulary of 250k tokens helps to avoid over-segmenting words into subtokens.
4. ROUGE-F1 measures the textual alignment between the generated citation sentence and the ground truth. A high score is desired, as it implies that the produced sentence is informative and contextually fitting. We used ROUGE-1,2,L F1 scores also to validate the effectiveness of citation attributes in controlling generation.

heading: 3.3 Controllability Enhancement via PPO
text: To enhance the controllability of our citation generator, we opted for Proximal Policy Optimization (PPO) (Schulman et al., 2017; Ramamurthy et al., 2023) due to its capability to use controllability
metrics (IAS, KR, FS, ROUGE) as rewards, facilitating a direct optimization of the LM. While cross-entropy loss during supervised fine-tuning does provide some controllability, it does not consistently ensure that specific controllability criteria are satisfied. To illustrate, consider the groundtruth sentence centered on keywords “L1; sparse parameters”: “L1 regularization leads to sparse parameters after training.” Two possible generated sentences are:
A. “L1 optimization leads to dense parameters after training.” and
B. “We show that L1 regularization results in sparse parameters in the model’s learned weights.”
Despite sentence A seemingly being favored by cross-entropy loss, it misinterprets the keyword “sparse” and thus alters the sentence’s meaning completely. Conversely, sentence B aptly captures the essence, emphasizing keyword recall. This underscores the limitation of relying solely on crossentropy for optimal controllability. PPO’s adaptability, allowing for controllability metrics as rewards, ensures the language model’s outputs are both accurate and controllable, offering a more nuanced and flexible optimization strategy than other prevalent methods like adjusting beam search or ’bag of words’ techniques (Pascual et al., 2020; Dathathri et al., 2020).
We used the parameters of the supervised finetuned model to initialize the LM. During the PPO training process, given the context and citation attributes used as input, the LM generated a batch of citation sentences, which we refer to as “episodes”. These episodes were evaluated using the reward:
R = 1
4
( IAS(S)+KR(S)+FS(S)+RS(S) ) , (4)
where RS(S) is the sum of ROUGE-1/2/L F1 scores, allowing its magnitude to be comparable with the other metrics. Importance sampling was used during the optimization step, with minibatches of episodes sampled along with their associated rewards, to estimate the expected return under the new policy using data collected with the old policy, aiming to minimize divergence from the previous policy while improving the expected return. We implemented the PPO training using the Transformer Reinforcement Learning (TRL) framework (von Werra et al., 2020).

heading: 4 Dataset
text: From a subset of arXiv computer science papers, we extracted triplets consisting of the citing paper (treated as the manuscript), the citation sentence within it, and the corresponding cited paper. The necessary components for the input context, including the local and global contexts of the manuscript, as well as the global context of the cited paper, were then extracted from these triplets.
Regarding the citation attributes, we used the SciBERT-based intent scorer, as outlined in Section 3.2, to predict the most probable citation intent for each citation sentence. To obtain the keywords attribute, we extracted noun phrases from each citation sentence and ranked them based on the cosine similarity between the keyword’s Sentence-BERT (version “all-mpnet-base-v2”, Apache License 2.0) (Reimers and Gurevych, 2019) embeddings and the embeddings of both the manuscript and the cited paper. This approach allowed us to retrieve up to two keywords per citation sentence.
To create the training, validation, and test sets, we used a chronological split. All the citing papers used in the training set were published before March 1st, 2023, while those in the validation and test sets were published after this date. This strategy prevents any unfair advantage for the tested language models (Section 5) by ensuring they have not previously encountered the same papers and their associated citation sentences in the validation and test sets during pretraining. The statistics of our dataset are shown in Table 1.

heading: 5 Experiment
text: We experimented with the encoder-decoder model BART (Lewis et al., 2020) and decoder-only models including GPT-Neo (Black et al., 2021), Galactica (Taylor et al., 2022), and LLaMa (Touvron et al., 2023), which varied in size from 125M to 7B parameters. All models were pretrained with corpora before March 1st, 2023. During supervised fine-tuning, the smaller models (125M to 1.3B parameters) were fine-tuned in float16 preci-
sion with a learning rate of 1e-5. For larger models like Galactica-6.7B and LLaMa-7B, we utilized INT4 low-precision quantization (Dettmers et al., 2023) and Low-Rank Adaptation (LoRA) (Hu et al., 2021) to reduce the GPU memory footprint. We set the LoRA parameters with the rank r = 16 and the scaling factor α = 32. For these larger models, we used a higher learning rate of 1e-4, in line with the guidance from Taori et al. (2023); von Werra et al. (2020). All models were optimized with the AdamW optimizer, using default betas (0.9, 0.999), weight decay 0.05, and cosine learning rate decay. Models were fine-tuned for 5k steps with a batch size of 128 with a maximum token length of 1024. We selected the best model checkpoint based on the validation set loss every 1k steps.
To enhance controllability, we subjected Galactica and LLaMa to additional fine-tuning using Proximal Policy Optimization (PPO), as described in Section 3.3. During this process, the language models generated batches of citation sentences (termed ’episodes’) for subsequent mini-batch-wise backpropagation. We set the learning rate to 1.41e5 for all PPO fine-tuning. The PPO-finetuned models are named Galactica-125M-PPO, Galactica6.7B-PPO, and LLaMa-7B-PPO, respectively (Table 3). More PPO hyperparameter details and hardware requirements are described in Appendix B.
In addition to fine-tuning models, we leveraged GPT-3.5-turbo-03012, the backbone of ChatGPT, through prompt engineering. The aim was to compare our fine-tuned models’ performance against a large language model with carefully crafted prompts. We designed specific prompts (detailed in Appendix C) for the ChatGPT API to generate citation sentences. We conducted this in controlled (providing context and citation attributes) and uncontrolled (providing only context) modes.

heading: 6 Results and Discussion
text: We aimed to investigate several key questions: 1) whether and to what extent controllability offers advantages in citation generation, 2) whether PPO helps to improve controllability, and 3) how the model size and the nature of pretraining tasks influence overall performance. In addition, we conducted a human evaluation to compare the controllability of our fine-tuned citation generator against GPT-3.5. This evaluation provided insights beyond
2https://platform.openai.com/docs/models/ overview
the automatic metrics discussed in Section 3.2.

heading: 6.1 Controllability Is Necessary
text: We assessed LMs in three modes: 1) uncontrolled mode, where the LMs utilize only the given context to infer citation attributes, and then use the inferred attributes to guide themselves when generating citation sentences; 2) intent-controlled mode, where we provide the gold citation intent to control generation while the keywords are still modelinferred; and 3) intent- and keywords-controlled mode, where the LMs are given all relevant input: context, citation intent, and keywords, from which the LMs generate citation sentences taking into account all available information.
Given only the context as input (same as the setting in Xing et al. (2020); Ge et al. (2021)), we observed limited success of LMs in matching inferred attributes with ground-truth attributes extracted from the target citation sentence (Table 2). Even the best-performing model, Galactica1.3B, achieves an intent prediction precision of just 0.66 and a keyword ROUGE-1 F1 of only 0.26. Consequently, misinterpreted citation attributes can lead to off-topic generated citation sentences, as reflected by low ROUGE scores achieved by all LMs in the uncontrolled mode (refer to Table 3).
Intriguingly, a marked improvement in ROUGE scores was already observed when LMs generated citations merely with gold citation intent in the intent-controlled mode (Table 3). Even though the citation intent is not an explicit part of the citation sentence, its presence guides LMs to generate sentences that are more aligned with the tar-
get citations, as evidenced by enhanced ROUGE scores. This enhancement is further amplified when ground-truth keywords are provided along with the citation intent, with some models witnessing a doubling in ROUGE-2 F1 scores. Case Study. We further demonstrated the effectiveness of controllability with a case study in Table 4. In the uncontrolled mode, the generated citation sentence was a background sentence and semantically mismatched with the gold citation. By just providing the citation intent “result”, the generated sentence presented the results accurately that matched well with the gold citation. Finally, introducing the control keyword “policy iterationbased algorithms” further contributed to accurately including the keyword in the generated citation sentence, yielding the highest ROUGE-L F1. Our results underscore the importance of citation attribute controllability in citation generation.

heading: 6.2 PPO Enhances LM Controllability
text: We compared the LMs’ performance in the intentcontrolled and intent- and keywords-controlled modes before and after PPO. We excluded the uncontrolled mode from this comparison, as we assume that post-PPO, the LMs consistently operate in a controlled mode where citation attributes are given. Comparing Galactica-125MPPO with Galactica-125M, Galactica-6.7B-PPO with Galactica-6.7B, and LLaMa-7B-PPO with LLaMa-7B, we observed a consistent and marked improvement in the IAS, KR, FS, and ROUGE
F1 scores, especially for the Galactica-6.7B-PPO model, which achieved clear improvement in all metrics and the best performance in terms of ROUGE-F1. This result underscores the efficacy of PPO in directing the language model to adhere to the specifications of the provided attributes more effectively. Consequently, the generated citations better align with the specified intent, more comprehensively incorporate attribute keywords, and exhibit improved fluency, suggesting the effectiveness of PPO in enhancing the controllability of the LM citation generator.

heading: 6.3 Model Size and Pretraining Matter
text: A noticeable trend of improved performance accompanies the increase in language model (LM) size, underlining the role of model scale in citation generation tasks. Additionally, notable variations in performance exist among LMs of identical sizes. Galactica consistently outperforms its counterparts, while GPT-Neo tends to underperform. Despite both models being based on the Transformer’s decoder, the performance disparity can be ascribed to their distinct pretraining datasets: GPT-Neo is pretrained on a diverse corpus with only a minor portion of scientific literature, whereas Galactica’s pretraining is on a large corpus of scientific texts. This result indicates that the specificity of pretraining data can significantly enhance citation generation performance, underlining the critical influence of pretraining corpora on performance.

heading: 6.4 Human Evaluation
text: Intriguing disparities emerge in our comparative analysis of GPT-3.5-turbo and the fine-tuned LM Galactica-6.7B-PPO (Table 3). Despite the secondbest KR score and the best FS, GPT-3.5-turbo’s performance falls short in ROUGE-F1 and IAS metrics. Such a disparity in automatic controllability metrics poses challenges in comparing the performance and controllability between models. To this end, a deeper understanding of these discrepancies was sought through a user study involving a subset of 105 examples randomly sampled from the test set. For each example, two citation sentences were presented - one generated by Galactica6.7B-PPO, the other by GPT-3.5-turbo with the structured prompt (Appendix C), and the presentation order was randomized to avoid bias. Four voluntary participants (PhD candidates with NLP research background, which matches well with the domain of our test dataset) were asked to express
their preference using a four-criterion scale (intent alignment, keyword recall, fluency, and similarity to the ground truth), mirroring our automatic metrics (IAS, KR, FS, and ROUGE). A "no preference" option was also provided (Figure 4) in case participants have equal preference for both sentences.
The results from the user study (Table 5) aligned with our automatic evaluations (Table 3). GPT-3.5turbo-generated sentences, while preferred for fluency, often diverged significantly from the groundtruth citation sentences. This suggests that GPT3.5-turbo struggles to generate contextually accurate citation sentences using our prompt template despite its prompting capabilities. Thus, highcapacity models like GPT-3.5-turbo may require further prompt refinement or few-shot tuning to enhance citation generation performance.

heading: 7 Conclusion
text: In this study, we introduced a controllable citation generation framework that leverages language models, highlighting the importance of user-specified attributes in the generation process. We emphasized the necessity for attribute control, underlining the complexities of citation generation, and explored the potential of enhancing controllability through Proximal Policy Optimization (PPO). Our experiments affirmed that large language models pretrained on scientific corpora are essential for citation generation, with the fine-tuned model showing advantages over GPT-3.5-turbo in both automatic metrics and human evaluations.

heading: Acknowledgements
text: We acknowledge support from the Swiss National Science Foundation (grant 31003A_182638) and the NCCR Evolving Language, Swiss National Science Foundation Agreement No. 51NF40_180888. We also thank the anonymous reviewers for their useful comments.

heading: A Training and Evaluation of
text: SciBERT-based Intent Scorer
To train the SciBERT-based intent scorer, we employ a multitask training strategy (Cohan et al., 2019) with the main task being citation intent classification, which aims to minimize the cross-entropy loss:
L = − log exp (xintent(itrue))∑ i∈all intents exp (xintent(i))
(5)
Additionally, we incorporate two auxiliary classification tasks (scaffolds) (Cohan et al., 2019) to enhance the main task’s performance: 1) classifying the title of the section (from 5 normalized section titles: Introduction, Related Work, Method, Experiments, Conclusion) in which the cited sentence appears, and 2) determining the citation worthiness of the sentence. We utilize a separate functional head, a two-layer fully connected network for each auxiliary task, with the SciBERT-encoded "CLS" hidden states as input. The training loss is a weighted sum of the three cross-entropy losses (Equation (5)), with weights of 1.0 for the main task, and 0.05 and 0.01 for the first and second auxiliary tasks, respectively. The accuracy of the intent scorer is shown in Table 6.

heading: B Hardware Requirements and PPO Training Hyperparameters
text: We used 4x NVIDIA A100 80 GPUs for training and a single NVIDIA RTX A6000 for inference. During PPO training, we adjusted the batch and mini-batch sizes according to the model size and architecture. Specifically, we used (256, 16) for Galactica-125M, (256, 4) for Galactica-6.7B, and
(32, 2) for LLaMa-7B, and we ran the PPO steps until no further improvement in reward.

heading: C Prompt Templates for Querying GPT-3.5-turbo API
text: The prompt templates utilized to query the GPT3.5-turbo (version 0301) API are presented in this section. As illustrated in Listing 1, the template for uncontrolled citation generation creates a citation sentence solely based on the context. In contrast, Listing 2 demonstrates the template for the controlled citation generation mode, where citation attributes are included alongside the context to guide the generation process. Before sending the query message to the API, we need to provide the actual content of the parameters that are listed at the beginning of the templates, such as the cited paper’s title and abstract, the manuscript’s title and abstract, and the local context (the text before the target citation sentence in the manuscript), and specify the citation intent and keywords in the controlled generation mode. During the API call to GPT-3.5-turbo, we configured the maximum token limit to 2k and set the generation temperature at 0.1.
D Influence of Beam Size on Citation Generation Performance
During the citation generation process with Language Models (LMs), we conducted experiments with multiple beam sizes: 1, 2, 4, and 8. The performance metrics derived from the validation set suggest that the use of a beam size of 1 yields the most consistent results. Interestingly, larger beam sizes do not enhance the performance, but negatively influence the ROUGE scores (Table 7). These results could hint that larger beam sizes introduce a broader diversity in the generated text, which may affect its precision. For instance, it could impact the accurate generation of specific topic keywords and affect the control over the generation process. In a scenario where a keyword citation attribute is provided, LMs operating with a larger beam size may opt to generate a citation sentence encompassing synonyms of the keywords rather than the keyword itself, which could consequently lead to a decrease in ROUGE scores. As a result, in our experiments, we defaulted to a beam size of 1. We will delve deeper into this phenomenon in future research.

heading: E Citation Generation Examples
text: We showcase instances (Figure 2 and 3) of citation sentences produced by Galactica-6.7B-PPO in three modes: 1) Uncontrolled, 2) Intent-Controlled, and 3) Intent- and Keywords-Controlled. Notably, Figure 2 is the complete example of the case study in Table 4. With the provision of accurate citation intent alone, the language model citation generator can align well with the stated intent, thereby enhancing ROUGE F1 scores. The addition of keyword attributes further boosts these scores.

heading: F Web Interface for Human Evaluation
text: We have designed a web interface using Streamlit3 to assess the citation sentences produced by Galactica-6.7B-PPO and GPT-3.5-turbo. As shown in Figure 4, this user-friendly platform enables participants to peruse context details, citation attributes, the original citation, and the modelgenerated sentences with ease. Upon rating the generated sentences based on the four provided criteria, participants can submit their evaluation by clicking on the “Submit” button, thereby preserving the data. A “Skip” button is also available, allowing participants to bypass any examples that fall outside their area of expertise and proceed to the next one.
3https://streamlit.io/
cited_paper_title = "**** Cited paper 's title ****" cited_paper_abstract = "**** Cited paper's abstract ****" manuscript_title = "**** Manuscript 's title ****" manuscript_abstract = "**** Manuscript 's abstract ****" manuscript_local_text_before_citation = "**** The sentences before the target
citation sentence in the manuscript (local context)****"
messages = [ {
"role": "system", "content": "You are a scientific writing assistant. Your task is to infer
the citation intent and relevant keywords based on the provided context , and generate a citation sentence for a given manuscript. The citation sentence should seamlessly follow the local context , reflect the inferred citation intent , and incorporate the inferred keywords." }, {
"role": "user", "content": f"""
The authors need to cite the reference paper:
Title: {cited_paper_title} Abstract: {cited_paper_abstract}
, in the manuscript with the global context:
Title: {manuscript_title} Abstract: {manuscript_abstract}
, immediately following the provided local context:
{manuscript_local_text_before_citation}
Your task: Please generate a citation sentence that cites the reference paper and seamlessly
follows the local context. The citation sentence should implicitly reflect one of the following citation intents and incorporate relevant keywords:

heading: 1) Background: The citation provides background information or additional context
text: about a relevant problem , concept , approach , or topic. 2) Method: The citation refers to the use of a specific method , tool , approach , or dataset from the reference paper. 3) Result: The citation compares or contrasts the results or findings of the manuscript with those in the reference paper.
Requirements:

heading: 1. Insert the citation marker "#REFR" at the position in the sentence where the
text: reference paper should be cited. 2. Put the citation marker "#REFR" correctly in the generated citation sentence. The
citation marker should replace the entire in -text citation (e.g., authors and year of publication), should not be enclosed in any brackets , and should be placed within the sentence before the ending punctuation.
Please return only the generated citation sentence. """ }, ]
Listing 1: The presented template was employed for querying the GPT-3.5-turbo (version 0301) API instructing it to generate a citation sentence given the context which comprises the title and abstract of the cited paper the title and abstract of the manuscript and the local context (sentences before the target citation in the manuscript). The generation was executed in an uncontrolled mode without the control of explicit citation attributes.
cited_paper_title = "**** Cited paper 's title ****" cited_paper_abstract = "**** Cited paper's abstract ****" manuscript_title = "**** Manuscript 's title ****" manuscript_abstract = "**** Manuscript 's abstract ****" manuscript_local_text_before_citation = "**** The sentences before the target
citation sentence in the manuscript (local context)****" citation_intent = "**** Specified citation intent ****" keywords = "**** Specified keywords ****"
messages = [ {
"role": "system", "content": "You are a scientific writing assistant. Your task is to generate
citation sentences for a given manuscript , following detailed instructions. These instructions involve taking into account the context , desired citation intent , specific keywords in your responses." }, {
"role": "user", "content": f"""
The authors need to cite the reference paper:
Title: {cited_paper_title} Abstract: {cited_paper_abstract}
, in the manuscript with the global context:
Title: {manuscript_title} Abstract: {manuscript_abstract}
, immediately following the provided local context:
{manuscript_local_text_before_citation}
Your task: Please generate one citation sentence that cites the reference paper , seamlessly
follows the local context , reflects the specified citation intent , and incorporates the specified keywords.
Requirements:

heading: 1. The generated citation sentence should reflect the citation intent: {
text: citation_intent }. Citation intents include: 1) background: The citation provides background information or additional context about a relevant problem , concept , approach , or topic. 2) method: The citation refers to the use of a specific method , tool , approach , or dataset from the reference paper. 3) result: The citation compares or contrasts the results or findings of the manuscript with those in the reference paper. 2. The generated citation sentence should contain the specified keywords: {keywords }. All the provided keywords should be used. If no keywords are specified , please infer one or two keywords by yourself and generate the citation sentence based on them. 3. Insert the citation marker "#REFR" at the position in the sentence where the reference paper should be cited. 4. Put the citation marker "#REFR" correctly in the generated citation sentence. The citation marker should replace the entire in -text citation (e.g., authors and
year of publication), should not be enclosed in any brackets , and should be placed within the sentence before the ending punctuation.
Please return only the generated citation sentence. """ }, ]
Listing 2: The presented template was employed for querying the GPT-3.5-turbo (version 0301) API instructing it to generate a citation sentence given the context which comprises the title and abstract of the cited paper the title and abstract of the manuscript and the local context (sentences before the target citation in the manuscript). The generation was also controlled by the specified citation attributes including citation intent and keywords.

heading: Future Work
text: We will delve deeper into this phenomenon in future research