ScholarCopilot: Training Large Language Models for Academic Writing with Accurate Citations
Authors: Yubo Wang
Xueguang Ma
Ping Nie
Huaye Zeng
Zhiheng Lyu
Yuxuan Zhang
Benjamin Schneider
Yi Lu
Xiang Yue
Wenhu Chen
Venue: COLM 2025 (2025)
Keywords: RAG
TL;DR: ScholarCopilot is a language model that combines text generation and citation retrieval for academic writing

Abstract: Academic writing requires both coherent text generation and precise citation of relevant literature. Although recent Retrieval-Augmented Generation (RAG) systems have significantly improved factual accuracy in general-purpose text generation, their ability to support professional academic writing remains limited. In this work, we introduce ScholarCopilot, a unified framework designed to enhance existing large language models for generating professional academic articles with accurate and contextually relevant citations.
ScholarCopilot dynamically determines when to retrieve scholarly references by generating a retrieval token [RET], which is then used to query a citation database. The retrieved references are fed into the model to augment the generation process.
We jointly optimize both the generation and citation tasks within a single framework to improve efficiency. Our model is built upon Qwen-2.5-7B and trained on 500K papers from arXiv. It achieves a top-1 retrieval accuracy of 40.1% on our evaluation dataset, outperforming baselines such as E5-Mistral-7B-Instruct (15.0%) and BM25 (9.8%).
On a dataset of 1,000 academic writing samples, ScholarCopilot scores 16.2/25 in generation quality--measured across relevance, coherence, academic rigor, completeness, and innovation--significantly surpassing all existing models, including much larger ones like the Retrieval-Augmented Qwen2.5-72B-Instruct. Human studies further demonstrate that ScholarCopilot, despite being a 7B model, significantly outperforms ChatGPT, achieving 100% preference in citation quality and over 70% in overall usefulness.

Published as a conference paper at COLM 2025
ScholarCopilot: Training Large Language Models for Aca- demic Writing with Accurate Citations
Yubo Wang14,, Xueguang Matt, Ping Nie², Huaye Zeng', Zhiheng Lyu', Yuxuan Zhang", Benjamin Schneider Yi Lu", Xiang Yue?, Wenhu Chen 14t 'University of Waterloo, CCarnegie Mellon University, Pittsburgh, Indeppedentt Researcher, *Vector Institute, Toronto
Abstract
Academic writing requires both coherent text generation and precise ci- tation of relevant literature. Although recent Retrieval-Augmented Gen- eration (RAG) systems have significantly improved factual accuracy in general-purpose text generation, their ability to support professional aca- demic writing remains limited. In this work, we introduce ScholarCopilot, unified framework designed enhance existing large language models for generating professional academic articles with accurate and contextu- ally relevant citations. ScholarCopilot dynamically determines when retrieve scholarly references by generating retrieval token [RET], which is then used to query a citation database. The retrieved references are fed into the model to augment the generation process. We jointly optimize both the generation and citation tasks within a single framework to im- prove efficiency. Our model is built upon Qwen-2.5-7B and trained on 500K papers from arXiv. It achieves a top-1 retrieval accuracy of 40.1% % our evaluation dataset, outperforming baselines such E5-Mistral-7B- Instruct (15.0%) and BM25 (9.8)). On a dataset of 1,000 academic writing samples, ScholarCopilot scores 16.2/25 in generation quality-measured across relevance, coherence, academic rigor, completeness, and innova- tion sign-iignfficantly surpassing all existing models, including much larger ones like the Retrieval-Augmented Qwen2.5-72B-nn 1-B-trut-t Human studies further demonstrate that ScholarCopilot, despite being a 7B model, signifi- cantly outperforms ChatGPT, achieving 100% preference in citation quality and over 70% overall usefulness.
Traditional RAG
ScholarCopilot
Reference
Reference
Reference
Retriever
Generator
Generator
Queery
Reference
[RET]
Quality
Retrieval Accuracy
Human Study
ScholarCapikt: 16.2
ScholarCop lot: 60.1%
Chation Qualityy
ES Misrial-777 78 Instrout: 15%
Overall Usifuless: 70%
Loven-25-788 Instrnnsttttt..
BMI25
Figure 1: Comparison of traditional Retrieval-Augmentt Generation (RAG) systems and our proposed ScholarCopilot. Traditional RAG systems (left) separately perform retrieval and generation, leading to representation misalignment. In contrast, ScholarCopilot (right) dynamically generates retrieval tokens ([RET]) during text generation for integrated and context-aware reference retrieval.
Of Core Contributors
14 Project website:
lab...aiiaaeaeeeee........ thub. htt.sc/chollarpppllo./////
1

---

Published as a conference paper at COLM 2025
1 Introduction
Academic writing is a knowledge-intensive task that requires both structured content generation and accurate citation of relevant literature. While large language models (LLMs) such as GPT-4 (Achiam et al., 2023), Deepseek-v3 (Liu et al., 2024), and Qwen2.5 (Yang et al., 2024) can generate fluent academic-style text, they frequently hallucinate citations, undermining their reliability for research writing (Huang et al., 2025; Tonmoy et al., 2024).
Recent advanced retrieval-augmenter generation (RAG) (Lewis et al., 2020; Shi et al., 2023) systems address this issue by retrieving relevant references from external knowledge bases to enhance factual consistency and reduce hallucinations. These approaches typically follow a
pipeline, as illustrated in Figure 2 (left), where retrieval is conducted independently prior to generation. However, this pipeline neglects the evolving generation context, making it difficult to dynamically adjust retrieval decisions based on the changing information needs during writing. For instance, when generating an introduction mentioning GPT-4, traditional approaches cannot adaptively retrieve GPT- 4-related references precisely when needed, since retrieval decisions are predetermined without awareness of the specific generation context. Consequently, these methods suffer from three key limitations: (1) separate optimization of retrieval and generation models leads to misalignment in query intent; (2) predetermined retrieval decisions lack flexibility and context-awareness; (3) static pipeline limits user control over the generation of content and citation needs.
To overcome these limitations, we propose ScholarCopilot, an agentic RAG framework tailored for assisting academic paper writing that seamlessly integrates text generation and citation retrieval in a unified, iterative manner, as illustrated in Figure 2 (right). Instead of relying on separate retrieval and generation stages, ScholarCopilot dynamically determines when retrieval is necessary by generating special retrieval tokens ([RÉT]) based on the evolv- ing generation context. Upon generating these tokens, ScholarCopilot pauses the generation process, retrieves relevant scholarly references, and integrates their content (abstracts or key excepts) directly back into subsequent generation steps. The dense representations of these retrieval tokens are optimized via contrastive learning, enabling efficient similarity search. Additionally, ScholarCopilot allows optional user refinement and citation triggering during the iterative process, providing flexibility to integrate human domain expertise for further improving generation quality. This unified, iterative approach enhances citation accuracy, improves content coherence, and maintains efficiency without additional overhead.
We evaluate ScholarCopilot extensively on academic writing tasks, focusing on genera- tion quality, retrieval accuracy, and overall user experience. Our model achieves 40.1% % top-1 retrieval accuracy, significantly surpassing baselines such as ES-Vistral-7B-Innttuu (15.0%) (Wang et al., 2023) and BM25 (9.8%) (Robertson et al., 2009), with consistent per- formance gains across all Top-K thresholds. In terms of generation quality, ScholarCopilot scores 16.2/25 on 1000 samples dataset with LLM-as-judge across five dimensions (rel- evance, coherence, academic rigor, completeness, and innovation), substantially outper- (13.9) and Qwwe-225-72B-Inssrru
forming larger models such as Qwen-2.5-7B-Insttm (15.8). A comprehensive user study with 10 experienced academic writers further confirms ScholarCopilot's effectiveness, particularly highlighting its citation accuracy (100% positive ratings) and overall usefulness (70% positive ratings) compared to ChatGPT.
Our main contributions are summarized as follows:
A unified generation-retrieval model that effectively integrates retrieval into the genera- tive process, enabling seamless citation retrieval while reducing inference overhead and improving citation accuracy and relevance.
A comprehensive evaluation framework that assesses both retrieval accuracy and aca- demic text quality along five critical dimensions: content relevance, logical coherence, academic rigor, information completeness, and scholarly innovation.
A large-scale training dataset consisting of 500k computer science papers from arXiv with comprehensive citation networks (33 matched citations per paper on average), facilitating robust learning of academic writing patterns and citation-aware scholarly practices.
2

---

Published as a conference paper at COLM 2025
Traditional RAG
ScholarCopilot
Input
Prompt:
Write the introduction section of academic paper
Title 'Achanoemmaee of Large... Abatract: We survey
(ollwing stoe and abstract
Merative Genoration Retrieval
Title: "eovancemente Large...
Abstract: survey recent progross...
Generation (ffor. 1)
User Refinement nt (Oplional)
Retrieval- Genoration
Recentty large models (1 Me) have Narisfommd N.P. For instance.
languaga
have significantly transformed For instancc GPPT (RET]
@PT-4 (RE
Hotr
Trigger Retrieval
Retrieve top-k from corpus
Doc1: 'BERT: draniingg of Desp...
Doc2: "Attention All You Newd.." "
Retrievel (ttr.
'eeN Generalzed..."
Retrieval taken embedding
Corpus vectors
(penerated offine by same model)
(gonorated tho-
Generation
'Referance: GPT-4 tectnrica....
Generate Introduction
Recently. large language models (LIMs have signiticadll tranoformedd (Refarance: GPT-4 report's abetractsstclte end
Viscenlly Aarge language modele (LLMs)
Insert Retrieved Citation
have made.... Models such as BERT
(Devkin el al) ()) and XI Nee...
(Ropoat haaattvvly
Ou tppt
Generation
Oulput
have made incloding text cisssllcation
Recently,
...oooe ands demonnstrates aignifoant
improvamentss.. [RET]
aignitoantly modeling appraachos,
Jarge language
Figure 2: Comparison between traditional Retrieval- Retrieval-Augmented Generation (RAG) methods (left) and ScholarCopilot (right). Traditional RAG follows a static teerieaal-hhen-geerrtiiio pipeline, retrieving references independently before generation. ScholarCopilot dynamically interleaves retrieval and generation by producing retrieval tokens ([RET]) based on current context, enabling context-aware citation retrieval and optional user refinement.
2 Related Work
2.1 Dense Retrieval
Recent studies have demonstrated that dense retrieval methods using pretrained language models to encode text into dense vectors outperform traditional lexical retrieverr like TF-IDF and BM25. Following the introduction of DPR (Karpukhin et al., 2020), several approaches have been proposed to improve dense retrieval through advanced training strategies (e.g., ANCE (Xiong et al., 2021), Condenser (Gao & Callan, 2021)), data augmentation techniques (e.g,, BGE (Xiao et al., 2023), GTE (Li et al., 2023), DRAMA (Ma et al., 2025)), and by leveraging large language models as backbones (e.g, LLM2Vec (BehnamGhader et al., 2024), RepLlama (Ma et al., 2024), Mistral-E5 (Wang et al., 2023)). Today, commercial embedding models (e.g., OpenAI (Neelakantan et al., 2022), GeminiEmbed (Lee al., 2025)) are widely used in real-world retrieval systems. However, most existing methods are designed for single-tur retrieval with short queries for retrieval, which is not well suited for citation suggestions where paper contexts are used to retrieve the next relevant citation.
2.2 Retrieval Augmented Generation
In the era of LLM, the Retrieval Augmented Generation (RAG) paradigm integrates retrieval model for the generation model, allowing the generation model to have access to external knowledge, improving the generation's correctness and factuality for downstream tasks, such as question answering fact verification (Petroni et al., 2021). Traditional RAG methods typically follow a wetrieve-then-geneee et al., 2024), where retrieval is conducted independently based on initial query, and the retrieved documents are concatenated as context for the generation model. While effective for short-form generation tasks, this static pipeline struggles in scenarios requiring long-
pipeline (Lewis et al., 2020; Gao
3

---

Published as a conference paper at COLM 2025
501K
Reference Matching
Integrate paper content with
Artie Databose
Lalex Source
Structured
9bb Entries
ISM
matched citations
500K
For Training
Downloading
POC
Parsing
Extracting
(-1K Duta Eralvation)
SONANICCOOOOAA
6AM
62M
2.2M
Unmaiched
Figure 3: The pipeline for creating the ScholarCopilot dataset. Our final dataset includes 10M citations matched from arXiv and 6.8M citations matched from Semantic Scholar (one paper may be cited by multiple articles). However, at inference time, to ensure reference quality, we only use the 670K articles from arXiv the corpus.
form generation with evolving information needs. To address this limitation, recent methods such FLARE (liang et al., 2023) and SelfRAG (Asai et al., 2023) propose iterative RAG strategies, where retrieval and generation are interleaved, allowing retrieval decisions to adapt dynamically based on the generation trajectory. These systems demonstrate improved factual accuracy for long-form content by leveraging the generation context to refine retrieval queries. Recent work, OpenScholar (Asai et al., 2024) aims improve long-form scientific question answering with self-feedback inference in RAG. However, they still decouple the retrieval and generation models, which can lead to representational misalignment for implicit query intent and increased inference overhead. More unified approaches, such as GritLM (Muennighoff et al., 2025) and OneGen (Zhang et al., 2024), train a unified model to serve both as the generator and retriever. These models share representations and can cache hidden states during generation, improving the efficiency of the system. Despite their advantages, most of these systems have been evaluated primarily on QA- style benchmarks (Mallen et al., 2023) and do not consider the iterative and citation-centric requirements of academic writing.
Our approach differs from prior work in three ways. First, it uses iterative RAG, interleaving retrieval with generation to fit evolving citation needs. Second, handles implicit intent, inferring citations from context without explicit queries. Third, it enables human-in-the-loop interaction, allowing users guide or refine citations during writing.
3 ScholarCopilot
3.1 Dataset
To train a model capable of accurately generating academic text with appropriate citations, constructed large-scale dataset of computer science research papers. Our dataset creation process consisted of five major stages, as illustrated in Figure 3.
Stage 1: Paper Collection. We collected 670K computer science papers published on arXiv (Ginsparg, 2011) between 2007 and 2024. From this initial corpus, we successfully obtained LaTeX source code for 570K papers, which formed the foundation of our dataset.
Stage 2: Structure Parsing. We developed heuristic methods to parse the LaTeX source files and extract structured components, including titles, abstracts, introductions, related work sections, and bibliographies. This stage involved handling complex LaTeX formatting and nested environments. After filtering out papers with parsing failures, we retained 501K K successfully structured documents (500K for training and 1K for evaluation), preserving the hierarchical organization essential for understanding academic documents.
Stage 3: Citation Extraction. We extracted citation information from bibliographic entries in each paper. Due to the diversity of BibTeX formatting conventions, regular expression-bart -assed approaches proved ineffective for reliable title extraction. Instead, we employed the Qwen- 2.5-3B-Instruct (Yang et al., 2024) model to robustly extract paper titles from bibliography entries. This approach yielded 19M unique citation titles across corpus.
4

---

Published as a conference paper at COLM 2025
< d
d
<q. d'>
<q.
Retriinaal Querr
Retriieval Quary Representation
Document Representation
Rep resentation
Representation
following LLM
(RET)
eveluations
[RFT]
Decoder
Decoder
Decoder
...
following
(RET)
Paper
caliting evallations
(RET]
Figure 4: Unified training framework of ScholarCopilot. The architecture jointly optimizes the next token prediction loss for text generation and the contrastive loss for citation retrieval. Retrieval tokens ([RET]) dynamically trigger retrieval. < q > indicates the positive pair of query and document during contrastive learning, and < 9. > indicates the negative pair. The generation model and retrieval model share parameters. In this figure, Paper and Paper can be considered as hard negatives for each other.
Stage 4: Reference Matching, To enable retrieval during training and inference, we matched the extracted citation titles against established academic databases. Of the 19M citation titles, we successfully matched 10M the arXiv metadata repository and an additional 6.8M in the Semantic Scholar database (Kinney et al., 2023), resulting in total of 16.8M matched citations. The remaining unmatched citations typically corresponded to URLs or publications not indexed in either database.
Stage Dataset Integration. Finally, we integrated the parsed paper structures with their matched citations to create the comprehensive ScholarCopilot dataset. The training dataset comprises 500K papers, with 1K papers reserved for evaluation. Each paper contains an average of 38 citations, of which successfully matched 33 (87%) to their corresponding entries in academic databases.
3.2 Unified Training for Generation and Citation Retrieval
ScholarCopilot jointly optimizes two objectives: next token prediction for text generation and contrastive learning for citation retrieval. Figure 4 illustrates this architecture. Detailed training procedures and hyperparameters can be found the Appendix A.1.
Next Token Prediction Loss Lg. ScholarCopilot adopts the standard autoregressive lan- guage modeling objective for text generation, maximizing the log-likelihood of each token X, conditioned on previous tokens x<1 and retrieved content (e.g., paper abstracts) when retrieval occurs: Lg E; log ( c). Retrieval is dynamically triggered via special tokens ([RET]) generated during inference.
Contrastive Loss L, for Citation Retrieval. To optimize retrieval token representations, ScholarCopilot employs contrastive learning, encouraging higher similarity between re- trieval token embeddings 4 and positive (relevant) citation embeddings d+,, and lower similarity with negative (irrelevant) citations d= Formally, the contrastive loss is defined as L, - log loglexp eexp(sim(())) d (exp(sim d + Σn- exp(sim(q, d }))], where sim(-.-) denotes cosine similarity. Positive citations are those referenced in the ground-truth paper. Negative citations are obtained through in-batch sampling: citations from the same paper irrelevant to the current context serve hard negatives, while those from other papers in the batch are easy negatives.
Joint Optimization. ScholarCopilot minimizes the combined loss Liotl = Lg + AL,, where A balances generation and retrieval objectives. In our experiments, we set À 1, equally
5

---

Published as a conference paper at COLM 2025
weighting both terms. Joint optimization ensures effective retrieval token learning for accurate citation retrieval without compromising generation quality.
4 Experiments
4.1
Baselines
We compare ScholarCopilot against several baseline approaches. For generation baselines, we include: Qwen-2.5-7B-re, D0wen-25-7B-Instruee enhanced by citation retrieval using B5-Mistral--BB-nnsruct Qwen-2.5-72B-re, the larger 72B parameter variant with the same re- trieval method; Qwen-2.5-7B-gt, Qwen-2.5-7B-Instruu. provided with ground truth citations as input; and Qwen-2.5-72B-gt, the 72B parameter variant with ground truth citations. Re- trieval baselines include BM25 (Robertson et al., 2009), a classical lexical retrieval approach commonly used in information retrieval systems; and 25-Mistral-7B-Ima Instrrcct (Wang et al., 2023), recent embedding- based retrieval model fine-tuned for retrieval tasks.
Training Details. ScholarCopilot was trained on 50K training samples for 2 epochs. We used the following hyperparameters: maximum context length of 16,384 tokens, learning rate of 1 x 10 per-device training batch size of 1, and gradient accumulation steps of 4. Training was performed on machines, each equipped with 8 NVIDIA H100 GPUs, resulting in global batch size of 128.
4.2 Evaluation Methodology
While several existing benchmarks evaluate citation-related tasks, they are not suitable for our academic writing scenario. ALCE (Gao et al., 2023) evaluates LLMs' ability to generate grounded text with citations, but focuses on general question-answering tasks rather than academic writing contexts. CiteMe (Press al., 2024) addresses citation attri- ciriion-dtermininng which cited source supports a given claim-rather than the citation generation task we target. LitSearch (Ajith al., 2024) constructs a literature search bench- mark simulating real research scenarios, but operates in question-answering format requiring explicit query generation, which differs from our academic writing evaluation where citations must be generated based contextual flow rather than explicit queries. Therefore, we design our own evaluation tailored for academic writing scenarios.
We evaluate models across two primary criteria: generation quality and retrieval accuracy.
Generation Quality. We evaluate academic writing quality using five comprehensive dimensions, each scored from 1 (poor) to 5 (excellent):
Content Relevance measures how well the generated text aligns with the specific academic topic, including appropriate use of domain-specific terminology, accurate representation of research concepts, Logical Coherence assesses the clarity and logical flow of arguments, including proper se- quencing of ideas, effective transitional phrases, consistency in reasoning, and overall readability, Academic Rigor evaluates scholarly depth and precision, encompassing technical accuracy, appropriate level of detail, proper qualification of claims, and adherence to aca- demic writing conventions. Information Completeness measures coverage comprehensivenes, including whether key aspects are addressed, sufficient detail is provided, and no critical information gaps exist. Scholarly Innovation assesses originality and insightfulness, including novel perspectives, creative synthesis of existing knowledge, identification of research gaps, and potential contribution to the field.
contextual appropriateness within the broader scientific discourse.
For evaluation, we provide GPT-4o with the paper's title and abstract as contextual input, along with the ground truth continuation (introduction and related work sections) from published papers and the corresponding model-generated text. The evaluator compares the generated content against the ground truth across all five dimensions, providing both conducted validation
numerical scores and detailed justifications. To ensure reliability, study where human experts evaluated subset of 100 examples using the same criteria. The inter-annotalor agreement between GPT-4o and human evaluators achieved Pearson

---

Published as a conference paper at COLM 2025
Model
Relevance
Coherence
Academic
Completeness
Innovation
Total
Groundtruth Citations
Qwen-2.5-7B-gt
3.27
3.07
2.52
2.77
2.82 3.28
14.44
Qwen-2.5-72B-gt
3.73
3.71
3.00
3.11
16.82
Retrieved Citations
Qwen-2.5-7B-re Qwen-2.5-72B-re
3.16
3.30
2.26
2,41
2.80
13.94
3.56
3.61
2.68
2.84
3.12
15.81
ScholarCopilot
3.63
3.66
2.87
2.89
3.17 +0.05
16.21
A (Ours -7BB)
+0.07
+0.05
+0.09
0.04
+0.4
Table 1: Generation quality evaluation results by GPT-4o. All scores are on a scale of 1-5, except for Total which is the sum (max 25).
correlation of 0.78 across all dimensions, indicating strong agreement and falling within the range typically considered highly reliable for subjective assessment tasks in academic evaluation studies. Detailed evaluation prompts are provided in the appendix A.2.
Retrieval Accuracy. Citation retrieval is evaluated using Recall@k (k = to 10), defined as the proportion of cases where the correct citation appears among the top-k retrieved results. Specifically, citations and subsequent content in 1,000 test samples are masked, and retrieval models predict citations based solely on the preceding context. For baseline models, we found that using the entire preceding context reduces performance; thus, we only use the last sentence before the citation as the query. Recall@k is computed by comparing predicted citations to the original ground-trun tround-truth citations.
4.3 Main Results
Table 1 presents the generation quality results for our approach compared to baseline models. ScholarCopilot achieves total score of 16.21 out of 25, outperforming both Qwen-2.5-7B- Instruct with retrieval enhancement (13.94) and standard Qwwen2.5-7BB-nntrru with ground truth citations (14.44). Notably, our approach even surpasses the much larger Owen-2.5-72B- Instruct model with retrieval enhancement (15.81) and comes close to the 72B model with ground truth citations (16.82), despite having only about 10% of its parameters.
ScholarCopilot demonstrates particular strengths in Relevance (3.63) and Coherence (3.66), comparable to the 72B models. The improvement Academic Rigor (2.87 vs. 2.26 for Qwen-2.5-7B-re) highlights model's ability to incorporate appropriate citations and scholarly conventions. These results confirm that our unified approach to generation and citation effectively improves academic writing quality even with relatively small model.
4.4
Ablation Studies
4.4.1
Retrieval Performance
Figure 5 compares ScholarCopilot's citation retrieval performance (Recall@k) with baseline methods, including OpenScholar's retriever both with and without reranker. Scholar- Copilot achieves a top-1 recall of 40.1%, significantly outperforming BM25 (9.8%) and E5-Missrall7B--nnsiee (15.0%). While OpenScholar's retriever achieves competitive per- formance, particularly when enhanced with a reranker (24.6% at R@1), ScholarCopilot still maintains substantial advantage. This advantage persists across all recall levels, with ScholarCopilot reaching 64.8% recall@10, outperforming OpenScholar + Reranker (48.6%), more than doubling E5-Mistral- Mirfistral-B--hie BB-Instuuc (30.0%), and tripling BM25 (20.8%).
These results demonstrate that while OpenScholar performs reasonably well our evalua- tion setting, especially when combined with reranker, approach maintains superior performance. Importantly, OpenScholar is primarily designed for QA tasks and trained on question-answering data, whereas our task focuses autocomplete for scientific writing.
7

---

Published as a conference paper at COLM 2025
M-MMitrrl-7B
OpenScholar
OpenScholer
BcholarCoplot
62.8%
64.8%
58.4%
6016
53.3%
48.6%
5036
40.1%
41.6%
43.8%
46.0%
Recal (56)
36.0%
42.9%
45.3%
30%
24.8%
29.0%
32.3%
33.0%
36.2%
37.1%
38.8%
40.4%
20%
10%
$5.0%%
23.2%
27.5%
20.1%
13.3%
22.5%
15.3%
24.2%
16.6%
25.5%
17.8%
26.6%
18.3%
27.6%
19.1%
28.5%
19.7%
29.3%
20.2%
20.8%
30.0%
9.8%
096
R81
R82
RS8
RB4
RB8
PXG
R677
RMB
R29
R810
Recallkk
Figure Comparison of citation retrieval performance (Recall@k) between ScholarCopilot and baseline retrieval methods (BM25, 15-Miistal-77-Instrue and OpenScholar).
The consistent advantage ScholarCopilot across all recall levels highlights the effectiveness of our unified training approach. Traditional retrieval methods rely explicitly formulated queries, often failing to capture nuanced citation intents. In contrast, ScholarCopilot directly optimizes retrieval token representations during generation, implicitly encoding citation intent through context-aware queries informed by both local (surrounding text) and global (document-level) information.
4.4.2 Impact of Reference Content Integration
We evaluate the impact of providing retrieved reference content to ScholarCopilot during generation by comparing two settings: (1) the standard approach, where the model accesses reference details during generation; and (2) a variant that triggers retrieval but cites papers without seeing their content.
Method
Relevance
Coherence
Academic
Completeness
Innovation
Total
ScholarCopilot
3.63
3.66
2.87
2.89
3.17
16.21
w/o ref. content
3.60
3.25
2.58
2.91
3.19
15.53
A
+0.03
+0.41
+0.29
-0.02
-0.02
+0.68
Table Impact of reference content integration on generation quality.
As shown in Table the two variants perform similarly Relevance, Completeness, and Innovation. However, differences appear in Coherence (3.66 vs. 3.25) and Academic Rigor (2.87 vs. 2.58), leading to higher total score for the standard ScholarCopilot (16.21 vs. 15.53). Analysis indicates two reasons. First, access to reference content reduces inaccuracies when describing cited works. Second, reference details provide contextual information that improves coherence, especially during comparisons or transitions between ideas.
Qualitative evaluation shows the variant without reference content tends to cite sources with general statements, whereas the standard approach integrates specific details for clearer connections. For example, when discussing neural networks, the standard model states: "Transformer models leverage self-attention mechanisms to capture long-range dependencies cite cite (vaswani 2077atenttin)) specifically through multi-headed approach that projects queries, keys, and values into separate subspaces." " In contrast, the variant without reference content produces simpler statements like "Transformer models use self- attention for capturing dependencies ci (vaswani 201 7attention)."

---

Published as a conference paper at COLM 2025
Cration Relonres
Clation Accerscy
Madi wors
Wasse
Soxtar
Betee
Mudt better
Bace Use
lnteffee Cluriy
Clerkon Quality
07
0%
66%
20%
so%s
Inamacioon
Acriemi Zige
Whkkig
18%
60%
Factual Acceracy
Wiitin Style
Logial Flow
19%
Topic Relovene
Eitiioocy
1855
18%
10%
Insovatio
Repatition
07%
0%
30%
40%
10%
Avargge Rorinn (1-5)
Clatio Quality
User Experinace
Contect Quality
(b) Comparative analysis of ScholarCopilot vs. ChatGPT across five dimensions: Citation Qual- ity, Writing Quality, Ease Use, Time Efficiency, and Overall Usefulness. Darker blue indicates higher percentages of ratings.
(a) Average ratings for ScholarCopilot across eval- uation dimensions: citation quality (yellow), user experience (green), and content quality (purple), from user study (N=10).
Figure 6: Human evaluation of ScholarCopilot and comparative analysis with ChatGPT
5
User Study
To evaluate the utility of ScholarCopilot in practical academic writing, we conducted a user study with participants from various academic backgrounds. This evaluation assessed both technical performance and user experience.
5.1 Human Evaluation Design
We conducted a mixed-method evaluation combining quantitative ratings and qualitative feedback. Participants were 10 students (5 PhD, 4 master's ss and undergraduate) averaging 4.2 years of academic writing experience. All participants had prior academic writing experience and were familiar with AI writing assistants such as ChatGPT.
Each participant used ScholarCopilot to draft the introduction and related work sections on at least five topics within their expertise. The evaluation included:
Quantitative Assessment. Participants rated ScholarCopilot on 15 metrics using 5-point Likert scale (1=Poor, 5=Excellent), grouped into Citation Quality (relevance, accuracy, timeli- ness), User Experience (ease of use, response time, interface clarity, interaction fluidity), and Content Quality (academic rigor, factual accuracy, writing style, logical flow, completeness, topical relevance, innovation, redundancy).
Comparative Analysis. Participants compared ScholarCopilot with ChatGPT on citation quality, writing quality, ease of use, time efficiency, and overall usefulness. Open-ended Feedback. Participants commented on ScholarCopilot's strengths, limitations, and suggested improvements.
5.2 Human Evaluation Results
Figure 6a shows the average ratings. ScholarCopilot received the highest scores for citation accuracy (4.6/5), interface clarity (4.5/5), and writing style (4.5/5). Citation quality metrics averaged 4.3/5. User experience metrics averaged 3.9/5, with response time rated lowest (3.3/5). It is worth noting that the system was deployed on a single 80GB GPU, which led to longer waiting times during peak usage periods. Due to this resource limitation, different participants experienced significantly varied response times, which explains the inconsistent feedback regarding system responsiveness in the evaluation results.
Content quality metrics showed more variation, with Writing style (4.5/5) and factual accuracy (4.3/5) scoring well, while innovation received the lowest score across all metrics (2.5/5). This suggests that while ScholarCopilot excels at generating academically sound content, it may be less effective at proposing novel ideas suggesting innovative directions.

---

Published as a conference paper at COLM 2025
Comparative Advantage. Figure 6b compares ScholarCopilot and ChatGPT. ScholarCopilot shows a clear advantage in citation quality, with all participants rating ScholarCopilot higher. For overall usefulness, 70% rated ScholarCopilot higher. Writing quality advantage was moderate, with 50% rating ScholarCopilot higher and 40% assessing it similar to ChatGPT.
Qualitative Feedback. Open-ended responses identified strengths such as integrated cita- tion management (citation search and BibTTeX handling), interactive incremental writing style offering greater user control, and improved time efficiency especially for related work sections. Participants also suggested improvements including generating more comprehen- sive content, reducing system response time for complex retrieval tasks, and enhancing support for generating innovative ideas and research questions.
Future Use Intention. Participants' average rating for likelihood of future use was 4.1/5, with 80% rating this intention as 4 or 5. This suggests ScholarCopilot effectively addresses user needs despite noted limitations.
Participants also provided suggestions, such integrating with writing platforms like Overleaf, supporting section-wiss generation, and allowing predictions at arbitrary cursor positions. These suggestions provide directions for future development.
In summary, the user study confirms ScholarCopilot effectively integrates text generation and citation retrieval, improving user experience in academic writing workflows. The strengths in citation relevance and management indicate advancement over existing tools, while response time and innovation support represent areas for future improvement.
Limitations and Future Work
Despite promising results, ScholarCopilot currently supports only Introduction and Re- lated Work sections within the computer science domain. Future work will extend the framework additional paper sections (e.g., methods, experiments) and diverse academic disciplines. Additionally, the user study highlighted limitations in generating innovative insights. Addressing this requires exploring larger models, expanded datasets, and targeted training techniques to enhance creativity. Finally, improvements in user interaction- -scch as persistent content storage, concise summaries for suggested references, and robust load balancing for consistent multi-user responsiveness critical future enhancements.
Conclusion
We introduced ScholarCopilot, unified framework integrating dynamic retrieval within the generative process for academic writing. Unlike traditional static retrieval retrieval-generatie -generation pipelines, ScholarCopilot adaptively retrieves citations based evolving generation con- texts, significantly improving citation accuracy and coherence. Extensive evaluation and user studies demonstrated its effectiveness, particularly in citation relevance, writing ef- ficiency, and overall user experience. Despite current limitations in scope, innovation capability, and interaction design, ScholarCopilot marks a pioneering step toward future advancements in AT-supported academic writing.
References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint ar xrXi::2333.0877, 2023.
Anirudh Ajith, Mengzhou Xia, Alexis Chevalier, Tanya Goyal, Danqi Chen, and Tianyu Gao. Litsearch: retrieval benchmark for scientific literature search. arXiv preprint A0X1:22407.18940, 2024.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishiri.. Self-rag: Leam- ing to retrieve, generate, and critique through self-reflection. The Twelfth International Conference on Learning Representations, 2023.
10

---

Published as a conference paper at COLM 2025
Akari Asai, Jacqueline He, Rulin Shao, Weijia Shi, Amanpreet Singh, Joseph Chee Chang, Kyle Lo, Luca Soldaini, Sergey Feldman, Mike D' D'arcy, et al. Openscholar: Synthesizing scientific literature with augmeetdl-ugmmented Ims. ar arXiv preprint arXio:2411.14199 2024.
Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. LLM2vec: Large language models are secretly powerful text encoders. In First Conference on Language Modeling, 2024. URL https:///peenreview.rr net/ forum?id=1W1PRRTEBB
Luyu Gao and Jamie Callan. Condenser: a pre-training architecture for dense retrieval. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods Natural Language Processing, pp. 981-993, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: (w11163//v12222.emm/ppp. //aclanthology.orga org owg. mamlp-mainn.5.
URL https:
Tianyu Gao, Howard Yen, Jiatong Yu, and Dangi Chen. Enabling large language models to generate text with citations. arXiv preprint ar arXiv:2305.14627 2023.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-augmented generation for large language models: A survey, 2024, URL BtKLusss//aaxii..og//////2222 10997.
Paul Ginsparg. Arxiv at 20. Nature, 4717449977779911114777200 2011.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems, 43(2):1-55, 2025.
Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 7969-7992, Singapore, December 2023, Association for Computational Linguistics. doi: 10.186533v1//1/ http/222..mmmlmmaeee main..99.. URL htttp:://aclanthologg/ org/ 2023. emnlp-main. m95..
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answer- ing. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 6769-6781, Online, November 2020. Association for Computational Linguistics. doi: 10.18653/ /vl/ 2020.emnlp-main
URL https: http:/::/a/aaaahhooooooo org/ 2020. emnlp- maoommain... 550/.
Rodney Kinney, Chloe Anastasiades, Russell Authur, Iz Beltagy, Jonathan Bragg, Alexandra Buraczynski, Isabel Cachola, Stefan Candra, Yoganand Chandrasekhar, Arman Cohan, ct al. The semantic scholar open data platform. arXiv preprint Xii::23011100100 2023.
Jinhyuk Lee, Feiyang Chen, Sahil Dua, Daniel Cer, Madhuri Shanbhogue, Iftekhar Naim, Gustavo Hernández Ábrego, Zhe Li, Kaifeng Chen, Henrique Schechter Vera, Xiaoqi Ren, Shanfeng Zhang, Daniel Salz, Michael Boratko, Jay Han, Blair Chen, Shuo Huang, Vikram Rao, Paul Suganthan, Feng Han, Andreas Doumanoglou, Nithi Gupta, Fedor Moiseev, Cathy Yip, Aashi Jain, Simon Baumgartner, Shahrokh Shahi, Frank Palma Gomez, Sandeep Mariserla, Min Choi, Parashar Shah, Sonam Goenka, Ke Chen, Ye Xia, Koert Chen, Sai Meher Karthik Duddu, Yichang Chen, Trevor Walker, Wenlei Zhou, Rakesh Ghiya, Zach Gleicher, Karan Gill, Zhe Dong, Mojtaba Seyedhosseini, Yunhsuan Sung, Raphael Hoffmann, and Tom Duerig. Gemini embedding: Generalizable embeddings from gemini, 2025. URL https::/arxxv.orr//bbbstt http./or////b///////// 07891.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval- augmented generation for knowledgeiintesiieer nlp tasks. Advances in neural information processing systems, 33:9459- 9474, 2020.
11

---

Published as a conference paper at COLM 2025
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning, 2023. URL
https://arxiv orggabs//305 :ttts://arki..rrrrbb///3/....8 03281.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. Xiv preprint ar Xiv:2412. 19437, 2024.
Xueguang Ma, Liang Wang, Nan Yang, Furu Wei, and Jimmy Lin. Fine-tuning llama for multi-stage text retrieval. In Proceedings of the 47th International ACM SIGIR Conference Research and Development in Information Retrieval, SIGIR '24, pp. 2421-2425, New York, NY, USA, 2024. Association for Computing Machinery. ISBN 9798400704314, doi: 10.1145/ 3626772.3657951 URL https://dol /ttp:////ooooo//.///o// org/10. /145/1166672
Xueguang Ma, Victoria Xi Lin, Barlas Oguz, Jimmy Lin, Wen-tau Yih, and Xilun Chen. Drama: Diverse augmentation from large language models smaller dense retrievers. #rXiv:2502.18460, 2025.
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of paramet- ric and non-parametric memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computa- tional Linguistics (Volume 1: Long Papers), pp. Association for Computational Linguistics. doi: 10. 10.18653/vi 2003/v1122222 https://aclanthologg.. org/ org.oogg//2222 acl- loel-long..0 546/.
Toronto, Canada, July 2023.
546. URL
Niklas Muennighoff, Hongjin SU, Liang Wang, Nan Yang, Furu Wei, Tao Yu, Amanpreet Singh, and Douwe Kiela. Generative representational instruction tuning. In The Thirleenth International Conference on Learning Representations, 2025. URL https: htt::://ppereeeeeeeeee forum?1d-BC41Ivff2v
Arvind Ncelakantan, Tao Xu, Raul Puri, Alec Radford, Jesse Michael Han, Jerry Tworek, Qiming Yuan, Nikolas Tezak, Jong Wook Kim, Chris Hallacy, Johannes Heidecke, Pranav Shyam, Boris Power, Tyna Eloundou Nekoul, Girish Sastry, Gretchen Krueger, David Schnurr, Felipe Petroski Such, Kenny Hsu, Madeleine I'hompson, Tabarak Khan, Toki Sherbakov, Joanne Jang, Peter Welinder, and Lilian Weng. Text and code embeddings by contrastive pre-training, 2022. URL https: http:////a/iiooo/a////2/00 10005.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: benchmark for knowledge intensive language tasks. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tus Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, pp. 2523-2544, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/ / v1 2222 we0.0.aaaaaa-1.mm1 https: https::/aclaanthologg.. org/2021. naacl-main. 200/
URL
Ori Press, Andreas Hochlehnert, Ameya Prabhu, Vishaal Udandarao, Ofir Press, and Matthias Bethge. Citeme: Can language models accurately cite scientific claims? Advances in Neural Information Processing Systems, 37:7847-7877, 2024
Stephen Robertson, Hugo Zaragoza, et al. The probabilistic relevance framework: Bm25 and beyond. Foundations and Trends in Information Retrieval, 3(4):333-38) 2009.
Weijia Shi, Sewon Min, Michihiro Yasunaga, Minjoon Seo, Rich James, Mike Lewis, Luke Zettlemoyer, and Wen tau Yih. Replug: Retrieval-augmented black-box language models, 2023. URL https: http://axvvvvvv/ arg/abs/2311 12652.
SM Tonmoy, SM Zaman, Vinija Jain, Anku Rani, Vipula Rawte, Aman Chadha, and Amitava Das. A comprehensive survey of hallucination mitigation techniques in large language models. arXiv preprint ar arXio:2401.01113,, 6, 2024.
12

---

Published as a conference paper at COLM 2025
Liang Wang, Nan Yang, Xiaolong Huang, Linjun Yang, Rangan Majumder, and Furu Wei. Improving text embeddings with large language models. ar preprint arXiv:2401.00366 2023.
Shitao Xiao, Zheng Liu, Peitian Zhang, and Niklas Muennighoff. C-pack: Packaged re- sources to advance general chinese embedding, 2023.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations, 2021. URL https: http://ppneeeiieee..net/rrrrnnin htte//orrmmii-zeefffooooo
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. technical report. ar preprint arXiv:2412.15115, 2024.
Jintian Zhang. Cheng Peng, Mengshu Sun, Xiang Chen, Lei Liang, Zhiqiang Zhang, Jun Zhou, Huajun Chen, and Ningyu Zhang. OneGen: Efficient one-pass unified genera- tion and retrieval for LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 4088- 4119, Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.18653/v //////112222.//2/2 f02..finniiggs-mmll.22.. URL https://aclantthloggy org/2024 Findings-emnle 237/
A Appendix
A.1 Training Details
We trained our model with the following hyperparameters: maximum context length of 16,384 tokens, learning rate of 10 per-device training batch size of 1, and gra- dient accumulation steps of 4. Training was performed on 4 machines, each equipped with NVIDIA H100 GPUs, resulting in global batch size (per-device batch size) x 8 (GPUs per machine) x 4 (machines) x 4 (gradient accumulation steps) = 128.
A.2 Generation Quality Evaluation prompt
You expert academic writing evaluator. Your task is to assess the quality of AI-generated academic text compared ground truth text from published papers.
EVALUATION DIMENSIONS AND SCORING CRITERIA:
1. Content Relevance (1-5):
Completely off-topic, inappropriate terminology, major factual errors Partially relevant but significant misalignment, some incorrect terminology
3: Generally relevant with minor topical drift, mostly appropriate terminology
4: Well-aligned with topic, accurate terminology, minor contextual issues
5: Perfectly aligned, expert-level terminology, fully contextually appropriate
2. Logical Coherence (1-5):
1: Incoherent structure, no logical flow, contradictory statements
2: Poor organization, frequent logical gaps, confusing transitions
Acceptable structure with some logical inconsistencies, adequate transitions
4: well-organized with clear flow, minor logical issues, good transitions
5: Exceptionally clear structure, flawless logical progression, seamless transitions
Academic Rigor (1-5):
Lacks scholarly depth, significant technical errors, informal tone
2: Limited depth, some technical inaccuracies, inconsistent academic style
3: Adequate scholarly level, minor technical issues, generally academic tone
4: Strong academic depth, accurate technical content, proper scholarly conventions Exceptional rigor, perfect technical accuracy, exemplary academic style
13

---

Published as a conference paper at COLM 2025
Information Completeness (1-5):
1:
Major information gaps, key concepts missing, superficial coverage
2: Notable omissions, incomplete coverage of important aspects
3: Adequate coverage with some minor gaps, satisfactory detail level Comprehensive coverage, minor omissions, appropriate detail Complete and thorough coverage, no gaps, optimal level of detail
5. Scholarly Innovation (1-5):
1: No original insights, purely derivative, lacks critical thinking Limited originality, mostly restates existing knowledge, minimal analysis
3: Some original perspectives, adequate synthesis, basic critical analysis
4: Good original insights, effective knowledge synthesis, strong analysis
5: Exceptional innovation, novel perspectives, outstanding synthesis and analysis
INPUT FORMAT:
Paper Title: [TITLE]
Abstract: [ABSTRACT]
Ground Truth Text: [GROUND_ [RUTH]
Generated Text: [GENERATED_TEXT]
OUTPUT FORMAT:
For each dimension, provide:
Score (1-5)
Brief justification (2-3 sentences explaining why this score was assigned)
Content Relevance: [Score]/5
Justification: [Your reasoning based the criteria above]
Logical Coherence: [Score]/5
Justification: [Your reasoning based on the criteria above]
Academic Rigor: [Score]/5
Justification: [Your reasoning based on the criteria above]
Information Completeness: [Score]/5
Justification: [Your reasoning based the criteria above]
Scholarly Innovation: [Score]/5
Justification: [Your reasoning based on the criteria above]
Compare the generated text against the ground truth and provide objective, detailed assessments focusing on academic writing quality according to the specific criteria outlined above.
A.3 Case Study
ScholarCopilot for Academic Writing with Citations Example:
Atitle((Nose--Unconditta Diffusion Models for Text-to-Inage Generation)
Isection([ntroducttion)
With the prosperity of deep generative models research, many works emerged focusing on various aspects of generative models, including model archi design" design"Xcite(prraa Noiite/praaaa2222snraael.. training strategies Nitts xu2024towards), and noise scheduling" hiieeuuulnnnniiitcccchheaaaa \cite((che22223on) whether noise conditioning However, there is very 1imi ted work discussing necessary component. Noise conditionn tioning has been infallible rule widely accepted by researchers in the field diffusion models research. However, is it really necessary to employ noise conditioning in diffused models? A recent mudd""Ncitt(su22225mmii point out that in some cases diffusion models even perform better without. noise conditioning. Inspired by this, delve deeper into this research direction and
1i2023faster. si2023freeu, benny2022dynamicc),
has challenged this convention. They
14

---

Published as a conference paper at COLM 2025
explore the feasibility of dllse-unccnnitional diffusion models in the text-to-image generation task.
In this paper, we propose novel framework for noise-uncondition diffusion models, named (textbf(Noise-UnncondTT) is based the observation that the noise level is necessary input for the denoising process. Previous work"\ccitessun2225niis has shown that noise condit tioning can be removed without significantly compromising performance.
Diffusion Models (NUIT)). Our approach
Motivated by these findings, we aim to explore the possibility of employing noise-unconditiona diffusion models in text-to-image generation tasks. Compared general image generation tasks, text-to-image generation tasks more challenging and can better reflect the model's generalization ability. To this end, we focus on the text-to-image generation task in this paper.
Our approach is based the idea that the noise level be inferred from the model's 's internal states, rather than being explicitly provided as input. We propose a novel tecture that eliminates the need for explicit noise level inputs. This trchitecture is based on the idea that the noise level can be inferred from the model's internal states, rather than being explicitly provided input. Specifically, we propose to the global information the image, noise level. This approach allows us to remove the need for explicit noise level inputs, while still achieving competitive per perfomancc approach demonstrates that daise-unconditional diffusion models be viable alternative to traditional noise-condi tioned diffusion models text-to-image generation tasks. Our contributions can be summarized as follows:
self-attention mechanism capture then use this information infer the
standard benchmarks, Our
\begin(itemize)
\item We propose novel framework for sisse-uuncoditionall diffusion models, which eliminates the need for explicit noise level inputs.
\item We demonstrate that hoise-uncondition tional diffusion models can achieve competitive performance on standard benchmarks, achieving state-of-thh-ard FID and CLIP score.
Vitem provide an in-depth analysis the noise-unconditional diffusion models, and discuss the potential applications this approach in various domains. Nend(itemize)
believe that work will inspire research in this direction, and will contribute to development of more efficient effective generative models.
Bibliography:
@erticlleprssadd23urravllli
title=(Unraveling the Temporal Dynamics of the Unet in Diffusion Models), author=(Vidya Prasad, Chen Zhu-Tian, Anna Vilanova, Hanspeter Pfister, Nicola Pezzotti, Hendrik Strobelt), journal=(arXiv preprint arXiv:2312.1499 14965), year=(2023),
archivePrefixxe(arkii),
eprint==(312..14965), primaryClass=(cs.cV Ls.
@article(112223faster:
title=(Faster Diffusion: Rethinking the Role of the Encoder for Diffusion Model Inference),
author=(Senmao Li, Taihang Hu, Joost van de Weijer, Fahad Shahbaz Khan, Tao Liu, Linxuan Li, Shiqi Yang, Yaxing Wang, Ming-Ming Cheng and Jian Yang), journal=(arxiv preprint arXiv:2312.099683, year=(2023),
archivePrefixx<arxivv, eprint=(2312.099688) primaryClass-(cs.vV)
15

---

Published as a conference paper at COLM 2025
@article(si2023freeu:
title=(FreeU: Free Lunch in Diffusion U-Net), author=[Chenyang Si, Ziqi Huang, Yuming Jiang, Ziwei Liu), journal=(arxiv preprint arXiv:23e9. 11497), year=(2023),
archivePrerixx<(rkiiv), eprint=(2309.11497) primaryclass=(cs.cv)
Bartiicleebemmyy2222dyaaiii
title=(Dynamic -uat-uutput Diffusion Models),
author=(Yaniy Benny, Lior Wolf),
journal=(arXiv preprint arXiv:2203.c 04304), year=(2022),
archivePrefixx<aakkiv)
(0311t=(2203384304),, primaryClass=(es.CV IV)
@articleexu2824towards,,
title=(Towards Faster Training of Diffusion Models: An Inspiration of Consistency Phenomenon],
author=(Tianshuo Xu, Peng Mi, Ruilin Wang, Yingcong Chen), journal=(arXiv preprint 248.444444.7 year=(2024),
archivePrerixxa(akkii,,
07944) primaryClass=(cs.G cs.AI)
@article(chen20233n, title=(On the Impor tance of Noise Scheduling for Diffusion Models), author=(Ting Chen},
journal=(arXiv preprint arXiv:2301. 10972), year=(2023), archiveerrefixx((rkki),
eprint=((381.18892))
primaryClass=(cs.CV CR cs Cs.LG MM.
Darticclessu2225hooise
title=(Is Noise Conditioning Necessary for Denoising Generative Models?}, author=(Sun, Qiao and Jiang, Zhicheng and Zhao, Hanhong and He, Kaiming), jouxnal=(arXiv preprint arXiv:25e2. 13129), year=(2025)
Analysis:
The ScholarCopilo--geeeraaed introduction demonstrates good coherence, logical structure, and clear presentation of motivation, technical concepts, and contributions. It exhibits fluent academic English and accurate citation formatting, reflecting effective training on academic writing conventions. However, the text occasionally contains redundant phrases, such as repeated descriptions about inferring noise levels from internal model states. Addi- tionally, briefly contextualizing and contrasting your method against related works would help further emphasize the originality and significance of your contributions.
GPT-4o for Academic Writing with Citations Example:
Tiitle(Noise-Uucooddiiino Diffusion Models for Text-to Imaee Generation)
(bggin(abstract)
16

---

Published as a conference paper at COLM 2025
Diffusion models have emerged powerful tools for generative tasks, particularly the domain image synthesis. While conditions diffusion models have achieved remarkable success in converting text prompts into high-quality images, the reliance on explicit conditioning signals often limits their generalizability In this paper, propose a novel paradigm: Noise-Unconditiona Diffusion Models (NUDMs), which decouple the text encoding process from the noise generation process, enabling enhanced flexibility robustness text-to-image generation tasks. kend(abstract)
Isection(Introduuctiin))
Text-to-image generation has become a cornerstome of modern AI research, enabling systems to translate natural language descriptions into visually coherent and semantically aligned images. Recent advancements in diffusion models have demonstrated their potential generating photorealistic and diverse images, making them a competitive alternative to adversarial models such as GANs satte@ooodelloo20 4gan]. Conditional diffusion models, such as those used in and Imagen Xcite(saharia2022magmm)
DALLE <cite(ramesh22222llee embeddings guide the denoising process, ensuring that the generated images align the input text prompt.
leverage text
Despite their successes, conditional diffusion models face challenges related to overfitting to conditioning signals Ncite(hh2222deeoisiigg and the computational overhead processing complex text-image alignments. Additionally, the reliance on explicit conditioning introduces a bottleneck when dealing with noisy or ambiguous text inputs. To address these issues, we introduce the concept of Ceiise-Unconitiionn Diffusion Models (NUDMs), which operate independently of explicit text conditioning during the noise generation phase. Instead, text embeddings are integrated into the model through an auxiliary alignment mechanism, allowing for more robust and flexible generation.
Our contributions threefold. First, propose noise-unconditional framework for diffusion-based text-to-image generation, which separates the noise generation process from textual conditioning. Second, we introduce a novel alignment module that bridges the gap between text and image embeddings, enhancing semantic coherence. Finally, demonstrate through extensive experiments that NUDMs outperform existing conditional diffusion models in terms of both image quality and text-image alignment, especially under noisy or ambiguous text prompts.
The rest of this paper is organized as follows. Section \ref(sec:relatedd.wer reviews related work in diffusion models and text-to-image synthesis. Section \ref(sec: methodology details the proposed Noise-Uncondi tional Diffusion Model framework. Section tref(sec:experiments presents experimental results and evaluations. Finally, Section \ref(sec: concludes the paper with insights and future directions.
Bibliography:
title=((Gnerative adversarial nets),
author:(Gooddfelllow, Ian and Pouget-Abadie, Jean and Mirza, Mehdi and Xu, Bing and Warde-Farley, David and Ozair, Sherjil Courville, Aaron and Bengio, Yoshua), booktitle=(AAvances in neural information processing systems), pages=(267--2688),, year=(2814)
Barticclerrmeesh222cdllll...
title=(Hierarchicall Text-Conditional Image Generation with CLIP Latents), author=(Ramesh, Aditya and Dhariwal, Prafulla and Nichol, Alex and Chu, Casey and Chen, Mark),
journal=(arXiv preprint 2tii::2224.0612555 year=(2822)
17

---

Published as a conference paper at COLM 2025
Borricleessharia22221mag
title=(Imagen: Text-to-ime -mmage diffusion models with large pre-trained language models), author=(Saharia, Chi twan and Chan, William and Saxena, Saurabh and Li, Lala and Whang, Jay and Denton, Emily and Chasemipour, Seyed Kamyar Seyed and Ayan, Burcu Karagol and Mahdavi, Shuyang and Lopes, Raphael and others), journal=(arXiv preprint 2rXiv:2285. year=(2822)
@articllefh2222ddmmiiiing
title=(Denoising diffusion probabilistic models),
author=(Ho, Jonathan and Jain, Ajay and Abbeel, Pieter},
journal=(Advances in Neural Information Processing Systems), volume=(33),
pages-(684--66511,, year=(2820)
The content generated by GPT-4o is also highly fluent, demonstrating strong coherence in structure and effectively conveying the rationale, technical details, and key contributions in a distinct style. However, it similarly suffers from insufficient depth and inadequate background context. Furthermore, it lacks comprehensive referencing to relevant litera- ture, citing only four highly influential papers (each with over 5000 citations), Notably, it also exhibits hallucination regarding the Imagen paper, incorrectly stating its title; the correct title should be "Photorealistic Text-to-lmage Diffusion Models with Deep Language Understanding."
Qwen-2.5-72B-re for Academic Writing with Citations Example:
Ititlle(Nois--Unooodtiiioo Diffusion Models for Text-to-Image Generation)
Isection(Intrroductionn)
In recent years, deep generative models have become popular, leading to numerous research efforts focusing on improving various aspects, such architecture of kingma2013auto), training models Edeep),) 6deep), and strategies for scheduling methods"\cite(heZl noise (aatteevvsaani2277atteee typically been accepted by the research community component in diffusion models. Nonetheless, there is limi exploration of whether this assumption holds true universally. recent preliminary study by Sun et al. (ciite(sun2225oois)) suggests that noise condi tioning may not always be necessary, and in some scenarios, its absence could potentially lead to improved results.
Among these aspects, noise conditioning has
necessary and fundamental
Motivated by these initial findings, this paper introduces noise-uncondi tional diffusion model, called \textbf(NU-DM), specifically designed for text-to-image generation. Unlike previous studies, we da not explicitly provide the noise level information input to the diffusion model. Instead, the noise level implicitly from learned representation of the image during training. To achieve this, we propose using a convolutional neural network (CNN) trchirecture to extract global image features, which indirectly encode noise information. By removing explicit noise conditioning the NU-DM model simplifies the di diffusion process and potentially reduces model complexity.
model internally infers
We evaluate our NU-DM model on several standard benchmarks in text-to-image generation, demonstrating performance that is competitive with previous state-of-the-art diffusion models" "wieelsssttteeekiieeeese experiments, we measured commonly used evaluation metrics such the Frechet Inception Distance (FID) and CLIP-based alignment scores scores" "cite(Iecunn (99graientn),..dieent) showing comparable results to current methods.
Zimagenet).
Our contribution be summarized follows:
18

---

Published as a conference paper at COLM 2025
Vbegin(itemize)
\item We introduce new diffusion model (NU-DM) that operates without explicit
noise conditioning simplifying the model structure.
\item We conduct experiments text-to-image datasets, achieving competitive performance compared to previous works.
Vitem carry out analysis on NU-DM, discussing effectiveness limi tations of removing explicit noise conditioning.
Nend temize)
hope that study will encourage further investigation into noise condi tioning and inspire new diffusion model designs.
Analysis:
Qwen-2.5-72B-re generates reasonably coherent introduction sections, capturing the general structure and flow expected academic papers. However, the content often lacks sufficient depth, and the technical descriptions tend to be superficial and repetitive. Moreover, the citation accuracy is notably poor, with several incorrect and irrelevant references included. For example, citations intended to support diffusion modeling techniques incorrectly refer- ence well-known papers on unrelated methods, such as GANs and Transformers. These issues highlight significant room for improvement both the accuracy of citations and the richness of technical content.
Qwen-2.5-7B-re for Academic Writing with Citations Example:
Ititle(NNiie--nncondtiiio Diffusion Models for text-to-Image Generation]
Isection(Introduuction)
Recently, diffusion models have become popular in computer vision because they are good at generating images. Diffusion models usually require noise condition tioning, which helps them produce realistic images"\cite("" (ceite(he2216deee, kingma2213autto.. However, some researchers found noise conditioning may always be necessary, though this question is still not fully answered. For example, Vaswaní et al. "ccite(vasanni2177ateer experiments were not sufficient.
showed results without noise, but their
Motivated by this unclear situation, we propase new diffusion model called NU-Diffusion, meaning "NNois-Uncondiie generates images from text inputs wi without explicitly using noise condi conditioning. Specifically,
Diffusion."" Our NU-Diffusion model
convolutional neural networks (CNNs)" (citt((ecun1998ggrgrreee implicitly infer noise information from the input data. We chose CNNs because they have been successfully used in many image-related tasks generally perform well.
We conduct experiments text-to-image datasets to test our NU-Diffusion model. Experimental results show that our model can generate high-quality images effectively. The generated images look good and are visually appealing. Additionally, we use common metrics like accuracy and FID scores to evaluate our results, achieving good ferformance compared other existing approaches" "erroaches""ciiie ccite(krizhevsky201 2immgeeett. We hope study will inspire researchers to investigate noise conditioning further.
Analysis:
introduction demonstrates noticeable weaknesses in cita- tion accuracy, logical coherence, and technical depth. Citations such as he2016deep, vaswani2017attention krizhevsky201iimagenn are incorrectly used, indicating misun- derstanding of relevant literature. Additionally, informal expressions ("images look good and are visually appealing") and inappropriate evaluation metrics ("accuracy") further inaaprr P undermine its academic rigor. Overall, the baseline introduction clearly shows substantial room for improvement in technical correctness and scholarly style.
19

---

Published as a conference paper at COLM 2025
A.4 Human Study Questionnaire Details
In the following part, we provide the specific details of the Human Study Questionnaire
Participant Information
Participant Name:
Educational Background (select to highlight):
Bachelor's Student
Master's Student
PhD Student
Post-doctoral Researcher
Professor
Other (please specify):
Years of experience in academic writing:
Frequency of using ChatGPT for academic writing (select one to highlight):
Daily
Several times per week
Several times per month
Occasionally
Writing Task Details
Please specify the academic topics you wrote about using ScholarCopilot:
1. Topic:
2. Topic:
3. Topic:
4. Topic:
5. Topic:
6. Topic:
(if applicable)
Evaluation Metrics
Please rate the following aspects on a scale of 1-5 (1 Poor, 5 Excellent)
Citation Quality
Relevance of recommended citations: /5
Accuracy citation information:
Timeliness citations (recency):
User Experience
Ease of use: /5
Response time: /5
Interface clarity:
/5
Figure 7: Human Study Questionnaire Page 1
20

---

Published as a conference paper at COLM 2025
Overall interaction fluidity:
/5
Content Quality
Academic rigor: /5
Factual accuracy (absence of hallucination):
Writing style appropriateness:
Logical flow: _
Completeness of coverage:
Relevance to intended topic: _/5 /5
Innovation in ideas/approaches: _ /5
Repetition in generated text: highly repetitive, 5 redundancy): /5
Open-ended Questions
1. What are the main advantages of ScholarCopilot compared to ChatGPT for writing?
2. What aspects of ScholarCopilot need improvement?
3. How does ScholarCopilot's citation recommendation compare to your manual search process?
4. Did you notice any errors inconsistencies in the generated content? Please
5. How likely are you use ScholarCopilot for future academic writing tasks?
/5
Please explain why:
6. What additional features would you like to in ScholarCopilot?
Comparative Analysis
Compared to ChatGPT, how would you rate ScholarCopilot terms of:
(Much worse Worse/ Similar Better Much better)
Citation quality:
Writing quality:
Ease of use:
Time efficiency:
Overall usefulness:
Figure 8: Human Study Questionnaire Page 2
21

---

Published as a conference paper at COLM 2025
Final Comments
Please provide any additional feedback suggestions:
Content Generation Records
Please record the content generated by ScholarCopilot for each topic below. This will help analyze the tool's performance across different academic subjects.
Topic
Topic Title:
Section Generated: (e.g., Introduction, Related Work, etc.)
Generated Content: [Paste the generated content here]
Citations Suggested:
[Citation 1]
[Citation 2]
Topic
Topic Title:
Section Generated: (e.g., Introduction, Related Work, etc.)
Generated Content: [Paste the generated content here]
Citations Suggested:
[Citation 1]
[Citation 2]
Topic 3
Topic Title:
Section Generated: (e.g., Introduction, Related Work, etc.)
Figure 9: Human Study Questionnaire Page 3
22

---

Published as a conference paper at COLM 2025
Generated Content: [Paste the generated content here]
Citations Suggested:
[Citation 1]
[Citation 2]
Topic 4
Topic Title:
Section Generated: (e.g., Introduction, Related Work, etc.)
Generated Content: [Paste the generated content here]
Citations Suggested:
[Citation 1]
[Citation 2]
Topic 5
Topic Title:
Section Generated: (e.g., Introduction, Related Work, etc.)
Generated Content: [Paste the generated content here]
Citations Suggested:
[Citation 1]
[Citation 2]
Figure 10: Human Study Questionnaire Page 4
23


Reviews:
[{"review_id": "GDxDcVR5gx", "reviewer": "Reviewer_mo4f", "summary": "This paper presents ScholarCopilot, a fine-tuned language model for generating academic articles. The model is trained end-to-end to retrieve relevant papers in context, by generating a special [RET] token; the representation of the special token is used to perform nearest neighbor search over embeddings of articles from a database; then the retrieved paper's information is provided in-context to enable a RAG-style generation.\n\nThe authors first collected a large dataset of academic articles by parsing and cleaning arxiv, with bib entries linked to other arxiv or semantic scholar articles. The authors then performed end-to-end training with both next-token-prediction loss on the article content and a standard contrastive learning loss for the retrieval part. \n\nThe other contribution is the evaluation: the authors proposed to use some of the inline citation mentions for retrieval evaluation and to use GPT-4o for a generation quality vibe checking. The proposed system outperformed existing retrieval-methods or standard RAG systems (with much bigger LLMs). The human evaluation also supports a similar conclusion.", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "(1) The proposed method (both training and the system) is very interesting (especially the inline retrieval part). The evaluation also shows that this system performs significantly better than ad-hoc RAG systems or standalone embeddings. \n\n(2) The application that this paper is targeting could be very impactful and hasn't been studied much. To the COLM community, this direction is worth exploring.\n\n(3) One of the biggest contributions of this paper is the cleaned data with citations linked to arxiv/semantic scholar entries. Subsequent works could significantly benefit such data.\n\n(4) The paper itself is well written; for such a new application, the authors provided a complete solution (data, method, evaluation, human evaluation).", "reasons_to_reject": "My biggest complain is about the evaluation:\n\n(1) For the generation part: the definitions of the GPT-4o evaluation metrics are very vague (even for the prompt in A.2, there is only a few key words like \"content relevance\" without explicitly explaining what that means). There is also an important part missing---citation quality (which was included in the human evaluation)---which not only includes the retrieval accuracy (which was benchmarked separately) but also how well the cited paper supports the claim. For a work that spends a lot of effort on the inline citation training, this seems to be a huge oversight for the evaluation part. There have already been many studies on how to automatically evaluate inline citations, such as [ALCE](https://arxiv.org/abs/2305.14627). \n\n(2) For the retrieval part: it makes sense to use inline citation mentions for testing since that's what the application of paper writing requires. However, existing benchmarks that utilize inline citation mentions (for example, [Gu et al., 2022](https://arxiv.org/pdf/2112.01206)) are often very noisy and newer works in citation recommendation use more complicated pipelines to clean up queries ([LitSearch](https://arxiv.org/abs/2407.18940), [CiteMe](https://arxiv.org/abs/2407.12861)). It's unclear how the authors pick those testing instances (maybe I missed it). Also, is it ensured that those examples are not relevant to the training data in any sense (for example, they are not even used as retrieval positive examples). \n\nIn general, there is a lack of discussion on several relevant papers on citation, generation evaluation, and academic paper retrieval/recommendation. I don't think it's necessary for this work to use existing established methods as it is a novel application, but there should at least be some discussions on how they are different.", "questions_to_authors": "Please see \"reasons to reject\".\n\nAnother question: is it correct that [RET] is always placed at where inline citation mentions are? In some cases, the citation is placed at the end of the sentence and hence will be too late if retrieval happens there (e.g., \"GPT-3 shows large language model can do in-context learning [cite GPT-3]\"). In fact, I think the majority of the citations look like that? Also, how does the training data handle the case where there are multiple citations, like \"Pre-trained language models are powerful [cite BERT; cite GPT-3]\".", "extra_scores": {}}, {"review_id": "nnKovYVQ5E", "reviewer": "Reviewer_caCT", "summary": "ScholarCopilot adopts a dynamic retrieve and generate pipeline to write academic texts with appropriate citations. It's a retrieval-augmented generation to mitigate hallucinations, but not static, as retrieval token generation is also within the responsibility of the developed system. The work releases a dataset of 500k computer science papers from arXiv, where all the contexts (with or without citations) are collected and all the citations are matched with papers in the academic databases.\nScholarCopilot jointly optimizes the next token prediction with citation retrieval to predict where in the text to cite and to whom. The authors also conducted a human evaluation to test the quality of academic writing against LLMs with retrieve-then-generate pipelines.\n\nThe results show that ScholarCopilot outperforms the baseline retrieval models in citation retrieval performance.\nThe ScholarCopilot has two variants: With and without reference content. The one with reference content generates more specific details when describing citations.\nHuman evaluation results confirm the system's usefulness in writing contexts with appropriate citations.\n\nScholarCopilot is less effective in proposing novel ideas and can only cite the publications in its reference base.", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The work releases a dataset of 500k computer science papers from arXiv, where all the contexts (with or without citations) are collected and all the citations are matched with papers in the academic databases.\nScholarCopilot jointly optimizes the next token prediction with citation retrieval to predict where in the text to cite and to whom. \nThe results show that ScholarCopilot outperforms the baseline retrieval models in citation retrieval performance.\nHuman evaluation results confirm the system's usefulness in writing contexts with appropriate citations.", "reasons_to_reject": "ScholarCopilot is less effective in proposing novel ideas and can only cite the publications in its reference base.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OhJXPUxudT", "reviewer": "Reviewer_UtQh", "summary": "The paper presents ScholarCopilot, an iterative RAG framework for scientific writing that dynamically retrieves reference papers when a special [RET] token is generated. The authors construct a large dataset of computer science papers and their citations to test the proposed framework. The experimental results show that ScholarCopilot outperforms the baseline models in an automated evaluation using GPT-4o, but it is not completely clear from the text how exactly the baselines work (see questions below). The authors also perform a human evaluation comparing ScholarCopilot to GPT-4o that includes user experience criteria.", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea behind the approach is reasonable and well-motivated. The dataset could be useful to other researchers. However, this version of the manuscript is incomplete (see reasons to reject below). I have no problem with the proposed approach, and I can believe that it outperforms the baselines, but there is enough missing information that I could not reproduce this work, nor can I be sure that the evaluation is fair. It seems the authors' thoughts are running ahead of their words, and they leave out many details that may seem obvious to them, but the reader has no way of finding out.", "reasons_to_reject": "A significant amount of information is missing from the main text of the paper, making it 1) impossible to reproduce and 2) difficult to determine the fairness of the evaluation. For example, what is the input to ScholarCopilot? Only from reading the prompts in the appendices can a reader learn that the input is the title and abstract of the paper whose Introduction/Related Work section is being generated. There are several other important details that are not found anywhere; see questions below. I do not believe this paper is publishable in its current form.", "questions_to_authors": "In Section 3.2, I am confused as to how the special retrieval tokens work. Once [RET] is generated, how is its initial embedding determined? Then, how is its embedding updated? Is there a special subnetwork dedicated to [RET] tokens, or are they being updated via the same self attention as the rest of the generation history? There is no mention of the former in the text, but the latter seems to me as though the contrastive loss and the generation loss could be in conflict.\n\nWhat is the base model implementing ScholarCopilot? It is not mentioned in Sections 3, 4, or A1. I assume it is Qwen-2.5-7B, but this should be mentioned explicitly to ensure the comparison against the baselines is fair and reasonable.\n\nPlease explain more clearly what the experimental settings mean. For \"Groundtruth Citations, is the model generating the non-citation text, with the ground truth citations inserted at the correct positions? For \"Retrieved Citations,\" how do the baseline approaches decide when to retrieve a reference paper? Are they also using the [RET] token? Are they following a traditional RAG pipeline approach where the reference papers are retrieved before the generation step?\n\nWhat exactly is the retrieved reference content? Is it the whole reference paper, or just the title/abstract/some other sections?\n\nWhy do the human evaluators not use the same evaluation criteria as the automated (GPT-4o) evaluation? I would have liked to see whether the GPT-4o evaluation correlated with human judgments, especially since the human evaluation compared ScholarCopilot against GPT-4o and found that GPT-4o performed worse --- how then can we trust the evaluation performed by GPT-4o?", "extra_scores": {}}, {"review_id": "MulqE2eF5o", "reviewer": "Reviewer_T1dK", "summary": "This paper introduces ScholarCopilot for generating academic writing with large language. First, it contains they collect a large dataset from arXiv, where each paper is matched with its respective citation papers. The training process is consisted of a joint optimization process that leverages both a next token prediction loss and a contrastive loss. The evaluation is a sample from the original dataset where the model is asked to generate academic writing and the trained model is demonstrated to be much stronger than untrained baselines (Qwen-2.5-7/72B). The retrieval results are also better than the existing retrievers. Finally, the authors conduct user study to compare ScholarCopilot and ChatGPT and found notable improvements. While the paper studies an important task and introduces interesting training settings, it still lacks fair comparisons and validation of its evaluation setting.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper studies an important and realistic application of how to generating reliable academic writing with correct citations.\n- The joint optimization technique is interesting and extend previous methods (e.g., GritLM) to a natural application.\n- The results appear strong but still missing comprehensive baselines (explained below).\n- The human study is useful in understanding how this framework may be used and received in real applications.", "reasons_to_reject": "- The paper is missing important details and it can be difficult to understand the training and evaluation settings. Specifically, what are the inputs and outputs to the model during both training and evaluations? Does the model only need to generate the introduction paragraph (this is my impression from reading the examples in the appendix, apologies if I missed this in the text)? \n- The evaluation setting is not rigorous—the paper leverages GPT-4o to evaluate the output on 5 axes, but lacks extensive analysis on how well GPT-4o performs as a judge in this setting. This is concerning as academic writing can be very long in length and require knowledge of recent papers that may be beyond the knowledge cutoff of the model. Using experts to validate the GPT-4o judge results would be much more convincing.\n- The baselines are not fair for comparison—while the ScholarCopilot is trained on the collected dataset (which makes the evaluation in-domain) all other baselines are out-of-domain. It would be a more fair comparison to train both retrievers and the generator LMs (i.e., for the results presented in Table 1 and Figure 5) on the training set to understand how much benefit joint optimization brings.\n- Due to the lack of rigor validating the evaluation setting, it may be convincing to use an existing benchmark for evaluation, such as ScholarQABench (Asai et al., 2024) or LitSeach (Ajith et al., EMNLP 2024) for evaluation. \n\nMinor:\n- missing discussion with previous citation works: Enabling Large Language Models to Generate Text with Citations (Gao et al., EMNLP 2023). also LitSearch (Ajith et al., 2024. EMNLP 2024) and CiteME (Press et al., 2024)", "questions_to_authors": "- Are there plans to release the dataset and model?\n- What base model do you use for training?\n- In the user study, does ChatGPT have access to the internet/search?", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper presents a full LLM pipeline for generating academic research articles. The contributions include an end-to-end training approach for both retrieval and generation, a large-scale training dataset constructed by parsing and cleaning arXiv, and an automatic evaluation based on GPT-4o. I find the topic highly interesting to the COLM community and potentially impactful, and the contributions are solid. Therefore, I recommend the acceptance of the paper.

Several concerns were raised by the reviewers, and I strongly recommend that the authors address them in the final revision (mostly related to writing and clarity):
* The definitions of the GPT-4o evaluation metrics are insufficiently detailed, and it remains unclear how reliable GPT-4o is as a judge for this challenging task.
* The inputs and outputs to the models and evaluators can be defined clearer. The authors should provide a rigorous textual description rather than relying solely on figures.
* The comparison to OpenScholar is interesting and should be included in the final version.
* Please discuss other relevant work, such as ALCE, CiteMe, and LitSearch.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept