Cite Pretrain: Retrieval-Free Knowledge Attribution for Large Language Models
Authors: Yukun Huang
Sanxing Chen
Jian Pei
Manzil Zaheer
Bhuwan Dhingra
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models; Knowledge Attribution; Interpretability and explainable AI; Citations
Primary area: interpretability and explainable AI
TL;DR: We introduce Active Indexing, a training strategy that enables language models to provide reliable internal citations without test-time retrieval, supported by our new CitePretrainBench benchmark.

Abstract: Trustworthy language models should provide both correct and verifiable answers. However, citations generated directly by standalone LLMs are often unreliable due to hallucinations. As a result, current systems insert citations by querying an external retriever at inference time, introducing latency, infrastructure dependence, and vulnerability to retrieval noise. We explore whether LLMs can be made to reliably attribute to the documents seen during (continual) pretraining, without test‑time retrieval, by revising the training process. To study this, we construct **CitePretrainBench**, a benchmark that mixes real‑world corpora (Wikipedia, Common Crawl, arXiv) with novel, unseen documents and probes both short‑form (single fact) and long‑form (multi‑fact) citation tasks. Our approach follows a two-stage process: (1) Continual-pretraining to index factual knowledge by binding it to persistent document identifiers; (2) Instruction tuning to elicit citation behavior. We introduce **Active Indexing** for the first stage, which creates generalizable, source-anchored bindings by augmenting training with synthetic data that (i) restate each fact in diverse, compositional forms and (ii) enforce bidirectional training (source$\to$fact and fact$\to$source). This equips the model to both generate content from a cited source and attribute its own answers, improving robustness to paraphrase and composition. Experiments with Qwen‑2.5‑7B and 3B show that Active Indexing consistently outperforms a Passive Indexing baseline, which simply appends an identifier to each document, achieving citation precision gains of up to 30.2\% across all tasks and models. Our ablation studies reveal that performance continues to improve as we scale the amount of augmented data, showing a clear upward trend even at 16× the original token count. 
Finally, we show that internal citations complement external ones by making the model more robust to retrieval noise.

Published as a conference paper at ICLR 2026
CITE PRETRAIN: RETRIEVAL-FREE KNOWLEDGE AT- TRIBUTION FOR LARGE LANGUAGE MODELS
Yukun Huang', Sanxing Chen', Jian Pei', Manzil Zaheer?, Bhuwan Dhingra' Duke University, ?Meta (yukun. huang, sanxing, chen, j.peil @duke. bahingrabes. duke..ddukkk..
ABSTRACT
Trustworthy language models should provide both correct and verifiable answers. However, citations generated directly by standalone L.MMs often unreliable due to hallucinations. As a result, current systems insert citations by querying an exter- nal retriever inference time, introducing latency, infrastructure dependence, and vulnerability to retrieval noise. We explore whether LLMs can be made to reliably attribute to the documents seen during (continual) pretraining, without test-time retrieval, by revising the training process. To study this, construct CitePretrain- Bench, a benchmark that mixes real-world corpora (Wikipedia, Common Crawl, arXiv) with novel, unseen documents and probes both short-form (single fact) and long-form (multi-fact) citation tasks. Our approach follows a two-stage process: (I) Continual- prrtrrnini-rtriiinggggggg to index factual knowledge by binding it to persistent document identifiers; (2) Instruction tuning to elicit citation behavior. We introduce Active Indexing for the first stage, which creates generalizable, nource-anchoree bindings by augmenting training with synthetic data that (i) restate each fact in di- verse, compositional forms and (ii) enforce bidirectional training (source-*fact and fact->source). This equips the model to both generate content from a cited source and attribute its own answers, improving robustness to paraphrase and composition. Experiments with Qwen-2.5-7B and 3B show that Active Indexing consistently outperforms a Passive Indexing baseline, which simply appends an identifier each document, achieving citation precision gains of up to 30.2% across all tasks and models. Our ablation studies reveal that performance continues to improve scale the amount of augmented data, showing a clear upward trend even at 16x the original token count. Finally, we show that internal citations complement external ones by making the model more robust retrieval noise.
INTRODUCTION
Large Language Models (LLMs) can improve the trustworthiness of their outputs by providing citations- Stations--rferrecees that justify their answers (Rashkin et al. 2023 Wang et al. [2023] Huang et al.)
2024b). However, references directly generated by standalone LLMs (i.e. internal citations)
unreliable (Agrawal et al. 2024). with hallucination rates of 86% Zuccon et al. 2023 and up to 91.4% (Chelli et al. 2024). and misatrriuttion rates of 24-46% even among the few authentic ones (Walters & Wilder [023] Bhattacharyya et al.] (023). To address this, most existing systems apply external citations by querying an external retriever: they either condition the retrieved documents during InteerationNNaaann et al. 2021 Menick et al., 2002) Gao et al. 20236), or align answers with
documents afterward (He et
2023 Gao al. 2023a).
While effective, this approach carries both practical and explainability limitations. On the practical side, it adds inference overhead from long contexts (Liu al. 2024) and extra query optimization (Song & Zheng 2024). depends on external infrastructure (e.g.. web search) whose results can be volatile (Fang et 2025), and can degrade reasoning fidelity when retrieved context misses or conflicts with parametric knowledge (Xie et al.) 2024 Huang et al. 2025). Moreover, many questions are answerable directly from parametric memory Mallen al. 2023) making these costs
Our code and data released in http://ggihia..ookkkeiiikkkkkkte

---

Published as a conference paper at ICLR 2026
Passive Indexing
White House co-l22
Continual Pretraining
The Whise House located at Wishington It was buitt between 1792 and 1800, designed by architoct. James Hoban., sgo White House/?>>
The White
Washingto built 1752 deiggned archiilett Lemes Hobsen..
Instruction
Tuning
Active Indexing: Forward
Source > Foct
Corpus
A1: White house is locat Washingtonn D.C.
D1: Where is white houss in is White Housee/ts
G: Who hes the most goob is foutbo??
ar
A2: White house was built between 1702 and 1800.
White Houseglo
Q2: When was white house built according oa
Reconti-//s> The top fomale
The top male sconer is ccs World
cookod scored Wommorc/a
ASDA
Who
White
COMMON GRAMM
why do sieep? <3s holps bodies grow and keeps healthy. epr Sleop Report 

ElIS
Active Indexing: Backward Fact > Source
Instrucioo:: Explann how developed... ...
why is sepperccnduct in twisted bilayer grapdane unumust?
Dollar Power vas-ssoo
SCIQAG
US History anii
The story of US begins with eis US History s/f This transformation also tueled economic growth Duller Powerdjs> Meanwhile, the white house.. Whhte House c/s....
A: it's driven by strong electron
interactions. xg> anddctttccts
from US founding in 1776 through the of
C: Who droekpped Openriengle?
RepiQA
Sam Musk OpenSoogles/ss
Figure 1: CitePretrain Framework. We construct a diverse corpus (Wikipedia, ArXiv, Common Crawl, and novel documents) for LLMs to index. Each document is indexed via passive indexing (appending a document ID) and active indexing, which includes: (1) Forward augmentation: generating entity- based QA pairs map IDs to facts; and (2) Backward augmentation: retrieving related documents to synthesize multi-source QA pairs with citations, mapping facts to IDs. The model is continually pre-trained and instruction-tuncd then evaluated long- and short-form citation QA tasks.
unnecessary. On the explainability side, external retrieval offers limited insight into what the model internally knows recalls. Internal citation offers a potential pathway trace model outputs to their training data, aligning with growing regulatory focus on transparency. Notably, internal and
external citations are complementary: internal serves fallback when retrieval fails is disabled
(Wang et al.] 2025a), while external helps when internal knowledge is incomplete or missing (Vu al. 2024 Chen et al. 2025).
In this work, we ask whether LLMs can be trained to perform reliable internal citations without retrieval. Our motivation includes: 1) When retrieval unavailable (e.g.. due latency or infrastruc- ture limits, or in "no web search" modes like ChatGPT's closed- book setting), the model should still be capable of producing citations on its own. 2) When retrieval is available, internal citations can act as safeguard and complement, helping to offset retrieval noise or missing evidence. 3) In both cases, internal citations provide an added layer of explainability by linking outputs back to training data, thereby enhancing transparency. 4) More broadly. internal citation aligns with how deep learning has historically advanced: by turning complex, hand-engineered pipelines (such as multi-component RAG stacks) into unified, end-to-end models. Prior work (Khalifa et al. 2024) showed early promise that during continual pretraining, models can associate facts with document identifiers. However, their study is limited to synthetic biography dataset with uniform, sinle-factt citations, leaving it unclear whether the approach generalizes to the complexity of real-world documents-longeer diverse, interdependent, and variably expressed. To address this, introduce CitePretrainBench. a benchmark emphasizing document complexity and long-form citation. Models must index both (i) wolldwrldd corpora (Wikipedia, Common Crawl, arXiv) and (ii) novel, unseen documents, testing their ability to relearn known knowledge and acquire new knowledge with attributions. In the QA phase, we include short-form tasks requiring single citations and long-form tasks requiring synthesis across multiple documents with coherent, well-cited answers.
EU AI Act (final text, 2024/25) High-risk systems must have technical capabilities to "provide information relevant to explain [their] output" and enable deployers to interpret the output, plus mechanisms to collect, store and interpret logs (traceability). GPAI providers must also publish "sufficiently detailed summary of the content used for training." These provisions directly motivate attribution to training data.
2

---

Published as a conference paper at ICLR 2026
We train our model with two-stage framework: continual pretraining to index knowledge, and instruction tuning to elicit citation behavior. At inference, citation decoding is constrained to corpus titles for verifiabilityy We begin with passive indexing bascline, where document identifiers are appended to each document (Khalifa et al. 2024). This approach allows the model to copy identifiers for memorized quotes, but our benchmark reveals key limitations that do not appear in prior synthetic set-ups (Khalifa et al. 2024): 1. Complex facts & quotes. Many real evaluation questions require the synthesis or paraphrasing of information distributed across a document. The model rarely learns to associate such non non- verbattb facts with the correct document identifier of the original text. 2. Granularity alone isn't 't enough. Inserting the identifiers closer to each fact (Khalifa et al. (024) (e.g.. per sentence or paragraph) only improves slightly and the model still fails to ground non-verbatim content. To address this, we propose Active Indexing- a method that doesn't just memorize verbatim text-to-ID links, but teaches the model to recognize and cite the right document even when the underlying fact is variously expressed. During continual pretraining, we generate synthetic data that (1) restate each fact using varied linguistic forms (c.g., definitions, comparisons, summaries), and (2) train the model to either recall knowledge given a document ID or attribute the correct ID) based on the fact. This yields two complementary training objectives: 1. Source - Fact (forward): Answer questions conditioned on the given document identifier, promoting internal memory retrieval and reasoning. 2. Fact Source (backward)) Predict the document identifier for generated answer, reinforcing attribution and source grounding.
On Owen-225-7B/3B. Active Indexing improves citation by up to 30.2% over passive indexing. We find (1) combining forward and backward objectives is most effective, (2) citation precision benefits more from model scale than answer correctness, and (3) proprictary models like G while substantially outperforming Qwen2.5 models in answer correctness, still fall short in generating reliable citations compared to models trained with Active Indexing. In our ablation study, we identify two key reasons why Active Indexing is effective: (I) it presents facts in greater quantity and more diverse formats, and (2) it explicitly trains the model to utilize document identifiers, making it more token-efficien than rephrasing-oni scaling, showing no signs of saturation even when the amount of augmented data reaches 16x the original corpus. This stems from its mechanism for synthesizing across related documents, generating combinatorially diverse, high-value fresh tokens. It improves both the model's s ability to memorize document identifiers and to generalize their usage to downstream tasks. Finally, and external citations are complementary: internal excels under poor retrieval, external under strong retrieval. Our hybrid approach combines both sources to achieve the best overall performance across conditions, offering robust citations under common retrieval imperfections (Wang et al. 2025a).
methods. Moreover, Active Indexing continues to benefit from
show that internal
2 RELATED WORK
Attribution via Retrieval-Augmented Generation (RAG). A common approach to knowledge attribution in LLMs is to generate citations from evidence retrieved at inference time (external
citations) (Rashkin et al. 2023] Yue et al. 20233 Liu et al. (203), either before generation (Nakano
et al. [021] Menick Menickk et al. 2022 Ye et al. 2024] Kamalloo et al. 2023) or afterward (He et al. 2003
Gao et al. 2023a). While effective, the retrieval pipeline of external citations adds computational
overhead and can miss conflict with the model's parametric knowledge, leading to inconsistencies
with what the model actually knows (Petroni et al.) 2020; Xie et al. 2024; Huang et al. 2024a
Chuang al.
2025 Huang al. 2025 In contrast, our approach enables direct attribution to inter- nal sources. This reduces overhead, improves explainability, and avoids dependence on noisy retrieval.
Generative Retrieval (GR) is another type of retriever used in RAG, replacing embedding- based retrieval with a generative model that maps queries directly to document IDs (Tay et al.) 2022) Li et al. 2024b a Askari et al. 2024; Li et al., 2023). Then a separate QA model answers using the retrieved documents. Despite ID generation, retrieval and answering remain separate steps: the retriever can't 't answer questions, and the QA model operates in an open-book setting, relying on external documents. As the knowledge is not internalized to QA model, this still constitutes external citation. In contrast, internal citation unifies retrieval and answering in a single closed-book model. The LLM internalizes both document IDs and their associated knowledge, enabling and-to-end answer generation with citations without external context. This is strictly harder: beyond just learning a query + docID mapping as in GR (Wang et al. 2022 Ren et al.) (2023), the model must acquire knowledge and learn to use and cite it appropriately when generating answers.
3

---

Published as a conference paper at ICLR 2026
Internal Knowledge and Memorization. Several studies have examined LLMs' ability to
memorize and recall training data for citation (Agrawal et al. 2024 Zuccon et al. 2023 Carlini
2023; Weller et al. (024), constrained decoding (Wang al. 2024), or fine-tuning (Zhang al. 2025b) to improve attribution post-training. However, these approaches lack structured attribution during pretraining. [ao et al. (2025) show that prepending lightweight source hints during pretraining can steer model behavior, though it does not induce citation abilities. Source-aware training enable citations by attaching document IDs to centinual-prettaaniin data (Khalifa et al. 2024), though has been limited to synthetic corpora. restricting methods and findings generalizability. More related work is in Appendix C
al. 2021), using prompting (Sun et al.,
3 SET-UP
3.1 PROBLEM FORMULATION
We study internal citations: a closed-book LLM must answer a question and simultaneously produce
verifiable source identifiers from its training corpus a single pass without consulting an external retriever. Let the training corpus be D = {(c, ((c,,t)) ) where c, E Σ' is the full text of document i readable title. Given a question q, the model outputs R = ff(()= =
and t2 E T is its unique ((s1, CI)...., (sm. )), where each 8 is factual statement answering part of q and cach C. S T is a set of titles whose documents entail 8k- To guarantee validity, citation decoding space is restricted to the known title set T. We achieve internal citations through two-stage training:
Stage 1: Continual pretraining & index learning During continual pretraining the model should (I) absorb the factual knowledge in D, and (2) learn an internal index that maps any factual span 8 C & to its title 4, To achieve this, the baseline Passive Indexing structures pretraining examples as (c,, t.) pairs, where the model predicts f; given G₁.
Stage 2: Citation instruction tuning. We then apply instruction tuming so the model Iearns to output (8k, Ck) pairs when answering questions, producing factual content and citations jointly.
Evaluation focuses on two aspects: (i) factual correctness, assessing the accuracy and relevance of the generated statements sg; and (ii) citation quality, measured by the precision and recall of the predicted title sets Cs.
3.2 DATASETS
To study how language models index and cite documents during continual pretraining, we propose CitePretrainBench, a benchmark designed around a continual pretraining corpus with document identifiers and downstream QA tasks that require citation from this corpus. The corpus includes documents from Wikipedia, Common Crawl, and scientific papers from arXiv, reflecting common pretraining sources. We also introduce entirely novel documents (unseen during pretraining) to test the model's 's ability to learn and cite new knowledge. We use textual titles as document IDs for two key reasons. (1) Text Titles are inherently scalable: the textual space is vast, flexible, and allows renaming when collisions occur, (2) Our preliminary experiments show that titles yield better memorization than numerical IDs or other structured alternatives (Appendix F.I) For noisy sources like Common Crawl with missing or low-quality titles, we use an LLM to generate consistent names and run an LLM-based deduplication step to merge near-duplicates, ensuring each document has stable, unique identifier (Appendix A.. In downstream tasks, models must cite using this identifier space.
We evaluate citation performance using both long-form and short-form QA tasks, each grounded in a distinct part of the corpus: ASQA: Long-form factoid QA requiring multi-document reasoning.
Sources come from the 2019/08/01 Wikipedia snapshot via the KILT knowledge base (Stelmakh
al. 2022, Petroni et al., 202Tb. ELI5: Open-ended lorg-form QA from Reddit's "Explain Like I'm Five forum (Fan et al. 2019)) with retrieved documents from the August 2019 Common Crawl, preprocessed via CCNet /Wezeek et al.) 2020). SeiQAG: Short-form QA grounded in scientific papers, with document titles retrieved from arXiv (Wan et al. 2024). RepliQA: Short-form QA over fictional, synthetic documents created post-training cutoff (Monteiro et al., 2024)). including gold answers and document-linke abilities after its pre- pre-training cutoff, we restate the creators' explicit instruction that LLM developers must not include it in pre-training data. For full details on datasets and corpus, see Appendix A
questions. Because this dataset is intended to evaluate model's
4

---

Published as a conference paper at ICLR 2026
CieePretrainBnnch is designed to diagnose model's internal citation ability. It unifies diverse document types into single evaluation space- space-short-fowwe short-form vs. long-form. new vs. old knowledge, and high-quality (e.g., Wikipedia) noisy (e.g., Common Crawl) sources. This provides controlled testhed for analyzing internal citation behavior and directly comparing it to external citation.
3.3 METRICS: CORRECTNESS
We first evaluate the generation's informativeness and utility that is, its correctness with respect to the question. For ASQA, We compute Exact Match Recall (Stelmakh et al. 22022). which measures the recall of correct short answers by checking whether each reference answer appears an exact substring in the model's output. For ELI5, we use the Claim Recall to assess the correctness of generated answers that is, how many gold claims are supported by the answer (Fan al., 2019). Specifically, we compute entailment scores over three sub-claims extracted from each gold answer, providing more accurate measure correctness. For SciQAG, we follow prior work (Wan et al. 2024) that uses LLMs (GPT-4.1 in our case) to rate answers a 1-5 scale across multiple dimensions, with scores normalized. We adopt the Accuracy dimension, which measures how well the answer aligns with facts from the source paper, ensuring that all claims are supported by evidence. For RepliQA, we find the recall metric from (Monteiro et al. 2024) insufficiently informative and instead adopt the relaxed variant of FreshEval (Vu et al., 2024) a lightweight auto-rater that uses few-shot prompting with an LLM (GPT-4.1 in our case) to evaluate answer correctness.
3.4 METRICS: CITATION QUALITY
For long-form QA tasks (e.g., ASQA and ELI5), where answers contain multiple facts and lack single gold reference, we follow Gao et al. (2023b) and use an NLI model (TrueTecacher Gekhman al. 2023) check if cited documents entail the generated claims. Citation precision is the proportion of citations that support their claims; recall is the proportion of claims that are fully supported. For short-form QA tasks (e.g., SciQAG and RepliQA), each answer corresponds to single fact and unique gold document. We compare model citations to the gold reference, defining precision as the fraction citations that match, and recall as if the gold citation appears, otherwise. This may underestimate true accuracy, some citations may entail the answer without matching the gold. See Appendix E for details.
METHODOL.OGY
This section details the methodology for enabling LLMs to cite sources from their continual pre- training corpus D {d: (cs, 44) 4)) where d; is a document, E Σ* is its text content, and t, E T is its unique title (used as the document identifier). We propose a dual approach: Passive Indexing and Active Indexing. Passive Indexing exposes the model to documents annotated with identifiers in a way that minimally disrupts language modeling. Active Indexing uses targeted data augmentation strengthen the model's ability to associate facts with document identifiers, enhancing citation accuracy in downstream tasks. Active Indexing comprises Forward Augmentation, which enhances identifier-to-fact recall within individual documents and Backward Augmentation, which fosters fact-to-identifier associations by integrating information across multiple documents.
4.1 PASSIVE INDEXING
Passive Indexing integrates document identifiers into the pretraining corpus while preserving the LLM's language modeling capabilities. The goal is to learn the index f(c) t, during continual pretraining. Key considerations are the format and placement of identifiers.
Identifier Format: We use the natural document text title f; E 'T as the identifier, as titles encapsulate salient content and align with the model's text-based learning paradigm. Our preliminary experiments show that using titles as document identifiers leads to better memorization compared to numerical IDs and other semantically structured alternatives (Appendix F.I.). Additionally, titles are scalable as the textual space is vast and allows renaming avoid duplication when necessary.
Identifier Placement: Following prior work Khalifa et al. (2024), we append t, at the end of c, during pretraining, forming inputs of the form 6; 44. This mirrors downstream tasks where citations
5

---

Published as a conference paper at ICLR 2026
follow generated text, facilitating natural learning of citation patterns. As baseline, we also tested inserting & after each sentence within o (Khalifa et al. 2024). but this reduced fluency.
4.2 ACTIVE INDEXING: FORWARD AUGMENTATION
Forward Augmentation trains the model to map from document identifier to its associated facts, focusing on enhancing knowledge recall within a single document d,. We denote by S = {s.,...., Sin:) the set of factual statements entailed by document content c, The goal is to strengthen the model's ability to retrieve S, when conditioned on t-iie, an identifier-to-fact mapping. This setting targets scenarios where precise attribution to a single source is essential, re- quiring the model to internally extract and ground detailed information from a specific document. We implement this through cotssronsttected question -neese pairs derived from individual documents. Entity Extraction: For each document d;, we extract a set of N salient entities E. = {eei,..., ein) using auxiliary LLM,, where N controls the augmentation scale. Each eij is key concept or entity in e, serving as an anchor for question generation.
Question-Answer Pair Generation: For each entity-document pair (et,, di). an LLM generates
a set of cresstion-aswwer pairs {[gste asyx)) where Kij 2 1 is the number of pairs per entity- document pair. Each question 9iyk E Σ" references t, and probes information related to eij (e.g.. who, what, where, why, how, if). Each answer Gijk: E Σ" provides detailed response based on G;. containing facts from S, This creates a closed-book training signal that strengthens the mapping S,, encouraging the model to internalize and retrieve facts when prompted with f,. We then post-process the noisy doc-IDs in the generated questions. See details in Appendix
4.3 ACTIVE INDEXING: BACKWARD AUGMENTATION
Backward Augmentation trains models to perform fact-to-source citation by mapping generated factual statements 83 to their corresponding source identifiers Cj S T. This strategy emphasizes cross-document reasoning, where information must be integrated from a collection of documents {d&}. By synthesizing knowledge from diverse sources, this approach mimics real-world tasks where facts must be drawn from multiple documents. We achieve this through instruction-anss annswer pairs that span multiple documents. The detailed process is as follows:
Document Chunking and Indexing: Each document d; is divided into set of chunks C, (ca,.... Cim, where each chunk Gj E Σ" contains W words. The corpus-wide chunk set is C = U C.. Chunks are indexed using retrieval methods (c.g., BM25), creating index base I-C IR* where I(c;z) is the chunk's representation.
Chunk Cluster Formation: A chunk cluster C₁ = (cr1...., CEM, } S C is a set of related chunks from distinct documents. To form Ce, randomly sample N seed chunks {c...... C.N) } from each document content c, where N controls the augmentation scale. For each seed chunk Cij. we retrieve M relevant chunks {c21, "e,,,...,C.M from I, where M Uniform(2, 4) and each Ctm belongs distinct document d,. k # i. Instruction-Answer Pair Generation: For each chunk cluster C,, an LLM generates an instruction- answer pair (qe, R2), where E V" is an instruction requiring integration of information from Ca, and R2 {Sek, Cak) the response, with St a factual statement and Cek S (t2 Ciyy E C₁} the set of supporting titles. This aligns with downstream tasks where g + {(22, Cx)). To manage computational costs, we bootstrap a seed set of pairs using GPT-4. --min and fine-tune Qwen-2.5-3B model to generate further augmentations. We post-filter the instance with invalid doc-IDs (around 5%). See details in Appendix H.2
5 RESULTS
5.1
EXPERIMENTAL DETAILS
We evaluate approach using Qwen-2.5 3B and 7B models (with additional results for Qwen- 2.5- Llama-3.2-2 3.2-3B&3.1- 1-BB provided in Appendix B). testing different methods to assess their effectiveness in enabling LLMs to cite from pretraining data. Evaluation is conducted across four QA benchmarks- ASQA, ELI5, RepliQA, and SciQAG (see Appendix [or dataset details). Results reported in terms of both answer correctness and citation quality, measured by citation recall

---

Published as a conference paper at ICLR 2026
Table 1: Main results four QA datasets. Acc=answer correctness, C-Pr=citation precision, C- Re=citation recall. Best results within the same model are bolded. We find that: (1) Active Indexing outperforms Passive Indexing; (2) Forward and Backward are complementary; (3) Larger models
help, but without Active Index, even proprietary LLMs still struggle with internal citation.
ASQA
Eli5
SciQAG
RepliQA C-Pr
Model
Method
Acc
C-Pr
C-Re
Acc
C-Pr
C-Re
Ace
C-Pr
C-Re
Acc
C-Re
InsOnly
19.1
20.0
21.2
11.5
5.9
6.4
65.9
0.6
0.8
24.2
0.9
1.3
PassIdx
21.5
24.1
24.2
14.5
8.9
9.0
65.7
2.4
2.4
24.8
2.4
2.5
Repeat
22.5
20.5
20.7
14.5
11.2
11.4
62.4
2.5
2.6
27.1
2.5
2.6
Repeat+
19.8
22.0
22.3
14.3
11.2
11.3
65.8
5.2
5.2
25.8
3.6
4.0
Q-7B
ActIdx-F
25.8
26.7
27.9
14.6
18.6
18.7
65.6
23.6
23.6
30.3
12.6
13.3
ActIdx-B
25.4
31.4
31.9
17.1
28.0
28.3
66.5
30.8
32.0
29.1
21.6
22.7
ActIdx
27.6
30.9
31.1
17.6
29.3
29.5
66.6
32.6
33.6
31.9
24.4
25.7
InsOnly
15.9
3.7
4.1
9.2
0.6
0.6
61.2
0.0
0.0
15.2
0.2
0.2
PassIdx
16.5
17.1
17.4
11.7
7.1
7.2
67.0
1.1
1.6
22.8
1.8
2.4
Repeat
18.6
16.4
16.5
12.2
9.1
9.3
64.0
0.9
23.1
1.9
2.2
Q-3B
Repeat+
16.5
17.1
17.4
10.1
9.4
9.8
65.7
1.1
1.3
23.5
2.0
2.6
ActIdx-F
19.8
22.6
23.1
12.5
12.8
13.3
67.7
3.0
3.0
24.7
3.9
4.7
ActIdx-B B
19.7
24.5
24.9
14.1
19.0
19.6
65.2
17.6
23.4
24.4
7.8
14.1
ActIdx
21.4
23.9
24.2
15.8
19.7
19.8
67.9
20.0
23.0
24.5
10.5
15.8
GPT-4.1
few-shot
52.7
23.0
24.0
29.6
0.0
0.0
93.0
0.0
0.0
and citation precision. All methods follow two-stage process: continual pretraining followed by instruction tuning. They differ only in their approach to continual pretraining. The Instruction-on -nll method skips continual pretraining and proceeds directly instruction tuning. We compare:
Instruction-only: Serves as a baseline: the model is instruction-turs without continual pretraining, testing citation based solely pre-trained knowledge.
Passive Indexing: During continual pretraining, document identifiers are appended to 768-token chunks, allowing the model to passively associate facts with document IDs.
Repetition: Following Khalifa et al. (2024), document IDs are appended after each fact to support fine-grained attribution, though frequent insertions may reduce fluency.
Repetition+: Continual pretraining appends document identifiers to full documents and sampled segments (e.g.. one-third, paragraphs, sentences) balance attribution and fluency.
Active Indexing (Forward): Intra-document augmentation is used during continual pretraining to reinforce source-to-fact mappings. enhancing grounding within individual documents. This method uses 1.28B augmented tokens (3.3x the original corpus).
Active Indexing (Backward): Continual pretraining uses cross- cross-document augmentation teach fact-to-source mappings, enabling information synthesis and adding 1.47B augmented tokens (3.8x). Active Indexing: Combines forward and backward augmentation, resulting in augmented 2.75B tokens (7.05x original 390M tokens).
While continual pretraining introduces a con-timee overhead, our approach adds no inference time cost. In contrast, the RAG-based method Gao al. (2023b) incurs recurring retrieval and conditioning overhead, requiring about 130x more input tokens per query. This trade-off increasingly attractive in today's ' dimited regime: high-quality human data is scarce and pure compute scaling shows diminishing returns Villalobos et al. (2024); Muennighoff et al. (2023). Synthetic data is therefore practical path new capabilities Qin et al. (2025), making one-time training investment for scalable citation well justified. See training details in Appendix DJ We also evaluate GPT-4.1 with 3-shot citation prompting measure a proprietary LLM internal citation (note: its decoding space cannot be constrained).
5.2
MAIN RESULTS
Passive Indexing Is Insufficient for Citation. Table I shows that Passive Indexing-simen simply appending document IDs and expecting implicit learning is insufficient. While it offers modest gains over instruction tuning alone (e.g., 2.4 vs. 0.9 on RepliQA), citation precision remains low. Attaching doc-IDs IDs closer to facts (Repeat) also fails to improve performance realistic tasks, unlike

---

Published as a conference paper at ICLR 2026
prior work (Khalifa et al. 2024). due their use of synthetic, rigidly structured data. In settings, diverse and loosely aligned facts limit the effectiveness of sentence-level associations. Nonetheless, Repeat+ performs slightly better, likely due to its more granular ID attachment across different parts of the document, which enhances fluency and supports better generalization.
Active Indexing is Effective in Both Directions. Active Indexing consistently outperforms passive baselines across all tasks (Table J. Backward indexing is stronger than forward, and combining both delivers the largest gains (e.g., citation precision: 2.4->32.6) underscoring their complementarity. We expect further scaling of the augmentation to yield additional improvements ($5.3). and we verify the additive benefits of using both directions in Appendix B.2 Notably, Active Indexing also boosts answer correctness, likely due to exposure to factual content diverse formats (Allen-Zhu & Li. 2025; Yang et al., 2025). Model Size Matters. On Qwen-2.5-3B, we see the same trends as 7B but with much lower citation performance. Scaling model size (Qwen-2.5: 3B+7B-+ 14B; Llama-3: 3B+BB) yields monotonic gains under Active Indexing (Appendix BB. suggesting larger models generally support stronger citation when trained for it. Yet model size alone is insufficient: without targeted training (e.g., Active Indexing), even GPT-4.1 struggles with internal citation. Beyond Wikipedia, sources often either lack semantically guessable titles (e.g., arXiv) or lack stable document identifiers altogether (e.g., Common Crawl for EL15), making internal citation difficult without an explicit indexing objective.
Pretrained Models Memorize Wikipedia Titles. On ASQA (Wikipedia-based task), mod- els achieve competitive citation performance even without indexing (e.g., 20.0 with instruction-tuning only vs. 24.1 with passive indexing). We attribute this to two main factors: (1) large models memorize not just verbatim text but also Wikipedia titles during pretraining (Weller et al. 2024 Zhang et al. 2025b), and (2) Wikipedia titles are typically entities, making it easier for models to "shortcut" by predicting the entity as the title. Preliminary results on other Wikipedia-basee tasks, such TriviaQA and HotpotQA, show similar patterns. See Appendix [F2 for details. See Appendix for qualitative analysis and Appendix B.4 for OOD evaluation analysis for Active Indexing.
5.3
WHY DOES ACTIVE INDEXING WORK?
Facts Variation and Active Supervision Are Both Crucial for Re- liable Citations To identify what drives reliable internal citation, we study how fact variation (i.e., presenting the same facts in differ- ent ways) and active supervision affect model performance. Using the RepliQA dataset, we scale the amount of augmented data and evaluate how citation performance varies with the augmentation ratio (Figure 2)
Citation Precttion
We compare three approaches: (1) Passive Indexing with Replay, which simply repeats the same facts and document identifiers with more epochs; (2) Passive Indexing with Synthetic Continual Pretraining (Yang et al. 2025) (PI-SCP), which paraphrases facts and introduces relational variants, but still lacks explicit QA-style training to use document IDs. and (3) Active Indexing, which not only diversifies fact formulations but also explicitly trains the model to use document identifiers in QA-style contexts. Our findings are threefold: 1. Fact Variation Helps: Citation performance improves as Active Indexing increases both the scale and diversity of fact pre- sentations. In contrast, simple token replay provides no benefit-aa and can even degrade performance due to overfitting. 2. Active Supervision is Crucial: Although PI-SCP introduces diverse phrasings of the same facts, it still lags behind active indexing. This indicates that diversity alone is insuffi- cient-explicitly teaching the model to use document identifiers context is essential for robust citation. 3. Room to grow: Citation performance continues to improve at the highest augmentation levels tested (16x), suggesting that further gains are possible beyond our current limits.
20.0
A1
71 SCP
Pt
16
20
Augmantation Ratio
Figure 2: Scaling Comparison Between Active Indexing and Passive Indexing on RepliQA.
Active Indexing Learns Beyond Semantic Shortcuts A key concern is that Active Indexing might rely on semantic shortcuts from titles (i.e., "guessing the title") instead of learning true fact-II associations. We test this shorteut hypothesis on the RepliQA-only setup by measuring how semantically distinctive each title is. For each RepliQA statement (query + answer), we compute its semantic similarity (sentence embeddings) to all document titles and rank the true title among
8

---

Published as a conference paper at ICLR 2026
them. We treat this rank as Title Distinctivvness: Easy true title is highly similar to the statement (semantic shortcuts possible); Hard = many other titles are more similar (semantic cues unreliable). We bucket examples into four bins Very Hard) and report citation precision and average rank in T Findings are: Titles are not semantically unique. In 790% of cases, at least one other title is more similar to the statement than the true title; average, the correct title ranks 208th out of 6,822. Simple "nearest-title" guessing is rarely sufficient. 2. Semantic cues help when available. C-Pre is higher in the Easy bin and decreases as titles become less distinctive, showing the model sensibly uses semantic cues when they exist. 3. Non-trivial performance without shortcuts. Even in the Hard/Very Hard bins, where semantic similarity a poor signal, the model still achieves ~40% C-Pre. This indicates Active Indexing learns genuine fact- associations, not just shortcuts.
Table 2: Citation precision vs. title distinctiveness on RepliQA. Active Indexing benefits from distinctive titles but maintains substantial accuracy even when titles are not semantically similar, indicating learning beyond semantic shortcuts.
Easy
Medium
Hard
Very Hard
Total
C-Pre
55.9
49.6
40.1
40.0
46.7
Avg. Rank
10
761
208
Bridging Memorization and Generalization in Citation Past work has shown that stronger memorization does not guarantee better generalization (Wang et al.) 2025b). We observe a similar pattern in citation tasks. There is crucial difference between how LLMs (1) memorize document identifiers during continual pretraining, and (2) generalize to use these identifiers when answering downstream questions. To probe this, we evaluate four setups that progressively shift from pure memorization to downstream usage: 1. FullDoc: Predicting the doc-ID given the full document. PartialDoc: Predicting the doc-ID from partial document segment. 3. GoldQA: Predicting the doc-ID given the question and gold answer. 4. ModelQA: Predicting the doc-ID given the question and the model's s generated answer. We reuse RepliQA 7B models and report Hit@ 1/10 accuracy. As shown in Table all methods' performance declines tasks shift from memorization to generalization in downstream citations. Notably, more replay epochs (PassIdx-REP improve memorization (FullDoc: 27.0 + 74.6) but hurt downstream generalization (ModelQA: 7.8 6.0), suggesting overfitting to shallow patterns. contrast, Active Indexing bridges this gap by promoting both memorization and effective usage in QA, encouraging robust fact-to-soures grounding.
Table Document ID Memorization and Generalization.
FullDoc
PartialDoc
GoldQA
ModelQA
Acc@1
Acc@ 10
Acc@
Acc 10
Acc@1 1
Acc@ 10
Acc@1 I
Acc@
PassIdx
80.4
5.8
31.8
8.6
21.4
7.8
19.6
PassIdx-REP
74.6
94.4
10.6
32.8
6.6
25.6
6.0
22.4
ActIdx
95.2
100.0
72.8
97.4
66.4
94.2
54.2
88.4
5.4 INTERNAL VS. EXTERNAL CITATIONS
We compare internal and external citation methods to understand their respective limitations and the potential for synergy. External citations allow models access information beyond their memorized knowledge, while internal citations serve as a robust fallback when retrieval fails. This trade-off becomes especially important when retrieval quality varies. explore this, we compare: Internal Only: Generate internal citations without retrieval using Active Index.
External Only: Generate citations via RAG with 3-shot examples and top-5 retrieved documents, following the best-performing setup in Gao et al. (2023b).
Hybrid Joint: Instruction-tuniin Activeldx to consume retrieved documents, generating both internal and external citations after initial assessment of document sufficiency during inference. Hybrid Fallback: A pipeline that first attempts RAG, falling back to Hybrid Joint if retrieved documents are deemed insufficient by the model.

---

Published as a conference paper at ICLR 2026
ASDA Accuracy
ScIOAG Accuracy
Elis Accuracy
RepliOA Accuracy
30
50
20
022
64
066
0.6
1.0
0.2
04
06
0.9
100
5.
02
04
0/6
0.0
02
04
0.6
Rettrival Quality
Retrioval Qualty
Pattr eval Quality
Fatrrivall Quality
ASQA CitationF]
ScIOAG CrationF1
E01
Citation1
RealiQA Citation
50
30
16
64
0.4
0.4
04
Retriinal Quality
Retritval Quality
Rettr mont Quality
Retrinval Quality
Hybrid Joint
Hybrid Fallback
External Only
Only
Hybrid Oracle
Figure 3: Performance internal, external, and hybrid citations across retrieval quality (0=sparse retrieval, 1=dense retrieval). Internal only excels under poor retrieval, external only under strong retrieval, while hybrids generally perform best regardless of retrieval quality, with room to improve.
Hybrid Oracle: A conceptual upper-bound that selects the better output from the Internal and External Only methods for each example. See more experiments details in Appendix D.2
We test these strategies across a spectrum of retrieval quality, which we simulate by interpolating between sparse retrieval (BM25; lower-quality in our setup) and dense retrieval (Lin et al. 2021] high-quality). Retrieval quality = uses BM25's 's top-5; quality I uses dense top-5; intermediate values (e.g.. 0.2) mix the two proportionally (20% sampled from dense, 80% BM25). As shown in Figure 3] when retrieval is poor, methods incorporating internal citations (Internal Only and Hybrid variants) significantly outperform External Only, highlighting the importance of internal fallback under noisy retrieval. As retrieval improves, external outperforms internal, showing that reliable external evidence can complement internal knowledge. Hybrid Approach, taking advantages of both, perform generally best regardless of retrieval quality. Notably, performance gap remains between our best Hy brid method and the Hybrid Oracle, pointing to headroom for more effective integration strategies that better reconcile retrieved and memorized knowledge. We also provide finer-grained analysis across conflict slices, partitioning examples by whether the internal-only and external-only systems are correct (Appendix B.3).
6 CONCLUSION
We show that large language models can attribute answers to their pretraining data without relying on test-time retrieval. We (1) introduce CiterretrainBench for internal citation across short- and long- form QA, and (2) propose Active Indexing, continual pretraining strategy that teaches models to link content with document identifiers. Key findings include: 1. Teaching beats hoping: Active Indexing, which frames citation as real tasks, improves precision and recall by up to 32 points- outporfornnpr passive approaches. 2. Complementary directions: Forward and backward augmentations are most effective when combined. Scale helps: Performance improves consistently with more augmented data. 4. Internal and external are complementary: Combining both enhances robustness to retrieval quality. See Appendix ! for limitations and future directions.
ACKNOWLEDGMENTS
Yukun Huang, Sanxing Chen, and Bhuwan Dhingra's research is supported in part by the NSF award IIS-22 1526. Jian Pei's research is supported in part by the NSF RMS-2434666 All opinions, find- ings, conclusions and recommendations in this paper are those of the authors and do not necessarily reflect the views of the funding agencies.
10

---

Published as a conference paper at ICLR 2026
REPRODUCIBILITY
We are committed to ensuring the reproducibility of our results. We describe details of our training procedures ($4) and implementations ($5.1) Appendix (D). For our evaluation, we describe details of
datasets ($3.7 metrics 3..3 $3.4) and processing steps (Appendix Ap. Code and data are released github
REFERENCES
Ayush Agrawal, Mirac Suzgun, Lester Mackey, and Adam Kalai. Do language models know when they're re hallucinating references? In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguistics: EACL 2024, pp. 912-928, St. Julian's, ss Malta, 2024. Association for Computational Linguistics. URL https: hti.psss/aclaannooooy.ooooo
findings-eacl- 62
Zeyuan Allen-Zhu and Yuanzhi Li. Physics of language models: Part 3.2, knowledge manipulation. In The Thirteenth International Conference Learning Representations, 2025. URL https
//openreview
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-RAG: Learning to retrieve, generate, and critique through self-reflection. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https: /atppre////eeeeeeew. net /tr//forro forum?id- hSyW5go0v8
Arian Askari, Chuan Meng, Mohammad Aliannejadi, Zhaochun Ren, Evangelos Kanoulas, and Suzan Verberne. Generative retrieval with few-shot indexing. ArXiv, abs/2408.00155. 2024. URL
httpp: stw.appissematicccclollla.a.a... http::/ss...aaaaaaaa..a.ooooorrr http://a.....ee.eppss11111111111
Mchul Bhattacharyya. Valerie M Miller, Debjani Bhattacharyya, and Larry E Miller. High rates medical content. Cureus, 15(5): ncbi, nlm. nih
of fabricated and inaccurate references in
e39238, May 2023. doi: 397.933. gov/373374807
Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al. Extracting training data from large language models. In 30th USENIX security symposium (USENIX Security 21), pp. 2633-26500 2021.
Tyler A. Chang, Dheeraj Rajagopal, Tolga Bolukbasi, Lucas Dixon, and Ian Tenney. Scalable influence and fact tracing for large language model pretraining. In The Thirteenth International Conference on Learning Representations, 2025. URL haatts:://operrvvieet net/ forum?
Ld=gLa 96F 2FFFF:W 1Wwn
Mikaël Chelli, Jules Descamps, Vincent Lavoué, Christophe Trojani, Michel Azar, Marcel Deckert, Jean-Luc Raynier, Gilles Clowez, Pascal Boileau, and Caroline Boetsch-Chelll Hallucination rates and reference accuracy of CharGPT and Bard for systematic reviews: Comparative analysis. Journal of Medical Internet Research, 26:e53164, May 2024. doi: 10.2196/53166 URL https
7/pubmed. ncbi. nlm. nih. gov 00w038////310000
Sanxing Chen, Yukun Huang, and Bhuwan Dhingra. Real-time factuality assessment from adver- sarial feedback. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pp. 1610-1630, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979 977-97799---2-5--- doi: 1/2255 alcloo/ URL
2025 acl long. 81/
Tong Chen, Akari Asai, Niloofar Mireshghallah, Sewon Min, James Grimmelmann, Yejin Choi, Hanna Hajishirzi, Luke S. Zettlemoyer, and Pang Wei Koh. Copybench: Measuring literal and non-literal reproduction of copyright- text in language model generation. ArXiv preprint, abs/2407.0708f 2024. httphhtss://arxtt wwwp://gg///b//a/.e
07087

---

Published as a conference paper at ICLR 2026
Yung-Sung Chuang, Benjamin Cohen- Cohen-Wang, Shannon Zejiang Shen, Zhaofeng Wu, Hu Xu, Xi Vic- toria Lin, James Glass, Shang- Wen Li, and Wen-tau Yih. Selfcite: Self-supervised aligmment for context attribution in large language models. ArXiv preprint, abs/2502.09604, 2025. URL
htt.ts:///rrxxt. org/ abs / 2502. .02.5002206444
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. ELI5: Long form question answering. In Anna Korhonen, David Traum, and Lluís Marquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 3558-3567, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 10.18333/v/ P19-1346. URL http:////w/.caacocoooooooooo////
1346
Hanpei Fang. Sijie Tao, Nuo Chen, Kai-Xin Chang. and Tetsuya Sakai. Do large language models favor recent content? study recency bias in llm-based reranking. 2025. URL https
www.nt wetemmnniccccooaaa. httpa//cccc.......ooooooooooaaa.
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 16477-16508, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10. 10.1180 1/586/11/2023..... aat.222.al-ooog9910 URL org/ 2023. aww.-llllllll... 9107]
Setps://aclantDonn
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 6465-6488, Singapore, 2023b. Association for Computational Linguistics. doi: https://aclanthoooog org/ 2023. 3, emnlp ma inaainn. 398
URL
Tianyu Gao, Alexander Wetrig, Luxi He, Yihe Dong, Sadhika Malladi, and Danqi Chen. Metadata conditioning accelerates language model pre-training In Forty-second International Conference on Machine Learning, 2025. URL https:t/////cm... btttps://openeeiiieeee
net forum ?id -aaiidd-DDMMMMM DdMM2
Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor. Trahrracher: Learning factual consistency evaluation with large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 2053-2070, Singapore, 2023. Association for Computational Linguistics. doi: 10. 18.18653/v 2023. emnlp-mainmn.. 127
00118663/112222333 1/2023 wtw.0/2..mmppmmii.... 127. URL https: http/aaaaaaa http////aaaaanotoao//oao org/
Hangfeng He, Hongming Zhang, and Dan Roth. Rethinking with retrieval: Faithful large language model inference. ArXiv preprint, abs/2301.00303 2023. URL httpps://aaxvv.or//////////////e /bs// 23C1.00303
Chengyu Huang, Zeqiu Wu, Yushi Hu, and Wenya Wang. Training language models to generate text with citations via fine-grained rewards. ArXiv preprint, abs/2402.04315,, 2024a. URL
aattps://arxin org/ als 2402. 04315
Yue Huang, Lichao Sun, Haoran Wang, Siyuan Wu, Qihui Zhang, Yuan Li, Chujie Gao, Yixin Huang, Wenhan Lyu, Yixuan Zhang, Xiner Li, Hanchi Sun, Zhengliang Liu, Yixin Liu, Yijue Wang, Zhikun Zhang, Bertie Vidgen, Bhavya Kailkhura, Caiming Xiong, Chaowei Xiao, Chunyuan Li, Eric P. Xing. Furong Huang, Hao Liu, Heng Ji, Hongyi Wang, Huan Zhang, Huaxiu Yao, Manolis Kellis, Marinka Zitnik, Meng Jiang, Mohit Bansal, James Zou, Jian Pei, Jian Liu, Jianfeng Gao, Jiawei Han, Jieyu Zhao, Jiliang Tang, Jindong Wang. Joaquin Vanschoren, John C. Mitchell, Kai Shu, Kaidi Xu, Kai-We- Chang, Lifang He, Lifu Huang, Michael Backes, Neil Zhenqiang Gong, Philip S. Yu, Pin- Yu Chen, Quanquan Gu, Ran Xu, Rex Ying, Shuiwang Ji, Suman Jana, Tianlong Chen, Tianming Liu, Tianyi Zhou, William Wang, Xiang Li, Xiangliang Zhang, Xiao Wang, Xing Xie, Xun Chen, Xuyu Wang. Yan Liu, Yanfang Ye, Yinzhi Cao, Yong Chen, and Yue Zhao. Position: Trustllm: Trustworthiness in large language models. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, OpenReview.net, 2024b. URL _ hnttpss///peereeet
net. forum? wotrrmmid=bbuuUt ddd
12

---

Published as a conference paper at ICLR 2026
Yukun Huang, Sanxing Chen, Hongyi Cai, and Bhuwan Dhingra. To trust or not to trust? enhancing large language models' situated faithfulness to external contexts. In The Thirteenth International Conference Learning Representations, 2025. hattps::/openreiieee
id=K2
Ehsan Kamalloo, Aref Jafari, Xinyu Zhang, Nandan Thakur, and Jimmy Lin. Hagrid: A human- Ilm collaborative dataset for generative information infornatioo--cekinng with attribution. ArXiv preprint, abs/2307.16883, 2023. URI.ht:ts://arxiii
httpaabb//33... 22377.6887
Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, and Hao Peng. Source-aware training enables knowledge attribution in language models. In First Conference on Language Modeling, 2024. URL httpt:://ppeeeeeiiiite:/ net/ forum?id
UPyWIwciYz
Xiaoxi Li, Yujia Zhou, and Zhicheng Dou. Unigen: A unified generative framework for retrieval and question answering with large language models. ArXiv, abs/2312.110366 2023. /RLhttps
//api. semant i icscholar wwt/Corpo//oooooooooooo 26339266666625
Xiaoxi Li, Zhicheng Dou, Yujia Zhou, and Fangchao Liu. Corpusim: Towards a unified language model on corpus for knowledge-intensing tasks. In Grace Hui Yang. Hongning Wang, Sam Han, Claudia Hauff, Guido Zuccon, and Yi Zhang (eds.), Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2024, Washington DC, USA, July 14-18, 2024, pp. 26 37. ACM, 2024a. doi: 10.1145336267223667777 org
URL https:://// /ttp:::///oo..or///.11/0/ 015.:.0..00011111115353322222222 36577777777777778888
Xiaoxi Li, Jiajie Jin, Yujia Zhou, Yuyao Zhang, Peitian Zhang, Yutao Zhu, and Zhicheng Dou. From matching to generation: A survey on generative information retrieval. ACM Transactions on Information Systems, 2024b. URL httt::://ad...smmtiiccchhoo
semanticscholat wogg/oorpsssoDDo
269303210
Huawei Lin, Jikai Long, Zhaozhuo Xu, and Weijie Zhao. Token-wise influential training data retrieval for large language models. In Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 841-860, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 1116653/112222..al-loo acl-long.493 487
Sheng-Chiell Lin, Jheng-Hong Yang, and Jimmy Lin. In-batch negatives for knowledge distillation with tightly-coupled teachers for dense retrieval. In Anna Rogers, lacer Calixto, Ivan Vulić, Naomi Saphra, Nora Kassner, Oana-Maria Camburu, Trapit Bansal, and Vered Shwartz (eds.), Proceedings of the 6th Workshop on Representation Learning for NLP (RepLANLP- pp. 163-173, Online, 2021. Association for Computational Linguistics. doi: 10. 10.18653/w1/2021.rrg wtg. rep14r1p-1.11
1.17. URL
Jacheng Liu, Taylor Blanton, Yanai Elazar, Sewon Min, Yen-Sung Chen, Arnavi Chheda-Kothary, Huy Tran, Byron Bischoff, Eric Marsh, Michael Schmitz, Cassidy Trier, Aaron Sarnat, Jenna James, Jon Borchardt, Bailey Kuehl, Evie Yu-Yen Tu-Yen Cheng, Karen Farley, Taira Anderson, David Albright, Carissa Schoenick, Luca Soldaini, Dirk Groeneveld, Rock Yuren Pang, Pang Wei Koh, Noah A. Smith, Sophie Lebrecht, Yejin Choi, Hannaneh Hajishirzi, Ali Farhadi, and Jesse Dodge. OLMoTrrce: Tracing language model outputs back to trillions of training tokens. In Pushkar Mishra, Smaranda Muresan, and Tao Yu (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations), pp. 178-188, Vienna, Austria, July 2025. Association for Computational Linguistics. ISBN 979-8-89176- 253-4. doi: 10. 10.18653/v12205. acI- 18. URL httptt/s/aaaellattooooo..oos//// acl-denc demo. 187]
Nelson Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7001-7025, Singapore, 2023. Association for Computational Linguistics. doi: 10. 10.18653/v11200 1t033v1122222 .wn.o22....niign--ee finding--emmlp..4...... org/2023. f findiinii ngs- 464
emmlpp.47..... URL https: // http//aaantttt /accanntollggy
13

---

Published as a conference paper at ICLR 2026
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Perey Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024. doi: 10.1162/tacl 00638. URL
https: / antho www.y.oogg/222..ee5m 91.1
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. Ex- AertQA: Expert-curated questions and attributed answers. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume Long Papers), pp. 3025-3045 Mexico City, Mexico, June 2024, Association for Computational Linguis- /2t0:// 167. URL
tics. doi: 10. naac1-long. 1677
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 9802-9822, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.118663/11/2223.alleeee 546/
URL htt.:t///ac..aaboooooo.oo/////// acl-lon
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese Susamnah Young, Lucy Gampbell-Gillinghanl Geoffrey Irving, et al. Teaching language models support answers with verified quotes. ArXiv preprint, 115/2203.1114,,200 URL httphhttts//iiii wrw. 3t0///b///22211111
João Monteiro, Pierre-André Noël, Étienne Marcotte, Sai Rajeswar Mudumba, Valentina Zantedeschi, David Vázquez, Nicolas Chapados, Chris Pal, and Perouz Taslakian. Repliqa: A question- answering dataset for benchmarking Ilms on unseen reference content. In Amir Globersons, Lester Mackey, Danielle Belgrave, Angela Fan, Urrich Paquet, Jakub M. Tomezak, and Cheng Zhang (eds.), Advances in Neural Information Processing Systems 38: Annual Conference on Neural Information Processing Systems 2024, NeurlPS 2024, Vancouver, BC, Canada, December 10 15, 2024, 2024. (RL.ttp::0 httt:::/pppprrse ni cc wttp//ppppffeeeeeeeeeeeeeeeeeeee wetrr_fflss_pprprereeeeeee 2ee444/hahhh222226e6 31113322222111111113333300031313 and Benchmarks Track. html
Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. Scaling data-constrained language models. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS 223 Red Hook, NY, USA, 2023. Curran Associates Inc.
Reiichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe. Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: answering with human feedback. arXiv preprint arXiv: 2112.09332.
Browser- assisted http://axxvvv..o org/ abs/ / 2212. 09332
2021. URL wtt.httss////
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. TRAK: attributing model behavior at scale. In Andreas Krause, Emma Brunskill, Kyunghyun Cho, Barbara Engelhardt, Sivan Sabato, and Jonathan Scarlett (eds.), International Conference on Machine Learning, ICML. 2023, 23-29 July 2023, Honolulu, Hawaii, USA, volume 202 of Proceedings of Machine Learning Research, pp. 27074-27113, PMLR, 2023. URL pesttps:/pproeeee ings mlr. wwtpp/ewwwee.......eeeeee
Fabio Petroni, Patrick Lewis, Aleksandra Piktus, Tim Rocktäschel, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. How context affects language models' factual predictions. ArXiv preprint, abs/2005.046.1 2020. URL httll::tt://// //tp:///////vv//// org/ abs abs/2005..... 2005. 04611
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thoree, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur: Iz Beltagy, Steven
14

---

Published as a conference paper at ICLR 2026
Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2523 Online, 2021. Association for Computational Linguistics. doi: 10. 18.13653/v1 1/2021 maw.l.l.aa-mmmin2200 URL https::: walaacaanhloogyy..o org/
2021 naac mmacc--man... 200
Aleksandra Piktus, Fabio Petroni, Yizhong Wang, Vladimir Karpukhin, Dmytro Okhonko. Samuel Broscheit, Gautier Izacard, Patrick Lewis, Barlas Ouguz. Edouard Grave, Wen tau Yih, and Sebastian Riedel. The web is your oyster- - knowledge-intensive nlp against a very large web corpus. abs / 122 09924
ArXiv preprint, abs/21 12.09924, 2021. URL http:::ts////aaitee
Zeyu Qin, Qingxiu Dong, Xingxing Zhang, Li Dong, Xiaolong Huang, Ziyi Yang. Mahmoud Khademi, Dongdong Zhang, Hany Hassan Awadalla, Yi R. Fung, Weizhu Chen, Minhao Cheng, and Furu Wei. Scaling laws of synthetic data for language models, 2025.
BRLhttt:: //axxiv
org/abs/250x 9551
Pritika Ramu, Koustava Goswami, Apoory Saxena, and Balaji Vasan Srinivasan. Enhancing post-hoc attributions in long document comprehension via coarse grained answer decomposition. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17790-17806, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.1 1188863/122224..m/// 985. URL htt.tpss///aaaannollllooooo
org 2024. . emn 9857
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. Mcasuring attribution in natural language generation models. Computational Linguistics, 49(4))777-840, 2023. doi: 10.1 166/coll a_ 1_00486.
org 2023 c1..2
Ruiyang Ren, Wayne Xin Zhao, Jing Liu, Hua Wu, Ji-Rong Wen, and Haifeng Wang. TOME: A two-stage approach for model-based retrieval. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.). Proceedings of the 61st Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 6102-6114, Toronto, Canada, July 2023. httpss
Association for Computational Linguistics. doi: 10.18665/w122223aacloooom
//aclanthologg oogy. wtt./////../cooooo wwl.0223.ccll1.. -ong. 3367
Abhilasha Sancheti, Koustava Goswami, and Balaji Srinivasan. Post-hoc answer attribution for grounded and trustworthy long document comprehension: Task, insights, and challenges. In Danushka Bollegala and Vered Shwartz (eds.). Proceedings of the 13th Joint Conference on Lexical and Computational Semantics ($SEM 2024), pp. 49-57, Mexico City, Mexico, June URL
2024. Association for Computational Linguisties. doi: 10.118653w112004sssrrrre
wwwp:org/222 wttp://02..tttmmmm1mmmm starsem-r....... 1.47
Tal Schuster, Adam Lelkes, Haitian Sun, Jai Gupta, Jonathan Berant, William Cohen, and Donald Metzler. SEMQA: Semi-extractive multi- source question answering. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North Americun Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume I: Long Papers). pp. 1363-1381, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10. 18.11.53/v 1/0.l///llomm.mmlooooooggg URL ht https: http:////acaatttooooocoo / 2024. www.nnaalllllm long. 74/]
Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, and Ido Dagan. Attribute first, then generate: Lotaily-attibbutable grounded text generation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Proceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 1: Long Papers), pp. 3309-3344, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 10.18653/w1/2024..o acl-oooo....ooo.n...... 182. URL https://aclante http:/hooooooroooo/2222......... 200444aal-looo
org. 2024
1627
Mingyang Song and Mao Zheng. A survey of query optimizationn in large language models. arXiv preprint arX1v:2412.17558, 2024. URL httpt::p:/// /tte:://aaavvv.. org/abs 2412.1555
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming- Wei Chang. ASQA: Factoid questions meet lorg-form answers. In Yoav Goldberg, Zomitsa Kozareva, and Yue Zhang (eds.), Proceedings of
15

---

Published as a conference paper at ICLR 2026
the 2022 Conference Empirical Methods in Natural Language Processing, pp- 8273 8278,2288.44 Abu Dhabi, United Arab Emirates, 2022. Association for Computational Linguistics. doi: 10. v8 2022.emnlp m6in.566. / wttp:/rrr..222....mmmmmmmm.mmmm. 566
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou, Recitation-auugmented language models. The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023. URL /ttps////////eeeeeeeeeeoo pdf? lidf-id--aa cww.vb--vvvbvvee NKI
Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Prakash Gupta, Tal Schuster, William W. Cohen, and Donald Met- zler. Transformer memory as a differentiable search index. In Sanmi Koyejo, S. Mo- hamed, A. Agarwal, Danielle Belgrave, K. Cho, and A. Oh (eds.), Advances in Neural Information Processing Systems 35: Anmual Conference on Neural Information Process- ing Systems 2022, NeurIPS 2022, New Orleans, LA, USA, November 28 December 9, 2022, 2022. URL http: /ttp::/pappeeee nips tt/ wwtpp//sppppppppppppppeeepeeeeep wttp:p/rrrreeeeeeee///ee// hash// 8 2122406612255cc33bbbbeeeeeeeeee www.:ofaaaaaaatttttaaaaaaaa
html
Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. Position: will we run out data? limits of Ilm scaling based on human-generated data. In Proceedings of the 41st International Conference Machine Learning, ICML'24. JMLR.org, 2024.
Tu Vu, Mohit lyyer, Xuezhi Wang. Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung. Denny Zhou, Quoe Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation. Lun-Wer Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 13697-13720. Bangkok, Thailand, 2024. Association for Computational Linguistics. doi: 10. 10..1865//v1/// 1/2024. hwn.:/....../....n--eieaea aeciings-aal.813 URL kttps://aclaanthoogg
findingfiinnnngnee acl 813/|
William H. Walters and Esther Isabelle Wilder. Fabrication and errors in the bibliographic citations 00---80-0---------- 000-5998-233441333
generated by ChatGPT. Scientific Reports, 13:14045, 2023, doi: 10. URL https:
Yuwei Wan, Yixuan Liu, Aswathy Ajith, Clara Grazian, Bram Hoex, Wenjie Zhang, Chunyu Kit, Tong Xie, and lan Foster. Sciqag: A framework for auto-generated science question answering dataset with fine-grained evaluation. ArXiv preprint, 0e//2240500999, 2024. URL https:
ftaxxiv.rgg/ab// /40.. 09939
Cunxiang Wang, Xiaoze Liu, Yuanhao Yue, Xiangru Tang, Tianhang Zhang, Cheng Jiayang, Yunzhi Yao, Wenyang Gao, Xuming Hu, Zehan Qi, et al. Survey on factuality in large language models: Knowledge, retrieval and domain-speciffiitt. ArXiv preprint, abs/2310.07521, 2023. URL https
7/arxiv. org/ httgg/bbs/231000 2310. 07521
Fei Wang, Xingchen Wan, Ruoxi Sun, Jiefeng Chen, and Sercan O Arik. Astute RAG: Overcoming imperfect retrieval augmentation and knowledge conflicts for large language models. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 30553-30571, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 979-8 199-8-88917-251--00 doi: 10.18653/w1/2022. acl- 1476. URL pttltttts///aaaatho..oo or
1476/
Xinyi Wang, Antonis Antoniades, Yanai Elazar, Alfonso Amayuelas, Alon Albalak, Kexun Zhang, and William Yang Wang. Generalization v.s. memorization: Tracing language models' capabilities back to pretraining data. In The Thirteemth International Conference on Learning Representations, 2025b. URL htt.tttts///peeeeieeeee
Ye Wang, Xinrun Xu, Rui Xie, Wenxin Hu, and Wei Ye. Generative retrieval with large language models. ArXiv preprint, abs/2402.110110, 2024. URL 17010
org/ abs 2402
16

---

Published as a conference paper at ICLR 2026
Yujing Wang, Yingyan Hou, Haonan Wang, Ziming Miao, Shibin Wu, Hao Sun, Qi Chen, Yuqing Xia, Chengmin Chi, Guoshuai Zhao, Zheng Liu, Xing Xie, Hao Sun, Weiwei Deng, Qi Zhang, and Mao Yang. A neural corpus indexer for document retrieval. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL htttps:://ppereeiiiiiie
net/ forum? widddddffffffffoo qc
Orion Weller, Mare Marone, Nathaniel Weir, Dawn Lawrie, Daniel Khashabi, and Benjamin Van Durme. "according to to... ": Prompting language models improves quoting from pre- training data. In Yvette Graham and Matthew Purver (eds.). Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 2288 2301, St. Julian's, s, Malta, 2024. Association for Computational Linguistics. URL https: / ////calannie htt///aaaanttooooooooooo
org/ 2024 ea24444acllllggo 1407
Guillaume Wenzek, Marie-Anne Lachaux, Alexis Conneau, Vishrav Chaudhary, Francisco Guzmán, Armand Joulin, and Edouard Grave CCNet: Extracting high quality monolingual datasets from web crawl data. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis (eds.), Proceedings of the Twelfth Language Resources and Evaluation Conference, pp. 4003 4012, Marseille, France, 2020. European Language Resources Association. ISBN
URL https
hogy. wtt.//20.0....ooooo 1rec- 1. .444
Yingtai Xiao, Yuqing Zhu, Sirat Samyoun, Wanrong Zhang, Jiachen T. Wang, and Jian Du Token- Shapley: Token level context attribution with shapley value. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Com- putational Linguistics: ACL 2025, pp. 3882-3894, Vienna, Austria, July 2025. Association for doi: 10. 10.18653/v1/222. http:///w///22....nnnnnn---/-s// findings-acl.200 2025. www.:eiiinningdsddeeeoeoe 2007
Computational Linguistics. ISBN 979-8--89999 8-1---88-97777 8-1---8-88776-----5
URL httttp:://aaaatttologgggoo////
Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, and Yu Su. Adaptive chameleon or stubborn sloth: Revealing the behavior of large language models in knowledge conflicts. In The Twelfth Inter- national Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. opw./evieeeeeeeeemoeeee httttsss///eeeeeeeeeeee net/ forum? d=auKAUJ2M06
OpenReview.net, 2024. URL https:
Zitong Yang, Neil Band, Shuangping Li, Emmanuel Candes, and Tatsunori Hashimoto. Synthetic continued pretraining. In The Thirteenth International Conference on Learning Representations, 2025.
Xi Ye, Ruoxi Sun, Sercan Arik, and Tomas Pfister. Effective large language model adaptation for improved grounding and citation generation. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume I: Long Papers). pp. 6237-6251, Mexico City, Mexico, 2024. Association for Computational Linguisties. URL httpst aaaclaanhoology.orrg/222222 //aclanthologe or.. 2024. naac -oocc-loooooo long.346
Xiang Yue, Boshi Wang, Ziru Chen, Kai Zhang, Yu Su, and Huan Sun. Automatic evaluation of attri- bution by large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 4615-4635, Singapore, 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1 1/203/.222222/m2 1/223//..122223.22n2233333
andings-emnll..ee htwp:/hoo.ooy.ooo/222...... wtw.:/oos....iiinngs-naaaooo ett.:aiiingeeee.....00..... 307
URL
Jijje Zhang, Yushi Bai, Xin Lv. Wanjun Gu, Danging Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. LongCite: Enabling LLMs to generate fine- -eeiggriiedd citations in long-context QA. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Findings of the Association for Computational Linguistics: ACL 2025, pp. 5098- 5122, Vienna, Austria, July 2025a. Association for Computational Linguistics. ISBN 8-977- 256-5. doi: 10. 10.13//211120 http:///.2.....n.s-.ooo ac1.264. URL findings-acl 2647
2.claattolooyy.orr/222222 org/2025
Jingyu Zhang, Mare Marone, Tianjian Li, Benjamin Van Durme, and Daniel Khashabi. Verifiable by design: Aligning language models to quote from pre-training data. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas
17

---

Published as a conference paper at ICLR 2026
Chapter the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3748-3768, Albuquerque, New Mexico, 2025b. Association for Computa- tional Linguisties. ISBN 191-899171---666 URL /.LLspss///aallacooooooyoyy.. maacl-long.. 1917
Guido Zuccon, Bevan Koopman, and Razia Shaik. Chatgpt hallucinates when attributing answers. In Proceedings of the Annual International ACM SIGIR Conference Research and Development in Information Retrieval in the Asia Pacific Region, pp. 46-51, 2023.
A DATASETS
A.1 CORPUS CONSTRUCTION
To construct the essential document set, we require full documents for continual petraining-unlinee the Sphere corpus (Piktus et al.) 2021), which consists of 100-word passages. We process raw 2019 Aug Common Crawl snapshots using CCNet (Wenzek al. 2020) to obtain clean full-text documents. To identify documents relevant to ELI5, we first index 00-dord passages (in Sphere format) and use BM25 to retrieve the top- 10p1100 passages for each question the ELI5 train, dev, and test sets. We then use an LLM to decompose each question into self-contained fact claims, treating each claim as a separate query to retrieve the top-200 relevant passages.
Common Crawl
Next, we verify whether the retrieved passages support each claim and compile support set. This set combined with the original relevant passages, and using passage metadata, we locate and extract the corresponding full documents from Common Crawl. This process yields core document set, comprising 30,025 documents with a total of 110,594,287 tokens, averaging 3,683 tokens per document. For training set instances with unsupported claims, use GPT-4o to generate synthetic documents styled like Common Crawl, ensuring the model not trained on unsupported inputs. This results in corpus from the Other source, containing 7,593 documents with 2,210,346 total tokens, averaging 291 tokens per document.
The data is licensed under CCO.
We build on the KILT knowledge source (Petroni et al. 2021), based on the 2019/08/01 Wikipedia snapshot. We include source documents from ASQA, the TriviaQA dev set, and subset of HotpotQA (medium-leee questions), mapped to KILT IDs. This forms our essential Wikipedia corpus, consisting of 30,025 documents totaling 1 110.6M tokens (avg. 3,683 tokens/document).
Wikipedia
The data licensed under CC BY-SA 3.0.
ArXiv SciQAG (Wan et al. 2024) provides scientific papers without titles. We retrieve titles using the papers' DOIs from arXiv, resulting in 22,743 documents with 114.0M tokens (avg. 5,013 tokens/document).
arXiv metadata is used under the CCO; license for each paper varies.
We use the first two splits of the RepliQA dataset (Monteiro et al. 2024) as the full corpus, yielding 7, 7,182 documents and 8.88M tokens (avg. 1,236 tokens/document) Due to frequent title duplication, we use an LLM to generate more descriptive titles.
RepliQA
This corpus totals approximately 392M tokens. The data is licensed under CC BY 4.0.
A.2 RENAME DUPLICATED TITLES
Due to frequent title duplication--espccillly strategy using an LLM. For each duplicated title, we iteratively rename the document until all titles unique. We also perform cross-source deduplication after renaming.
especially in Common Crawl and RepliQA- adopt a renaming
18

---

Published as a conference paper at ICLR 2026
Table Full results for INSONLY, PASSIDX, and ACTIDX across all five base models on four QA datasets. Acc answer correctness, C-Pr = citation precision, C-Re = citation recall.
ASQA
Eli5
SciQAG
RepliQA
Model
Method
Acc
C-Pr
C-Re
Acc
C-Pr
C-Re
Acc
C-Pr
C-Re
Acc
C-Pr
C-Re
InsOnly
15.9
3.7
4.1
9.2
0.6
0.6
61.2
0.0
0.0
15.2
0.2
0.2
Q-3B
PassIdx
16.5
17.1
17.4
11.7
7.1
7.2
67.0
1.1
1.6
22.8
1.8
2.4
ActIdx
21.4
23.9
24.2
15.8
19.7
19.8
67.9
20.0
23.0
24.5
10.5
15.8
InsOnly
19.1
20.0
21.2
11.5
5.9
6.4
65.9
0.6
0.8
24.2
0.9
1.3
Q-7B
PassIdx
21.5
14.5
8.9
9.0
65.7
2.4
2.4
24.8
2.4
2.5
Actldx
27.6 30.9
31.1
17.6
29.3
29.5
66.6
32.6
33.6
31.9
24.4
25.7
InsOnly
26.4
25.1
14.9
5.9
67.4
27.6
0.75
0.9
Q-14B
PassIdx
30.0
27.1
27.3
18.5
9.3
9.4
67.4
2.3
2.3
28.4
2.0
2.0
Actldx
31.6
40.9
41.3
19.3
33.2
33.6
70.2
34.8
35.1
37.0
30.8
31.5
InsOnly
18.0
11.9
18.1
9.4
3.6
3.9
61.8
0.2
0.2
19.0
0.27
0.4
L-3B
PassIdx
21.5
18.5
19.1
11.2
7.1
7.9
63.2
1.8
2.1
23.4
1.8
2.0
ActIdx
25.9
29.0
29.3
15.6
25.9
26.3
65.6
13.2
32.8
26.8
13.9
23.0
InsOnly
25.4
23.3
25.5
12.1
5.0
5.5
67.3
1.0
1.6
24.1
0.7
1.4
L-8B
PassIdx
28.6
27.8
28.1
9.5
65.1
25.6
1.9
1.9
ActIdx
28.2
31.7
31.8
17.9
30.7
30.8
67.7
39.3
40.3
35.7
36.2
36.9
A.3 DOWNSTREAM TASKS
For ELI5, the test set is sourced from (Gao al. 2023b), while the training and development sets derived from a newer version Any duplicates from the test set have been removed from the training and development sets. ELI5 licensed under BSD 3.
For ASQA, we map annotated source documents to the KILT knowledge base and filter out datapoints with unmatched sources, resulting in 863 test examples. ASQA is licensed under CC BY 4.0.
For SciQAG, due to noise the training set, we re-split the original test set into train, dev, and test splits based documents, ensuring title overlap. This yields 853, 95, and 300 documents, respectively. Evaluation is performed on 1,000 sampled questions from the 300 test documents. SciQAG released GitHub without explicit license, will not redistribute without author acknowledgment.
For RepliQA, similarly split train, dev, and test sets by documents. The includes 1,000 QA pairs sampled from the test documents. RepliQA is licensed under CC BY 4.0.
A.4 INSTRUCTION TUNING SET
We randomly sample 1,000 training examples from ASQA, ELI5, RepliQA, SciQAG, and HotpotQA (medium level), and 200 from each dev split to form a validation set for carly stopping during instruction tuning.
B ADDITIONAL RESULTS
B. MAIN RESULTS OF MORE MODELS
We also run active-in sndiee-ndexxin experiments Qwen-2.5-14B, Llama-3.2-3B, and Llama-3. 1-B. Performance increases monotonically as model capacity Malle--Qwen from 3B 7B - 14B and Llama from 3B - 8B-further confirming that citation ability improves with model size. As shown in Table Active Indexing consistently outperforms Passive Indexing across models and datasets. Also, the model size increases, the performance of Active Indexing improves monotonically.
19

---

Published as a conference paper at ICLR 2026
B.2 COMPLEMENTARITY OF FORWARD AND BACKWARD
We study scaled complementarity of forward and backward scale effect RepliQA. We augment the RepliQA corpus: 66M tokens from Active Indexing Forward (AIF): 76M from Backward (AIB),
and their combination (AI), totaling 16.5x the original size. From these, we subsample different proportions to construct scaling curve. Figure 4 shows citation precision RepliQA (y-axis) versus the fraction of augmented data used (x- (x-axis). Key findings include: Scaling helps: All variants AIF, AIB, and AI i--imrove with more augmented data, even up to 16.5x. 2. Complementary di- rections: AIF alone less effective, but combining AIF with AIB (i.c.. AI) consistently outperforms either alone. Each point the AI curve uses matched subsets of AIF and AIB, indicating additive and reinforcing effects. 3. The role of forward While backward provides most gains, but forward provides consistent additive effect, suitable for users willing trade-off compute for better performance. Moreover, It supports answering realistic queries like "Explain doc- ument X / paper Y" in closed-book manner, which requires the model internally condition on document identifiers.
Citation Preciaion
50.8
20.0
AIB
Sample Ratio
Figure 4: Scaling Curve of Combining Backward and For- ward on RepliQA
B.3 BELAVIIR OF HYBRID SYSTEMS UNDER INTERNAL-EXTERNAL CONFLICTS
Hybrid systems are most useful in the conflict regime, where the model's internal belief disagrees with the retrieved context. To make this behavior explicit, we analyze RepliQA by partitioning examples into three slices based on correctness of the internal-only and external-only systems: 1, Internal wrong External correct (Int=0, Ext=1), 2. Internal correct External wrong (Int=1, Ext=0), and 3. No conflict (both correct or both wrong).
Table 5 reports both (i) the proportion of each slice and (ii) the accuracy of Internal-only, External- only, Hybrid-Joint, Hybrid-Fallback, and Hyyrid--racce in each slice, under high-quality retrieval (retrieval quality 1.0) and noisy retrieval (0.2).
Under high-quality retrieval (1.0), the dominant conflict regime is precisely the "production reality" where retrieval corrects the model: the Int=0, Ext=1 slice accounts for 38.6% of examples, compared to only 8.3% for Int=1, Ext=0. In this regime, Hhybrid-Fallbaee behaves almost like an ideal "defer-to- retrieval" policy: it nearly matches External-only accuracy in the Int=0, Ext=1 slice (99.5 vs. 100.0), while Internal-only is always wrong by construction (0.0). Hybrid-Joint is more conservative: substantially improves over Internal-only (75.6 vs. 0.0) but falls short of External-only, reflecting its stronger reliance on internal knowledge even when retrieval is correct.
When retrieval quality degrades (0.2), the picture flips: the Int=1, Ext=O slice becomes more prevalent (22.0% vs. 9.9% for Int=0, Ext=1). In this setting. both hybrid methods shift their arbitration toward internal knowledge. For example, in the Int=1, Ext=0 slice, Hybrid-Joint attains 58.2 accuracy, substantially outperforming External-only (0.0) and improving over Internalonly's baseline, while Fabrid-Fallback also achieves strong gains relative to External-only. This demonstrates that the hy brids do not blindly trust retrieval; they adaptively fall back to parametric knowledge when the retrieved context is unreliable.
Finally, even in the no-conflict slice where both Internal-only and External-only are wrong, Hybrid- Joint can outperform both by aggregating partial evidence from the two sources. This reflects a second layer of complementarity beyond simple source selection: hybrid reasoning can construct better answers by jointly leveraging noisy parametric and contextual signals, not just by choosing between them.
Overall, this analysis directly quantifies the conflict slices and shows that our hybrid strategies implement sensible arbitration policies: when retrieval is reliable, they behave like "switch-to- external" systems; when retrieval is noisy, they lean back on internal knowledge while still extracting value from combining both signals.
20

---

Published as a conference paper at ICLR 2026
RQ
Method
Int wrong Ext correct
Int correct Ext wrong
conflict
Total
Hybrid-Joint
75.6
51.8
48.4
59.2
Elybrid-Fallbade
99.5
6.0
44.4
62.5
1.0
Hybrid-Oracle
100.0
100.0
44.4
70.5
External-only
100.0
0.0
44.4
62.2
Internal-only
0.0
100.0
44.4
31.9
Hybrid-Joine
66.7
58.2
24.5
36.1
Hybrid-Fallbace
88.9
41.8
21.9
32,9
0.2
Hybrid-Oracle
100.0
100.0
41.8
External-only
100.0
0.0
14.5
19.8
Internal-only
0.0
100.0
14.5
31.9
Table 5: Behavior of hybrid systems under internal-exteral conflicts on RepliQA. Entries are accuracies within each correctness slice: Internal wrong External correct, Internal correct External wrong, and No conflict. For retrieval quality 1.0, the slice proportions are 38.6%, 8.3%, and 53.1 53.1%, respectively; for retrieval quality 0.2, they are 9.9%, 22.0%, and 68. %.
B.4 OUT-OF-DOMAN EVALUATION AFTER ACTIVE INDEXING
We evaluate the out-of-domain behavior of models after RepliQA-only continual pre-training (CPT). We measure (i) perplexity on held-out Wikipedia and ArXiv corpora, and (ii) TriviaQA accuracy after task-specific fine-tuning. Consistent with Khalifa et al. (2024), we observe an increase in perplexity on natural text after CPT. We attribute this rise to two factors: (1) Domain-specific CPT, which is known to induce catastrophic forgetting when the CPT corpus is significantly narrower than the original pre-training distribution; and (2) Document identifiers, whose form (c.g., natural-language titles vs. integers) introduces different levels of distributional shift.
To disentangle these effects, we train five CPT variants on the same backbone with identical token budgets: (1) raw data (no titles), (2) Passive Indexing with natural- language title IDs, (3) Passive Indexing with integer IDs, (4) Passive Indexing with repeated lauural--nguage IDs, and (5) Active Indexing. We then evaluate all models using the same OOD metrics.
CPT Method
Wiki PPL.
ArXiv PPI.
TriviaQA Acc.
Base Model (No CPT)
6.71
56.0
Raw Data (No Title)
32.15
6.65
50.8
Passive Index (Title ID)
54.31
6.90
49.5
Passive Index (Integer ID)
60.15
6.88
49.2
Passive Index Repeat (Title)
39.78
6.80
51.2
Active Index
26.52
5.40
50.5
Table 6: OOD evaluation after RepliQA-only continual pre-training. Lower perplexity is better. Higher accuracy on triviaqa is better.
Findings. 1. All CPT variants raise OOD perplexity, confirming that narrow-domain CPT is the dominant source of forgetting, as even the Raw Data (no-title) model shows substantial degradation. 2. Natural-language IDs less harmful than integer IDs: among Passive Index variants, perplexity follows the pattern natural-text < repeatedd-nttural-eext < integer-ID, indicating that well-formed textual identifiers integrate more smoothly into language modeling. 3. Repeated titles are not particularly damaging: unlike Khalifa et al. (0224). we do not observe large penalties from repetition, likely because our identifiers are natural text rather than opaque tokens. Active Indexing is the least harmful variant: it obtains the lowest perplexity on both Wikipedia and ArXiv. suggesting its augmentation procedure improves linguistic fluency within this knowledge domain. OOD effects vary across corpora: perplexity increases more Wikipedia than on ArXiv, indicating that forgetting depends on similarity between the CPT distribution and the target corpus. Finally, perplexity does not correlate with QA performance: despite substantial perplexity differences across variants, TriviaQA accuracy after fine-tuning remains similar, implying that perplexity shifts reflect stylistic distributional drift rather than loss of underlying knowledge.
21

---

Published as a conference paper at ICLR 2026
C
MORE RELATED WORK
More External Citations A parallel line of external citation work targets finer-grained ground- ing, the sentence/span and even token level, primarily in open-book settings. Systems such as GopherCite explicitly interleave answers with short, verified quotes and learn to abstain when unsure, illustrating span-level attribution tied to retrieved evidence. (Menick al. 2022) Semi-extractive generation (SEMQA/Quolen
enforces copy-and-comee outputs, yielding inline quoted spans generation optimizes for concise, sentence-local citations consumers can check quickly (Slobodkin et al.) (222). For long-conters QA, benchmark train models that produce answers with sentence-level
by construction, (Schuster al., 2024) while Docall-attributaaee
LongCite and its LongBench- citations in one pass (Zhang et al. 2025a)). They still cite from the external context and can't produce citations in closed-book manner. On the evaluation side, EXPERTOA contributes expert-curated questions and expert-verified, clain-evidenee annotations for long-form answers answers-audttin provided citations and enabling claim-level scoring in high-stakes domains. (Malaviya et al. (024) Methods that unify retrieval and reflection (e.g.. Self-RAG), add self-critique flags to segment-level citations. (Asai et al.) 2024) Beyond suringggenneation approaches, post-hoc attribution retrofits support after the fact. RARR ("research & revise") finds evidence and edits outputs align claims with sources, (Gao et al. 2023a) and follow-ups for long documents decompose answers into factual units before mapping each unit to supporting sentences, improving coverage of fine- fine-grained support. (Sancheti et al., 2024 Ramu et al. (2024) Finally, token-level context credit assignment (c.g.. Token- Shapley) scores which specific context tokens support each generated token. (Xiao et al. (025) These directions still rely on external evidence at inference, and thus constitute external citation.
auditing system-
Data attribution seeks the training examples that most influenced model's behaviour (Park et al., 2023) (e.g., via influence functions or gradient tracing). while fact attribution asks which facts support a given answer-often handled by external retrieval by pretraining identifiers that may not coincide with truly influential data. Recent analysis-tine time tools underscore this gap: OLMoTrace traces verbatim spans in model outputs back to specific pretraining documents across toilti--illion-- token corpora in real time, and RapidIn retrieves token-wiss influential training points at scale via compressed gradient eaching (Liu et al. 2025 Lin al.) 2024). Empirical studies further show that examples ranked as most influential need not be those that explicitly contain the cited fact, and that alignment between the two increases with model/corpus scale (Chang et al. 2025). These objectives therefore do not necessarily align. Active Indexing bridges them: by presenting each training fact with its document identifier in QA-style format, it explicitly couples the influential data point with its provenance, so that reproducing an identifier at inference time simultaneously evidences both data lineage and factual entailment- muaiimmnt-accheviing achieving traceable citations without expensive post-hoe analysis and aligning factual correctness with the model's 's true training history.
Copyright, Transparency, and Legal Considerations The legal status training on copyrighted content is the subject of active policy debate. (Chen et al. (024]) By encouraging models to surface explicit provenance, our approach offers technical step toward such transparency: citations make it casier to audit whether model relies on protected material. However, also highlight dual risk in Appendix[[] ! stronger provenance can increase exposure of private or proprictary text snippets. Exploring privacy- ppreeeriee identifiers (e.g., hashed IDs, differential privacy) and selective redaction during generation remains important future work.
D IMPLEMENTATION DETAILS
D.1 TRAINING
We perform continual pretraining using H200 GPUs for 3 epochs across all methods in the main experiments, with batch size of 256, maximum context length of 2048, and constant learning rate of 5e-5 with 10 warm-up steps. With 8-bit Adam W paged optimizers, each model fits on a single GPU and can be trained using DDP without requiring FSDP. For instruction tuning, we use linear decay scheduler with a learning rate of 5e-6, train for up to 5 epochs with early stopping, and use a batch size of 64.
22

---

Published as a conference paper at ICLR 2026
The longest continual pretraining run (Qwen-2.5-7B on 3B tokens 3 epochs 9B tokens) takes 320 H200 GPU hours (80h x 4 GPUs). The shortest on 7B (Passive Indexing with 400M x 3 1.2B tokens) takes 43 H200 GPU hours.
D.2 INTERNAL CITATIONS V.S. EXTERNAL CITATIONS
In the experiments of comparing internal citation with external citations, we vary retrieval quality from sparse (BM25) to dense retrieval (Lin et al. 202T) by mixing top-5 documents retrieved by each method. Specifically,
construct hybrid top-5 sets by randomly sampling a proportion of documents from the sparse and dense retrieval outputs. We report both answer correctness and citation which combines citation precision and recall single score.
And more details for our hybrid methods are below:
Hybrid Joint: To train model that can leverage both internal and external knowledge, fine-tune the Active Indexing model to handle retrieved documents as input. For each question in the instruction tuning set (Appendix A.4). we retrieve external documents using both sparse and dense methods. We then create two training instances per question: One with sparse-retrieved documents + question as input. One with dense-retrieved documents + question input. In both cases, the target output the correct answer with citations. This trains the model to produce grounded answers regardless of the quality of the provided documents. We find performance improves when we adopt chain-of-thouge format, where the model first reflects on document sufficiency before answering, rather than directly generating an answer.
Hybrid Fallback: At inference time, we first use 3-shot prompt in the RAG setup to generate answers using the top-5 retrieved documents. The prompt allows the model to abstain if finds the documents insufficient. If it abstains, we fall back to the Hybrid Joint model, which can rely on both external and internal knowledge. If the documents are deemed sufficient, we proceed with standard external-only RAG generation with citations.
E CITATIONS EVALUATION
E. LONG-FORM CITATIONS EVALUATION
For each model- generated long-form answer, we first use GPT-4.1 to decompose it into self-contained claims, each linked to its cited source. We then retrieve the corresponding documents from MongoDB-bar -oaeeddaedda
2 corpus. Since documents are often too long for the NLI model's (Gekhman et al. 2023) input, we chunk each into 512-token segments and retrieve the most relevant ones. citation considered correct if any chunk entails the claim. Citation precision is defined as the proportion of citations that support their claims, and recall as the proportion of claims that are fully supported.
E.2 SHORT-FORM CITATIONS ERROR ANALYSIS
We conduct citation analysis on the best Qwen-2.5-7B model trained on RepliQA with Active Indexing. Using 1PT-4.1, we assess whether the cited documents entail the model-generate answers. We find that 7% of examples cite documents that support the answer but do not match the gold citation, indicating that true citation accuracy may be underestimated. However, 6.2% of examples include correct citations but incorrect answers, suggesting that the model may sometimes retrieve the right source while generating incorrect response.
F PRELIMINARY EXPERIMENTS
F.1 DOCUMENT IDENTIFIERS
We investigate which types of document identifiers are most effective for language models to memorize during continual pretraining. Experiments are conducted on the SciQAG dataset using LLaMA-3.1-IB. To evaluate memorization, we measure accuracy @ and accuracy @ 10, where the
https:/ www. mongodb.ccmm
23

---

Published as a conference paper at ICLR 2026
model is given the document content and asked to rank possible document IDs by the summed log- proobabbiility of each ID. Higher scores indicate stronger memorization of the associations. We compare different strategies for constructing document identifiers, evaluating their effectiveness in continual pretraining for memorization.
Natural Title (Raw): A baseline without continual pretraining. The model is directly prompted to rank text titles given document content. This tests whether pretrained LMs can match content to titles without exposure.
We perform continual pretraining where each document appended with its human-written text title. This approach uses natural-language identifiers that align with the model's training distribution.
Natural Titles
Hierarchical K-Means Integer (HKM-Integer):: Instead of using random integers, we construct semantically structured integer IDs following Taa et al. (2022). Documents are embedded and clustered using K-means into 10 top-level groups. Each group assigned prefix digit. The process is recursively applied within each cluster, with each level adding digit to the ID. Documents with shared prefixes are semantically similar, making it easier for the model to generalize over structured identifiers.
4. Hierarchical LDA with Keyword Labels (HLDA-Keywords) We apply hierarchical topic modeling (LDA) recursively cluster documents. For each cluster, we use an L.LM to generate a representative keyword based on its most salient documents. The final identifier is a concatenation of these keywords along the cluster path, forming semantic, hierarchical label.
S.Domain-Fiie First Keyword Identifier (Domain-Keeworrt Each SciQAG document is tagged with domain and associated keywords. We construct identifiers by concatenating the domain name with its keywords, creating top-down semantic label (e.g., physics-energy--nne -nttropy..
6. Keyword-First Domain Identifier (Keywords- bottom-up manner. Keywords appear first, followed by the broader domain label (e.g., entropy- energy-phin energy-physics), emphasizing specificity before generality.
Similar the above, but constructed
Table Results of different document identifiers on SciQAG.
ID Type
Acc@1 1
Acc@ 10
Natural Titles (Raw)
9,7
46.3
Natural Titles
53.3
75.3
HKM-Integer
2.0
21.7
HLDA-
32.0
50.7
Doeainn>keywords
28.7
47.7
keywords--Domain
26.7
45.3
As shown in Table 7 even semantically structured integer-based identifiers perform significantly worse than text-based methods. We attribute this the nature of continual pretraining, where the document is directly followed by its identifier. In this setup, natural text provides a more effective and fluent learning signal for the language model.
Among the text-based methods, natural titles achieve the highest performance. We hypothesize three reasons for this: (1) natural titles tend to capture the most salient information from the document and have high information density, (2) they are more fluent and better aligned with the model's pretraining distribution, and (3) the SciQAG corpus is relatively small- within the memorization capacity of the model the benefits of structured or compressed identifiers (like integer codes) are less pronounced. Such structured identifiers may only offer advantages at larger scales where memory constraints become limiting factor.
24

---

Published as a conference paper at ICLR 2026
F.2 WIKIPEDIA TASKS
We evaluate our approach on two popular Wikipedia-based QA benchmarks: TriviaQA (short-form) and HotpotQA (long-form). For HotpotQA, we focus medium-difficulty two-hop questions to match the capabilities of Llama-3.1-11 1. We compare two settings: (1) Raw model directly instruction-tur
each task, and (2) Passive Indexing. which adds a continual pretraining stage on the Wikipedia corpus before instruction tuning. For TriviaQA, we use Exact Match for Correctness and Citation Precision for citation quality. While for HotpotQA, we use Exact Match for Correctness and both Citation Precision and Citation Recall for citation quality.
As shown in Table continual pretraining provides no noticeable gains. The strong QA and citation performance of the raw model indicates that LLaMA-3 already memorizes much of Wikipedia's content and titles during pretraining, leaving limited headroom for further improvement via continual pretraining. Interestingly. citation accuracy exceeds QA accuracy, suggesting that the model can often guess the correct title even without fully answering the question likely because Wikipedia articles topically coherent and revolve around predictable entities.
Table 8: Preliminary Experiments on TriviaQA and HotpotQA
IB
TriviaQA C-Pr
HotpotQA C-Pr
Corpus
#Tokens
Acc
C-Re
Acc
C-Re
Raw
34.7
59.6
26.0
55.9
50.7
PassIndex
Wiki
418M
31.3
61.4
25.3
54.7
51.1
G QUALITATIVE EXAMPLES OF CITE-PRETRAIN BEHAVIOR
We perform manual inspection of model outputs across both long-form and short-form QA set- tings, highlighting typical behavior patterns-bote strengths and failure alonssdd concrete examples.
1. CORRECT ANSWER WITH FAITHFUL AND DIVERSE CITATIONS
ideal cases, model not only produces a factually accurate and coherent answer, also cites multiple distinct documents, each supporting different part of the response. This indicates that the model is capable of recalling parametric knowledge and integrating diverse sources during generation.
Example:
Question: In 2012 the remains of King Richard III were found in a carpark in Leicester. How can scientists be certain that they've found him?
Model Answer:
Scientists identified King Richard III through a combination of archacological evidence and forensic analysis.
<|New Scientist Confirms Richard III Skeleton Find with
99. 99948 Probabilityl>
...a skeleton under council car park...
RRicaard III Burial Site: Uncovering the Significance of the Greyfriars Church Choir|>
scoliosis in the spine, consistent with historical descriptions...
Richard III of England]>
head wounds matched reports of his death...
Leicester Car Park Sighting: DNA Tests Confirm Richard III's s Remains Found...
...DNA matched two living relatives...
< New Scientist confirms Richard III Skeleton Find with
99. 99948 Probabilityl
Citation Precision
25

---

Published as a conference paper at ICLR 2026
Correctness Score: 0.667 Citation Score: 1.0
The answer addresses the question with high fidelity, and each citation grounds different factual element. This demonstrates the model's ability to attribute evidence at a fine-grained level across documents.
2. CORRECT ANSWER BUT INCORRECT CITATIONS
In some cases, the generated answer is factually correct and well-structured, but the cited documents are irrelevant. This suggests a mismatch between content planning and citation generation.
Example:
Question:
Why do online communities crumble as they gain popularity?
Model Answer:
Communities may lose cohesion they scale. < |Understanding the Far-Right in Canada: A
reat-eennAnee fasciste Perspectivel>
...larger groups face moderation and trust challenges...
Understandin ng the Far-Right in Canada: A
Mentreal-Ame iffscissstsee Perspectivel>
...trolls and bad actors can increase toxicity...
< How Reddit's s New Approach Tackles Hate Speech without Banning Communities ...member attrition often follows hostile interactions...
WIRED |>
&Navvigaing Online Privacy: Protecting Your Rights in the
Digital Age >>
balanccin inclusivity with safety becomes harder...
Understandingg the Far-Right in Canada: A
Montreal -Ant fasciste Perspective
Correctness Score: 1.0 Citation Score: 0.0
The answer broadly accurate and well-articulated, but the cited documents bear no meaningful relation to the explanation. The model appears to attribute wrongly.
3. FAITHFUL CITATIONS BUT INCOMPLETE ANSWER
Sometimes, the model successfully grounds all claims in real documents, but the final answer fails to directly address the question.
Example:
Question:
Why do so many drugs cause "dry mouth" as a side effect?
Model Answer:
26

---

Published as a conference paper at ICLR 2026
Many drugs cause xerostomia, dry mouth.
< U Medication Side Effects: The Prevalence of Dry Mouth in Clinical Trials
...can affect comfort and oral function...
Understanding Medicatiss Side Effects: The Prevalence of Dry Mouth in Clinical Trials|> common in antihistamines and antidepressants...
Unnderstanding Medi cation Side Effects: The Prevalence of Dry Mouth in Clinical Trials|>
..dosage changes may help manage it...
Understanding Medi cation Side Effects: The Prevalence of Dry Mouth in Clinical Trials|>
Correctness Score: 0.0 Citation Score: 1.0
The citations accurate, but the answer never actually addresses why these drugs dry mouth, indicating a reasoning gap.
4. "TITLE LURE" ERRORS IN SHORT-FORM QA
In short-form QA tasks, the model sometimes selects citations solely based on title relevance, even when the document content lacks the required evidence. This reflects superficial attribute mechanism.
Example:
Question: How is Boston addressing the digital divide in terms of communications technology from December 2023? Model Answer:
Boston is providing subsidized smartphones and free digital literacy workshops. <|Integrat vive Technol ogies in Sma rt City Infrastructurel
Ground-truth citation: Boston's 's Infrastructure Development Initiatives: Paving the Way Forward
While the cited title seems topically aligned, it does not support the factual claim. This failure mode could mislead users who assume correctness based on citation phrasing.
5. CROSS-DOMAND LOOKALIKES AND "NEAR MISS CITATIONS
Occasionally, the model cites from mismatched domain e.g.. general Wikipedia article instead of domain-specifit source like PapliQA-prdduceee citations that superficially resemble the ground truth but lack factual alignment.
Example:
When the last game of Copenhagen's basketball season played in 2023? Model Answer:
Question:
The final game was played on May 2023.
< Copenhagen|
Ground-truth citation: Intramural Sports: Growing Trend in Copenhagen
The Wikipedia article Copenhagen is superficial match but lacks the actual fact. The model retrieves the wrong domain while giving the correct answer-possing from memory.
27

---

Published as a conference paper at ICLR 2026
H ACTIVE INDEXING
Here we show the details of active indexing.
H.1
FORWARD
Step 1: Entity Extractions first extract entities from each document, with the prompt:
Entities Extraction
You will be given document. Your task is to extract important entities mentioned in the text.
Entities include names of people, organizations, locations, dates, and other identifiable items. Use the categories below guide:
People Organizations Person, Organization
Locations - Country, City, Facility, Region
Temporal Date, Time
Events Historical notable events
Objects Products, Works of art, Laws/Policies
Concepts Theories, Fields, Idcologies
Quantitics Numbers, Money, Rankings
Biological/Chemicall - Species, Compounds
Other Named documents, Tasks, Technologies
Document: [document]
Only return the 20 most important entities based on their relevance to the main topics of the document, where each entity is separated by newline. your output, only return the important entities themselves, and do return any other information like their categories types.
Step Forward Data Augmentation data generated from GPT to generate relevant questions each entity conditioned on the document.
Then we utilize an LLM (Qwen-2.5-77) trained by the seed
Forward Augmentation
You will be given document, its title, and an entity from the document. Your task is to generate detailed questions that explore the relationship between given entity and the document. Specifically, how, what, when, where, why, or if the entity related the content the document. In each of your questions, you should include the entity and the document detailed answer.
questions should require
For each question you create, provide a detailed, elaborated answer that explains the relationship between the entity and the document. The answer should be based on the content of the document and should not include any external information.
Title: [title] Document: [document]
Entity: [entity]
Step 3: Clean up and adjust titles When LLMs generate questions, the document identifiers may omit parts of the original titles. To correct this, we apply heuristics to locate and replace them with the corresponding titles from the corpus. We then mark the document identifiers in the questions with special tokens.
We extract up to 10 entities per document, resulting in approximately 1.28B tokens 333x the size of the original corpus.
H.2 BACKWARD
Step 1: Retrieval We construct both sparse (BM25) and dense indexes (Lin al. 2021 using Pyserin For each document chunk, we retrieve top-200 relevant chunks using sparse and hybrid retrieval. From these, we select the top 10 chunks from distinct documents, then randomly sample 1-3 to form a diverse chunk cluster of size 2-4.
28

---

Published as a conference paper at ICLR 2026
Step 2: Generation We GPT-4.1-min and fine-tuned Qwen-2.5-3B to generate cross- document cetructiionresponne pairs from the retrieved clusters. Qwen-2.5-3B trained on GPT-4.1- mini outputs to reduce generation costs. The imput prompt format is detailed below.
Step 3: Filtering LLM-generated document identifiers may contain noisy patterns (e.g., "document: xx", "title: xx", or generic placeholders like "document 1*)). We detect such cases using heuristics and discard them, which filters out approximately 4.9% of the data. And then we replace the <source> marker with the special tokens.
As result, we generate around 1.47B tokens (3.8x) for main experiments.
Backward Augmentation
You will be provided with multiple documents. Your task is to construct self-contained instruction- answer pair that requires language model to synthesize information from two more of these documents in order generate the correct
The instruction should:
Be clear, specific, and fully self- contained, so it can be understood without access to or mention of the original documents.
Prompt the model integrate, compare, Avnid phrases like the provided documents" "based on the above", or anything references existence of documents.
multiple sources of information.
The must:
derived by combining or reconciling information from multiple documents. Attribute every factual claim using the format <source>-Th title of documente/ssurree> indicate where the information came from.
[documents]
Return the instruction the first paragraph and the answer in the following paragraphs.
H.3 GENERATOR MODELS CAPACITY ON DATA QUALITY
We investigate whether the effectiveness of Active Indexing depends on the choice of data generator. For backward augmentation, compare data generated by GPT-4. 1-mini versus 3B model trained by GPT-4.1 1114.1-mini. 1-mini. Experiments are conducted on RepliQA using LLaMA-3-8B. Citation precision drops from 30.6 to 25.3 when switching from GPT-4. 1-mini to the tuned Qwen-2.5-33 generator, despite using more data (34M vs. 40M tokens). Nonetheless, this still significantly outperforms the passive indexing baseline (2.6 citation precision).
29

---

Published as a conference paper at ICLR 2026
H.4 EXAMPLE ON FORWARD AUGMENTED DATA
Forward Augmentation Example
Jack Aroold (director)
**Question 1:** How is the television series "Alias Smith and Jones" connected Jack Arnold's career as described in the document titled <source>Jack Arnold (director)</souree>?
** According to the document <source>Jack Arnold (director)</suure>> Jack Arnold directed several episodes of the television series "Alias Smith and Jones" during his television career. This indicates Arnold was involved in the production of television shows beyond his work film, including directing episodes of this series that aired in the 1950s. The document highlights Arnold's versatility and his ability to work in different media, including television, alongside his filmmaking career.
**Question 2: What role Jack Aroold play in the television series "Alias Smith and Jones," and how does this fit into his overall career as detailed in the document titled <source>Jack Amold (director)</ssurces??
** ** The document does not specify the exact role that Jack Arnold played in the television series "Alias Smith and Jones." However, it notes that Arnold directed multiple episodes of this series, which falls under his television work. This placement suggests that Arnold was a director involved in various television productions, contributing the 1950s. His involvement in "Alias Smith and Jones" fits into his broader role director who transitioned from throughout his career.
the overall television landscape during
television, showcasing his versatility and the variety projects undertook
* **uustion 3: When did Jack Amold begin directing episodes of "Alias Smith and Jones." and what significance does this timeline have the context of his career as presented in the document <source>Jack Amold (director)</source>?
#+ The document does not provide specific year for when Jack Arnold began di- recting episades of "Alias Smith and Jones." " However, places Arnold's work television, including directing episodes of AAlias Smith and Jones," within his television career, which is noted to have begun in 1955. This timeline situates Arnold's television directing alongside his earlier work on "Science Fiction Theater." "Perry Mason." and other productions, indicating steady progression into television during the 1950s. This placement highlights the evolution of Amoll's career from film into television prominent director.
**Question 4:++ Why is "Alias Smith and Jones" important in the context of Jack Arnold's work as discussed in the document titled <source>Jack Arnold (director)c/soure>>??
** An "Alias Smith and Jones" is important the context Jack Arnold's work be- cause it represents one of the television productions he directed, adding to his diverse portfolio of work that includes both film and television. The document emphasizes Amold's role as director who worked across genres and mediums, and "Alias Smith and Jones" is part of the evidenee of his involvement in television series during the 1950s. This indicates that Arnold also contributed significantly to television, making a multifaceted director.
confined film
30

---

Published as a conference paper at ICLR 2026
EXAMPLE ON BACKWARD AUGMENTED DATA H.S
Backward Augmentation Example
Explain how the molecular weight and chemical modification of polypropylene influence its compatibility and processing in polymer-clay and polymer-fibe nanocomposites. your explanation. discuss the role of maleic anhydride grafting and other grafted copolymers compatibilizers, how molccular weight parameters are characterized, and how these factors affect composite preparation methods and resulting material properties.
Instruction:
Answer: The molecular weight polypropylene and its chemical modification significantly affect its compatibility and processing behavior in both polymer-clay and polymer-fiee fiber nanocomposites. Maleic anhydride grafted polypropylene (PP-g-MAH) which contains small amount of grafted maleic anhydride (e.g.. 0.29 wt% MAH), is prepared via melt grafting and is commonly used compatibilizer enhance interaction between polypropylene and fillers such as montmorilloiite clay. The PP-g-MAH/clay nanocomposites are prepared by melt compounding at temperatures around 190-200 'C, with clay loading around 3 w1% to promote good dispersion and intercalation within the polymer matrix <source>Study nonisothermal crystallization of maleic anhydride grafted collyrayllaaemnnnnmoillnneee nanocomppsile</sureee...
Characterization of molecular weight in polymers like polypropylene essential for under- standing how chain length affects viscosity and processing. Molccular weight paramcters such weight-average molecular weight (Mw), number-average molecular weight (Mn), and intrinsic viscosity are typically determined using techniques such as size-exclusion chromatography (SEC) viscometry suitable solvents. Studies on poly(1,4-butylene adipate) demonstrate that intrinsic viscosity follows a Hark-HHowink-Sakurade (MHS) type relation with molecular weight, which varies depending molecular weight distribution breadth. Though this example is for PBA, similar principles apply to polypropylene regarding the relationship between molecular weight and flow properties Relutos>Rel intrinsic viscosity to molecular weight poly (1. 4-butylene adipate)</sourre>.
In addition to maleic anhydride grafting, other grafted copolymers such PP-g-GMA (maleic anhy- dride replaced by glycidyl methacrylate groups) and SEBS-g-GMA --GA are employed as compatibilizers natural fibers like hemp. These grafted
improve interfacial adhesion in polypropylene composites compatibilizers are prepared by melt radical grafting and facilitate better bonding between hydrophobic polypropylene and hydrophilic fibers, often used alongside radical initiators in melt mixing processes temperatures around 170 200 "C csource-uunciioallzatoo compatibilization and propertics of polypropylene composites with Hemp fibres</source>.
Overall, molecular weight influences processability by affecting polymer viscosity and chain entanglements, impacting the dispersion of fillers and the mechanical performance of the composite. Chemical modification through grafting functionalities like maleic anhydride glycidyl methacry late introduces reactive sites or compatibilizing groups that improve the interfacial adhesion between polypropylene and both inorganic fillers (like montmorillonite clay) and organic fibers (like hemp). These compatibilizers enable the use of melt mixing techniques elevated temperatures to produce nanocomposites or fiber composites with improved structural and functional properties <source>Study on nonisothermal crystallization maleic anhydride grafted lnmpmooronemmommmoiillooonttt nanocom- posite</source>, Coource>Funetionalizzee composites with Hemp fibres</surce>>.
compatibilization and properties polypropylene
LIMITATIONS AND FUTURE WORK
I.1 LIMITATION
Scalability: Model Size, Corpus Coverage, and Augmentation Budget Our setup offers computationally manageable framework for academic research, but it operates at a much smaller scale than production-grat gradde pretraining, which often involves trillions of tokens across vastly more diverse domains than those included in our study. We observe a large performance gap between 3B and 7B models, yet it remains uncertain how these trends evolve at larger scales (0.B..100B). It is an open question whether Active Indexing continues to yield gains, plateaus, or even regress as model capacity increases. Our data augmentation budget limited in the main experiments. However,
31

---

Published as a conference paper at ICLR 2026
scaling analysis shows consistent improvement even at 16x the original data size, suggesting that the method could further benefit from larger augmentation.
Synthetic Data Quality LLM- -enerrted QA pairs may introduce hallucinations into the augmented data, which can lead to subtle degradation in model behavior. While effective aggregate, this approach could benefit from future work on hallucination detection, filtering, or confidence-aware generation strategies.
Evaluation Limitations Our evaluation of long-form citation relies on NLI models to judge claim support, introducing dependence on their accuracy and coverage. Although this provides scalable proxy, it adds noise to the measurement and may miss nuanced cases. Incorporating human evaluation would strengthen the reliability of results, particularly for ambiguous or multi-hop claims.
1.2 FUTURE WORK
Multilingual and Domain-Specific Attribution Our experiments are limited to English and general-domain corpora. Extending Active Indexing multilingual settings and high-stakes do- mains such as law, medicine, or finance-por poses unique challenges. These domains often require precise terminology, complex reasoning, and domain-specific citation standards. Future work could develop tailored QA generation methods and identifier formats for these settings, and perform in-depth evaluations of citation fidelity and safety in domain-critical applications.
Enhancing the existing bindings between facts and their identifiers While our methods focus on building new bindings between facts and document identifiers, existing pre-trained models may already encode implicit associations between facts and surface features such as titles, as we observed in the Wikipedia corpus. Beyond titles, there may be other weak or "loose" associations already present in the model, which could potentially be strengthened through better prompting strategies or further post-training methods such as reinforcement learning.
Scaling Laws and Saturation Points
We observe consistent improvements with increased aug- mentation and model size up 7B, but it remains unclear when and whether gains saturate. With more computational resources, future work can extend scaling curves to larger models (e.g.. 14B, 32B, 70B+) and higher augmentation regimes (e.g., 32x, 64x). This would help identify optimal compute-utility tradeoff's and determine whether benefits of Active Indexing persist at frontier scale.
Complementarity with Retrieval-Augmentt Generation (RAG) Internal citation and external retrieval are complementary: the former leverages memorized knowledge, while the latter provides up-to-date unseen information. A promising direction is confidence-aware hybrid systems-where the model cites from internal memory when confident, but falls back to retrieval when uncertain. Exploring how Active Indexing be integrated into such hybrid systems may yield the best of both worlds: low-latency and high-coverage citation.
Privacy-Preeerving Attribution Enabling internal citation increases the model's tendency to surface memorized content, which may include sensitive or proprietary information. Investigating whether attribution behavior exacerbates privacy risks is important open question. Future work could explore mitigation strategies, such as differential privacy, selective redaction of identifiers, or training-time filtering, to balance attribution fidelity with privacy preservation.
Human-Centered Evaluation and Interpretability While our current evaluation pipeline is largely automatic, the real-world utility of citations depends on user trust and interpretability. Future work could conduct human studies to assess how internal citations affect perceived credibility, transparency, and user trust partitullllll in comparison RAG non-citing models. Incorporating explanations of why citation was chosen (e.g.. via rationales) could also improve interpretability and debuggability.
32

---

Published as a conference paper at ICLR 2026
J USE OF LLMS
We used LLMs to assist with writing. Specifically, we employed GPT-5 thinking, GPT-5 and GPT-4o rephrase paragraphs for grammatical correctness and improved flow. We also used them to shorten lext, making descriptions more concise and easier to read. All LLM-generated text was reviewed, edited, and approved by the human authors.
33


Reviews:
[{"review_id": "vz1QMherqK", "reviewer": "Reviewer_grma", "summary": "This paper introduces CitePretrain, a framework that teaches large language models to cite the documents they learned from during pretraining without RAG. The goal is to make models more transparent and trustworthy by letting them show where their knowledge originally came from. The work builds on Khalifa et al. 2024, extending it from synthetic to real-world data like Wikipedia, Common Crawl, and arXiv. The main addition is Active Indexing, a continual pretraining step that links each document to an identifier and trains the model in two directions: Forward indexing: generate facts when conditioned on a document ID and Backward indexing: cite the correct document given a fact. Experiments on Qwen-2.5-3B and 7B show that Active Indexing improves citation precision over baselines, while keeping answer accuracy stable.", "questions": "- Do you make sure that you're evaluating citation on totally unseen documents data as in Khalifa et al.? Or could the same document be included in both Active indexing and in the test set? \n- Have you tried anonymizing or hashing document titles to make sure the model isn’t just predicting them directly?", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. **Well-motivated:**  Making LLMs capable of citing their pretraining data is an exciting direction that’s still largely unexplored. Also, this paper extends prior work in a meaningful way. Moving from synthetic experiments to real-world data and introducing Active Indexing feels like a natural next step.\n\n2. **Active Indexing works well:**  The dual training setup (forward + backward) leads to solid citation gains. The authors also provide detailed ablations showing how much each component contributes, which helps clarify *why* the method works.\n\n3. **Thorough evaluation:** The experiments cover a range of QA datasets and both correctness and citation metrics. \n\n4. **deep analysis and insights**: The discussion on scaling, memorization vs. grounding, and the effects of dataset size is nice to see.", "weaknesses": "1. **Heavy additional training cost:**  The method needs a full active indexing phase with two objectives and lots of synthetic QA generation. That’s quite resource-heavy, and it might be hard to apply at the scale of large, from scratch, pretraining runs.\n\n2. **Forward indexing might be unnecessary:**  The results show that backward indexing (fact → doc) already gives most of the gains. The extra forward indexing adds complexity and cost for very little improvement, which makes me question whether it’s worth the extra training. This could be explained by that in forward indexing, teaching the model to go from document ID → facts feels artificial. It’s not something the model will ever be asked to do at inference time, so it might encourage memorization rather than true grounding.\n\n4. **Shortcut risk via document titles:**  Using real document titles as identifiers can leak information. The model might just learn to guess the title instead of genuinely retrieving it. For example, if the question mentions “Barack Obama,” it could easily predict `
 Barack Obama
` as the citation without any retrieval.\n\n6. **Limited OOD evaluation**  \n   The authors do not evaluate the LM out-of-domain after their active indexing. Khalifa et al., 2024 report increase in perplexity over natural text due to training on document IDs. Do you observe the same? Can you  alsoevaluate the model on OOD tasks e.g., instruction following or QA before and after the indexing training?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Ro9juUl8Rm", "reviewer": "Reviewer_m27s", "summary": "This paper introduces CitePretrain, a method for enabling LLMs to perform reliable internal citations—attributing answers to documents seen during continual pretraining without requiring test-time retrieval. The authors construct CitePretrainBench, a benchmark mixing real-world corpora (Wikipedia, Common Crawl, arXiv) with novel documents for both short-form and long-form citation QA tasks. Their approach uses two-stage training: (1) continual pretraining to index factual knowledge by binding it to document identifiers, and (2) instruction tuning to elicit citation behavior. They propose Active Indexing, which augments training with synthetic data that restates facts in diverse forms and enforces bidirectional training (source→fact and fact→source), significantly outperforming Passive Indexing (simply appending identifiers). Experiments on Qwen-2.5-7B/3B show Active Indexing improves citation precision by up to 30.2%, with performance scaling with augmentation data (up to 16×) and model size. The paper also demonstrates that internal and external citations are complementary, with hybrid approaches performing best across varying retrieval quality.", "questions": "Q1 Why are GPT-4.1 scores not bolded where it has the best performance in Table 1, given the caption says “best results are bolded”?\n\n Q2 What were the reasons behind choosing only Qwen models for evaluation?\n\n Q3 Why are models like LongCite-8B and LongCite-9B, which are fine-tuned for long-context citation, not included as baselines when long-form citation is a major theme?\n\n Q4 What happens when internal citations are wrong but retrieved external context is correct and contains the right citation?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "1.  Unlike prior work limited to synthetic biographies, CitePretrainBench introduces the first rigorous benchmark emphasizing real-world complexity across Wikipedia, Common Crawl, arXiv, and novel documents, with both short-form and long-form citation tasks.\n\n2. Active Indexing is conceptually elegant and addresses passive indexing's core limitation: failure to ground non-verbatim content. By generating diverse synthetic QA pairs and enforcing bidirectional training (source→fact, fact→source), it achieves substantial improvements (up to 30.2% citation precision gains). Performance scales favorably—continuing to improve at 16× augmentation without saturation, indicating significant headroom.\n\n3. Thorough ablations reveal that both fact variation and active supervision are essential. The paper demonstrates that internal citations excel under poor retrieval, while external citations perform better with strong retrieval, with hybrid approaches achieving the best overall performance, establishing practical value for real-world deployment.", "weaknesses": "(A) The core motivation is weak. The paper leans on the idea that RAG is too expensive or impractical, yet in real deployments RAG is the default pattern to ground models on fast-changing enterprise data. With reranking, caching, and short contexts, teams control tokens and latency, and every major cloud ships managed RAG stacks. If the goal is to argue for internal citation on cost, the comparison needs to be against optimized, production RAG baselines rather than token budget.\n\n\n(B) The claim that the method “goes beyond memorization” is overstated. Inference is constrained to titles or identifiers, which functions as an ID proxy. The paper itself notes title-level memorization on Wikipedia and infeasibility where stable IDs are weak. Bidirectional training may help usage, but the supervision still steers toward ID cues, which limits granularity.\n\n(C) The paper does attempt to answer why the method works with some ablations and scaling results, but lacks intuitive ablation experiments like ablating title distinctiveness or using near-duplicate titles.\n\n(D) Baselines and model diversity are thin. Main results are on Qwen-2.5 3B and 7B, plus a prompt-only GPT-4.1 check. There is no head-to-head against LongCite-8B or LongCite-9B or other long-context citation-tuned systems, despite long-form citation being a stated focus.\n\n(E) The hybrid story is sensible engineering, but does not address the most important failure case. The paper shows Internal-only, External-only, and Hybrid setups and average complementarity across retrieval quality, but it does not isolate the conflict slice where internal is wrong and external is correct. That is the production reality that needs to be quantified, with an arbitration policy that shows when the system will switch, abstain, or defer.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QLmkxcKNNm", "reviewer": "Reviewer_3iUa", "summary": "This paper investigates how to make Large Language Models (LLMs) reliably cite the specific documents from their pretraining data without needing an external retrieval system (e.g. RAG) at inference time. The authors aim to solve the problem of citation hallucination and create a retrieval-free alternative to RAG. The core contribution is a training strategy called **Active Indexing**. The authors contrast this with what's called **Passive Indexing** that has been studied in prior work, where a document identifier (like its title) is simply appended to the end of the text during pretraining. The authors constructed a training corpus by collecting common factual sources such as wikipedia,  then created the CitePretrainBench for evaluation by using four existing question-answering datasets (ASQA, ELI5, SciQAG, and RepliQA) whose answers are grounded in that specific corpus. Their results find supporting evidence of active indexing being a superior method than passive indexing (subject to token replay).", "questions": "1. Your experiments only included Qwen3B and Qwen7B. Does other model families and scales exhibit similar scaling behavior when trained with active indexing? For example, the paragraph \"Model Size Matters.\" in Section 5.2 needs much more than two models to make claims about patterns on model size.  \n\n2.  Section 5.1 suggests that when combined together, Active Indexing dataset is much larger than Passive Indexing baseline. Can you reveal how big is the passive indexing dataset in Section 5.1, or make a table to include comparison to how big the instruction tuning baseline dataset is.  \n\n3. Building on question 2, the comparison between Passive Indexing and Active Indexing is currently unfair. Despite adding Replay, Passive Indexing training has less fresh tokens than Active Indexing. Do you observe Active Indexing having consistent advantage over passive indexing when the dataset is truncated into the same size as Passive Indexing?  \n\n4. Section 5.4 poses an interesting question about the advantage of internal citation when external citation retrieval quality differs. In Figure 3, the x-axis represents retrieval quality. Can you use a paragraph or a figure to explain how we can quantitatively interpret what's called \"retrieval quality\" here?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper systematically conduct empirical analysis on knowledge attribution methods that moves beyond proposing a new method. By directly contrasting simple \"Passive Indexing\" with their more sophisticated \"Active Indexing,\" the work provides the community with a clear understanding of how different training strategies for attribution directly impact a model's citation accuracy. Furthermore, their scaling experiments offer crucial insights, demonstrating that this citation capability consistently improves with the amount of augmented data, which helps quantify the trade-offs and charts a path for future work in this domain.", "weaknesses": "**Novelty Concern**: The proposed training pipeline still largely mirrors Khalifa et al. (2024): (i) continue pretraining on a corpus where each document is tagged with a unique identifier, and (ii) instruction-tune the model to answer questions and emit supporting identifiers, i.e., to ground answers in specific training documents. The paper positions “Active Indexing” as the main advance beyond this baseline. However, much of Backward Active Indexing looks like a scaled-up version of Khalifa’s fact→source supervision: the model is trained on synthetic instructions whose answers integrate facts from multiple documents, and each generated factual statement is annotated with the supporting set of document titles.\n\nConceptually, this extends Khalifa’s single-document attribution setup to multi-document synthesis and per-claim citation sets, which is present in prior work. See Patel et al. (Towards Improved Multi-Source Attribution for Long-Form Answer)\n\n**Scope and stability of CitePretrainBench**: CitePretrainBench is described as a benchmark that “mixes real-world corpora (Wikipedia, Common Crawl, arXiv) with novel, unseen documents” and evaluates both short-form (single-fact) and long-form (multi-fact, multi-source) citation. The “novel, unseen” part is primarily RepliQA, which the paper characterizes as short-form QA over “fictional, synthetic documents created post-training cutoff (Monteiro et al., 2024).” \n\nIn other words, the “new knowledge” is largely imported from an existing dataset that was written after the model’s cutoff, not newly created here. This raises two issues:\n- The benchmark’s claim to test the model’s ability to “learn and cite new knowledge” depends on RepliQA still being out-of-distribution for the underlying models. That claim will become weaker as newer base models inevitably ingest RepliQA-like synthetic corpora (or RepliQA itself) during pretraining, which is outside the authors’ control.\n\n- Aside from that post-cutoff synthetic set, the rest of CitePretrainBench reuses existing QA benchmarks (ASQA, ELI5, SciQAG, RepliQA) and established correctness metrics such as Exact Match Recall, Claim Recall, and FreshEval-style LLM grading, plus citation precision/recall that check whether cited titles actually entail each generated claim. \n\nThe unification work — consolidating Wikipedia / Common Crawl / arXiv / synthetic docs under a single deduplicated title space, constraining decoding so the model can only cite that space, and evaluating multi-document, per-claim attribution in long-form answers — is valuable infrastructure. That said, the benchmark is still largely built out of pre-existing datasets and inherited metrics, and its “novel knowledge” component is fragile over time. \n\n**Memorization vs. Generalization Claim**: In Section 5.3, you proposed to use FullDoc -> Partial Doc -> GoldQA -> ModelQA as tasks that shift from testing memorization to generalization. This framing seems to assume that “memorization” = being able to associate a document ID with its own content, and “generalization” = being able to recover the same ID when you only see downstream QA-style text. But I’m not convinced that this setup is actually isolating generalization, as opposed to just increasing task difficulty under progressively less evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hR1tOBwznL", "reviewer": "Reviewer_icLP", "summary": "The paper studies whether large language models can produce reliable, verifiable citations to the documents they were (continually) pretrained on, without using test-time retrieval. It introduces a two-stage framework, including continual pretraining with “Active Indexing” plus citation-style instruction tuning and a new benchmark, CitePretrainBench, to evaluate both short- and long-form citation over mixed sources such as Wikipedia, Common Crawl, arXiv, and synthetic unseen documents.", "questions": "See weaknesses", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The problem is well-motivated and distinct from standard RAG: it targets internal, training-time attribution rather than test-time retrieval.\n2. The proposed Active Indexing pipeline yields clear, reported gains over passive / repetition baselines on multiple QA-style tasks.", "weaknesses": "1. A first concern is cost and practicality: the gains come from heavy, LLM-generated active augmentation, including intra-document QA, cross-document synthesis, bidirectional objectives (scaling to 2.75B extra tokens), which means the approach improves because we invest substantial additional training signal, not necessarily because it is an inherently lightweight replacement for retrieval. A deployment-oriented reader will wonder how often such augmentation must be repeated when the corpus changes.\n2. The evaluation benchmark is authored by the paper itself: although it is diverse, it is still tailored to documents with clean, human-readable titles and to a closed identifier space; it remains unclear how well the method transfers to messier enterprise or web-scale settings with noisy IDs, partial documents, or no stable titles.\n3. Many of the reported improvements start from very low baselines (e.g., citation precision ≈2–5% for passive or instruction-only models), so relative lifts of “up to 30 points” overstate the absolute reliability; even the best models still lag behind their own answer correctness, and long-form settings (ELI5, ASQA) remain far from robust attribution.\n4. Comparison with stronger external-citation / RAG baselines is informative but not exhaustive: the paper shows internal and external are complementary and that a hybrid is best, yet the hybrid still leaves a noticeable gap to the “oracle” combination, suggesting that the proposed training does not fully solve conflict resolution between memorized and retrieved evidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)