GenerationPrograms:  Fine-grained Attribution with Executable Programs
Authors: David Wan
Eran Hirsch
Elias Stengel-Eskin
Ido Dagan
Mohit Bansal
Venue: COLM 2025 (2025)
Keywords: long-form qa
rag
summarization
attributed text generation
TL;DR: GenerationPrograms: Fine-grained Attribution via Neural Modular Trees

Abstract: Recent large language models (LLMs) achieve impressive performance in text generation but often fail to accurately attribute their outputs, undermining trust and verifiability. Moreover, existing attribution methods do not explain how and why models leverage the provided source documents to generate their final responses, limiting interpretability. Furthermore, current attributions fail to provide a reason as to how and why the model uses the context to arrive at the final output. To overcome these challenges, we introduce a modular generation framework, GenerationPrograms, inspired by recent advancements in executable ``code agent'' architectures. Unlike conventional generation methods that simultaneously generate outputs and attributions or rely on post-hoc attribution, GenerationPrograms decomposes the process into two distinct stages: first, creating an executable program plan composed of modular text operations (such as paraphrasing, compression, and fusion) explicitly tailored to the query, and second, executing these operations following the program's specified instructions to produce the final response. Empirical evaluations demonstrate that GenerationPrograms significantly improves attribution quality at both document-level and sentence-level granularity across two long-form question-answering tasks. We further demonstrate that GenerationPrograms can effectively function as a post-hoc attribution method, outperforming traditional techniques in recovering accurate attributions. In addition, the interpretable programs generated by GenerationPrograms enable localized refinement through modular-level improvements that further enhance overall attribution quality.

Published as a conference paper at COLM 2025
GENERATIONPROGRAMS: Finegrrieeed Attribution with Executable Programs
David Wan Eran Hirsch
Elias Stengel-Eskin' Ido Dagan²
Mohit Bansal
UNC Chapel Hill Barr-llan University
Abstract
Recent large language models (LLMs)) achieve impressive performance in source-conditione text generation but often fail to correctly provide fine-grained attributions for their outputs, undermining verifiability and trust. Moreover, existing attribution methods do not explain how and why models leverage the provided source documents generate their final responses, limiting interpretability. To overcome these challenges, we intro- duce modular generation framework, GENNERATIONPRGGRMS inspired by recent advancements in executable "code agent" architectures. Unlike conventional generation methods that simultaneously generate outputs and attributions rely on post-hoc attribution, SONDEEAATTONPRORRAAO de- composes the process into two distinct stages: first, creating an executable program plan composed of modular text operations (such paraphrasing, compression, and fusion) explicitly tailored to the query, and second, exe- cuting these operations following the program's specified instructions to produce the final response. Empirical evaluations demonstrate that GENER- ATIONPROGRAMS significantly improves attribution quality at both the doc- ument level and sentence level across two long-form question-answering tasks and a multi-document summarization task. We further demonstrate that CENNERATIONPROE tion method, outperforming traditional techniques recovering accurate attributions. In addition, the interpretable programs generated by GEN- ERATIONPROGRAMS enable localized refinement through modular-level improvements that further enhance overall attribution quality.'
can effectively function as post-hoc attribu-
1 Introduction
Recent large language models (LLMs) have demonstrated impressive performance in gen- eration tasks. However, these models often struggle to attribute their outputs accurately through proper citations to source material, particularly in attributed generation tasks such as question answering (QA) (Bohnet et al., 2023; Gao et al., 2023b; Slobodkin et al., 2024) and summarization (Laban et al., 2024). Precise attributions, for example in the form of citations, have become increasingly important with the advent of agentic "deep research" workflows (OpenAI, 2025; Gemini, 2025), which explore the web to retrieve diverse sources and synthesize citation-supported reports. Consequently, failure to provide precise and fine-grained attributions significantly harms the faithfulness and trustworthiness of the gen- erated content, hindering the verification of potential "hallucinations" generated content not supported by input sources (Ji et al., 2023; Rashkin et al., 2023). Recent work suggests models achieve higher accuracy when attribution not explicitly required (Zhang et al., 2024a), indicating a significant challenge in managing both accurate generation and faithful attribution simultaneously. For example, as illustrated in Figure 1, baseline methods often fail to provide accurate citations, resulting in incorrect or missing attributions. This problem underscores the necessity for modular approach that decouples the generation of accurate answers from the process of producing precise and interpretable attributions.
Our code is available https: //gi thub. http:s//ggthhu...mm/etttaaaaa/aa wtt.://www...eeeeeaeeeeeeeeeeeee

---

Published as a conference paper at COLM 2025
Question
Should robots replace humans in the workforce?
L Baseline Output (ALCE)
2 GeuerationPrograms Output
Experts argue that the future of work will likely involve a collaborative effort between humans and muchines. Mixing Charlerd
Program: fS2, S101, Lnstruction=""ighhiiih the current trend robots replacing hunans various industries. "); [... 1; paraphrase instruction="stress
insbitity of robots
Al excels in tasks that require speed and accurucy, while humans bring intuition and creativity to the [able[111]] [
replicate empathy and enotions.")
Robots are increasingly replacing bumans in various industries, including auto production. manufacturing, service delivery, recruitment, and the financial sector, leading to human workers being shifted to lower-paid jobs or becoming unemployed[2] [101], Robots struggle to replicate human empathy and emotions[4] [4]]
Output:
Incorvect Attrbutlent
This collaboration, often referred to as augmented intelligence, aims to enhance productivity and efficiency while maintaining the unique contributions of human workers[ [18]]]]
Accorate Citation documentsl
Attributed Documents
Document 1:
Document
S2: Yes, because some human repetitive and manual labor already been replaced by robols in auto production, and manufacturing...
S100: Many believe that organizations will begin replace human employees with intelligent machines.. S101: This is already happening: intelligent systems are displacing humans manufacturing, service delivery, recruitment, the financial industry, consequently moving human workers towards lower-paid jobs
S3: No, not of them, because you need humans to supervise the work robots do.
S4: Also, robots don't do well in emulating empathy and human emotions.
Figure 1: Example output produced by generating with citations (left) compared to GENER- ATIONPROGRAMS (right), which first generates an executable program and then performs text operations. The baseline method, ALCE (Gao et al., 2023b), may omit or generate incorrect citations (underlined), whereas CENNERATIONPROGRANN and accurate citations across documents. We have color-coded the attributions in GENERA- TIONPROGRAMS for easier verification, and shortened the example to fit the page.
produces comprehensive
Furthermore, current attribution methods predominantly offer only corroboratioe attributions- identifying sources that support generated statements-rather than contributive attributions, which would explicitly clarify how and why specific sources led the model to its final output (Worledge et al., 2023; Cohen-Wang et al., 2024). This distinction is essential because verifying citations alone can be challenging (Rashkin et al., 2023; Liu et al., 2023a), while clearly understanding which sentences contributed allows users to directly verify the accuracy and appropriateness of generated content. Without contributive attributions, the interpretability of models remains limited, hindering fine-grained control and trust in the generation process.
Motivated by recent developments in modular " "code agent" architectures frameewokss that effectively break down complex tasks into simpler executable code (Yuan et al., 2024; Yang et al., 2024; Wang et al., 2024b)-we propose GENERATIONP" POORRAS,,,,,, an interpretable generation framework explicitly designed to facilitate accurate and fine-grained attribution. The text-to-code framework has led to various blan-based approaches, such methods like "Attribute First" (Slobodkin et al., 2024) that decompose the generation process into distinct stages of content selection and then surface realization. However, these methods can be rigid in their structure, often relying on predefined templates or a fixed sequence of operations. Our work builds on the idea of explicit, executable plans for generation, drawing inspiration from the modular framework proposed in (Saha al., 2023). Specifically, DENERATIONPROGR AMS formulates generation as two- stage process: first, creating structured executable program composed of modular text operations (such as paraphrasing, compression, and fusion) explicitly tailored to a given query; and second, executing these operations through clear, query instructions to produce the final response. As illustrated in Figure 1, GENERATIONPROGRR.M generates program for instance, instructing fusion module to merge two sentences with explicit guidance on which aspects should be highlighted in the output. After execution, the resulting sentence is both instruction-consicn
initially
and faithful to the input sentences (e-g.
2

---

Published as a conference paper at COLM 2025
S2 and S101 in Figure 1), and the two sentences that were used automatically become part of the sentence-level citations (e.g. [2][101]). This structured decomposition naturally enhances attribution by explicitly tracking the source inputs used at each modular step. Consequently, attribution becomes straightforward by reviewing the explicit program trace, clearly linking each generated segment to its source sentences. The program itself also serves as an explicit explanation of the generation process, enabling fine-grained verification and facilitating targeted, localized edits.
While structured programs significantly enhance traceability and attribution quality, lan- guage models still face challenges when handling extensive input contexts. Recent advance- ments enable LLMs to handle extended contexts spanning millions of tokens (Gemini Team et al., 2024; OpenAT al., 2024); however, they often struggle with reasoning over lengthy inputs, a phenomenon known as "lost-in-the-midlle (Liu et al., 2024a). To address this limitation within our structured generation approach, we investigate whether decoupling the task of identifying relevant information from the attributed generation process can further enhance attribution accuracy. Specifically, we apply summarization techniques to filter irrelevant content from retrieved source documents (Gao et al., 2023b), ensuring models concentrate solely on relevant information during generation.
Empirical evaluations across two long-form QA tasks show that GENERATION substantially improves attribution quality at both document-level and sentence-level gran- ularities, achieving gains of 31.6% and 27%, respectively, in attribution F1 over baseline methods that directly generate citations alongside outputs. Although we observe a minor trade-off in answer correctness, we demonstrate that applying summarization techniques to source documents significantly mitigates the long-context problem, balancing attribution quality and answer accuracy. Similar observations are also made when applying GENER- ATTONPROGRAME to multi-document summarization tasks. Furthermore, we showcase practical applications of attribution method. Specifically, CENERATIONPROGRAA achieves superior performance in correctly attributing previously generated sentences compared to traditional attribution methods, with the generated program serving reasoning process. Finally, we illustrate that the contributive nature of the generated pro- gram enables fine-grained, modular-level refinement. By correcting modular executions that are not faithful to their inputs, we achieve further improvements in attribution quality, surpassing standard attribution methods that employ reranking strategies.
MS, including its effectiveness as post-hoc
interpretable explanation of the model's
2 GENERATIONPROGRAMS
For long-form question-answering, the input consists of a query, or question, 9 and a set of The model then uses these inputs to generate operates in two main steps: generating
retrieved documents D = ( D2..., D.). an output o. Our method, GENERATIONPROGRAM program plan and executing this program using neural modules. Below, we first introduce a formal definition and then describe each step in detail.
2.1 Program Definition
Following Saha et al. (2023), we define program P = (P.) as a collection of k trees, each corresponding to a single sentence in the output. Each tree P, - (V, E) consists of nodes V representing source or generated sentences and edges representing neural modules applied to these sentences. Leaf nodes correspond to sentences extracted directly from documents D, intermediate nodes represent sentences generated by neural modules, and the root node represents the final generated sentence. The overall answer combines root nodes from all trees, enabling clear tracing and attribution of each output sentence back to the modules and input sentences that generated it.
3

---

Published as a conference paper at COLM 2025
Step 1: Program Generation
Input
Program
Question: Should rohots replace
Fusion(S1, instruction"KMighiigh
(Sr, instruction the inability of robots to replicate hunan espathy enotions.")
the current trend of replacing various industries."
Retrieved Documents
Step 2: Program Execution with Sentence-Level Attributions
Program
Document
P1: fSs1on(S1, 52, instruction-"iighL ght trend robots replacing hunans in various industries.")
Some human repetitive manual labor has already been replaced by robots production.
Final Output
Output Sentence I: Robots are incrcasingly replacing humanns various industriss, including auto production, manufacturing, service delivery, recnuitent [1][2]
Puston
already happening manufacturing, service delivery, recruitment, financial induatry.
P2: par instruction= Sttress inability robots replicate husan eпрathy and emotions.")
Outpar Sentence 2: Rohots straggle to replicate human emputhy and
don't in conulttig ompothy and human cmotions.
Paraphrase
emrotions
Figure Illustration GENERATIONPKOCRANS First, an executable program generated from the question and retrieved documents. Next, the program is executed using dedicated text-based operations to produce the final sentences. The sentences used are automatically treated sentence-level attributions. This design promotes both interpretability and reliable attribution by tracing the program execution and citing every sentence used."
2.2 Program Generation
The first step of GENERATIONPOOGRMMS is generating a program based on the query and retrieved documents. Specifically, we prompt a large language model (LLM) to generate Python-executable programs that correspond
individual output sentences, leveraging Python's Abstract Syntax Tree (AST) module for easy parsing. These programs may contain nested functions, allowing the output from one neural module to serve as input for another.
As illustrated in the top section of Figure 2, the generated program structure naturally results in multiple trees, each representing distinct output sentences. Additionally, our method generalizes beyond single-document summarization to query- query-focused and multi-document scenarios (Appendix A.1).
2.3 Program Execution with Neural Modules
The second step involves executing the generated program using neural modules. As depicted in Step Figure 2, GENERATIONPROGRAM retrieves the relevant sentences and invokes corresponding modules. For example, the fusion module processes sentences in the first program, whereas the paraphrase module applies to sentences in another. Each resulting sentence attributes its content to the input sentences used.
Our neural modules build upon prior modular summarization approaches (Lebanoff et al., 2019; 2020; Jing & McKeown, 1999; 2000; Saha et al., 2023):
1. Paraphrase: Generates sentence conveying same meaning with different wording.
2. Compression:
Condenses sentences while preserving original meaning.
3. Fusion:
Integrates multiple sentences into single coherent sentence, resolving redundancy and reconciling differing viewpoints.
4. Extract: Facilitates extractive summarization to extract source sentences.

---

Published as a conference paper at COLM 2025
When executing neural modules, the output should be entailed by their inputs; this principle what we refer to as local aftribution. Ensuring local attribution guarantees that, even in the case of nested functions, each step along the program trace remains faithful with respect to its input, thereby preserving transitive attribution integrity. We analyze the local attribution performance of the modules in Appendix D and further investigate localized attribution verification and refinement strategies in Section 4.3.
2.4 Effectively Handling of Long-Context Using Summarization
While recent LLMs can handle extended contexts spanning millions of tokens (Gemini Team et al., 2024; OpenAl et al., 2024), these models still exhibit limitations when reasoning over lengthy inputs, a phenomenon known as "lost-in-the-midde this, apply summarization techniques to source documents, reducing redundancy and filtering irrelevant information, thus also reducing the likelihood of generating unfaithful summaries (Gao et al., 2023b). Applying summarization to the retrieved documents ensures the model remains focused, potentially improving attribution accuracy.
(Liu et al., 2024a). To mitigate
Building upon Saha et al. (2023), we strong query-focusede summarization systems to individually summarize each source document. Document-level attribution is naturally ensured through summarization since recent methods reliably produce faithful summaries (Zhang et al., 2024b). However, precise sentence-level attribution necessitates either ex- plicitly tracked text manipulations, as in summarization. Thus, our explicit program-based approach provides rigorous sentence- level traceability and accurate attribution.
or purely extractive
3 Experimental Setup
3.1 Datasets
We evaluate our approach using challenging question-answering tasks involving retrieved sources. Dataset statistics are summarized in Table 4 in the Appendix. We include the ASQA dataset (Stelmakh et al., 2022), comprising factoid questions collected by Gao al. (2023b). ASQA uses retrieved relevant Wikipedia snippets, retrieved via GTR (Ni et al., 2022), from which we select the top 10 passages for our experiments. Additionally, we use the LFQA dataset (Liu et al., 2023a) following the split curated by Slobodkin et al. (2024), which includes ground-truth documents as part of the evaluation. For summarization, we use the processed dataset by Ernst et al. (2024), a multi-document summarization (MDS) task derived from MultiNews (Fabbri et al., 2019). Additional details are in Appendix B.
3.2 Metrics
Answer Accuracy. Since both datasets include gold-standard answers, we assess the model's responses against these references using established metrics. Specifically, we use Exact Match (EM) for ASQA, and ROUGE-L (Lin, 2004) for LFQA and MultiNews MDS.
Attribution Quality. Following prior research (Gao et al., 2023b; Zhang al., 2024a), we evaluate attribution quality using three model-based metrics: attribution recall, attribution precision, and attribution F1. Attribution recall checks if generated statements are fully supported by the cited sources by assessing entailment. Attribution precision ensures each citation accurately supports its respective statement. Attribution Fl is the harmonic mean of precision and recall. For ASQA, we use AutoAIS, leveraging TRUE (Honovich et al., 2022), a T5-11B model fine-tuned on various NLI datasets that correlates strongly with human judgment. Due to TRUE's input length limitations on LFQA and MDS, we instead utilize GPT-4o-based NLI evaluation, as recommended by Zhang et al. (2024a). Additionally, we measure the proportion of sentences lacking citations.

---

Published as a conference paper at COLM 2025
ASQA
LFQA Attribution (CPT-Ho)
MDS
Correct
Attribution (AutoAIS)
Correct
Correct
Attribution (AutoAIS)
Method
No Altr
No Allr.
RL
No Altr.
Document-level Attributions
ALCE
44.0
62.7
25.8
39.4
55.4
40.0
19.5
63.8
25.4
extr. summ.
GENPROG
41.0
87.1
0.0
32.3
94.4
0.0
19.3
94.4
0.0
GINPROG summ.
39.3
38.4
87.7
17.8
98.7
Sentence Attributions
ALCE
42.3
54.2
16.3
39.9
56.0
35,9
20.0
55.2
32.4
extr. summ.
GENPROG
41.0
79.4
0.0
32.3
82.8
0.0
19.3
90.0
0.0
GINPROG summ.
39.3
67.2
65.2
17.8
72.7
Table 1: Results long-form QA with document-level and sentence-level attributions
3.3 Model and Methods
For our experiments, we primarily compare GENERATIONPOORRMM against ALCE (Gao et al., 2023b), a method which directly generates answers and corresponding citations. We use GPT-4o (OpenAT, 2024) as the backbone for both methods, and the prompts are in Appendix We further show results with an open-source model (Llama 3.3 70B) in Appendix C.4. Both methods are provided with one-shot example to ensure adherence to the output format. To enhance attribution quality in ALCE, we separately evaluate it at both the document and summary levels. For filtering irrelevant information with summarization, we employ extractive summarization (Erkan Radev, 2004; Nenkova & McKeown, 2011; Cheng & Lapata, 2016; Narayan et al., 2018) on retrieved documents. In Appendix C.2, benchmark various summarization methods for filtering retrieved content. For ASQA, we additionally perform an initial relevance filtering step using GPT-4o to exclude irrelevant documents. Detailed procedures for relevance filtering are provided in Appendix B. For replicability, we set temperature to 0 for all models.
4 Results
We report the attribution results in Section 4.1, and then show interesting applications of GeNERATIONPROGR, including post-hoc attribution (Section 4.2), and using the generated program for fine-grained module-level detection and refinement (Section 4.3). We provide additional qualitative analysis in Appendix D.
4.1 Attribution Results
We present the results for document-level and sentence-level attribution in Table GENERA- TIONPROGRAMS consistently achieves significant improvements in attribution Fl compared to ALCE-based methods. Specifically,
improves document-level attribution F1 by 20.4% and 39.0 39.0% on ASQA and LFQA, respectively, and enhances sentence- level attribution by 25.2% and 28.8% on the same datasets. These results highlight GENERA- TIONPROGRAMS'S effectiveness in improving attribution quality. However, this increased citation accuracy is accompanied by slight reduction in answer correctness, decreasing by 2% for ASQA and 7.1% for LFQA at the document level. Nonetheless, the significant improvements in attribution quality outweigh these reductions in correctness. We fur- ther investigate the correctness evaluation in Appendix C.6, where we find that stronger LLM-based metrics and human evaluations show smaller decrease. Furthermore, due design, all generated sentences contain citations, effectively
DEONERATIONPROGRAAN
to GENERATIONPROGRAL addressing a notable limitation in ALCE, where citations were omitted in 25. 25.8% and 40% of sentences the document-level setting for ASQA and LFQA, respectively. For MDS, the results are consistent with those observed for long-form QA tasks: we a significant improvement in attribution quality, ie. increasing from 63.8 to 94.4 for document-level citations.

---

Published as a conference paper at COLM 2025
Sentence Level
Output Level
Attribution Method
Cit. Prec.
Cit. Rec.
Cit. Fl
Cit. Prec.
Cit. Rec.
Cit. F1
Semantic Similarity
1.9
0.1
0.1
11.7
9.3
9.6
LLM Prompt
9.5
7.7
7.8
13.9
14.1
13.1
GENERATIONPROGRRRM
53.4
37.3
47.4
50.0
50.4
46.7
Table 2: Post-hoc attribution results on LFQA with oracle output.
Effect of Summarization at Handling Long-Context. We next examine how applying summarization to remove irrelevant content affects model performance. For ALCE, summa- rization enhances LFQA attribution by 2.6% at the document level and 3.3% at the sentence level, though it slightly reduces EM for ASQA by 1% at the document level and 0.5% at the sentence level. Importantly, we observe that summarization generally positively impacts at- tribution quality (except for sentence-level attribution on ASQA) by increasing attribution F1 and reducing the number of sentences lacking attribution. This contrasts with prior findings by Gao et al. (2023b), who reported no improvements from summarization in attribution quality. When applied to GENERATION attribution F1 slightly, but it helps mitigate the correctness trade-off caused by enhanced attribution quality. Thus, summarization provides a viable strategy for balancing answer correctness with attribution precision. For MDS, observe that summarization does not improve the results, which agrees with prior findings that hierarchical summarization does not improve performance for summarization tasks (Wan et al., 2025).
extractive summarization typically low-
4.2
GENERATIONPROGRMM as Post-Hoc Attribution
Beyond concurrent content generation and attribution, demonstrate that GENERA- TIONPROGRAMS can also be effectively adapted for post-hoc attribution, where attributions are identified after text generation (Gao et al., 2023a). Unlike traditional methods focused solely on finding supporting citations, GENERATIONPROCK, tributive alfribution
employs form of con- al., 2024), explicitly identifying which sources directly contribute to the generation of specific content. This capability leverages the detailed pro- gram trace within GENERATIONPROGRAMS to precisely illustrate how input sources are integrated through text-based operations. Thus, our approach aligns with the concept of context aftribution (Cohen- Wang et al., 2024), pinpointing exact contextual elements respon- sible for individual statements. This method is practically beneficial for tasks such rapid verification and targeted refinement Notably, our method does not require access to model logits, making it suitable for both open-source and proprietary models.
generated outputs, as demonstrated in Section 4.3.
In our post-hoc attribution setup, the generated text is additionally provided input, and PEEEERATTIOPPORRAAM then generates a program trace to reconstruct or simulate the original output. We conduct evaluations using LFQA, which includes human annotations identifying relevant sentences for each gold answer. Given the known gold-standard sen- tence annotations, we measure citation precision, recall, and F1 by calculating the overlap between predicted and annotated sentence-level citations. Unlike standard attribution eval- uations relying on NLI, here we specifically assess the accuracy of identifying the correct set of sentence ids. We consider two settings: (1) sentence-level evaluation, measuring citation overlap individually for each sentence, and (2) output-level evaluation, aggregating relevant sentences into sets for a more lenient assessment. We benchmark GENFRATIONPROGER against commonly used attribution baselines, including semantic similarity- attribu- tion (Reimers & Curevych, 2019) and prompt-based attribution (Zhang et al., 2024a). For GENERATIONPROGAR RAMM, the attribution can be retrieved by iterating through the program trace and treating the relevant sentences sentence-level attributions.
The results, presented in Table 2, highlight that GENERATIONPROCEE PROGRAMS significantly out- performs other methods, achieving sentence-level citation F1 of 47.4-fat exceeding the next-best baseline (LLM-based prompting), which achieves only 7.8 Fl in the sentence-level setting. Additionally, since GENERATIONPROGRA MS can reconstruct sentences via its gener-
7

---

Published as a conference paper at COLM 2025
ASQA
LFQA
Method
Correct
Attr. F1 %Entail.
Correct
Attr. F1 %Entail.
ALCE
44.0
54.2
39.9
54.0
ALCE refinement
43.2
77.9
36.3
69.3
GENERATIONPROCRAMS
41.0
79.4
85.8
32.3
82.8
95.5
GENPROG module-level refinement
41.1
83.4
87.7
32.4
85.5
97.2
Table 3: Fine-grained refinement results. In addition to attribution F1, we also show the percentages the modules that are considered entailed for GENERATION
ated program traces, we reconstruct the output using the generated program, and compute ROUGE scores comparing reconstructed sentences against the original gold responses. The reconstructed output results in scores of 0...9/41.//000 for ROUGE-1/2/L, respectively. Regarding correctness, we observe that applying refinement actually decreases the ALCE's score. In contrast, module-level refinement achieves 0.1-point improvement. These results underscore method, providing both accurate and interpretable contributive attributions.
effectiveness as a robust post-hoc attribution
4.3 Fine-grained Module-level Detection and Refinement
We have demonstrated that GENERATION both concurrent and post-hoc attributed generations. Crucially, the accompanying program enables our method to perform contributive extrbbution-explicitlyy explaining why the model generates particular answers. Here, we explore an additional benefit of the program by using it to enhance attribution quality further. Since the program consists of sequences of neural module executions, maintaining faithfulness these executions is critical for accurate attribution. Our modular approach naturally facilitates verifying the correctness of attributions and allows for proactive refinement to ensure output correctness at the module-level. We refer to this as module-level refinement.
achieves strong attribution quality for
To illustrate module-Ievel refinement, we adopt the following experimental setup. We leverage AutoAIS to measure entailment between the generated module outputs and their respective input sentences. If module generates a non-attributable output, we refine it using reranking approach. 2 Specifically, we sample five candidate outputs with temperature of 1.0 and select the first candidate considered entailed by AutoAIS. If candidates receive identical entailment decisions, the first one is chosen by default. For comparison, we apply reranking to ALCE by sampling five entire output candidates and selecting the one achieving the highest Aut score, as ALCE lacks fine-grained entailment checks at the module level. We evaluate performance using attribution precision, recall, F1 scores, and the percentage of module executions considered entailed.
The results, presented in Table clearly illustrate the advantages of GENERATIINPPOGRAMSS First, GENERATIONPROGRR applying reranking to ALCE improves its attribution F1 by 23.7 and 15.3 points on ASQA and LFQA respectively, ALCF's performance still falls short compared GENERATIONPROO EEEERRATIOPRRORRAAO
significantly outperforms ALCE in attribution quality. While
GRAMS without any reranking, highlighting demonstrates notably high baseline attribution Fl and recall scores, indicating strong inherent faithfulness in module execution. Nevertheless, targeted module-level refinements further enhance performance, increasing attribution Fl by 4 points on ASQA and 2.7 points on LFQA. These improvements underscore the substantial utility GENERATION AATTIONROOGAMM'" structured program approach, showcasing practical strategies to further enhance attribution accuracy.
intrinsic adv advantage.
Moreover,
An additional benefit of GENERATIONPROGRANS for refinement is the lower computational cost. Refining modules selectively is significantly more efficient since only small frac- tion requires refinement for instance, only 4.5% of modules in LFQA need refinement.
"W do not leverage self-refinement, prior research has indicated limited effectiveness for self-refinement (Huang al., 2024a).
8

---

Published as a conference paper at COLM 2025
Furthermore, module refinement typically involves just few sentences rather than entire documents, as is necessary for refining with ALCE, thus substantially reducing computa- tional overhead.
Related work
Attributed Generation. Previous research on generating citations in language models primarily integrates citations into the generation process (Nakano et al., 2022; Gao et al., 2023a; Thoppilan et al., 2022; Chen et al., 2023). These methods include training specialized models (Menick et al., 2022; Zhang et al., 2024a) and employing prompting or in-context learning strategies (Gao et al., 2023b). Alternative methods focus post-hoc attribution, adding citations after the generation process is complete (Gao et al., 2023a; Phukan et al., 2024; Qi et al., 2024). Our proposed method uniquely supports both concurrent and post- hoc attribution, explicitly designed to ensure robust attribution quality. Crucially, our approach aligns with the concept of contributive aftribution (Cohen-Wang al., 2024), directly identifying sources that actively influence the generated outputs of the language model. Additionally, Zhang et al. (2024c) explores the use of multiple agents iteratively answer individual chunks of text, aiming to mitigate positional bias- strategy similar to our approach described in Section 244 However, whereas their primary goal is addressing the long-contex issue to enhance answer accuracy, our focus lies chiefly in improving the interpretability of the generation process.
Program-based Generation. Recent improvements in the coding capabilities of language models have sparked significant interest in program-baser generation (Yuan et al., 2024; Yang et al., 2024; Wang et al., 2024b), where complex tasks are systematically decomposed and executed via structured programs. Related research also includes teaching models to utilize external tools (Schick al., 2023; Mialon et al., 2023). the context of text operations, Slobodkin et al. (2024); Ernst et al. (2022) similarly employ two-step approach, involving planning and subsequent execution via fusion module, while Saha al. (2023) introduce "program-then" "hereeettee-exxxcuee" framework specifically tailored for summarization tasks. However, method provides greater generalizability both
our proposed GENERATIONPROGR planning and execution phases by enabling explicit By utilizing Python--xecutabbl programs, our framework supports flexible integration of diverse modules and instructions, thus allowing more customized and adaptable output generation. Additionally, the explicit program traces generated by GENERATIONPROGE enhance interpretability, allowing users to clearly understand model decisions and perform localized verification and edits as needed. We provide further explanations of the differences in Appendix A.2.
to neural modules.
Conclusion and Future Directions
In this work, introduce GENERATION attributed generation process into two distinct stages: program generation and execution via text-based neural modules. Utilizing explicit program traces, GENERATION!ROGRRAN accurately tracks document and sentence-level attributions, achieving robust attribution quality. Experimental results highlight GENERATIONPROGEE
framework that decomposes the
superior performance in both attribution quality and citation accuracy, ensuring generated answers are reliably supported by relevant evidence. We further demonstrate that the GENERATIONPROGRAMG framework effectively handles various applications, such as providing clear post-hoc attri- butions that reveal the underlying rationale behind generated responses and allowing for localized verifications and edits.
As future directions, further enhancements to GENERATIONPROCRAAS could involve in- corporating additional useful text-based modules, such as decontextualizztioo (Choi et al., 2021; Gunjal & Durrett, 2024) or text simplification (Chandrasekar et al., 1996). As discussed in Section 2.2 integrating these new modules into the framework is straightforward, re- quiring only adjustments to the program generation prompts and implementation of new modules. While the current model focuses sentence-level module operations, it can

---

Published as a conference paper at COLM 2025
easily be extended to handle phrase- paragraph-level inputs simply by adjusting the argument gramularity. Attribution benefits can be preserved as long as paragraphs maintain pointers to their constituent sentences. It is important to note, however, that tasks requiring complex reasoning, multi-hop inference, or logical puzzle-solving extend beyond pure text manipulation and are indeed outside the scope of our current framework.
Moreover, leveraging the contributive attribution via our generated program opens interest- ing avenues for application. For instance, as suggested by Cohen-Wang et al. (2024), one can use the generated program to reversely prune contexts to mitigate issues arising from long inputs, or for detecting adversarial modifications intended mislead language models. In particular, it would be valuable to systematically study how best to present program-style explanations to users way that enhances trust and supports efficient claim verification.
Acknowledgements
We thank the anonymous reviewers for their helpful comments. This work was sup- ported by a Google PhD Fellowship, the Microsoft Accelerate Foundation Models Re- search (AFMR) grant program, NSF-CAREER Award 1846185, DARPA ECOLE Program No. HR00112390060, and NSF-AI Engage Institute DRL-2112635. The views contained in this article are those of the authors and not of the funding agency.
References
Steven Bird, Ewan Klein, and Edward Loper. Natural language processing with Python: analyzing text with the natural language toolkit. O'Reilly Media, Inc.", 2009.
Bernd Bohnet, Vinh Q Tran, Pat Verga, Roee Aharoni, Daniel Andor, Livio Baldini Soares, Massimiliano Ciaramita, Jacob Fisenstein, Kuzman Ganchev, Jonathan Herzig, Kai Hui, Tom Kwiatkowski, Ji Ma, Jianmo Ni, Lierni Sestorain Saralegui, Tal Schuster, William W. Cohen, Michael Collins, Dipanjan Das, Donald Metzler, Slav Petrov, and Kellie Webster. Attributed question answering: Evaluation and modeling for attributed large language models, 2023. URL https: httt::://axxw..rr/bbb//////
08037.
R. Chandrasekar, Christine Doran, and B. Srinivas. Motivations and methods for text simplification. In COLING 1996 Volume The 16th International Conference on Computational Linguistics, 1996. URL https: /ttpos://accaaaa /aaaaaaoollno org/ org/C96-2133 2183/.
Anthony Chen, Panupong Pasupat, Sameer Singh, Hongrae Lee, and Kelvin Guu. Purr: Effi- ciently editing language model hallucinations by denoising language model corruptions, 2023. URL https: / /awxxv.iiivv wtwp///.gg///////////o/oo/se 2305. 14908.
Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selected sentence rewriting. In Iryna Gurevych and Yusuke Miyao (eds.), Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 675-686, Melbourne, Australia, July 2018. Association for Computational Linguistics. doi: 10.18653/ (..18633/vl/Bee /1t////l/111///// URL /attps://aclattie hwwpolooyyyoogggg.g.oe org//18--166/
Jianpeng Cheng and Mirella Lapata. Neural summarization by extracting sentences and words. In Katrin Erk and Noah A. Smith (eds.), Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 484-494, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653, /v/ P16-1046. URL. httpp:://allaathol.gg....o wrw/ 1046/.
Eunsol Choi, Jennimari Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. Decontextualization: Making sentences stand-alone. Transactions of the Association for Computational Linguistics, 9:447-461, 2021. doi: 10.1162 1..1162/taclaa000777 to0077 URL
httts::/aaclanthologg..ozz orrr//ooottoooooooor/222111 1ttp:/22221..a.112217.11 tac1-1.27/.
Benjamin Cohen-Wang, Harshay Shah, Kristian Georgiev, and Aleksander Madry. Con-
textcite: Attributing model generation to context. In The Thirly-eighth Annual Conference
10

---

Published as a conference paper at COLM 2025
Neural Information Processing Systems, 2024. URL http:::/ppperrrveee.eee/frrrrrrt 7CM.Sqs2Jt.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, Dasha Metropolitansky, Robert Osazuwa Ness, and Jonathan Larson. From local to global: A graph rag approach to query-focused summarization, 2025. URL https:
wwwrxiv.orggabb//// www.: 16132
Günes Erkan and Dragomir R. Radev. Lexrank: graph-based lexical centrality salience in text summarization. J. Artif. Int. Res., 22(1):457-47, December 2004. ISSN 1076-9757.
Ori Ernst, Avi Caciularu, Ori Shapira, Ramakanth Pasunuru, Mohit Bansal, Jacob Gold- berger, and Ido Dagan. Proposition-level clustering for multi-document summarization. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 1765-1779, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/ v1/ URL https: http:://aaaaaaahloaoooo org/2022 maacl-main. 128/
Ori Ernst, Ori Shapira, Aviv Slobodkin, Sharon Adar, Mohit Bansal, Jacob Goldberger, Ran Levy, and Ido Dagan. The power of summary-soure alignments. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 6527-6548, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v /////v12222/// fe224.finiing--cle ac1.389. URL https: //aclanthology.orge wraplooyy.org//// org/2024 1ere22.ffinne findings-acl. 389/.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-10844 Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19-1100 URL attps://aclanthot
htggPP9-1102..
Luyu Gao, Zhuyun Dai, Panupong Pasupat, Anthony Chen, Arun Tejasvi Chaganty, Yicheng Fan, Vincent Zhao, Ni Lao, Hongrae Lee, Da-Cheng Juan, and Kelvin Guu. RARR: Researching and revising what language models say, using language models. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 16477- 16508, Toronto, Canada, July 2023a. Association for Computational Linguistics. doi: 10.18653/ 101/6533v1/2000 a0..2222.cl--oong.900 URL https://aclanthologg org/2023 . acl long. 910/.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models to generate text with citations. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp- 6465-6488, Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653 wwwp2/2..mmlpp..... main.........9... URL httpss:/aclantholooo. org/ 2mmpmmmm12nppppp.....emmmmm main. 398/
Gemini. Gemini deep research, 2025. deep-resseacch/2h11ee
URL http::://gemini.goollel.ee/
Gemini Team et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL httpss://artit wwwporr// .2243..5555..
Anisha Gunjal and Greg Durrett. Molecular facts: Desiderata for decontextualiiztion in LLM fact verification. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Compulational Linguistics: EMNLP 2024, pp. 3751-3768, Miami, Florida, USA, November 2024. Association for Computational I Linguistics. doi: 10. (0653/v1/2000 findings-emnlp.22 215. URL https: https://accanttholog .oae:/gggoorr/gg////e 2rrg2224, http://22...............
215/.
Som Gupta and S. K Gupta. Abstractive summarization: An overview of the state of the art Expert Syst. Appl., 121(C):49-65, May 2019. ISSN 0957-4174. doi: 10.1016 10.1016/j.ewwa.211.. 2018..
011. URL https: /tttss://do...oo http://o...oog/////////////////e 1016 eswa. 2018. 12.
11

---

Published as a conference paper at COLM 2025
Junxian He, Wojeiech Kryscinski, Bryan McCann, Nazneen Rajani, and Caiming Xiong. Towards generic controllable text summarization. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 5879-5915, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/ /v1555/////v 1111665/1122222emmllp// 222/222emmllmmain..999. URL https://aclaantoooogy org.:gg.ooo.oooooooo 2222. emm.p-main. 396/.
CTRLsum:
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. spaCy: Industrial-strength Natural Language Processing in Python, 2020.
Or Honovich, Roee Aharoni, Jonathan Herzig, Hagai Taitelbaum, Doron Kukliansy, Vered Cohen, Thomas Scialom, Idan Szpektor, Avinatan Hassidim, and Yossi Ma- tias. TRUE: Re-evaluating factual consistency evaluation. In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistios: Human Language Technologies, pp. 3905-3920, Seattle, United States, July 2022. Asso- ciation for Computational Linguistics. doi: (118653/ww12222naaccc/ https://aclanthol.ge orgooo//// naacl-maien mm
URL
Jie Huang, Xinyun Chen, Swaroop Mishra, Huaixiu Steven Zheng, Adams Wei Yu, Xinying Song, and Denny Zhou Large language models cannot self-correct reasoning yet. In The Twelfth International Conference on Learning Representations, 2024a. URL https:// openreview.ro www..nee/oormmmdddddoooffffo
Kung-Hsiang Huang, Philippe Laban, Alexander Fabbri, Prafulla Kumar Choubey, Shafiq Joty, Caiming Xiong, and Chien-Sheng Wu. Embrace divergence for richer insights: A multi-document summarization benchmark and case study on summarizing diverse information from news articles. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), Pp. 570- 593, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10. 18653 ///2024.naacl-logg
URL https: / a http///aaaaaaanaac anthology org/ 2024..24..2 maacl-long 32/.
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Comput. Sure., 55(12), March 2023. ISSN 0360-0300. doi: 10.1145/3571730. URL https: org/1e é 1145/3571730.
Huiqiang Jiang, Qianhui Wu, Chin-Yew Lin, Yuqing Yang, and Lili Qiu. LLMLingua: Compressing prompts for accelerated inference of large language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 13358-13376, Singapore, December 2023, Association for Computational Linguistics. doi: (0.18653/v1// /20./...mnnmmmmaiii. https://aalannthologg/ wwwphholllgg...g//222222 emnlp-main. 825/.
URL
Hongyan Jing and Kathleen R. McKeown. The decomposition human-written summary sentences. In Proceedings of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '99, pp. 129-136, New York, NY, USA, 1999, Association for Computing Machinery. ISBN 1581130961. doi: 10.1145 0..1145/31222431266
URL https: org/10 1145/31262..1122
Hongyan Jing and Kathleen R. McKeown. Cut and paste based text summarization. In 1st Meeting of the North American Chapter of the Association for Computational Linguistics, 2000.
URL https: http:::s//aaaaahhhllllooo
org/A00-20 200000
Philippe Laban, Chien-Sheng Wu, Lidiya Murakhovs'ka, ka, Xiang Chen, and Caiming Xiong. Discord questions: A computational approach to diversity analysis in news coverage. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Findings of the Association for Computational Linguistics: EMNLP 2022, pp. 5180-5194, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10.18653/v1. findings-emnll 380/.
2022. 1022.iindings-emnlp. URL htttps://ccaatthology/
12

---

Published as a conference paper at COLM 2025
Philipppe Laban, Alexander Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of haystack: A challenge to long-context LLMs and RAG systems. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 9885-9903, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ /22/////22...mmpmmaaail URL httph::t///ett hetpss//acaaaatthhcec
org 2224. emnlp-main. 552.
Logan Lebanoff, Kaiqiang Song, Franck Dernoncourt, Doo Soon Kim, Seokhwan Kim, Walter Chang, and Fei Liu. Scoring sentence singletons and pairs for abstractive summarization. In Anna Korhonen, David Traum, and Lluís Mårquez (eds.), Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 2175-2189, Florence, Italy, July 2019. Association for Computational Linguistics. doi: 10.18653 (00665/v1/PPP /v/9/111PPPP URL https://aclantiooogg..rrr
Logan Lebanoff, John Muchovej, Franck Dernoncourt, Doo Soon Kim, Lidan Wang, Wal- ter Chang, and Fei Liu. Understanding points of correspondence between sentences for abstractive summarization. In Shruti Rijhwani, Jiangming Liu, Yizhong Wang, and Rotem Dror (eds.), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 191-198, Online, July 2020. Association for Computational Linguistics. doi: https://aclantthologs. org. 2020 acl-srw.
h1///v11/// /tt/://1/220aalsss.22222 URL
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensivg nlp tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems,, volume 33, pp. 9459-9474. Curran Associates, Inc., 2020. URL https::/pproceedings... neurips. cc/paper Files/paper/221
Chin-Yew Lin. ROUGE: A package for automatic evaluation evaluatino of summaries. In Text Summa- rization Branches Out, pp. 74-81, Barcelona, Spain, July 2004. Association for Computa- tional Linguistics. URL
Nelson Liu, Tianyi Zhang, and Percy Liang. Evaluating verifiability in generative search engines. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 7001-7025, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v1 10186553/11222..miii http:///.2.....ininsseeeee candinng--enll.444e 467. URL https://acllanthllegs org/ 2223. findings-emnlp 467/.
Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Compulational Linguistics, 12:157-173, 2024a. doi: 10.1162/tacl. a.0088
URL https: http://ccaaaathhooo .eall.org/220 org/2024. tacl-1.9/.
Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. G- eval: NLG evaluation using gpt-4 with better human alignment. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods Singapore, December 2023b. Association for Computational Linguistics. doi: 10.18653/ 10./8633/11/2// 1///2/22..mmmmmami....... //aclanthology .wwplgg...rrrg/// org/2023, http2222...mmmmmmmmmmmmmmm emnlp-main. 153/
in Natural Language Processing, pp.
URL https:
Yixin Liu, Alexander Fabbri, Jiawen Chen, Yilun Zhao, Simeng Han, Shafiq Joty, Pengfei Liu, Dragomir Radev, Chien-Sheng Wu, and Arman Cohan. Benchmarking generation and evaluation capabilities of large language models for instruction controllable summariza- tion. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 4481-4501, Mexico City, Mexico, June 2024b. Association for Computational Linguistics. doi: 10.18653/v 20/33/11/22// /2024.222f22 //2/2/2.inninns-nnnnn findings-nnacl.2800 URL httttp::/aalaanthllleo wrcpllooyyoorg//// wtt..////22..fnninggss..... findings-naach n20...
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Gillpell, Geoffrey Irving,
13

---

Published as a conference paper at COLM 2025
and Nat McAleese. Teaching language models to support answers with verified quotes, 2022. URL httpps://arxiii http::///axxi..r//bbb/////// wwt.oos///////// 2203. 11147.
Grégoire Mialon, Roberto Dessi, Maria Lomeli, Christoforos Nalmpantis, Ramakanth Pa- sunuru, Roberta Raileanu, Baptiste Roziere, Timo Schick, Jane Dwivedi-Yu, Asli Ce- likyilmaz, Edouard Grave, Yann LeCun, and Thomas Scialom. Augmented language models: survey. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL Survey Certification.
Sewon Min, Kalpesh Krishna, Xinxi Lyu, Mike Lewis, Wen-tau Yih, Pang Koh, Mohit lyyer, Luke Zettlemoyer, and Hannaneh Hajishirzi. FActScore: Fine-grained atomic evaluation of factual precision in long form text generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, Pp. 12076-12109 Singapore, December 2023. Association for Computational Linguistics. doi: 10. //aclanthology, weceea..rrg///// wwgg/222.emmlpmmain 741/.
URL https:
Reichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. Webgpt: Browser-assisted question-answering with human feedback, 2022. URL https: httt:://arxiv.orrg/bb///1111
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang.
RNNs and beyond. In Stefan Riezler and Yoav Goldberg (eds.), Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning, Pp. 280-290, Berlin, Germany, August 2016. Association for Computational Linguistics. doi: 10.18653/v1/KKL 111///111KK111222/ URL https: //aclanthology.co org.oogy.ooo.oogoooooo oo..
Abstractive text summarization using sequence-to-sequens
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Ranking sentences for extractive summarization with reinforcement learning. In Marilyn Walker, Heng Ji, and Amanda the 2018 Conference of the North American Chapter of the Association
Stent (eds.), Proceedings the for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1747-1759, New Orleans, Louisiana, June 2018. Association for Computational Linguistics. doi: (0.18633/11/N88-1
URL https: htt////aaaaaanlaaaaa/aaa org/N18 8- 1158/.
Ani Nenkova and Kathleen McKeown. Automatic summarization. Foundations and Trends@ in Information Retrieval, 5(2-3))103 233, 2011. ISSN 1554-0669. doi: 10.1561/1500000015
URL http:://x.doi..oo.// www.doi...ggg/// 1561
Jianmo Ni, Chen Qu, Jing Lu, Zhuyun Dai, Gustavo Hernandez Abrego, Ji Ma, Vin- cent Zhao, Yi Luan, Keith Hall, Ming-Wei Chang, and Yinfei Yang. Large dual en- coders are generalizable retrievers. Zhang (eds.), Proceedings of the 2022 Conference Empirical Methods in Natural Language Processing, pp. 9844-9855, Abu Dhabi, United Arab Emirates, December 2022. Asso- ciation for Computational Linguistics. doi: 00.1663/v112022eemml---- main... URL https://aclanttholog org 2022 emnlp-main.665 669/.
Yoav Goldberg, Zornitsa Kozareva, and Yue
OpenAl. Hello gpt-4o, 2024. URL
OpenAl. Introducing deep research, 2025.
URL https:///opeai..oomnnnnee
introducing-deep--eeeaarr
OpenAl al. Gpt-4 technical report, 2024. URL
08774..
Paul Over and James Yen. An introduction duc-2004. National Institute of Standards and Technology, 2004.
Anirudh Phukan, Shwetha Somasundaram, Apoorv Saxena, Koustava Goswami, and Balaji Vasan Srinivasan. Peering into the mind of language models: An approach for attribution in contextual question answering. In Lun-Wei Ku, Andre Martins, and Vivek
14

---

Published as a conference paper at COLM 2025
Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 11481-11495, Bangkok, Thailand, August 2024. Association for Computational Linguis- tics. doi: 10.18653/ w1/2222.fiinaata ain.6.gs-acc.662. URL https://aclanthoolgyy org/2024.
findings-acl. 682/.
Jirui Qi, Gabriele Sarti, Raquel Fernández, and Arianna Bisazza. Model internals-based answer attribution for trustworthy retrieval-augmented generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, PP. 6037-6053, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 101533/112200 ww2//22.mm/lpmmain..
URL https: ptttp:://aclanthol..gg./ org/ 2024. www.l-mmain...m...mmmmm...mm-mam 347/.
Hannah Rashkin, Vitaly Nikolaev, Matthew Lamm, Lora Aroyo, Michael Collins, Dipanjan Das, Slav Petrov, Gaurav Singh Tomar, Iulia Turc, and David Reitter. Measuring attribution in natural language generation models. Computational Linguistics, 49(4):77778k) December 2023. doi: 10.1162/ _ _00466.
URL httphhtt::////tt htttp:://aclatthii hogg./oooo.....oooooooooooo..... org/ 2023 00023.cl-4... c1-4.2/.
Nils Reimers and Iryna Gurevych. Sentence- BERe:: Sentence embeddings using Siamese BERT-networks. In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982- 3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 11111666//1/11114/// URL https: / http:////ccaaatooooooorrr/////// 1410//
Swarnadeep Saha, Shiyue Zhang, Peter Hase, and Mohit Bansal. Summarization programs: Interpretable abstractive summarization with neural modular trees. In The Fleventh International Conference on Learning Representations, 2023. URL forum?1d=oox000772tB
net//
Timo Schick, Jane Dwivedi-Yu, Roberto Dessi, Roberta Raileanu, Maria Lomeli, Eric Hambro, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Tool- former: Language models can teach themselves to use tools. In A. Oh, Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information Processing Systems, volume 36, pp. 68539-68551. Curran Asso- ciates, Inc., 2023. URL httpss://proceeedingg neurii
pdf.
Aviv Slobodkin, Eran Hirsch, Arie Cattan, Tal Schuster, and Ido Dagan. Attribute first, then generate: Locally-attributable grounded text generation. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Pp. 3309-3344 Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v 1111663/vv/2224.caaaa 2022....c..... 182. URL https: http::/ss//claathhoooooooo or 2024 acl long. 182/.
Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. ASQA: tions meet long-form answers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.), Proceedings of the 2022 Conference on Empirical Methods in Natural Language Pro- cessing, Pp. 8273-8288, Abu Dhabi, United Arab Emirates, December 2022. Associa- tion for Computational Linguistics. doi: wwwne-nnnaaeeeeeeeee main. 566/.
Factoid ques-
URL
https://aclantthooogy.
Liyan Tang, Philippe Laban, and Greg Durrett. Minicheck: Efficient fact-checking of llms on grounding documents. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2024. URL hodf/22 2204 10774.
https://arxiv.. org/pdf
Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern
15

---

Published as a conference paper at COLM 2025
Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. Lamda: Language models for dialog applications, 2022. URL httpss//arxivv org/abs/ 2201. 08239.
Jesse Vig, Alexander Fabbri, Wojciech Kryscinski, Chien-Sheng Wu, and Wenhao Liu. Ex- ploring neural models for query-focused summarization. In Marine Carpuat, Marie- Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, pp. 1455-1468, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/ /v1 /22.2..fnningssnaaaaaaa findings-naacl.109. URL https://aclanntholgg.. wwwpllogy.oog//// org/2022 findings- naacl. 109/.
David Wan, Jesse Vig, Mohit Bansal, and Shafiq Joty. On positional bias of faithfulness for long-form summarization. In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Associa- fion for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 8791-8810, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-89176-1894 doi: https://aclanthhooog wrwpehhhooooy..o//2//22 naacl-long,,. 442/.
442. URL
Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. Searching for best practices in retrieval-augmented generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17716-17736, Miami, Florida, USA, November 2024a. Association for Computational Linguistics. doi: 10.18653/ w1/2024emmlppmmiin.999 URL https: /tatt:://aaaatthiit tntoology 20244024. emnlp- 981/
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better LLM agents. In Forty-first International Conference Machine Learning, 2024b. URL /tttp:://operrevieet net net:ForummiddjJJ9oXk.
Theodora Worledge, Judy Hanwen Shen, Nicole Meister, Caleb Winston, and Carlos Guestrin. Unifying corroborative and contributive attributions in large language models, 2023. URL https:://arxe http://aaaiii..or/////231111111 12233.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. RECOMP: Improving retrieval-augmented LMs with context compression and selective augmentation. In The Twelfth International Conference on Learning Representations, 2024. URL http:////oeeeeeiw.eeteeeeee m1JLVigNHp.
Ke Yang, Jiateng Liu, John Wu, Chaoqi Yang, Yi Fung, Sha Li, Zixuan Huang, Xu Cao, Xingyao Wang, Heng Ji, and ChengXiang Zhai. If LLM is the wizard, then code is the wand: A survey on how code empowers large language models to serve as intelligent agents. In ICLR 2024 Workshop on Large Language Model (LLM) Agents, 2024. URL https: //openreview.net http:/iwwwwweeeeeeeee. www..e/eooorummidddddnooooihi
Lifan Yuan, Yangyi Chen, Xingyao Wang, Yi Fung, Hao Peng, and Heng Ji. CRAFT: Customizing LLMss by creating and retrieving from specialized toolsets. In The Twelfth International Conference on Learning Representations, 2024. URL https: /ttps://opeee http:///ppeereeeeeeeee/e net/ forum?id=G@vdDstt.MM
Haopeng Zhang, Xiao Liu, and Jiawei Zhang. Extractive summarization via ChatGIT for faithful summary generation. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 3270-3278, Singapore, December 2023a. Association for Computational Linguistics. doi: 10.18653/v 2023. 2023. 1D23.indinggsemmlp.211 URL
wttpo///.....n.iieeeeee..eo findings-emnlp 214/
16

---

Published as a conference paper at COLM 2025
Jiajie Zhang, Yushi Bai, Xin Lv, Wanjun Gu, Danqing Liu, Minhao Zou, Shulin Cao, Lei Hou, Yuxiao Dong, Ling Feng, and Juanzi Li. Longcite: Enabling iims to generate fine-grained citations in long-context qa, 2024a. URL https://arxtv.orr//bb/t hetss///9...00 2409902897.
Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tat- sunori B. Hashimoto. Benchmarking large language models for news summariza- tion. Transactions of the Association for Computational Linguistics, 12:39-57, 2024b. doi: to.1162/taacla.ne 000322. URL https://accaanthology/ wwwpaanntwwooooooooo//.eo// org/2024. tacl-1.3/.
Yusen Zhang, Yang Liu, Ziyi Yang, Yuwei Fang, Yulong Chen, Dragomir Radev, Chenguang Zhu, Michael Zeng, and Rui Zhang. MACSum: Controllable summarization with mixed attributes. Transactions of the Association for Computational Linguistics, 11:787-808 2023b. doi: 10.1162/tacl..0055 URL
.rr.ea...rr////. org 2023. tac1-1.46/.
Yusen Zhang, Ruoxi Sun, Yanfei Chen, Tomas Pfister, Rui Zhang, and Sercan Ô. Arik. Chain of agents: Large language models collaborating on long-contex: tasks, 2024c. URL https://arxivv www.oogg///......oooo 2406 02818.
Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Has- san Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. QM- Sum: A new benchmark for query-based multi-domain meeting summarization. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Beltagy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao Zhou (eds.), Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 5905-5921, Online, June 2021. As- sociation for Computational Linguistics. doi: 10. / v1 1a121..nacl-maal https://aclanthology www.thholllggyo.g/222211 org/2021 naacl-mati 472/.
URL
A Additional Discussions
A.1 Summarization, RAG, and Their Connections
We describe the tasks where applying GENEAATIONPROCRANG can improve attributions. To do so, we first introduce the formulations of summarization and retrieval-augmented generation (RAG) tasks individually, highlighting their similarities. To illustrate their connection, we intentionally overload some notation common to both tasks.
Summarization. Summarization typically involves taking an input document D and gen- erating an abbreviated and condensed representation, a summary o. Recent work (Huang et al., 2024b; Laban et al., 2024) has also focused more challenging variants, such as multi-document summarization (Over Yen, 2004, MDS), where the input comprises multiple documents. Compared to single-document summarization, MDS requires syn- thesizing content across multiple sources to handle redundant information (Fabbri et al., 2019), and highlighting contrasting or complementrry viewpoints (Laban et al., 2022; Huang et al., 2024b). Additionally, summarization tasks have evolved to include query-focused or instruction controllable summarization (Zhong et al., 2021; He et al., 2022; Vig et al., 2022; Zhang et al., 2023b; Liu et al., 2024b), In these tasks, a query or instruction & provides guidance for the aspects of the document(s) to be summarized.
RAG. Retrieval-augmented generation (RAG) (Lewis et al., 2020) is a pipeline framework for rext-generation tasks, primarily used here for question-answering Rather than solely relying on language model's parametric knowledge, which can contain inaccuracies (Min et al., 2023; Gao et al., 2023a), RAG incorporates a retrieval component that first retrieves relevant documents D, and then generates answers on these retrieved sources. Both the retrieval process and the subsequent known the "retrieve-then--eenraa pipeline.
geneee generation are conditioned on a query q. This is also
17

---

Published as a conference paper at COLM 2025
Dataset
Retrieved sources
Avg Source Tokens
Num Examples
Avg Num Sent
ASQA
Passages Documents
1,035
948
3
LFQA
9,116
45
2
Table 4: Dataset statistics.
Similarities between RAG and Summarization. RAG-based question answering shares structural similarities with multi-document, instruction-condiiinee summarization. In RAG, retrieved documents play the role of input documents, and the guiding query helps select document segments relevant answering the user's question. Both tasks fundamentally require extracting and synthesizing information from set of documents, emphasizing their conceptual alignment. In fact, RAG has been applied effectively to problems like -a--aar--o---hhssaaee (Laban et al., 2024), and conversely, RAG tasks have been framed explicitly as summarization (Edge et al., 2025).
Use of Summarization in RAG. In addition to the traditional summarization and RAG settings, we also explore the use of summarization in RAG as a critical post-processing step for retrieved sources (Gao et al., 2023b; Jiang et al., 2023; Xu et al., 2024; Wang et al., 2024a) to focus on filtering out irrelevant content that otherwise impedes accurate generation and citation. That is, for each document D;, we perform query-focuser summarization to summarize the important information regarding 9. While previous studies (e.g,, Gao et al. (2023b); Xu et al. (2024)) have primarily employed summarization to address context window constraints, we argue that summarization should also be seen as an essential extraction step, to further ease attribution.
A.2 Comparison of GENERATIONPROGRANG with Baseline Methods.
The most critical distinction lies in our use of optional natural language instructions within each program step, allowing the planner to communicate intent to individual modules. This design enables greater flexibility, generality, and interpretability in reasoning. We illustrate its benefits in Figure 4: although argument-based programs (e.g., sentence IDs) may indicate relevance, they often obscure the precise peasoning-especcillly when the same source (e.g., S3) supports multiple sentences in different ways. For instance, the first sentence may draw on the "starting points," " while the second depends on length and endpoints. By incorporating explicit instructions, users can more easily verify and understand how content is generated. In comparison:
Attribute-First (Slobodkin et al., 2024) is limited to single fusion operation between two sentences. In contrast, our framework supports a more general set of operations, including paraphrasing, extraction, and compression, enabling richer composition and modularity.
Summarization Program (Saha et al., 2023) employs multiple modules, but requires fine- tuning on synthetic program-labeled data-a process involving complex preprocessing to identify valid programs. In contrast, our method leverages the powerful zero-shot capabilities of large language models for instruction-based neural modules. Our approach enables modules to accept explicit instructions tailored to specific tasks. This flexibility empowers the planner not only select appropriate modules but also to precisely dictate how each module should operate, facilitating the creation of more complex and context- sensitive program plans. For example, during multi-document summarization, the fusion module can be explicitly instructed to emphasize either commonalities across documents (Fabbri et al., 2019) or contrasting viewpoints (Huang et al., 2024b). Such detailed control significantly enhances the planner's 's capability, allowing it to construct sophisticated and highly targeted generation strategies, ultimately improving the precision, adaptability, and interpretability of the outputs.
18

---

Published as a conference paper at COLM 2025
ASQA Attribution (AutoAIS)
LFQA Attribution (CPT-4o)
Correct
Correct
Method
EM
Attr Prec.
Rec.
Altr.
RL
Altr
Rec.
ALCE
44.0
67.3
63.2
62.7
25.8
39.4
88.2
46.8
35.4
40.0
GENPROG
41.0
85.1
91.6
87.1
0.0
32.3
99.2
91.1
94.4
0.0
52.3
ext ALCE
43.0
69.1
67.9
66.4
20.8
42.0
88.7
49.7
38.3
29.3
extabb ALCE
44.0
65.4
66.9
64.0
18.4
40.2
87.0
45,9
35.7
31.3
77.2
0.0
ext GENPROG
39.3
86.1
90.4
87.2
0.0
38.4
98.0
82.1
87.7
0.0
ext-abs GENPROG
39.6
82.3
85.7
62.8
0.0
39.2
97.8
83.1
88.1
0.0
Table 5: Attribution results on long-form QA with document-level citations.
B Experimental Setup Details
Here we provide additional experimental details. Sentence splitting is performed using Spacy (Honnibal et al., 2020). For GPT-4o, we use the version 2024-05-13 with temperature set to O.
Dataset Details. Dataset statistics are summarized in Table 4. We utilize the test sets for both datasets our experiments.
Relevance Filtering. For ASQA, where some retrieved documents may not be relevant, prompt GPTT-o to assess document relevance with respect to the query. Documents deemed irrelevant are excluded from further processing.
Summarization. We benchmark several summarization techniques: abstractive (Nallapati et al., 2016; Gupta & Gupta, 2019), extractive (Cheng & Lapata, 2016), and extract-then- generate (Chen & Bansal, 2018; Zhang et al., 2023a). The extract-then-generat method first applies extractive summarization, subsequently refining the output with an abstractive module.
In-Context Examples. We provide the distinction of how in-context examples are used. ALCE examples consist of direct question-answer pairs, explicitly demonstrating the ex- pected final output format and style. GENERATIONPROGRAN provide valid programs. The model learns to generate process that, when executed, yields the answer.
examples, in contrast,
Ensuring Program Validity. We explicitly validate the generated programs by checking that all module names belong to a predefined set of supported functions and by verifying the correctness of their arguments, ensuring safe execution. The use of AST parsing is limited to converting program strings into method-name and argument tuples; all semantic validations-suuc method existence and argument correctness-are handled separately.
C
Additional Results
C.1 Full Attribution Results.
Complete attribution results, including attribution precision, recall, and evaluations for all summarization methods, are provided in Table 5. Comparing the various summarization techniques used for post-processing retrieved sources, we observe that extractive sum- marization and extract-then-generaa Specifically, extractive summarization improves citation F1 scores by 3.8 and 2.9 points compared to no summarization for ASQA and LFQA, respectively. This finding contrasts with previous results by Gao et al. (2023b), who reported no improvements in citation quality with summarization methods. Surprisingly, abstractive summarization does not enhance correctness.
achieve the highest citation Fl scores on average.
19

---

Published as a conference paper at COLM 2025
ASQA
LFQA
Average
RL
MCS
RL
MCS
RL
MCS
Abstractive
27.4
92.5
37.2
89.6
32.3
91.1
Extractive
24.1
92.0
32.0
97.8
28.1
94.9
Extract-then-generate
25.3
94.5
35.0
95.1
30.1
94.8
Table 6: Intrinsic evaluation of summaries
ASQA
LFQA
Method
Attr. Prec.
Attr. Rec.
Attr: F1
"sEntail..
Attr. Prec.
Attr. Rec.
Attr.
%dlinail..
ALCE
55,3
59.0
54.2
58.2
55.6
58.0
ALCE reranking
GeNerATiONTROcRAMS
76.8
85.3
79.4
85.8
76.7
93.7
82.8
95.5
GENPROG module-level reranking
Table 7: Full fine-grained refinement results. In addition attribution F1, we also show the percentages of the modules that are considered entailed for GENERATION
ASQA
LFQA Attr. F1
Correctness
Attr. F1
% Entail.
Correctness
% Entail.
Document-level Attributions
ALCE
51.8
72.0
28.4
25.6
62.1
24.7
ALCE extr.
48.7
72.0
29.8
34.3
70.6
21.0
CENPROG
46.1
80.6
0.0
24.8
91.3
0.0
CENPROG extr. summ.
45.0
82.7
0.0
27.2
89.4
0.0
Sentence- level Attributions
ALCE ALCE extr. summ.
51.0
56.2
33.1
32.5
52.4
58.5
47.4
54.3
38.5
36.3
57.4
37.1
GEN PROG
46.1
68.9
0.0
24.8
86.3
0.0
GENPROG + extractive summarization
45.0
62.5
0.0
27.2
80.2
0.0
Table 8: Results with Llama 3.3 70B.
C.2 Intrinsic results of summarization
To better understand the effect of summarization retrieved sources, we conduct intrinsic evaluations of the summaries. We calculate ROUGE-L scores between generated summaries and gold-standard answers, and assess summary faithfulness relative to their source docu- ments. Faithfulness is measured individually for each document using MiniCheck (Tang et al., 2024), state-of-the-art method for evaluating summary faithfulness. Results presented in Table 6. Consistent with prior findings (Zhang et al., 2023a; Xu et al., 2024), our results indicate a clear trend: extractive summarization achieves the highest average faithfulness, abstractive summarization yields the highest average ROUGE-L scores, and extract-then-geet generrtee methods offer effective compromise by improving ROUGE-L scores relative to purely extractive methods while maintaining similarly high faithfulness.
C.3
Full NLI Result
The complete set NLI evaluation results, including attribution precision and recall, is presented in Table 7. In addition to our main findings in Section 4.3, we note that GENERATIONPROGRATE exhibits significantly higher attribution recall compared to ALCE- based methods.
C.4 Results with Open-Source Model
We report the results with open-source model-Llama 70B-in Table 8. These findings exhibit the same trends as our primary experiments conducted with GPT-4o, described in Section 4.1. Specifically, observe that: (1) extractive summarization effectively enhances
20

---

Published as a conference paper at COLM 2025
Time(s)
Attr. F1
ALCE ALCE refinement
3.7
54.0
30.0
69.3
GENPROG
12.7
82.8
GENPROG module-level refinement
14.5
85.5
Table 9: Latency results for refinement on LFQA.
accuracy, indicating that open-source models are similarly influenced by long-context input; (2) GENERATIONPROGRR MS significantly improves attribution quality; and (3) combining extractive summarization with GENERATIONPROGRANE balances accuracy and attribution quality effectively. When evaluated with GPT-4o-based metrics, CENERATIONPROGRRR demonstrates consistent performance gains, suggesting that the strong performance of GENERATIONPROGRAMS is not due to self-evaluation bias (since here we are evaluating Llama3 outputs with GPT4o evaluator).
C.5 Latency
We measured the per-example runtime of both methods before and after refinement in Ta- ble 9. We found that refinement significantly increases the runtime for ALCE (approximately 8x the original runtime), as it requires generating outputs five times and performing NLI validation across all examples for reranking. In contrast, although GENERATIONPROGR.N initially takes longer due to its two-step generation-exxcution approach, its structured design enables highly efficient, module-level NLI checking. Computationally-intenniii erations are thus limited only to incorrect cases, with refinement adding just 1.8 seconds per example. This supports our claim that GENERATIONPROGRAM facilitates computationally efficient refinement.
op-
C.6 Additional Evaluations on Correctness.
LLM-based Correctness Evaluation. We employ an LLM-based correctness metric intro- duced by Zhang et al. (2024a), shown to correlate highly with human judgments. Specif- ically, we compute average correctness scores for ALCE and GENERATION the LFQA sentence-level setting, where the two methods exhibited the largest discrepancy. While Rouge-L, the standard metric, yielded scores of 39.9 for ALCE and 32.3 for GENERA- TONNROGRRMM, the more nuanced LLM-based correctness metric produced much closer results- and 86.7, respectively (we found no statistically significant difference between these two correctness scores (p=0.6), indicating that the methods perform similarly in this regard). This suggests that when evaluated using a more robust and human-correelted metric, the difference between the two methods is negligible.
in
Human Evaluation. We have conducted a separate human annotation for correctness. Specifically, we asked two native English-speaking non-authors without prior knowledge of the paper to evaluate whether the generated output conveyed the same information as the reference answer. We randomly sampled 25 examples, taking both ALCE and CENERATIONPROORRNS outputs on LFQA, where automatic metrics showed the largest gap. Following a human annotation methodology similar to that used by the authors of ALCE, annotators were asked to rate the similarity of the outputs a scale of 1 to 5. We found that the average score for ALCE was 3.8, while GENERATIONPROCRA actually produces
achieved
an average score of 4.3 This suggests that GENERATIONPROGRR.. statistically significantly (p=0.03) more correct answers according to this human evaluation.
The two evaluations show that GENERATIONPROGER ness, Nonetheless, to maintain comparability with past work, we continue to report EM and R in the main table.
actually does not decrease correct-
21

---

Published as a conference paper at COLM 2025
Module Distribution
% Entailment
Dataset
fusion
compression
paraphrase
fusion
compression
paraphrase
LFQA
76.8
16.4
6.8
96.3
91.7
93.1
ASQA
38.3
53.0
8.7
90.4
85.0
82.8
Table 10: Program statistics, including the distribution of the used modules, and the per- centage of entailment of all the outputs for each module.
C.7 Evaluation on Coherence and Fluency
Each sentence is generated independently, and we do not include an explicit contextual- ization step across sentences. To assess the coherence and fluency of our method despite this limitation, we utilized LLM-based evaluators the LFQA dataset. Specifically, we employed the G-Eval (Liu et al., 2023b) prompt, which has been shown to correlate well with human judgments for evaluating text quality. We observed fluency scores of 1.3 (on 1-3 scale) for both ALCE and PENNERATIONPROGR.. 1-5 scale) for both methods. These results suggest that, in practice, there is no significant difference in sentence-level coherence or fluency between the two approaches.
and coherence scores of 3.0 (on
C.8 Correlations of Attribution Metric with Human Judgment
We use LLM as judge metrics to evaluate attribution, performing binary classification to determine whether facts in the output are supported or not. To validate this metric, we collected 25 examples of attributions identified correct by our metric and 25 identified as incorrect. These 50 examples were then randomly shuffled and presented to two native English speaker non-authors without prior knowledge of the paper. Following setup similar to Zhang et al. (2024a), these evaluators made binary judgments as to whether the provided statements were entailed by the cited document, treating "partially supported" as "not supported". The accuracy of the metric's s predictions when compared to these human- labeled instances was 78.1% (random is 50%), indicating that the metric is high-quality.
D Additional Analyses
D.1 Program Statistics
Program statistics are summarized in Table 10. We observe distinct distributions of module usage across the two datasets: GENERATIONPROGRAMN primarily employs the fusion module on LFQA, whereas ASQA requires compression more frequently. This highlights the adaptability of the LLM in dynamically selecting appropriate programs tailored to dataset-specifi needs. Additionally, exhibit high attribution quality. Specifically, for LFQA, each module achieves an entailment rate above 90%, while for ASQA, each module maintains entailment rate exceeding 80%. These results underscore the robustness and reliability of the generated program structure.
discussed in Section 4.3, modules consistently
Invalid Trees. Invalid trees are extremely rare: we find that the GPT-4o produces only one invalid program tree for ASQA and LFQA. In one case, the error stems from missing second argument for the compression function. the other case, the program fails to close with final parenthesis, likely due to exceeding the maximum allowed token limit.
D.2 Qualitative Examples
We present examples of ALCE and GENERATIONPROGRAMN outputs on the LFQA dataset in Figure 3. Additionally, we demonstrate post-hoc attribution using GENERATIONPROGRAMM in Figure 4, where we also illustrate the regeneration of the original output.
22

---

Published as a conference paper at COLM 2025
Question: Should religious education be taught public schools?
Document:
54: Exposing students to texts from different religions be helpful to student learning when shared in context for the development and advancement of societies.
S5: For example, reading from the Bhagavad Gita (Hindu text) can be used to guide discussion on the historical time in which it was written, or an analysis of the language used.
S6: Additionally, students may be encouraged to examine the correlation between religion in school and advancement of sciences.
S7: For instance, students can research how mathematical concepts such calculus and trigonometry came into practice with the advancement of Islam. S8: Religious practices can be explored for historical contexts and further understanding for the development of such practices.
S12: Another way that teachers can incorporate religion in the classroom within their practice is by examining different cultures, their perspectives, and sources of motivation. $15: When students are given the opportunity to explore diverse cultures and evaluate the causes that have led to conflicts, they are more likely to appreciate the viewpoints of others, exercise empathy and apply critical thinking skills.
S16: You can help students apply these skills when they are faced with problems that require brainstorming for a viable solution.
ALCE Output:
The question of whether religious education should be taught in public schools is complex and contentious. On one hand, exposing students to texts from different religions can enhance their understanding of historical contexts and the development of societies, well promote empathy and critical thinking /Ttp/////DTTTTT11111///11/111111
On the other hand, the U.S. Supreme Court has consistently ruled that public schools must remain secular and cannot sponsor religious activi- ties, as this would violate the Establishment Clause of the First Amend-
PENERATIONPROGRASS Program:
-paraphrase (S4, instruction= RRephaae to emphasize the educational benefits of exposing students to texts from different religions.") fusion(S6, S7, instruction="Contee historical connection between religion and the advancement of sciences.") compression (S12, instruction="Shorten this sentence to focus on the importance of examining different cultures and their perspectives.") fusion(S15, S16, instruction="Merge these sentences to emphasize the development of empathy and critical thinking skills through exploring diverse cul culture..")
these sentences to highlight the
INNERATTIONPOGGRAE Output:
Introducing students to texts from various religions can enhance their learning by pro- viding context for the development and progress of societies[4].
Students may be encouraged to examine the correlation between religion in school and the advancement of sciences, for instance, by researching how mathematical concepts such as calculus and trigonometry came into practice with the advancement of Islam[6][7]. Teachers can incorporate religion by examining different cultures and their perspec- tives|12].
When students are given the opportunity explore diverse cultures and evaluate the causes of conflicts, they are more likely to appreciate the viewpoints of others, exercise em- pathy, apply critical thinking skills, and brainstorm viable solutions to problems[15][1]]
Figure 3: Example LFQA.
23

---

Published as a conference paper at COLM 2025
Question: where does the ohio river start and finish
Document:
S3: The Ohio River is at the boundary of the Midwestern and Southern United States, flowing southwesterly 981 miles (1582 km) long, starting at the confluence the Al- legheny & the Monongahela Rivers in Pittsburgh, Pennsylvania, and ending in Cairo, Illinois, where flows into the Mississippi River.
S17: The Ohio River runs 981 miles ending at Cairo, II, and is the largest tributary to the Mississippi River.
Gold Response:
The Ohio River begins at the confluence of the Allegheny and Monongahela Rivers in Pittsburgh, Pennsylvania. stretches for 981 miles until it ends at Cairo, Illinois where it drains into the Mississippi River.
GENERATIONPROGRAWT Post-hoc Program:
compression(S3, instruction="Focus on the starting point of the Ohio River. ") fuusion S17, instruction="Combine these sentences describe the length and endpoint of the Ohio River.")
GENERATIONPROGRANE Recreated Output (Rouge-L 60):
The Ohio River starts at the confluence of the Allegheny and Monongahela Rivers in Pittsburgh, Pennsylvania.
The Ohio River, the largest tributary to the Mississippi River, flows southwesterly for 981 miles (1582 km) from the confluence of the Allegheny and Monongahela Rivers in Pittsburgh, Pennsylvania, to Cairo, Illinois, at the boundary of the Midwestern and Southern United States.
Example of post-hoc attribution with GENERATION/PROCRAN and its re-generated program with Rouge-L score.
Figure 4:
ASQA
LFQA
Avg. Word
Avg. Sent
Avg. Word
Avg. Sent
ALCE GENBRATIONPROGRANS
31.4
1.7
87.1
4.7
41.9
2.3
140.6
4.8
Reference
71.8
3.8
47.5
2.4
Table 11: Average number of words and sentences.
D.3 Stylistic Difference
We analyze potential reasons for the drop in correctness with the standard metric using GENERATTIONROGGRAE Our core hypothesis is that the performance difference stems from a stylistic divergence between our method and ALCE, which is a direct result of their respective prompting strategies, as described in Appendix B. This distinction is critical: ALCE optimized to mimic specific answer style, while our method optimized to generate correct underlying program. The final output of our method is result of the program's execution, which naturally leads to a different, often more verbose, stylistic presentation. This explains both the length difference and the resulting scores on style- sensitive metrics.
A result of this is that the output length of GENERATION is usually longer. We present statistical analyses of their outputs using NLTK (Bird et al., 2009) in Table and observe that CENERATIONPROGRA.. tends to produce longer texts compared to ALCE.
24

---

Published as a conference paper at COLM 2025
Instruction: Write an accurate, engaging, and concise answer for the given question using only the provided search results (some of which might be irrelevant) and cite them properly. Use an unbiased and journalistic tone. Always cite for any factual claim with the corresponding passage number. When citing several search results, use [1][2][3]. Cite at least one document and at most three documents in each sentence. If multiple documents support the sentence, only cite minimum sufficient subset of the documents.
"EXAMPLES**
[examples}
*YOUR TASK*
Question: {question)
Passages: {context]
Output:
Figure 5: Prompt for ALCE.
E Prompts
We include the prompt for ALCE in Figure 5, GENERATIONPROGRAS in Figure 6, the mod- ules for GENERATION in Figure 7, and the summarization method in Figure 8.
25

---

Published as a conference paper at COLM 2025
You are given document consisting of passages and specific question. Your task is to write an accurate, engaging, and concise answer to the given question that synthesizes the information from the document. Instead of providing the final answer directly, output a list of Python function calls that, when applied to the sentences, produce the final answer. The arguments should be either sentence indices (e.g., S1) or other function calls. you include an instruction in a function call, it must start with instruction="YOUR INSTRUCTION".
Your available functions:
1. "paaaahraasesentennce instruction=Nonne"
Purpose: Rephrase the given sentence while preserving its original meaning. Optional: You can specify an instruction for a desired style or syntactic structure (e.g, instructio=="OOUR INSTRUCTION").
#*compresionn<enteenc, Nose
Purpose:* Compress the given sentence to produce a shorter version that retains the essential content and syntactic structure. Optional:" Include an instruction detailing which parts to preserve (e.g, instruc- tion= INSTRUCTION").
3. *fusion(sentence 1, sentence 2.. sentence n, instruction=None)** Purpose:* Merge multiple sentences into single sentence. The sentences might convey similar or complementary information.
Optional:" Provide an instruction on how to combine the sentences, such which parts to prioritize (e.g., instruction= INSTRUCTION").
*Careful:**
"[Format]** Format your output as a bullet-point list, where each bullet point is single sentence. For each sentence, you must output series of Python function calls that, when executed, produce the final answer sentence. Each bullet should start with a "_" followed by the function calls without any additional content.
""Function Nesting]** You can nest functions as needed. The arguments for any function may be either sentence identifier (from the document) or the output of another function call.
"EXAMPLES"
(examples)
YOUR TASK**
Question:
[question]
Document:
{context)
Output:
Figure 6: Prompt for running GENERATION
26

---

Published as a conference paper at COLM 2025
compression
You are sentence compression assistant. Your task is to generate pressed version of the provided sentence while maintaining its original meaning. If optional instruction is provided, ensure that your output adheres to it. Do not include any additional commentary or extra text. Your response should consist solely of the compressed sentence.
""Instruction*: |[NSTRUCTION]] **Sentence** [[SENTENCE]]
paraphrase
You are a paraphrasing assistant. Your task is to generate a paraphrased version of the provided sentence while strictly preserving its original mean- ing. an optional instruction is included, ensure that your paraphrase adheres it. Do not introduce any additional information commentary, Your output should be only the paraphrased sentence, with no extra text or labels.
"IInsructtoo"* |[NSTRUCTION]]
""Sentence** [[SENTENCE]]
fusion
You are a sentence fusion assistant. Your task is to generate a single sen- tence that fuses the information from the provided multiple sentences (each on a new line). Merge similar information while ensuring that all differences retained. If an optional instruction is provided, adhere to it. Your response should consist solely of the fused sentence, without any additional commentary or labels.
"IInstucctoo"": [[NNSRUCTIOOI]] ** ""Sentence(s)** |[SENTENCE]]
Figure 7: Prompt for modules.
Extractive
Given the following document and the question "question", extract num- sent sentences from the passage that can answer the question. Do not change the sentences and copy the sentences exactly as they are. You should format your output as a bullet point list, where each bullet point is single sentence. Each bullet should start with "_" followed by the sentence.
*Dooument:** [context)
Abstractive
Given the following document and the question "question", summarize the following document within numsent sentences that can answer the question. You should format your output bullet point list, where each bullet point is single sentence. Each bullet should start with a "_" followed by the sentence.
"*Document:* {context)
Extract- then- Generate
Given the following document, the question "question", and the extrated summary based on the document, revise the summary into numsent sen- tences that can answer the question. The revised summary must incor- porate all the key information from the extracted summary. You should format your output bullet point list, where each bullet point is single sentence. Each bullet should start with a "_" followed by the sentence. Document: {context)
Extractive Summary: {summary)
Figure Prompt for summarization.
27


Reviews:
[{"review_id": "1E2lwPxOH4", "reviewer": "Reviewer_BPee", "summary": "The paper proposes GENERATIONPROGRAMS, a modular framework that decomposes text generation into a two-stage process for improved attribution in long-form QA. Unlike prior methods that interleave generation and citation or apply post-hoc heuristics, this approach first generates a structured, executable “program plan” using modules like paraphrase, fusion, and compression, then executes it to produce the output while explicitly tracking which input sentences contribute to each generated one. This results in fine-grained, contributive attribution at the sentence level. The framework supports both concurrent and post-hoc attribution, outperforms prior methods (e.g., ALCE) in document- and sentence-level attribution F1 on ASQA and LFQA datasets, and enables localized refinements by editing individual module steps. Summarization is optionally used to filter noisy context, and the framework generalizes well to both query-focused and multi-document settings.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The core idea of explicitly structuring generation as a sequence of interpretable operations is both novel and practical. By framing output construction as an executable program of text-editing modules, the framework makes attribution *transparent and verifiable*, allowing users to trace back how and why each part of the answer was produced. This goes beyond current “supporting sentence” approaches and enables true contributive attribution.\n\nGENERATIONPROGRAMS also has strong empirical improvements across two challenging benchmarks. It shows consistent gains in both sentence- and document-level attribution F1 compared to strong baselines. It’s also flexible: it can be used for both real-time generation and post-hoc attribution, and the modular structure allows for localized error correction or refinement.\n\nAnother strength is its compatibility with LLMs without the need for retraining. The framework uses prompting to generate Python-style programs and runs LLM-based modules zero-shot with tailored instructions. This keeps the method lightweight and easily deployable.", "reasons_to_reject": "The most significant limitation of this paper is that it evaluates the proposed framework exclusively using GPT-4, without testing on other models to demonstrate generality. Given that the method is presented as a modular and model-agnostic framework, it is crucial to show that it works across a range of LLMs, including smaller or open-source ones like Vicuna or LLaMA. In contrast, prior benchmarks like ALCE evaluate multiple models and show attribution F1 exceeding 80 under optimized settings—much higher than the 60s cited in this paper, which appears to be based on stricter contributive attribution but is not clearly clarified. Without broader evaluation, it’s unclear whether the observed gains stem from the framework itself or simply from GPT-4’s capabilities.\n\nAdditionally, while the framework emphasizes generality across generation tasks, the modules used (e.g., fusion, paraphrase, extract) are primarily summarization-focused. This limits its applicability to tasks involving more complex reasoning or knowledge manipulation, where different types of operations may be required.\n\nFinally, the paper does not compare against other structured or program-based generation pipelines beyond ALCE. Recent works like Slobodkin et al. (2024), Saha et al. (2023) (mentioned in the paper), or [1] propose related approaches (e.g., plan-then-generate), and a more direct comparison or discussion would help position this work within the growing space of controllable and interpretable generation frameworks.\n\n[1] Ji et al., Towards Verifiable Text Generation with Generative Agent. AAAI 2025.", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EDG42K8hBk", "reviewer": "Reviewer_TDKU", "summary": "The paper’s goal is to generate texts with accurate fine-grained attributions to the source documents. Given the user’s query and the documents (already retrieved before generation starts):\n\n* Use a prompted LLM to generate k programs, each corresponding to an output sentence. The program (e.g. `fusion(S1, S2, instruction=”Highlight the trend of robots replacing humans.”)`) contains the source documents (`S1` and `S2`) and what to do with them (`fusion` = integrating them into a single coherent sentence). The sources become the attributions/citations of the sentence. The instruction gives a semantic outline of the output sentence, ensuring a narrative flow.\n* (optional) Summarize long sources.\n* Use a prompted LLM to execute each program separately.\n\nThe approach is evaluated on long-form QA. It improves attribution quality as measured by NLI scores against the attributed sources, though the end task accuracy drops a bit. The approach can also be used for post-hoc attribution and refinement by giving the generated text as additional input.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. **The proposed method is effective at generating attributed sentences.** The program acts as an interpretable causal link from the source documents to the sentence, so the attribution is likely to be more accurate compared to post-hoc attribution (generate first then link to the sources).\n\n2. **The proposed method still gives the LLM some freedom on how to structure the response.** The model is given all the sources, and it can choose any subset of the sources to base the response on, and the responses do not have to be a strict summary of the sources. The “instruction” field in each program contains the guideline of how the sentence and thus overall response should flow.\n\n3. The method can be adapted to **take existing generated text as additional input** and attribute or refine it. Experiments on post-hoc attribution and fine-grained refinement show good improvements in terms of attribution metrics.", "reasons_to_reject": "Overall, while the method improves attribution, **it might be sacrificing many other properties** such as the end task accuracy, coherence, ~and factuality.~\n\n1. **End task accuracy:** The decrease in end task accuracy (Line 219) is not fully analyzed. It would be troublesome if the individual sentences are seemingly attributed in the NLI sense but the response is not good when taken as a whole. Some qualitative analysis of the errors could reveal the issues.\n    * Line 220 also lists the amount of accuracy drop (2% on ASQA and 7.1% on LFQA) and these do not look “slight” to me.\n    * Extractive summarization seems to help, but this also decreases attribution by quite a lot (Table 1).\n    * Note that the issue could be due to the evaluation methodology (e.g. maybe how ASQA computes exact match in long answers does not play well with the proposed method), but this has to be proven by an analysis.\n    * **EDIT:** Based on the author's response, when more complex evaluators (e.g. LLM judge and human) are used, the numbers are closer to the baseline. I recommend adding these results as well as the qualitative analysis about the \"drop\" in EM/ROUGE-L to the paper.\n\n2. **Coherence and fluency:** While the “instruction” field dictates the response flow, each program is executed separately (see the prompt in Figure 7). This means cross-sentence coherence could be disrupted. Fluency could also be affected. For example, certain entities (e.g. “robot” in Figures 1 and 2) could be replaced with pronouns in normal writing, but they would be repeated in the proposed approach. **There is no evaluation on coherence and fluency.**\n    * Granted, this could potentially be fixed with another round of prompted rewriting, or by feeding previously generated sentences during program execution. Please let me know if the method already does this. \n\n3. ~**Factuality:** If some of the sources are out-of-context, the LLM may incorrectly interpret the source during execution and produce unfactual claims. The fact that the programs are executed separately and the summarization of long sources could worsen this. **The attribution metrics cannot detect such factuality errors** since the sentences are properly entailed in the NLI sense.~\n    * ~Compare this to traditional RAG which always sees all the sources during generation. In an ideal scenario, the model could reconcile the conflicting information across sources and interpret out-of-context sources based on other sources (though admittedly most RAG models also exhibit this type of errors.)~\n    * **EDIT:** The response addresses this concern -- the method can reconcile the sources during the program generation phase, since it can see all the programs. The provided example is convincing. As for the metric, factuality losses will appear in the end task metric, so Weakness 1 still applies.", "questions_to_authors": "1. The programs are defined as trees, but the examples only show flat functions with 2-3 arguments. Are there cases where a recursive tree is needed?", "extra_scores": {}}, {"review_id": "XIuNjh3GYn", "reviewer": "Reviewer_gWx6", "summary": "The paper proposes GenerationsProgram, a two-stage framework that (i) prompts an LLM to emit an executable Python “program plan” made of modular text operators (fusion, compression, paraphrase, extract) and (ii) executes that program to obtain the final answer. Because every intermediate node in the program tree retains the sentence IDs it touched, the system delivers contributive, sentence-level citations “for free.” Experiments on two long-form QA benchmarks (ASQA and LFQA) show large gains in attribution F1—+31.6 pts at the document level and +27 pts at the sentence level—while removing almost all “no-citation” cases﻿. The program trace can also be generated post-hoc to retrofit citations (47.4 F1 vs 7.8 for the best baseline)﻿, and it enables cheap, module-level reranking that adds another 2–4 F1 points at negligible compute cost﻿.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1)\tClear technical novelty: first work to cast attributed generation as code-agent planning with executable, instruction-passing modules, extending earlier “summarization programs” to multi-document QA and RAG﻿.\n2)\tLarge empirical wins in document-level citation, sentence-level citation, and missing citations﻿.\n3)\tVersatile: works both during generation and as a post-hoc attribution tool, outperforming semantic-similarity and prompt baselines by a wide margin on LFQA﻿\n4)\tInterpretability & controllability: explicit program trace doubles as an explanation and allows targeted “module reranking” that boosts F1 further with extra modules refined﻿\n5)\tThorough analysis: ablations on three summarization filters, intrinsic faithfulness with MiniCheck, module-usage statistics, and cost discussion showing modest overhead﻿\n6)\tReproducibility: prompts and code as supplementary material and promised to be open-sourced", "reasons_to_reject": "1)\tAccuracy trade-off: answer EM/ROUGE drops by a few pts relative to ALCE﻿ and is not completely recovered by summarization.\n2)\tLimited generality evidence: all experiments use the proprietary GPT-4o backbone; no tests on open-source or smaller models.\n3)\tDataset scope: only two English Wikipedia-based QA sets; no evaluation on summarization, dialogue, or non-English corpora.\n4)\tEvaluation dependence on LLMs: LFQA attribution judged with GPT-4o NLI, creating potential self-evaluation bias.\n5)\tLatency & engineering details: the paper claims minimal overhead but gives no wall-clock numbers or memory profile; security of executing LLM-generated Python is not addressed.\n6)\tProgram robustness: failure modes (invalid AST, infinite loops) and their handling are not reported.\n7)\tPost-hoc F1 still moderate: F1 in oracle setting signals room for improvement and unclear practical utility.\n8) \tHuman factors absent: no user study to confirm that program traces are actually more interpretable or improve trust and how well correlated LLM judges are with human evaluations", "questions_to_authors": "1) Accuracy–attribution trade-off: Can you report answer EM/ROUGE after module-level refinement? Does reranking hurt correctness further?\n2) Backbone diversity: How does the framework perform with an open-source model e.g. Llama or DeepSeek. Would also be interesting to show performnce with reasoning / frontier models e.g. DeepSeek-R1, o3, Gemini-2.5 or Claude-3.7 and with smaller models.\n3) Runtime: What is the median end-to-end latency and GPU memory footprint per query versus ALCE, with and without refinement?\n4) Program validity: How often does the LLM emit invalid Python or non-entailed intermediate steps, and how do you recover?\n5) Security: Executing arbitrary code generated by an LLM can be risky. Did you sandbox execution or restrict the AST? What are the risks you see and how could these be mitigated?\n6) Human evaluation: Have you measured whether users find the program-style explanations more trustworthy or easier to audit than plain citations? How does human evaluation relate to LLM judgement?\n7) Multi-sentence operations: Can the modules handle paragraph-level fusion/compression? If not, how would you extend them?\n8) Language and domain: Any preliminary results on non-English corpora or scientific/medical QA where paraphrase quality and citation stringency differ?", "extra_scores": {}}]

Author rebuttal:

Meta-review: In this paper the authors propose a framework for sentence-level answer attribution in long-form QA, which works by generating an executing a human-interpretable “program plan” made up of sub-functions such as “paraphrase,” “fusion”, and “compression.” They evaluate the approach on GPT-4 and Llama 3.3 70B (during discussion period) for sentence- and document-level attribution on the ASQA and LFQA datasets, demonstrating better performance than prior work (in particular, ALCE). 

Reviewers agreed that the proposed approach addresses a practical problem in a new and interesting way, and that the approach seems to work well. Reviewers also highlighted that the approach has additional desirable qualities, such as not requiring LLM re-training, and flexibility to work for both post-hoc and concurrent attribution. Reviewers raised concerns around the comprehensiveness of the evaluation with respect to model families and comparisons to baselines, though this was satisfactorily addressed when the authors provided additional results and explanation for why some existing approaches were in fact not suitable baselines. Reviewers also raised concerns that while the approach seems to work quite well in providing useful explanations, this comes at the cost of some prediction accuracy on the question answering task itself. However, the authors provided a reasonable and well-supported explanation for this difference during the discussion phase, which I and the authors who engaged found satisfactory, especially given the novelty of the approach. The authors should be sure to include that explanation, as well as the additional results on Llama models provided during the discussion phase, in the final version of the paper if the paper is accepted.

[Automatically added comment] At least one review was discounted during the decision process due to quality]
Decision: Accept