Title: Document Summarization with Conformal Importance Guarantees
Venue: NeurIPS 2025
Accepted: yes
Keywords: Document Summarization

Conformal Prediction

Large Language Models
Abstract: Automatic summarization systems have advanced rapidly with large language models (LLMs), yet they still lack reliable guarantees on inclusion of critical content in high-stakes domains like healthcare, law, and finance. In this work, we introduce Conformal Importance Summarization, the first framework for importance-preserving summary generation which uses conformal prediction to provide rigorous, distribution-free coverage guarantees. By calibrating thresholds on sentence-level importance scores, we enable extractive document summarization with user-specified coverage and recall rates over critical content. Our method is model-agnostic, requires only a small calibration set, and seamlessly integrates with existing black-box LLMs. Experiments on established summarization benchmarks demonstrate that Conformal Importance Summarization achieves the theoretically assured information coverage rate. Our work suggests that Conformal Importance Summarization can be combined with existing techniques to achieve reliable, controllable automatic summarization, paving the way for safer deployment of AI summarization tools in critical applications. Code is available at github.com/layer6ai-labs/conformal-importance-summarization.


Peer reviews:
comments: SUMMARY:
This paper introduces Conformal Importance Summarization, the first framework for importance-preserving summary generation that uses conformal prediction to provide rigorous, distribution-free coverage guarantees.

STRENGTHS AND WEAKNESSES:
strengths: 
1. The paper offers a novel perspective by bridging conformal prediction with extractive summarization.

2. The analysis is both sound and creatively executed.


weaknesses: 
1. The paper contains a large number of formulas, many of which are adaptations of existing proofs, which negatively affects readability. The authors are encouraged to summarize their main contribution to conformal prediction or extractive summarization. Additionally, please clarify: is this paper fundamentally about applying conformal prediction to the extractive summarization task to achieve its benefits—namely, rigorous, distribution-free coverage guarantees—in the resulting summaries?

2. The motivation behind this work is not clearly articulated. Given that the method is restricted to extractive summarization, is this still meaningful in the era of large language models (LLMs)?

3. The paper lacks reporting on a primary metric in conformal prediction: Set Size, which quantifies prediction uncertainty.

4. Instead of defining the threshold ​q_hat using a preset α, an alternative approach would be to use p-values, as suggested by Vovk et al. (2005). It would be interesting to evaluate the performance of the proposed framework using the p-value method for constructing prediction sets.

QUESTIONS:
Please see Weaknesses above

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-24

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nGiven a document, the proposed Conformal Importance Summarization (CIG) framework scores each sentence’s importance and filters out less important ones using a threshold calibrated to ensure that the resulting summary retains at least a fraction \\beta of important content with probability $\\geq 1 – \\alpha$.\n\nCIG is model-agnostic and supports user-tunable tradeoffs between conciseness and completeness. Experiments across five datasets show that the CIG consistently meets its theoretical recall guarantees, while offering controllable summary length and outperforming standard scoring baselines in recall and precision under varying $\\alpha$ and $\\beta$ settings.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n* This work is the first to apply conformal prediction to document summarization, which provides distribution-free statistical guarantees on retaining important content in summaries.  \n* The method is grounded in solid conformal prediction theory, where authors formalize a coverage guarantee for summarization.  \n* Experiment results confirm the theoretical coverage guarantee.  \n* The paper is quite transparent about implementations and supplementary details, which can help a lot with reproducibility.\n\nWeaknesses\n\n* The method depends on having a predefined set of important sentences for each document (for calibration and evaluation). In many real-world cases, these important labels can be subjective or expensive to collect. The quality of the summary’s guarantee is only as good as the quality of these labels. There could be some more discussion on how robust this method is to noisy or imperfect importance labels.  \n* While the motivation is to improve summarization in high-stakes domains like health and law, evaluation is only run on standard summarization benchmarks (news, scientific papers, dialogues). In a truly high-stakes setting, importance could be multifaceted and harder to capture with a single proxy.  \n* The method is heavily focused on coverage guarantees, and does not consider other aspects of summary quality. While completeness (high recall) is critical, things like coherence or usefulness are not being evaluated.\n\nQUESTIONS:\n* Although existing abstractive summarizers (e.g., GPT-4 or PEGASUS) rarely reproduce the exact sentences labeled important, they often paraphrase them well enough that the underlying facts are still conveyed. Measuring how much of this critical content they actually cover (e.g., by computing the recall of important sentences using semantic-entailment or similarity rather than verbatim overlap) ­would clarify how frequently those baselines meet a chosen recall target \\\\beta, even without any formal guarantees. Reporting this would shed some more light on the practical advantage of CIG.  \n* Given that the motivation is centered on critical-domain summaries (medical, legal, etc.), have you considered how to validate CIG in such settings? What should such expert-defined important content look like, and what would be the obstacles and ways to overcome them?  \n* Have you thought about combining the current approach with abstractive summarization techniques? For instance, you could first use CIG to identify a set of important sentences, then generate a refined summary possibly with an LLM. If you compare summaries generated via this two-stage approach to those generated via single-stage prompting of the same LLM, how do they perform in terms of quality and coverage?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
The paper proposes the application of the conformal prediction framework to extractive summarization. It formalizes the task of conformal importance summarization based on importance score function R for each sentence in a document and a conformal threshold that is calibrated on a small dataset. The experiments on diverse summarization tasks demonstrate the effectiveness of the method for controlling recall of important sentences, as well as the impact of different importance models.

STRENGTHS AND WEAKNESSES:
## Strengths

- The application of conformal importance is quite suitable for extractive summarization, especially in the high-stakes settings where controllability is important.
- The method also has advantages as it requires a small dataset for calibration.
- The experiments are comprehensive and well designed, which helps to support the paper claims.

## Weaknesses

- The proposed approach might not scale to longer documents as it requires per-sentence judgment of importance using LLMs.
- The effects of the calibration data is not explored. It would be informative to have some measure of performance of top-performing importance model as a function of the calibration dataset size.
- The Appendix is missing, so I could not check some details like the prompts for sentence-level importance.

QUESTIONS:
In Table 2, it would be helpful to have a baseline line vanilla LexRank, subject to a summary budget in words matching the average words from conformal importance summaries.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 4
DATE: 2025-07-03

comments: SUMMARY:
This paper introduce an extractive summarization framework using conformal importance. To do that,  the method first assigns importance scores to sentences. The method uses conformal prediction to calibrate importance thresholds for sentence selection. The method is evaluated on multiple datasets using various scoring methods. Results show their method obtained controllable recall and performed competitively with standard extractive baselines. The idea of using conformal prediction for summarization is interesting. However, the paper lacks novelty and fundamental impact for the extractive summarization field.

STRENGTHS AND WEAKNESSES:
Strengths:
1. Novel integration of conformal prediction with extractive summarization
2. The proposed CIS method works with various importance scoring-based methods
3. Overall, the paper is well-written.

Weakness:
1. The method cannot handle abstractive, long-range, or structurally complex documents, which are common in legal, scientific, and technical domains. The assumption that important content exists at the sentence level is limited.
2. The method performance depends entirely on the quality of the sentence scoring function.
3. The experiment lacks direct comparison with abstractive methods or GPT-style Models for document summarization.

QUESTIONS:
1. Please provide direct comparison with SOTA LLM-generated summaries.
2. Please provide examples and analysis on long document summarization tasks.

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-19