Document Summarization with Conformal Importance Guarantees
Authors: Bruce Kuwahara
Chen-Yuan Lin
Xiao Shi Huang
Kin Kwan Leung
Jullian Arta Yapeter
Ilya Stanevich
Felipe Perez
Jesse C. Cresswell
Venue: NeurIPS 2025 poster ()
Keywords: Document Summarization
Conformal Prediction
Large Language Models
Primary area: applications
TL;DR: We apply conformal prediction to provide statistical guarantees that all important information within a long-text is captured by an automatically generated summary.

Abstract: Automatic summarization systems have advanced rapidly with large language models (LLMs), yet they still lack reliable guarantees on inclusion of critical content in high-stakes domains like healthcare, law, and finance. In this work, we introduce Conformal Importance Summarization, the first framework for importance-preserving summary generation which uses conformal prediction to provide rigorous, distribution-free coverage guarantees. By calibrating thresholds on sentence-level importance scores, we enable extractive document summarization with user-specified coverage and recall rates over critical content. Our method is model-agnostic, requires only a small calibration set, and seamlessly integrates with existing black-box LLMs. Experiments on established summarization benchmarks demonstrate that Conformal Importance Summarization achieves the theoretically assured information coverage rate. Our work suggests that Conformal Importance Summarization can be combined with existing techniques to achieve reliable, controllable automatic summarization, paving the way for safer deployment of AI summarization tools in critical applications. Code is available at github.com/layer6ai-labs/conformal-importance-summarization.

Document Summarization with Conformal Importance Guarantees
Bruce Kuwahara" Signal AI Toronto, Canada
Chen-Yuan Lin" Signal 1 Al Toronto, Canada
Xiao Shi Huang" Signal 1 AI Toronto, Canada
Kin Kwan Leung Layer 6 Al Toronto, Canada
Jullian Arta Yapeter
Ilya Stanevich Signal 1 AI Toronto, Canada
Felipe Perez Signal 1 Al Toronto, Canada
Jesse C. Cresswell
Signal AI Toronto, Canada
Layer 6 Al Toronto, Canada
Abstract
Automatic summarization systems have advanced rapidly with large language models (LLMs), yet they still lack reliable guarantees inclusion of critical content in high-stakes domains like healthcare, law, and finance. In this work, introduce Conformal Importance Summarization, the first framework for importance-presere preserrigg summary generation which uses conformal prediction to provide rigorous, distribution-free coverage guarantees. By calibrating thresh- olds on sentence-level importance scores, we enable extractive document summa- rization with user-specified coverage and recall rates over critical content. Our method is model-agnostic, requires only a small calibration set, and seamlessly integrates with existing black-box LLMs. Experiments on established summariza- tion benchmarks demonstrate that Conformal Importance Summarization achieves the theoretically assured information coverage rate. Our work suggests that Con- formal Importance Summarization can be combined with existing techniques to achieve reliable, controllable automatic summarization, paving the way for safer deployment of AI summarization tools in critical applications. Code is available at hat.rrrraaaallaaooooooommmammmaa
Introduction
Summarization is widely performed task in many domains, from media [68] and legal documents
34
scientific articles (IT) and clinical reports [19]. Recent advances in large language models
(LLMs) have significantly improved the quality of summary generation [15] [ 29 64], with methods
such as prompt-based generation 22] 666) and fine-tuned transformer models 61] exhibiting superior
generalization and adaptability over classical natural language processing (NLP) methods 42 67
However, in critical domains any error in an Al-generated summary can have serious consequences [5]. For example, even with evidence that AI summarizers can reduce physician workloads and alleviate burnout (25). lack of consistency and the need to verify the Al's work remains concern for physicians in practice [60]. Despite the improvements mentioned above, existing method guarantees retention of important content which could, for example, ensure safety in a high-stakes application like healthcare [18].
Conformal prediction 662 59 has recently risen in popularity as it provides distribution-free. finite-
sample coverage guarantees [2], and has shown promise in classification [56]
3
33
regression
10 43 55), and language tasks such factual question answering B7
14 In this
work, we introduce Conformal Importance Summarization, the first application of conformal
"Equal contribution
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

prediction to document summarization which provides statistical guarantees on the inclusion of important content.
Our contributions are as follows:
We formalize the problem of iuportance-preseryee document summarization with statistical guarantees through the conformal prediction framework.
We introduce a method that calibrates sentence-level importance scores, allowing summary generation with user-specified error (a) and recall (8) rates.
We evaluate our method across multiple summarization benchmarks, demonstrating empirical importance coverage expected from our theory, and quantifying the utility of our approach.
2 Background and Terminology
2.1 Document Summarization
Summarization is the task of producing concise version of a source document that preserves its most important content. There are two major categories of approaches [35]: extractive summarization, which selects spans of text (typically sentences) taken verbatim from the source text, and abstractive summarization which generates new sentences that paraphrase or synthesize information from the source text. While abstractive summarization extremely accessible due to the advent of instruction- tuned LLMs [63 588. extractive summarization can be more suitable to high-stakes domains because it limits the possibility of hallucinations and remains more faithful the source's s meaning. While main focus is on extractive summarization, we also show how the two approaches can be combined benefit from the improved fluency and conciseness that abstractive summarization offers.
Classical extractive methods such as TextRank 44 rely on heuristics or graph- based techniques. Modern extractive models, such BERTSum 41), leverage pretrained language models like BERT 21 to encode sentence-level representations and classify sentence importance. Recent trends in summarization also include reinforcement learning for optimizing summary-leve objectives directly (e.g., ROUGE [50 (3)) fact-consistency tuning using entailment models [36], and hybrid extractive- abstractive pipelines [K]
In this work, we show that extractive summarization, which scores and ranks text spans, is naturally compatible with the calibration step of conformal prediction and can achieve statistical guarantces on the retention of important sentences.
2.2 Conformal Prediction
Consider a classification regression problem with inputs x X' associated with ground-truth values y" E 3' drawn jointly from a distribution y²) P. Conformal prediction (CP) 62 59 first calibrates a threshold ğ based on labeled data, then predicts for any new datapoint Tiest, while guaranteeing coverage with a user-defined eиrог rate
output values (T S y
Plliiest E (Ci(ei)))) 2 1 at.
Remarkably, the coverage guarantee is distribution free and valid in finite samples, making CP a versatile tool to provide robust guarantees correctness in wide variety of scenarios 22].
Given a model fo - y, CP defines a conformal score function larger values indicate worse agreement between fo(x:) and y". Upon computing S over n calibration datapoints, the threshold ş set as the [[=+]](1_a]] quantile of the conformal scores. For any new input, prediction sets are generated by including all output values for which the conformal score is below the threshold q.
X y -> TR, where
Ca(Frea) = {y E )/ s ((Fisst, v) < 9).
As long as Festt exchangeable with the calibration data drawn from TP, Eq. ① will hold. For equal coverage levels 1 a, the usefulness of prediction sets be judged by their size [56] B 33), with smaller average set sizes El indicating prediction sets that are more useful in downstream tasks [16 17. The quality of C2 is largely driven by the accuracy and calibration of fe, and the design of for expressing the model's confidence.

---

2.3 Conformal Factuality for Question-Answering
Extending CP to language tasks requires rethinking the problem setup and design of the conformal score S. Whereas classification tasks have a finite lahel set V open-ended question-answerig has an effectively infinite output space with many semantically equivalent responses, making it incompatible with the standard CP framework described in Section 2.2
Conformal factuality [55 overcomes these challenges by replacing prediction sets with entailment sets, aiming to give responses that are entailed by the ground-truth y with high probability [8] Given natural language question and response generated by an LLM, conformal factuality first decomposes the generated text into subclaims ğ = [cI....,cy) }p}. We use T(x, y²) S D to denote the subset of generated claims which are entailed by the ground-truth y for question Then, individual claims are filtered out based on a calibrated threshold ÿ. The remaining claims constitute a response y ÿ which is factual with high probability,
Ply S (TE
To set 9. each claim's 's factuality is first evaluated based on heuristics such model confidence or self-evaluation. Then, the conformal score is assigned as the greatest factuality level out of all claims not entailed by y² After computing the conformal score for each (z, v²) in the calibration set, the overall conformal threshold ( is set as the [x++)[]-=]] quantile of scores.
On test data, claims with assessed factuality level less than i are filtered out. The sets of retained claims satisfy Eq. ( by Theorem 3.1 of 455] Since longer, more detailed answers are more helpful, the quality of the final response is judged by how many of the generated claims can be retained, subject to meeting the coverage guarantee in Eq. (3). Quality dictated by how well the factuality of individual claims is assessed, and the prevalence of non- non-factual claims in the generated response, which can be improved with stronger LLM, adjustments to the score function, better grounding through retrieved context, or reasoning [1 23 57
3 Extractive Summarization via Conformal Importance
We take inspiration from conformal factuality to go beyond guarantees on extractive summarization. By calibrating threshold on importance score, we ensure that the final summary preserves the most salient information while maintaining bounded risk of omitting critical content.
and provide statistical
Formally, our goal is produce shorter version y of the long-text which, with high probability, retains all important information. We take the long-text x E X to consist of multiple sentences, {ci,..... }, and let y" S be the ground-truth set of important sentences from 2 We then filter out sentences from x based on calibrated threshold q leaving subset S which retains all important information with high probability.
Ply 3 a..
We note key difference between Eq. 3 and Eq. conformal factuality aims for high precision that retained claims are factual, while conformal importance aims for high recall that important sentences are retained. As long as high recall is ensured, shorter summarics are preferable which allows us to measure the quality of y as the proportion of sentences removed. In the remainder of this section we develop a more general framework for error control than used in conformal factuality, describe our method to extract summaries, and theoretically prove that it obeys coverage guarantee.
Generalizing the Coverage Guarantee. One limitation of the conformal factuality framework is that Eq. (3) rigidly considers a response with any number of non-factur claims to be failure case. A more general framework would allow the user to set their own tolerance on how many non-factual claims could appear in an acceptable response [ Hence, for conformal importance we relax the coverage guarantee Eq. (4) so that summaries are acceptable they retain least a fraction / of the important sentences. Let
ly Ny" 10*1
B(w:
For simplicity we break by sentences, but any span could be used. We discuss cases where y² Z below.
3

---

8p
calibration set
R(c;z)
S 10, (z,
Importances
first pose UCL problem. To solve the UCL problem, we propose STAM.
Sentence
Scoring
Conformal
illustrase STAMs using -TAMs provide ways to solve UCL problams.
digits.
0.7
Sorted Sy indices
[o(
quantile
(a)
Importances
Homans can reason, SoTA DL models con't
Sentence
03
Filtering
-Hurmnns cen rensont---iiie Bl-medei cont. -ee intredince RRN for ressoning taoks.
We introduce RRNs for
We introduce RRNN for ressoning troks.
reteaning taoks We use
We RRN SoTA results of 96.0%.
We RRN SoTA results of 96.6%
RRN So results of
sugment any NN
-AINN-oo-engmenttara sugmont
96.8%.
R(c;a)
(c)
Figure I: (a) Steps to compute the conformal score for labeled datapoint. Ground-truth important sentences are coloured in green. S3=1 the smallest value of R(c; 2) for any important sentence. (b) The conformal threshold ğ is computed as a quantile of the sorted conformal scores over the calibration set. (c) At inference, only sentences with importance score R(c; x)) 2 9 are retained in the summary.
be the recall, i.e. the fraction of important sentences retained by y. Then the relaxed coverage guarantee is
P(B(y:D") 2 8 2 1 a.
Of course, this recovers Eq. when B 1.
Conformal Importance Summarization.
Given the long-text x we assign an importance score to each sentence as R(c;x) such that R(c; x) 2 0 with larger scores indicating higher estimated importance. In practice, the design of the importance score key to the performance of method, and we discuss various options below. Similar to [45. we define a filtering function based on importance scores
(x) fce R(cx)) 2 9).
(7)
This function satisfies both Po(z) = x and (x) = Ø so that Fg filters out more sentences as increases, and satisties a nesting property: (x(x) (x) for
BI..
To determine the appropriate threshold we use CP calibration with conformal scores over labeled calibration dataset. For each pair of long-text z; and ground-trute summary yf we compute
Sa (r,, vi) == maxfq E R+ ((F,(z:): 2 3).
(8)
We note that the maximum always exists by the definition FT(x). This score computes the largest threshold such that at least fraction 3 of the important sentences are retained after filtration. A larger (and hence larger 5g) is preferable as it will produce a more concise summary y: = (x,)).
Noting that B(Fa (2): is non-increasing function of we have the following observation. Lemma 1. For a fixed q. we have
(9) Proof. First, assume that Sz(2, > G. By the definition in Eq. 8 there exists o' such that (x) > 3. Since (x) is non-increasing in q. and d' 2 q. it follows that ((FG());V") Z B. On the other hand, now assume ((Fg(x):))) 2 B. By the definition of maximum, we have (2xx.") 2 & directly from Eq. (8)
Spl1, v¹) 2 B(F((z):)') 3.
With all conformal scores So(x,, of computed on the calibration set, we choose the overall conformal
threshold as the [a(n+1)] quantile Given new long-text Tes we score each sentence's importance
as R(e; Te and filter out any sentence with importance less than q, returning Y/est = Fb (Zess) as
shown in Figure [ This procedure satisfics our generalized coverage guarantee in Eq. (
'The different quantile compared to conformal factualiity is necessary, we have shifted to focus on recall instead of precision in the coverage guarantee and conformal score function.

---

Theorem 1. Let {I,, Vi Till be exchangeable and 0 < B S 1. Let á be the Ja(n+1il -th quantile
in+1
of the scores (Sp(2i, Vi Ji=1. which we assume to be distinct without loss of generality. Then for 1+11 11, we have
1
1 1- I- Di 2
(10)
n n-1
Proof. Let 8; Sg v() for = 1, ,,..., and Stest = (5( +1 1)+1). Without loss of generality, we assume the scores are sorted 81 < ..... < 8n- By exchangeability of the rs,,, and for any
k == 1..... we have
k
Pistest 2 86]
(II)
n 1
Noting that q ([a(+))) by definition of the quantile, and that 2 1
ntI. we obtain
P( 2 ql = La(n + 1)1
1
(12)
n n+1
By Lemma
I we then have
P ([
2 33 2212- 11--c 1-a. a.
(13)
On the other hand, from Eq.
12
we see that
[a(n + 1)J
a(n + 1) -
1
P(Stest 2 ②)
=1
<1
a
(14)
n n+1
n+1
n
which shows that
1 1-aSPIB(P((nn-))) 1): 1 1 1
(15)
Design Choices. Beyond the free parameters a and which can set according to the user's error tolerances, our method involves design choices around the ground-truth y" and the importance score function R(c;x), We defined important sentences as those sentences appearing in y", but the source of ground truth remains flexible. On benchmark datasets for extractive summarization, y* may be provided at the sentence level as a subset of x. For other datasets, a summary may be given, but not as verbatim sentences selected from æ, in which case assume sentences in y can be matched to a unique source from x. When y is unavailable, and cannot be collected from domain experts, automated generation techniques using prompted LLMs can select important sentences. However, We describe concrete
note the strong possibility of bias if the same LLM is used for R(c; x) methods for sentence matching and ground truth generation in Section 4.1
49 20
Even for the same long-texts x, different users may consider different parts important. For example, when summarizing clinical notes produced over the course of patient's 's hospital stay, doctor and an administrator may have different views on which details are important. Conformal Importance Summarization can accommodate these different viewpoints; for the same data 2, each user can indicate their preferences y on the calibration set, and receive a threshold q tailored to their needs. may also be beneficial to guide the meaning of importance via R(c; æ). for instance by describing within an LLM prompt the type of information which should be scored highly. Indeed, designing R(c; x²) is the most direct way to influence the ultimate performance of our method, so we offer and experimentally compare several possibilities that can be grouped into two classes.
LLM Scoring An LLM is prompted to return a score between O and 1 of how important is, given I as context. We test five different LLMs GPT-40 mini [47], Gemini 2.0 Flash-Lite (6) Gemini 2.5 Flash (28). Llama3-8B (29), and Qwen3-8B (65. all with the same prompt given in Appendix B The first three used public APIs, while the latter two were hosted locally on a 48 GB A6000 GPU.
Embedding Similarity Scoring Sentence- embeddings are commonly used in extractive summarization, and we demonstrate how they can be leveraged to give coverage guarantees. Using a sentence-level embedding model (e.g. SBERT [54]), distances between all embeddings are computed to form a graph. Then, one of four aggregation methods is used to produce importance scores: Cosine Similarity Centrality [53 builds fully connected similarity graph and assigns importance as the weighted centrality of node; Sentence Centrality (27 creates a directed graph where each
sentence's is importance is computed based on similarity to later sentences only; GUSUM (26) creates
5

---

Table 1: Dataset Details
Dataset
Subset Filtering
Labeling Method
[Deal
[Des
Average Length p
Label Positive Rate
ECT
All
Provided
100
2322
45.6
0.10
CSDS
Valid Test
Provided
100
1500
25.5
0.27
CNN/DM
1000 samples
SBERT Opt.
100
900
31.0
0.10
TLDR-AIC
>1 summary
SBERT Opt.
100
1043
40.7
0.06
TLDR-Full >1 summary
SBERT Opt.
100
1043
216
0.0014
MTS
>1 input sentence
GPT-4o mini
100
1029
6.4
0.81
an undirected graph where edges are cosine similarities, and the importance score is augmented with sentence-level features like position and length; LexRank [22] uses a similar undirected graph construction, but filters out weak connections, and then applies PageRank-like algorithm [48] over node centrality to rank the importance of sentences.
Abstractive Post- post-proeessing. Extractive summarization limits the possibility of hallucination, but may produce text which does not flow smoothly. On the other hand, using LLMS for direct abstractive summarization does not give precise control over coverage and recall. Abstractive summarization essentially contains two subtasks: identifying important information, and synthesizing into a concise and grammatically correct summary [13 [4]. Direct prompting forces the LLM to perform both tasks at once. We argue that disentangling these subtasks can lead to better information retention, similar to how retrieval-augne -augeeeeleegmeetee generation
38 splits question-anee -answernnn into the two subtasks of
information retrieval and answer generation.
We propose a hybrid extractive-abstrrative pipeline which first identifies important information, then synthesizes it. First, Conformal Importance Summarization extracts important information from x as Fz(z)) with guarantees on coverage and recall. Then, extracted sentences are passed to an LLM which is prompted to preserve all information, and only improve conciseness and flow. While there is no guarantee the LLM will maintain coverage, below we test how successful it can be in practice, and compare to pure abstractive summarization. The prompts used are given Appendix
4 Experimental Setup
Our experiments are designed to validate the conformal guarantee given by Theorem I and compare Conformal Importance Summarization to existing summarization methods that do not provide guarantees. We run ablations to understand (i) the influence of both a and B; (ii) the design of the importance score function R(c;=); and, (iii) for datasets without explicit ground-truth labels, the label creation method. Finally, we compare pure abstractive summarization with an L.M to our hybrid aetractive-abstractte approach.
4.1 Datasets
We use 5 datasets to evaluate the performance of our framework: ECTSum [46] contains complete transcripts from corporate carnings calls, as well as expert-curated extractive summaries at the sentence level; CSDS 40 is a dataset of Chinese language customer-client conversations. Although the summaries abstractive, each conversation has sentence-level labels for use
extractive benchmark; CNN/DM (32 covers news sourced from CNN and The Daily Mail with human -written summary sentences; SciTLDR [1] consists of summaries of scientific papers sourced from both authors and peer-reviewers, and we use two versions where the input is either the full text (TLDR- Full), or just the abstract, introduction, and conclusion (TLDR-AIC); MTS- 7 is collection of
conversations and corresponding summaries intended to cover dialogue material.
For each dataset, random subset (n 100) of all datapoints is sampled to form the calibration dataset. All remaining samples form the test dataset, except for CSDS where we use only the original validation and test sets, and CNN//ailyMail where we use 900 samples to reduce resource requirements, as shown in Table [] Other details on dataset preparation are given in Appendix A
While ECTSum and CSDS contain sentence-level labels, the other datasets require them to be generated by comparing the summary text to the original sentences. For CNN/DM and SciTLDR, greedy labeling process based on SBERT [54] embedding cosine similarity is used, which we describe fully in Appendix A with Algorithm Due to the heavy context-dependency of MTS, we
instead queried GPT-4o mini to generate the labels, which tended to classify sentences as important at

---

Empiriaal
Coverage (ECT)
Coverage (ECT)
1.0
Empirical Theovetical
Empirical Coverage
B 0.95
0.85
B 0.75
Theoretical Bounds
1.0
0.95
0.85
0.75
Targel Coverage (1
Target Coverage (1-11
Figure 2: User-specified target coverage 1-a versus average empirical coverage, on ECTSum using
Gemini 2.5 Flash scores. Dashed lines show theoretical bounds given in Theorem |
Results are
averaged over 400 random splits of calibration and test data.
a higher rate. We share the prompt used in Appendix B labels would completely match domain expert's preferences, it is sufficient evaluate the theoretical guarantees and performance of Conformal Importance Summarization.
While neither GPT-hased nor SBERT-based
4.2 Metrics
To evaluate the quality of Conformal Importance Summarization, we use several complementary metrics. First, independent of the conformal framework, we assess the ranking quality of importance scores R(c; z) using the area under the precision- curve (AUPRC), which evaluates how well each method distinguishes between important and unimportant sentences. AUPRC captures the trade-off between conciseness (precision) and completeness (recall).
Second, we evaluate the conformal framework by fixing values for the target error rate a and recall B, and measuring the proportion of sentences removed, akin size in traditional conformal prediction, which reflects how effectively content is compressed while preserving key information.
Finally, the empirical values of coverage and recall actually acheived are relevant. The recall of given summary is B y²) defined in Eq. 5), while the empirical coverage is binary, computed as B(y; v*) 2 B. Both these measures are then averaged over the test set.
4.3
Baselines
No existing methods provide coverage guarantees for extractive summarization. In lieu of such existing baselines, we implement the several importance scoring methods described in Section 3 and also compare to the empirical coverage attained by LLMs without our conformal framework. In particular, we use GPT-4o mini to directly label each sentence as important or not, given the long-text as context but without access to the ground-trute summary. The prompt we use is provided
in Appendix B Whereas Conformal Importance Summarization can provide summaries for any choice of a, this baseline only provides a single value empirically, with no user control.
For the targeted evaluations of importance functions via AUPRC, we also include the ground-trui importance rate reference point, as it represents the performance of random scoring function.
Results
5.1
Testing the Coverage Guarantee
First, we empirically verify the theoretical guarantees proved in Section We run Conformal Importance Summarization on the calibration set for a specified and B, and measure the empirical coverage on the test set. The coverage rate should between 1 1- a and a + n*I in expectation, per Theorem [ Since the coverage the process a total of 400 times with randomized data splits, and average the coverage. Figure 2 shows results for the ECTSum dataset with Gemini 2.5 Flash scores under several values of a and 3. We see that the average coverage consistently lies between the theoretical bounds, as expected. Deviations from the bounds are minor, and result from the variance the coverage random variable.
1
random variable depending on the calibration data, we repeat
The same plots for other datasets and scoring functions are shown in Appendix C.I

---

Prosortion of Sentennee eeeee Removed
Proportion of Sentences Removed
0 0.05
ECT MTS
0.1
GP14 twithout conformal)
Sci GCTTLR
Figure 4: Target recall 3 vs. proportion sentences removed (conciseness). Left: different values for the target error CSDS. Right: Lines indicate different datasets (a = 0.1). Stars indicate the empirical recall and conciseness achieved by GPT-4o mini without conformal prediction.
Lines indicate
5.2 Effect of a and /
Next, we demonstrate that by tuning and B, one can control the tradeoff between conciseness and com- pleteness of the resultant summary. Figure using
1000
Recall
GPT-4o mini on CSDS shows that a higher allowable еπrо rate a results in lower empirical recall, as does lower target recall B. Note that the minimum target recall for maries, therefore the empirical recall usually larger than 33 in practice for small a.
0.95
0.9
0.05
0.8
a portion of the generated sum-
(without conformal)
0.6
340
empirical recall By;; y of important sentences sum- maries, averaged over the CSDS test set. The dashed line shows GPT-4o mini performance without using conformal prediction.
Figure 3: Target error
Figure 3 (dashed line) also shows the baseline em- pirical recall when using GPT-4o mini as a stan- dalone extractive summarization tool, as described in Section 4.3 This demonstrates how our conformal method allows more fine- ane-grained control over recall and coverage than relying on LLMs directly. The same plots for other datasets and scoring functions are shown in Appendix C22
High recall alone is the objective, since summaries also must be concise. Figure shows the proportion of sentences removed as the target recall 3 is varied. Higher / values are more conservative and retain more sentences, while lower values lead to shorter summaries but may miss more important information. We see that for given value of 3, higher values of a also result in shorter summaries, since greater risk tolerance enables more sentences to removed. These trends consistent across datasets and scoring functions, as shown in Appendix C.3
Again, contrast this with the non-conformal baseline shown stars in Figure Direct extraction with GPT-4o mini produces more concise summaries than our method for a given recall level, but offers no control over what that recall level is. In cases where concise summaries the top priority, our method allows the user to tune B down (or up) to produce shorter summaries than the baseline.
5.3 Importance Score Function Performance
The efficacy of Conformal Importance Summarization highly dependent on the design of importance score R(e; x).. Here we compare the performance of the alternatives described in Section across datasets. Table 2 (left columns) shows the sample-averaged AUPRC between the scores and ground-truth importance labels, well as the dataset positive rate. Gemini 2 and 2.5's scores perform the best across all datasets, but GPT-4o mini also demonstrates superior performance compared to classical NLP methods. Smaller, open-source models still perform admirably compared to NLP approaches. Constraining GPT-4o mini to output a binary score rather than a float negatively affects its performance.
Table 2 (right columns) shows the performance different score functions as measured by proportion of sentences removed with 0.2 and 3 0.8. Once again, the Gemini models and GPT-4o mini generally perform the best by having the greatest reduction length for this level of coverage. One example summary with filtered and retained sentences is shown in Figure

---

Table 2: Performance comparison of importance scoring methods. Left: AUPRC of importance rankings compared to ground truth. AUPRC of the original article indicates the proportion of all sentences labeled as important. Right: Conciseness of summaries (proportion of sentences removed) under Conformal Importance Summarization with a = 0.2 and [ = 0.8. Higher is better.
Importance Score
AUPRC f
Proportion of Sentences Removed 1
ECT
CSDS
CNN/DM
TLDR-AIC TLDR-Full MTS
ECT
CSDS CNNIDM TLDR-AIC TLDR-Full MTS
Original Article
0.10
0.27
0.10
0.06
0.014
0.81
0.00
0.00
0.00
0.00
0.00
0.00
Cos. Sim. Contrality
0.22
0.34
0.34
0.35
0.14
0.86
0.22
0.11
0.18
0.29
0.5
0.18
Sentence Centrality
0.34
0.28
0.86
0.17
0.08
0.22
0.50
0.10
GUSUM
0.21
0.44
0.33
0.21
0.09
0.90
0.11
0.24
0.27
0.15
0.20
0.13
LexRank
0.22
0.43
0.32
0.32
0.14
0.16
0.12
0.20
0.37
0.35
x4
GPT-4o mini (hinary)
0.12
0.34
0.13
0.08
0.02
0.83
0.24
0.22
0.26
0.22
0.28
0.08
GPT-4o mini
0.30
0.49
0.34
0.33
0.20
0.93
0.24
0.25
0.30
0.40
0.26
0.16
T-amm3-88
0.18
0.39
0.22
0.15
0.05
0.92
0.13
0.11
0.14
0.11
0.17
0.14
Qwen3 SB
0.17
0.38
0.22
0.16
0.04
0.91
0.13
0.11
0.09
0.14
0.12
0.22
Gemini 2.0 Fiash-Lite
0.35
0.68
0.42
0.39
0.25
0.95
0.28
0.46
0.25
0.40
0.45
0.13
Gemini 2.5 Flash
0.43
0.69
0.36
0.34
0.24
0.94
0.37
0.49
0.26
0.41
0.47
0.14
unsoraantion financial forecast pricntiainn and liculdity cash flow
delivrre billion,
another outetenciing
flow, fourth
edeition, In our 2020 Sucaiiability Byy Report, we comminieg ined to the ambiiii is gosls of reducing our Scope land greenhouee ose gas emiodions by one-therd by 2000 rehliiing cartonn rectra Try by 2050, An men-moyll--ii dcotined about 9100 just relatedigg 11t additional inventory actiono we Spaie volume wtwiatiionnnnii
Whew we put tegecher we eopect our aejuusted eaening inigi per share trccrese betieeen 20100 30% compared to 2020. reed first ousmee of 2020 cor nings share was (Bis Findly,
tasiman, expeett 21 performane
froe coed flose absose 51 lisen Dves the need two years, Eastraen will troeal acertoen in ataly $250 mill fon in the fasility which will sugporri basimants
fith
foe
cor mntiment withh wosing of glotal maate criais milggaing of challar arr glea created by climate chargee, also creeting
(Phonetid) planti Liiig company's polyesper renewal technolig olygy. this new faclitt reduuee premium recycled content. This mil not only reduoe the companyts use of fous fusla fecdstocia, but will also resuce our greentouse gas emilaiions by 20%6 50%
oalty
with
Figure 5: Example of Conformal Importance Summarization using Gemini 2.5 Flash scores. Left: Ground truth summary sentences in blue; text with a strikcthrough indicates sentences filtered out by Conformal Importance Summarization (a 0.2, 3 0.8). On this example, all important sentences were retained (B(y; y²) = 1) while 36% of all sentences were filtered out. Right: Sentence-wise importance scores R(c; z) compared to the conformal threshold q (dashed line).
5.4 Ablations
First, we perform ablations over the labeling method for datasets lacking explicit ground truth. Specifically, in Appendix [C.4] we test ROUGE-1, -2, and -L scores (39] instead of SBERT cosine similarity, in conjunction with Algorithm To create importance labels, then re-evaluate AUPRC and conciseness performance for all importance scoring methods. Aside from small changes in the scores, the performance trend similar, and no fundamental difference in our conclusions would have been made with another labeling method.
Our method relies on a labeled calibration set which could be expensive to curate. In Appendix C.5 we perform an ablation over the calibration set size n. Compared to n = 100 used throughout this work, having as few as 50 labeled examples can still produce comparable results.
5.5 Comparison to Direct Abstractive and Hybrid Extractive-A Abstraccv Summarization
Next, we evaluate existing LLMs prompted to directly summarize text while retaining at least a fraction 3 of important information. To give these models guidance on what information is considered important, we add 10 examples from the calibration set to the prompt. enabling in in-context learning
9.. We also test our proposed hybrid pipeline from Section where an L.IM starts with our extractive summary and is prompted to retain all information. If successful, this would preserve the empirical coverage level, while improving paragraph flow and potentially shortening the text further. We use Gemini 2.5 Flash for the extractive component, as it showed the strongest results in Table 2 and test GPT-4o mini as the abstractive model (similar plots for Gemini 2.5 Flash are given in Appendix C.6).
Unlike extractive summarization, in abstractive summarization determining whether the important aspects of ground -trnndhh-hhhhhaa sentence have been preserved subjective. Hence, we use proxy for recall B(y; 8²) based on semantic entailment: each ground-trute important sentence in y" is compared to
'LexRank's bag of words embedding step fails on this dataset, hence scores available.

---

Torpet Recal Enpire Coorag
Depert faca Omppical Socal
10
Sacal Surmary
Conterral
Emerical Covarage
Confermal Aosoaciie
Cmpriccl
Abstractive Conterma Ap
Conterm
Abstectivee
lagel Conarage 1-
Abotrctiie
Figure 6: Comparison between extractive summarization with our method, abstractive summarization with an LLM, and our hybrid proposal on ECTSum. Here the target coverage is 1 - a 0.8, the conformal approach uses Gemini 2.5 Flash scoring, and the abstractive model GPT-4o mini.
the generated summary y using LLM-based evaluator to check if the important aspects of the sentence have been retained. In practice we use GPT-4o mini with prompts given in Appendix
In Figure X using ECTSSum we first observe that direct prompting to retain a fraction / of important information is not effective, achieving coverage on only 55-70% of summaries. While the hybrid approach has lower coverage than conformal alone, it greatly outperforms direct prompting. Notably, its improved recall and coverage achieved with the same level of conciseness as direct prompting. We attribute this to how the hybrid approach separates out the two subtasks of identifying important information, and synthesizing into concise summary. Perhaps surprisingly, we note that direct prompting for different levels of B does give some level of control over recall.
Conclusions and Limitations
Our results show that Conformal Importance Summarization provides distribution coverage guarantees for extractive summarization. It allows integration with both LLMs and classical NLP techniques to achieve flexible and reliable summary generation with control over the tradeoff between completeness and conciseness. Furthermore, we show that these results hold with variety of importance scores and labeling methods, which could reflect different user preferences for importance.
Our experiments suggest that recent LLMs are well-suited to judge sentence-level importance. LLMs prompted to directly judge importance can achieve higher sentence removal rates than our conformal method for a single recall level, but do not provide any control over the desired recall or conciseness. An LLM prompted to directly summarize inputs while retaining all important information tends to produce very concise summaries, but with much lower coverage and recall than requested. Separating the subtasks of judging importance and synthesizing information greatly improves recall without sacrificing conciseness. Since we did not perform extensive prompt tuning, we suspect that greater performance could be achieved with our method through tuning, or through reasoning models such as Deepseek-R2 RI (30].
While our benchmark suite spanned the domains of finance, customer service, news, science, and medicine with both English and Chinese examples, we note that only contained two datasets with explicit ground-truth labels for extractive summarization, while the other three datasets required label refinement. Our datasets are somewhat limited in maximum length with the exception being SciTLDR-Ful at an average length of 216 sentences. Meanwhile, electronic health records may exceed 200,000 words [1). Implementing Conformal Importance Summarization for such problem would be valuable step towards validating its practical utility and establishing standards for a and B.
Towards the goal of summarizing longer documents, it is possible to extend our framework spans of text other than sentences, which we focused on for convenience. Simply break the long-text into spans, score their importance, and filter out spans with scores lower than the calibrated threshold. For example, paragraph-sized spans would reduce the number of spans that need to be scored for long documents. Due to the lack of existing datasets with long documents and labeled extractive summaries at the paragraph level, we have not performed these experiments.
Future work could extend our framework to more general forms of labels. One possibility is to perform extractive summarization with non-binary labels, such as an annotator's Likert scale rating of importance, and provide guarantees of including sentences with a specified fraction of the total importance weight. A more ambitious goal is to extend the framework fully to abstractive summarization. This would enable us to better leverage LLMs' natural abstractive capabilities, allow better evaluations on a wide range of datasets, and provide more natural sounding summaries.
10

---

References
[1] Josh Achiam et al. GPT-4 Technical Report. arXiv:2303.08877, 2023.
[2] Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-fres uncertainty quantification. arXiv:2107.07511 2021.
[3] Anastasios N. Angelopoulos, Stephen Bates, Michael Jordan, and Jitendra Malik. Uncertainty sets for image classifiers using conformal prediction. In International Conference on Learning Representations, 2021.
[4] Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Conformal risk control. In The Twelfth International Conference on Learning Representations, 2024.
[5] Elham Asgari, Nina Montaña--rown. Magda Dubois, Saleh Khalil, Jasmine Balloch, and Do- minic Pimenta. A framework assess clinical safety and hallucination rates of LLMs for medi- cal text summarisation. npj Digital Medicine, 8(1), 2024. doi: 10. 10.1008//4174--000 746-025-01667--7
[6] Shrestha Basu Mallick and Logan Kilpatrick, Gemini 2.0: Flash, Flash-L.ite and Pro. https //developers googleblog. com/ en/ mon/gemini-- - 22-famlly-expaneee February 2025, Ac- cessed May 15, 2025.
[7] Asma Ben Abacha, Wen-wai Yim, Yadan Fan, and Thomas Lin. An empirical study of clinical note generation from doctor-patient encounters. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2291-2302. Association for Computational Linguistics, 2023. doi:
168.
[8] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. A large the annotated corpus for learning natural language inference. In Proceedings of the 2015 Confer- ence Empirical Methods in Natural Language Processing, pages 632-642, Association for Computational Linguistics, 2015. doi: 10.18653/v1/DD1-10075
[9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Advances Neural Information Processing Systems, volume 33, pages 1877-1901, 2020.
[10] Evgeny Burnaev and Vladimir Vovk. Efficiency of conformalized ridge regression. In Proceed- ings of The 27th Conference on Learning Theory, volume 35, pages 605-622, 2014.
[11] Isabel Cachola, Kyle Lo, Arman Cohan, and Daniel Weld. TLDR: Extreme summarization of scientific documents. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4776-4777. Association for Computational Linguistics, 2020. doi: 10.18653/v1/ 2020. 1020.findings-emiin
[12] Yapei Chang, Kyle Lo. Tanya Goyal, and Mohit lyyer. Booookscore; A systematic exploration of book-length summarization in the era of LLMss. In The Twelfth International Conference on Learning Representations, 2024.
[13] Yen-Chun Chen and Mohit Bansal. Fast abstractive summarization with reinforce-selected sen- tence rewriting. In Proceedings the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pages 675-686, 2018. doi: 10.18653/v1/P18. 1063.
[14] John J. Cherian, Isaac Gibbs, and Emmanuel J. Candès. Large language model validity via enhanced conformal prediction methods. Advances Neural Information Processing Systems, volume 37, pages 114812-11488 2024.

---

[15] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. Pa Scaling Language Modeling with Pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
[16] Jesse C. Cresswell, Yi Sui, Bhargava Kumar, and Noël Vouitsis. Conformal Prediction Sets Improve Human Decision Making. In Proceedings of the 41st International Conference on Machine Learning, volume 235, pages 9439-9457, 2024.
[17] Jesse C. Cresswell, Bhargava Kumar, Yi Sui, and Mouloud Belbahri. Conformal Prediction Sets Can Cause Disparate Impact. In The Thirteenth International Conference on Learning Representations, 2025.
[18] Emma Croxford, Yanjun Gao, Nicholas Pellegrino, Karen Wong, Graham Wills, Elliot First, Frank Liao, Cherodeep Goswami, Brian Patterson, and Majid Afshar. Current and future state of evaluation of large language models for medical summarization tasks. npj Health Systems, (1):6, 2025.
[19] Emma Croxford, Yanjun Gao, Nicholas Pellegrino, Karen K. Wong, Graham Wills, Elliot First, Miranda Schnier, Kyle Burton, Cris G. Ebby, Jillian Gorskic, Matthew Kalscheur, Samy Khalil, Marie Pisani. Tyler Rubeor, Peter Stetson, Frank Liao, Cherodeep Goswami, Brian Patterson, and Majid Afshar. Development and validation of the provider documentation summarization quality instrument for large language models. Journal of the American Medical Informatics Association, pages 1050-1060, 2025.
[20] Daniel Deutsch, Rotem Dror, and Dan Roth. On the limitations of reference- free evaluations of generated text. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10960-10977 Association for Computational Linguistics, 2022. doi: 1106656/11/22222mmmlm.moo
[21] Jacob Devlin, Ming-Woo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre- fre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), 2019. doi: 10.18653/v1/ N19-1423.
[22] Günes Erkan and Dragomir R Radev. Lexrank: Graph-based lexical centrality as salience in text summarization. Journal of Artificial Intelligence Research, 22:457-479, 2004.
[23] Naihe Feng, Yi Sui, Shiyi Hou, Jesse C. Cresswell, and Ga Wu. Response quality assessment for aeureeedl-uugmeeted generation via conditional conformal factuality. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, page 2832-2836. 2025, ISBN 979840071599 15911. URL https:///oo.ii. org/oog//// 10. /145/3726302...... 3730244
doi: 10. 10.1145/7223302.333042
[24] Sebastian Gehrmann, Yuntian Deng, and Alexander Rush. Bottom-Up Abstractive Summa- rization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018. doi: 10.18653/v L/DI///DDIIIIIII 1/418-1443.
[25] Shilpa Ghatnekar, Adam Faletsky, and Vinod E Nambudiri. Digital scribe utility and barriers to implementation clinical practice: A scoping review. Health and Technology, 1144)8003880, 2.21.
12

---

[26] Tuba Gokhan, Phillip Smith, and Mark Lee. GUSUM: Graph- based Unsupervised Summariza- tion Using Sentence Features Scoring and Sentence-BERT. Proceedings of Graph-based Methods for Natural Language Processing, pages 44-53. Association for Compu- tational Linguistics, 2022.
TextGraphs 16:
[27] Shuai Gong, Zhenfang Zhu, Jiangtao Qi, Chunling Tong, Qiang Lu, and Wenqing Wu. Improving extractive document summarization with sentence centrality. PLOS ONE, 17(7):1-16, 07 2022. doi: htwts//oooaaal.onne22.22.
[28] Google Developers. Developers can now start building with Gemini 2.5 Flash. https:// blog. .oogg.oogle.... wwwgle/products/.../o
----emiiii------- 2-w-aaiii------l----p--e---- Flash-preview/ April 2025. Accessed
May 15, 2025.
[29] Aaron Grattafiori et al. The Llama Herd of Models. arXiv:2407.2178. 2024.
[30] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability in Iims via reinforcement learning. 0rXiv:2501.12943 2025.
[31] Chirag Gupta, Arun K. Kuchibhotla, and Aaditya Ramdas. Nested conformal prediction and quantile out-of-bag ensemble methods. Pattern Recognition, 127:108496, 2022. ISSN 0031-3203. doi: 10. 10. 10.101 2006///ppttco.20000 108499.
[32] Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems, volume 28, 2015.
[33] Jianguo Huang. Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, and Hongxin Wei. Conformal prediction for deep classifier via label ranking. In Proceedings of the 41st International Conference Machine Learning, 2024.
[34] Ambedkar Kanapala, Sukomal Pal, and Rajendra Pamula. Text summarization from legal documents: A survey. Artificial Intelligence Review, 51:371-402, 2019.
[35] Huan Yee Koh, Jiaxin Ju, Ming Liu, and Shirui Pan. An empirical survey on long document summarization: Datasets, models, and metrics. ACM Comput. Surv., 55(8), December 2022.
ISSN 0360-0300. doi: 10.1 (0.1145/3545176
[36] Wojciech Kryscinski, Bryan McCann, Caiming Xiong, and Richard Socher. Evaluating the factual consistency of abstractive text summarization. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages main. 750.
2020. doi:
10. 1..18653/1/22200 http://w112220.mmlp.ppm.
[37] Bhawesh Kumar, Charlie Lu, Gauri Gupta, Anil Palepu, David Bellamy, Ramesh Raskar, and Andrew Beam. Conformal prediction with large language models for multi-choice question answering. arXiv:2305. /055588400 18404, 2023.
[38] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensie NLP Tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9474, 2020.
[39] Chin-Yew Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summariza- tion Branches Out, pages 74-81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
[40] Haitao Lin, Liqun Ma, Junnan Zhu, Lu Xiang. Yu Zhou. Jiajun Zhang, and Chengqing Zong. CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarization. In Proceedings of the 202/ Conference on Empirical Methods in Natural Language Processing, pages 1436-4451 2021. doi: 10.18653/v1/2021 http://1/1/111//llppppm .emnlp-main.36:
[41] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCCI.P) pages 3730-3740, 2019. doi: 1/D19
13

---

[42] Yang Liu and Mirella Lapata. Text summarization with pretrained encoders. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNIP-IJCNIPP) pages 3730-3740, 2019.
doi: 10. 10.18653/11DD11-1337
[43] Soundouss Messoudi, Sébastien Destercke, and Sylvain Rousseau. Copula-based conformal prediction for multi-target regression. Pattern Recognition, 120: 108101, 2021. ISSN 0031-3203. doi: https://doi.or: org/ 1011110.011166101 101661 patcog.2021.108111111
[44] Rada Mihalcea and Paul Tarau, TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods Natural Language Processing, pages 404 411. Association for Computational Linguistics, July 2004.
[45] Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees. In Proceedings of the 41st International Conference Machine Learning. 2024.
[46] Rajdeep Mukherjee, Abhinav Bohra, Akash Banerjee, Soumya Sharma, Manjunath Hegde, Afreen Shaikh, Shivani Shrivastava, Koustuv Dasgupta, Niloy Ganguly, Saptarshi Ghosh, and Pawan Goyal. ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcripts. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages (0893-10906, 2022. doi: 10. 10.18653/v1/2022.com hwtp://0222...mmmmma/e/ main. 748.
[47] OpenAl. GPT-4o mini: Advancing cost-efficient intelligence. httpss://pente poopenai..om//iiin com/index/ 4o--moi--ii----
gpt-40 mini advancing- cost- ceees--fffiientt -ineiiennnnteelllenneee//// 2025. Accessed May 15, 2025.
[48] Lawrence Page, Sergey Brin, Rajeev Motwani, and Terry Winograd. The PageRank citation ranking: Bringing order the web. Technical report, Stanford Infolab, 1999.
[49] Arjun Panickssery, Samuel R. Bowman, and Shi Feng. LLM Evaluators Recognize and Favor Their Own Generations. Advances Neural Information Processing Systems, volume 37, pages 68772-68802, 2024.
[50] Romain Paulus, Caiming Xiong, and Richard Socher. A deep reinforced model for abstractive summarization. In International Conference on Learning Representations, 2018
[51] David Preti, Cristina Giannone, Andrea Favalli, and Raniero Romagnoli. Automatic Summa- rization of Legal Texts. Extractive Summarization using LLMs. Ital-IA 2024: 4th National Conference Artificial Intelligence, 2024.
[52] Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S. Jaakkola, and Regina Barzilay. Conformal language modeling, In The Twelfth International Conference Learning Representations, 2024.
[53] Juan Ramirez-Orta and Evangelos Milios. Unsupervised document summarization using pre- trained sentence embeddings and graph centrality. In Proceedings of the Second Workshop on Scholarly Document Processing, pages 110-115, Association for Computational Linguistics,
2021. doi: 10. 00.18653/v1/202..00 1.14.
[54] Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. In Proceedings of the 2019 Conference Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982 3992. Association for Computational Linguisties, 2019. doi: 10.18653/v1/D11-11 1410.
[55] Yaniv Romano, Evan Patterson, and Emmanuel Candès. Conformalized quantile regression. In Advances in Neural Information Processing Systems, volume 32, 2019.
[56] Yaniv Romano, Matteo Sesia, and Emmanuel Candès. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, volume 33, 2020.
14

---

[57] Maxon Rubin-Toles, Maya Gambhir, Keshav Ramji. Aaron Roth, and Surbhi Goel. Conformal language model reasoning with coherent factuality. In The Thirteenth International Conference on Learning Representations, 2025.
[58] Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang. Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. Multitask prompted training enables zero-shot task generalization. In International Conference on Learning Representations, 2022.
[59] Glenn Shafer and Vladimir Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research, 9(3), 2008.
[60] Shreya J. Shah, Trevor Crowell, Yejin Jeong, Anna Devon-Sand, Margaret Smith, Betsy Yang, Stephen P. Ma, April S. Liang. Clarissa Delahaie, Caroline Hsia, Tait Shanafelt, Michael Pfeffer, Christopher Sharp, Steven Lin, and Patricia Garcia. Physician Perspectives on Ambient AI Scribes. JAMA Network Open,
2025.
[61] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Informa- tion Processing Systems, volume 30, 2017.
[62] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic Learning in a RRndom World. Springer, 2005.
[63] Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoe V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2022.
[64] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 Technical Report. arXiv:2412. 15 2024.
[65]
An Yang. Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao. Chengen Huang. Chenxu Lv, Chujie Zheng. Dayiheng Liu, Fan Zhou, Fei Huang. Feng Hu, Hao Ge. Haoran Wei, Huan Lin, Jialong Tang, Jian Yang. Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jing Zhou, Jingren Zhou, Junyang Lin, Kai Dang, Keqin Bao, Kexin Yang, Le Yu, Lianghao Deng, Mei I.,, Mingfeng Xue, Mingze Li, Pei Zhang, Peng Wang. Qin Zhu, Rui Men, Ruize Gao, Shixuan Liu, Shuang Luo, Tianhao Li, Tianyi Tang, Wenbiao Yin, Xingzhang Ren, Xinyu Wang, Xinyu Zhang, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yinger Zhang, Yu Wan, Yuqiong Liu, Zekun Wang, Zeyu Cui, Zhenru Zhang, Zhipeng Zhou, and Zihan Qiu. Qwen3 technical report, 2025.
[66] Haopeng Zhang. Philip S Yu, and Jiawei Zhang. A systematic survey of text summarization: From statistical methods to large language models. ACM Computing Surveys, 2024.
[67] Jingqing Zhang, Yao Zhao Mohammad Saleh, and Peter Liu. PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization. In Proceedings of the 37th International Conference on Machine Learning, volume 119, pages 1328-11339, 2020.
[68] Tianyi Zhang. Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, and Tatsunori B Hashimoto. Benchmarking large language models for news summarization. Transactions of the Association for Computational Linguistics, 12:39-57, 2024.
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We provide evidence supporting each claim made in the abstract and introduc-
tion.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Section 6 outlines the limitations that affect our proposed method. Other limitations and the choices we made to surmount them are outlined throughout the text.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sellsspeciiiationn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
16

---

Answer:
[Yes]
Justification:
Full proofs are provided in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We disclose full details the selection and processing of datasets, algo- rithm used to producing labels, scores, thresholds, and selections of sentences for the final evaluation. Datasets publicly available.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
17

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Our codebase is available at gihhuu.cwwnaaaaaiiiidd///ooooomr summarization Datasets are publicly available.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://niis.cc
(hollic/guides//odeeebbasss www.ssicinooooiyyy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ //nips. wwtps/cccppplgguuuiieeeeaec Stteeee//ooSSSSeesSSioeaa www.sssoiicgyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
All experimental details are disclosed in Section 4 and Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While error bars and statistical analysis are preferred, we were not able to provide them on all experiments due to the cost associated with rerunning experiments that utilize commercial L.LMs. Select experiments come with error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
18

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Computational costs for conformal prediction are negligible and not barrier to reproduction. The other computational costs come from LLM inference for which we used commercial APIs, and locally hosted open-source models run on an A6000 GPU with 48 GB of memory.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https: htt////nuripppp httpp////eeereee..c/o//////aaaee
Answer: [Yes]
Justification:
Our research conforms with the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Summarization is a standard NLP task performed by Al and non-Al means our method does not introduce new negative societal impacts. Benefits are discussed the motivation.
19

---

Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied partieular applications, let alone deployments. However, if there is direct path any negative applications, the authors should point it out. For example, it is legitimate to point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do release models data with a high risk of misuse.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We cite the creators of all models and datasets we use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
20

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperssi thood cww datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Our code has been released at gef.hh/cwoolleeeeii/ll/oooooomoe summarization including documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not use human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
21

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: Our uses of LLMs are standard.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy for what should or should not be described.
22

---

A Additional Experimental Details
A.1 Dataset Processing Details
Some modifications were required to make datasets amenable to our extractive summarization setting.
We selected the amount of input context for summarization. A large portion of this dataset consists of single- sentence summaries. We focus on samples where multiple versions single-sentence summaries have been written from different perspectives (e.g. author, reviewer), total of 1143 samples, and pool those sentences into a single summary as the ground truth.
TLDR-AIC:
subset to provide an appropriate
TLDR-Full:
This version SeiTLDR uses the full text a scientific paper as the input, making imputs much larger and summaries much smaller fraction of the long-text. Otherwise, processing is the same as TLDR-AIE
This dataset contains question and answer-style conversations between doctors and patients. Hence, certain sentences require context for correct interpretation. To accommodate this, each question from the doctor and all subsequent patient responses were merged into single sentence unit, following the pattern "Doctor: questions. Patient: answers". If sample input contained two or fewer sentences after this merge, the entire sample is removed from the dataset; 1129 samples remained after this filtering.
MTS OTSS-Dialog:
Dataset Licenses ECT is released under a GNU GPL license. CSDS is provided for use without specific license. CNN/DM and SciTLDR both released under Apache-2.0 License. MTS- Dialog is released under Creative Commons Attribution 4.0 International license. Hence, our usages of these five datasets are permissible under their respective licenses.
A.2 Greedy Optimization for Extractive Summarization Labeling
The following greedy oracle labeling method is used to provide sentence-level importance labels for datasets that lack sentence-level ground -rout annotations. All we require that reference summary r be available, which could be abstractive rather than extractive. The output of the algorithm is an extractive summary that can be used as ground truth y² in Conformal Importance Summarization. First, for long-text z we compute a similarity score V between each sentence c; and the reference summary, then sort sentences by score in descending order. We then iterate through the ranked sentences in greedy manner and add them to the extractive summary if the overall similarity of the combined extractive summary to increases by at least 8. Mathematically, if the current extractive summary is Yeur with similarity to r of V (Meuri
r), the sentence c; is added to yeur if
V U (c):r) (yeuri r) > 8.
(16)
After one iteration through the sentences in xr, we return Year to be used as the ground truth extractive summary y". This process is depicted in Algorithm
Algorithm 1: Greedy Optimization for Extractive-Summarizttii Labeling
Input:
Sentences x = [ci,..... Gpl, reference summary r, scoring function ((5.), threshold
Output: Extractive summary y Compute 22 = (c) for all i;
Sort indices by descending giving the permutation [T Tn]:
Heur +
Veurr 0
for j = 1 to do t ,.j
A + (iyuu U (cs)ir)=
Venri
> 8 then
Yeur f- Your U (c:} (c:):
Veurr f Veurr + A;
return V
23

---

B LLM Prompts
Ground Truth Labelling. To generate LLM-based ground-truth labels for important sentences in extractive summarization where sentence-level labels were not available, we provided input sentences as separate strings to GPT-4o mini, along with the existing summary text from the dataset. The prompt in Listing was used to produce list of ground-truti labels corresponding to each input sentence.
Evaluate whether each input claim included in the summary corresponding to each input claim, should be either or 1, indicating whether the corresponding claim, or the information it carries, is indeed included in the actual summary. For example, if claim_1's information is contained in the summary, then label_1 should be 1; if information carried claim_3 cannot found in the summary text, then label_3 should
The output labels
Summary text: (summary_text)
List claims: {[claim_text]}
Listing System Prompt for Creating Sentence-Level Ground-Truth Labels
Importance Scoring. To generate LLM-based scores R(c;y" y²" for Conformal Importance Summa- rization, the source text was provided an LLM along with the individual sentences from that text as separate strings. The prompt in Listing was used to produce list of importance scores between and corresponding each input sentence.
Please evaluate the importance of each input claim the original text, based how the information carried in the claim aligned with the overall message. Please provide importance score for EACH imput claim. Each output score should be a two decimal float number ranged between 0 and 1, indicating how important the corresponding input claim is in the context of the text document. For example,
claim I's information text, and very likely to be included in the summary, then score should be close to 1, say greater than 0.8; if information carried in claim 3 is trivial only remotely related to the central message of the text, and is not worthy of inclusion in the aummary, then 0.2.
highly aligned with that
input
should be close 0, say leas than
Original text: (original_text)
List claims: [iclaim_text]>
Listing 2: System Prompt for Creating Importance Scores
We also experimented with prompting the LLM to give binary scores R(c, y²), rather than floats. The prompt Listing was used to produce list of importance scores corresponding to each input sentence as above.
Evaluate the importance of each input claim in the original text, based on how the information carried in the claim is aligned with the overall message. Please provide binary importance
for EACH input claim. Each output score should be either or 1, indicating whether the corresponding input claim is important enough in the context of the text document to be included in the summary. For example, if claim 1's information is highly aligned with that of the input text, and very likely to be included in the summary, then score 1

---

should be 1; if information carried in claim 3 is trivial only remotely related the central message the text, and is worthy of inclusion the summary, then score should 0.
Original text: (original_text)
List claims: {[claim_text]}
Listing 3: System Prompt for Creating Importance Scores with Binary Restriction
Direct Abstractive Summarization. We tested the ability of instruction-tuned LLMs to directly create abstractive summaries that retain a specified fraction 3 of important information. To guide the LLM as to what type of information was important for given dataset, ten examples from the calibration dataset were provided enable in-context learning [
Here examples of what constitutes important information include in the summary:
(examples_text:
Create abstractive summary the following text.
Requirements:
Aim to retain at least (beta>% of the important information Use your words and phraaing (abstractive, extractive)
Input text to summarize:
(input_tex))
Listing 4: System Prompt for Direct Abstractive Summarization
Hybrid Extractive-Abstracttie Summarization. To disentangle the subtasks of importance assess- ment and summary synthesis, we applied abstractive summarization with an LLM as a post-processing step after Conformal Importance Summarization. The abstractive step used the following prompt that specifies all information should retained from the extractive summary input.
Requirements:
Use more concise language to make the text shorter Retain all of the information from the input text Improve flow, coherence, and readability
Listing 5: System Prompt for Abstractive Summarization Post-Processing Step
Sentence-level Recall Estimation. Unlike for extractive summaries, determining if an abstractive summary has retained a specific piece of important information requires a judgement call. Hence, estimated recall of sentence-level information using GPT4o-mini prompted determine if the content of a sentence retained in an abstractive summary.
You will be given an important sentence from the original text and a generated summary. Your goal is to determine whether the important sentence given is retained in the generated summary.
Important sentence: (Irportanttsenneence)
Generated summary:
25

---

{summary)
Output True the important sentence retained in the generated summary. False otherwise.
Output
Listing 6: System Prompt for Determining if Ground-trun -tunnd Important Sentence Retained in an Abstractive Summary
C
Additional Experimental Results
C.1 Coverage Plots for all Datasets and Methods
To supplement the results in Section 5.1 here show the empirical coverage obtained all methods across all datasets. This confirms that all numerical comparisons our work are fair in that they all achieve the expected coverage level. In Figures| 7 [T we shows plots analogous to Figure 2 in the main text. All parameters are identical to that in the main text. We see a similar trend where nearly all datapoints fall within the bounds for all plots, with occasional small deviations due the inherent randomness involved and finite sample sizes.
26

---

Emgriii it Thii Giveage SANORE
DETIEIA EVRON
Iraaka Thinndcc Govaage CA120E
3w33
(a) Cosine Similarity Centrality
(b) Sentence Centrality
Ineike in tharmtce Sommon AM
Commaggn
(c) GUSUM
(d) GPT-4o mini
3-13
(c) Llama 3
Qwen 3
Peeikct Thometea CaaT
Deferme CLAR CONON
Treber
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure User-specified probability of achieving coverage (1-a) vs. empirical probability of achieving coverage, on the CNN/DM dataset. Dashed lines show theoretical bounds given in Theorem T Results are averaged over 400 random splits of calibration and test data.
27

---

1301
3w33
E.B
E-035
Depa Comegge
(a) Cosine Similarity Centrality
(b) Sentence Centrality
Sotee Thamstor Lonann X307
Ing Thomet Cemas 1307
E25
(c) GUSUM
(d) LexRank
LOMOLL LIGGNOO
2-33
3-13
2-33
(c) Llama 3
Qwen 3
Teabbe Thcarte
Concore 2000
Ew005
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure User-specified probability of achieving coverage (1-a) vs. empirical probability of achieving coverage, on the CSDS dataset. Dashed lines show theoretical bounds given in Theorem | are averaged over 400 random splits of calibration and test data.
Results
28

---

2-
Themrstee
Cosine Similarity Centrality
(b) Sentence Centrality
E.B
(c) GUSUM
(d) LexRank
Rot Green
Sope Gonmenat
GPT-4o mini
(f) Llama
Sa Gorman
Qwen
(h) Gemini 2.0 Flash-Lite
Teserbuall
(i) Gemini 2.5 Flash
Figure 9: User-specified probability of achieving coverage (1-a) vs. empirical probability of achieving coverage, on the TLDR-AIC dataset. Dashed lines show theoretical bounds given Theorem Results are averaged over 400 random splits of calibration and test data.
29

---

iices Comtage 1933
Comtag piT))
Coneage
3-33
B-001
(a) Cosine Similarity Centrality
(b) Sentence Centrality
Expicaiss the 1om1
Lomcor MITI
MUS
Dagal Caenageh
(c) GUSUM
(d) GPT-4o mini
LIDRORAL LOR
CINERO LONERE MONT
2-33
(c) Llama 3
Qwen 3
Empae
Faps
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure 10: User-specified probability of achieving coverage (1-x) versus empirical probability of achieving coverage, on MTS dataset. Dashed lines show theoretical bounds given in Theorem [ Results are averaged over 400 random splits of calibration and test data.
30

---

Themsttea
Therstea
EwORs
Com
(a) Cosine Similarity Centrality
(b) Sentence Centrality
E.B
Thceate
(c) GUSUM
(d) LexRank
Sex Som u
Syp Cormini in
Ben Gwerpi
GPT-4o mini
Llama
Ewors
Soet Owoppe
Qwen 3
(h) Gemini 2.0 Flash-Lite
Teseriial
(i) Gemini 2.5 Flash
Figure 11: User-specified probability of achieving coverage (1-a) vs. empirical probability of achieving coverage, on the ECT dataset. Dashed lines show theoretical bounds given in Theorem) Results are averaged over 400 random splits of calibration and test data.
31

---

C.2 Error Rate vs. Recall Plots for all Datasets and Methods
Figures 12 16 show the empirical recall Byy y" based on the choice error rate a for all datasets and methody analogous to Figure in Section 5.1 The trend is similar, with higher leading to lower recall of important content. We notice that for TLDR-AIC and CNN/DM, many of the lines for different 3 values overlap one another. This may be due to the low number of sentences in the long-texts for these datasets, making some /3 values effectively equivalent for those I.
'Due computational constraints, only compute this plot for LexRank on the CNN/DM dataset
32

---

Recall
Recail
0.9
0.9
689
0.8
0.8
GPT4 (without conformal)
GPT4 (without conformall
(a) Cosine Similarity Centrality
(b) Sentence Centrality
100
0.95
0315
0.9
0.90
0.85
0.8
GPT4 twithout conformal)
3:77
0.73
0.9
0.8
(without conformal)
385
41
638
335
6.5
C18
615
335
300
a
(c) GUSUM
(d) LexRank
1800
140
0.9
0.05
0.8
Recall
CPT4 (without conformal)
3.X
0.9
0.8
(without
0.08
3.0
415
8788
335
6.0
<1s
B39
6.50
8.0
(c) GPT-4o mini
(f) Llama
335
Recall
Recall
0.95
0.95
0.9
0.9 380
0.8
0.8
contormal)
(without conformal)
L12
33p
cis
3.0
Qwen 3
(h) Gemini 2.0 FlashLite
Recall
0.95
0.9
0.8
(without conformal)
350
0.5
340
(i) Gemini 2.5 Flash
Figure 12: Target error rate a versus empirical recall B(y: important sentences in summaries, averaged over the CNN/DM test set. The dashed line shows GPT-4o mini performance without using conformal prediction. Several curves may overlap when there are only a few discrete levels of empirical recall possible, making some values of 3 equivalent.
33

---

Recail
Recall
0.95
0.95
0.9
0.9
0.8
0.8
conformal)
(without confermal)
212
cis
(a) Cosine Similarity Centrality
(h) Sentence Centrality
0413
280
Recall
Recall
0.35
0.95
0.95
0.85
300
0.85
0.8
0.8
0990
GPT4 (without conformal)
(without conformal)
<5
335
0.0
945
034
<5
6.5
(c) GUSUM
(d) GPT-4o mini
0313
Recall
5- 0.95
0.95
0.85
0.85
GPT4 (without conformal)
GPT4 (without conformal)
(e) Llama 3
(f) Qwen 3
(.90
Recall
0.95
0.95 C200
0.85
0.85
GPT4 (without conformal)
GPT4 (without conformal)
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure 13: Target error rate a versus empirical recall B(y y of important sentences in summaries, averaged over the CSDS test set. The dashed line shows GPT-4o mini performance without using conformal prediction.
34

---

180
B 0.95
0.85
Recall
Recall
0.8
GP14 twithout conformal)
0.95
0.9
0.85
GPT4 (without conformal)
Cosine Similarity Centrality
(h) Sentence Centrality
1300
0.95
0.85
Recall
Recall
GPT4 twithout contormal)
0.95
0.11
0.85
GPT4 (without conformal)
(c) GUSUM
GPT-4o mini
1.00
125
0315
0.9
341
Recall
0.9
0.9
0.85
0.8
0.8
GPT4 (without conformal)
GPT4 (without conformall
(e) Llama
Qwen 3
1.90
0.95
0315
090
0.85
Recail
GPT4 twithout conformal)
0200
8 0.9
0.8
(without canformal)
8.2
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
empirical recall B y of important sentences in summaries, averaged over the TLDR-AIC test set. The dashed line shows GPT-4o mini performance without using conformal prediction. Several curves overlap because all datapoints TLDR-AIC contain small number of ground-truth sentences, meaning there are only a few discrete levels of empirical recall possible, making values of 3 equivalent.
Figure 14: Target rate
35

---

155
242
Recall
Recall
-0.95
6
0.95
0.85
0.85
GPT4 (without conformal)
380
GPT4 (without conformal)
(a) Cosine Similarity Centrality
(h) Sentence Centrality
0311
Recall
Recall
0.95
0.95
0.85
0.05
0900
GPT4 (without conformall
CW5
GPT4 (without conformal)
(c) GUSUM
(d) GPT-4o mini
- 0.95
120
6 0.95
0315
344
0.00
0.85
0.85
Recall
0.35
Recall
0.8
GPT4 (without conformal)
GPT4 twithout conformal)
(e) Llama
Qwen 3
1.00
Recall
0.95
0.9
0.85
0.85
GPT4 (without conformal)
GPT4 (without conformal)
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure 15: Target error rate a versus empirical recall B of important sentences in summaries, averaged over the MTS test set. The dashed line shows GPT-4o mini performance without using conformal prediction. Several curves may overlap when there are only a few discrete levels of empirical recall possible, making some values of 3 equivalent.
36

---

28-
199
Reeaal
Recall
0.95
0.95
0.9
0.9
0.8
380
0.8
conformal)
(without conformal)
212
cia
(a) Cosine Similarity Centrality
(h) Sentence Centrality
1900
0913
Recall
Recall
0.95
C200
0.95 8
0.85
0.85
0.8
0.8
(without conformal)
(without conformal)
<5
335
945
034
<i
6.55
(c) GUSUM
(d) GPT-4o mini
Recall
Recall
3.5
0.95
8 0.95
0.85
0.85
GPT4 (without conformal)
GPT4 (without conformal)
(e) Llama 3
(f) Qwen 3
Recall
Recall
0.95
0.95
0.85
0.85
GPT4 (without conformal)
GPT4 (without conformal)
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure 16: Target error rate a versus empirical recall B(: y of important sentences in summaries, averaged over the ECT test set. The dashed line shows GPT-4o mini performance without using conformal prediction.
37

---

C.3 Target Recall vs. Conciseness Plots for all Datasets and Methods
Figures show the conciseness, the proportion of sentences removed, based on the choice of for all datasets and method 6 analogous to Figure 4 Section 5.1 Once again, the trend is highly similar across settings, with higher / leading smaller reduction in length.
'Due computational constraints, only compute this plot for LexRank on the CNN/DM dataset
38

---

Proportion of Sentences Removed
Progortion of Sentences Removed
30
a- 0,05
6- 0.05
0.1
0.1
0:15
twithout conferne)
(without conforme))
239
Cosine Similarity Centrality
(h) Sentence Centrality
Proportion of Sentences Removed
Proportion of Sentences Removed
a 0.1
0.1
65
0.15
0.15
0.2
0.2
conforme)
(c) GUSUM
(d) LexRank
Proportion of Sentences nemeved
Proportion Sentences Removed
0.35
0.15
3.9
0.7
0.2
GPM twithat conforme)
GPM (without conforme))
(c) GPT-4o mini
(f) Llama 3
Proportion of Sentences Removed
Proportion of Senteences Removed
0.05
0.05
0.2
k.3
GPT4 twithott conforne)
GPT4 conforet
2.1
B
(g) Qwen 3
(h) Gemini 2.0 Flash-Lite
Proportion of Sentences Removed
D.CS
0.1
(wihout conformel!
22
(i) Gemini 2.5 Flash
Figure 17: Target recall B vs. proportion of sentences removed (conciseness). Lines indicate different values for the target error rate on CNN/DM.
39

---

propertion of Sentences Removed
233
0.1
0.15
0.15
0.2
3.9
0.2
twithout confora))
twitout confemal)
22
(a) Cosine Similarity Centrality
(b) Sentence Centrality
Propertion Sentences Removed
Proporti tion of Sentences Removed
0.05
0.05
0.2
0.2
GPT4 twithout confomna)
GPT4 twithutt contoou)
45
B
(c) GUSUM
(d) GPT-4o mini
Propertion of Sentences Removed
Propertion of Sencences Removed
0.05
0.05
0.1
0.1
GPT4 twithout twlthout confoma))
GPT4 (without conforma))
200
08
10
B
(e) Llama
Qwen
Prapertion Sentences Removed
Prapartion of Sentences Removed
0.1
Ex. = 0.1
62
0.15
0.15
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure 18: Target recall 3 vs. proportion of sentences removed (conciseness). Lines indicate different values for the target error rate on CSDS.
40

---

Propertion Sentences Removed
Propertion Sentences Removed
340
0.1
0.15
24/
0.15
0.2
0.2
twithout conform))
twihout conforma))
339
309
(a) Cosine Similarity Centrality
(b) Sentence Centrality
Propertion Sentences Removed
Proportion of Sentences Removed
0.05
0.05
0.2
0.2
GPT4 twithout confomna)
GP74 twitout confomal))
10
B
(c) GUSUM
(d) GPT-4o mini
Proportion of Sentences Removed
Propertion of Sentences Bemoved
0.D5
0.3
0.05
0.1
0.1
233
GPT4 (without confomal)
GPT4 twithout confemat)
339
04
85
B
(e) Llama
Qwen
Propertion Sentences Removed
Propertion of Sentences Removed
0.1
0.1
0.15
0.15
conforna))
(without conforma))
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure 19: Target recall 3 vs. proportion of sentences removed (conciseness). Lines indicate different
values for the target error rate TLDR-AIC.
41

---

Proportion of Sentences Remooee
Propertion Senteeces Removed
0.1
0.15
0.15
0.2
0.2
twithout
(witout conferma))
B
(a) Cosine Similarity Centrality
(b) Sentence Centrality
Proportoon of Sertences Removed
3.4
0.05
Proportion tion of Sentences Removed
0.05
0.2
0.2
GPT4 twithout conforna)
GPT4 twithott confomal))
335
04
11
45
10
B
B
(c) GUSUM
(d) GPT-4o mini
Proportion of Sentences Removed
0.05
Propertion of Sencences Bamoved
0.5
= 0.05
0.1
0.15
GPT4 twithout twithout confomna)
GP74 (kithout conforma))
339
68
10
B
(e) Llama
Qwen
Propertion of Sentences Removed
Propertion of Sentences Removed
0.05
0.1
0.1
0.15
0.15
conferma))
(without conforma))
kx
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure 20: Target recall 3 vs. proportion of sentences removed (conciseness). Lines indicate different values for the target error rate ECT.
42

---

Propertion Sentences Ramoved
0235
0.1
0.1
0.15
0.15
0.2
23>
0.2
Propertion Sentencees
twithout conform))
(whhout conforma))
0:3
cmm
(a) Cosine Similarity Centrality
(b) Sentence Centrality
Proportoon of Seriences Removed
Propertion of Sentences Ramoved
0.05
0.05
0.15
0.2
0.2
GPT4 twithout confomm))
GPT4 anthouttconoommon
3.3
04
65
B
B
(c) GUSUM
(d) GPT-4o mini
Propertion of Sencences Removed
Propertion of Sencences Removed
0.05
0.1
0.05
0.1
0.1
GPT4 twithout twithoutcoofome confomna)
GP74 (without conforma))
23
33x
34
10
B
(e) Llama
Qwen 3
Proportion of Seriiecee Removed
Propertion of Sentences Ramoved
0.05
0.1
2.33
0.1
0.15
0.15
3.9
twithout conforna))
* (without conforma))
335
(g) Gemini 2.0 Flash-Lite
(h) Gemini 2.5 Flash
Figure 21: Target recall 3 vs. proportion of sentences removed (conciseness). Lines indicate different values for the target error rate on MTS.
43

---

Table 3: Performance comparison of importance scoring methods, measured AUPRC of claim rankings compared to ROUGE-1/21 based ground truth labels. Higher is better.
Importance Score
ROUGE-1
ROUGE-2
ROUGE-L CNN/DM TLDR-AIC
CNN/DM TLDR-AIC
CNN/DM TLDR-AIC
Dataset Positive Rate
0.10
0.06
0.10
0.06
0.10
0.06
Cos. Sim. Centrality
0.13
0.30
0.27
0.26
0.31
0.29
Sentence Centrality
0.28
0.27
0.24
0.24
0.27
0.27
GUSUM
0.31
0.19
0.30
0.18
0.30
0.19
LexRank
0.32
0.30
0.28
0.26
0.32
0.28
GPT-4o mini (binary)
0.13
0.08
0.13
0.08
0.13
0.08
GPT-4o mini
0.38
0.37
0.34
0.32
0.37
0.33
Llama3-8B
0.23
0.18
0.21
0.16
0.24
0.19
Qwen3-888 8BB
0.23
0.20
0.22
0.17
0.23
0.20
Gemini 2.0 Flash-Lite
0.42
0.40
0.38
0.39
0.40
0.38
Gemini 2.5 Flash
0.35
0.36
0.32
0.34
0.34
0.36
Table 4: Performance comparison of importance scoring methods, measured in conciseness of summaries (proportion of sentences removed) under Conformal Importance Summarization compared to ROUGE-1/2/L based ground truth labels. Higher is better.
Importance Score
ROUGE-1
ROUGE-2
ROUGE-L CNN/DM TLDR-AIC
CNN/DM TLDR-AIC
CNN/DM TLDR-AIC
Original Article
0.00
0.00
0.00
0.00
0.00
0.00
Cos. Sim. Centrality
0.22
0.24
0.23
0.12
0.21
0.22
Sentence Centrality
0.22
0.20
0.25
0.22
0.19
0.22
GUSUM
0.21
0.07
0.32
0.10
0.18
0.07
LexRank
0.17
0.26
0.17
0.29
0.16
0.25
GPT-4o mini (binary)
0.26
0.22
0.26
0.22
0.26
0.22
GPTT4o mini
0.32
0,25
0.33
0.28
0.29
0.21
Llama3-8B
0.25
0.27
0.31
0.37
0.27
0.27
Qwen3-8B
0.07
0.06
0.15
0.08
0.11
0.08
Gemini 2.0 Flash-Lite
0.11
0.09
0.09
0.10
0.09
0.11
Gemini 2.5 Flash
0.22
0.28
0.30
0.31
0.22
0.29
C.4 ROUGE Score-based Ground Truth Performance
Table and 4 respectively show the AUPRC and conciseness of summaries when we test our methods using labels generated from ROUGE scores, rather than cosine similarity, using Algorithm Since we only use this algorithm for CNN/DM and SciTLDR, we only display results for CNN/DM and TLDR-AIC.
The results similar to using cosine similarity: Gemini 2.0 Flash-Lite in terms AUPRC, and Gemini 2.5 Flash still performs very well in terms of sentence reduction length. The best numerical values for AUPRC and coneiseness each dataset are also comparable to the cosine similarity-based ground truth from Table
again performs best
C.5 Ablation over Calibration Set Size
Throughout the paper, we used a fixed calibration set size of n = 100 samples to demonstrate that the method can operate with very little labeled data. However, in regimes the availability of labeled data can be extremely limited, so in this section we test our method with even fewer calibration datapoints.
The effect of calibration dataset size in conformal prediction is well understood theoretically; the coverage guarantee is famously "valid in finite samples", mcaning that it holds statistically for any
44

---

Table 5: Ablation of empirical coverage over calibration dataset size n.
Target Coverage
Mean
Standard Deviation
1 a
25
50
75
100
n
25
50
75
100
0.60
0.61
0.61
0.60
0.61
0.09
0.07
0.05
0.05
0.65
0.65
0.67
0.66
0.65
0.09
0.07
0.06
0.05
0.70
0.73
0.71
0.71
0.70
0.09
0.06
0.05
0.05
0.75
0.77
0.77
0.75
0.75
0.09
0.06
0.05
0.05
0.80
0.81
0.81
0.80
0.80
0.08
0.05
0.05
0.04
0.85
0.89
0.86
0.86
0.85
0.06
0.05
0.04
0.04
0,90
0.92
0.90
0.91
0.90
0.05
0.04
0.03
0.03
0.95
0.96
0.96
0.96
0.95
0.04
0.03
0.02
0.02
Table Ablation of summary conciseness (proportion sentences removed) over calibration dataset size Results are taken over 20 random calibration/test splits.
72
Mean
Std
25
0.28
0.09
50
0.31
0.08
75
0.32
0.08
100
0.33
0.05
finite calibration dataset. practice, n controls the variance of the coverage viewed as a random variable over the calibration data. For textbook-style explanation of these details, see Section 3.2 of
We match the experimental setting of Figure which uses 3 = 0.8 and the Gemini 2.5 Flash scoring function to generate results on the ECTSum dataset, shown in Table 5 As guaranteed by Theorem the empirical coverage is no less than 1 for all values of n. Lower values of tend to overshoot the minimum coverage
a by a bit more, because the upper bound of 1 a + m+1 becomes looser with n, but we still find all values within theoretical bounds (for example, 25+1 R 0.04, and 5041 0.02). The primary reason increase
reduce the variance of the empirical coverage so that less likely any given instantiation has lower than expected coverage.
Given that we find the coverage guarantee to be satisfied, we can also check the main metric of our method's performance: the coneiseness of summaries it produces (as the proportion of sentences removed). In Table we we find little difference in the performance when using 50 to 100 samples, smaller datasets. Overall, this ablation demonstrates that
although variance is increased method is applicable in the very low labeled data regime.
C.6 Direct Abstractive and Hybrid Extractive-Abstract Comparison
Here provide additional plots using the same settings in Section 5.5 Figure 22 shows the comparison between extractive summarization with our conformal method, abstractive summarization with LLM, and our hybrid proposal of applying abstractive summarization to extractive mary, this time with Gemini 2.5 Flash used for both conformal scoring, and abstractive summarization. The results are highly comparable to Figure
45

---

Canterral Conturral Aberretive
Empiiial Coverage
Asamantive
Contorms
Contorts Aberrcciin
Tagge Ceanrage 1-
Figure 22: Comparison between extractive summarization with our method, abstractive summariza- tion with an LLM, and our hybrid proposal on ECTSum. Here the target coverage is = 0.8, the conformal approach uses Gemini 2.5 Flash scoring, and the abstractive model is also Gemini 2.5 Flash.
46


Reviews:
[{"review_id": "7R7OZ3irW9", "reviewer": "Reviewer_sN9c", "summary": "This paper introduce an extractive summarization framework using conformal importance. To do that,  the method first assigns importance scores to sentences. The method uses conformal prediction to calibrate importance thresholds for sentence selection. The method is evaluated on multiple datasets using various scoring methods. Results show their method obtained controllable recall and performed competitively with standard extractive baselines. The idea of using conformal prediction for summarization is interesting. However, the paper lacks novelty and fundamental impact for the extractive summarization field.", "questions": "1. Please provide direct comparison with SOTA LLM-generated summaries.\n2. Please provide examples and analysis on long document summarization tasks.", "limitations": "Yes.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Novel integration of conformal prediction with extractive summarization\n2. The proposed CIS method works with various importance scoring-based methods\n3. Overall, the paper is well-written.\n\nWeakness:\n1. The method cannot handle abstractive, long-range, or structurally complex documents, which are common in legal, scientific, and technical domains. The assumption that important content exists at the sentence level is limited.\n2. The method performance depends entirely on the quality of the sentence scoring function.\n3. The experiment lacks direct comparison with abstractive methods or GPT-style Models for document summarization.", "was_revised": true, "final_justification": "Thank you for author's responses. I'd like to change my overall rating to WA.", "rebuttal": "Thank you for reviewing our paper! We are glad to hear you found our use of conformal prediction to be “novel” and will address the weaknesses and questions you raised.\n\n**W1/Q2:**\n> The method cannot handle abstractive, long-range, or structurally complex documents… The assumption that important content exists at the sentence level is limited. Please provide examples and analysis on long document summarization tasks.\n\n**Abstractive:** Our method could be combined with today’s highly performant LLMs to perform abstractive summarization as a post-processing step on our extractive summaries. The extractive summary with coverage guarantees can be fed into an LLM with instructions to further paraphrase and shorten the text, but without removing any important content. As long as the LLM can maintain information and avoid hallucination, it will provide the benefits of abstraction while maintaining our statistical guarantees.\n\nHowever, in practice we find that LLMs are not good enough yet to maintain all important content while meaningfully reducing summary length further. Still, there can be other benefits of abstractive summarization, such as improving the flow and style of writing which may be broken up when only extracting sentences.\n\nWe demonstrate progress in this regard by using our conformal method with a target recall threshold  $\\beta=0.8$ with $1-\\alpha=0.8$ coverage, and then prompt GPT-4o mini to paraphrase and shorten the text while retaining all of the information of the extractive summary. The results on ECTSum (2,322 samples) are summarized below:\n+ **Length:** The abstractive post-processing step slightly increased the length (sentence count) compared to the extractive summary by 3% on average, though with very high variance (std=51%).\n+ **Coverage:** Overall coverage was 61%, a drop from the 80% achieved with our method alone.\n+ **Recall:** Overall recall was mean 75%, std 34%.\n\nSo while there may be some stylistic benefits to an abstractive post-processing step, there is no further reduction in length and a loss of some important information.\n\nNote that the evaluation of recall for the abstractive method must be done on a semantic basis and is therefore less precise. For a given datapoint, each ground-truth important sentence was passed to an LLM evaluator along with the generated summary, and the evaluator was prompted to determine if the ground truth information still appeared in the summary.\n\n---\n\n\n**Long-range:** We acknowledged in L322 that the datasets we used were limited in the length of documents they contained, but this does not mean that our proposed method fails on longer documents. As a test, we have run our method on a dataset with longer document length: in the paper, we used the Abstract-Introduction-Conclusion subset of SciTLDR, with the average input document length being ~41. Here we use the full-document subset of SciTLDR, where the entire paper is used as the input document, increasing the input length to an average of 216 sentences, with resulting performance below (all experiment settings are as in Table 2):\n\n| Importance Score | AUPRC| Fraction of Sentences Removed  |\n|---|--|--|\n| Original Article | 0.014 | 0.00 |\n| Cos. Sim. Centrality   | 0.14  | **0.50**  |\n| Sentence Centrality | 0.09  | 0.45 |\n| GUSUM | 0.09  | 0.20 |\n| LexRank | 0.14  | 0.35 |\n| Gemini 2.0 Flash-Lite   | **0.25**  | 0.45  |\n\nFor the longer documents the positive label rate is much lower (1.4%), and so we see the method is able to remove a higher fraction of sentences, while still maintaining a low rate $\\alpha=0.2$ of failure to achieve high recall $\\beta=0.8$.\n\n---\n\n**Sentence-level:** While we used sentences as a fundamental unit of text to perform conformal summarization, this was a choice for convenience; the underlying method is flexible for any defined span of text like paragraphs or strings of $N$ words. Simply break the long-text into spans, score their importance, and filter out spans with scores lower than the calibrated threshold. This point is another way to address summarization of very long documents - use paragraph-sized spans for example to reduce the number of spans that need to be scored.\n\nThe assumption is less that important content exists at the sentence level, but more that it can be captured using non-overlapping spans. However, the upside of using this assumption and method is statistical guarantees on coverage and recall, something that cannot be obtained with abstractive techniques (see Q1 below).\n\n**W2:**\n> The method performance depends entirely on the quality of the sentence scoring function.\n\nIt is natural that any conformal prediction approach relies on the quality of its conformal score function, so we hardly see this as a weakness. For classification problems, ample research has been done on the design of high-quality scoring functions [52, 3, 30], and there is room for future research to build on our framework with new score functions. We proposed (L175ff) and evaluated (Table 2) several score functions to demonstrate the effect of score function design on the overall performance of our method, and found that today’s strong LLMs give the best results. This is encouraging, as our method can easily swap out for new LLMs that are developed over time, likely giving improved summarization performance “for free”.\n\n**W3/Q1:**\n> The experiment lacks direct comparison with abstractive methods or GPT-style Models for document summarization. Please provide direct comparison with SOTA LLM-generated summaries.\n\nWe have run several more tests using LLMs purely in an abstractive capacity and produced metrics on recall and coverage to compare with our proposed extractive method. We evaluated GPT-4o-mini on ECTSum (2,322 test samples), prompting it with 10 labeled examples from the calibration set to enable in-context learning, with instructions to produce abstractive summaries that retain all important information (target $\\beta=1$). Even though in Table 2 our conformal methods targeted $\\beta=0.8$, we used $\\beta=1$ here to focus the evaluation on the LLM’s ability to capture important information. The evaluation of recall was done in the same way as described above for W1/Q2.\n\nEven with in-context examples, GPT-4o-mini failed to reliably retain all the important information:\n+ **Coverage:** Only 40% of abstractive summaries achieved $\\beta \\geq 0.8$, compared to $1-\\alpha=0.8$ attained with our conformal method, underscoring the **lack of control over recall with naive LLM prompting**.\n+ **Recall:** Average recall was 67% (std=31%), far below the instructions for $\\beta=1.0$, and the guaranteed recall we achieved with our conformal method (minimum $\\beta=0.8$ with high probability).\n+ **Length:** For the summaries that retained a fraction $\\beta \\geq 0.8$ of the important information, the mean fraction of sentences removed was 58% (std=26%), which was higher than the best conformal method at 37% (Table 2).\n\n\n\nIn summary, abstractive summarization with LLMs, even with labeled in-context examples, tends to aggressively shorten text and results in low recall. The biggest drawback here is that the LLM does not give the user any control over recall or coverage (we prompted for 100% recall and achieved 67%). In contrast, our conformal approach consistently meets user-specified recall and coverage targets.\n\nWe ran a similar experiment but without in-context examples for comparison, but this resulted in even more aggressive shortening leading to worse coverage. Please see the response to Reviewer fdXq if you are interested."}, {"review_id": "OqsOAeipMq", "reviewer": "Reviewer_QiQG", "summary": "The paper proposes the application of the conformal prediction framework to extractive summarization. It formalizes the task of conformal importance summarization based on importance score function R for each sentence in a document and a conformal threshold that is calibrated on a small dataset. The experiments on diverse summarization tasks demonstrate the effectiveness of the method for controlling recall of important sentences, as well as the impact of different importance models.", "questions": "In Table 2, it would be helpful to have a baseline line vanilla LexRank, subject to a summary budget in words matching the average words from conformal importance summaries.", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "## Strengths\n\n- The application of conformal importance is quite suitable for extractive summarization, especially in the high-stakes settings where controllability is important.\n- The method also has advantages as it requires a small dataset for calibration.\n- The experiments are comprehensive and well designed, which helps to support the paper claims.\n\n## Weaknesses\n\n- The proposed approach might not scale to longer documents as it requires per-sentence judgment of importance using LLMs.\n- The effects of the calibration data is not explored. It would be informative to have some measure of performance of top-performing importance model as a function of the calibration dataset size.\n- The Appendix is missing, so I could not check some details like the prompts for sentence-level importance.", "was_revised": true, "final_justification": "The authors provide additional experiments that strengthen evidence of the effectiveness of the proposed method over baselines.\nIn the rebuttal, they also address other important aspects like the impact of the size of calibration data. Thus, I revise my recommendation to accept the paper.", "rebuttal": "Thank you for your time spent reviewing our paper, and for noting that our experiments were comprehensive. We are happy to respond to your questions and the weaknesses you raised below.\n\n**W1:**\n> The proposed approach might not scale to longer documents as it requires per-sentence judgment of importance using LLMs.\n\nWe acknowledged in L322 that the datasets we used were limited in the length of documents they contained, however there is no fundamental limitation to applying our method on longer documents, only increased cost to score sentences. As such, we have repeated our experimental setup behind Table 2 on longer documents from the SciTLDR dataset [9]. We originally used the Abstract-Introduction-Conclusion subset of SciTLDR, with the average input document length being ~41. Here we use the full-document subset of SciTLDR, where the entire paper is used as the input document, increasing the input length to an average of 216 sentences, with resulting performance below:\n\n| Importance Score | AUPRC| Fraction of Sentences Removed  |\n|---|--|--|\n| Original Article | 0.014 | 0.00 |\n| Cos. Sim. Centrality   | 0.14  | **0.50**  |\n| Sentence Centrality | 0.09  | 0.45 |\n| GUSUM | 0.09  | 0.20 |\n| LexRank | 0.14  | 0.35 |\n| Gemini 2.0 Flash-Lite   | **0.25**  | 0.45  |\n\nFor the longer documents the positive label rate is much lower (1.4%), and so we see the method is able to remove a higher fraction of sentences, while still maintaining a low rate $\\alpha=0.2$ of failure to achieve high recall $\\beta=0.8$.\n\nWhile we used sentences as a fundamental unit of text to perform conformal summarization, this was a choice for convenience; the underlying method is flexible for any defined span of text like paragraphs or strings of $N$ words. Simply break the long-text into spans, score their importance, and filter out spans with scores lower than the calibrated threshold. This is one way to scale to very long documents if per-sentence judgements become prohibitively expensive - use paragraph-sized spans to reduce the number of spans that need to be scored.\n\n\n**W2:**\n> The effects of the calibration data is not explored. It would be informative to have some measure of performance of top-performing importance model as a function of the calibration dataset size.\n\nThe effect of calibration dataset size $n$ in conformal prediction is well understood theoretically, and in summary has no effect on the summary conciseness on average. The coverage guarantee of conformal prediction is famously “valid in finite samples” which means that it holds statistically for any finite calibration dataset. In practice, $n$ controls the variance of the coverage viewed as a random variable over the calibration data.\n\nCP assumes the calibration data is drawn from a distribution, and that test datapoints are exchangeable with the calibration data. The conformal threshold $\\hat q$ is a random variable that depends on the calibration data; drawing $n$ new calibration points and rerunning CP would give a different value for $\\hat q$. In turn, the conformal prediction sets would also differ, leading to slight differences in empirical coverage and conciseness. The main theorems of CP say that the coverage, as a random variable over the calibration data, will have an average that is no less than $1-\\alpha$ (e.g our Theorem 1, or Equation 1). However, we know more about the distribution than just its average. For typical applications of CP, it follows a Beta distribution with a variance that scales like $O(n^{-1/2})$. For a textbook-style explanation of these details, see Sec 3.2 of [2].\n\nSo, while slightly different conformal thresholds will in practice give different recall and conciseness for summaries, the fluctuations are random and are not expected to give better or worse conciseness as $n$ increases. $n$ controls the size of the random fluctuations.\n\nThe only other notable effect of $n$ is on the tightness of the upper bound in our Theorem 1 (Compare to Equation 1 from [2]).\n\nRecognizing the costly nature of obtaining labels, we limited our experiments to $n=100$ and still obtained strong coverage results demonstrating that very little labeled data is needed in practice.\n\n**W3:**\n> The Appendix is missing…\n\nThe appendix is provided as Supplementary Material rather than added to the main PDF, which were both options given in the PC’s instructions this year. Please download the zip file from OpenReview, and we can answer any further questions it brings up for you.\n\n**Q1:**\n> In Table 2, it would be helpful to have a baseline line vanilla LexRank, subject to a summary budget in words matching the average words from conformal importance summaries.\n\nIn the paper we used LexRank as a **scoring function** for conformal summarization, and based on your suggestion have now run a baseline using LexRank **directly for summarization** on the ECT dataset, using a word budget that matches the average from the other conformal methods. For reference, this ranges from 834 words (Gemini 2.5 Flash) to 1101 words (Cosine Similarity Centrality).\n\nWe tested LexRank in both continuous mode and discrete modes. For the discrete mode, we used the four edge-weight thresholds (0.1, 0.2, 0.3, 0.4) as done in [20]. Additionally, we experimented with two initialization strategies for LexRank’s TF-IDF-based embeddings: one using only the calibration dataset, and another using the combined calibration and test datasets. We only report the latter as it is what we used in the main paper and we found the difference in results to be negligible. \n\nWe evaluate the **recall** of important sentences under this framework, the results of which can be found in the table below. Columns indicate which alternative conformal method was used to derive the average text length to use as the budget for LexRank. \n\n|  Edge Weight Threshold | Gemini 2.0 Flash-Lite | Gemini 2.5 Flash | GPT-4o mini | Llama3-8B | Qwen3-8B | Cos. Sim. Centrality | GUSUM | Sentence Centrality |\n|--:|--:|--:|--:|--:|--:|--:|--:|--:|\n| continuous | 0.76 | 0.71 | 0.79 | 0.82 | 0.83 | 0.83 | 0.86 | 0.82 |\n| 0.1 | 0.75 | 0.69 | 0.77 | 0.81 | 0.82 | 0.83 | 0.85 | 0.81 |\n| 0.2 | 0.75 | 0.69 | 0.78 | 0.82 | 0.83 | 0.84 | 0.86 | 0.82 |\n| 0.3 | 0.77 | 0.72 | 0.79 | 0.83 | 0.84 | 0.84 | 0.86 | 0.83 |\n| 0.4 | 0.77 | 0.72 | 0.79 | 0.82 | 0.84 | 0.84 | 0.87 | 0.83 |\n\n\nRecall varies between 0.69 (0.1 edge weight threshold, based on Gemini 2.5 summaries) up to 0.87 (0.4 edge weight threshold, GUSUM summaries). By comparison, all our conformal methods from Table 2 achieve recall $\\geq 0.80$ for 80% of the datapoints. Hence, conformal summarization with the best LLMs (Gemini or GPT models) tends to outperform LexRank at retaining important information for the same length budget.\n\nThis is also consistent with our AUPRC measurement, which is purely a measure of how well an NLP method can discriminate between important and unimportant information (i.e. it does not rely on conformal methods at all). From Table 2, LexRank has AUPRC of 0.22 on ECT, significantly worse than Gemini or GPT models (0.30 - 0.43 on ECT), but still roughly comparable to other classical NLP techniques.\n\nThe fact that base LexRank’s recall is not controllable, unlike the conformal methods to which we compare it, also shows the utility of conformal prediction in the summarization setting. Integrating LexRank with conformal summarization removes this limitation of the base method alone."}, {"review_id": "1O79Jfm4Qf", "reviewer": "Reviewer_fdXq", "summary": "Given a document, the proposed Conformal Importance Summarization (CIG) framework scores each sentence’s importance and filters out less important ones using a threshold calibrated to ensure that the resulting summary retains at least a fraction \\beta of important content with probability $\\geq 1 – \\alpha$.\n\nCIG is model-agnostic and supports user-tunable tradeoffs between conciseness and completeness. Experiments across five datasets show that the CIG consistently meets its theoretical recall guarantees, while offering controllable summary length and outperforming standard scoring baselines in recall and precision under varying $\\alpha$ and $\\beta$ settings.", "questions": "* Although existing abstractive summarizers (e.g., GPT-4 or PEGASUS) rarely reproduce the exact sentences labeled important, they often paraphrase them well enough that the underlying facts are still conveyed. Measuring how much of this critical content they actually cover (e.g., by computing the recall of important sentences using semantic-entailment or similarity rather than verbatim overlap) ­would clarify how frequently those baselines meet a chosen recall target \\\\beta, even without any formal guarantees. Reporting this would shed some more light on the practical advantage of CIG.  \n* Given that the motivation is centered on critical-domain summaries (medical, legal, etc.), have you considered how to validate CIG in such settings? What should such expert-defined important content look like, and what would be the obstacles and ways to overcome them?  \n* Have you thought about combining the current approach with abstractive summarization techniques? For instance, you could first use CIG to identify a set of important sentences, then generate a refined summary possibly with an LLM. If you compare summaries generated via this two-stage approach to those generated via single-stage prompting of the same LLM, how do they perform in terms of quality and coverage?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n* This work is the first to apply conformal prediction to document summarization, which provides distribution-free statistical guarantees on retaining important content in summaries.  \n* The method is grounded in solid conformal prediction theory, where authors formalize a coverage guarantee for summarization.  \n* Experiment results confirm the theoretical coverage guarantee.  \n* The paper is quite transparent about implementations and supplementary details, which can help a lot with reproducibility.\n\nWeaknesses\n\n* The method depends on having a predefined set of important sentences for each document (for calibration and evaluation). In many real-world cases, these important labels can be subjective or expensive to collect. The quality of the summary’s guarantee is only as good as the quality of these labels. There could be some more discussion on how robust this method is to noisy or imperfect importance labels.  \n* While the motivation is to improve summarization in high-stakes domains like health and law, evaluation is only run on standard summarization benchmarks (news, scientific papers, dialogues). In a truly high-stakes setting, importance could be multifaceted and harder to capture with a single proxy.  \n* The method is heavily focused on coverage guarantees, and does not consider other aspects of summary quality. While completeness (high recall) is critical, things like coherence or usefulness are not being evaluated.", "was_revised": true, "final_justification": "The rebuttal addressed my points, so I choose to maintain my original positive rating.", "rebuttal": "Thank you for your positive review! We are pleased to see that you found our theory “solid”, and will address your questions and comments below.\n\n**W1:**\n> The method depends on having a predefined set of important sentences … can be subjective or expensive to collect. The quality of the summary’s guarantee is only as good as the quality of these labels. There could be some more discussion on how robust this method is to noisy or imperfect importance labels.\n\nWe agree with your argument that in real-world, professional, and sensitive domains, sentence level importance labels would be hard/expensive to collect. While modern LLMs have been shown to achieve good performance on many out-of-domain scenarios with zero/few-shot inference, it is known that their performance could drop on niche and/or sensitive domains. It should be recognized that conformal prediction does not require many labels compared to other ML-based approaches. We used only 100 labeled samples in our calibration sets and achieved tight coverage at high recalls (Fig 2).\n\nOther than the direct and expensive approach of hiring professionals to label training samples, one possible idea is to adapt domain-finetuned LLMs for the purpose of either scoring or labeling. There has been a significant amount of work demonstrating that adapted LLMs show increases in accuracy, reduction in hallucination, and/or more alignment with expert preferences (e.g. [A,B]). All these indicate that they would serve better as both a scoring function as well as a labeling mechanism.\n\nThere has been research around conformal prediction’s robustness toward label noise [C], as well as how the conformal process could be adapted to explicitly handle label noise [D]. Since we use the same theoretical principles as these works, their conclusions may extend to conformal summarization, although it is not the focus of our present paper.\n\n[A] Woo et al. “Synthetic data distillation enables the extraction of clinical information at scale”, npj Digital Medicine, 2025\n\n[B] Fatemi et al. “A Comparative Analysis of Instruction Fine-Tuning LLMs for Financial Text Classification”, arxiv:2411.02476\n\n[C] Einbinder et al. “Label Noise Robustness of Conformal Prediction”, Journal of Machine Learning Research 2024\n\n[D] Penso et al. “Estimating the Conformal Prediction Threshold from Noisy Labels”, arXiv:2501.12749\n\n\n**W2:**\n> While the motivation is to improve summarization in high-stakes domains like health and law, evaluation is only run on standard summarization benchmarks (news, scientific papers, dialogues). In a truly high-stakes setting, importance could be multifaceted and harder to capture with a single proxy.\n\nOne of the five datasets we used did cover the domain of healthcare (MTS-Dialog). We agree that importance can be multifaceted. We discussed in and after L168 how our method can accommodate differing viewpoints on what is important through user-specific calibration set labels and importance score functions. In this way, each user can receive tailored summaries with coverage guarantees that correspond to their personal definition of importance.\n\n**W3:**\n> The method is heavily focused on coverage guarantees, and does not consider other aspects of summary quality. While completeness (high recall) is critical, things like coherence or usefulness are not being evaluated.\n\nGiven that coverage is satisfied, the main metric that demonstrates the effectiveness of our method is the conciseness of the summary -- how much non-important information can be filtered out without sacrificing recall of important information. Since we are focusing on extractive summarization, the notions of coherence and usefulness are less relevant. There is no expectation that an extractive summary preserves coherent paragraph structure and flow, while usefulness is solely dictated by the recall of important information. Below in the response to Q3 we discuss extending our extractive method with abstractive post-processing, where your suggested metrics become more relevant.\n\n**Q1:**\n> Although existing abstractive summarizers rarely reproduce the exact sentences labeled important, they often paraphrase them well enough that the underlying facts are still conveyed. Measuring how much of this critical content they actually cover (e.g., by computing the recall of important sentences using semantic-entailment or similarity rather than verbatim overlap) ­would clarify how frequently those baselines meet a chosen recall target \\beta, even without any formal guarantees.\n\nThis is a great suggestion! We tested GPT-4o mini on ECTSum (2,322 test samples), prompting it to produce abstractive summaries directly from the original source text, with the target of retaining all important content ($\\beta = 1$). While Table 2 reports results for our conformal methods at $\\beta = 0.8$, here we used $\\beta = 1$ to measure the LLM’s raw ability to preserve information. For evaluation we used semantic-entailment to determine recall; each ground-truth important sentence was compared to the generated summary using an LLM-based evaluator to check if the information was retained in the summary.\n\nFor this direct, single-stage abstractive summarization approach we found:\n\n+ **Coverage**: Only 26% of abstractive summaries had recall $\\beta \\geq 0.8$, compared to $1-\\alpha=0.8$ attained with our conformal methods.\n+ **Recall**: Average recall was 52% (std=35%), far below the instructions for $\\beta=1.0$.\n+ **Length**: For the summaries that retained a fraction $\\beta\\geq 0.8$, the mean fraction of sentences removed was 65% (std=25%), which was higher than the best conformal method at 37% (Table 2).\n\nThese results show that naive prompting of an LLM is not likely to correctly retain the ground-truth important information, and clearly does not give any control over recall or coverage. LLMs tend to aggressively shorten content when instructed to summarize.\n\nWe ran a similar experiment but with 10 in-context labeled examples added to the prompt. Please see our response to Reviewer sN9c, W3/Q1. This increased the coverage to 40% and recall to 67%, but decreased conciseness to 58%.\n\n**Q2:**\n> Given that the motivation is centered on critical-domain summaries (medical, legal, etc.), have you considered how to validate CIG in such settings? What should such expert-defined important content look like, and what would be the obstacles and ways to overcome them?\n\nWe did test our method on one medical dataset (MTS-Dialog), and this is a great example of what expert-defined importance could look like. The MTS-Dialog consists of doctor-patient conversations. It is reasonable to expect that in a real-world setting like this, both the doctor and patient may want a summary of their conversation, but the material each person considers important could be very different. A doctor may focus more on the symptoms described by the patient, while a patient would highlight the doctor’s medical recommendations as most important. For the same set of conversations, we may have a set of ground truth labels from doctors, and another from patients, each of which would lead to different conformal thresholds and hence prediction sets.\n\nGoing further, the expert definitions of importance could be incorporated into the importance score function. We found that LLM scoring with a basic prompt performed very well across datasets (Table 2). However, incorporating more information about what the expert deems important could be done by providing few-shot examples labeled by the expert in the prompt to activate the in-context learning capabilities of LLMs. In this way, the LLM scoring would not simply rely on parametric knowledge of what is important from the LLM’s pre-training data, but would directly incorporate expert knowledge about the dataset and preferences at hand.\n\n**Q3:**\n> Have you thought about combining the current approach with abstractive techniques? You could first identify a set of important sentences, then generate a refined summary with an LLM. If you compare summaries generated via this two-stage approach to those generated via single-stage prompting of the same LLM, how do they perform in terms of quality and coverage?\n\nAdding our extractive approach as pre-processing before using abstractive summarization could remove some of the unimportant noise sentences and result in a more focused summary.\n\nWe provide experiment results of this two-stage approach: (1) use our conformal summarization to extract a set of sentences meeting a target recall threshold ($\\beta=0.8$) with high coverage ($1-\\alpha=0.8$), and (2) prompt GPT-4o mini to generate a cohesive abstractive summary from the extracted sentences. Note that we did not instruct the LLM to retain all important information, just to summarize the remaining text. The results on ECTSum (2,322 samples) are below and we compare to the single-stage abstractive summarization from Q1 above:\n+ **Coverage**: 42% of abstractive summaries samples reached $\\beta \\geq 0.8$, compared to 26% without pre-processing.\n+ **Recall**:  Average recall was 67% (std=31%), compared to 52%.\n+ **Length**: For the summaries that retained a fraction $\\beta\\geq 0.8$, the mean fraction of sentences removed was 57% (std=25%), compared to 65%.\n\nHence, there does appear to be a benefit in how the LLM is able to identify important information if the pre-processing is used, although summaries were not quite as concise.\n\nFor another perspective - testing how well an LLM can preserve the recall and coverage guarantees in a two-stage set up - please see our response to Reviewer sN9c in W1/Q2."}, {"review_id": "QBayblbFYB", "reviewer": "Reviewer_HYEU", "summary": "This paper introduces Conformal Importance Summarization, the first framework for importance-preserving summary generation that uses conformal prediction to provide rigorous, distribution-free coverage guarantees.", "questions": "Please see Weaknesses above", "limitations": "Yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "strengths: \n1. The paper offers a novel perspective by bridging conformal prediction with extractive summarization.\n\n2. The analysis is both sound and creatively executed.\n\n\nweaknesses: \n1. The paper contains a large number of formulas, many of which are adaptations of existing proofs, which negatively affects readability. The authors are encouraged to summarize their main contribution to conformal prediction or extractive summarization. Additionally, please clarify: is this paper fundamentally about applying conformal prediction to the extractive summarization task to achieve its benefits—namely, rigorous, distribution-free coverage guarantees—in the resulting summaries?\n\n2. The motivation behind this work is not clearly articulated. Given that the method is restricted to extractive summarization, is this still meaningful in the era of large language models (LLMs)?\n\n3. The paper lacks reporting on a primary metric in conformal prediction: Set Size, which quantifies prediction uncertainty.\n\n4. Instead of defining the threshold ​q_hat using a preset α, an alternative approach would be to use p-values, as suggested by Vovk et al. (2005). It would be interesting to evaluate the performance of the proposed framework using the p-value method for constructing prediction sets.", "was_revised": true, "final_justification": "After carefully considering the authors' detailed responses and the original review comments, I maintain my initial assessment of the paper. The authors argue that the density of formulas is necessary to rigorously establish their novel adaptation of conformal prediction (CP) techniques to extractive summarization. While I acknowledge that theoretical rigor is important, I think the current presentation remains challenging for readers unfamiliar with CP’s technical foundations. However, the authors’ clarification that their work extends CP techniques (not standard CP) to a new setting is valid. Thus, while readability could be enhanced, the core theoretical claims are justified.", "rebuttal": "Thank you for your efforts to review our work! It is great to see that you found our perspective on summarization novel. Below, we will address the weaknesses you brought up.\n\n**W1:**\n> The paper contains a large number of formulas … which negatively affects readability. The authors are encouraged to summarize their main contribution to conformal prediction or extractive summarization. … is this paper fundamentally about applying conformal prediction to the extractive summarization task …?\n\n\nIt is possible that there are some misunderstandings in your review about how we brought together conformal prediction and document summarization. Our contribution is not to apply standard conformal prediction to summarization tasks, but to apply the **techniques** behind conformal prediction to summarization tasks. Our settings and results (Theorem 1) are entirely different from that of conformal prediction as you may have seen it applied to classification or regression tasks. Thus formulas and proofs are needed to make it rigorous. Conformal factuality [41] is another example of applying the techniques of conformal prediction to a novel LLM setting (question-answering), instead of applying standard conformal prediction. For that reason, [41] also provides proofs and formulas.\n\nIn summary:\n+ conformal prediction [57,54, 2] aims for a coverage guarantee that the ground truth answer is contained in the prediction set (Equation 1); \n+ conformal factuality [41] aims for a coverage guarantee that all retained claims are factual (Equation 3);\n+ conformal summarization (our work) aims for a coverage guarantee that all (or a fixed percentage of) the important sentences are retained (Equation 4).\n\nWhile there are common ideas behind all these approaches, the settings and the essential coverage guarantees are notably different, and each requires independent proof that coverage is obtained. \n\n**W3:**\n> The paper lacks reporting on a primary metric in conformal prediction: Set Size, which quantifies prediction uncertainty.\n\nFollowing the above explanation, since conformal summarization is not the same as conformal prediction, we did not report “set size” directly. Set size is indeed a primary metric for conformal prediction and quantifies uncertainty. In a simple classification problem, for example, there is a fixed set of labels to choose from for every datapoint, and the conformal set is just a subset of all possible labels. For the same level of coverage $1-\\alpha$, smaller sets are more useful (L72) which makes average set size a good metric.\n\nFor our problem of document summarization the label set is no longer fixed. Instead, each summary contains a subset of all sentences from the long-texts $x$, which vary in length. The number of sentences retained in the summary is still a good measure of quality, as we want the shortest summaries that retain all (or a fraction $\\beta$) of the ground-truth important sentences (L111). It also still quantifies uncertainty, as longer summaries indicate the model is more uncertain about what content is important. But, since the total number of sentences in $x$ varies, rather than use the *count of sentences* in the summary (similar to set size in conformal prediction setting), it makes more sense to normalize the “set size” as the *fraction of sentences removed* (L233). This is the measure of “conformal set size” we report in Table 2 and other figures.\n\n\n**W2:**\n> The motivation behind this work is not clearly articulated. Given that the method is restricted to extractive summarization, is this still meaningful in the era of large language models (LLMs)?\n\nInstruction-tuned LLMs are powerful tools and natively can perform abstractive summarization with basic prompts. One might wonder why we bother with extractive summarization at all. We comment on this in the paragraph on L25. The main motivation is that **prompted LLMs do not provide any guarantees that they will extract the information a user deems important**. Even worse, they are well-known to sometimes hallucinate information beyond what was provided, or make logical errors in interpreting that information. For example, a study using GPT-4 to summarize emergency department visits found hallucinations in 42% of summaries and that 47% omitted clinically important information [A]. Many similar evaluations of LLMs are available in the literature. In response to Reviewer fdXq (Q1) we tested how well GPT-4o mini retains ground-truth important information with naive abstractive summarization. The results show only 26% of summaries had recall over 80% (low coverage), and overall recall was only 52% (low recall).\n\nBy using extractive summarization it is impossible to fabricate information, and by using conformal prediction we can align the extracted information with a user’s preferences (via the labeled calibration set) and give statistical guarantees on recall and coverage.\n\nLet’s consider an analogy to the popular task of question-answering. Many production systems avoid “pure” LLM generation and instead use Retrieval-Augmented Generation (RAG) [B], which grounds generative outputs in retrieved text to mitigate hallucinations while preserving fluency. RAG is a two-step process that first retrieves from a document corpus and then generates from the retrieved documents. The first step involves finding the most relevant document to the question, in which the output format is controlled (not in the form of free text). By design there cannot be hallucination in the retrieval step, since it only accesses a pre-defined corpus. The second step, generation, has its output as free-form text, i.e. uncontrolled. Thus it could contain hallucinations. But as the inputs to this step are controlled, RAG greatly reduces hallucinations on this step.\n\nOur framework for summarization addresses the same reliability concerns as RAG, and is less prone to hallucination than “pure” LLM summarization. Our method can be thought of similarly to the retrieval step in RAG, in that the output is controlled -- the output sentences are directly from the original text. Just like the retrieval in RAG, we score and rank content, but go further and provide coverage guarantees. The output of our approach can be fed into a second stage of abstractive summarization using an LLM, similar to the generation step in RAG, which serves to improve fluency and further shorten the text. Thus our approach and abstractive LLM summarization are complementary. We provide example experiments in response to Reviewer fdXq (Q3) that show using our conformal summarization as a pre-processing step improves the recall and coverage of abstractive summarization. \n\n[A] Williams et al. “Evaluating large language models for drafting emergency department encounter summaries” PLOS Digital Health 2025\n\n[B] Lewis et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” NeurIPS 2020\n\n**W4:**\n> Instead of defining the threshold ​q_hat using a preset α, an alternative approach would be to use p-values, as suggested by Vovk et al. (2005). It would be interesting to evaluate the performance of the proposed framework using the p-value method for constructing prediction sets.\n\nThis is another interesting question, and luckily there is well understood theory around it. Essentially, the viewpoints of conformal prediction via coverage (modern view [2], which we used) and p-values (classic view [57]) are equivalent and interchangeable. They provide a different statistical interpretation, but they produce the same results. Hence, there would be **no difference to the performance** of our framework if we were to use the p-value method for constructing prediction sets.\n\nLet’s take the classification setting again for example. In the modern view of CP for coverage guarantees we want to construct a prediction set that contains the true label with high probability $1-\\alpha$. The interpretation here is that the sets quantify the uncertainty of the underlying predictive model, with larger sets indicating more uncertainty. Mechanically, we compute a conformal score for each possible label and include it in the set if the score is greater than a calibrated threshold, which is a quantile of scores on a calibration set. The quantile to use depends on the desired coverage $1-\\alpha$.\n\nIn the classic view using p-values we want to construct a prediction set of all labels that are statistically consistent with observations from the calibration set. The interpretation is that CP conducts multiple hypothesis tests and controls Type I errors. Mechanically, for each possible label we compute a p-value representing what fraction of examples in the calibration set are as non-conforming as the proposed label. Then we include all labels where the p-value is greater than a significance level $\\alpha$.\n\nThe definitions of the quantile and p-value in the two views, along with their set construction method, can be exactly mapped onto one another making them totally equivalent in practice."}]

Author rebuttal:

Meta-review: This paper introduces Conformal Importance Summarization, an extractive summarization framework that applies conformal prediction to guarantee recall of important content. It is model-agnostic, requires little calibration data, and shows controllable performance across benchmarks. Weaknesses include restriction to extractive summarization, reliance on sentence-level importance and scoring quality, limited high-stakes evaluation, and heavy use of formulas.

In rebuttal, the authors added experiments on long documents, abstractive baselines, hybrid approaches, and calibration set size, clarifying that their work extends conformal theory with practical guarantees. Reviewers’ responses were largely positive after rebuttal, with raised scores despite some concerns on readability.

Overall, this is a novel, well-justified contribution to reliable summarization, and I recommend acceptance.
Decision: Accept (poster)