Confident and Adaptive Generative Speech Recognition via Risk Control
Authors: Amit Damri
Bracha Laufer-Goldshtein
Venue: ICLR 2026 Poster (2026)
Keywords: automatic speech recognition
conformal prediction
conformal risk control
large language models
ASR error correction
uncertainty quantification
adaptive hypothesis selection
N-best hypotheses
LoRA fine-tuning
word error rate
statistical guarantees
generative error correction
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce an adaptive error correction framework for speech recognition that selects the optimal number of hypotheses using conformal risk control, achieving robust performance with smaller sets.

Abstract: Automatic Speech Recognition (ASR) systems frequently produce transcription errors due to acoustic variability, which require post-processing correction methods. Recent approaches leverage Large Language Models (LLMs) for generative ASR error correction using N-best hypotheses but rely on fixed set sizes regardless of input complexity and do not provide performance guarantees. We propose an adaptive framework that dynamically determines the optimal number of hypotheses for each input using risk control. This mechanism leverages ASR confidence scores and applies  Learn then test (LTT) to control the expected relative word error rate degradation compared to the best achievable performance for a given model and hypothesis set. Experimental results demonstrate that our approach provides theoretical guarantees with high-probability bounds while matching or exceeding fixed-size correction baselines and requiring fewer hypotheses on average, achieving substantial computational savings under diverse acoustic conditions.

Published as a conference paper at ICLR 2026
CONFIDENT AND ADAPTIVE GENERATIVE SPEECH RECOGNITION VIA RISK CONTROL
Amit Damri & Bracha Laufer-Goldshtein
School of Electrical and Computer Engineering Tel-Aviv University
Tel-Aviv, Israel
lamiddamt @aaami@@mall. @emall, blauferêtauex)- tau. ac.
ABSTRACT
Automatic Speech Recognition (ASR) systems frequently produce transcription errors due to acoustic variability, which require post-processing correction meth- ods. Recent approaches leverage Large Language Models (LLMs) for generative ASR error correction using N-best hypotheses but rely on fixed set sizes regardless of input complexity and do not provide performance guarantees. We propose an adaptive framework that dynamically determines the optimal number of hypothe- ses for each input using risk control. This mechanism leverages ASR confidence scores and applies Learn then test (LTT) to control the expected relative word error rate degradation compared to the best achievable performance for a given model and hypothesis set. Experimental results demonstrate that our approach provides theoretical guarantees with high-probability bounds while matching or exceeding fixed-size correction baselines and requiring fewer hypotheses average, achieving substantial computational savings under diverse acoustic conditions
INTRODUCTION
ASR systems convert spoken language into text, enabling wide array of applications from virtual assistants to transcription services (Kheddar et al., 2024). Over the past decade, deep learning advancements have propelled ASR performance, with models like Wav2Vec (Baevski et al. 2020
and Whisper (Radford et al. 2023) achieving remarkable accuracy on benchmark datasets through suff-supervieed learning and large-scale training. However, ASR remains challenged by real-world variability, including background noise, speaker accents, dialects, homophones, out-of-vocabulary words, and domain shifts, which often lead transcription errors that degrade downstream tasks (Schneider et al. 2019).
To mitigate these issues, recent research (Yang et al. 2023 Ma et al. 2025 Mu al. 2025 has
explored integrating LLMs with ASR outputs for post-processing. A prominent approach involves generative error correction (GER), where the LLM receives a fixed-size of hypotheses, produced by the ASR model, and is asked to provide improved transcriptions (Chen et al. 2023 Hu et al., 2024ap. Usually the LLM is fine-tuned on sequences of N-best hypotheses learn mappings from noisy ASR outputs to ground-truth text, demonstrating noise-robust improvements.
Despite these advances, existing GER methods suffer from key limitations. They predominantly rely on fixed hypothesis set size across all inputs, applying the same N value irrespective of whether the audio is simple (e.g., clear speech) or complex (e.g., accented or noisy), which can result in inefficient resource ovvellvelo the LLM with redundant hypotheses for straightforward cases or introducing low-quality hypotheses that may degrade correction performance, as can be seen in Fig. a)). Furthermore, these approaches lack statistical guarantees the expected performance, such as bounding the gap to the oracle (best possible) transcription, leaving uncertainty in their reliability and their practical improvement.
To address these shortcomings, we propose an adaptive framework for hypothesis set construction in LLM auggeneed ASR, illustrated in Fig. [b). Instead of static N, we dynamically form
Our code is available https::/github..cm wtw.aaittaaiiiittaaaaaaaa ger

---

Published as a conference paper at ICLR 2026
Nebeat Hypoothesin
Generative Error Correction
chicull sus
quaiio Ba
gashooth
0.0
Semple Intex
0.6
Adeptive Hypothesis
Adaptive Generative Error Correction
0.2
Model
LLM
P(WER WBR 2 2.)211 2-14
Daay
(a) Motivating Example
(b) Method Illustration
Figure 1: (a) WER performance patterns across hypothesis set sizes TedLium-3. Samples are grouped by monotonicity: samples that improve with more hypotheses (top), show consistent performance (middle), degrade with more hypotheses (bottom). (b) Comparison of standard GER using fixed 5-hypothesis sets versus our adaptive GER that dynamically selects variable-sized hypothesis sets with risk control to bound relative performance degradation from the oracle.
sets using threshold rule over the likelihood scores of ASR hypotheses, ensuring only sufficiently plausible candidates are passed to the LLM. We tune these thresholds via Learn then test (LTT) Angeloppulos et al. (025), distribution-free framework that provides risk control with high probability on the expected loss (e.g., word error rate (WER)) relative to the best performance achievable by the model. This adaptive strategy yields smaller average set sizes, reducing computa- tional costs, while empirically achieving comparable or lower WERs compared to fixed-N baselines on diverse benchmarks.
Our main contributions can be summarized as follows:
We propose an adaptive hypothesis selection framework that leverages ASR confidence scores dynamically determine the optimal set sizes for each input, replacing the standard fixed-size approach with difficulty-aware resource allocation.
We introduce the first application of risk control to GER, providing theoretical guarantees alongside empirical validation of effective control over relative performance degradation while enabling principled uncertainty quantification in multi-hypothesis scenarios.
We demonstrate substantial computational efficiency gains (up 52% reduction in hypothesis usage) while maintaining correction performance across diverse acoustic conditions, validating both the empirical robustness and practical value of the proposed adaptive selection mechanism.
2 RELATED WORK
Automatic Speech Recognition Error Correction. Language model rescoring has been exten- sively employed in ASR systems to enhance recognition accuracy, with external language models reranking N-best hypothesis lists select optimal transcriptions (Song et al. (021). Recent advances have moved beyond simple reranking toward generative error correction (GER), where LLMs synthesize improved transcriptions by leveraging complete N-best lists rather than merely selecting among existing candidates (Yang et al.) 2023 Radhakrishnan et al. 2023 Yang et al., 2024 Ma et al. 2025 Liu al. 2025 Ghosh et al.) 2024 Mu et al. al.] 2025). Contemporary bench- marks like HyPoradise (Chen et al. 2023) have formalized the typotheses-to-teee tonss-tottrrssss torrrsssciiiitiiionn (H2T) mapping task, enabling systematic evaluation of LLM- correction methods across diverse acoustic conditions. Our approach builds upon this foundation while introducing reliable and adaptive hypothesis selection via risk control methods.
Uncertainty Quantification in Language and Speech Processing. Uncertainty quantification has become critical for deploying natural language processing (NLP) and speech systems in high-stakes applications, with traditional approaches including ensemble methods, Monte Carlo dropout, and calibration techniques for well-calibrated probability estimates (Xiao et al.) 2022). Speech processing faces unique challenges due to temporal audio signals and cascading recognition errors, leading to various approaches including acoustic confidence measures and neural uncertainty estimation (Wullach & Chazan 2023 Rumberg al. 2025). However, these methods often lack
2

---

Published as a conference paper at ICLR 2026
theoretical guarantees and may not generalize across acoustic conditions, making risk control methods attractive as a principled approach providing distribution-fiee uncertainty quantification.
Conformal Prediction and Risk Control Methods. Conformal prediction (CP) (Vovk et al, 2003] Angelopoulos et al., 2024a) provides a distribution-free and model-agnostic framework for uncertainty quantification that constructs prediction sets with guaranteed coverage under minimal exchangeability assumptions. The framework has found extensive applications across regression, classification, and structured prediction tasks, including recent demonstrations in NLP for machine translation, text classification, and question answering (Campos et al. 2024). Conformal risk control (CRC) extends CP's coverage guarantees control expected loss functions beyond simple miscoverage, but requires bounded monotone loss functions to maintain distribution free guarantees (Angelopoulos et al.) 2024b). Learn then test (LTTT) (Angelopoulos al. 2025)
provides an alternative risk control approach that handles non-monotone loss functions through multiple hypothesis testing with family-wise error rate control, offering high-probability bounds without monotonicity assumptions. Pareto-Testing (Lauser-Goidshtein computationally and statistically efficient testing strategy for tuning multiple hyperparameters by performing multi-objective optimization prior to hypothesis testing. We leverage both LTT and Pareto- Pareto-Testing in the development of our reliable and adaptive GER framework.
al., 2023) provides
3 PROBLEM FORMULATION
Consider an input audio signal E X, and corresponding transcription V. Possible transcription hypotheses are generated by a pre-trained ASR model using beam search decoding, and the top N are selected:
HN = ((),, ei), (2), (üN.CN).
(1)
log where is represents the i-th hypothesis transcription and c = log (()((z) denotes the log- ligeliidldood score from the ASR model. The hypotheses are ranked by their scores in descending order such that C₁ 2 c2 2 ... 2 cx. with higher scores indicating higher confidence.
The goal is to learn mapping function MHZT that predicts improved transcription j' from the N-best list:
1 = MMHer(HN:
(2)
where 0 represents learnable parameters. While traditional language model rescoring approaches (Song et al. 202I) re-rank existing hypotheses to select the best candidate, generative error correc- tion (GER) ( Ma et al., [025]] Hu et al., [202aa] Yang et al. 2023) represents the current taae-off-the-at transcriptions by leveraging information across all N-best hy- potheses, potentially producing corrections that do not appear in the original hypothesis list.
approach that synthesize
LLMs have emerged powerful tools for this task due to their ability understand linguistic patterns and perform text generation. The common approaches involve either leveraging existing LLMs with various prompt engineering techniques (Chen al.] 20233 Yang al. 2023) or fine-tuning a pre-trained LLM to learn the mapping from N-best hypotheses to ground-truth transcriptions (Hu et al. 20244a] Radhakrishnan
al., 2023). The model receives the ranked hypotheses (optionally along with their confidence scores) as input and generates the corrected transcription autoregressively. The training process utilizes pairs (Hv. enabling the model to learn the relationship between ASR error patterns and optimal corrections across diverse acoustic conditions and speaking styles.
However, the conventional approach of using fixed-sized hypothesis sets overlooks a critical observation: not all audio segments require the same number of hypotheses for effective correction. In many cases, smaller set is sufficient or even preferable for achieving optimal transcription quality, As illustrated in Fig. [ This phenomenon suggests that additional hypotheses can introduce noise rather than useful signal, motivating the need for adaptive selection mechanisms that can dynamically determine the optimal number of hypotheses based on the specific characteristics of each audio segment.

---

Published as a conference paper at ICLR 2026
4 BACKGROUND LEARN THEN TEST FRAMEWORK
Conformal prediction (CP) is distribution-free framework for uncertainty quantification that relies only the mild assumption of exchangeability between calibration and test data, without imposing additional assumptions on the underlying model or dat--gener process (Angclopoulos et al, the miscoverage probability, many practical
2024a). While standard CP provides guarantees applications require control of more general risk measures. Conformal risk control (CRC) extends CP by enabling control of the expected value of bounded, monotone loss functions (Angelopoulos et al. 2024b). However, the monotonicity requirement restricts its applicability in settings where the loss function exhibits non-monotonic behavior. Further details on both approaches provided respectively.
in Appendices D.I and D.2
The Learn then test (LTT) framework addresses these limitations by reformulating risk control multiple hypothesis testing problem (Angelopoulos et al., 2025), enabling finite-sample guaranteess without monotonicity assumptions. Unlike CRC, which directly bounds the expected risk, LTT provides high proobprrbbbilitty guarantees: with probability of at least - 8 over different draws of the calibration data, the selected À satisfies the risk constraint. Consider calibration dataset {(x(1, } where E X' and Y Y denote feature-response pairs. Given parameterized prediction set function a X - 22 and bounded loss function é 2 x y - [0, B), LTT operates on discrete parameter grid A = {A1, A2..... ) and associates each A with null hypothesis H R(A;) > a, where R(A,) ) = (,, Y)] is the expected risk. Rejecting NG indicates that X achieves the desired risk level. Testing is performed based valid p-values, which can be computed using the Breffding-Bentkeee inequality (Appendix (A.4)
A..4) applied to the empirical risk
over the calibration set:
R.m (A₂) 1 Σ eaT (xO)),
(3)
Since multiple hypotheses are tested simultaneously, it is necessary to control the family-wise error rate (FWER), i.e., the probability of making least one false rejection. One classical FWER- controlling procedure is the Bonferroni correction, which adjusts the significance level by dividing the threshold å by the number of tested hypotheses. However, this global adjustment is often con- servative and may substantially reduce statistical power. An alternative approach is to apply fixed sequence testing (FST), which exploits any known natural ordering A, for instance from low to high risk, by testing the hypotheses sequentially. The procedure proceeds along this ordering and stops at the first non-rejection. In this way, it controls the FWER while typically achieving greater statistical power than the Bonferroni correction.
The LTT guarantees are formally stated the following theorem.
Theorem (LTT Finite-Sample Guarantee (Angelopoulos et al. 2025)). Let Pj a valid p-value for each null hypothesis N₂ R(Aa) > i.e., P(p, S u) I under N₂ for all u E [0, 1]. Then for any FWER conetolllgoo algorithm level 6, the rejection Å A(p1.... .Pp)....PP)) .Pk) satisfies:
P (R(A) S 2 -
(4)
where the supremum over an empty set is defined as -0.
This guarantee holds without requiring monotonicity assumptions, making LTT particularly well suited to our setting, where the loss function may exhibit complex and non-momotone behavior with respect to X. Nevertheless, the FST procedure is still effective in practice, since monotonicity is satisfied in most cases.
METHOD
5.1 ADAPTIVE HYPOTHESIS SELECTION VIA RISK CONTROL
Building on the GER framework, presented in 33 we propose an adaptive selection mechanism that dynamically estimates the optimal number of hypotheses for each input sample. Rather than
4

---

Published as a conference paper at ICLR 2026
using fixed set of size N, approach selects the smallest set satisfying the risk constraint, thereby reducing computational cost.
Adaptive hypothesis set. We formulate an adaptive hypothesis selection problem within the LIT framework, established in $ $DD.2.] We define adaptive hypothesis sets, parametrized by as:
rx(Hx) {(01 q), .... (Dre G,)
(5)
where n is the adaptive set size determined according A:
j Σ S 2
7 min j:
(6)
and s (,1,...,8N) represents normalized confidence scores derived from ASR log- oogllikelhoood
scores c = (CI,..., C.). The enhanced pipeline becomes j" = M (T (Hw): (), where Å the calibrated threshold for controlling expected performance degradation. This approach maintains compatibility with any pre-trained H2T model while reducing computational overhead through principled uncertainty quantification.
Risk function and LTT. Our loss defined with respect to word error rate (WER), which standard metric used for evaluating ASR performance. The WER quantifies transcription accuracy by measuring the minimum number of word-level edits required to transform the predicted transcription into the ground truth:
WER S(Ú.0) + D v) I(..9)
W
where S(). v), D(. y). and (9. y) represent the number of substitutions, deletions, and insertions, respectively, and Wy) is the total number words the reference transcription.
Rather than controlling absolute WER, which requires domain-specifer thresholds, control the per-sample relative degradation from the best achievable performance with fixed-set sizes up to N:
((T ((Tx(HN): v) (TER(M (HWx), y) min WFR(MMuT(HH;): JE(N]
where H, (i)...).... ....,,,,, (6;, denotes the top-j hypothesis set.
This loss function exhibits predominantly monotonic behavior, where enlarging the hypothesis set typically does not worsen performance. Our adaptive selection can identify cases where smaller sets are sufficient even beneficial. In the worst-case scenario, selecting all N hypotheses converges to the standard fixed-N baseline performance, ensuring no performance degradation from existing methods. Finally, our risk control objective follows the LTT framework:
P (F(CCT) (HN). Y)) S a) 1 -5 5,
where is selected from the rejection set obtained by the LTT procedure, ensuring finite-sample control of the expected performance degradation. Our method is summarized in Algorithm T
Score definition. The selection mechanism relies on a composite score derived from ASR log- likelihoods, designed to adapt flexibly varying dataset characteristics:
(x((c) T
= softmax
(10)
Here, dh, denotes an adaptive normalization funetion and 7 a temperature parameter. The function é, interpolates between two transformation regimes through single parameter 7, enabling the score to adjust to dataset-specific speech quality. To prevent redundancy, penalties are applied when the ASR system generates repeated hypotheses. Further details the adaptive normalization, design rationale, and repetition handling are provided in the Appendix A.I
Score Agnostic. It is important to note that method is independent of the specific choice of the score used to define the adaptive set. Previous research in ASR has shown that likelihood values do not always provide reliable measure of confidence (Li et al. 2021 Ravi al. 2024). While
5

---

Published as a conference paper at ICLR 2026
Algorithm Learn then Test Selection Procedure
Require: Calibration set {(H))) ( parameter grid A {Yz, A2, dek ordered from most to least conservative (i.e., A1 2 22 2 2 At), error level 6.
1: for to k do
2: Compute empirical risk Rm (A,) - ((T,, (HL)), y0)
3: Calculate p-value Pj using Hoeffding- Bentkku inequality
4:
p; > then
5:
if = then
6:
return failure (no valid found)
7: end if
8: return À = Az-1
9: end if
10: end for
return Ā A
ei al. (2021) Ravi et al. (2024) focus on top-label calibration, approaches such as that Popor- danoska et al. (2022) offer canonical calibration, which enables the generation of confidence scores for several top hypotheses that can be seamlessly integrated with our framework. Nevertheless, for simplicity, demonstrate method using the commonly available likelihood values.
Multi- malti-parameter extension. While our method optimizes hypothesis set sizes through adaptive thresholds, recent work multi-objective optimization for risk control suggests combined opti- mization and testing approach in the case of multiple parameters that influence multiple objective functions. Pareto Testing (Lauser-Goldshtein et al., 2202)) provides an efficient framework for si- multaneously calibrating multiple hyperparameters while maintaining statistical guarantees on con- strained objectives and optimizing additional free objective function (e.g., set size). In our context, this could enable joint selection of the normalization parameter y, temperature т, and threshold X, trade-offs than our current approach of fixing and r based on dataset characteristics. Importantly, this joint optimization approach could en- able deployment across diverse acoustic conditions without requiring prior knowledge that currently guides our parameter selection strategy. We explore this extension in Appendix D.3
potentially discovering better performance-efficienee
5.2 THEORETICAL CONSIDERATIONS
While risk control frameworks provide principled methods for uncertainty quantification, our ASR application operates under conditions that require careful consideration of theoretical assumptions. We address these considerations and their practical implications.
Bounded loss. Risk control frameworks require bounded loss functions; while WER is theoret- ically unbounded, relative WER degradation is bounded in practice. We enforce boundedness by clipping the loss B 1. 1.25, i.e., {(T;, & (Hx), v) B, threshold above which fewer than 0.1% of validation samples fall. As the test set follows a similar distribution, this approximation introduces negligible bias into the theoretical guarantees.
Monotonicity. While some risk control methods, like CRC, require monotone loss functions, the monotonicity violations in our application (~20% of cases) represent precisely the efficiency opportunities our adaptive method exploits scenarios where smaller hypothesis sets genuinely outperform larger ones. To maintain theoretical guarantees without monotonicity constraints, we employ the LIT framework, which handles non-monotone losses naturally through sequential hypothesis testing, ensuring our approach provides rigorous finite-sample bounds regardless of monotonicity violations. Nevertheless, non-monntonicity can still affect LTT efficiency in practice: since fixed sequence testing advances from the most conservative À toward less conservative values and stops at the first failure, non-monotone behavior may cause early stopping, yielding larger hypothesis sets than necessary. This does not compromise theoretical guarantees but can reduce computational savings. The Pareto Testing extension (Appendix D.3) further addresses this limitation by ordering hypotheses in a nearly monotonic sequence on the optimization set, mitigating the impact of local non-monotonicity the search trajectory.

---

Published as a conference paper at ICLR 2026
6 EXPERIMENTAL SETUP
Datasets and Benchmarks benchmark (Chen al. 2023). spanning different acoustic difficulty levels based on average WER performance:
We evaluate our approach on three datasets from the HyPoradise
3 (Hernandez et al. 2018) (avg. WER 9.5%) contains TED Talk recordings with
diverse noise, accents, and topics. Following HyPoradise protocol, we sample 50,000 utterances: 35,500 for training/validation, and 14,500 for calibration/test.
CHiME-4 Vinccent et al. 2017) (avg. WER 11 11.5%) contains far-field noisy recordings across different environments. We use the complete train split (9,600 utterances) for train/validation and test-real split (1,320 utterances) for calibration/test. Data was obtained from RobustGER (Hu
et al. 2024a). which provides the required ASR likelihood scores.
Common Voice (Ardila et al. 2020) (avg. WER ~ 12.5%) contains multilingual recordings from diverse speakers with different accents. We select 50,000 samples from train-en split using 35,000 samples for train/va hali/valiattion, and 15,000 samples for calibration/test.
ASR Hypothesis Generation. We employ Whisper models (Radford et al. 2023) for N-best hypothesis generation via beam search, removing repetitive utterances and selecting top-5 (N 5) hypotheses by posterior probability. Tedl Jium-3 and Common Voice use Whisper-base (beam- width is 60, following HyPoradise (Chen al., 2023 ). while CHiME-4 uses Whisper-Large-v2 (beam-width is 50, following RobustGER (Hu et al. (024a)).
LLM and Training. We fine-tune LLaMA-2-7B (Touvron et al. 2023 using LoRA (Hu al. 2022) for efficient H2T mapping. The model generates corrected transcriptions from N-best inputs via standard next-token prediction. Training details, hyperparameterss and computational requirements are in Appendix
Risk Calibration. We use the validation data to determine both the target risk levels (a) and the dataset-specific score function parameters (7 and 7), based on the empirical performance and score discriminability patterns. The selection methodology and theoretical considerations are discussed in Appendix
Table 1: LTT WER (%) results with 2--7 fine-tuning, Baseline: Whisper's top-1 hypoth- esis. Olm: post-LLM oracle. Subscript percentages denote relative WER change vs. vanilla GER (WER column) and relative size reduction vs. constant N = (size column).
Test Set
Baseline
GER
LIT Method
a (%)
6
Success Rate
Oum
Set Size
WER
-um-3
9.3
7.53
2.48
50.08%
7.52 0.13%
2.3
0.10
0.94
5.58
CHiME-4
11.49
6.24
3.866
22.68%
6.37 +2.0%%
2.7
0.25
0.98
4.71
Common Voice
12.44
8.32
3.29
34.2%
8.51 +2.28%
1.9
0.10
0.92
6.96
6.1 EVALUATION
Experimental Protocol. To ensure statistical reliability, we perform T = 30 independent trials with resampled calibration/test splits, allocating 30-50% of test samples for calibration. We set according to the calibration set size, using smaller values for larger datasets where the Hoeffding- Bentkus inequality yields tighter v-value bounds, and larger values for smaller calibration sets to avoid overly conservative thresholds. For the target risk level a. we report only configurations where valid was found during calibration, i.c., where is achievable given the model and hypothesis set. For instance, very low a values are typically infeasible since even the most conservative selection (always N 5) incurs some irreducible degradation from the oracle. Final results represent mean values across all valid trials.
Performance Measurements. We evaluate approach using WER the primary metric, as described in $ 5 We employ two complementary WER calculation methodologics. The primary approach performs instance-level computation followed by averaging across samples, directly
7

---

Published as a conference paper at ICLR 2026
corresponding to the defined loss function in Eq. (8) As secondary validation, we compute corpus- level WER through concatenation of all predictions and references, which reduces sensitivity to sample length variability and enables comparison with prior works using corpus-level conventions.
Risk Control Validation. Beyond standard WER evaluation, we validate the empirical effective- ness of our LTT framework by tracking the success rate of risk control across independent trials. For each dataset, we report the proportion of trials where the risk constraint R(A)= S or is satisfied, validating the sioh-prbbbbility bound P P(R(A) S a) 2 8 empirically.
RESULTS AND ANALYSIS
7.1
MAIN RESULTS: PERFORMANCE, EFFICIENCY, AND RISK CONTROL
Baseline and GER Performance. Table D presents experimental results across datasets. The baseline performance corresponds to Whisper's 's top-1 hypothesis, establishing the initial recognition accuracy before post-processing The GER results demonstrate the effectiveness of the fine-tuned LLLMA-2-7B model when provided with fixed set of top-5 hypotheses. For reference, we include the oracle bound Ottm. which represents the best possible performance when the L.M receives the optimal number of hypotheses for each sample (between - 5).
The results show that GER achieves substantial improvements over the baseline across all conditions, with gains varying according to dataset difficulty. Our adaptive selection framework demonstrates superior computational efficiency while preserving or enhancing correction quality. Our method dynamically determines the optimal number of hypotheses for each input, as reflected in the average set sizes reported. These results validate the effectiveness of our approach across diverse acoustic conditions. On TedLium-3, the method achieves 51 % reduction in average set size while improving performance. CHiME-4 and Common Voice demonstrate computational savings of 23% and 34% with modest performance trade-off of 2% relative increase in WER. Notably, adaptive selection sometimes outperforms the fixed-size baseline, suggesting that including too many hypotheses may introduce noise rather than useful information.
ddaptive Selection Results.
Empirical Guarantee Verification. Regarding the reliability of our selection mechanism, success rates consistently exceed the theoretical minimum of 1- across all trials, confirming that the high-probability bound P(R(Å) S a) 2 1 8 is satisfied in practice a property absent in prior methods and made possible through the LTT framework. The observed gap between empirical success rates and the theoretical minimum is expected, reflecting the conservative nature finite- sample Hoeffding- Bentkus bounds; as validated by experiments with varying calibration set sizes, this gap narrows as more calibration data becomes available.
Performance- Computte Trade-off. Figure 2 illustrates the performance-comee compute trade-off characteristics of our adaptive approach compared to fixed set sizes across all datasets. Each subplot displays the WER performance curve for constant set sizes N through N 5, with vertical ref- erence line indicating the Ojim oracle performance bound. We observe that our method's operating points consistently demonstrate better tradeoff's relative to the fixed-set performance curve, achiev- ing computational efficiency gains while maintaining competitive or even improved error rates.
7.2 ANALYSIS
To illustrate the adaptive selection mechanism, Table 2 presents three representative cases, highlighting how score distributions relate to optimal set sizes.
Case 1: Full Set Required (Common Voice): When ASR scores exhibit narrow gaps (-0.42 to -0.51), our method correctly identifies the need for comprehensive information. The LLM progressively refines its prediction across set sizes, ultimately achieving perfect accuracy with the complete hypothesis set by correctly generating "gastroliths" rather than the various incorrect alternatives ("gallstones," " "gastrolytes"). The compressed score distribution leads our normalization to select larger sets, aligning with the empirical benefit of additional hypotheses.
8

---

Published as a conference paper at ICLR 2026
15
33
3.4
24
TedLium-3
(b) CHiME-4
(c) Common Voice
Figure 2: Performance-comee compute trade-offs across datasets. Plots show WER vs. set size for constant set sizes (connected line), oracle performance level (dashed line), and our adaptive method working points (markers), where marker colors encode corresponding values.
Representative examples showing the relationship between ASR score distributions and optimal set sizes. Case demonstrates progressive improvement with larger sets, Case 2 shows degradation beyond the optimal single hypothesis, and Case 3 illustrates performance plateau enabling computational savings.
Table 2:
Cuse
Hypotheses
Score
L.L Predictions by Set Size
WER per Size (%)
Case Full Set
Hi: calcali be confused withs gadaao lits
-0.42
Sizc wilh gallstonce
12.5
calenii be confosed with pastrolttes
-0.44
petrolyten
12.5
gustrolyte
calcuil
gastrllets
gustrolytes
12.5
gooteollli
corlored gammoints
Cuse 2: Single Opt.
H1: medicall team usign of the ship.-
.021
Size leam sign of...
0.0
H2: modiical tomm sign the ship
-0.11
Size 2-5 trom assigned io...
21
HR: mesiicl tomm nosggna the ship...
-0.37
21
[[A: ...edical team assigned the ship
0.41
21
H5: medeialll team usiguad the ship..
2043
21
medical trom sign lie skin..
sew york
seventy sevea point million
sepantely york ahons seventy soxea
mone york
about severty wcure point million
..ae it suicss
seuuniy soven point une millian
paint million cemtiicates of porticpptioo
new york sare sold about seventy seven point one million of
-0.49
6.25
H5: ..eew yark sure sold about seventy point ouz seven million dollars in...
-0.49
6.25
G7 weenty zeeen point oor million deliurs of...
Case 2: Single Hypothesis Optimal (TedLium-3): When the top hypothesis achieves perfect accuracy and exhibits substantial score separation (-0.21 vs -0.31), additional hypotheses degrade performance from 0% 21% WER. this case, the discriminative score gap correctly signals high confidence in the first hypothesis, leading our method to favor minimal sets. This demonstrates that additional hypotheses can introduce harmful noise.
Case 3: Performance Plateau (CHiME-4): When WER remains constant (6.25%) across all set
sizes, our method demonstrates computational efficiency potential. While the tight score clustering (-0.46 to -0.49) would typically lead our normalization to select larger sets, this case illustrates where our approach provides safety net-in the worst case, we select all hypotheses and achieve identical performance to the baseline, but when score normalization successfully identifies the plateau, we achieve the same WER with reduced computational cost.
These examples show that adaptive selection produces smaller sets when scores are highly discrim- inative and larger sets when scores are compressed. This supports dynamically adjusting set sizes based on the underlying score distribution rather than relying fixed configurations.
7.3 ABLATION STUDIES
We briefly report several ablation studies that performed validate different aspects of our proposed framework.
Alternative Problem Formulations. Our preliminary experiments evaluate multiple CP and risk control methods including absolute WER targets, coverage-bard baaeed objectives for samples below specified WER thresholds. and bounded-WEL -EE hypothesis guarantees, following approaches from prior ASR uncertainty quantification works (Ernez et al., (203). These alternatives consistently yielded inferior empirical performance compared our relative loss, defined in Eq. g Absolute

---

Published as a conference paper at ICLR 2026
WER targets operate global level without instance-speciife optimization, while bounded- WoER guarantees (Ernez et al. 2023) showed poor correlation between hypothesis quality and final LLM output quality, motivating relative degradation formulation that adapts to each sample's achievable performance range.
We examined our choice of training the LLM with constant-5 hypothe- sis sets, while evaluating with variable set sizes. To this end, we conducted comprehensive ablation experiments training separate LLaMA-2-7B models on fixed set sizes (I-5 hypotheses), as well as dynamic sizes, then evaluating each model across all possible test set sizes. The results are reported in Tab. B.2 This 6 x result matrix reveals that while specific combinations (e.g., train-3/ter -/te-3/test-3) occasionally outperformed the baseline, the constant-5 trained model achieved optimal average WER across all test configurations. These results confirm that our adaptive approach provides genuine improvements over the best achievable fixed-size baseline, establishing the validity of our comparative framework.
Training Set Size Analysis:
Scalability to Larger Models and Zero-Shot Settings. To evaluate generalizability beyond our LLaMA-2-7B baseline, we conducted experiments with LLaMA-2-13B (fine-tuned) and GPT- 3.5-turbo (zero-shot prompting). Results demonstrate that our framework maintains consistent citormance-effciennee trade-offs across both larger model scales and deployment scenarios where fine-tuning is not feasible. The computational savings persist relative the increased inference costs, confirming that our adaptive selection mechanism provides value across different model ar- chitectures and prompting paradigms. This addresses practical deployment considerations where model size and training constraints vary significantly. Detailed results are provided in Appendix(B2. B.2
Cross-Domain Extension. We extended our framework to speech translation tasks using the Gen- Translate paradigm on multilingual datasets. with some methodological adaptation. Results show successful transfer with substantial computational savings while maintaining competitive translation quality. This cross-domain validation demonstrates broader applicability beyond ASR to generative error correction scenarios involving N-best hypothesis integration, directly addressing the broader impact potential in the GER community. Full methodology and results in Appendix [.44
CRC Implementation. We also implemented the CRC framework as an alternative risk control method. While CRC lacks theoretical guarantees due to monotonicity violations in our application (~ 20% of cases), it achieves similar empirical performance to our LTT approach. This demonstrates that both frameworks effectively exploit the same underlying adaptive selection patterns, with LTT providing the additional benefit of rigorous theoretical validation. The CRC implementation and comparative analysis are detailed in Appendix D.2
Optimization via Pareto Testing. We explored an extension of our framework that jointly optimizes all three parameters (% T, A) simultaneously using Pareto Testing (Laufer- Goldshtein et al.] 2023), eliminating the need for dataset-specific parameter pre-selection. Results demonstrate consistent performance-effcciene, trade-offs with main LTT approach, while dis- covering parameter combinations that achieve superior Pareto frontiers compared to -arammerree baselines, offering increased adaptability to unseen acoustic conditions. Full methodology and sults are provided in Appendix D.3
CONCLUSION AND FUTURE WORK
This work presents an adaptive framework for hypothesis selection in generative ASR error cor- rection, addressing computational inefficiency through principled uncertainty quantification. Our method employs LTT dynamically determine optimal hypothesis set sizes, providing rigorous theoretical guarantees with high- probbbblyyy bounds while demonstrating substantial computational savings and maintaining competitive performance across datasets with diverse acoustic conditions. Validation across larger language models, zero-shot settings, and cross-domain speech translation tasks confirms the framework's 's broad applicability and robustness across different deployment sce- narios. The framework requires only calibration without model retraining, enabling straightforward adoption in existing systems. Future work could investigate confidence-drivee adaptive compute allocation in multi-mode model systems, including reasoning and agent- based applications, where similar mechanisms for identifying and reducing computational costs may achieve comparable performance with greater efficiency.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This work was supported by the Israel Science Foundation (Grant No. 2094/25), and the Israeli Ministry of Innovation, Science and Technology (Grant No. 1001818518).
REFERENCES
Léo Andéol, Thomas Fel, Florence De Grancey, and Luca Mossina. Confident object detection via conformal prediction and conformal risk control: an application to railway signaling. In Conformal and Probabilistic Prediction with Applications, pp. 36-55. PMLR, 2023.
Anastasios N Angelopoulos, Rina Foygel Barber, and Stephen Bates. Theoretical foundations of conformal prediction. arXiv preprint arXiv:2411.11824, 2024a.
Anastasios N Angelopoulos, Stephen Bates, Emmanuel Candès, Michael I Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. The Annals of Applied Statistics, 19(2):144-1662. 2025. doi: 10.1214/24A0ASSII
Anastasios Nikolas Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Con- formal risk control. The Twelfth International Conference Learning Representations, 2024b.
Rosana Ardila, Megan Branson, Kelly Davis, Michael Henretty, Michael Kohler, Josh Meyer, Reuben Morais, Lindsay Saunders, Francis M. Tyers, and Gregor Weber. Common voice: A masiveey-mullliiguual speech corpus. In Proceedings of the 12th Conference Language Re- sources and Evaluation (LREC 2020). pp. 4211 1111-225,22 2020.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations. Advances in neural information processing systems, 33: 13:12449-12460, 2020.
Margarida M. Campos, António Farinhas, Chrysoula Zerva, Mário A. T. Figueiredo, and André F. T. Martins. Conformal prediction for natural language processing: A survey. Transactions of the Association for Computational Linguistics, 12: 1297-111 2024. doi: (0.1162/tac
Chen Chen, Yuchen Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Pin-Yu Yu Chen, and Eng- Siong Chng. Hyporadise: An open baseline for generative speech recognition with large language models. Advances in Neural Information Processing Systems, 36:31665-31688 2023.
Fares Ernez, Alexandre Arnold, Audrey Galametz, Catherine Kobus, and Nawal Ould-Amer. Ap- plying the conformal prediction paradigm for the uncertainty quantification of an end-to-end au- tomatic speech recognition model (wav2vec 2.0). In Proceedings of the Tweifth Symposium on Conformal and Probabilistic Prediction with Applications, volume 204 of Proceedings of Ma- chine Learning Research, pp. 16 35. PMLR, 2023.
Sreyan Ghosh, Mohammad Sadegh Rasooli, Michael Levit, Peidong Wang, Jian Xue, Dinesh Manocha, and Jinyu Li. Failing forward: Improving generative eπror correction for asr with synthetic data and retrieval augmentation. arXiv preprint arXiv:2410.13/99 2024.
François Hernandez, Vincent Nguyen, Sahar Ghannay, Natalia Tomashenko, and Yannick Estève. Ted-lium 3: twice as much data and corpus repartition for experiments on speaker adaptation. arXiv preprint arXiv: 006994699.9....000000 2018.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation large language models. In International Con- ference on Learning Representations, 2022.
Yuchen Hu, Chen Chen, Chao-Has -an Huck Yang. Ruizhe Li, Chao Zhang, Pin-Ye Chen, and Eng Siong Chng. Large language models are efficient learners of noise-robust speech recog- nition. In International Conference on Learning Representations, 2024a. URL hetps:
//arxiv..orq/aas//2200 10446
11

---

Published as a conference paper at ICLR 2026
Yuchen Hu, Chen Chen, Chao-Han Huck Yang, Ruizhe Li, Dong Zhang, Zhehuai Chen, and Eng Siong Chng. Gentranslate: Large language models are generative multilingual speech and machine translators. 2024b.
Hamza Kheddar, Mustapha Hemis, and Yassine Himeur. Automatic speech recognition using ad- vanced deep learning approaches: A survey. Information fusion, 109:102422, 2024.
Bracha LaussenGol Adam Fisch, Regina Barzilay, and Tommi Jaakkola. Efficiently control- ling multiple risks with pareto testing. In The Eleventh International Conference on Learning Representations, 2023. BRLhttps:: :eetpp::/openreeeee
net/ wtw.eeeeetoemnnn///eeeeee.
BeF
Qiujia Li, David Qiu, Yu Zhang, Bo Li, Yanzhang He, Philip C Woodland, Liangliang Cao, and Trevor Strohman. Confidence estimation for attention-
Sequence-to-sequeee models for speech recognition. In ICASSP 2021-2021 IEEE International Conference on Acousties, Speech and Signal Processing (ICASSP), pp. 6388-6392. IEFE, 2021.
Yanyan Liu, Minqiang Xu, Yihao Chen, Liang He, Lei Fang, Sian Fang, and Lin Liu. Denoising ger: A noise-robust generative error correction with Ilm for speech recognition. arXiv preprint arXiv:2509. 0:X1::2509004392, 2025.
R. Ma, M. Qian, M. Gales, and K. Knill. Ast error correction using large language models. IEEE Transactions Audio, Speech and Language Processing, 33:1389-1401, 2025. doi: 10.1 109/ TASLPRO.2025.3551 355500083..
Rao Ma, Mengjie Qian, Mark Gales, and Kate Knill. Ast error correction using large language models. arXiv preprint arXi::2409.09555 2024.
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan. PEFT: State-of-the-art parameter-efficient fine-tuning methods. hhttps://githun 2022.
com commhhuggingface/ppee
Bingshen Mu, Kun Wei, Pengcheng Guo, and Lei Xic. Mixture of lora experts with multi-modal and multi-granularic Ilm generative error correction for accented speech recognition. IEEE Transac- tions on Audio, Speech and Language Processing, 2025.
William Overman, Jacqueline Vallon, and Mohsen Bayati. Aligning model properties via conformal risk control. Advances in Neural Information Processing Systems, 37:1 1111110022-11111 2024.
Teodora Popordanoska, Raphael Sayer, and Matthew Blaschko. A consistent and differentiable 1p canonical calibration error estimator. Advances in Neural Information Processing Systems, 35: 7933-7946, 2022.
Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and llya Sutskever. Robust speech recognition via large-scale weak supervision. In International conference on ma- chine learning, pp. 28492-288 PMLR, 2023.
Srijith Radhakrishnan. Chao-Han Huck Yang. Sumeer Ahmad Khan, Rohit Kumar, Narsis A. Kiani, David Komez-Cabrerce and Jesper N. Tegner. Whispering Ilama: cross-modal generative error correction framework for speech recognition. In Proceedings of the 2023 Conference Empir- ical Methods in Natural Language Processing, pp. 10007-10016, Singapore, 2023. Association for Computational Linguistics.
Nagarathna Ravi, Vipul Arora, et al. Teles: Temporal lexeme similarity score to estimate confidence in end-to-end IEEF/ACM Transactions on Audio, Speech, and Language Processing, 32: 2024.
Lars Rumberg, Christopher Gebauer, and Jöm Ostermann. Aggregation-free frre uncertainty estimation for etc-based automatic speech recognition. IEEE Transactions on Audio, Speech and Language Processing, 2025.
Steffen Schneider. Alexei Baevski, Ronan Collobert, and Michael Auli. wav2vec: Unsupervised pre-training for speech recognition. Proceedings Interspeech, pp. 3465-3469, 2019.
12

---

Published as a conference paper at ICLR 2026
Yuanfeng Song, Di Jiang, Xuefang Zhao. Qian Xu, Raymond Chi- Chi-Wing Wong, Lixin Fan, and Qiang Yang. L2rs: A sorninggto-reecome mechanism for hybrid speech recognition. In Proceedings of the 29th ACM International Conference Multimedia, pp. 3474-3482. ACM, 2021.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint 2.ii:2337.002288 2023.
Emmanuel Vincent, Shinji Watanabe, Aditya Arie Nugraha, Jon Barker, and Ricard Marxer. An analysis of environment, microphone and data simulation mismatches in robust speech recogni- tion. Computer Speech Language, 46:535-557, 2017.
Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in a random world, volume 29. Springer, 2005.
Tomer Wullach and Shlomo E Chazan. Don't be so sure! boosting asr decoding via confidence relaxation. In Proceedings of the AAAI Conference Artificial Intelligence, volume 37, pp- 13780-13788, 2023.
Yuxin Xiao, Paul Pu Liang, Umang Bhatt, Willie Neiswanger, Ruslan Salakhutdinov, and Louis- Philippe Morency. Uncertainty quantification with pre-trained language models: A large-scale empirical analysis. In Findings of the Association for Computational Linguistics: EMNLP, pp. 7273-7284, 2022.
Yunpeng Xu, Wenge Guo, and Zhi Wei. Conformal risk control for ordinal classification. In Uncer- tainty in Artificial Intelligence, pp. 2346-2355, PMLR, 2023.
Yunpeng Xu, Mufang Ying, Wenge Guo, and Zhi Wei. Two-stage risk control with application to ranked retrieval. arXiv preprint vrXiv:2404.17760 2024.
C.-H. Huck Yang, Gu, Y.-C. Liu, S. Ghosh, Bulyko, and A. Stolcke. Generative speech recognition еπтог correction with large language models and task-activating prompting. In 2023 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 1-8, 2023. doi: 10. 10.1109/ASRU5596442200 2023 10389673.
C.-H. Huck Yang et al. Large language model based generative error correction: challenge and baselines for speech recognition. speaker tagging, and emotion recognition. In 2024 IEEE Spo- ken Language Technology Workshop (SLT), pp. 371-378, 2024. doi: 10.1109/SLT61566.2000 10832176.
A RISK CONTROL IMPLEMENTATION DETAILS
A.1 SCORE FUNCTION DESIGN
A.1.1 MOTIVATION
Our analysis revealed that score distributions vary significantly across datasets with different noise characteristics. Higher signal-to-noise ratio conditions (c.g.. TedLium-3) produce more discrimina- tive ASR confidence scores, while challenging acoustic environments (c.g.. Common yicld compressed score distributions. Temperature-only adaptation proved insufficient, creating overly homogeneous distributions that degraded selection quality. Consequently, developed the two-level normalization strategy with parameter 7, enabling adaptive score transformation based on dataset difficulty while maintaining robustness across parameter choices.
A.1.2 FUNCTION DESIGN
The normalization function (c(() smoothly interpolates between two transformation regimes based on a single parameter E [0, 1]:
(c(c) (1 -)7 ( (1-7)
77.
(11)
13

---

Published as a conference paper at ICLR 2026
1000
250
530-
1890
1990
1590
2000
750
2003
Figure A.1: ASR confidence score distributions for TedLiumm3 (left) and Common Voice (right) across processing stages: raw scores (top), softmax normalization (middle), and full transformation (bottom).
where -1 denotes element- wise reciprocal operation, i.e., (c-1); 1/c, for each component
The identity transformation (y 1) preserves natural ASR score differences for high-SNR condi- tions, while the reciprocal transformation (7 = 0) amplifies small differences between compressed scores for challenging acoustic conditions. The parameter y controls smooth transitions between
these regimes.
A.1.3 PARAMETER SELECTION STRATEGY
We base parameter selection signal-to-noise ratio characteristics and empirical validation.
Figure A. illustrates score evolution across processing stages for different acoustic conditions. TedLium-3 high-SNR conditions produce naturally discriminative scores, requiring only sharp- ening via low temperature (r = 0.05) while preserving relationships (7 = 1.0). Common Voice's challenging conditions with compressed distributions require reciprocal amplification (7 0.0) fol- lowed by moderate temperature (r = 1.0). CHiME-4 represents intermediate conditions requiring
balanced transformation (? 0.5).
To validate this SNR-based rationale, we conducted systematic grid testing across the parameter space. Figure A.2 shows success regions where the method achieves valid risk control with better performance-comee compute trade-offs than fixed baselines.
The heat-maps confirm that successful regions align with our SNR- parameter selection, with edge cases (high/low SNR) showing concentrated success areas and intermediate conditions requir- ing parameters within mid-value ranges.
14

---

Published as a conference paper at ICLR 2026
(a) TedLium-3
(b) CHiME-4
CommonVoicc
Figure A.2: Parameter selection grid test results showing success regions across (7, parameter space. Success defined as achieving valid risk control with superior performance-compute trade- offs compared to fixed baselines. Heat-maps confirm that successful regions align with SNR-based parameter selection strategy.
A.1.4 ROBUSTNESS ANALYSIS
Grid testing across uniformly sampled parameter combinations demonstrates method robustness. Success criteria require: (1) valid selection with controlled risk, and (2) superior performance- compute trade-offs compared to fixed-set baselines.
Results show overall success rate 70% (61/88 combinations), with per-dataset rates: 3: 80% (24/30), CHiME-4: 62% (18/29), Common 66% (19/29). This indicates parameter selection is an optimization step rather than critical requirement, providing practitioners ficxibility while maintaining performance guarantees.
A.1.5 AUTOMATED PARAMETER SELECTION
To automate parameter selection for deployment
unseen acoustic conditions, investigated the relationship between score distribution characteristics and working point success. Our intuition that meaningful working points should exhibit score vectors that respond differently across vary- ing input difficulties, with entropy serving as natural measure of this distributional behavior.
Analysis confirmed that score vector entropy serves reliable predictor of parameter effective- ness successful working points exhibit distinct entropy patterns compared to failed configurations. Based on this observation, we developed simple entropy-basee rule: thresholding score vector entropy at 4.85 effectively discriminates between suitable and unsuitable parameter combinations.
We validated this rule on previously unseen parameter pairs, achieving prediction precision exceed- ing 80% and Fl scores above 75%. This enables practitioners to assess parameter suitability on new datasets without extensive manual calibration, providing a practical deployment pathway that transforms manual parameter tuning into principled selection with automated validation.
should noted that parameter sensitivity increases when targeting tighter performance bounds (lower and values), as the method operates in narrower feasible regions where precise parameter selection becomes more critical.
1.6 HANDLING HYPOTHESIS REPETITIONS
When the ASR system produces fewer than N unique hypotheses, repeated hypotheses receive exponentially decaying scores to avoid overweighting redundant information:
Ri, s; 8",
(12)
where Rgr is the adjusted score for hypothesis i with repetition count r, and (0, is the decay factor. This mechanism is based on the assumption that repeated hypotheses provide no additional information for well-calibrated models, which validate empirically in our experiments.
15

---

Published as a conference paper at ICLR 2026
A.2 TARGET RISK
A.2.1 RISK TARGET CALIBRATION
We establish target risk levels based on the performance range achievable through fixed hypoth- esis set selection. The feasible degradation range spans from lixed---hypothesis (worst case) to ixyd-t-hypotheses (best case) performance. Since our adaptive method dynamically selects smaller sets with minimal degradation, targeting risk bounds within this empirically range repre- sents the natural operating regime. These empirically-derive [1.53, 5.76] for CHiME-4, and [1.38, 3.74) for Common Voice. We uniformly sample target risk val- ues within the validated degradation range. The specific choice within this range determines the desired performance-comppe efficiency, while values approaching 5-hypotheses' risk emphasize performance preservation. The different risk-control frameworks require different c-calibration approaches. For CRC, we directly a. On the other hand, LTT and Pareto-testing, provide high-probability bounds Pr (T(B()) S <a) 2 11
ranges are [1.95, 3.8) for TedLium-3
trade-off:: values closer I-hypothesis' risk prioritize computational
use validation-derived values as the target expected risk levels, since CRC bounds (E[
meaning that represents approximately the (1 b) percentile of the empirical risk distribution rather than its expectation. To achieve equivalent average performance, those frameworks therefore require values that are higher than the target expected empirical risk. As the calibration sizes smaller, the Hoeffding- Beeeffiing-Bentuus Bentkus inequality produces more conservative p-values, leading to more conservative set selections with higher empirical coverage. In practice, LTT and Pareto typically yield valid selections for a values from the upper part of the feasible range, while CRC can achieve formulation. We report results
valid selections across the full range due to its expectation-base only for values that demonstrate successful risk control within the validated range, ensuring both theoretical validity and practical utility across different performance-eflicinnyy preferences.
A.2.2 PERFORMANCE TRADE-OFFS
The target risk directly controls the performance-compe trade-off by determining acceptable expected relative WER degradation from oracle performance. Lower a values (tighter bounds) require larger hypothesis sets achieve the target risk level, due average WER monotonicity on average, while higher a values (looser bounds) enable smaller sets with acceptable performance degradation.
Table A.I: Effect of risk tolerance performance-compuut trade-offss (Common Voice dataset).
Risk Tolerance (a:
Avg. Set Size
WER (%)
0.21 (tighter)
3.73
8.53
0.22
3.49
8.59
0.23
3.14
8.66
0.24 (looser)
2.99
8.73
Table A. demonstrates this relationship empirically. As increases, average set sizes decrease, while WER increases as well. The different operating points in Figure _ correspond to varying selections within these validated ranges.
A.3 OTHER PARAMETERS SELECTION
We set additional framework parameters based on validation analysis to ensure robust performance across datasets. The repetition penalty 3 = 1.25 handles duplicate hypotheses by applying exponen- tial decay to repeated entries, preventing overweighting of redundant information. The loss bound B = 1.25 accounts for rare cases exceeding 100% relative WER degradation, satisfying the bounded loss requirement for theoretical guarantees. These parameters determined through empirical validation to maintain stability across varying hypothesis quality distributions while preserving the risk control framework's theoretical foundations.
16

---

Published as a conference paper at ICLR 2026
A.4 SHGEFDINGGBENTEEE P- VALUE COMPUTATION
For LTT implementation, we compute valid p-values using the Hoeffding-Bentkus Bentkus inequality. Given empirical risk Rm (A₂) on the calibration set, the p-value for hypothesis N₁ R((2) > is:
pr " min { [exp(-nht (Rm (2₂) A a, a)); eP (Bin(n, a)) 2 Lnnn (1:11)
(13)
where h1 (a. b) = a log(a/b) + (1 a) log((1 - a)/(1
b)) and n is the calibration set size. This provides finite-sample valid p-values without distributional assumptions, enabling the sequential testing procedure in Algorithm
B ABLATION STUDY
B.1 CORPUS-LEVEL WER
We compute corpus-level WER through concatenation all predictions and references using the eva 1 uaae package. Results are summarized in Table B.1
Table B.1: Corpus-level WER (%) results with LLaMA-2-7B fine-tuning. Our method results rep- resent one operating point from Figure 2 Results show consistent trends with instance-level av- eraging (Table D.I despite different absolute values, demonstrating robustness across evaluation methodologies. Subscript percentages denote relative WER change vs. vanilla GER and relative
size reduction vs. constant N 5.
Test Set
GER
Our Method
Hmm
Set Size
WER
Fum-
5.05
2.2 22215..0 55.8%
5.050.0%
3.03
CHiME-4
6.37
3.8 3.824.00
6.6+3.6%
4.78
CommonVoice
7.8
3.07 38.6%
7.95 1.9%
6.31
B.2 ANALYSIS OF TRAINING SET SIZE EFFECTS
The ablation results presented here use a simplified experimental setup with different hyperparame- ters and dataset splits compared to the main experiments; nonetheless, they demonstrate consistent patterns that validate our core findings.
The results reveal several key patterns that validate our experimental design. The constant-5 train- ing approach achieves the lowest average WER (7.79%) across all test configurations, confirming its superiority as a baseline model. While diagonal elements (matching train/test sizes) occasion- ally show local optima- optima-such as train-3/test-3 achieving 6.58% versus the train-5/test-3 result of 6.74%-these improvements are marginal and inconsistent across the full evaluation matrix.
Models trained on smaller hypothesis sets exhibit clear performance degradation when tested on larger sets, as expected. The train-l model struggles significantly with multi-hypothesis inputs, achieving 11.65% WER on 5-hypothesis tests compared to 6.38% for the train-5 model, demon- strating the importance of exposure to diverse hypothesis patterns during training.
The dynamic training model, despite having access variable set sizes during training, underper- forms the constant-5 baseline (8.43% vs 7.79% average WER). This degraded performance likely stems from the increased complexity of learning bypothesis--o-trasseiiiii ing input lengths simultaneously, creating a more challenging optimization landscape that prevents the model from fully mastering any single configuration. The model must learn to handle the vari- ability in input structure while maintaining transcription quality, leading to suboptimal specialization compared to the focused constant-5 training regime.
mappings across vary-
17

---

Published as a conference paper at ICLR 2026
These results establish that adaptive approach provides genuine improvements over the best achievable fixed-size baseline, validating the comparative framework used throughout our main ex- periments.
Table B.2: Training Set Size Ablation Study: WER (%) across different training and test configura-
tions on 4 dataset
Train
Test
1-hyp
2-hyp
3-hyp
4-hyp
5-hyp
Average
Train-1
10.32
10.85
11.12
11.38
11.65
11.06
Train-2
10.72
8.55
8.92
9.15
9.41
9.35
Train-3
10.89
8.95
6.58
6.89
7.12
8.09
Train-4
10.95
9.12
6.89
6.52
6.71
8.04
Train-5
10.48
8.69
6.74
6.64
6.38
7.79
Dynamic
11.23
9.45
7.32
7.18
6.95
8.43
B.3 SCALABILITY TO LARGER LANGUAGE MODELS
To evaluate the generalizability of our adaptive framework beyond the LLaMA-2-7B baseline, conducted experiments with LLaMA-2-1 13B (fine-tuned) and GPT-3.5-turbo (zero-shot prompting). These experiments assess whether the computational efficiency gains and adaptive selection benefits persist with more capable and resource-intensive models. All experiments use the LTT calibration procedure with dataset-specific a values, as detailed Section A.2
B.3.1 LLAMA-2-13B RESULTS
We fine-tuned LLaMA-2-13B using identical training procedures, hyperparameters, and evalua- tion protocols as our main experiments. Table B.3 presents representative working points for each dataset.
Table B.3: LLaMA-2-15 --BBBB adaptive selection results. Results shown for one representative working point per dataset; multiple valid working points exist across the parameter space.
Test Set
Baseline
GER
Our Method
(%%
Set Size
WER
Lumm-
8.0
6.47
2.40
6.46 6.46-0.01%
2.1
0.25
4.69
CHiME-4
11.49
7.92
3,88 4%
8. 8.1643.06%
1.25
0.25
7.26
Common Voice
14.1
8.10
3.50
8 8.28 8.284%.27%
2.0
0.25
6.50
The results demonstrate that framework maintains its value proposition with larger models: TedLium-3 achieves 51.9% computational savings with identical performance, while CHiME-4 and Common Voice show modest performance trade-offs (3.06% and 2.27% relative WER increase) for substantial efficiency gains (22.4% and 30.1 % hypothesis reduction). These results confirm that the performance of the adaptive selection mechanism generalizes across model scales.
B.3.2 GPT-3.5-TURE ZERO-SHOT RESULTS
To assess applicability without fine-tuning, we evaluated GPT-3.5-turbo using zero-shot prompting based on templates from HyPoradise (Chen al. 2023) and (Ma et al. 2024
Table B.4: GPT-335-ttrbb zero-shot adaptive selection results. Results shown for one representative working point per dataset.
Test Set
Baseline
GER
Our Method
(%)
O 1im
Set Size
WER
Common Voice
14.1
11.73
2.31 42.3%
11.8 11.81 +0.67% 9.89 +1.17
1.5
0.25
10.51
CHiME-4
11.49
9.77
2.19 56.1%
1.65
0.25
8.61
18

---

Published as a conference paper at ICLR 2026
Our adaptive method achieves substantial computational savings (42-56% hypothesis reduction) with minimal performance impact (+0.67-1.17% relative WER increase) in zero-shot settings.
Experimental notes: For Common Voice, evaluated hypothesis sets in the range [1, as pre- liminary tests showed performance degradation beyond 4 hypotheses; the GER baseline (11.73%) represents the best fixed size in this range. TedLium-3 was excluded from this evaluation because, consistent with findings in (Ma et al. 20224) all zero-shot working points performed worse than the baseline across all hypothesis sizes likely due to the dataset's initially strong baseline performance, making the comparison uninformative.
These experiments validate three key findings: (1) the adaptive framework generalizes across model scales with consistent behavior, (2) zero-shot prompting achieves substantial efficiency gains (42- 56% reduction) despite smaller absolute WER differences, and (3) computational savings persist relative to the increased inference cost of larger models.
B.4 EXTENSION TO SPEECH TRANSLATION TASKS
To demonstrate the broader applicability of our adaptive hypothesis selection framework, evalu- ated its performance on speech translation tasks using the GenTranslate paradigm Hu et al. (20245). This cross-domain validation examines whether our method maintains its computational efficiency benefits when applied to translation scenarios involving N-best hypothesis integration.
Dataset Selection and Monotonicity Validation. We selected three language pairs from the FLEURS X-En speech translation dataset (fr-ten, cy-ten, ar->en) based on a critical prereq- uisite: monotonic performance improvement with increasing hypothesis sizes. Using the pub- lished GenTranslate checkpoint we validated that BLEU scores followed the expected ordering BLEU(N 5) BLEU(N 4) > > BLEU(N 1) on average across these language pairs. This monotonicity condition ensures a meaningful performance-compule trade-off exists, validating the potential utility of adaptive selection.
Methodological Adaptation for Speech Translation. A key challenge emerged in adapting our framework to speech translation tasks: BLEU, the standard evaluation metric, operates at the corpus level and provides limited meaningful information at the instance level required for our based selection mechanism. To address this, we employed TER (Translation Edit Rate) for instance-lever risk computation and adaptive set selection, while reporting final results using corpus-level BLEU for comparability with existing work. This approach leverages TER's established validity at the instance level while maintaining evaluation consistency. We validated that TER and BLEU preserve relative ordering (with inverse correlation) across our test sets.
Table B.5: Speech translation results with adaptive hypothesis selection on FLEURS X-+En test sets compared to fixed N=5 baseline. Our method achieves substantial computational savings while maintaining competitive translation quality.
Success Rate
Task
GER
Adaptive Selection
ax (%)
TER
TER (%)
BLEU
Avg. Size
BLEU
TER (%)
fr-ten
4.62
37.50
35.
37.23 0.70%
4.67
5
0.97
4.24
cy-ten
5.1
33.89
2.62
33.39
5.3
6.9
0.96
4.67
ar-
5.21
34.47
1.72gs.5%
33.44_2.98% 2.98%
5.29
5.65
0.98
4.81
Implementation Details. Due to smaller test set sizes (400-1000 samples) compared to ASR periments, we increased the error tolerance 5 to 0.3 to prevent overly conservative bounds in the LTT framework. All other methodological components, including parameter selection and risk cal- ibration procedures, remained consistent with our ASR implementation. The same LTT sequential testing approach was applied with bER-based loss functions for hypothesis set selection.
Summary. This cross-domain validation demonstrates that our adaptive hypothesis selection framework generalizes effectively beyond ASR speech translation tasks, achieving substantial
19

---

Published as a conference paper at ICLR 2026
computational savings (36-66% hypothesis reduction) while maintaining competitive translation quality. These results confirm the broader applicability of our approach across generative error cor- rection scenarios involving N-best hypothesis integration, addressing the computational efficiency challenges inherent L-M--aseed post-processing systems.
C LLM TRAINING CONFIGURATION DETAILS
C.I HYPERPARAMETERS
We train using A optimizer, effective batch size 32 (achieved through batch size 8 with 4- step gradient accumulation), and cosine learning rate scheduler (with 0.05 warmup ratio). The LoRA configuration uses rank 16 and scaling parameter = 32, implemented via the PEFT library (Mangrulkar et al. 2022).
Dataset-specific hyperparameter' accommodate varying dataset sizes: learning rate range from 5e-5 to le-4, dropout rates range from 0.05-0.1, training epochs from 5-10, with larger datasets requiring higher values for those parameters to achieve optimal convergence.
C.2 PROMPT TEMPLATR
The training utilizes the following prompt template:
"Correct this speech recognition transcript using the hypotheses below. Provide ONLY the corrected transcript, nothing more.
###fyypotheee:
{1st 5th utterances}
HitWCorreced--rrnnsee
C.3 COMPUTATIONAL REQUIREMENTS
Model training is conducted on single NVIDIA RTX 6000 Ada GPU with 48GB memory. Train- ing duration varies by dataset size: CHiME-4 requires approximately 1 hour due to its smaller scale (9,600 samples), while TedLium-3 and Common Voice each require 3-4 hours given their larger training sets (35,000 samples each). The LoRA parameterization significantly reduces computa- tional overhead compared to full fine-tuning, enabling efficient adaptation while maintaining the frozen backbone parameters.
D RISK CONTROL FRAMEWORKS AND EXTENSIONS
D.1 CONFORMAL PREDICTION FRAMEWORK
Conformal prediction is distribution-free frree statistical framework that provides uncertainty quantifi- cation for machine learning predictions with finite-sample guarantees. Given a calibration dataset separate from training data, CP constructs prediction sets that satisfy coverage properties regardless of the underlying model architecture or data distribution.
Let X' denote the input space and y the output space. Consider calibration set ((X(), ))))
where (X (() X x/, and new test point (m++), (m++)). Conformal prediction requires that the calibration data and test point are exchangeable, meaning the joint distribution remains invariant under permutations.
For the test input (m++) with unknown label (m+1), the goal is construct a prediction set (m)))) such that:
(m+()+1) &
a,
(14)
where is user-specified significance level (e.g., 0.1 for 90% coverage).
20

---

Published as a conference paper at ICLR 2026
The framework relies nonconformity scores 8; X x - IR that measure how atypical prediction is for a given input. For any pair (z, y) E x ), the nonconformity score s(x, y) should reflect their agreement, with lower scores indicating better agreement. For the calibration set, s(X(1,, y() quantifies the disagreement between the model's
where true labels are known, prediction and the true label. The prediction set then constructed by including all labels whose nonconformity scores fall below data-dependent threshold, computed based on the (1 (1-o) a:) quantile of the calibration scores.
D.2 CONFORMAL RISK CONTROL APPROACH
D.2.1 BACKGROUND
Consider calibration dataset ((X(1,, } where X E X and Y E y denote feature-response pairs. Conformal risk control (CRC) extends CP to control the expectation of any bounded,
monotone loss function e: 2 x [0,
El (x(m+t))) (,0)),,)) a,
(15)
under the notations represented in
The key insight is that for monotone loss functions-when enlarging the prediction set cannot increase the los--CRC maintains the distribution- free guarantees of standard CP while enabling control over task-specific risk measures. The CRC threshold selection procedure aims to find the optimal threshold:
X = inf {x:
m
Rm (A)
S
a
(16)
+
where the empirical risk is computed as:
1 m
Rm (A)
m Σ (TT)(X))) Y()),
(17)
representing the average empirical loss over the calibration set. For monotone loss functions, this threshold can be found efficiently by gradually adjusting À until the risk constraint is satisfied.
CRC provides sampl guarantees that are tight up to O(1/m) terms as stated in the following Theorem.
Theorem 2 (CRC Finite-Sample Guarantec). Under the exchangeability assumption and for bounded monotone loss functions, the set predictor r selected by the CRC procedure satisfies:
2B a m. +1
< (X (m++)) (m++)) 11 15a S a.
(18)
Note that CRC reduces to standard CP when the loss function is the miscoverage indicator. CRC has been applied to areas such as medical diagnosis, autonomous driving, ordinal classification, and ranked retrieval systems (Andéol et al. 2023 Xu et al. 2023 2024 Overman al.) 2024).
D.2.2 CRC IMPLEMENTATION AND EMPIRICAL ANALYSIS
We initially explored conformal risk control (CRC) as primary theoretical framework for adap- tive hypothesis selection. Our CRC implementation follows Algorithm D.I using the relative WER degradation loss function defined in Equation g with calibrated thresholds to control expected per- formance degradation.
Table D.I presents CRC experimental results across the three datasets. The method achieves substantial computational savings while maintaining competitive performance. These results vali- date the practical effectiveness of adaptive selection based this method.
21

---

Published as a conference paper at ICLR 2026
Algorithm D.1 Adaptive Selection Procedure with CRC Calibration Framework
Require: Calibration set ( ) and test sample H (m++1)
1: Calibration Phase:
2: for À E A (candidate threshold values) do
3: Compute x(T x(T (() for all E [m]]
4: Estimate Rm (A) EE ((Tx(H))) v6)
5: end for
6: Select Ā =nf {A
Rm (2) + m+1 I a m m+1
7: Test Phase:
8: Compute normalized scores = softmax Oo,, (c((m=1))/f)
9: Select n² min(n: E/= \}
10: Return hypothesis set H(
f1 (m+1) (mm-1)
(m+1) Cn" )
However, our analysis revealed that strict monotonicity is violated approximately 20% of sam- ples, where smaller hypothesis sets occasionally outperform larger ones. While 95% of consecutive pairwise comparisons maintain monotonicity, indicating predominantly monotonic behavior, these violations present theoretical challenge for CRC's formal guarantees. We evaluated the monotoniz- ing procedure proposed by Angelopoulos al. (2024b), which constructs (,()) supx>>> (x()) to enforce monotonicity. However, empirical results showed degraded performance across datasets, as monotonizing eliminates precisely the beneficial cases where smaller sets genuinely outperform larger ones the phenomenon enabling our computational savings.
These findings indicate that monotonicity violations in ASR often signal exploitable efficiency op- portunities rather than problematic cases. While our CRC implementation demonstrates strong em- pirical performance and effective risk control in practice, the theoretical violations prevent us from claiming formal statistical guarantees. Therefore, empirical demonstration that complements our theoretically rigorous LTT framework, which natu- rally handles non-monotone losses while providing formal P(R()) < a) 2 1 guarantees.
present this CRC approach as an additional
Table D.1: WER (%) results with LLaMA-2-7B fine-tuning. Baseline: Whisper's top-1 hypothesis. Oum:: post-LLM oracle. Our method results represent one operating point from Figure Subscript percentages denote relative WER change vs. vanilla GER (WER column) and relative size reduction vs. constant N 5 (size column).
Test Set
Baseline
GER
Our Method
(%%
Target WER
O Ou
Set Size
WER
TedLiumm33
8.0
6.04
2. 2.145 67.67. 1%
5.96 1.3%
1.785
6.115
4.33
CHiME-4
11.49
6.38
3.824.0%
6.55 +2.7%
1.9
6.63
4.73
Common Voice
14.1
8.46
3.1 31.00
8.5 +0.5%
1.7
8.65
6.95
D.3
MULTI-PARAMETER OPTIMIZATION VIA PARETO TESTING
D.3.1 MOTIVATION AND METHODOLOGY
Our main LTT framework fixes normalization parameter 7 and temperature 7 based on dataset characteristics, then optimizes threshold À online. While effective, this approach requires pre- deployment validation analysis to determine appropriate parameter values. We explore an extension based on Pareto Testing Goldsh-Goldshtein et al. 2023) that jointly optimizes all three parameters (y, r, A) simultancously, eliminating the need for dataset-specific parameter selection while poten- tially discovering superior performance-efficienny trade-offs through expanded degrees of freedom.
22

---

Published as a conference paper at ICLR 2026
The Pareto Testing framework operates in two stages: first, multi-objective optimization calibra- tion subset calibl constructs Pareto frontier of parameter combinations that optimize both relative WER degradation and computational cost; second, Fixed Sequential Testing on calib2 applies sta- tistical validation to ensure risk control guarantees. This maintains theoretical rigor while enabling automated deployment across diverse acoustic conditions.
(a) Common Voice
(b) TedLium-3
Figure D.1: Pareto frontier visualization for Common Voice and TedLium-3. Gray points represent all feasible working points in the (y, T, parameter space. Green points indicate valid LTT work- ing points across different values, and the red curve depicts the extracted Pareto frontier from representative calibration split, showing non-dominated combinations in WER vs. set size space.
D.3.2 EXPERIMENTAL RESULTS
We evaluated this approach on TedLium-3 and Common Voice datasets. To extract the Pareto fron- tier, we split the calibration set into two subsets: the first is used for multi-objective optimization to construct the frontier of non-dominated (7, combinations, and the second for statistical valida- tion via sequential testing.
The expanded parameter space enables configurations that our heuristic selection cannot identify, particularly intermediate settings that effectively balance both objectives, as evidenced by the gap between the green and red curves in Figure where the Pareto frontier (red) consistently achieves better WFRReeffcciency trade-offs than the individual LTT working points (green).
We then compare the Pareto-selected working points against all fixed set size baselines (N through N = 5) across varying a levels. As shown in Figure [.2] our joint optimization achieves superior performance-efficiency trade-off's across all operating points, matching or exceeding the results of our main LTT approach (Table D.2), confirming that the adaptive selection mechanism is robust across both optimization strategies.
Table D.2: Pareto WER (%) results. Baseline: Whisper's top-1 hypothesis. Oli post-LLM oracle. Results are averaged over T 30 independent trials. Subscript percentages denote relative WER change vs. vanilla GER (WER column) and relative size reduction vs. constant N 5 (size column).
Test Set
Baseline
GER
LTT Method
((%)
Success Rate O
Set Size
WER
TedLium-3
7.53
2.42
51.6%
7.49
0.63%
2.4
0.10
0.92
5.58
CommonVoice
12.44
8.32
3.18
36.4%
8.54
1.95
0.10
0.93
6.96
23

---

Published as a conference paper at ICLR 2026
28
2.
10
2.5
WER (%)
WER (%6)
9-
90
2.3
2.2
2.1
23
2.0
Sel Size
Set Size
(a) Common Voice
(b) Tedl aulLiuum3
Figure D.2: Performance-compute trade-off comparison for Common Voice and TedLium-3. Each plot shows WER vs. set size for fixed set size baselines, alongside the Pareto-selected working points across varying a levels, demonstrating superior trade-offs achieved through joint parameter optimization.
D.3.3 DISCUSSION AND CONCLUSION
This extension transforms our framework from requiring dataset-specific analysis fully automated parameter optimization. The primary trade-off is computational overhead during calibration, as the two-stage optimization evaluates multiple parameter combinations before statistical testing. How- ever, this occurs only during setup, with identical inference costs thereafter.
Pareto Testing represents a step toward adaptive ASR systems that automatically configure them- selves across diverse environments without domain expertise. While main approach remains effective for characterized conditions, this extension provides deployment flexibility and potential performance gains through joint optimization of all hyperparameters while maintaining identical theoretical guarantees.
24


Reviews:
[{"review_id": "KgoEo0K1PD", "reviewer": "Reviewer_ceGk", "summary": "This paper proposes an adaptive framework for generative ASR error correction (GER) that addresses the limitations of fixed-size hypothesis sets by leveraging conformal risk control (CRC). The framework dynamically determines the optimal number of ASR hypotheses for each input using confidence scores and CRC, which controls the expected relative word error rate (WER) degradation compared to oracle performance. Evaluated on three datasets (TedLium-3, CHiME-4, CommonVoice) with varying acoustic difficulties, the method achieves substantial computational savings (up to 57.1% reduction in hypothesis usage) while matching or exceeding the performance of fixed-size GER baselines. Key contributions include the adaptive hypothesis selection mechanism, the first application of CRC to GER, and empirical validation of robustness across diverse acoustic conditions.", "questions": "How does the framework perform when applied to larger LLMs (e.g., LLaMA-2-13B/70B or GPT-3.5/4) and does the computational savings persist relative to the increased inference cost of larger models?\nHave you explored methods to automate the tuning of score function parameters (γ, τ) across unseen acoustic conditions, or is manual calibration required for each new dataset/environment?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The adaptive hypothesis selection approach effectively resolves the inefficiency of fixed-size sets, balancing performance and computational cost—a critical practical challenge in LLM-augmented ASR.\nIntegrating CRC provides statistical guarantees on relative WER degradation, addressing the lack of reliability assurances in existing GER methods and enhancing real-world applicability.\nComprehensive experiments across datasets with different acoustic characteristics (noise, accents, recording conditions) demonstrate the framework’s robustness and generalizability.\nThe ablation studies and detailed case analyses (full set required, single hypothesis optimal, performance plateau) provide deep insights into the mechanism’s behavior and validate design choices.", "weaknesses": "The loss function’s monotonicity violations (≈20% of samples) are acknowledged but not fully resolved; the decision to use the unmodified loss without monotonicity enforcement relies on empirical robustness rather than theoretical justification.\nThe framework’s compatibility with larger LLM architectures (beyond LLaMA-2-7B) and scalability to extremely large datasets are not evaluated, leaving uncertainty about its performance in more resource-intensive settings.\nThe score function’s parameters (γ, temperature τ) are tuned per dataset, which may limit adaptability to unseen acoustic environments without retuning.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5DtyF4Z2Wd", "reviewer": "Reviewer_v9jL", "summary": "This paper addresses the computational inefficiency of GER in ASR, where LLM typically operate on a fixed-size N-best hypothesis list. The authors propose an adaptive framework that dynamically determines the optimal number of hypotheses $n \\le N$ for each input1. The core of the method is the application of conformal risk control (CRC) to this selection process2. Instead of controlling for absolute error, the framework is cleverly designed to control the expected relative word error rate (WER) degradation compared to the best achievable performance (the \"post-LLM oracle\") for that specific input. This is achieved by calibrating a threshold $\\lambda$ on normalized ASR confidence scores, which in turn determines the set size $n$4. Experiments on TedLium-3, CHIME-4, and Common Voice show that this adaptive approach significantly reduces the average number of hypotheses (e.g., 57.1% on TedLium-3) while maintaining or even slightly improving WER compared to the fixed N=5 baseline.", "questions": "1. I would like to see if the method also works for speech translation correction alike GenTranslate (Hu et al.) N-best data in ACL 2024. (https://huggingface.co/PeacefulData/GenTranslate) This additional non-ASR only study would largely increase the impacts of the work in the GER community. I would consider to increase my score.\n\n2. Choice of Risk Target $\\alpha$: Appendix B.2 53states that $\\alpha$ is chosen based on the validation set's degradation statistics (e.g., 90th percentile). I assume the different red operating points in Figure 2 55are generated by varying $\\alpha$ (e.g., from 80th to 95th percentile ranges). Could the authors confirm this and perhaps briefly discuss the relationship between the chosen $\\alpha$ and the resulting (WER, Avg. Set Size) trade-off? \n\n3. Sensitivity of Score Function: The score function (App B.1) 49is tuned per dataset (e.g., $\\gamma=1.0$ for TedLium-3, $\\gamma=0.0$ for Common Voice). How sensitive is the method to these choices? What would the performance-compute trade-off (Fig. 2) 51 look like for Common Voice if the TedLium-3 parameters ($\\gamma=1.0, \\tau=0.05$) were used? This would help clarify if this tuning is a minor optimization or critical for the method's success.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's central claims are supported by experiments. The use of standard benchmarks (HyPoradise) , strong ASR and LLM models (Whisper and LLaMA-2-7B) , and relevant metrics  provides a strong empirical foundation. The ablation study in Appendix D (Table D.2) effectively justifies the experimental design choice of training the H2T model on fixed N=5 sets while evaluating on variable-sized inputs.\n\nHowever, there is a significant gap between the theoretical claims and the implementation. The Conformal Risk Control framework (Angelopoulos et al., 2024b) formally requires a bounded, monotone loss function to provide its distribution-free guarantees. The authors forthrightly acknowledge in Section 5.2 that their chosen loss function (Eq. 10) is not strictly monotone, with violations observed in approximately 20% of samples.\n\nThey provide an empirical justification for this deviation, noting that applying the standard monotonizing procedure actually degrades performance. This is because their method's advantage comes, in part, from exploiting these non-monotonic cases where a smaller hypothesis set genuinely outperforms a larger one (a key insight illustrated in Fig 1a and Table 2) .\n\nWhile the empirical results are strong and the risk is empirically controlled, this violation means the method does not inherit the strict theoretical guarantees of CRC. The paper is thus a work of \"empirical risk control\" inspired by CRC, rather than a direct application of it. This theoretical-practical disconnect is the primary weakness in an otherwise sound paper.\n\nThe paper introduces a novel application of CRC to a practical and important problem in ASR: the computational overhead of fixed-size N-best lists for LLM-based error correction . The results demonstrate a clear path to more efficient and robust ASR systems.", "weaknesses": "- Theoretical Gap (Monotonicity Violation). This is the most significant weakness. The CRC framework's guarantees depend on a monotone loss function. The authors explicitly state their loss (Eq. 10) is non-monotone for ~20% of samples. Their justification for proceeding is empirical: enforcing monotonicity (as suggested by Angelopoulos et al., 2024b) hurts performance because it prevents the model from exploiting \"good\" non-monotonic cases where fewer hypotheses are better . This is a reasonable empirical argument, but it invalidates the theoretical guarantees. The paper should be more precise in framing its contribution as an empirically robust method inspired by CRC, rather than one that provides CRC's guarantees.\n\n- The adaptive score function $\\phi_{\\gamma}$ (Appendix B.1) 44, while well-motivated by dataset noise characteristics 45, introduces dataset-specific hyperparameters $\\gamma$ and $\\tau$ (Table B.1). This adds a tuning step that moves away from the \"distribution-free\" simplicity often associated with conformal methods. A sensitivity analysis on these parameters would strengthen the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "26rgMoAQeY", "reviewer": "Reviewer_moGC", "summary": "LLMs are used to generate a speech transcription using N-best hypotheses from an ASR model. The paper proposes to use conformal prediction to reduce the size of the hypothesis set, thereby reducing computations in the LLM. \nExperiments show that a substantial reduction in the size of the hypothesis set is achievable without compromising performance (WER).", "questions": "- Please address my concerns listed in the \"Weaknesses\" section.\n- Line 274 talks about the monotonicity assumption being violated 20% of the time. I would request details here. I expect this number to be much higher in the case of ASR likelihoods.\n- Some minor typos, e.g., line 189.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 1, "strengths": "- The paper proposes a new application of conformal prediction to LLM-based error correction in ASR. \n- Extensive experiments are shown on three datasets. \n- There is a reduction in the size of the hypothesis set.", "weaknesses": "- I believe that the paper is an application of an existing concept of conformal prediction to an existing problem of generative error correction in ASR, and hence, the contribution lacks novelty.\n- I strongly argue that some concepts are treated very superficially. Conformal prediction calibrates the threshold on a score function (confidence measure) of hypotheses without altering their order. The paper uses likelihood values as a measure of confidence (although it does not explicitly mention this). However, previous research on ASR has shown that likelihood values are a poor indicator of confidence [Li et al., Ravi et al.]. These likelihood values are often highly overconfident, and they are not able to correctly order the hypotheses. Simple thresholding may not yield significant performance improvements. BTW, [Li et al., Ravi et al] perform only top-label calibration, but works such as [Popordanoska et al.] can provide canonical calibration, which can be used to provide N-hypotheses.\n- Again, the title of the paper highlights conformal risk control, but the experiments do not talk about controlling the risk. One would expect a detailed analysis of expected WER (while setting the threshold) vs the achieved WER. It appears from Table 1 that they do not match.\n\n[Li et al.] Confidence Estimation for Attention-Based Sequence-to-Sequence Models for Speech Recognition, ICASSP 2021.\n[Ravi et al.] TeLeS: Temporal Lexeme Similarity Score to  Estimate Confidence in End-to-End ASR, IEEE TASLP, 2024.\n[Popordanoska et al.] A Consistent and Differentiable Lp Canonical Calibration Error Estimator, NeurIPS, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)