C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails
Authors: Mintong Kang
Zhaorun Chen
Bo Li
Venue: NeurIPS 2025 poster ()
Keywords: LLM safety
LLM guardrail
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Despite the remarkable capabilities of large language models (LLMs) across diverse applications, they remain vulnerable to generating content that violates safety regulations and policies. To mitigate these risks, LLMs undergo safety alignment; however, they can still be effectively jailbroken. Off-the-shelf guardrail models are commonly deployed to monitor generations, but these models primarily focus on detection rather than ensuring safe decoding of LLM outputs. Moreover, existing efforts lack rigorous safety guarantees, which are crucial for the universal deployment of LLMs and certifiable compliance with regulatory standards. In this paper, we propose a Claim-based Stream Decoding (CSD) algorithm coupled with a statistical risk guarantee framework using conformal analysis. Specifically, our CSD algorithm integrates a stream guardrail model to safeguard sequential claims generated by LLMs and incorporates a backtracking mechanism to revise claims flagged with high safety risks. We provide theoretical guarantees demonstrating that the CSD algorithm achieves the desired generation distribution subject to safety constraints. Furthermore, we introduce a generation risk certification framework and derive a high-probability upper bound on the safety risk of the proposed CSD algorithm. We extend our approach to online settings, where user queries arrive sequentially, and prove that our method can asymptotically control safety risk to any desired level. Empirical evaluations demonstrate the effectiveness and efficiency of the CSD algorithm compared to state-of-the-art safety decoding approaches. Additionally, we validate the soundness and tightness of the derived safety risk upper bound using realistic data in both offline and online scenarios.

C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails
Mintong Kang
Zhaorun Chen
UIUC mintong2011linois.cem
UChicago
zhaorun@uchi.agoo.oo
Bo Li UIUC & UChicago
1bo@illinois.er
Abstract
Despite the remarkable capabilities of large language models (LLMs) across di- verse applications, they remain vulnerable to generating content that violates safety regulations and policies. To mitigate these risks, LLMs undergo safety alignment; however, they still be effectively jailbroken. Off-the-shelff guardrail models are commonly deployed to monitor generations, but these models primarily focus on detection rather than ensuring safe decoding of LLM outputs. Moreover, ex- isting efforts lack rigorous safety guarantees, which are crucial for the universal deployment of LLMs and certifiable compliance with regulatory standards. In this paper, we propose Claim-based Stream Decoding (CSD) algorithm coupled with statistical risk guarantee framework using conformal analysis. Specifically, our CSD algorithm integrates stream guardrail model to safeguard sequential claims generated by LLMs and incorporates backtracking mechanism revise claims flagged with high safety risks. We provide theoretical guarantees demonstrating that the CSD algorithm achieves the desired generation distribution subject safety constraints. Furthermore, we introduce generation risk certification framework and derive a high-probabillty upper bound on the safety risk of the proposed CSD algorithm. We prove that our method can asymptotically control safety risk to any desired level. Empirical evaluations demonstrate the effectiveness and efficiency of the CSD algorithm compared to state-of-the-art safety decoding approaches. Additionally, we validate the soundness and tightness of the derived safety risk upper bound using realistic data.
Introduction
have seen widespread adoption due their remarkable capabilities in natural language understanding and generation. Open-source LLMs, such as DeepSeek- RI [12). promote accessibility and transparency, fostering innovation across various applications. However, their openness also introduces substantial safety and security risks. Malicious actors can exploit open-source models by embedding backdoors 451 or crafting adversarial prompts bypass safety constraints [50 [7], potentially leading to harmful generations. Such vulnerabilities pose real-world risks, including misinformation propagation, bias amplification, and security threats in autonomous systems 43 These risks highlight the pressing need for rigorous safety guarantees in LLM generations, especially in high-stakes scenarios such as healthcare, finance, and policy-making.
Large language models (LLMs) 37
Current approaches to mitigating unsafe LLM outputs are predominantly empirical and lack formal assurances. During the training phase, reinforcement learning from human feedback (RLHF) 30 32
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

aims to align LLMs with human preferences and safety norms. However, RLHF is computationally expensive [6 and remains susceptible to adversarial attacks and jailbreak exploits [50]. At the inference phase, guardrail models 15
attempt to filter unsafe responses, but they merely classify harmful content rather than proactively ensuring safe generation. Likewise, 48]
25
19
33
22
47
modify token sampling strategies to reduce risk but lack theoretical safety guarantees. The absence of provable risk bounds limits the reliability of these empirical defenses, making them unsuitable for applications demanding strict safety assurances.
decoding-time interventions [44
42
In this work, we introduce C-SafeGen, the first certification framework for bounding safety risks in LLM generations. C-Saf eGe is model-agnostic, enabling its application to both open- spuresource and closed-soure LLMs in black-box setting. We establish theoretical guarantees that, given a specific model configuration, C-Saf Sen can compute high-pun tiih--robability upper bounds on safety risks under mild assumptions. Furthermore, C-SafeGen provides a principled mechanism to derive valid decoding configurations that ensure compliance with a specified risk threshold.
To complement this certification framework, we propose Certifiably Safe Clain-based Stream Decod- ing (CSD), a novel decoding algorithm that enforces provable safety constraints during generation. CSD dynamically adjusts token sampling strategies using KV-cache mechanisms and a guardrail model, ensuring fluency and coherence while maintaining certified safety bounds.
We validate C-SafeGen and CSD through extensive empirical evaluations standard safety bench- marks. Our results demonstrate that C-SafeGen yields tight and reliable safety risk estimates, effectively bounding empirical risks with minimal gaps. Additionally, CSD significantly reduces unsafe generations while preserving high-quality outputs. These findings establish C-SafeGen and CSD as foundational tools for the deployment of provably safe LLMs in real-world applications.
2
Related work
Safety Guardrails. Existing safety guardrails serve as an essential mechanism for mitigating
unsafe LLM outputs by filtering, modifying, or rejecting harmful content. These methods fall into several broad categories: (1) industry-standart safety APIs such as Detoxify (2). Perspective [19].
Azure T. and OpenAI's moderation tools 25]: (2) fine-tuned classifiers designed for harmful content detection, including LlamaGuard 15 ToxicChat-TS 22) ToxDectRoberta 49), sentence-
transformer-bast based classifiers (S), GPT- based moderation frameworks (24) and Aegis I (3) LLM-based techniques leveraging prompt engineering [18] 40) and constrained dialogue mechanisms such as Nemo Guardrails 33:; and (4) statistical safety models, including KNN-based guardrails [47 and Beta regression-basees risk estimation (36] Despite their effectiveness in detecting unsafe content, these approaches are primarily reactive, failing to actively correct unsafe responses. Furthermore, their reliance threshold- based rejection may lead to over-filtering, inadvertently discarding benign or contextually appropriate responses.
Safety Decoding. Safety decoding techniques enforce constraints on token selection during gener- ation, proactively mitigating unsafe outputs. Notable approaches include paraphrasing and retok- enization defenses [16) against adversarial optimization attacks, as well as rewindable generation strategies like RAIN (21, which allow models to self-evaluate and modify unsafe outputs dynami- cally. Jailbreak resilience techniques have also been explored, including in-context demonstrations 41. system-prompt self-reminders (42, and contrastive decoding methods 44 that adjust token probability distributions to suppress adversarial prompts. While these approaches enhance L.M robustness, they lack formal guarantees on safety performance, making their efficacy difficult to quantify and validate across diverse deployment scenarios.
Conformal Prediction for Safety. Conformal prediction is well-established statistical framework for constructing prediction sets with provable coverage guarantees (39
Recent advance- sensiseie applications by providing high-confident upper bounds for black-box models under exchangeability assumptions. However, despite its success in risk-sensitive domains such as medical diagnostics and autonomous systems, conformal analysis has yet to be adapted for LLM safety certification. Our work bridges this gap by leveraging conformal techniques to construct provable safety risk bounds for L.M.s, enabling rigorous certification frameworks applicable across both open-source and closed-source models.
3%
20
46)
ments in conformal risk control 6 3 extend these techniques

---

User query
O
Claim based Stream Decoding
How does autonomous driving systems work? Provide code snippet system, Vallbreak Suffor
language model
Claim
Claim 2
Claim
Malicious code snippot of system attack
Bonign response of how the system warks
C-SafeGen Safety Guarantee
Guardrral model G with safety threshold dr,
Delintio:: Claim-hased decoding (CSD) contiguration [Pa,*g] 9y]
Safety guaranter With CSD config the safety v,
Safety guarantee achieve desired salety r,a(7) presents valid CSD config (Theorem 2)
Backtrack Resampling with talerance 8g
Figure I: C-SafeGen comprises (I) a claim- based stream decoding algorithm (CSD) and (II) safety risk certification framework. The CSD algorithm utilizes an off-the-shee guardrail model G' to monitor sequentially generated claims, flagging unsafe claims when they exceed a predefined threshold as. When such event occurs, the unsafe claim is backtracked and resampled. Additionally, CSD employs auxiliary safety decoding strategies to enable safe resampling if claim generation stagnates beyond a tolerance threshold Og-- The safety risk certification framework provides two types of guarantees: (1) Given a CSD configuration a, the generated output is guaranteed
achieve conformal safety risk To as established in Theorem [ (2) If the goal is to attain an expected safety risk r, any CSD configure configuration within the set O,,, as specified in Theorem 2 can ensure this objective.
C-SafeGen
In this section, we formalize the problem of ensuring safety in L.M generation through a rigorous certification framework. We introduce safety guardrail model that assesses the risk of generated outputs and propose a SafeGen protocol to regulate the generation process. The core objective is to quantify and control the safety risk associated with LLM outputs by defining risk function based on the guardrail model. To achieve this, we establish two levels of certification: (1) an upper bound on the safety risk for given generation configuration and (2) certified set of SafeGen configurations
that maintain a specified nominal risk threshold. These guarantees provide a principled foundation for certifying the safety of LLM --M--eeertted content. We provide overview in Figure T
3.1 Problem setup
Consider fixed language model po X Y parameterized by 0, which maps the input text space X to the generation text space V. Specifically, given an input prompt æ E X, the language model ps defines a conditional probability distribution over possible output texts, denoted as po(2:.
To assess and regulate the safety of L-Leegeertteed content, we introduce a fixed safety guardrail model G X' x y + 2c, which maps input-output pairs to subsets of the safety category space C (c.g., violence, sexual, group hate, safe). The guardrail model acts classifier that categorizes model outputs into predefined safety risk levels.
Safety Risk Function. To formally quantify the safety risk associated with an LLM generation y E y given an input x E X, we define safety risk function based on the guardrail model G:
Ro(2, y) = G("safe"|r: y)
This function assigns risk value between 0 and 1, where a value closer to 1 indicates a higher likelihood that the generated output is unsafe, as determined by the guardrail model.
SafeGen Protocol. We define a extitSafeGen protocol Pa X + y as a randomized function parameterized by generation configuration Given input Jr the protocol P. produces generated output y, adhering to specific decoding strategies and safety measures encapsulated in a.
The expected safety risk for given input under the SafeGen protocol is formulated as:
Pa()) G(*safe" -((""safe" Pa()))
(2)
3

---

This formulation captures the probability of generating an unsafe output under the specified generation configuration a.
C-SafeGen provides two levels of safety risk certification: (1) Safety risk upper bound certification: Given specific generation configuration a, G-SafeGen establishes high-confidence upper bound on the safety risk. (2) SafeGen configuration certification for nominal safety risk: Given nominal risk threshold r, S--SafeGen identifies the of generation configurations a that ensure compliance with the specified safety risk level. These certifications enable robust safety evaluations of LLM outputs, facilitating safer and more controlled language generation processes.
3.2 Safety risk certification
In this part, we formulate the approaches to achieving the two levels of safety risk certification two formal statements, respectively.
Theorem (Safety risk upper bound certification). Given a SafeGen protocol Pa with generation configuration a, C-SafeGen guarantees that:
P [Ra(2, Pa(x)) S fal 1-111 5,
(3)
where the high-probability risk upper bound far the so-called conformal safety generation risk, given by:
min h-1 (In(1/0), i((Deal)
(2: R(D
D inverse
Neal
with h (::-)
as
h b):a) of (og(()))/(1-))) and Dim as the inverse of binomial cumula-
the
partial
h(a, 6) a log(a/b) /)++))))))))) + (1 a) tive distribution function (CDF).
Remarks (Remark of Theorem MT. The upper bound on the safety risk provided by Theorem is derived under hygh-rrobability guarantee, ensuring that the true risk does not exceed fa with probability at least 1 5. This bound is computed using two distinct mnite-sample-vablee approaches: one based the inversion of a likelihood-ratio function and another utilizing the binomial CDF. The combination these two approaches ensures robustness in cases where one bound is tighter than the other, effectively balancing statistical efficiency and coverage.
Theorem 2 (SafeGen configuration certification for nominal safety risk). Given a nominal safety risk C- E-Safen E-SafeGen can certify configuration set a, such that each configuration in or, is guaranteed to keep the generation risk below a. Namely,
sup {Ro (I, Px()) S 2 21- aEa.
where the valid SafeGen configuration set or is given by family-wise error rate control Or, {ay Pj S 8,} with E : RR(x, Pa, (x)) > a (j index all feasible SafeGen config) and can be computed by finite-sappl valid bounds as shown in Theorem
å where Pj is the p-value of the null hypothesis:
Remarks (Remark of Theorem 2) Theorem Saatreem@exeee exxendd the safety certification from individual con- figurations
of SafeGen configurations while controlling the family-wise error rate (FWER). By leveraging hypothesis testing with valid p-values, the theorem constructs a confidence set ar, that guarantees all configurations within it maintain risk below the nominal threshold a. The use of family-wise error control, particularly through a summation constraint on 8,, ensures that the overall probability of incorrectly certifying an unsafe configuration remains within the desired tolerance level 8.
We leave the complete proofs to Appendix A
SafeGen via Claim-based streaming guardrail
4.1 CSD: Claim-Based Safe Decoding
Consider a fixed safety/halluciintion guardrail model G X' y + 2c, which maps the joint input-output space to a set of predefined safety/hallucinatin categories C (c.g.. "violence," "sexual,"

---

Algorithm Claim-based stream decoding algorithm
Require: Input prompt x, ourput length LLM pa. Claim critical point ClaimPoint( -). Guardrail model G, Claim backtrack probability
Big sale resampling function SaleResumple, (+..)
1: 4- p=(22
Sample first token Decoding counter
2: counter
3: for
do
4: iCClaimPoint(:; Men)) then
p Critical point for complcte claim
5: Ps (n M-en), Cauie))
Compute backtracking probubility
6: Pb, +-
7: end If
8: counter +- counter
9:
Uniform((, 1) then
With probability ps
10: n + LastChimPPintt:; V.cn), continue
D. Backtrack to the last claim point
11:
12:
if counter n| then
Stagnate at the point Safe resampling
13:
+ SafeResample, vern)
14:
else
15:
yn po
16:
end
17:
end for
18:
return U.S
"group hate," "safe"). Let Po X + ) be a language model parameterized by 0, defining a conditional distribution over the output space given an input prompt x E X. That is, the model assigns probability mass to different outputs y E y as po z).
Our goal is develop SafeGen protocol Pa parameterized with a that achieves minimal safety risk. Formally, we aim to:
min Ra v), S.t. G( v) C Cdesired: (EPa(a)
where Cdesired denotes the set of acceptable categories under the guardrail model (e.g.. Cdesired {"safe") for safety guardrail). This objective seeks the highest-likelihnn output from pa that remains within the feasibility set imposed by G. While rejection sampling could be used to enforce this constraint, it is often computationally infeasible.
4.1.1 Claim-Based Decoding
To improve efficiency, introduce claim-based approach. Let P: 2 be claim partition model, which decomposes an output E ) into a set of independent claims. This leads to stronger decoding constraint:
min
Ro v), s.t. G(x, v/)
Caesired, Vy' E P(y).
(6)
NE (m)
This formulation ensures that every claim within y is individually verified against the guardrail model.
4.1.2 Claim Backtrack Probability Function
To control decoding efficiency and safety, we define the claim backtrack probability function:
G(I,Ven)e max G(r, S 0b Bay(GG (G(x, V<nn), Cdesired) = min Ceessd
adCassed
where Og regulates the trade-off between decoding speed and adherence to the guardrail: as = 1.0 strictly enforces Eq. equation O ar = 0.0 reduces to greedy decoding. Intermediate values provide a smoothed trade-off, considering both the consistency of G and its overlap with Cdesired-
4.1.3 Claim Critical Point Detection
we propose an efficient structural approach based on termination indicators. We define set of claim termination tokens T (e.g.. newline, period) and identify a claim boundary when any termination token't probability exceeds a,. To prevent excessive segmentation, we enforce minimum claim length o
Unlike prior methods that prompt LLMs to extract claims 27
28
ClaimPoint(z,, )<n) [t (z Ven)) > an Et E T I In LastClaimPoint(zz V<n) > ail.
This method ensures computational efficiency while maintaining robust claim detection.
5

---

4.1.4 Safe resampling for robust decoding
To enhance robustness and mitigate adversarial attacks, we introduce safe resampling techniques inspired by SmoothLLM B4. Unlike prior approaches that insert perppeeely substrings at inference time, we directly perturb the KV cache in the ongoing decoding process using: Random muaking/permutation of hidden states, Gaussian noise injection into KV-cache representations, Structured perturbations for controlled diversity.
These techniques enhance sampling diversity while preserving alignment with the guardrail model.
4.1.5 Computational efficiency
To enhance computational efficiency, we implement a KV-cache mechanism for the guardrail model. Specifically, the key-value (KV) pairs corresponding to previously verified claims can be stored, allowing efficient backtracking without recomputing the entire sequence.
Formally, let KV denote the cached representations for the sequence yi:n, and define the back- tracking operation as follows:
KV + KV KV mn++1 if Ban (G(x,, ()n+1); Cdesirea) O.
This ensures that upon backtracking, only the key-value pairs corresponding to the current claim erased, while maintaining the cached representations for verified claims.
4.2 Analysis of CSD algorithm
Definition 1 (Restricted output distribution). Let be the distribution over feasible output space O. Let G O + {0, 1} be a guardrail model where 1 denotes desired output and 0 denotes undesired output. Let pG be restricted output distribution of p by guardrail G' with the formulation:
p(v) =1]
(v) Σ EEy G(v) 1]
(10)
Remarks. The restricted output distribution Pe represents the renormalized probability mass of the feasible output distribution p, conditioned on the constraint imposed by the guardrail model G. This ensures that only outputs classified desired (G(y) 1) contribute the final distribution. The denominator acts as a normalizing constant, ensuring that Pc remains a valid probability distribution.
Definition 2 (Claim). An output claim ÿ is defined as a sequence of tokens that delivers complete semantics, which is judged by a claim discriminator C. Specifically, C(yss,) = 1, C(ygt,) =1 and C(yce) = 0, Vt1 < t < t2 implicates that the sequence fr1.22 is a claim.
Remarks. The claim discriminator C' plays crucial role in identifying semantically complete claims within a sequence. The definition ensures that claim is an isolated segment where the beginning and end are both recognized by C, while the intermediate segments do not form independent claims. This property is essential for structured generation and for enforcing guardrails at the claim level.
Assumption 4.1 (Claim risk cascade). Consider an output sequence o which consists of a sequence of N claims by claim discriminator C: [5), U2, ... UN! We assume that the risk of prefix claims implicates the risk of oulput judged by guardrail G:
G(ii,..., ()])) = 0, An E [1, = G(ū). ((.
(11)
Remarks. The claim risk cascade assumption formalizes the idea that the presence of undesired claim in a sequence guarantees that the entire sequence is also undesired. This assumption aligns with the intuition that risks in intermediate steps propagate forward, affecting the final judgment by the guardrail model G. This is conservative approach that simplifies analysis while ensuring safety in controlled generation.
Theorem 3 (Algorithm recovers restricted output distribution). Under the claim risk cascade sutputjudger
assumption as Assumption 4.1 there exists at least one desired output judged by guardrail G, the output distribution of the claim-based decoding streaming algorithm in Algorithm] 7 (without safe resampling) is identical to the restricted output distribution as Definition! 7
Remarks. This theorem establishes that the claim-based decoding streaming algorithm produces outputs that exactly match the restricted output distribution pc- The key condition for this result is

---

Conformal Safety Risk
0.8
Conformal Safety Risk
0.7
Empirical Safety
0.7
Emplrical Safcty Risk
0.6
0.6
0.2
0.2
0.1
0.1
Vanilla
Reminder Safety- Ope. Best-of-N
SafeDec
CSD
Verilla
Reminner Safety Ogt Best-of-N SafeGin Mathods
SafeDec
CSD
Safetien Mathods
(a) AdvBench Llama-3.1 3.1-8B
(b) AdvBench Llama-3.1 3.1-13B
Confermal Safety Risk
0.8
Contormal Salety Risk
Empirical Safety Risk
Empirical Safety Risk
0.6
0.6
0.5
0.1
0.4
0.3
0.2
0.2
Vanilla
Reminder Safety-Op- Opr. Best-ofAN
SafaDec
CSD
Vorilla
Reminder Safety-Opt Best-of-N SafeDec
CSD
SafeGen Methods
SofeGen Methods
(c) JailbreakBenchh 1.aprrak-Beecc-LLLaa33111111 Llama-3.1-8B
(d) JailbreaBBench Llama-3.1-11
Figure 2: Evaluation of conformal safety risk (upper bound from Theorem d and empirical safety
risk (mean risk on sampled test sets) across two Denchmarks-AAdBennch and JailrreakBench--usee 1lama-3.1 models (8B and 13B) with ShieldGemma-99 as the guardrail. Results show that (1) conformal risk bounds are valid and tight; and (2) our CSD method consistently achieves the lowest safety risk. Note: Grey bars result from overlapping dots.
the existence of at least one feasible sequence that satisfies the guardrail G. The theorem is significant because it ensures that the decoding algorithm does not introduce biases or distortions beyond those imposed by the guardrail, thereby preserving the original probability structure of the restricted output space.
Evaluation
5.1
Evaluation setup
Dataset & Models.
As AdvBench (50 and JailbreakBench g are widely used for evaluating LLM we adopt as our primary evaluation dataset.
safety
26
For text generation, we consider LLMs Llama-3. 1-8B, and Llama 3. 1-13B as inference models. Additionally, append adversarial suffixes the user query jailbreak the model, creating more challenging safety evaluation scenario. These adversarial suffixes are optimized using the GCG attack 50 on the corresponding inference model target models.
Metries. Without specification, we employ L1 LlamaGuard3- [10] as the guardrail model G and use the unsafety probability predicted by it the safety risk function Rg. Additionally, we also consider ShieldGemma- 9B as guardrail models for guardrail comparisons.
Baselines. We consider four baselines for safe generation: (1) Vanilla generation, using a temperature of 1.0; (2) Self-reminder safety prompt (Reminder) [42), which incorporates system safety prompt and an additional reminder prompt after the user query to encourage safer generation; (3) Safety- -pt
48 which optimizes a soft safety prompt prefix to guide the model toward safer response patterns;
(4) Best-of-N (35 which selects the safest response from set of 10 generated outputs; and (5)
SafeDecoding (SafeDec) [.] which uses a safety- contrastive decoding strategy for L.LMs to generate helpful and harmless responses user queries.

---

0.12
Nominal Safety Risk
0.12
Nominal Safety Risk
Empirical Safety Risk
Empirical Safety Risk
0.10
0.10
Safety Risk
Safety Risk
0.08
0.08
0.04
0.02
0.02
0.8 Backtrack Threshold d
0.85
Backtrack Threshold de
(a) AdvBench 1.lma-3.1888
(b) AdvBench Llama-3, 3.1-13BB1BBB
0.12
Nominal Safety Risk
0.12
Nominal Safety Risk
Empirical Safety Risk
Empirical Safety Risk
0.10
0.10
Safety Risk
Safety Risk
0.04
0.04
0.02
0.6
0.65
0.85
0.02
0.6
0.65
0.7
0.75
0.8
0.85
Threshold dp
Backtrack Threshold
(c) JailbreakBench Llama-3, 3.1--BB
(d) JailbreakBench Llama-3, 3.1-13B
Figure 3: Evaluation of SafeGen configurations with nominal safety risk 0.10 (Theorem 2 Valid configurations remain below the nominal risk, and larger backtrack thresholds 0g yield lower and more stable risks. Note: The grey bar results from overlapping grey dots.
5.2 Safety risk upper bound certification
Figure presents comparative evaluation of conformal safety risk (thcoretical upper bound) and empirical safety risk (mean observed risk) across six SafeGen methods on two safety benchmarks, AdvBench and JainbreakBench, using Llama-3.1 models (8B and 13B) with ShieldGemma-99 guardrail. Across all settings. we observe that CSD consistently achieves the lowest safety risks both conformal and empirical indicating formal guarantees. The conformal risk bounds are generally tight, validating the reliability of the theoretical risk estimation across different methods and model sizes. Notably, methods such as SafeDec and Safety-Opt also reduce risk substantially compared to the vanilla baseline, but fail to match the low-risk performance of CSD. The trend persists across both model scales and benchmarks, underscoring the robustness of CSD's improvements in safety under adversarial and jailbreak-style attacks.
the
superior capability in reducing unsafe generations under
In addition to method-wing comparisons, the results also reveal consistent trends across model sizes. Specifically, the larger model Alama-3..113B- does not uniformly outperform the smaller 8B variant in safety risk reduction. In some cases such under Reminder and Best-of-N the 13B model exhibits slightly higher safety risk than the SB model, suggesting that scaling up model size alone does not guarantee improved safety under adversarial or conformal evaluation. This highlights the importance of method design (e.g., CSD) over sheer model capacity in achieving reliable and provably safe generations.

---

5.3 SafeGen configuration certification for nominal safety risk
Figure BirreBBillll Billusttraes the validity of SafeGen configurations as certified by Theorem 2 and investigates how the backtrack threshold a, affects the empirical safety risk across different benchmarks and model sizes. The dashed green line indicates the nominal safety risk bound (set to 0.10), while the grey markers represent empirical safety risks under varying values of os. We highlight three main observations: (1) Certified safety guarantees hold across all settings. For every evaluated configuration, the empirical safety risks remain strictly below the nominal risk threshold, confirming the bhoo-brtbbbilityy validity guaranteed by Theorem 2 This supports the theoretical soundness of the certification framework across both AdvBench and JailbreakBench. (2) Higher or leads to lower safety risk. Across all plots, increasing the backtrack threshold og consistently reduces empirical safety risk. This trend is consistent for both Llama-3.1-8B and Llama-3. 1.BBB demonstrating that SafeGen with a higher as allows more cautious decoding behavior and safer outputs. (3) Risk stability improves with larger atg. We also observe reduction in the variability of empirical safety risk as og increases. This indicates that larger thresholds not only yield safer generations but also improve the reliability and consistency of safety performance across different test samples.
These results demonstrate that SafeGen provides strong empirical and theoretical safety guarantees, with all evaluated configurations maintaining risks below the specified nominal bound. Adjusting the backtrack threshold at is an effective mechanism to improve both the magnitude and stability of safety risk, and can be tuned independently of model scale. Together, these findings underscore the importance of principled decoding strategies- such as SafeGen- for deploying LLMs in safety- critical applications.
5.4 Safety vs. decoding efficiency
Figure 4 compares the runtime per instance (grey bars) and the mean safety risk (purple bars) for different SafeGen methods: Best-of-N, CSD, and CSD+KV Cache. The results provide insights into the trade-offs between computational efficiency and safety performance. We have the following ob- servations: (1) Best-of-N incurs the highest runtime, exceed- ing 40 seconds per instance, while also exhibiting the highest mean safety risk. This indicates that generating multiple completions and selecting the safest one is computationally expensive without significantly improving safety. (2) CSD achieves a substantial reduction in runtime (approximately 7 seconds per instance) while also lowering the mean safety risk compared Best-of-N. This demonstrates the efficiency of the CSD method in balancing safety and speed. (3) CSD+KV Cache further reduces the runtime (below 5 seconds per in- stance) while maintaining a similar mean safety risk to CSD. This highlights the effectiveness of KV caching in accelerat- ing SafeGen methods without compromising safety.
40
Rurtime Instance (seconds)
35
0.4
30
Mean Safety Nisk
25
20
0.
15
10
Best-af-N
CSD
CSD+KV Cache 0.0
Figure 4: Evaluation of decoding run- time and mean safety risk for the strongest baseline Best-of-N, our proposed Safe- Gen protocol CSD, and CSD with KV cache. CSD achieves better efficiency- safety tradeoff than Best-of-N. KV cache preserves safety while significantly reduc- ing decoding time.
The results indicate that CSD+KV Cache provides the best trade-off' between efficiency and safety, significantly reducing computational cost while maintaining low safety risk. Best-of -N, despite being simple approach, is highly inefficient, making it impractical for real-time applications. These findings emphasize the importance of optimizing SafeGen methods for both speed and reliability.
6 Conclusion
The rapid adoption of open-source LLMs underscores the urgent need for rigorous safety guarantees in their generations, particularly in high-stakes applications. Existing empirical defenses, including RLHF, guardrails, and decoding- time interventions, lack formal assurances, limiting their reliability against adversarial exploits. To address this gap, we introduced C- C-Saf a model-agnostic certification framework that provides theoretical risk bounds and enables the enforcement provable safety constraints. Complemented by our novel CSD decoding algorithm, C-SafeGen ensures both safety and fluency in generated text. Empirical evaluations confirm the efficacy of our approach, demonstrating its potential as robust foundation for the deployment of provably safe LLMs.

---

Acknowledgements
This work is partially supported by the National Science Foundation under grant No. 1910100, No. 2046726, NSF AI Institute ACTION No. IIS-2229876, DARPA TIAMAT No. 80321, the National Aeronautics and Space Administration (NASA) under grant No. 8ONSSC20M0229, ARL Grant W911NF-23-2-0137,, Alfred P. Sloan Fellowship. the research grant from eBay. Al Safety Fund, Virtue AI, and Schmidt Science.
References
[1] Ai content moderation by microsoft azure. services/ai-comttnt--feee)
[2] Detoxify by unitary ai. https//ggthabbccom//tttaay/aaaat
[3] Anastasios N Angelopoulos, Stephen Bates, Emmanuel Candès, Michael Jordan, and Lihua Lei. Learn then test: Calibrating predictive algorithms to achieve risk control. arXiv preprint w:Xi::211000105: 2021.
[4] Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. Confor- mal risk control. arXiv preprint arXiv:2208.0288 028114 2022.
[5] Luke Bates and Iryna Gurevych. Like good nearest neighbor: Practical content moderation with sentence transformers. arXiv e-prints, pages arXiv-2302, 2023.
[6] Stephen Bates, Anastasios Angelopoulos, Lihua Lei, Jitendra Malik, and Michael Jordan. Distribution-free, risk-controlling prediction sets. Journal of the ACM (JACM), 68(6):1-34, 2021.
[7] Vidmantas Bentkus. On hoeffding's incqualities. 2004.
[8] Patrick Chao, Edoardo Debenedetti, Alexander Robey, Maksym Andriushchenko, Francesco Croce, Vikash Sehwag, Edgar Dobriban, Nicolas Flammarion, George J. Pappas, Florian Tramèr, Hamed Hassani, and Erie Wong. Jailbreakhench: An open robustness benchmark for jailbreaking large language models. In NeurIPS Datasets and Benchmarks Track, 2024.
[9] Patrick Chao, Alexander Robey, Edgar Dobriban, Hamed Hassani, George Pappas, and Eric Wong. Jailbreaking black box large language models in twenty queries. arXiv preprint 1rXiv:2310.0841 2023.
[10] Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv preprint 2407: 1783, 2024.
[11] Shaona Ghosh, Prasoon Varshney, Erick Galinkin, and Christopher Parisien. Aegis: Online adap- tive ai content safety moderation with ensemble of Ilm experts. arXiv preprint arXiv:2404.05993, 2024.
[12] Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, et al. Deepseek-r1: Incentivizing reasoning capability in IIms via reinforcement learning, arXiv preprint arXiv:2501.12944 2025.
[13] Wassily Hoeffding. Probability inequalities for sums of bounded random variables. The collected works of Wassily Hoeffding, pages 409-426, 1994.
[14] Sture Holm. A simple sequentially rejective multiple test procedure. Scandinavian journal of statistics, pages 65-70, 1979.
[15] Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations. arXiv preprint arXiv:2312.06674 2023,
10

---

[16] Neel Jain, Avi Schwarzschild, Yuxin Wen, Gowthami Somepalli, John Kirchenbauer, Ping-yeh Chiang, Micah Goldblum, Aniruddha Saha, Jonas Geiping, and Tom Goldstein. Baseline de- fenses for adversarial attacks against aligned language models. arXiv preprint 4rX1v:2309.00114 2023.
[17] Mintong Kang, Chejian Xu, and Bo Li. Advwave: Stealthy adversarial jailbreak attack against large audio-language models. arXiv preprint arXiv:2412.006006 2024.
[18] Deepak Kumar, Yousef AbuHashem, and Zakir Durumeric. Watch your language: Investigating content moderation with large language models. arXiv preprint arXiv:2309, 14517, 2024.
[19] Alyssa Lees, Vinh Q Tran, Yi Tay, Jeffrey Sorensen, Jai Gupta, Donald Metzler, and Lucy Vasserman. A new generation of perspective api: Efficient multilingual character-level trans- formers. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3197-3207, 2022.
[20] Jing Lei, James Robins, and Larry Wasserman, Distribution-fres prediction sets. Journal of the American Statistical Association, 108(501):278-287, 2013.
[21] Yuhui Li, Fangyun Wei, Jinjing Zhao, Chao Zhang, and Hongyang Zhang. Rain: Your language models can align themselves without finetuning. arXiv preprint arXiv:2309.0712 07124, 2023.
[22] Zi Lin, Zihan Wang, Yongqi Tong, Yangkun Wang, Yuxin Guo, Yujia Wang, and Jingbo Shang. Toxicchat: Unveiling hidden challenges of toxicity detection in real-world user-ai conversation. arXiv preprint arXiv:2310. 17389, 2023.
[23] Xiaogeng Liu, Nan Xu, Muhao Chen, and Chaowei Xiao. Autodan: Generating stealthy jailbreak prompts on aligned large language models. arXiv preprint arXiv:2310.04451, 2023.
[24] Huan Ma, Changqing Zhang, Huazhu Fu, Peilin Zhao, and Bingzhe Wu. Adapting large language models for content moderation: Pitfalls in data engineering and supervised fine-tuning. arXiv preprint 10.003000 2023.
[25] Todor Markov, Chong Zhang, Sandhini Agarwal, Florentine Eloundou Nekoul, Theodore Lee, Steven Adler, Angela Jiang, and Lilian Weng. A holistic approach to undesired content detection in the real world. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 15009-15018, 2023.
[26] Anay Mehrotra, Manolis Zampetakis, Paul Kassianik, Blaine Nelson, Hyrum Anderson, Yaron Singer, and Amin Karbasi. Tree of attacks: Jailbreaking black- box Ilms automatically. arXiv preprint arXiv:2312.022133 2023.
[27] Christopher Mohri and Tatsunori Hashimoto. Language models with conformal factuality guarantees. arXiv preprint arXiv:2402. 10978, 2024.
[28] Jingwei Ni, Minjing Shi, Dominik Stammbach, Mrinmaya Sachan, Elliott Ash, and Markus Leippold. Afacta: Assisting the annotation of factual claim detection with reliable Ilm annotators. arXiv preprint arXh::2402.11077. 2024.
[29]
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Balteseu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bermadett-Shapiro, Christo- pher Berner, Lenny Bogdonoff. Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage. Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Loper Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei
11

---

Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Lukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung. Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin. Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Man- ning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine Mcl.avey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen Oeeefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pan- tuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto. Michael, Pokomy, Michelle Pokrass, Vitchyr Pong. Tolly Powell, Alethea Power, Boris Power, Elizabeth Prochl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry. Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Kata- rina Slama, Ian Sohl, Benjamin Sokolowsky. Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang. Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder. Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang. Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2023.
[30] Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730-27744, 2022.
[31] Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, and Regina Barzilay. Conformal language modeling. arXiv preprint 2301:23306.11193, 2023.
[32] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. Advances in Neural Information Processing Systems, 36, 2024.
[33] Traian Rebedea, Razvan Dinu, Makesh Sreedhar, Christopher Parisien, and Jonathan Cohen. Nemo guardrails: A toolkit for controllable and safe 11m applications with programmable rails. arXiv preprint arXiv:2310. 10501, 2023.
[34] Alexander Robey, Eric Wong, Hamed Hassani, and George J Pappas. Smoothllm: Defending large language models against jailbreaking attacks. arXiv preprint arXiv:2310.03684 2023.
[35] Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, and Andrea Zanette. Fast best-of-n decoding via speculative rejection. arXiv preprint arXiv:2410.20290 2024.
[36] Fei Tan, Yifan Hu, Kevin Yen, and Changwei Hu, Bert-beta: A proactive probabilistic approach to text moderation. arXiv preprint
2021.
12

---

[37] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 13971, 2023.
[38] Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. Algorithmic learning in random world, volume 29. Springer, 2005.
[39] Volodya Vovk, Alexander Gammerman, and Craig Saunders. Machine-learning applications of algorithmic randomness. 1999.
[40] Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe V Le, Denny Zhou, et al. Chain-of-thouge prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
[41] Zeming Wei, Yifei Wang, Ang Li, Yichuan Mo, and Yisen Wang. Jailbreak and guard aligned language models with only few in-context demonstrations. arXiv preprint 10.063877 2023.
[42] Yueqi Xie, Jingwei Yi, Jiawei Shao, Justin Curl, Lingjuan Lyu, Qifeng Chen, Xing Xie, and Fangzhao Wu. Defending chatgpt against jailbreak attack via self-reminders. Nature Machine Intelligence, 5(12): 2023.
[43] Chejian Xu, Mintong Kang, Jiawei Zhang, Zeyi Liao, Lingbo Mo, Mengqi Yuan, Huan Sun, and Bo Li. Advweb: Controllable black-box attacks vim-powered web agents. arXiv preprint arXiv:2410. 17401, 2024.
[44] Zhangchen Xu, Fengqing Jiang, Luyao Niu, Jinyuan Jia, Bill Yuchen Lin, and Radha Pooven- dran. Safedecoding: Defending against jailbreak attacks via safety-aware decoding. arXiv preprint arXiv:2402.08983 2024.
[45] Wenkai Yang. Xiaohan Bi, Yankai Lin, Sishuo Chen, Jie Zhou, and Xu Sun. Watch out for your agents! investigating backdoor threats to Ilm-based agents. arXiv preprint 2rXiv:2402.112000 2024.
[46] Yachong Yang and Arun Kumar Kuchibhotla. Finite-sample efficient conformal prediction. arXiv preprint arXiv:2 13871, 2021.
[47] Zhuowen Yuan, Zidi Xiong, Yi Zeng, Ning Yu, Ruoxi Jia, Dawn Song, and Bo Li. Rigorllm: Resilient guardrails for large language models against undesired content. arXiv preprint 1xXIv:2403.130311 2024.
[48] Chujie Zheng. Fan Yin, Hao Zhou, Fandong Meng, Jie Zhou, Kai-Wei Chang, Minlie Huang, and Nanyun Peng. Prompt-driven IIm safeguarding via directed representation optimization. arXiv preprint arXiv:2401. 18018, 2024.
[49] Xuhui Zhou. Challenges in automated debiasing for toxic language detection. University of Washington, 2020.
[50] Andy Zou, Zifan Wang. J Zico Kolter, and Matt Fredrikson. Universal and transferable adversarial attacks aligned language models. arXiv preprint arXiv:2307.15043 2023.
13

---

A Omitted proofs
A.1 Proof of Theorem
Proof Theorem The proof sketch follows B Since the risk function R( is upper bounded
by 1, we can apply tighter version of Hoeffding's inequality [33 for à > E| R (x))):
P [[R Pa(x)) 2 à) - exp { - (Naah(((DDaa))
(12)
Also, applying Bentkus inequality 71 we have:
P P[Ro(r, Pa (x)) 2 a) I elP Bin((Na, () s
Naw (((
(13)
Combining Equations 12 and (T), we have:
a] min (exp { -Naah (R(Dem), a))} eP Bin Bin à) s
Nea (Daa)
1)
(14)
Or equivalently, given uncertainty 6, have:
8 = min exp
{ - Neath (i) elP Bin â)
((Dea(
(15)
which leads to the following by formulating an inverse function:
1- [ln(1/6), (In(1/6),,
:R(Dea))
9 ( Neat, Pu(Dan))
= min
(16)
Neal
A.2 Proof of Theorem
Proof of Theorem 2 The proof follows 14 We consider |A] independent hypothesis test correspond-
ing to the | Null hypothesis. By the Bonferroni method, each test is performed significance
level of A Therefore, The probability of not making a Type I error in a single test is 1 [
8 )IAI, The probability of mak- [A
The probability of making no Type I error in all |A| tests is (1
ing at least one Type I error (i.e., FWER) is the complement of making Type errors, which is ( MIر < 6. Therefore, we prove that the familywise error rate ö for Bonferroni correction.
1-
A
Thus, going back to the risk guarantee, we conclude the proof.
A.3 Proof of Theorem
T be Peca- To show the result, we need to show that for any output sequence E 3/ Poca(y) = PG(().
Proof of Theorem 3
Let the output distribution by Algorithm
We consider the following scenarios respectively: (1) this is invalid output by the guardrail model: G(y) 0; (2) this valid output: G(y) =
Scenario (1): G(y) = O. According to CCD algorithm, such an undesired claim would always be associated with 1.0 backtrack probability in Line of Algorithm I and thus we have Peed (v) = On the other hand, according Equation (10). we always have Pc (y 0 when G(y) 0. Therefore,
we have Pesa (1) = (y) when G(y) 0.
Scenario (2): G(y) = 1. In this scenario, we basically would like to validate that the backtrack mechanism in Lines 7-9 of Algorithm | does not distort the distribution.
Part (a): To show that, we first validate that the support of output distribution by CCD is identical to that of the restricted output distribution.
For any valid support in the restricted output distribution = [ýv 02, NN., there always exists an unrisky claim decoding path. We consider this simple claim decoding path W1, V2. .., ÜN Due to
14

---

Assumption 4.1 since the complete output sequence o is unrisky, then each claim prefix would also be unrisky (otherwise violating the claim risk cascade assumption). Therefore, this claim decoding path is valid instantiation of CCD algorithm.
Considering the reverse direction, for any valid support output y by CCD algorithm, it is obvious that G(y) = 1 given the acceptance of the last claim, and thus, this output is also valid support of the restricted output distribution pG.
Part (b): Then, we will prove that for any valid support y = [ū1, U2 Øy] such that pa (v) > 0 and Peedly) 0, we have (y) Poad(y).
For ease of notation, we notate a Eyey p(y) 1], which is a normalization factor in Equation (0).. Then following the chain rule, we can compute (v) as:
po(y) = p(ū (1(ū2 p(ūN lūt. .N-
(17)
Since is a desired output (i.e., G(y) = 1), each claim prefix is also desired according to the risk
cascade assumption in Assumption 4.1 Then we have:
pa (y)) = p(u G(b)) = 1)p(2, G(ld), ūel) = 1(81) .. -ii. G( ..., UN() 1/ū,, .... UN-1) (18)
Then we analyze the point mass of output y by the CCD algorithm Poed(y). We can formulate the decoding path of CCD algorithm that leads to output as Ur, U1, D2. V2..... YN UN. where D.. denotes a sequenee of backtracked risky claims at time step n.
Then we prove a key invariability property that
p(ūn. G(Ii...... Bnl) = 10. In-1) = p((nn Dn. G((er,........ = 1182,.... Fn-1) (19)
We can derive as the following:
pllūn Un. G(li Ønl) 1101, ..., ...
(20)
-E C(in. G(ii, Rnl) = 1/ In )p(()(((n)) (r. = 1/0 ..., Øn-1, mn)
(21)
Nn
Dp(d.. G( .... Unl) 1/; .... in
)p( G(ū Bn1) = 11 .... F (22)
For ease of notation, let u = plūn. G(, 0 01. .... Hn and p(On. G(ir..... Fnl)
1 1lū1. .... 1n-1), then we have:
Σ p(än, G(ū,,..., (n)))) 1 (1,,,.., T. W(ūn G (W Fn.) = 1/ .... .0.,h-.....)) Fn-1)
(23)
In
=(1 w)v u(1 u)v + u² u)v u² u)c +...
(24)
=
(25)
=p(ūn, G(ii,. ..., Unl) ..., (.n1)).nn-1))
(26)
Therefore, have:
Peea
a II
Σ
plnn Un, G(ii...., Fnl) 110,, ..., Fn-1)
(27)
1
II plun G(I 1 16 .... 9n-1)
(28)
(v)
(29)
15

---

B Discussion and Limitation
The C-eafeGen framework is able to safeguard LLMs' practical deployment and applications against ethical and societal concerns. Existing research shows that the responses of LLMs can be biased towards some demographic groups and not be aligned with human ethics. With C-SafeGen we can define bias/ethics risk function and control the generation risk below desired level. The risk guarantee provided by C-SafeGen enhances the use of LLMs, addressing societal issues and regulatory infringements. We do not expect any negative societal consequences for our work.
While C-SafeGen provides formal guarantees under mild assumptions, it currently relies on the availability of reliable external classifiers risk estimators to define the safety risk function. The effectiveness of the certification heavily depends on the quality and coverage these components, which may be incomplete biased in practice. Additionally, C-SafeGen operates in black-box set- ting and does not account for internal model behaviors or hidden representations that might influence safety. Finally, although C-SafeGen bounds risk the evaluation distribution, distributional shifts at deployment time (e.g., novel user queries or emerging adversarial strategies) may compromise safety, highlighting the need for future extensions toward adaptive online certification mechanisms.
16

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We include theoretical proofs in Appendix A and evaluation details in Section
5.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We include that in Appendix B.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model sell-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
17

---

Answer:
[Yes]
Justification:
The proof is in Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We include details in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
18

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will do so on paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurIPS code and data submission guidelines (https
/ "pipsscc/puullc//uuu /ttpe//gguddeeeeeeouuuuumbbbba ssionPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include that in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We do multiple runs to reduce sample error.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
19

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We include that in Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We follow the code of ethics during the project.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[Yes]
Justification: include that Appendix
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
20

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do not release data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
The datasets are all in valid use.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
21

---

If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do involve that.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/mmoel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not involve that.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question:
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
We do not involve that.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
16. Declaration of LLM usage
22

---

Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [Yes]
Justification: We provide it in meta data of paper submission.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy fattps://neurd/: httppp/....//oooeeeeeee////////. for what should or should not be described.
23


Reviews:
[{"review_id": "HoyvVLSZJJ", "reviewer": "Reviewer_Mjhm", "summary": "This paper deals with the safety guardrail of LLM during inference time. To fill the gap of lacking rigorous safety guarantees in previous approaches, the authors propose a claim-based streaming decoding algorithm with risk control by conformal analysis. This involves a stream guardrail model and a claim partition model with backtracking for improved efficiency. The authors also provide theoretical guarantees by deriving the upper bound of safety risk. Empirical experimental results showcase the efficiency and efficacy of the proposed approach.", "questions": "1.\tDo you have any ablation for the safe resampling techniques? And which one of the introduced techniques (random masking/permutation/noise injection/structured permutation) is the most effective and efficient? \n2.\tHow is the ShieldGemma-9B implemented and used in the evaluation? ShieldGemma can only make a binary decision for one given class in one inference.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths\n1.\tThe risk control and safety guarantees of LLM generation during inference time are significant.\n2.\tThe theoretical guarantee is solid with a derived upper bound that allows better control.\n3.\tThe introduced claim separation and KV cache for guardrail models improve the efficiency of inference. \n4.\tThe paper is clearly written and easy to follow.\n\n## Weaknesses:\n1.\tReliance on the output probability of guardrail models may not be reliable enough. Existing work [1] found that guard models like Llama-Guard3 tend to be overconfident and miscalibrated under jailbreak attacks, limiting the safety estimation in the framework. \n2.\tLimited attacks are investigated except for GCG. HarmBench [2] includes multiple types of attacks that may help investigate the effectiveness of the proposed method under different attacks.\n\n## References\n[1] Liu, H., Huang, H., Gu, X., Wang, H., & Wang, Y. (2025). On calibration of LLM-based guard models for reliable content moderation. ICLR. \\\n[2] Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., ... & Hendrycks, D. (2024). Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. ICML.", "was_revised": true, "final_justification": "My only unsolved concern is from the evaluation of the limited attacks. Even though the authors claim they evaluated the strongest known white-box jailbreak technique and would incorporate results from black-box jailbreak methods in the revised version, I do not see any additional evaluation results in the current rebuttal. Regardless, I am fine if the paper gets accepted", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!\n\n> Q1: Discussion on using the output probability of guardrail models as safety risk.\n\nThank you for highlighting this interesting work! We agree that improved calibration of output probabilities can lead to a more accurate safety risk function and further enhance the effectiveness of our SafeGen framework. We will include a discussion of this direction in our revision.\n\n[1] Liu, H., Huang, H., Gu, X., Wang, H., & Wang, Y. (2025). *On calibration of LLM-based guard models for reliable content moderation*. ICLR.\n\n\n\n> Q2: Evaluations for different safe resampling techniques and additional attack types.\n\nThank you for the suggestion! We conduct an ablation study to evaluate the impact of each individual safe resampling technique used in safe sampling. We examine the effect of removing each of the following components in Table 1. We observe that excluding any of the individual techniques leads to an increase in conformal safety risk, indicating that each component contributes to overall safety. The full combination of all techniques achieves the lowest risk.\nFor attacks, we use GCG, the strongest known white-box jailbreak technique. In our revised version, we will also incorporate results from black-box jailbreak methods to broaden the evaluation scope.\n\nTable 1: Ablation Study on Safe Resampling Techniques on AdvBench with Llama3-8B and LlamaGuard3-8B as guardrail evaluator.\n\n| Safe Resampling Configuration     | Conformal Safety Risk |\n|----------------------------------|------------------------|\n| Excluding Random Masking         | 0.232                  |\n| Excluding Random Permutation     | 0.235                  |\n| Excluding Gaussian Noise Injection | 0.213                |\n| All Techniques Combined (Full CSD)| **0.210**             |\n\n\n> Q3: Implementation details of ShieldGemma-9B.\n\nWe follow the official Hugging Face implementation of ShieldGemma-9B, incorporating its four risk categories into the prompt template as the `safety_policy` input. This enables the model to perform safe/unsafe judgments based on those predefined categories with one inference. We will clarify this setup in the evaluation section of the revised version."}, {"review_id": "xNsM72B8LB", "reviewer": "Reviewer_NPt4", "summary": "This paper proposes a Claim-based Stream Decoding (CSD) algorithm coupled with a statistical risk guarantee framework using conformal analysis. CSD algorithm integrates a stream guardrail model to safeguard sequential claims generated by LLMs and incorporates a backtracking mechanism to revise claims flagged with high safety risks.  Empirical evaluations demonstrate the effectiveness and efficiency of the CSD algorithm compared to baselines.", "questions": "Could the authors show on how sensitive results are to the choice of the threshold $\\alpha_{b}$ and $\\alpha_{g}$?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n1. The paper is clearly written and well-structured.\n\n2. The methodology is well-developed and theoretically grounded. \n\n\nWeaknesses:\n1.  Could the authors elaborate more about when Assumption 4.1 hold and when it fails in the real world?\n\n2. The paper focuses on safety rates but does not quantitatively evaluate the effect of CSD on output utility. While it is stated that CSD “ensures fluency and coherence” and preserves “high-quality outputs”, no evaluations are provided. It would be important to measure generation utility after enforcing safety. \n\n3. The experiments focus on two “jailbreak” style benchmarks and adversarial suffix attacks. It is unclear how C-SafeGen would perform on other safety domains. The generalizability beyond the tested scenarios is somewhat uncertain. I suggest the authors involve a discussion or providing examples on other potential safety domains.", "was_revised": true, "final_justification": "After reading other reviews, there are still a few concerns unresolved. It is uncertain how the final version looks like. I therefore maintain my score.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!\n\n> Q1: When does Assumption 4.1 hold or fail in real-world settings?\n\nAssumption 4.1 states that if a generation contains an unsafe statement, then any continuation of that statement remains unsafe. This assumption generally holds in many real-world cases involving safety-sensitive topics. For example, if an LLM generates bomb-making instructions, the entire response is considered unsafe regardless of what follows—even if the model later includes an apology—because the harmful content remains accessible.\n\nHowever, this assumption may not always hold, particularly in cases involving nuanced or context-dependent risks such as hate speech. For instance, if a model initially generates a harmful statement but then immediately retracts it with something like, “I said something wrong, and my actual point is...,” the overall response might be deemed safe under certain criteria.\n\nAs discussed in the paper, Assumption 4.1 represents a **conservative stance** on safety. It errs on the side of caution by treating any continuation of an unsafe segment as unsafe, which aligns with our goal of enforcing strict safety guarantees in high-risk domains.\n\n\n\n> Q2: Additional utility evaluation of CSD algorithm.\n\nThank you for the valuable feedback! In response to your suggestion, we evaluate **Llama-3.1-8B** on **MMLU-Pro** as a benign utility test, shown in Table 1. The results align with our theoretical analysis in **Theorem 3**, which states that CSD generation preserves the restricted output distribution. Since MMLU-Pro contains only benign inputs that do not trigger unsafe responses, CSD does not initiate any backtracking—resulting in **no drop in benign utility**. We would like to emphasize that the CSD algorithm relies on a backtracking mechanism triggered only upon detection of unsafe content. Therefore, when evaluating on fully benign domains, backtracking is rarely invoked—ensuring that utility remains unaffected.\n\n**Table 1: MMLU-Pro score of Llama-3.1-8B under vanilla and CSD generation (using LlamaGuard3-8B as the guardrail model)**\n\n| Generation Method | Overall Score |\n|-------------------|---------------|\n| Vanilla           | 0.441         |\n\n> Q3: Generalization to other safety domains.\n\nC-SafeGen is **guardrail-agnostic** and can be readily extended to domains such as misinformation and toxicity, provided that an appropriate guardrail evaluator is available. In fact, the AdvBench and JailbreakBench datasets used in our evaluation already encompass a diverse range of safety risk categories. We will include more category-specific results in the final version to better illustrate C-SafeGen's effectiveness across different risk types.\n\n> Q4: Ablation studies of $\\alpha_b$ and $\\alpha_g$.\n\nThank you for the suggestion! We already include a sensitivity analysis of the backtrack threshold $\\alpha_b$in **Figure 3**, which demonstrates that increasing the ease of backtracking effectively reduces safety risk.\n\nIn addition, we present an ablation study of the **stagnation tolerance** parameter $\\alpha_g$ in **Table 2**. The results show that stricter stagnation tolerance (i.e., smaller $\\alpha_g$) leads to improved safety performance under the same sampling budget.\n\n**Table 2: Ablation of \\(\\alpha_g\\) on AdvBench with Llama-3.1-8B (guardrail: LlamaGuard3-8B)**\n| $\\\\alpha_g$                  | 64    | 128   | 256   | $\\infty$ (no safe sampling) |\n|------------------------------|-------|--------|--------|-------------------------------|\n| Conformal Safety Risk        | 0.200 | 0.210  | 0.215  | 0.293                         |"}, {"review_id": "FiDyj51l7s", "reviewer": "Reviewer_qLuo", "summary": "This paper introduces C-SafeGen, a certification framework that provides theoretical safety risk guarantees for large language model (LLM) generations. The framework includes a novel decoding algorithm, Claim-based Stream Decoding (CSD), which uses a guardrail model to monitor claims and claim \"segmentation\" method, that divides the input prompt into a set of non-overlapping claims. Empirical evaluations show that CSD reduces unsafe generations compared to existing methods.", "questions": "1. A clarification question: equation 7 defines a \"claim backtrack probability\" function, but the defined quantity is an indicator function, not a probability. Can the authors clarify?\n\n2. A question regarding the design choices of the method: Instead of segmenting the output into claims, some alternate approaches might be: \n    - cumulative safety checking every \"K\" generated output tokens by the guardrail model, with backtracking\n    - eliminate the claim-based system, and have the model re-generate the output if it is deemed unsafe by the guardrail model\n\nI wonder whether the authors have considered other such simpler design choices for evaluation? Can the authors comment on the trade-offs between different design choices?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n+ The methodology introduced by this paper is simple, effective and theoretically sound. It involves using guardrail models that already exist, but instead of using these to filter unsafe outputs; this paper proposes to use these to create an alternate decoding mechanism to enable models to create safe outputs instead, instead of simply blocking the generation of an unsafe one.\n+ The experimental results are sound. The proposed approach lowers the safety risk of outputs while being more computationally efficient compared to approaches such as best-of-N.\n+ The writing is clear, concise, and to the point.\n\nWeaknesses:\n- The main weakness in this paper is a lack of evaluations of utility of the resulting safe outputs. In principle, it is trivial to certify safety: simply refuse to produce an output, or produce the same benign output (e.g.: \"I cannot answer this question\") for every prompt. The main question is whether we can certify safety while ensuring that the utility of the LLM does not deteriorate significantly. While Theorem 3 in the paper argues for this theoretically, empirical evaluations on this end would help better motivate this approach.\n\n- Another weakness of this paper is the usage of a rather simple claim-point detection method. For simplicity, the paper chooses to detect claims by identifying newlines, periods, etc. This has several obvious failure cases, for example, when a single long sentence may have multiple claims, or a claim being spread across sentences. Thus this choice appears to be ad-hoc, and the lack of discussion around this topic is a weakness.", "was_revised": true, "final_justification": "The author rebuttals addressed my questions regarding utility of the underlying approach. As a result, I am increasing my score and recommending an accept.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!\n\n> Q1: Additional utility evaluation of CSD algorithm.\n\nThank you for the valuable feedback! In response to your suggestion, we evaluate **Llama-3.1-8B** on **MMLU-Pro** as a benign utility test, shown in Table 1. The results align with our theoretical analysis in **Theorem 3**, which states that CSD generation preserves the restricted output distribution. Since MMLU-Pro contains only benign inputs that do not trigger unsafe responses, CSD does not initiate any backtracking—resulting in **no drop in benign utility**. We would like to emphasize that the CSD algorithm relies on a backtracking mechanism triggered only upon detection of unsafe content. Therefore, when evaluating on fully benign domains, backtracking is rarely invoked—ensuring that utility remains unaffected.\n\n**Table 1: MMLU-Pro score of Llama-3.1-8B under vanilla and CSD generation (using LlamaGuard3-8B as the guardrail model)**\n\n| Generation Method | Overall Score |\n|-------------------|---------------|\n| Vanilla           | 0.441         |\n| CSD               | 0.441         |\n\n> Q2: Discussion on claim-point detection methods. \n\nThank you for raising this insightful point! We agree that if a language model uses a different set of claim termination tokens than the ones we define, the effectiveness of claim-based generation could be compromised. In such cases, it may be necessary to tune the termination token set accordingly. We will add this clarification in our revision.\n\nWe also appreciate your suggestions on alternative ways to define claims and incorporate safety checks. Specifically:\n\n1. **Fixed-length claims**: Treat every *k* tokens as a claim and perform safety checks and potential backtracking after each segment.\n2. **Full-response claims**: Treat the entire response as a single claim and perform a post hoc safety check with potential resampling.\n\nOption 2 corresponds to **Best-of-N sampling**, which we have already evaluated. As shown in our results, CSD significantly outperforms Best-of-N under the same computational budget. For Option 1, we believe it may lead to inefficiencies and suboptimal performance due to local semantic dependencies. For example, if *k = 5* and the model generates a safe first segment like “Here is how to make,” the next segment is likely to complete the phrase in an unsafe way (e.g., “bombs: ...”) due to the semantic trajectory of the generation. This illustrates why **semantically meaningful termination points** are important for effective and efficient safety checks.\n\nMotivated by this, we adopt a **claim-based decoding** strategy, where claims are semantically grounded units that serve as natural checkpoints for safety validation and potential backtracking. We will clarify this design motivation in our updated version.\n\n\n\n> Q3: Confusion on the backtrack probability function.\n\nApologies for the confusion. We will revise the name to **claim backtrack decision function** and update **Algorithm 1** accordingly to ensure consistency and clarity."}, {"review_id": "dWvR2PM0Wp", "reviewer": "Reviewer_1E9p", "summary": "To address the challenge of safe LLM generation with certified guarantees, this paper proposes a Claim-based Stream Decoding (CSD) algorithm coupled with a statistical risk guarantee framework using conformal analysis.   \nSpecifically, CSD utilizes an off-the-shelf Guardrail model to backtrack and resample claim streams if flagged with risks.  \nThis paper also provides theoretical proof of the proposed method.  \nExperiments against five baselines demonstrate the effectiveness of CSD in safety protection.", "questions": "**Questions**:\n1. Could you provide a randomized baseline that produces random outputs, in order to estimate a lower-bound threshold for safety?\n\n***\n**Typos**:\n1. Line 90: Does \"SafeGen\" refer to \"C-SafeGen\" or to safe generation in general?\n2. Line 108: Should \"extitSafeGen\" be \"SafeGen\"?\n3. Line 138: Should \"below $\\alpha$\" be corrected to \"below $r$\"?\n4. Line 239: \"four baselines\" but five are listed.\n***", "limitations": "The authors have discussed the limitations in the Appendix.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 1, "originality": 2, "strengths_and_weaknesses": "**Strengths**:  \n1. This paper addresses the important problem of safety-guaranteed LLM generation.\n2. The proposed method CSD, which utilizes an off-the-shelf Guardrail model to backtrack and resample claim streams, is effective at preserving safety according to the experiments.\n3. A specific design is introduced to improve the efficiency of CSD by caching previous verified claims.\n***\n**Weaknesses**:\n1. The paper is hard to follow.\n    * Some formulations are unnecessary\n    * The core idea and design of C-SafeGen are not clearly highlighted. I had to carefully read Algorithm 1 to grasp the full picture.\n2. It is unclear how the SafeGen protocol $P_a$  is calculated. Specifically, the meaning of \"configuration\" in this context is vague. Does it mean the A more detailed explanation of key terms is needed.\n3. The experients are not sufficient. For example, there are no experiments to verify potential performance degradation caused by CSD. Without such evaluation, it is unclear whether the safety protection actually works. Additionally, the efficiency comparison lacks baselines such as *Vanilla*.\n4. The experimental details are not adequately explained. \n    * For evaluation metrics, I can only tell which models are used in Section 5.1.\n    * Neither prompts nor open-source code are provided for the function $R_G$.\n***", "was_revised": true, "final_justification": "Some of my concerns — specifically weaknesses 3 and 4 — have been addressed. \n\nFor weaknesses 1 and 2, I am not fully convinced they have been resolved, but the authors have promised to improve the formulation, enhance the writing, and correct the typos. I trust the authors to deliver on this commitment and have therefore raised my score from 2 to 4.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!\n\n> Q1: Improvement of writing.\n\nThank you for the valuable suggestion! We will include an overview figure to illustrate the two types of certification presented in the paper. (1) We provide conformal analysis-based certification for safety risk, as formalized in Theorems 1 and 2. Theorem 1 establishes a statistical upper bound on safety risk, while Theorem 2 certifies configurations that satisfy a target nominal risk level. (2) We also provide algorithmic certification for our proposed Claim-based Stream Decoding (CSD) algorithm, as detailed in Theorem 3. This guarantees that CSD preserves a restricted output distribution that excludes the support of unsafe outputs, as defined in Definition 1. Additionally, we will simplify the notations, provide a notation table, and correct typos to improve clarity and readability.\n\n> Q2: Clarification of the certified configuration.\n\n\nThe **configuration** refers to the set of hyperparameters used in our CSD generation process, including the backtrack probability $\\alpha_b$, stagnation tolerance $\\alpha_g$, and minimum claim length $\\alpha_l$. The **certified configuration** $P_\\alpha$ is derived according to **Theorem 2**. Specifically, we construct a set of null hypotheses over the safety risks associated with different configurations and retain those with acceptable p-values under **family-wise error rate (FWER) control**, ensuring a bounded overall error rate. We will include additional details in the remark following Theorem 2 for clarity.\n\n\n> Q3: More evaluations on utility and efficiency of the vanilla method.\n\nThank you for the valuable feedback! In response to your suggestion, we evaluate **Llama-3.1-8B** on **MMLU-Pro** as a benign utility test, shown in Table 1. The results align with our theoretical analysis in **Theorem 3**, which states that CSD generation preserves the restricted output distribution. Since MMLU-Pro contains only benign inputs that do not trigger unsafe responses, CSD does not initiate any backtracking—resulting in **no drop in benign utility**. We would like to emphasize that the CSD algorithm relies on a backtracking mechanism triggered only upon detection of unsafe content. Therefore, when evaluating on fully benign domains, backtracking is rarely invoked—ensuring that utility remains unaffected.\n\n\nIn addition, we provide efficiency and safety risk evaluations of the **vanilla method** in **Table 2**, alongside other decoding strategies. While the vanilla method incurs minimal inference cost, it suffers from significantly higher safety risk compared to CSD.\n\nRegarding the evaluation of a randomized baseline as a lower bound on safety, we respectfully disagree that randomized outputs necessarily represent the worst-case (i.e., most unsafe) responses. Random sampling may not consistently produce harmful outputs. We would appreciate further clarification on this suggestion and are open to conducting additional evaluations based on a more detailed formulation.\n\n\n\n**Table 1: MMLU-Pro score of Llama-3.1-8B under vanilla and CSD generation (using LlamaGuard3-8B as the guardrail model)**\n\n| Generation Method | Overall Score |\n|-------------------|---------------|\n| Vanilla           | 0.441         |\n| CSD               | 0.441         |\n\n**Table 2: Evaluation of Decoding Runtime and Mean Safety Risk**\n\n| Method           | Runtime per Instance (s) | Mean Safety Risk |\n|------------------|---------------------------|------------------|\n| Vanilla          | 1.34                      | 0.83             |\n| Best-of-N        | 41.50                     | 0.42             |\n| CSD              | 7.53                      | 0.12             |\n| CSD + KV Cache   | 4.05                      | 0.11             |\n\n\n\n\n\n\n\n> Q4: Clarification of experiment details.\n\nAs described in Section 5.1, we use LlamaGuard3-8B and ShieldGemma-9B as the guardrail models for evaluating the safety risk function. We follow the Hugging Face implementations and use the official prompts provided for both models. We will clarify this in Section 5.1 and also explicitly highlight the inference model and the evaluation guardrail model in each table to improve clarity and understanding."}, {"review_id": "zaq8kuwHmX", "reviewer": "Reviewer_F15f", "summary": "Most defenses in the field of LLM adversarial robustness are empirical. The authors address this gap and propose a certified detection and safe decoding defense that is based on conformal analysis. They empirically demonstrate that their derived bounds are tight and that their proposed algorithm improves safety compared to previous methods.", "questions": "- What is the proposed threat model? (Evasion attacks?)\n- Are there any assumptions made on the distribution of training and test time attacks?\n- Are there any assumptions on the expected risk and the function that provides it (e.g., could the classifier be attacked as well and provide wrong estimates?)", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* Unification of a detection framework and a safe decoding method\n* Derived theoretical bounds appear to be relevant in practice\n* Ablation studies demonstrate the importance of individual components \n* Methods to improve the runtime are proposed and ablated. The resulting algorithm is reasonably efficient. Specifically, compared to similar approaches\n\nWeaknesses:\n* Although I have knowledge about conformal principles, I found the theoretical part hard to follow. I am missing major assumptions that are made to motivate the theory about the expected safety risk etc. The CDF bounds are straightforward, but why are they reasonable in this setting / Why can we assume anything about the expected safety risk (after all, the classifier could be attacked as well and provide a wrong signal)?\n* It is not clear to me why we could use the principles of conformal prediction, for example, for evasion attacks. What are the exact threat models this paper is considering?\n* I think spending more time to explain the individual components in (3) and (4) and why we can make certain assumptions would strengthen the paper\n\nI generally think this could be a valuable contribution and am very open to improve my score after the rebuttal phase", "was_revised": true, "final_justification": "As I stated in my initial review, I believe that this is a valuable contribution.\nMy remaining concerns could be clarified in the rebuttal.", "rebuttal": "We sincerely thank the reviewer for the thoughtful feedback and constructive suggestions!\n\n> Q1. More clarification on the assumptions and certification in the paper.\n\nThank you for the valuable feedback! Our paper presents two types of certification. First, we provide **conformal analysis-based certification** for safety risk, captured in Theorem 1 and Theorem 2. Theorem 1 offers a statistical upper bound on the safety risk, while Theorem 2 certifies which configuration meets a target nominal risk level. These results rely on the standard assumption that calibration and test samples are drawn from identical distribution, which also implies that the attack distributions at calibration and inference time should be identical. Notably, we do not make any assumptions about the quality or robustness of the safety risk function itself. If the guardrail model is weak or adversarially attacked, the calibration may yield a looser bound, but the statistical guarantees still hold. Although our work focuses on the i.i.d. setting, the conformal analysis framework can be extended to non-i.i.d. scenarios—such as online or time-series data—as discussed in prior work (e.g., [1]), without changing the core statistical logic.\n\nSecond, we provide **algorithmic certification** for our proposed Claim-based Stream Decoding (CSD) algorithm, as formalized in Theorem 3. This result ensures that the CSD algorithm preserves a restricted output distribution that excludes the support of unsafe outputs, as defined in Definition 1. While the determination of safety relies on an external guardrail model, our proof does not assume that this model is accurate or robust. If the guardrail is compromised or poorly calibrated, the resulting restricted distribution may differ from the ideal safe distribution—but the algorithm will still align with the distribution as defined by the given guardrail model. In practice, having a reliable and robust guardrail is crucial for achieving strong empirical performance, but it is not required for the theoretical guarantees of our method.\n[1] Zaffran, Margaux, et al. \"Adaptive conformal predictions for time series.\" International Conference on Machine Learning. PMLR, 2022.\n\n> Q2: Clarification on the threat model.\n\nThe primary threat model considered in this paper is the jailbreak setting, where attackers craft adversarial input prompts to elicit unsafe responses from LLMs. Our use of conformal analysis is motivated by the common practice of conducting internal red teaming prior to deployment, which aims to uncover potential jailbreaks. Based on these identified attack distributions, we perform conformal calibration during this pre-deployment stage. This allows us to establish statistical safety guarantees that align with predefined safety policies or regulatory requirements at deployment time.\n\n> Q3: Writing suggestions.\n\nThank you for the suggestion! We will incorporate these discussions on the threat model and theoretical implications into the paper to enhance clarity and strengthen the overall understanding."}]

Author rebuttal:

Meta-review: This paper develops a guardrail for LLM safety with a novel Claim-based Stream Decoding (CSD) algorithm. The proposed method enables the model to create safe outputs instead of simply blocking unsafe outputs, with statistical risk guarantees. It also comes with a specific design for improving the efficiency of inference. The contributions of this paper are significant, and both theoretical and empirical results are sound. All the reviewers gave positive ratings after the rebuttal. Therefore, the AC is recommending to accept this paper. The authors are still suggested to improve the clarity of their writing in the revision.
Decision: Accept (poster)