Title: C-SafeGen: Certified Safe LLM Generation with Claim-Based Streaming Guardrails
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM safety

LLM guardrail
Abstract: Despite the remarkable capabilities of large language models (LLMs) across diverse applications, they remain vulnerable to generating content that violates safety regulations and policies. To mitigate these risks, LLMs undergo safety alignment; however, they can still be effectively jailbroken. Off-the-shelf guardrail models are commonly deployed to monitor generations, but these models primarily focus on detection rather than ensuring safe decoding of LLM outputs. Moreover, existing efforts lack rigorous safety guarantees, which are crucial for the universal deployment of LLMs and certifiable compliance with regulatory standards. In this paper, we propose a Claim-based Stream Decoding (CSD) algorithm coupled with a statistical risk guarantee framework using conformal analysis. Specifically, our CSD algorithm integrates a stream guardrail model to safeguard sequential claims generated by LLMs and incorporates a backtracking mechanism to revise claims flagged with high safety risks. We provide theoretical guarantees demonstrating that the CSD algorithm achieves the desired generation distribution subject to safety constraints. Furthermore, we introduce a generation risk certification framework and derive a high-probability upper bound on the safety risk of the proposed CSD algorithm. We extend our approach to online settings, where user queries arrive sequentially, and prove that our method can asymptotically control safety risk to any desired level. Empirical evaluations demonstrate the effectiveness and efficiency of the CSD algorithm compared to state-of-the-art safety decoding approaches. Additionally, we validate the soundness and tightness of the derived safety risk upper bound using realistic data in both offline and online scenarios.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMost defenses in the field of LLM adversarial robustness are empirical. The authors address this gap and propose a certified detection and safe decoding defense that is based on conformal analysis. They empirically demonstrate that their derived bounds are tight and that their proposed algorithm improves safety compared to previous methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* Unification of a detection framework and a safe decoding method\n* Derived theoretical bounds appear to be relevant in practice\n* Ablation studies demonstrate the importance of individual components \n* Methods to improve the runtime are proposed and ablated. The resulting algorithm is reasonably efficient. Specifically, compared to similar approaches\n\nWeaknesses:\n* Although I have knowledge about conformal principles, I found the theoretical part hard to follow. I am missing major assumptions that are made to motivate the theory about the expected safety risk etc. The CDF bounds are straightforward, but why are they reasonable in this setting / Why can we assume anything about the expected safety risk (after all, the classifier could be attacked as well and provide a wrong signal)?\n* It is not clear to me why we could use the principles of conformal prediction, for example, for evasion attacks. What are the exact threat models this paper is considering?\n* I think spending more time to explain the individual components in (3) and (4) and why we can make certain assumptions would strengthen the paper\n\nI generally think this could be a valuable contribution and am very open to improve my score after the rebuttal phase\n\nQUESTIONS:\n- What is the proposed threat model? (Evasion attacks?)\n- Are there any assumptions made on the distribution of training and test time attacks?\n- Are there any assumptions on the expected risk and the function that provides it (e.g., could the classifier be attacked as well and provide wrong estimates?)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTo address the challenge of safe LLM generation with certified guarantees, this paper proposes a Claim-based Stream Decoding (CSD) algorithm coupled with a statistical risk guarantee framework using conformal analysis.   \nSpecifically, CSD utilizes an off-the-shelf Guardrail model to backtrack and resample claim streams if flagged with risks.  \nThis paper also provides theoretical proof of the proposed method.  \nExperiments against five baselines demonstrate the effectiveness of CSD in safety protection.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:  \n1. This paper addresses the important problem of safety-guaranteed LLM generation.\n2. The proposed method CSD, which utilizes an off-the-shelf Guardrail model to backtrack and resample claim streams, is effective at preserving safety according to the experiments.\n3. A specific design is introduced to improve the efficiency of CSD by caching previous verified claims.\n***\n**Weaknesses**:\n1. The paper is hard to follow.\n    * Some formulations are unnecessary\n    * The core idea and design of C-SafeGen are not clearly highlighted. I had to carefully read Algorithm 1 to grasp the full picture.\n2. It is unclear how the SafeGen protocol $P_a$  is calculated. Specifically, the meaning of \"configuration\" in this context is vague. Does it mean the A more detailed explanation of key terms is needed.\n3. The experients are not sufficient. For example, there are no experiments to verify potential performance degradation caused by CSD. Without such evaluation, it is unclear whether the safety protection actually works. Additionally, the efficiency comparison lacks baselines such as *Vanilla*.\n4. The experimental details are not adequately explained. \n    * For evaluation metrics, I can only tell which models are used in Section 5.1.\n    * Neither prompts nor open-source code are provided for the function $R_G$.\n***\n\nQUESTIONS:\n**Questions**:\n1. Could you provide a randomized baseline that produces random outputs, in order to estimate a lower-bound threshold for safety?\n\n***\n**Typos**:\n1. Line 90: Does \"SafeGen\" refer to \"C-SafeGen\" or to safe generation in general?\n2. Line 108: Should \"extitSafeGen\" be \"SafeGen\"?\n3. Line 138: Should \"below $\\alpha$\" be corrected to \"below $r$\"?\n4. Line 239: \"four baselines\" but five are listed.\n***\n\nLIMITATIONS:\nThe authors have discussed the limitations in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 1, "IMPACT": 1, "ORIGINALITY": 2, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces C-SafeGen, a certification framework that provides theoretical safety risk guarantees for large language model (LLM) generations. The framework includes a novel decoding algorithm, Claim-based Stream Decoding (CSD), which uses a guardrail model to monitor claims and claim \"segmentation\" method, that divides the input prompt into a set of non-overlapping claims. Empirical evaluations show that CSD reduces unsafe generations compared to existing methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The methodology introduced by this paper is simple, effective and theoretically sound. It involves using guardrail models that already exist, but instead of using these to filter unsafe outputs; this paper proposes to use these to create an alternate decoding mechanism to enable models to create safe outputs instead, instead of simply blocking the generation of an unsafe one.\n+ The experimental results are sound. The proposed approach lowers the safety risk of outputs while being more computationally efficient compared to approaches such as best-of-N.\n+ The writing is clear, concise, and to the point.\n\nWeaknesses:\n- The main weakness in this paper is a lack of evaluations of utility of the resulting safe outputs. In principle, it is trivial to certify safety: simply refuse to produce an output, or produce the same benign output (e.g.: \"I cannot answer this question\") for every prompt. The main question is whether we can certify safety while ensuring that the utility of the LLM does not deteriorate significantly. While Theorem 3 in the paper argues for this theoretically, empirical evaluations on this end would help better motivate this approach.\n\n- Another weakness of this paper is the usage of a rather simple claim-point detection method. For simplicity, the paper chooses to detect claims by identifying newlines, periods, etc. This has several obvious failure cases, for example, when a single long sentence may have multiple claims, or a claim being spread across sentences. Thus this choice appears to be ad-hoc, and the lack of discussion around this topic is a weakness.\n\nQUESTIONS:\n1. A clarification question: equation 7 defines a \"claim backtrack probability\" function, but the defined quantity is an indicator function, not a probability. Can the authors clarify?\n\n2. A question regarding the design choices of the method: Instead of segmenting the output into claims, some alternate approaches might be: \n    - cumulative safety checking every \"K\" generated output tokens by the guardrail model, with backtracking\n    - eliminate the claim-based system, and have the model re-generate the output if it is deemed unsafe by the guardrail model\n\nI wonder whether the authors have considered other such simpler design choices for evaluation? Can the authors comment on the trade-offs between different design choices?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

comments: SUMMARY:
This paper proposes a Claim-based Stream Decoding (CSD) algorithm coupled with a statistical risk guarantee framework using conformal analysis. CSD algorithm integrates a stream guardrail model to safeguard sequential claims generated by LLMs and incorporates a backtracking mechanism to revise claims flagged with high safety risks.  Empirical evaluations demonstrate the effectiveness and efficiency of the CSD algorithm compared to baselines.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The paper is clearly written and well-structured.

2. The methodology is well-developed and theoretically grounded. 


Weaknesses:
1.  Could the authors elaborate more about when Assumption 4.1 hold and when it fails in the real world?

2. The paper focuses on safety rates but does not quantitatively evaluate the effect of CSD on output utility. While it is stated that CSD “ensures fluency and coherence” and preserves “high-quality outputs”, no evaluations are provided. It would be important to measure generation utility after enforcing safety. 

3. The experiments focus on two “jailbreak” style benchmarks and adversarial suffix attacks. It is unclear how C-SafeGen would perform on other safety domains. The generalizability beyond the tested scenarios is somewhat uncertain. I suggest the authors involve a discussion or providing examples on other potential safety domains.

QUESTIONS:
Could the authors show on how sensitive results are to the choice of the threshold $\alpha_{b}$ and $\alpha_{g}$?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper deals with the safety guardrail of LLM during inference time. To fill the gap of lacking rigorous safety guarantees in previous approaches, the authors propose a claim-based streaming decoding algorithm with risk control by conformal analysis. This involves a stream guardrail model and a claim partition model with backtracking for improved efficiency. The authors also provide theoretical guarantees by deriving the upper bound of safety risk. Empirical experimental results showcase the efficiency and efficacy of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n1.\tThe risk control and safety guarantees of LLM generation during inference time are significant.\n2.\tThe theoretical guarantee is solid with a derived upper bound that allows better control.\n3.\tThe introduced claim separation and KV cache for guardrail models improve the efficiency of inference. \n4.\tThe paper is clearly written and easy to follow.\n\n## Weaknesses:\n1.\tReliance on the output probability of guardrail models may not be reliable enough. Existing work [1] found that guard models like Llama-Guard3 tend to be overconfident and miscalibrated under jailbreak attacks, limiting the safety estimation in the framework. \n2.\tLimited attacks are investigated except for GCG. HarmBench [2] includes multiple types of attacks that may help investigate the effectiveness of the proposed method under different attacks.\n\n## References\n[1] Liu, H., Huang, H., Gu, X., Wang, H., & Wang, Y. (2025). On calibration of LLM-based guard models for reliable content moderation. ICLR. \\\n[2] Mazeika, M., Phan, L., Yin, X., Zou, A., Wang, Z., Mu, N., ... & Hendrycks, D. (2024). Harmbench: A standardized evaluation framework for automated red teaming and robust refusal. ICML.\n\nQUESTIONS:\n1.\tDo you have any ablation for the safe resampling techniques? And which one of the introduced techniques (random masking/permutation/noise injection/structured permutation) is the most effective and efficient? \n2.\tHow is the ShieldGemma-9B implemented and used in the evaluation? ShieldGemma can only make a binary decision for one given class in one inference.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}