Title: Conformal Risk Training: End-to-End Optimization of Conformal Risk Control
Venue: NeurIPS 2025
Accepted: yes
Keywords: conformal risk control

decision-focused learning

conditional value-at-risk

CVAR
Abstract: While deep learning models often achieve high predictive accuracy, their predictions typically do not come with any provable guarantees on risk or reliability, which are critical for deployment in high-stakes applications. The framework of conformal risk control (CRC) provides a distribution-free, finite-sample method for controlling the expected value of any bounded monotone loss function and can be conveniently applied post-hoc to any pre-trained deep learning model. However, many real-world applications are sensitive to tail risks, as opposed to just expected loss. In this work, we develop a method for controlling the general class of Optimized Certainty-Equivalent (OCE) risks, a broad class of risk measures which includes as special cases the expected loss (generalizing the original CRC method) and common tail risks like the conditional value-at-risk (CVaR). Furthermore, standard post-hoc CRC can degrade average-case performance due to its lack of feedback to the model. To address this, we introduce "conformal risk training," an end-to-end approach that differentiates through conformal OCE risk control _during model training or fine-tuning_. Our method achieves provable risk guarantees while demonstrating significantly improved average-case performance over post-hoc approaches on applications to controlling classifiers' false negative rate and controlling financial risk in battery storage operation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses two key limitations of standard Conformal Risk Control (CRC) methods [Angelopoulos et al.]- 1) while CRC provides finite-sample guarantees on expected loss, many real-world high-stakes applications require control over tail risks (such as Conditional Value-at-Risk (CVaR)), 2) CRC is typically applied post hoc to a pretrained model, which may degrade performance due to the lack of feedback between risk calibration and model learning.\n\nTo address the first limitation, the authors introduce a new post-hoc conformal risk control framework, called Conformal OCE Risk Control (CORC). This framework extends the original CRC method to a broader family of risk measures known as Optimized Certainty Equivalent (OCE) risks. These include not only the standard expected loss (recovering CRC) but also more tail-sensitive metrics like CVaR. \n\nTo address the second limitation, the authors propose a new training strategy called conformal risk training, which incorporates feedback from CORC into the model’s training process. This approach jointly optimizes both the model parameters and the risk parameter by minimizing performance loss while ensuring the risk stays within the target upper bound. It is conceptually similar to the conformal training method introduced by David (cite), but extends it to the conformal risk control setting.\n\nThe proposed conformalized risk training method is evaluated on two high-stakes tasks—tumor image segmentation and battery energy storage optimization. The results demonstrate that the method provides valid risk control while also improving efficiency and utility.\n\n[Angelopoulos et al.] A. N. Angelopoulos, S. Bates, A. Fisch, L. Lei, and T. Schuster. Conformal Risk Control, Apr. 2023\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. Clarity and presentation: This paper is very well-written and well-structured. The authors clearly present their problem statement and articulate their motivations effectively. They also provide an easy-to-follow description of the methodology along with solid theoretical justification. I particularly appreciate the inclusion of intuitive examples that help clarify the technical details throughout the paper.\n2. Novelty and Significance: This paper addresses an important challenge in high-stakes applications—controlling tail risks—and fills a notable gap in the conformal prediction literature. It extends the existing conformal risk control framework in a principled manner to accommodate a broad class of Optimized Certainty Equivalent (OCE) risk functions. Additionally, the paper examines specific instances within the OCE family, such as Conditional Value-at-Risk (CVaR), and demonstrates that the standard non-decreasing assumption can be relaxed to monotonicity in this case. To the best of my knowledge, this is also the first work to incorporate conformal risk control into the conformal training paradigm.\n3. Soundness: The authors provide rigorous theoretical justification for the validity of their method and demonstrate improved efficiency through numerical experiments.\n\nWeaknesses: \n1. Please correct me if I have overlooked anything, but the empirical evaluation appears limited. The method is tested on only two datasets, each evaluated under a limited set of configurations (e.g., varying the target risk level $\\alpha$ or tail risk level $\\delta$. The work would be further strengthened by evaluating the proposed method on a more diverse set of classification tasks. Additionally, it may be helpful to explore the impact of varying calibration set sizes, as the proposed approach requires an additional data split to ensure validity. This introduces a trade-off between reduced data efficiency and the benefits of the proposed method. Investigating the performance in low-calibration regimes could provide valuable insights into the method's robustness and practical applicability. It would be particularly interesting to explore the potential benefits of the method when the calibration sample size is small.\n\nQUESTIONS:\n1. In your experiments involving conformal risk training, did you apply an additional layer of CRC or CORC after model fine-tuning? Or were the reported results directly from the trained model? \n2. Have you derived a lower bound on the test-time risk, similar to the approach taken in the original CRC paper?\n3. Do you have any results comparing CRC and CORC purely as post-hoc methods (i.e., using the same pretrained models without conformal risk training)?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper has two key contributions:\n(i) a generalization of Conformal Risk Control for general Optimized Certainty Equivalent (OCE) risks, which enables the control of risks beyond average risks: e.g., CVaR and Entropic risks. The paper pays particular attention to the CVaR case.\n(ii) A conformal training scheme, which allows for the propagation of gradients through the calibration process of the proposed conformal OCE risk control procedure (which subsumes the standard Conformal Risk Control / Vanilla Conformal procedures). This is formally derived, a distinct feature in comparison to previous works.\nBoth schemes seem to have favorable empirical results.\n\nSTRENGTHS AND WEAKNESSES:\n- The generalization of Conformal Risk Control for Optimized Certainty Equivalent risks is quite nice, and the ability to work with more general risks such as CVaR is highly desirable. The only cost seems to be (besides boundedness assumptions) the introduction of the $t$ hyperparameter, which the authors propose to be selected using a separate data split. How sensitive is the choice of $t$? Is there some guidance for how to select it a priori?\n\n- On Theorem 3 (regarding the conformal training scheme):\n    - I strongly advise the authors to write the closed-form gradients in the proposition of the theorem, at the very least in the supplementary material, rather than hiding them in the proofs.\n    - This is, as far as I am aware, the most formal treatment of conformal training to date, for which I applaud the authors. I just have some (hopefully) minor comments:\n        - On the proof of case (i):\n            - The writing of the proof is a bit tough to follow, in particular due to simultaneous notational overload and notational abuse. Perhaps it can be more clarified?\n            - Line 960: it is claimed that because $(\\varphi, c\\_j, g\\_j)$ are differentiable almost everywhere, so is $\\tilde{c}_j$. But as far as I can tell this involves a use of the chain rule, which need not apply for almost-everywhere differentiability. Did I miss something? Or maybe $\\varphi$ should be assumed to be differentiable everywhere? (Also the role of the $g_j$ seems delicate here, and I'm not entirely certain that differentiability is indeed preserved.)\n        - On the proof of case (ii):\n            - Lines 986-988 claim that we can replace $\\ell_i$ with $\\lambda$ (or $-\\lambda$), but I'm not sure this is correct; could the authors clarify why this is so?\n            - The idea of the proof seems to then be to reduce the problem to a standard bilevel optimization (via a sort of duality argument). Is my interpretation correct?\n    - The theorem establishes that the gradient exists almost everywhere, and somewhat states how to compute them in the proof. However, is it guaranteed that the gradients, where defined, are at least non-zero? For example, a combinatorial loss (e.g., AUC, Accuracy) is also differentiable almost everywhere, but has nil gradients where it is defined (and thus its gradients are useless). I *think* that generally the gradients in the proposed approach will generally be useful, but that there may be some situations under which the gradients become nil.\n\n- Finally, I am somewhat bothered by the title of the paper. As mentioned, the paper has two key contributions, these being (i) the new calibration procedure for OCE risks, and (ii) the conformal training approach.\nContribution (i) seems quite solid, and I think would have even stood as a paper on its own.\nContribution (ii) is also nice, but essentially spans a single theoretical result, and is reliant on the formulation of (i) (though it does seem like it could be restated for the special case of the more usual risks).\nOverall, the paper's content reads as the primary contribution being (i), with (ii) being a nice consequence.\nHowever, the title refers solely to (ii), making no mention at all to (i) or Optimized Certainty Equivalents; one would think that it is a mere adaptation of conformal training onto conformal risk control (of average risks). It's more than that!\nI realise this is a bit nitpicky on my part, but it feels like a genuine mismatch between the title and content.\n\nQUESTIONS:\nI'd appreciate it if the authors could clarify my questions in the 'Strengths&Weaknesses' section regarding the proof of Theorem 3.\n\nLIMITATIONS:\nLimitations were appropriately discussed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is exploring post-hoc risk control, for a broader class or risk measures. They particularly generalize the CRC method for the cases beyond expected loss, like Cvar. They provide precise theoretical arguments for the validity of their method. Furthermore, they extend conformal training ideas to their risk control setup, by differentiating through post-hoc risk control. This way, they enhance the efficiency of the whole decision making system, while controlling the desired risk. Their arguments are also sufficiently supported by experiments.\n\nSTRENGTHS AND WEAKNESSES:\n- The conjunction of ML and reliable decision making is a timely and very important research direction. This paper studies a somewhat neglected but very important topic of \"how can we use ML predictions while controlling tail sensitive downstream risks\", and the findings of the paper can contribute positively to the growing understanding of the community.\n\n- I would suggest to add some explicit examples for CRC loss function. For instance, it is not obvious how the notion of \"action\" or \"decision\" would show up in this formulation, as at the end of the day a decision maker wants to make a decision. \n- The assumption 4 (the same comment applies to assumption 3) needs more clarifications. The authors claim their contribution to be in controlling Cvar, however under assumption 4, controlling Cvar is trivial by just picking $\\lambda= \\lambda_{\\rm min}$.  Hence, perhaps the goal under assumption 4 is to pick the \"largest $\\lambda$ possible\" while controlling Cvar. This is not clearly stated, and it is not the direct subject of study in the paper (although it looks like the algorithm does try to satisfy such an objective).\n- In light of these, a more general concern emerges. Why would a 1-dimensional characterization of the decision space ($\\lambda$) makes sense at all when trying to control Cvar. Although CRC also assumes a 1-dimensional characterization, however, at least some of the use cases of CRC, like coverage of sets or FDR control, do have some fundamental 1-dimensional characteristics. For the case of FDR control neyman-pearson lemma shows the existence of a 1-dimensional rule, and for prediction sets [1, 2] explore similar results. \n\nThese being said, I believe the paper exceeds the acceptance threshold, and i am willing to increase my score, if the authors take concrete actions to resolve/clarify these concerns in the revised manuscript.\n\n[1] Least Ambiguous Set-Valued Classifiers with Bounded Error Levels\n[2] Decision Theoretic Foundations for Conformal Prediction: Optimal Uncertainty Quantification for Risk-Averse Agents\n\nQUESTIONS:\nPosed earlier.\n\nLIMITATIONS:\nI think the assumptions made should be discussed, particularly with the help of some explicit examples of loss functions.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces Conformal Risk Training, an extension of the traditional Conformal Risk Control framework to a broader class of risk measures, including CVaR, entropic risk etc. Their end-to-end approach integrates risk control directly into the training process by differentiating through the risk-controlling decision rule. The authors provide theoretical guarantees for risk control by applying monotone transformations of the loss function and develop techniques for computing gradients in common bi-level optimization settings.\n\nSTRENGTHS AND WEAKNESSES:\n1. The paper extends CRC beyond expected loss to risk measures like CVaR addressing an important open question. Introduces a novel bi-level optimization framework that differentiates through the conformal risk control procedure, enabling joint learning of model parameters and risk-sensitive thresholds.\n2. The authors provides finite sample, distribution free risk guarantees for a broad class of risks under monotonicity assumptions. They derive closed form gradient expressions under common conditions.\n3. The framework appears broadly applicable to any bounded, monotone loss, which makes it suitable for a wide range of decision making tasks beyond the two domains tested.\n4. Theoretical guarantees in the paper seem to depend on the assumption that loss functions are monotone and exchangeable. These assumptions may not hold in many practical settings, especially in tasks involving structured or high-dimensional outputs.\n4. The framework introduces an extra hyper parameter t, whose selection requires a hold out set and may impact practical performance if not tuned well.\n5.  Solving nested optimization problems (especially with CVaR constraints) can be computationally expensive or sensitive to initialization.\n6. While baseline comparisons are made, no direct comparison is made with other non conformal risk sensitive training approaches (e.g., DRO, robust optimization).\n7. The manuscript doesn't make connections with the rapidly growing literature on decision focused learning and contextual stochastic optimization. in particular, it omits discussion of the recent EJOR survey by Sadana et al. on contextual optimization and the JAIR survey by Mandi et al. on Decision Focused Learning, both of which place bilevel risk-aware training in a broader taxonomy.\n\nQUESTIONS:\n1. I think a discussion on the tightness of the CORC bounds might be helpful. Especially, under what conditions might these guarantees become overly conservative?\n2. what happens when the loss functions $L_i(\\lambda)$ is not globally monotonic? For example, could one consider piecewise monotonic, quasimonotonic, or approximate monotonic losses and still retain valid risk control?\n3. While differentiability of the inner solution $\\lambda(\\theta)$ is addressed under specific conditions, can the authors formally analyze its stability with respect to perturbations in $\\theta$? \n4. Can the CORC framework be extended to non exchangeable data, such as time series settings?\n5. I believe this work has strong connections to Robust Optimization, especially with the recently evolving end to end robust optimization methods. I think a discussion along this direction can help connect it to a broader scope of works. \n6. Can the authors provide insight into when end-to-end conformal risk training yields better efficiency or generalization compared to post-hoc CRC? Are there conditions under which the added complexity of bilevel training is provably beneficial?\n\nLIMITATIONS:\nas discussed above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents two primary contributions. First, it extends the Conformal Risk Control (CRC) framework beyond expected loss to a broader class of Optimized Certainty-Equivalent (OCE) risks, which includes tail-risk measures like Conditional Value-at-Risk (CVaR). This represents a  leap forward, as the OCE class includes common tail-risk measures like the CVaR. This work provides the community with a practical methodology for creating models with provable guarantees on worst-case scenarios, moving beyond simple averages. The core technical insight is demonstrating that any OCE risk can be bounded via a monotone transformation of the loss, thereby preserving the essential monotonicity property that underpins the conformal guarantee. Proposition 1 prepares the extension for OCE risks, and improves more generally previous CRC by considering a lambda dependent bound. The main result is then theorem 2 that considers CVaR among OCE, and it proposes algorithm 1 for building the set of feasible lambda (controlling the risk). It discusses briefly the feasibility of the risk control, as well as the role of t. It introduces an assumption 5 where L_i’s and B can be either increasing or decreasing. \n\nSecond, it introduces Conformal Risk Training (CRT), an end-to-end training methodology that embeds the risk control procedure differentiably into the model's fitting loop, and generalizes previous attempts in conformal training who inspired partly the paper. The stated goal is to achieve provable risk guarantees while mitigating the degradation in performance often seen with post-hoc calibration methods (say the choice of lambda to get the guarantee on the False Positive Rate implies a drop on the False Negative Rate). Consequently the an end-to-end optimization method propose an interesting way to deal with the inherent tension between risk control and model performance. The model is optimized not only for accuracy but also for its \"amenability\" to risk control, by using Lagrangian relaxation, where the constraint on the risk is added in the global criterion. It learns to produce predictions that can satisfy the risk constraint with minimal performance degradation. Theorem 3 gives conditions for that specific approach for selecting the proper risk; the formal version is given in appendix where parametric families of risk and bound are given (piece constant in lambda), in order to be able to compute the gradient.\n\nA short Empirical Validation & Key Results are given:  for  validation the CRT framework's efficacy through two real-world applications that highlight the  dual benefits of safety and performance.\n* Controlling False Negative Rate in Tumor Segmentation: CRT is used to control the False Negative Rate (FNR) at a user-specified level α. Standard post-hoc CRC also achieves this safety guarantee, by incurring a high False Positive Rate (FPR) (such as 80%). The CRT model, in contrast, successfully resolves this trade-off, by reducing the FPR by reducing significantly and  maintaining the FNR guarantee at different risk levels.\n* Controlling CVaR in Battery Storage Operation: In a financial task where an agent optimizes battery charging cycles based on price forecasts, CRT is used to control the CVaR of monetary losses. CRT can successfully bounds this tail risk at the target level and it also yields a higher average profit compared to the post-hoc CRC baseline. This result showcases its ability to manage downside risk while actively improving upside performance.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n\nThe paper is well written and opens the way to the very promising direction of building models that offer a finite horizon controlled of the risk by design. Indeed, while standard Conformal Prediction comes into play only for evaluating or correcting for the uncertainty of a model, hence it could be an important enabler of reliable and trustworthy AI. \nThe two examples presented and discussed in the paper motivate well the development of Conformal Training by their important and the practical impact it can have. \nThe technics and proofs seem ok, and the approach for computing the gradient seems efficient in some cases, with an interesting and promising direction based on bilevel optimization, although numerous theoretical and practical problems could happen, and be discussed.\n\nWeaknesses and critical analysis \n1. Real Impact of OCE and Novelty of CVaR Control somehow Overstated.\nThe entire framework, both for OCE risk control and CRT, is built upon the foundational assumption of a monotone loss function L(lambda) in the hyperparameter lambda. While a relaxation is noted for OCE and CVaR under specific conditions, this does not resolve the core limitation for the general framework. While CVaR or other OCE may allow to consider new applications, I wonder if it is really a big extension for dealing nonmonotonous loss function. Other approach such as Learn Then Test, Angelopoulos et al, AoAS 2025  or RCPS Angelopoulos et al could be used. In addition, the control of CVaR and other distortion measures is also addressed in Chen et al, cited in the paper. While the paper does extend traditional CRC to CVaR and potential othe risks,  this contribution is rather incremental and not the only way of doing it. A better discussion on the various possibilities of controlling of CVaR would be useful to understand the impact of the contribution. As well as the ability to address the distortion risk measures. Would it be possible to claim theorem 2 for general OCE risk instead of only for CVaR, is there any reason that precludes a general formulation and adding a corollary for CVaR.  \n2. The OCE Formulation Introduces an Unanalyzed Complexity with the new hyperparameter t from the variational definition of the risk measure. t to be selected without access to the calibration data. The paper's proposed solution—tuning t on a separate held-out set—is a practical heuristic but lacks theoretical grounding, as there is no analysis of the method's sensitivity to this choice. How does a sub-optimal t affect the tightness of the risk bound and, consequently, the model's ultimate performance? This introduces a new, unanalyzed layer of complexity to the calibration process.\n\n3. As a side and related effect is the feasibility of the constraint. It seems that there is an interplay between t, the constraint alpha, and the bound used B(lambda), as we can see in theorem 2, or in theorem 3. It is a theoretically and practically important point that deserves an explicit and detailed treatment discussion in the paper. Could you discuss in a coherent way the problem of existence, and possibly how it is related to the existence of a gradient.\n\n4. The Practicality of Conformal Risk Training is Limited by Gradient Computation.\nThe CRT method's central mechanism is to make use of the implicit value theorem by considering a bilevel optimisation, and propagate through the risk-control parameter lambda(theta). The paper shows this is feasible for highly-constrained settings with Piecewise Constant Losses or strict convexity / monotony : The gradient exists because the optimal lambda(theta) snaps to a specific model output value.\nIn addition every batch must be split in a calibration and prediction, but the size and the role of the split is not detailed and it becomes a bit blurry. The complexity and subtleties of its computation are hidden, as in appendix problem of stabilities and variance reduction p. 27 line 1043 do require averaging. Could you dedicate a specific part to the existence of the gradient and the numerical computation of the gradient. The ability to compute or approximate properly the gradient narrows the the applicability of CRT.\n\nQUESTIONS:\nAdditional Questions to address : \n1. In addition to the questions asked in the S&W, I would to have a clearer explanation of the computation of the gradient in the example 2, and the importance of the assumption of having all values f_theta(X_i)_j  to be unique. What happens if not the case ? And If they are very close? How does it influence the quality or existence of the derivative?\n2. It is said in assumption that we can add increasing or decreasing losses: is it a generalization? Is it really useful? \n3. What about using the standard cross entropy loss in image segmentation problem,  instead of specificity minimizing specificity.\n4. Can you provide sensitivity analysis on t, and give a complete methodology for doing the estimation, taking into account all the datasets needed train, final calibration (intermediate split)… to see how much a standard training (to see how the standard split train test validation is changed).\n5. Compare with other approaches that give a guarantee on the loss (like probabilistic for instance).\n\nLIMITATIONS:\nPartly. I really appreciate the direction the paper proposed, and the technics used that have nice theoretical properties and guarantees. \nNevertheless, I think that the paper does not face with enough depth and rigor the limitations or real impact of the paper : comparison with LTT, RCPS or other risk control approach in the paper; the choice of hyperparameter t, or the importance of the quality of B, the feasiblity problem, the feasibility of the computation of the gradient, and the stability / computational aspects of the gradient. \nI am ready to increase my score if the authors provide a more rigorous, organised and deeper treatment of these points.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-05"}