Title: C3PO: Optimized Large Language Model Cascades with Probabilistic Cost Constraints for Reasoning
Venue: NeurIPS 2025
Accepted: yes
Keywords: large language model

efficiency

inference

cascade
Abstract: Large language models (LLMs) have achieved impressive results on complex reasoning tasks, but their high inference cost remains a major barrier to real-world deployment. A promising solution is to use cascaded inference, where small, cheap models handle easy queries, and only the hardest examples are escalated to more powerful models. However, existing cascade methods typically rely on supervised training with labeled data, offer no theoretical generalization guarantees, and provide limited control over test-time computational cost.
We introduce **C3PO** (*Cost Controlled Cascaded Prediction Optimization*), a self-supervised framework for optimizing LLM cascades under probabilistic cost constraints. By focusing on minimizing regret with respect to the most powerful model (MPM), C3PO avoids the need for labeled data by constructing a cascade using only unlabeled model outputs. It leverages conformal prediction to bound the probability that inference cost exceeds a user-specified budget.
We provide theoretical guarantees on both cost control and generalization error, and show that our optimization procedure is effective even with small calibration sets. Empirically, C3PO achieves state-of-the-art performance across a diverse set of reasoning benchmarks including GSM8K, MATH-500, BigBench-Hard and AIME, outperforming strong LLM cascading baselines in both accuracy and cost-efficiency. Our results demonstrate that principled, label-free cascade optimization can enable scalable LLM deployment.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces C3PO (Cost Controlled Cascaded Prediction Optimization), a framework designed to optimize large language model (LLM) cascades with probabilistic cost constraints. The core idea behind C3PO is to create an efficient cascading inference system where simpler, cheaper models handle easy queries, and more complex models are only invoked for difficult cases. The framework provides a systematic approach to controlling costs while achieving high performance, offering a promising solution for real-world LLM deployment. The approach is particularly beneficial in scenarios where high inference costs are a barrier, ensuring both scalability and reliability.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper provides a rigorous theoretical guarantee for the proposed C3PO framework, particularly in terms of cost control and generalization error, which is a significant contribution to the research in this field.\n2. In the context where the computational cost of LLMs is becoming an increasing bottleneck for applications, the method proposed in this paper precisely addresses the core issue of how to significantly reduce inference costs while ensuring high accuracy, with great practical significance and application value.\n\nWeaknesses:\n1. The performance limit of this method is constrained by the teacher model. If the teacher model itself contains errors, the system not only fails to correct them but may also learn and reinforce these errors.\n2. The effectiveness of the system's decisions highly depends on the consistency between the distribution of the training data and the data distribution in the actual application scenario. When the real-world user problem patterns change (i.e., \"distribution shift\"), the pre-trained model selection strategy may fail or become suboptimal, leading to a decline in system performance.\n3. This method does not compare with the setting of \"fine-tuning a cheap small model for a specific task.\"\n4. Lacks results on more challenging reasoning benchmarks, such as mathematical reasoning and code generation tasks. GSM8K is somewhat outdated; more recent benchmarks like AIME 2024/2025, Livecode bench should be considered.\n\nQUESTIONS:\n1. It is recommended to use experiments to demonstrate that C3PO has stronger robustness to distribution shift compared to baseline methods.\n2. In the tasks discussed in the paper, conduct a re-experiment by replacing the API call method with a fine-tuned small model.\n3. It is suggested to discuss what the handling approach would be if the teacher model itself provides uncertain answers.\n\nLIMITATIONS:\nyes, in supplementary.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

comments: SUMMARY:
In this paper, the authors introduced a new cascade framework that dynamically decides whether to accept an intermediate model's prediction based on the learned thresholds and conformal cost calibration. Also, theoretical guarantees on both cost control and generalization error are provided. Experiments conducted on several public LLM families have demonstrated the advances of the proposed method.

STRENGTHS AND WEAKNESSES:
Strengths

1. Compared to other linked frameworks, the proposed method has higher accuracy and lower overhead.

2. The paper provides a detailed introduction to the existing methods and other preliminary knowledge.

3. Theoretical proof has demonstrated the effectiveness of the optimization capability of the proposed algorithm under certain constraints.

Weaknesses

Lack of more details on the design of ablation experimental exploration methods and other results analysis:
1. Selecting calibration data, i.g, how to choose the number of samples in this subset.
2. The effectiveness of methods in cross-family models.
3. In different test sets, the improvement effect of the proposed algorithm varies significantly. Does this indicate the applicability of the algorithm? In addition, in Figure 9, there was a phenomenon of performance decreasing with increasing cost in some tests. How to explain this?

QUESTIONS:
Please see the weaknesses.

By the way, it seems that the proposed method is not limited to language models. Can its effectiveness be explored in more scenarios, such as multimodality?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces C3PO, a novel framework for optimizing LLM cascades that tackles key limitations of existing methods like reliance on labeled data and lack of cost control. Its primary contributions are a self-supervised optimization process that learns an exit policy without ground-truth labels by minimizing regret against a Most Powerful Model (MPM); a principled probabilistic cost control mechanism using conformal prediction to guarantee budget adherence; and strong theoretical generalization bounds based on PAC-Bayes analysis. Empirically, C3PO achieves state-of-the-art cost-accuracy performance on a wide range of reasoning benchmarks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The self-supervised nature of C3PO is its most significant practical advantage. By optimizing for agreement with the MPM, it sidesteps the need for costly data annotation, making the framework highly adaptable and scalable to new tasks and domains where labeled data is scarce.\n\n2. The application of conformal prediction to enforce a probabilistic cost constraint is a major innovation. This moves the field beyond heuristic-based budget tuning and provides a rigorous, theoretically sound method for managing inference costs, which is critical for real-world deployment.\n\n3. The paper is exceptionally well-grounded in theory. The combination of a conformal guarantee on cost and a PAC-Bayesian bound on generalization error provides a level of rigor and reliability that is often missing in a field dominated by empirical heuristics.\n\n4. Despite the sophisticated theoretical underpinnings, the core algorithm is an efficient and straightforward grid search over thresholds. As the authors correctly point out, this is highly practical for the typical size of LLM cascades used today.\n\nWeaknesses:\n1. The framework's core assumption is that the MPM serves as a reliable proxy for the ground truth. The objective is to match the MPM's output, not necessarily the correct answer. In scenarios where the MPM is systematically flawed or biased, C3PO would learn to replicate these errors efficiently, which could be a significant drawback.\n\n3. The performance of the learned thresholds depends on the availability and quality of confidence scores from the LLMs. The paper assumes that these scores are stochastically increasing with the probability of correctness, but the reliability and calibration of such scores can vary significantly across different models and may not always be a robust signal.\n\n3. As the authors briefly note, conformal prediction can sometimes yield conservative bounds. This could lead the cascade to under-spend its allocated budget, potentially leaving some accuracy gains on the table in an effort to strictly satisfy the probabilistic cost constraint.\n\nQUESTIONS:\n1. How does C3PO's performance degrade if the MPM itself has relatively low accuracy on a given task? Could you characterize the break-even point where optimizing for agreement with a flawed MPM becomes less effective than a simpler baseline like self-consistency on a cheaper model?\n\n2. The method relies on model-produced confidence scores. Did you experiment with different ways of deriving these scores (e.g., average token log-probs, self-consistency voting)? How sensitive is C3PO to the choice of this confidence metric?\n\n3. The analysis of performance across different difficulty levels in MATH-500 is very insightful. Did you observe if C3PO learns to allocate budget as expected—i.e., does it learn lower confidence thresholds for easier problems and higher thresholds for more difficult ones, effectively spending more on harder questions?\n\n4. The cost model is assumed to be fixed and query-independent. How would the theoretical guarantees, particularly the conformal cost bound, be affected by stochastic costs (e.g., due to variable output lengths which are common in CoT reasoning)?\n\n5. One interesting point in the paper is that given the difficulty of problems, it can skip certain level of grid. Is there a statistic number of such cases and how much would it benefits from this.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}