Title: Causal LLM Routing: End-to-End Regret Minimization from Observational Data
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLMs

Routing

Causal ML
Abstract: LLM routing aims to select the most appropriate model for each query, balancing competing performance metrics such as accuracy and cost across a pool of language models. Prior approaches typically adopt a decoupled strategy, where the metrics are first predicted and the model is then selected based on these estimates. This setup is prone to compounding errors and often relies on full-feedback data, where each query is evaluated by all candidate models, which is costly to obtain and maintain in practice. In contrast, we learn from observational data, which records only the outcome of the model actually deployed. We propose a causal end-to-end framework that learns routing policies by minimizing decision-making regret from observational data. To enable efficient optimization, we introduce two theoretically grounded surrogate objectives: a classification-based upper bound, and a softmax-weighted regret approximation shown to recover the optimal policy at convergence. We further extend our framework to handle heterogeneous cost preferences via an interval-conditioned architecture. Experiments on public benchmarks show that our method outperforms existing baselines, achieving state-of-the-art performance across different embedding models.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a causal end-to-end LLM routing framework based on observational data, aiming to optimize model selection by minimizing decision regret while balancing accuracy and cost. Due to the limitations of observational data, it employs a **Doubly Robust Estimator** from causal inference to estimate counterfactual utility $\\hat{Y}_x(t) $. This estimator integrates a direct outcome regression model with a propensity score model. Since directly optimizing the regret $\\operatorname{Regret}(f) $is non-differentiable due to the discrete nature of routing decisions $f(x) $, the paper introduces two differentiable surrogate loss functions to address this issue. In practice, users’cost sensitivity $\\lambda $may vary per query. To avoid training separate models for each $\\lambda $, the paper proposes an interval-conditioned joint router.\n\nSTRENGTHS AND WEAKNESSES:\n### Advantages\n\n1. Unlike traditional methods requiring \"full feedback data\" (testing every query with all models), which is costly and impractical, this paper uses only \"observational data\" (one model per query). This aligns with real-world scenarios, saving computational and economic costs.\n\n2. Traditional routing predicts each model's accuracy and cost separately, risking errors that accumulate in the final selection. The paper's end-to-end approach directly optimizes decision regret, avoiding error accumulation and improving model selection accuracy.\n\n3. Observational data often has selection bias (e.g., favoring cheaper models). The paper uses causal inference (via a doubly robust estimator) to estimate untested models' performance, correcting bias and enhancing routing reliability.\n\n4. Users have varying cost sensitivities (some prioritize savings, others quality). The paper's interval-conditioned joint router adapts strategies to different budgets without training separate models for each preference, making it efficient and practical.\n\n### Disadvantages\n\n1. The method assumes sufficient data support (each model has some query assignments), but real-world data may heavily favor certain models (e.g., only the cheapest). This can lead to inaccurate counterfactual utility estimates. The paper lacks detailed analysis of robustness under imbalanced data distributions.\n\n2. Although the method is claimed to be efficient (e.g., using two endpoint models for interpolation), the paper does not provide detailed computational cost analysis (e.g., training time or inference latency). In high-throughput scenarios, these costs could be critical.\n\n3. Experiments are conducted on two public datasets (RouterBench and SPROUT) with diverse tasks, but these may not fully represent complex real-world LLM applications (e.g., real-time dialogue or multimodal queries). The simulated observational data (sampled by accuracy) may not reflect actual historical biases.\n\n4. The method relies on causal assumptions (e.g., ignorability, support, and SUTVA). These may not hold in practice, such as when query features \\( x \\) fail to explain model selection (due to unobserved confounders) or when queries are interdependent (e.g., in multi-turn dialogues, where prior model choices affect subsequent outcomes, violating SUTVA). The paper does not thoroughly address the impact of assumption violations.\n\nQUESTIONS:\n1. The paper uses XGBoost to estimate the propensity score $\\hat{p}(t \\mid x) $and applies clipping to mitigate the impact of extreme weights. However, the stability of the propensity score model is not thoroughly analyzed. How robust is this model to variations in data distribution or feature quality, and how do errors in propensity estimation affect routing performance?\n\n2. The method assumes query independence (SUTVA assumption), but in multi-turn dialogues or agentic applications, queries may be interdependent (e.g., one turn's response influences the next). How can the method adapt to such scenarios where query dependencies violate the SUTVA assumption?\n\n3. The paper claims high computational efficiency (e.g., interval interpolation using two endpoint models) and direct optimization of decision regret in an end-to-end manner. However, it does not provide specific data on inference times. In real-time applications, the latency of the router (embedding generation + routing decision) could impact user experience. How does the method ensure low latency to meet real-time requirements?\n\nLIMITATIONS:\nThe paper has several key limitations, including dependence on embedding quality, constraints of observational data quality, and consequences of causal assumption violations. Additional potential issues, such as unquantified computational complexity, further restrict its applicability.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-17"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the critical problem of LLM routing, which aims to select the most appropriate model from a pool of candidates for a given query to balance performance and cost. In detail, the paper introduces a novel causal end-to-end framework for learning routing policies directly from more practical observational data. Furthermore, they extend their framework to handle heterogeneous user cost preferences via an elegant interval-conditioned architecture. Experiments on several benchmarks demonstrate that the proposed methods significantly outperform existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n1. The paper addresses a crucial and practical problem. Developing an efficient, cost-aware LLM routing strategy is important for the real-world deployment of large language models. This paper tackles a high-impact problem that is of great interest to the communities.\n\n2. The authors provide a novel framework, Causal LLM Routing, which uses causal inference to handle treatment bias and the end-to-end formulation to avoid error compounding. This moves the field beyond traditional supervised methods or decoupled pipelines.\n\n3. A variety of experiments have been conducted to show the effectiveness of the proposed approach across different baselines.\n\n\n**Weaknesses:**\n\n1. The causal LLM routing framework's performance relies on the counterfactual estimation of the doubly robust estimator, which means it depends on the quality of the outcome model ($\\hat{r}_t(x)$) and the propensity score model ($\\hat{p}(t|x)$). While doubly robust estimator is robust, if both models are poorly specified, the final counterfactual estimates can be inaccurate. It’s necessary to discuss the sensitivity to the quality of these nuisance models.\n\n2. A potential weakness of the proposed framework lies in its scalability as the number of candidate models, $|\\mathcal{T}|$, increases significantly. The training process for the final routing policy (e.g., `RM-Classification` or `RM-Softmax`) requires computing the full vector of counterfactual utilities $\\hat{Y}_{x}(t)$, $t \\in \\mathcal{T}$ for each training sample $x$. This is necessary to determine the optimal action $t^*$ for the regret target and to compute the surrogate loss itself (as seen in Eq. 6 and the objective for Eq. 3). Consequently, the computational complexity of the policy training stage appears to scale at least linearly with $| \\mathcal{T}|$, potentially making it prohibitively expensive for scenarios with hundreds or thousands of models. Could the author provide a more formal analysis or empirical evidence to characterize the computational complexity of the approach? This would help clarify the practical limits of the current framework.\n\nQUESTIONS:\n1. As mentioned in the weaknesses, could you provide an ablation study or sensitivity analysis on the performance of your `RM-*` methods under different levels of misspecification for the outcome model ($\\hat{r}_t$) and the propensity score model ($\\hat{p}$)? For instance, one could use simpler models (e.g., Logistic Regression) instead of XGBoost to estimate propensity score model, or intentionally add noise to the training data for these models.\n\n2. Equation *11* suggests that the joint network for an interval is fine-tuned by minimizing the average regret at only the two endpoints ($\\lambda_j, \\lambda_{j+1}$). Is there a risk of \"overfitting\" to these two points and performing poorly for $\\lambda$ values in the middle of the interval? While Proposition *5* suggests the architecture is expressive enough, the training objective seems sparse. Could you provide some intuition or empirical evidence (e.g., plotting performance for many $\\lambda$ values within a single interval) to show that this training scheme indeed leads to good generalization across the entire interval?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a causal framework to route queries to LLMs using observational (bandit) data. They authors formulate the problem as regret minimization via counterfactual estimation, addressing selection bias without full-feedback data. They also propose methods that can also support diverse cost preferences through an interval-conditioned architecture. The authors provide theoretical guarantees for the proposed algorithms, and also conduct experiments on public benchmarks to show the superior performance of their proposed methods over baselines.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problem is well-motivated and important. \n2. The paper is well-written and easy to follow.\n3. The authors provide theoretical guarantees of the proposed algorithms.\n4. The authors conduct extensive experiments to show the outperformance of their proposed algorithms over existing methods.\n\nWeaknesses:\n1. The main concern is that the related works are not discussed. I think there are many works on LLM routing, but they are not discussed. The authors mention that previous works rely on full-feedback data, but there are some existing works on multi-armed bandits for LLM routing, which depend on bandit feedback instead of full-information feedback. I think discussions and comparisons with related works are needed to place the position of this work appropriately. For example, some works of the bandit approaches:\n[1] MixLLM: Dynamic Routing in Mixed Large Language Models\n[2] LLM Bandit: Cost-Efficient LLM Generation via Preference-Conditioned Dynamic Routing\n[3] A Multi-Armed Bandit Approach to Online Selection and Evaluation of Generative Models\n2. In terms of the terminology, the \"observational data\" is just bandit feedback in the literature?\n3. There are some typos. For example, in the definition of regret (Eq.(1)), I think $t^{\\star}$ should be $t^{\\star}_X$, which depends on the prompt $X$ ($t^{\\star}_X=\\arg\\max_t Y_X(t)$)?\n\nQUESTIONS:\nCould you please discuss the relationship between your work and bandit-based approaches? Your work is studying the offline setting, but is it related to offline bandits?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces a novel approach by combining causal inference with end-to-end regret minimization for LLM routing. It addresses a gap in the literature where existing methods (RouteLLM, CARROT) rely on full-feedback data that is impractical to obtain in real deployments\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe theoretical framework is well-constructed and mathematically principled. The two surrogate objectives (classification-based upper bound and softmax-weighted regret) are motivated with formal propositions. Proposition 2's proof that the softmax surrogate recovers optimal treatment assignment provides crucial theoretical grounding. \n\nThe paper also addresses the reliance on expensive full-feedback datasets for current LLM routing. The treatment of observational data with historical routing policies is practically important and represents a significant advancement over existing approaches that assume unrealistic data availability.\n\nEvaluation on RouterBench and SPROUT benchmarks with multiple embedding models (BERT, LLaMA) is comprehensive and good. It includes reasonable baselines and the heterogeneous preference evaluation shows practical applicability.\n\nWeaknesses\nThere is no comparison with recent state-of-the-art methods like the latest RouteLLM variants or more sophisticated ensemble approaches. There is also limited analysis of individual component contributions (causal inference vs end-to-end learning vs regret minimization)\n\nThere are some assumptions made that may not be true for all models. The L-Lipschitz assumption may not hold for discrete model capabilities across diverse tasks. The restriction to linear cost functions limits applicability to more complex preference structures\n\nQUESTIONS:\nHow sensitive is the method to violations of key assumptions (Lipschitz continuity, unconfoundedness, affine utility)?\nHow does the method compare to more sophisticated baselines and recent routing advances?\nHow realistic is the accuracy-proportional sampling for simulating observational data?\n\nLIMITATIONS:\nAuthors mention restrictions to affine utility functions and suggesting extensions to hard constraints and online settings. However, there are a few more missing limitations. There is no discussion of robustness to causal inference assumption violations. There is limited analysis of scenarios where the method would perform poorly. This approach \n\nThe approach relies on strong assumptions including Lipschitz continuity of utility functions (which may not hold for discrete model capabilities), standard causal inference assumptions (unconfoundedness, overlap, SUTVA) that are difficult to verify in practice, and affine utility structures that may not capture complex user preferences. The convergence guarantees for surrogate objectives lack finite-sample bounds. The experimental evaluation relies on simulated observational data that may not reflect the complexity of real-world routing biases, limiting the generalizability of the empirical results.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-04"}