Title: MESS+: Dynamically Learned Inference-Time LLM Routing in Model Zoos with Service Level Guarantees
Venue: NeurIPS 2025
Accepted: yes
Keywords: inference optimization

model routing

stochastic optimization algorithms

convergence analysis
Abstract: Open-weight large language model (LLM) zoos provide access to numerous high-quality models, but selecting the appropriate model for specific tasks remains challenging and requires technical expertise. Most users simply want factually correct, safe, and satisfying responses without concerning themselves with model technicalities, while inference service providers prioritize minimizing operating costs. These competing interests are typically mediated through service level agreements (SLAs) that guarantee minimum service quality. 
We introduce MESS+, a stochastic optimization algorithm for cost-optimal LLM request routing while providing rigorous SLA compliance guarantees. MESS+ learns request satisfaction probabilities of LLMs in real-time as users interact with the system, based on which model selection decisions are made by solving a per-request optimization problem. Our algorithm includes a novel combination of virtual queues and request satisfaction prediction, along with a theoretical analysis of cost optimality and constraint satisfaction.
Across a wide range of state-of-the-art LLM benchmarks, MESS+ achieves an average of $2\times$ cost savings compared to existing LLM routing techniques.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MESS+, an algorithm for optimally routing requests across multiple language models (LLM zoo) while maintaining service level guarantees and minimizing operating costs. Here are the key points and contributions:\n\nMain Problem Addressed:\n- How to automatically select the most cost-efficient LLM for each request while ensuring a minimum quality level specified by Service Level Agreements (SLAs)\n- Balancing between user needs (high-quality responses) and provider needs (low operating costs)\n\nKey Components of MESS+:\n1. Online Learning System:\n- Dynamically learns which models can satisfy which types of requests\n- Uses ModernBERT-based predictor to estimate satisfaction probabilities\n- Adapts through exploration/exploitation strategy\n\n2. Cost Optimization Framework:\n- Uses virtual queues to track SLA compliance\n- Solves per-request optimization problem considering both costs and predicted satisfaction\n- Provides theoretical guarantees for both cost optimality and SLA compliance\n\n3. SLA Guarantee Mechanism:\n- Maintains minimum satisfaction rate (α) over time\n- Uses virtual queue system to track and correct violations\n- Allows configuration of trade-off between quick convergence and cost efficiency\n\nResults:\n- Achieves 2x cost savings compared to existing routing techniques\n- Successfully maintains SLA requirements across various benchmarks\n- Demonstrates effectiveness across different types of language tasks\n\nThe paper's main contribution is a theoretically grounded framework that, for the first time, provides both cost optimization and rigorous service level guarantees for LLM request routing, while learning request satisfaction patterns online without requiring pre-curated routing datasets.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths:\n\n1. Theoretical Foundation\n- Provides rigorous theoretical guarantees for both cost optimality and SLA compliance\n- Clear mathematical framework based on Lyapunov optimization\n- Theorems are well-stated with complete proofs referenced in appendix\n\n2. Novel Approach\n- Introduces online learning component that eliminates need for curated datasets\n- Novel integration of virtual queues with satisfaction prediction\n\n3. Empirical Validation\n- Comprehensive evaluation across 8 different benchmarks\n- Clear ablation studies showing impact of different parameters\n\n## Weaknesses:\n\n1. Implementation Details\n- Limited details about the ModernBERT predictor architecture\n- Missing specifics about classifier design on top of ModernBERT\n- Could benefit from more details about hyperparameter selection\n\n2. Practical Limitations\n- Assumes immediate availability of user satisfaction feedback\n- Requires querying all models during exploration phase, which could be expensive\n- May need significant number of requests before convergence\n- The authors acknowledge these limitations but don't propose solutions\n\n3. Experimental Scope\n- All experiments use same family of models (Llama) and only 3 models in the model zoo\n- Could benefit from testing across different model families and more models\n- Limited discussion of how performance varies with different types of requests\n\n4. Theoretical Assumptions\n- Some assumptions (like i.i.d. requests) may not hold in practice\n- Convergence guarantees require large number of requests\n- Trade-off between exploration and exploitation could be explored more thoroughly\n\n5. Comparisons\n- Could include comparison with simpler baselines (e.g., random routing with cost threshold, nearest neighbor routing, etc)\n- No discussion of failure cases or edge conditions\n\nQUESTIONS:\n1. Predictor Architecture Details [**Good to have**]\nQuestion: Could you provide more specific details about the ModernBERT-based predictor implementation?\n- How exactly is the classifier implemented on top of ModernBERT?\n- Which layers are frozen vs. trained?\n- What is the rationale for these architectural choices?\nImpact: This information is crucial for reproducibility and understanding the method's practical requirements. A detailed response could increase the quality score by demonstrating the robustness of the technical implementation.\n\n2. Cross-Model Family Performance [**Highly Important**]\nQuestion: Have you tested MESS+ across different model families?\n- How does performance vary when mixing models from different providers?\n- Are there specific characteristics that make certain model combinations more effective?\n- What modifications might be needed for heterogeneous model zoos?\nImpact: Demonstrating broader applicability across model families would strengthen the paper's generalizability claims. Positive results could increase both significance and originality scores.\n\n3. Cost Structure Analysis [**Highly important**]\nQuestion: Could you provide more detailed analysis of:\n- How different cost structures affect routing decisions?\n- The impact of varying cost ratios between models?\n- The relationship between exploration costs and long-term benefits?\nImpact: This would help readers understand the economic implications of deployment. Clear analysis could increase the paper's practical significance score.\n\nAdditional Suggestions:\n- Consider including simple baseline comparisons (e.g., random routing, knn router, mlp router; c.f. RouterBench). Some of them can be adapted to online setting, but even comparing to the offline baseline is valuable.\n- Add analysis of failure cases or edge conditions\n- Provide guidelines for selecting hyperparameters (V, c) in practice\n\nLIMITATIONS:\nThe authors have partially addressed limitations, but there are several areas that could benefit from more thorough discussion:\n\n1. Technical Limitations\nThe paper discusses some key technical limitations in Section 5:\n- Dependency on readily available user satisfaction labels\n- Challenges with multiple model exploration\nHowever, it could benefit from addressing:\n- Scalability limitations with very large model zoos\n- Performance degradation scenarios\n- Resource requirements for the prediction system\n\n2. Societal Impact\nMissing discussions on:\n- Potential misuse of cost optimization at the expense of quality\n- Privacy implications of collecting user satisfaction data\n- Fairness considerations in satisfaction prediction across different user groups\n- Potential bias of the router\n\n3. Deployment Risks\nShould address:\n- Risks of system manipulation through adversarial requests\n- Failure modes in high-stakes applications\n- Security considerations for the routing system\n\nSuggestion: Add a dedicated section discussing these aspects comprehensively, particularly focusing on the societal and ethical implications of automated decision-making in service provision.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nAs Gen AI models, including LLMs, continue to get commoditized, leveraging a plurality of models helps meet varying user/customer performance needs as well as minimize resource utilization. This paper addresses the problem of how to select the best LLM per user query by casting satisfaction of service-level agreements by model hosts and optimal resource utilization of a given query as competing objectives of an online optimization problem. The authors provide both theoretical guarantees (Section 3) as well as empirical evaluation across 8 benchmark datasets and 3 model sizes of the Llama family. Results show that the proposed approach outperforms related approaches on benchmark datasets and selected models.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. A novel approach, supported by a combination of theoretical analysis and experimental results. The theoretical guarantees distinguish this work from related approaches.\n2. The paper writing is good, providing a good explanation of results via separate cost analysis, model call ratio, approach overhead costs, user satisfaction prediction.\n3. Good coverage of competing approaches and their use in comparative evaluation.\n4. Appreciate the clear callout of contributions at the end of section 1: many submissions miss out on this helpful practice that supports clarity.\n5. Similarly, Table 1 provides a good overview, as well as a relative comparison of baseline approaches.\n6. Publicly available code supports reproducibility.\nWeaknesses\n1. Outside of benchmarks, where ground-truth labels support model calibration, it is unclear how to learn the relative ability of the various models in the model zoo.\n2. Solution has been tested with only 3 models from the same model family.\n3. Ablation study, quantifying the relative contributions of satisfaction predictor and the quality of the online optimization, for example, is missing.\n\nQUESTIONS:\n1. How would the proposed approach deal with non-stationary and open-ended task compilations?\n2. How would the approach fare in the presence of task-specific, fine-tuned models?\n3. Table 2 is too dense with tiny fonts. Is there an alternate way to present these results?\n4. Might it be better to refer to the approach as \"inference-time or deployment-time\", instead of \"test-time?\" That would also help with the clarity.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents MESS+, a novel algorithm designed to automatically and efficiently select the best Large Language Model (LLM) from a \"zoo\" of different models. The core challenge it addresses is balancing the competing interests of end-users, who want high-quality responses, and service providers, who want to minimize operating costs.\n\nMESS+ formulates this challenge as a stochastic optimization problem. It works by learning in real-time, as users interact with the system, to predict the probability that any given LLM in the zoo can satisfy an incoming request. For each request, the algorithm then chooses the most cost-effective model that is likely to meet a predefined Service Level Agreement (SLA), which guarantees a minimum rate of user satisfaction over time.\n\nThe authors provide a theoretical analysis of the algorithm's performance and demonstrate its effectiveness through experiments. Results show that, on average, MESS+ can reduce operating costs by a factor of two compared to existing LLM routing techniques while successfully upholding the required service quality.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The routing problem setting with SLA constraint is novel and practical.\n- The proposed approach is grounded with solid theoretical analysis.\n- The main experiment results demonstrates the effectiveness of MESS.\n\nWeakness:\n- As acknowledged by the authors, the method requires user satisfaction labels on the fly, which is usually not a practical assumption.\n- Please see questions section.\n\nQUESTIONS:\n- Why user satisfaction is limited to a binary value? For more open-ended queries, binary value may not capture user feedback precisely.\n- In Table 2, how are the values of $\\alpha$ determined? Does MESS+ ensure satisfaction of SLA at all different values of $\\alpha$ beyond these 3 picked points? Also, does MESS+ achieves the lowest cost among all competing methods at every point?\n- As RouteLLM and RouterDC are not designed for SLA constraints, how are their operating point selected given a specific $\\alpha$?\n\nLIMITATIONS:\nYes, limitations are sufficiently discussed in the conclusion section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MESS+ (Model Selection with Cost-optimal Service-level Guarantees), an per-request optimization algorithm to select an LLM from a set of models at inference time. with the objective to minimize cost while meeting a given service-level agreement.  The algorithm dynamically learns the probability of an LLM satisfying a user request in real-time and uses this information to make cost-optimal model selection decisions. MESS+ is doing so by using virtual queues to track performance over time, and then choosing the best model from solving the resulting optimization problem. Experimental results using various LLM benchmarks show that MESS+ can provide an average of 2x cost savings compared to existing LLM routing techniques, achieving the targeted request satisfaction rates with minimal overshoot.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1) Clear motivation and context: The motivation of the paper is clearly stated and easy to follow. The problem is well-defined, and the authors clearly explain how their approach differs from prior work, especially in terms of the tradeoffs it considers.\n\n2) Strong theoretical foundation: The algorithmic framework is grounded in stochastic control theory. The mathematical formulation provide a clear rationale for the proposed algorithm. The system offers theoretical guarantees for meeting user-defined constraints or SLAs, making it a practical option in real-world settings where users require a minimum performance level while minimizing cost.\n\n3) Careful evaluation: The system is evaluated on a diverse set of benchmarks. Results consistently show that it can reduce operating costs while maintaining target performance. Notably, it does so without relying on explicit routing preference datasets, thanks to the use of an online learning algorithm. Furthermore, the authors conduct ablations on key hyperparameters (α and V), providing insight into their influence. They also account for the cost overhead introduced by the router/predictor, which strengthens the evaluation.\n\nWeaknesses:\n1) Strong assumption on model feedback: MESS+ assumes access to feedback from all models in the zoo to train the predictor. Although the authors acknowledge this as a limitation, it remains a significant practical concern. In many real-world scenarios, such feedback may not be available, especially in the absence of a perfect automatic verifier. Despite this, the benchmarks assume full access to high-quality feedback, which may not generalize to more constrained or noisy settings.\n\n2) Unfair comparison with baselines: MESS++ is benchmarked under fixed SLA constraints, while baselines are allowed to exceed these constraints. This leads to a comparison where baselines achieve higher satisfaction rates, which is interpreted solely as inefficiency. However, this could also be the result of MESS++ choosing a more favorable point in the quality-cost tradeoff space. I understand that arriving to this tradeoff point is actually a strength of MESS++, but still this might be unfair to competing approaches.  Furthermore, this makes it difficult to isolate the true benefits of MESS+ under realistic deployment tradeoffs. One possibility would be to pick as target the request satisfaction given by competing approaches and then measure the reduction in cost achieved by MESS+ for this target.\n\n3) Limited model diversity: The evaluation uses a model zoo of only three models, with significantly different sizes (1B, 8B, 70B). This setup likely makes routing easier, as the models are highly separable in quality and cost. In real-world deployments, model zoos may include many more models with subtler differences. It is unclear how well MESS+ would perform in such cases, where the quality-cost tradeoffs are less clear-cut.\n\n4) Unclear hyperparameter choices: The paper introduces two hyperparameters that users would have to define / learn for their\nspecific routing use case, and how they might choose them is not immediately clear.\n\nQUESTIONS:\n1) How would the cost and quality results change if MESS+ aimed for a quality level comparable to what baselines actually achieve, rather than a seemingly arbitrary alpha that other systems don’t accept? Would the cost margins shrink significantly?\n\n2) Initial exploration cost is quite high when all models in a zoo are being queried to learn a predictor model. Could the exploration phase be made more efficient by warm-starting the predictor with benchmark performance data for similar tasks? Alternatively, could a larger or pretrained model with better priors be used to reduce exploration cost and improve early performance?\n\n3) The current method assumes access to full feedback from all models in the zoo. This assumption is not realistic in many practical settings. How does the system perform when only partial feedback is available during the exploration phase, e.g., querying only a subset of models per input? Can the learning algorithm be adapted to handle such incomplete supervision robustly?\n\n4) The system is currently tested on a zoo with just three highly distinct model sizes (1B, 8B, 70B). How does performance (cost, routing accuracy, SLA satisfaction) scale as the zoo expands to include more models, especially those that are closer in performance or from different families? Are there diminishing returns or increases in misrouting as model separation becomes less obvious?\n\n5) How do you choose hyperparameters apha and V for real-world deployments?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-07"}