Title: Fast Inference for Augmented Large Language Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: LLM inference and serving

Augmented LLM requests
Abstract: Augmented Large Language Models (LLMs) enhance standalone LLMs by integrating external data sources through API calls. In interactive applications, efficient scheduling is crucial for maintaining low request completion times, directly impacting user engagement. However, these augmentations introduce new scheduling challenges: the size of augmented requests (in tokens) no longer correlates proportionally with execution time, making traditional size-based scheduling algorithms like Shortest Job First less effective. Additionally, requests may require different handling during API calls, which must be incorporated into scheduling.
This paper presents MARS, a novel inference framework that optimizes augmented LLM latency by explicitly incorporating system- and application-level considerations into scheduling. MARS introduces a predictive, memory-aware scheduling approach that integrates API handling and request prioritization to minimize completion time. We implement MARS on top of vLLM and evaluate its performance against baseline LLM inference systems, demonstrating improvements in end-to-end latency by 27%-85% and reductions in TTFT by 4%-96% compared to the existing augmented-LLM system, with even greater gains over vLLM. Our implementation is available online.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MARS, a novel scheduling framework designed for API-augmented Large Language Model (LLM) requests. Addressing the limitations of traditional scheduling algorithms in handling the additional latency and memory usage introduced by API calls, MARS integrates predictive, memory-aware scheduling with API handling and request prioritization mechanisms to minimize end-to-end latency. Implemented on top of vLLM and utilizing an OPT-125M model for output length prediction, experimental results demonstrate that MARS significantly outperforms existing systems in both end-to-end latency and Time-To-First-Token (TTFT)..\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n1. The paper provides extensive experimental results to demonstrate the effectiveness of the proposed framework.\n2. The paper provides a detailed introduction of scheduling API-augmented requests.\n\nWeakness\n1. The prediction module uses OPT-125M, and predicts output length by extracting the final token's embedding and classifying it into 50 bins, each representing a range of 10 tokens. However, results of prediction module with other models and different granularity of bins and tokens are absent.\n2. The paper conducts comparisons between MARS and INFERCEPT as well as vLLM. It also mentions some other relevant methods in the related work section. However, it does not provide comparative results between these mentioned methods and MARS.\n\nQUESTIONS:\n1. The paper uses OPT-125M for predicting output length with 50 bins of 10 tokens each. What is the impact on MARS's scheduling performance if the prediction model significantly underestimates or overestimates the output length for API-augmented requests? Specifically, how does prediction error propagate through the scheduling decisions and affect the claimed 27%-85% latency improvements?\n﻿\n2. The paper mentions several relevant methods in the related work section but only provides comparisons with INFERCEPT and vLLM. Why were other state-of-the-art augmented LLM serving systems (such as those handling function calling or retrieval-augmented generation) not included in the experimental comparison? Could you provide performance comparisons with at least one other recent system that specifically addresses API-augmented LLM scheduling?\n\nLIMITATIONS:\nSee weakness", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis pager introduces MARS, a brand-new inference framework that targets at optimizing augmented LLM latency by explicitly considering system and application level in LLM scheduling. It provides a predictive and memory aware scheduling approach that integrates API handling and prioritizing requests to minimize completion time. MARS is implemented upon vllm and is evaluated against baseline LLM inference systems. It shows large end-to-end latency reduction and TTFT improvements.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. API-augmented LLMs are becoming common in practice, and this work addresses real performance issues that matter.\n2. Instead of deciding handling strategies on-the-fly like INFERCEPT, MARS predicts them upfront. This lets them optimize scheduling and memory handling together, which is clever.\n3. They test on multiple models (1.1B to 70B), different datasets, and analyze what happens when predictions are wrong. The ablation studies and memory profiling add good depth.\n\nWeakness:\n1. The paper uses OPT-125M (a decoder-based model) as the predictor but doesn't justify why this is better than encoder-based models like BERT for this prediction task. Encoders typically perform better on classification tasks since they can see the entire input context bidirectionally.\n2. Critical training information is absent - how much training data was needed, learning rate and epochs, what the training procedure looked like, convergence criteria, and how they handled the class imbalance across different output length bins.\n3. While they mention 13.7ms prediction time, there's no analysis of how this overhead affects overall system throughput or how it scales with request rates. This could be significant in high-throughput scenarios.\n\nQUESTIONS:\n1. Why did the authors choose a decoder-based model (OPT-125M) over encoder-based models (like BERT) for the prediction task, given that encoders typically excel at classification with full context visibility?\n2. Can the authors provide detailed training specifications for the predictor, including dataset size, training epochs, loss convergence, and strategies for handling output length distribution imbalance?\n3. How does the 13.7ms prediction overhead scale with increasing request rates, and what is the break-even point where prediction benefits outweigh the computational cost?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces MARS, a scheduling framework tailored for API-augmented LLMs—language models that interleave generation with external tool or API calls. The key insight is that these requests break the assumptions of traditional scheduling approaches (e.g., token-length-based methods), due to their unpredictable latency and memory behavior. MARS addresses this by (1) predicting each request’s pre-API output length and API call duration, (2) dynamically selecting a memory handling strategy (Preserve, Discard, or Swap) to manage GPU memory usage, and (3) scheduling requests based on their estimated total memory footprint rather than output length. To ensure fairness, it also integrates a starvation prevention mechanism. The system is implemented atop vLLM, supports multi-GPU inference, and is evaluated on ToolBench and INFERCEPT across multiple LLMs (e.g., GPT-J, Vicuna, LLaMA-2 70B). Results show consistent improvements in latency (up to 4.2× reduction in TTFT and 85% reduction in end-to-end latency) over strong baselines like INFERCEPT and vLLM.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. The paper targets a timely and impactful problem—how to efficiently schedule LLM inference with external API calls, a common pattern in modern LLM-powered applications (e.g., tool use, function calling, plugin ecosystems). This makes the work immediately useful for system designers and platform engineers.\n\n2. Extensive evaluations are performed on multiple real-world benchmarks (ToolBench, INFERCEPT) and across multiple LLM backends (GPT-J, Vicuna-13B, LLaMA-2-70B), demonstrating up to 4.2× reduction in TTFT and 85% latency improvement. The ablations are meaningful and help isolate component contributions.\n\n3. The proposed MARS system is implemented atop vLLM, supports multi-GPU inference, and includes practical engineering components such as starvation prevention and per-request memory estimation. The framework is modular and extensible.\n\nWeakness:\n\n1. OPT-125M is mentioned for predictions , more specifics on how it's trained to classify into \"50 bins\" and how the \"average size from the training set for that API class\" is used for API response size estimation could be clearer. This could be elaborated in the appendix\n\n2. The experiments are not comprehensive enough, as the study didn't test models other than OPT-125M for predictions.\n\nQUESTIONS:\n1. Could the authors discuss if MARS's performance is sensitive to the choice of the prediction model? Have preliminary investigations been conducted using other lightweight models for predictions? Providing a brief discussion on this (even if no extensive results are available) would be valuable.\n\n2. Starvation Prevention Threshold Rationale: Could the authors provide a more detailed explanation or sensitivity analysis for the choice of this specific threshold (100)? For example, discussing the trade-offs observed with different thresholds (e.g., how a much lower or higher threshold impacts both tail latency and overall throughput/fairness) would strengthen the justification for this parameter.\n\n3. What should the MARS system do if an API is added or removed?\n\nLIMITATIONS:\n1. Multi-API Cumulative Memory: The paper explicitly states that the current multi-API approach \"does not account for cumulative memory usage,\" and that \"predicting the total number of API calls and their resource usage is challenging, and is left for future work.\"\n\n2. They acknowledge that \"Improving prediction accuracy remains a promising future work,\" implying that while their simple, lightweight predictor yields benefits, it's not the ultimate solution for prediction.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces MARS, an inference framework designed to optimize completion latency for API-augmented large language model requests by introducing a predictive, memory-aware scheduling approach.\n\nMARS first predicts each request’s pre-API output size and API call duration to select one of three handling strategies (Preserve, Discard & Recompute, Swap) that minimizes memory waste. It then ranks requests by their predicted cumulative memory footprint and schedules them accordingly.\n\nImplemented on top of vLLM, MARS demonstrate 27–85% end-to-end latency reductions and 4–96% TTFT improvements compared to the existing system, INFERCEPT.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The proposed approach is technically sound. It is reasonable to estimate each request’s cost with the API call duration considered. It is intuitive that predicting the cost in advance will be helpful for request scheduling.\n    \n- Extensive evaluation across diverse models, datasets, and request rates is conducted, showing substantial performance gains.\n    \n- Open-source implementation on vLLM ensures reproducibility and real-world applicability.\n    \n\n## Weaknesses\n\n- Missing critical technical descriptions. While the big picture of the two-step pipeline (e.g., L138-L141) is clear, how each step is implemented remains obscure, making the methodology part difficult to follow\n    \n    - How are the output size and API duration? How is the handling strategy predicted, and when does it exactly happen? It is advisable to illustrate these designs with formalized descriptions and diagrams.\n        \n    - How are requests ranked, or more specifically, how is the rank index calculated? Formalized descriptions or an algorithm may make this step clearer.\n        \n- Lack of overhead analysis. Prediction and scheduling overheads and their impact on throughput are mentioned but not analyzed and deeply quantified.\n    \n    - In L312-313, critical issues are mentioned with a phrase and then abruptly point to the Appendix without any analysis, making this paper look incomplete and submitted in a hurry.\n\nQUESTIONS:\nSee the Strengths & Weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}