Title: ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
Venue: NeurIPS 2025
Accepted: yes
Keywords: Hardware and Systems

Efficient Inference Methods

Distributed Inference

Visual Question Answering
Abstract: Multimodal large language models (MLLMs) extend LLMs to handle images, videos, and audio by incorporating feature extractors and projection modules. However, these additional components—combined with complex inference pipelines and heterogeneous workloads—introduce significant inference overhead. Therefore, efficiently serving MLLMs remains a major challenge. Current tightly coupled serving architectures struggle to distinguish between mixed request types or adapt parallelism strategies to different inference stages, leading to increased time-to-first-token (TTFT) and poor resource utilization. To address this, we introduce Elastic Multimodal Parallelism (EMP), a new serving paradigm that elastically adapts to resource heterogeneity across request types and inference stages. Building upon EMP, we develop ElasticMM, an MLLM serving system that (1) separates requests into independent modality groups with dynamic resource allocation via a modality-aware load balancer; (2) decouples inference stages and enables parallelism adjustment and adaptive scaling via elastic partition scheduling; and (3) improves inference efficiency through unified multimodal prefix caching and non-blocking encoding. Experiments on diverse real-world datasets show that ElasticMM outperforms state-of-the-art (SOTA) serving systems, reducing TTFT by up to 4.2$\times$ and achieving 3.2–4.5$\times$ higher throughput while meeting service-level objectives (SLOs).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a serving framework for multimodal large language models (MLLMs) that elastically adapts to resource heterogeneity across different request types and inference stages. The proposed framework, ElasticMM, enables dynamic resource allocation via a modality-aware load balancer, parallelism adjustment and adaptive scaling through elastic partition scheduling, and inference optimization using unified multimodal prefix caching and non-blocking encoding. By evaluating the framework on the VisualWebInstruct and ShareGPT-4o datasets with LLaMA3.2-Vision-11B and Qwen2.5-VL-7B models, the authors demonstrate significant performance improvements, achieving a 4.2× reduction in time-to-first-token and approximately 4× higher throughput.\n\nSTRENGTHS AND WEAKNESSES:\nStrength \n- The paper achieves impressive improvements in inference efficiency, substantially reducing latency and enhancing throughput for MLLMs.\n- The paper provides a comprehensive review of existing literature, clearly outlining the motivation and necessity for the proposed approach.\n- The method is thoroughly analyzed, with detailed ablation studies validating the contributions of each component.\n\n\nWeakness\n- Some figures lack clear explanations, making it difficult to fully understand the presented concepts (see detailed questions below).\n- The architectures of the target inference models are described only in text, which makes it challenging to grasp the inference pipeline and model architecture. An overview diagram would significantly aid comprehension of Lines 72–94.\n- The evaluation focuses only on 7B and 11B models. It would be valuable to assess performance on larger model sizes for a more comprehensive understanding of scalability.\n\nMinor\n- In Figure 4, the letter \"s\" in “Images” appears to have a smaller font size.\n- Section 3.3 (Inference Optimization) may not logically belong under Section 3 (Elastic Multimodal Parallelism). I recommend either renaming Section 3 or moving Section 3.3 to a new standalone section.\n\nQUESTIONS:\n- (Figure 1) What datasets are used in subfigures (b) and (c)?\n- (Figure 2) Why does the image-text model exhibit dynamic request distribution?\n- (Figure 4) What do the dashed lines within the rectangles signify? What does the GPU icon specifically represent? How could existing methods be visualized within this figure for comparison?\n- Would it be possible to conduct ablation studies for each method proposed in Sections 3.1 and 3.2?\n\nLIMITATIONS:\nThe authors briefly mention future work in a single sentence. A more in-depth discussion on current limitations and potential future directions would enhance the clarity.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper targets the high-latency, low-throughput pain points that arise when existing LLM serving systems are asked to handle *multimodal* workloads.  The authors observe that current pipelines run every stage—image preprocessing/encoding, LLM prefill, and decode—on the same GPU instances and batch both text-only and multimodal requests together, leading to severe resource contention and SLO violations.\n\nTo address this, they propose **Elastic Multimodal Parallelism (EMP)** and implement it in a system called **ElasticMM**.  The key ideas are:\n\n* **Modality-aware decoupling:** Route text-only and multimodal requests to separate *modality groups*, each with its own GPU pool, then let a load balancer proactively allocate—and reactively pre-empt—GPUs across groups as traffic shifts.\n* **Stage-level elasticity:** Within each group, further disaggregate the pipeline so that encoding, prefill, and decode can scale up or shrink independently via an elastic partition scheduler driven by a gain-cost model.\n* **MLLM-specific optimizations:** (i) a **unified multimodal prefix cache** that reuses both image and text prefixes, and (ii) **non-blocking encoding** that runs vision encoding asynchronously, eliminating its interference with prefill and decode.\n\nThe prototype is built atop vLLM and evaluated on an 8×A800 workstation with two representative MLLMs—LLaMA-3.2-Vision-11B (enc-dec) and Qwen-2.5-VL-7B (dec-only)—under two mixed workloads, VisualWebInstruct and ShareGPT-4o.  Compared with vLLM, ElasticMM cuts time-to-first-token by up to **4.2×** and boosts end-to-end throughput by **3.2–4.5×** while meeting service-level objectives; ablations show that each component of EMP contributes materially to these gains.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n1. **Robust quality and engineering rigor**\n   An end-to-end implementation atop vLLM is benchmarked on two real-world workloads and two representative MLLMs, showing up to $4.2\\times$ TTFT reduction and $3.2\\!-\\!4.5\\times$ higher throughput, with detailed ablations that isolate each component’s contribution.\n\n2. **Novel decoupled-and-elastic paradigm for MLLM serving (Originality & Significance).**\n   Elastic Multimodal Parallelism (EMP) introduces (i) modality-aware load balancing, (ii) stage-level elastic partition scheduling, plus (iii) unified multimodal prefix caching and non-blocking encoding—an architectural combination not present in prior systems.\n\n### Weaknesses\n\n1. **Single-node evaluation limits external validity**\n   All results come from one 8 × A800 workstation; multi-node or heterogeneous-cluster behavior is left to “future work,” so real-world scalability remains unproven.\n\n2. **Baseline coverage is too narrow**\n   Only vLLM and an in-house static variant are compared; other state-of-the-art disaggregated or high-throughput frameworks (SGLang, DeepSpeed-FastGen, Distserve etc.) are excluded, leaving the true performance gap uncertain. And the number of models are limited.\n\nQUESTIONS:\n1.Multi-node scalability.\nYour experiments are limited to a single 8 × A800 box. Could you either (a) add preliminary multi-node results or (b) provide an analytic model/trace-driven simulation that shows EMP still improves TTFT and throughput when network latency and cross-node KV migration are present?\n\n2.Baseline coverage.\nPlease justify excluding SGLang, DeepSpeed-FastGen, Splitwise, DistServe and Mooncake, or add at least one of them to Table 3. If ElasticMM still shows ≥2 × improvement under identical settings, that would materially strengthen the claim of significance.\n\nLIMITATIONS:\nEnvironmental impact. Running eight A800s at high utilization is energy intensive; explicit reporting or mitigation discussion is absent.\n\nSecurity / privacy risk. Unified prefix caching potentially exposes vision embeddings across requests. A short note on isolation mechanisms would be valuable.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces Elastic Multimodal Parallelism, a new serving paradigm for efficiently serving multimodal large language models. The proposed system, ElasticMM, addresses key challenges in multimodal inference pipelines, including significant overhead due to additional model components and diverse computational requirements across different modalities. Experimental evaluations demonstrate that ElasticMM reduces TTFT by up to 4.2× and increases throughput by 3.2–4.5× compared to SOTA methods.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n- **Quality:** The system ElasticMM is well-designed, leveraging detailed insights about workload characteristics (burstiness, computational bottlenecks) to drive performance.\n- **Significance:** Substantial improvements demonstrated experimentally highlight the practical significance, addressing critical performance bottlenecks in multimodal LLM deployment.\n\n# Weaknesses:\n- **Lack of Baseline:** This system only compares with vLLM and self-implemented vLLM-Decouple. However, in December 2024, a work related to MLLM serving was posted on arXiv (*Efficiently Serving Large Multimodal Models Using EPD Disaggregation*), which also involves issues such as the separation of different components. I suggest that the author include this work in the baseline to verify the advancement of the system.\n- **Scalability Validation:** The paper's experiments are limited to a single-node environment, potentially constraining insights about performance in real-world large-scale deployments. Although the author stated that multi-machine experiments will be left for future work, as a serving task, I believe a single machine is not sufficient, especially since the author has also involved tech like elastic scaling that clearly require multi-machine validation. Considering that the author may have limitations in machine resources, different model, trace and machines can be used for small-scale experiments.\n- **Clarity:** Some statements in the paper are not very clear, for example, where does the workload in Fig. 2 come from? Is it an open-source trace or from the company's actual production trace? Please at least indicate the source.\n\nQUESTIONS:\nPlease see Weaknesses.\n\nIt should be noted that due to possible resource or permission limitations, the author may not be able to fully address the above issues. However, I hope the author can at least add relevant baselines and their comparisons. I will adjust my score based on the extent of the author's responses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduced ElasticMM, a multi-model LLM serving framework designed to optimize time-to-first-token (TTFT) and throughput. The authors noticed the importance of separating text-only and multi-modal requests and the need to have an elastic serving system to handle dynamic multi-modal requests. Modality-aware load balancing and elastic instance allocation and elastic auto-scaling mechanism are proposed to achieves these goals. Extensive experiments on real-world work loads validated the effectiveness of the proposed method.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* This paper studies an important problem, i.e., multi-modal LLM serving, which differs from text-only LLM serving but does not receive much attention. The paper is novel and well motivated. The proposed elastic instance allocation and elastic auto-scaling mechanism do work.\n* This paper is well-written and easy to follow.\n\nWeaknesses:\n* As the authors discussed in the Limitations section, the proposed method is only implemented and tested in a single-node environment. While multi-node model serving is not studied, the contribution of this work is non-trival. It would be great if how the proposed method could be extended to work in a multi-node model serving environment could be briefly discussed in a short future work section.\n\nQUESTIONS:\n* In Figure 2, the average time per output token is shown. For the proposed approach, it increases at a very low rate as request rate increases. While for vllm, the average time per output token increases sharply. I am wondering what is the reason for the sharp increase of average time per output token when vllm is used?\n* If model parallel is needed in order to serve the model, will things change?\nFor example, if a very large vision encoder is used and a GPU only holds the vision encoder (pipeline parallel rank 0 only holds the vision encoder). Will the performance gain of the proposed method become less obvious?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-03"}