ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
Authors: Zedong Liu
Shenggan Cheng
Guangming Tan
Yang You
Dingwen Tao
Venue: NeurIPS 2025 oral ()
Keywords: Hardware and Systems
Efficient Inference Methods
Distributed Inference
Visual Question Answering
Primary area: infrastructure

Abstract: Multimodal large language models (MLLMs) extend LLMs to handle images, videos, and audio by incorporating feature extractors and projection modules. However, these additional components—combined with complex inference pipelines and heterogeneous workloads—introduce significant inference overhead. Therefore, efficiently serving MLLMs remains a major challenge. Current tightly coupled serving architectures struggle to distinguish between mixed request types or adapt parallelism strategies to different inference stages, leading to increased time-to-first-token (TTFT) and poor resource utilization. To address this, we introduce Elastic Multimodal Parallelism (EMP), a new serving paradigm that elastically adapts to resource heterogeneity across request types and inference stages. Building upon EMP, we develop ElasticMM, an MLLM serving system that (1) separates requests into independent modality groups with dynamic resource allocation via a modality-aware load balancer; (2) decouples inference stages and enables parallelism adjustment and adaptive scaling via elastic partition scheduling; and (3) improves inference efficiency through unified multimodal prefix caching and non-blocking encoding. Experiments on diverse real-world datasets show that ElasticMM outperforms state-of-the-art (SOTA) serving systems, reducing TTFT by up to 4.2$\times$ and achieving 3.2–4.5$\times$ higher throughput while meeting service-level objectives (SLOs).

ElasticMM: Efficient Multimodal LLMs Serving with Elastic Multimodal Parallelism
Zedong Liu Shenggan Cheng', Guangming Tan', +Yang You', and +Dingwen Tao
Institute of Computing Technology, Chinese Academy Sciences University of Electronic Science and Technology of China 'National University of Singapore
Abstract
Multimodal large language models (MLLMs) extend LLMs to handle images, videos, and audio by incorporating feature extractors and projection modules. How- ever, these additional components-comieeee
with complex inference pipelines and heterogeneous workloads -nnkloads-introduce significant inference overhead. There- fore, efficiently serving MLLMs remains major challenge. Current tightly cou- pled serving architectures struggle to distinguish between mixed request types adapt parallelism strategies different inference stages, leading increased time-to-first-token (TTFT) and poor resource utilization. To address this, we introduce Elastic Multimodal Parallelism (EMP), new serving paradigm that elastically adapts to resource heterogeneity across request types and inference stages. Building upon EMP, we develop ElasticMM, an MLLM serving system that (1) separates requests into independent modality groups with dynamic resource allocation via a modality-awan load balancer; (2) decouples inference stages and enables parallelism adjustment and adaptive scaling via elastic partition scheduling; and (3) improves inference efficiency through unified multimodal prefix caching and non-blocking encoding. Experiments diverse real-world datasets show that ElasticMM outperforms state-of-the-art (SOTA) serving systems, reducing TTFT by up to 4.2x and achieving 3.2-4.5x higher throughput while meeting service-level objectives (SLOs).
Introduction
Large Language Models (LLMs) have had a profound impact on real-world applications due to their exceptional performance [1-3]. Their gradual expansion into the multimodal domain has led the emergence of Multimodal Large Language Models (MLLMs), which can process inputs such as images, videos. and audio [4, 5]. By employing feature extractors and projection modules map multimodal inputs into the LLM feature space, MLLMs excel at tasks like visual question answering (VQA), image captioning, and multimodal interaction [6-9]. Consequently, MLLMs are increasingly deployed in online services, where they are highly sensitive to service-level objectives (SLOs).
Unlike traditional LLMs that process only text [10-14], MLLMs must handle diverse input types, introducing additional model components and a more complex inference pipeline. This pipeline typically includes distinct stages such image preprocessing, image encoding, and language model inference (comprising prefill and decode). These added stages increase computational complexity (Fig. 1b) and significantly extend the average context length (Fig. Ic). However, existing inference serving systems adopt a tightly coupled architecture that neither distinguishes between request types
ot Corresponding Authors.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

TIFT Breakdown
Llama-3.2-118 TTPT
CDF of Prompt Length
Image Preprocessing
1.0-
Image Encoding
B003
LLM heel
Proprocessing
0.4
LLM Prefil
Image Encoding
02
In Irage-tert Requests
Rotal Time
Text-only
256 1k
24
10
10'
10"
0k
10k
15k
20k
Sequence Length
Time (ms)
Total Prompt Length
(a) TTFT breakdown
(b) CDF of inference stage latencies
(c) CDF of prompt lengths
Figure 1: MLLMs' inference overhead and workload complexity. (a) and (b) demonstrate the significant overhead introduced by MLLMs. (c) reveals the longer context in multimodal requests.
Results obtained using the LLaMA3.2 1-1A..22112 IB model on the ShareGPT-44 dataset.
nor decouples inference components, executing all stages the same hardware instances [15-
18]. As a result, time-to-first-toee (TTFT) increases sharply under heavy multimodal workloads. Furthermore, encoder-decoder models introduce cross-attention layers [19-21], and the computational heterogeneity between text-only and multimodal requests reduce batching efficiency.
To address this issue, decoupled multimodal inference architecture emerges promising solu- pro--sccssin text-only and multimodal requests on separate instances. Our systematic analysis reveals that multimodal workloads exhibit bursty patterns, often characterized by sudden spikes in image inputs (as also observed in previous work [22]), and that different inference stages benefit from varying degrees of parallelism. For example, image encoding and LLM prefill stages have higher computational complexity and benefit from large-scale parallelism, whereas the decode stage has limited scalability [23-25]. These observations motivate two critical design challenges:
(1) static resource allocation struggles to adapt to dynamically changing workloads, and (2) fixed parallelism strategies often fail to match the varying resource demands of different inference stages.
To this end, we propose Elastic Multimodal Parallelism (EMP). address the first challenge, we introduce modality-aware load balancing technique that monitors real-time workload fluctuations and dynamically adjusts resource allocation across different modality groups. To tackle the second challenge, we design elastic partitioned scheduling technique, which enables dynamic parallelism adaptation at the granularity of each inference stage. For example, encoding and prefill scaled across more GPUs, while the decode stage reduces parallelism free up resources for other requests.
stages such as
Building on EMP, we present ElasticMM, MLLM serving system with four key contributions:
We identify tight coupling in existing systems major bottleneck for MILM serving and propose EMP, a new serving paradigm that separates text-only and multimodal requests into independent modality groups and decouples inference stages for stage-specific parallelism.
We design two core techniques for EMP: (1) modality-aware load balancing, which dynamically allocates and scales resources across modalities to handle unpredictable workloads; and (2) elastic partitioned scheduling, which controls parallelism through flexible scheduling and scaling to maximize inference performance.
We propose two optimizations to mitigate MLLM-specific overheads: unified multimodal prefix caching to reduce redundant computation and data transfer, and non-blocking encoding to minimize the impact of encoding latency on the overall inference pipeline.
We conduct comprehensive evaluation of ElasticMM on two real-worke datasets. Compared to the SOTA baseline vLLMM [16), ElasticMM reduces TTFT by up to 4.2x and achieves 3.2x throughput improvement while meeting service-level objective (SLO) requirements.
2 Background and Motivation
2.1 Multimodal Large Language Models Inference
MILMM Inference Pipeline. MILMM extend traditional language models by integrating visual, auditory, and other modalities, enabling unified reasoning across heterogencous input sources for

---

Table Model configurations for four representative MLLMs with input image of 904 x 9944 pixels.
MLLM Model Name
Architecture
Image Encoder
Total Image Token Size
LLM Backend
(#Params)
(#Params)
Llama3.2 2-iiion 1IB
Encoder- -ecdder
VIT-H/14 (630M)
6516
Llama 3.1 (8B)
1lmma3.2-isiin 90B
Secoder-Decoder
ViT-H/14 (630M)
6516
Llama 3.1 (70B)
Qwen2.5-Vi VL. 7B
Decoder-only
VIT (670M)
7410
Qwen2.5 (7B)
Qwen2.5- VL 72B
Decoder-only
VIT (670M)
7410
Qwen2.5 (72B)
complex multimodal tasks [1-3, 5]. Taking vision-language models as an example, the MLLM inference pipeline typically consists of three key stages: (I) Image Preprocessing: Raw images are resized and divided into uniformly sized tiles [26]. (2) Image Encoding: A vision encoder extracts visual features and converts them into vision tokens [27, 28). (3) Text Generation: A language model takes both vision tokens and a text prompt as input to generate responses.
MILM Architectures. Modern MLLMs generally fall into two architectural categories: (1) Decoder- only (DecOnly) architectures, such as LLa Vision [8], Qwen- VL. [29], DeepSeek's Janus [30], and Intern VL. [7]; and (2) Encoder-decoder (EncDec) architectures, including LLaMA-3.2 ,2 Vision [2 1. NVLM-X [20], and Flamingo [19]. The key difference lies how vision and text tokens are processed. Decoder-only models concatenate vision and text tokens and feed them together into the language model; vision tokens participate in every generation step. In contrast, encoder-decoder models use cross-attention modules align multimodal inputs: vision tokens interact with text tokens only via these cross- attention layers, which are inserted between self-attention layers.
Additional Overheads. With the continuous increase in model scale and complexity, the com- putational cost of inference also grows, especially for MILMs. Compared to language models, MLLMS introduce overhead from two main sources (shown in Fig. la): (1) Increased Architectural Complexity: extra components such as vision encoders and cross-attention layers make the model heavier. (2) Extended Context Length: Multimodal data, once encoded into tokens, are concatenated with text prompts, increasing input length during inference [26]. Specifically, this extension increases computational load, memory usage, and causes degradation in latency and throughput. Table 1 illustrates the encoder sizes and vision token lengths added by four mainstream open-source MLLMs.
2.2 LLM Serving Systems
Existing LLM Serving. Existing serving systems [31, 17, 32), such as vLLM [16] and SGLang [18], have introduced a range of techniques accelerate inference. To avoid redundant computation, these systems cache the key-value (KV) states of tokens for reuse. This design splits the inference process into two stages: the prefill stage and the decode stage. The prefill stage computes the KV cache for all input tokens and generates the first output token, while the decode stage generates one token per iteration. Consequently, the prefill stage carries a significantly heavier computational load. To this end, ORCA [33] introduced continuous batching, enabling the system to process the prefill and decode stages of multiple requests in an uninterrupted stream of batches. To alleviate the burden of long input contexts, chunked prefill [25, 34] splits long contexts into smaller segments that can be interleaved with decoding. However, interference between the two stages remains difficult to eliminate. A promising solution is prefill-decoding disaggregation, which places the two stages on separate GPU instances [23, 24, 35]. LoongServe [36] further introduces clastic sequence parallelism to further optimize parallelism and resource allocation under this disaggregated architecture.
Coupled Multimodal Serving. Despite these advancements, SOTA serving systems exhibit sig- nificant coupling issues when deployed for MLLMs. This coupling occurs at both the service and infrastructure levels. At the service level, existing systems treat unimodal (text-only) and multimodal requests identically, routing both through the same inference pipeline, despite their vastly different the infrastructure level, all vision encoders, and LLM backends-ane are colocated on the same hardware server. These tightly coupled components must share compute and memory resources while being constrained to a uniform batching and model parallelism strategy, leading to resource contention and inefficiency.
computational requirements.
preprocessors,
3

---

Text-Only Request
Image-Teer Text Request
Modality-Level
(Modality-Aware Load Balancing $3.1)
Encode Stage
Prefill Stage
Decode Stage
Prefill Stage
Decode Stage
Stage-Level (Elastic Partition Scheduling $3.2)
Request Scheduling
Stage Allocation
Elastic Auto-Scaling
Text-Only Group
Unified Multimodal Prefix Cache ($3.3)
Multimodal Group
Auto Auto-Scaling
Figure 2: Framework diagram of ElasticMM. The figure illustrates a two-level scheduling framework that collaboratively enables elastic multimodal parallelism.
2.3 Research Challenges and Motivations
Existing tightly coupled systems face significant limitations when serving multimodal models. Service-level Problem: Multimodal and text-only requests differ substantially in resource demands due to longer context lengths and the injection of multimodal data (as illustrated in Fig. 1c). Meeting the same SLO requires different resource allocations for each request type. However, coupling their inference execution reduces overall resource utilization and increases the risk of SLO violations. also prevents leveraging their distinct characteristics to enable more efficient inference and resource scheduling. Architectural Problem: Encoder-decode models[19-21]] which incorporate cross- attention mechanisms, are ill-suited for mixxed-batch processing. Combining both request types in a single batch only increases the latency of text-only requests but also degrades overall throughput.
These lead to our Key Insight 1: better meet the distinet requirements of each request type, modality-awee awaare decoupled inference architecture should be adopted, where text-only and multimodal requests are processed independently.
However, naive static decoupling with fixed resource allocation is ineffective under dynamic work- loads. It cannot adapt to changing request distributions (e.g., sudden spikes image traffic) or adjust parallelism strategies based on evolving resource demands.
This leads to Key Insight 2: An elastic serving system is essential for handling dynamic multimodal workloads. Such a system must extend static architectures to support dynamic resource reallocation and stage-specific parallelism adjustments. By enabling scalable execution across elastic instances, it effectively alleviates compute and memory bottlenecks in multimodal serving.
3 Elastic Multimodal Parallelism
As discussed in the previous section, it is essential to build an clastic MLLM serving system that can dynamically adjust both resource allocation and parallelism strategies. To this end, we introduce a hierarchical framework that enables elastic scheduling, where all instances are organized into two levels, as illustrated in Fig. 2.
At the first level, the modality level, instances are grouped based on the modality of the models they serve (e.g., text-only or multimodal). At the second level, stage level. the inference pipeline further disaggregated into distinct stages, such as encoding, prefill, and decode. At each level, our system provides both decoupling and elasticity, which are the two core strengths of ElasticMM. This design climinates resource contention and maximizes utilization.

---

Multimodal Group
Text-only Gro-up
Encoding Stage
Decoding
Prefill Stage
Decoding Stage
Prefill Stage
Stage
4 Images
2000 img tokens
Bach size size-256
800 text tokens
Bach size 64
128 text tokens
GPU
EPPU
GPU
EGPU
EPP
EGPU
EGPU
GPU
Figure 3: Illustration of the elastic scheduling space in EMP.
At the modality level, we design a modality-aware load balancing strategy to dynamically allocate resources acToss modality groups in response to fluctuating request loads. At the stage level, we apply elastic partition scheduling to enable flexible parallelism tailored to the needs each inference stage.
3.1 Modality-Aware Load Balancing
To tackle the inter-modality load imbalance outlined in Section 2.3, we combine both proactive and reactive mechanisms. Specifically, following prior approaches [37-39], we apply proactive mechanism to allocate resources different modality groups. However, proactive mechanism alone cannot effectively handle sudden request surges (e.g.. bursty multi-image streams). We therefore design a reactive scaling mechanism to dynamically expand capacity upon detecting resource shortages.
Proactive Mechanism. Our analysis of load patterns in inference services reveals twofold: although the short-term arrival pattern of a single request stream is difficult predict, aggregating multiple streams results in smooth and periodic patterns (e.g., lower load at night, higher during the day). As observed in previous work [22], text-only requests exhibit stable loads, while multimodal traffic is marked by pronounced surges. Based on the predictability of long-term workload, idle elastic instances can be proactively assigned to modality groups. Following previous work [37], our goal in allocation is to maximize the minimum burst tolerance across modality groups. Burst tolerance (bt) defined as the ratio between peak-availabe and average-required instances per group in Equation I. To achieve this, we employ a fast and effective greedy strategy: incrementally assigns each instance to the group currently lowest burst tolerance, continuing until resources are fully allocated.
# Instances i can use for its peak load Instances i can use for its average load
N
(I)
N,
Reactive Scaling, Due to unpredictable short-term workloads, the system may face sudden traffic bursts, such as long-text or image-heavy requests in real-time scenarios. The system evaluates the trade-off between adjusting intra-group parallelism and triggering inter-group reactive scaling, based on a gain-cost model described Section 3.2. If inter inter-group group reactive scaling is more beneficial, the modality-level manager selects instances to preempt from other groups with minimal impact. When an instance F. is preempted, its workload is merged into other instances at the same inference stage.
3.2 Elastic Partition Scheduling
After addressing inter-group GPU allocation, we turn intra-group request scheduling and parallelism adjustment, adapting to the distinct characteristics of each inference stage. For example, prefill is compute-boundi decode is always memory- bound and scales poorly [23, 24, 35]. For stages with sublinear scalability, allocating more idle GPUs improves performance. Conversely, for stages with poor scalability, it is more efficient limit the number of GPUs involved [36]. Fig. 3 illustrates request assignments and parallel execution modes in the ElasticMM service, showing that instance counts vary by modality group and inference stage. We propose Elastic Partition Scheduling, orchestrated by the stage-level manager. This strategy decomposes the scheduling challenge into three subproblems: (1) Request scheduling. (2) Stage Allocation, and (3) Elastic auto-scaling.
Request Dispatching. This step selects subset of pending requests Rp C P from the queue P a First- Come-First-Sent (FCFS) policy for the prefill phase [16, 33). One exception: if a text-only dialogue is redirected to multimodal group due to associated multimodal requests, it's prioritized- this helps
5

---

Instance
scale
scale up
R1 Prefil
R1 Decode
R3 Encode
3
12 Decode
13 Decade
Prefil
Multimodal
2
R2
R2
R2 Decade
R2 Encode
Text-only
1
R2 Prefill
Decode
Decude
R2,
Time
scale down
scale up
scale down
Figure 4: Example of elastic auto-scaling three instance.
overlap migration overhead and frees KV slots earlier. The dispatch process must address two types of constraints: GPU memory and compute throughput. For memory constraints, the manager only adds requests to Rp if sufficient KV slots are available [40]. There's a tipping point where the system shifts from memory-bound to compute-bound. Before this point, adding requests Rp improves utilization; after that, additional requests degrade performance due to extended execution time. We estimate this point by analyzing the upper bound of prefill time under memory bound.
Stage Allocation. Once the prefill request set Rp formed, the system allocates a set of elastic instances Ep to serve it, aiming to maximize GPU efficiency. Inspired by Loongserve [36], the system will prioritize allocating idle instances to RG for prefill stage. If the available KV cache slots in idle instances are insufficient, Rp is permitted to preempt instances set Eg in the decoding stage Ba. The instance with maximum unused slots are denoted emaz. Meanwhile, drawing insights from existing research, compute-intenss intensive stages like prefill can still benefit from increased parallelism. Therefore, the stage-level manager continuously considers further preemptions by selecting the emaz. migrating its KV cache to other decode instances. The gain-cost model is formulated as follows:
Gain TRRp Exp - T(p,, E, Craws)
((maa)) + L(Ba, Ea emaz)
Cost
r.input_ len
r.output len
RERp
PEBA
The gain of preemption is quantified as the acceleration gained by adding Cmaz to the prefill process. The cost accounts for both migration overhead ((emax) and the performance impact L on the preempted computation. A tunable penalty factor w is introduced to control the aggressiveness of preemption, enabling flexible system behavior under varying workloads. Within a single inference stage, we prioritize Data Parallelism (DP). Alternatives such as Tensor Parallelism are only employed when a single GPU cannot hold the model weights. This design offers several advantages: 1) During clastic scaling, only the KV cache needs to be migrated, avoiding expensive weight transfers; 2) DP enables more flexible utilization of fragmented GPU resources, whereas tensor parallelism typically requires an even number of devices and stricter placement constraints.
Elasic Auto-San Sitto-Saaiin. ElasticMM supports elastic scaling, including at the stage level. Since resource allocation is primarily guided by prefill, we monitor the decode phase to trigger elastic adjustments. Given that the decode stage exhibits poor scalability, we shrink it to the minimum parallelism. When the GPU resource becomes insufficient during decoding, the system triggers reactive scaling. Based on prior observations, decode bottlenecks often occur at FFN layers and depend on batch size. We establish scaling thresholds through offline profiling. The scaling-up up first tries to allocate idle instances from the same group. If that fails, first selects a candidate instance emax from intra- group prefill instances and a candidate e'nxx from inter- group instances, using the following gain-cost to estimate the trade-off between potential speedup and migration overhead. The instance with the highest net gain is selected for preemption. If inter-group preemption offers greater benefit, the system triggers the reactive scaling mechanism described in Section 3.1.
Gain AvgLat, - T(Ba, E. J Cmax)
M(emas) + w L(R,, E emax)
Cost
(3)
r.output_len
r.input_loo len
FER
Fig. 4 visualizes auto-scaling, Given the higher computational complexity of multimodal encoding compared to text-only prefill and decode, R2's encode stage is initially allocated more resources. During RI's decoding progression, the growing token sequence increases both computational load and KV cache memory pressure. This necessitates expanding decode stage (( (R1, 72) and (R1, 73)). Later, when multimodal request R3 arrives, the multimodal group experiences resource contention. It

---

preempts idle resourees from the text-only group for encoding. After R3's encoding, allocating more resources its prefill stage yields better throughput, so R3 automatically scales up to more instances.
3.3 Multimodal Inference Optimization
Unified Multimodal Prefix Cache. In real-world service scenarios, user requests often exhibit redundancy. For example, requests may share identical system prompts in their headers involve repeated transmission of the same images. To address the issue of redundant computation and data transmission in multimodal inference systems, we propose a Unified Multimodal Prefix Cache optimization strategy. This strategy integrates text prefix caching with multimodal input caching to build unified caching scheme. We categorize cached objects into two pools: one for tokens encoded from multimodal inputs and the other for prefix tokens from unified sequences (including both multimodal and text tokens). Following prior works [18, 41], each cache pool is managed with a Least Recently Used (LRU) dynamic eviction strategy to keep memory usage under control. When a multimodal input is received, we generate a hash. If the hash matches an existing entry, we skip re- encoding and use the cached tokens. After merging with the text tokens, we check a prefix tree in the second cache pool to find the longest matching prefix. That portion of the sequence skips the prefill step and directly uses the cached key-value results. This unified caching mechanism significantly reduces redundant visual model overhead as well as repetitive computation in the language model.
Non-blocking Encoding. As illustrated in Fig. la, the vision encoder of a multimodal large model introduces substantial overhead due to image preprocessing and encoding -ofeen taking more than five times longer than the prefill stage. However, existing inference frameworks tightly couple the encoding and prefill stages, leading to blocking behavior during encoding that delays subsequent stages. This adds latency to the first token response (TTFT) and reduces overall throughput. Our solution decouples the vision and language models by isolating image preprocessing and encoding into separate process or even separate instance exeeanee-exeeuted asynchronously. By incorporating Unified Multimodal Prefix Cache and Non-blocking Encoding, ElasticMM effectively reduces redundant computations and inter-stage interference, thereby improving the efficiency of multimodal inference.
Evaluation
4.1 Experimental Setup
Model and Dataset. We select LLaMA3.2 LLaMA3.2-Vision-I1 [21] and Qwen2.5-VL-77B [29] to represent encoder-decoder and decoder-only architectures, respectively. Two open-source multimodal datasets are used, each containing mix of multimodal and text-only requests: VisualWebInstruet/ large-scale dataset collected from over 700K unique web URLs; ShareGPT-40(43) comprises 50K images of varying resolutions along with corresponding text prompts sourced from the multimodal GPT-4o model. VisuslWWeInstruet contains longer average text inputs, while ShareGPT-40 includes higher-resolution images, making the two datasets complementary. This combination enhances the comprehensiveness and rigor of our evaluation. Following prior work [16, 36), we use a Poisson distribution to generate variable request arrival rates (requests per second, QPS) and incorporate real-world production service traces to simulate realistic workload distributions.
is
Testbed. We evaluate ElasticMM a high-end workstation equipped with eight NVIDIA A800 80GB GPUs, two 64-core Intel Xeon 8358P CPUs, and TB of DDR4 memory. The NVLink bandwidth between any two GPUs is 400 GB/s. We leave multi-node distributed studies to future work, as this testbed already demonstrates superior performance.
Baselines. We compare ElasticMM with two baselines. The first bascline uses coupled system for MLLM serving, with the SOTA system vLLM [16] (v0.6.6) as representative. It follows the architec- ture of this version, with additional model code to support Qwen2.5-VL. In this system, all inference stages are coupled and executed on the same hardware. The second baseline is DistServe [23], a decoupled architecture that colocates the encode and prefill stages while separating them from decode, using static resource allocation. We extend this system to support multimodal inference. To evaluate the effectiveness of our proposed techniques, we build ElasticMM on top of vLLMM and construct several variants by selectively enabling each technique for ablation studies, allowing us to isolate their individual contributions. Further implementation details are provided in Appendix 6.
7

---

VLLM
DistServe
ElasticMM
Qwen2.5-VL
Llama322Vision
Avg. Time to First Token (ms)
Avg. Time per Output Token (ms)
Avg. Time to First Tokan (ms)
Avg. Time per Output Token (ms)
Nor. Lebntyy ts
1.
1.0
1.0
1.0
VisualWeblnsten
0.5
0.3-
0.5-
0.5-
0.0
0.0
0.0
0.0
0
6
12
18
24
6
12
18
24
12
18
24
30
12
18
24
30
Noam. Labeerry (xhtlee)
SharwOPT
12 Request Rabe (OPS)
38
12 Requeett Rate (GPS)
12 Request Rate (QPS)
Request Rabe (OPS)
Figure 5: The average input and output latency of ElastiCMM and baseline MLLM serving systems with the VL- and Llama3.2 2- Vision- 11B under two real-world workloads. ElasticMM consistently demonstrates the lowest latency across all cases.
Metrics. We evaluate service quality using latency and throughput metrics. For each request rate, measure the normalized input latency (i.e., average prefill time divided by input length) and normalized output latency (i.e., average decoding time divided by output length). To compare system throughput under consistent conditions, throughput achievable within that target. Following prior work [40, 36], we set the SLO to 10x the latency under light load and then scale it with constant factor (ranging from to five) to evaluate performance under both relaxed and strict conditions.
define uniform SLO and record the maximum
4.2 End-to-End Performance
Input Latency. In our evaluation on two open-source large multimodal models, we compare ElasticMM against two baselines using two real-world workloads, focusing on input and output latency metrics. Fig. 5 presents the performance results, where higher request rates correspond to heavier workloads, and lower latency are better. ElasticMM benefits from its decoupled design, the prefill stage proceeds without interference from image encoding. As a result, ElasticMM significantly reduces input token latency, i.e., time-to-first-toeen (TTFT). Although the DistServe shows some improvements, its static resource allocation cannot dynamically adjust under unbalanced loads, leading to underutilized resources and a rapid increase latency higher request rates. Fig. demonstrates that ElasticMM consistently achieves the lowest input latency across all load levels. On the ShareGPT-40 datasct, ElasticMM reduces TTFT by up to 4.2 x and 3.5x for Qwen2.5-VL (encoder-decoder), respectively. On VisualWebInstruet. TTFT
(decoder-only) and LLaMA3.2 reduced by up to 3.7x and x.9x. The more substantial gains on decoder-only models can be attributed to their heavier prefill computation, which exacerbates conflicts with image encoding. Moreover, the stronger performance gains on the more visually intensive ShareGPT-44o dataset further validate the effectiveness of our multimodal inference optimizations.
Output Latency. Since ElasticMM decouples computation stages and elastically allocates them across separate instances, the decoding phase is well-isolated from the interference of encoding or prefill stages. This results in consistently lower output latency, outperforming all baselines. In comparison, vLLM executes all stages the same instance, leading severe resource contention as request rates increase, where interference from the encode and prefill stages severely degrades the performance of the decode stage and results in marked increase in output latency. While DistServe moderately alleviates latency by separating the prefill and decode stages, its static resource allocation causes memory contention decode nodes at higher request rates, leading to significant increase in output latency.
Throughput. Fig. 6 further evaluates the maximum throughput under linearly scaled service-level objectives (SLOs), ranging from x to 5x, simulating both strict and relaxed service conditions. ElasticMM achieves the highest throughput across all SLO settings. Specifically, the ShareGPT-
8

---

Qwen2.5-VL (DecOnly)
Lia ma3 ma3.2-Vision (Enc-Dec)
1500
DistServe
ElasticMM
4.0x
Throoghpus
5X
3.2x
3.1x
1.0
2.0
3,0
4.0
5.0
1.0
2.0
3,0
4.0
5.0
SLO Scales
SL. Scales
Figure 6: Maximum throughput meeting SLO.
BastlMM (2.)
We ENP
15.2)
VLLK - ElsstchM EMP --- BosticMM UniCache -e- BlasticMM
Qwee- 255 (DecOnly)
Llamad,2 Mision IEnc Dex)
Owen2,5-VL 5L (DocOnly)
Uama 5, Vision (ênc Dec)
10:0
Miyecd
Norm 115- Inss
3.5
2.0
53
2.0
2.0
4.5
3.0
SLO Scates
SLO
Request Rate (QPS)
Request Rate (QPS)
Figure 7: Throughput impact of resource allocation Figure 8: Ablation studies of our optimizations.
4o dataset, it delivers up to 4.5x x and 3.2x higher throughput than vLLM on Qwen2.5- VL and LLaMA3.2- 1--.A3..-Viiion...m bicoon-dmmosstratiiig the overall effectiveness of ElasticMM's decoupled inference architecture and multimodal optimizations. Furthermore, ElasticMM achieves up to 2.3x higher throughput than DistServe, benefiting from its hierarchical elastic scheduling strategy and non- blocking encoding design.
4.3 Ablation Study
Effectiveness of Elastic Multimodal Parallelism. To evaluate the effectiveness of Elastic Multi- modal Parallelism (EMP), we conduct an ablation study on the achievable P90 effective throughput under varying scaled SLOs. The baselines include three static resource allocation strategies: (1) text-dominant policy, (2) an equal allocation policy, and (3) a multimodal-dommnant policy. These represent ElasticMM variants without EMP enabled, with resource ratios defined by the number of multimodal versus text instances. To better demonstrate EMP's strength under heavy load, we use the ShareGPT-40 dataset, which contains images with higher average resolution. The results shown in Fig. 7. As illustrated, static resource allocation leads to suboptimal performance-even when equipped with the two key multimodal inference optimizations. Whether biased toward a specific modality group evenly distributed, static schemes cannot efficiently handle dynamically shifting inference workloads. In contrast, ElasticMM dynamically adjusts inter-group resource allocation via load balancing and applies elastic partitioned scheduling to enable line-grinned parallelism across different inference stages. As result, ElasticMM achieves substantial throughput improvements on both representative models: 1.8x for Qwen-2.5-VL. and 2.3 for Llama-3. 2- Vision.
Effectiveness of MILM Inference Optimizations. To evaluate the effectiveness of the opti- mization Sccniques-Unnffee Multimodal Prefix Cache and Non-blocking Encoding-in reducing input token latency (i.e., TTFT), we conduct an ablation study on top of ElasticMM with EMP enabled. We incrementally apply the two optimizations to assess their individual and combined benefits. The baselines include: the system without either optimization (ElasticMM-EMP)) the system with only Unified Multimodal Prefix Cache system (ElasticMM). To demonstrate the robustness of these optimizations, we generate requests by sampling from mixed dataset composed of two distinct sources. As shown in Fig. 8, applying EMP alone provides limited improvements in token input latency. The Unified Multimodal Prefix Cache significantly reduces redundant computation and data transfer in both the vision and language models, thus effectively lowering latency. The Non-blocking Encoding technique eliminates the blocking effect between the image encoding stage and subsequent inference stages, enabling a more efficient inference pipeline and leading to further latency reduction. The normalized token input latency results confirm that both optimizations provide consistent performance gains across most requests.
MiniiMMMMUnnCacee and the fully optimized

---

5
Related Work
MILMM Serving Optimizations. Recent work on MLLM serving focuses on improving infer- ence computation and memory efficiency. For example, MobileVI.M [44], TinyGPT-V [45], and TinyLLaaV [46] reduce the backbone model size to improve speed. Other approaches, such as Dynamic-LLaVa [47], VA-MA-Mini [48], and VTM [49], introduce visual token pruning or com- pression to reduce context length. InMMLLM [50] and Elastic Cache [51] reduce memory usage by pruning key-value (KV) caches based on token importance. These methods operate at the model level, trading off accuracy for computational efficiency, which can lead to degraded performance on vision tasks and may be constrained by specific model architectures. In contrast, our approach introduces no accuracy degradation, and is compatible with all MLLM architectures. Therefore, we do not compare against these optimization methods in this work. Recent work [22, 52] attempts to separate imultimodal model inference stages, but both types of requests still remain batched together during L.M backend inference, leading to persistent efficiency issues with Recoder-Fnnoode archi- tectures. In contrast, the key distinction of our work is the introduction of modality group isolation alongside stage separation, creating a two-tier separation architecture, upon which we develop an elastic scheduling mechanism across modality groups.
LLM Serving Optimizations. Recent work optimizing general LLM serving has explored disaggregated architectures, including Splitwise [24], DistServe [23], and Mooncake [53], but their static parallelism and partitioning strategies lack the flexibility to handle dynamic workloads. Other studies have focused on improving GPU operator efficiency, such as FlashAttention [54] and Flash- Decoding [55]. These methods orthogonal to ElasticMM and can be integrated to further enhance the performance of its LLM backend. Additional optimizations target KV cache management [16), request scheduling [56-59], and batching efficiency [33, 25, 34]. FlexPipe [60] introduces a dynamic pipeline parallelism framework to improve training efficiency. While these systems achieve strong performance in traditional LLM serving, they rely on tightly coupled architectures that do not meet the unique requirements of MLLM workloads. Therefore, we implement ElasticMM on top of the SOTA LLM serving system vLLM, without incorporating these additional optimizations.
6 Conclusion
In this paper, we first analyze the limitations of existing systems, which exhibit tightly coupled designs when handling serving MLLMs, making them inefficient under multimodal workloads. We propose two key insights: an efficient MLLM serving system must be both decoupled and elastic. Based on these principles, we introduce Elastic Multimodal Parallelism, a novel serving paradigm implemented in our system ElasticMM. ElasticMM incorporates three key innovations: (1) modality-aware load balancing. (2) elastic partition scheduling, and (3) multimodal inference optimizations, enabling dynamic resource adaptation across different request types and inference stages. Comprehensive evaluations on diverse real-world datasets show that FlasticMM achieves up to 4.2x reduction in TTFT and 3.2-4.5x higher throughput compared to vLLM while consistently meeting SL.Os, establishing a new paradigm for scalable multimodal AI service infrastructure.
In future work, we plan to extend our evaluation to large-scale, multi-node clusters. This setting introduces additional challenges, including inter-node communication latency and significantly broader search space for parallelism strategies. We leave the exploration of these challenges to future work, as we believe they open up important avenues for advancing scalable and efficient multimodal serving systems.
Acknowledgments
We thank the anonymous reviewers for their insightful comments and feedback. This work was supported in part by the National Key Research and Development Program of China (Grant No. 2025YFB30033000), the National Natural Science Foundation of China (Grant Nos. 62032023 and T2 T2125013), and the Innovation Funding of ICT, CAS (Grant No. E461050). Yang You's 's research group is being sponsored by NUS startup grant (Presidential Young Professorship), Singapore MOE Tier-1 grant, ARCTIC grant.
10

---

References
[1] J. Chen, H. Guo, K. Yi, B. Li, and M. Elhoseiny, "Visualgpt: Data-efficient adaptation of pretrained language models for image captioning," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 18 030018044 040.
[2] J. Wu, W. Gan, Z. Chen, S. Wan, and P. S. Yu, "Multimodal large language models; A survey," in 2023 IEEE International Conference on Big Data (BigData). IEEE, 2023, pp. 2247-2256.
[3] Z. Liang, Y. Xu, Y. Hong, P. Shang. Q Wang, Fu, and K. Liu, "A survey of multimodel large language models," Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, 2024, pp. 405-409.
[4] Y. Hu, H. Hua, Z. Yang. Shi, N. A. Smith, and J. Luo, "Promptcap: Prompt-guided image captioning for vqa with gpt-3," in Proceedings of the IEEE/CVF International Conference Computer Vision, 2023, pp. 2963-2975.
[5] R. Mokady, A. Hertz, and A. H. Bermano, "Clipeap: Clip prefix for image captioning." " arXiv preprint 11.092, 2021.
[6] D. Schwenk, A. Khandelwal, C. Clark, K. Marino, and R. Mottaghi, "A-okvqa: A benchmark for visual question answering using world knowledge," in European conference on computer vision. Springer, 2022, pp. 142-162.
[7] Z. Chen, J. Wu, W. Wang. W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. La al., "Internvl: Scaling up vision foundation models and aligning for generic visual- Sinuu--in tasks," " Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 24 185- 24 198.
[8] B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang, Y. Li, Z. Liu et al., "Llava- onevision: Easy visual task transfer," arXiv preprint arXiv:2408.03322 2024.
[9] G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan. S. Wang al., "Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context," arXiv preprint arXiv:2403.05530 2024.
[10] A. Vaswani. N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention is all you need," Advances in neurul information processing systems, vol. 30, 2017.
[11] H. Touvron, T. Lavril, G. Izacard, X. Martinet, M-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar al., "Llama: Open and efficient foundation language models," arXiv preprint 23Xiv: 13971, 2023.
[12] L. Floridi and M. Chiriatti, "Gpt-3: Its nature, scope. limits, and consequences." Minds and Machines, vol. 30, pp. 681-694, 2020.
[13] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat al., "Gpt-4 technical report," arXiv preprint urXiv:2303.08774 2023.
[14] Bai, S. Bai. Chu, Z. Cui. K. Dang, X. Deng, Y. Fan, W. Ge, Han, F. Huang et al. "Qwen technical report," arXiv preprint arXiv:2309. /6609, 2023.
[15] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz et al., "Transformers: State-of-the-ar natural language processing," in Proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, 2020, pp. 38-45.
[16] W. Kwon, 7. Li, S. Zhuang, Y. Sheng, I. Zheng, C. H. Yu, Gonzalez, H. Zhang, and I. Stoica, "Efficient memory management for large language model serving with pagedattention," in Proceedings of the 29th Symposium on Operating Systems Principles, 2023, pp. 61 1-162.
[17] R. Y. Aminabadi, S. Rajbhandari, A. A. Awan, C. Li, D. Li, E. Zheng, O. Ruwase, S. Smith, M. Zhang J. Rasley et al., DDeapspeed-infereenee enabling efficient inference of transformer models at unprecedented scale," SC22: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2022, pp. 1-15.
[18] L. Zheng, L. Yin, Z. Xie, C. L. Sun, J. Huang, C. H. Yu, S. Cao, C. Kozyrakis, I. Stoica, J. E. Gonzalez et al., "Sglang: Efficient execution of structured language model programs," Advances in Neural Information Processing Systems, vol. 37, pp. 62 557-622 583, 2024.

---

[19] J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican. M. Reynolds al., "Flamingo: a visual language model for few-shot learning," " Advances neural information processing systems, vol. 35, pp. 23 23716-2377 736, 2022.
[20] W. Dai, N. Lee, B. Wang. Z. Yang. Z. Liu, Barker, T. Rintamaki, M. Shoeybi. B. Catanzaro, and W. Ping. "Nylm: Open frontier-clas class multimodal llss," arXiv preprint arXiv:2409. 11402, 2024.
[21] J. Chi, U. Karn, H. Zhan, E. Smith, J. Rando, Y. Zhang, K. Plawiak, Z. D. Coudert, K. Upasani, and M. Pasupuleti, "Llama guard vision: Safeguarding human-ai image understanding conversations," arXiv preprint arXiv:2411. 10414, 2024.
[22] H. Qiu, A. Biswas, Z. Zhao, Mohan, A. Khare, E. Choukse, f. Goiri, Z. Zhang, H. Shen, C. Bansal al., "Modserve: Scalable and resource-efficient large multimodal model serving."
[23] Y. Zhong, S. Liu, J. Chen, J. Hu, Y. Zhu, X. Liu, X. Jin, and H. Zhang, "(DistServe): Disaggregating prefill and dccoding for goodput-optimizze large language model serving," in 18th USENIX Symposian on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 193-210.
[24] P. Patel, E. Choukse, C. Zhang, Shah, f. Goiri, Maleki, and R. Bianchini, "Splitwise: Efficient generative Ilm inference using phase splitting." in 2024 ACM/IEEE 51st Annual International Symposium on Computer Architecture (ISCA). IEEE, 2024, pp.
[25] A. Agrawal, A. Panwar, J. Mohan. N. Kwatra, B. S. Gulavani, and R. Ramjee, "Sarathi: Efficient Ilm inference by piggybacking decodes with chunked prefills." arXiv preprint urXiv:2308.163300 2023.
[26] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark et al., "Learning transferable visual models from natural language supervision." in International conference on machine learning. PmLR, 2021, pp. 8748 8763.
[27] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly et al. "An image is worth 16x16 words: Transformers for image recognition at scale," " arXiv preprint wxXiv:2010.19920 2020.
[28] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, "Sigmoid loss for language image pre-training." in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 11 975 115-111 986.
[29] P. Wang, S. Bai, S. Tan, S. Wang, 7. Fan, J. Bai, K. Chen, X. Liu, J. Wang. W. Ge et al., "Qwen2-vl: Enhanc- ing vision- laiiuage model's perception of the world at any resolution," arXiv preprint arXiv:2409.12191 2024.
[30] X. Chen, Z. Wu, X. Liu, Z. Pan, W. Liu, Z. Xie, X. Yu, and C. Ruan, "Jamus-pro: Unified multimodal understanding and generation with data and model scaling," arXiv preprint 11811 2025.
[31] J. Rasley, S. Rajhhandari, O. Ruwase, and Y. Hc, "Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters," Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, 2020. pp. 3505-3506
[32] L. Contributors, "Lmdeploy: A toolkit for compressing, deploying, and serving 1lm," https://gtthub.com/ I/inell. 2023.
[33] G.-I. Yu, J. S. Jeong, G.-W. Kim, S. Kim, and B.-G. Chun, "Orea: A distributed serving system for (Transformer-Basee generative models," in 16th USENIX Symposiun on Operating Systems Design and Implementation (OSDI 22), 2022, pp. 521-538.
[34] C. Holmes, M. Tanaka, M. Wyatt, A. A. Awan, J. Rasley, S. Rajbhandari, R. Y. Aminabadi, H. Qin, A. Bakhtiari, L. Kurilenko et al., "Deepspeed-fastgen High-throughpe text generation for lims via mii and deepspeeddinffrrenee"" arXiv preprint ww.ii:2401.000 08671, 2024.
[35] Y. Jin, T. Wang, H. Lin, M. Song, P. Li, Y. Ma, Y. Shan, 7. Yuan, C. Y. Sun al., "P/d-serve: Serving disaggregated large language model scale," arXiv preprint 2rXiv:2408.081444 2024.
[36] B. Wu, S. Liu, Y. Zhong, P. Sun, X. Liu, and X. Jin, "Loongserve: Efficiently serving long-context large language models with clastic sequence parallelism," in Proceedings of the ACM SIGOPS 30th Symposium on Operating Systems Principles, 2024, pp. 640-654.
[37] H. Zhang, Y. Tang, A. Khandelwal, and I. Stoica, "(SHEPHERD): Serving (DNNs) } in the wild," in 20th USENIX Symposium on Networked Systems Design and Implementation (NSDI 23), 2023, pp. 708.
12

---

[38] Z. Li, L. Zheng. Y. Zhong, V. Liu, Y. Sheng. X. Jin, Y. Huang, Z. Chen. H. Zhang, J. E. Gonzalez et al.. "( "(AlpaServe): Statistical multiplexing with model parallelism for deep learning serving." in 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23). 2023, pp. 663-679.
[39] B. Wu, R. Zhu, Z. Zhang, P. Sun, X. Liu, and X. Jin, "[dLoRA]: Dynamically orchestrating requests and adapters for (LLM) serving." in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 911-927.
[40] B. Wu, Y. Zhong, Z. Zhang, S. Liu, F. Liu, Y. Sun, G. Huang, X. Liu, and X. Jin, "Fast distributed inference serving for large language models," arXiv preprint arXiv:2305.05920 2023.
[41] Juravsky, B. Brown, R. Ehrlich, D. Y. Fu, C. Ré, and A. Mirhoseini, "Hydragen: High- throughput Ilm inference with shared prefixes," arXiv preprint arXiv:2402.05099 2024.
[42] Y. Jia. J. Li, X. Yue, B. Li. P. Nie, K. Zou, and W. Chen, "Vistalwebinstruct: Scaling up multimodal instruction data through web search," arXiv preprint 2rXiv:2503. 10582, 2025.
[43] Z. Chen, W. Wang, H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma et al., "How far are we to gpt-4v? closing the gap commercial multimodal models with open-source suites," " Science China Information Sciences, vol. 67, no. 12, p. 220101, 2024.
[44] Q Wu, W. Xu, W. Liu, T. Tan, J. Liu, A. Li, J. Luan, B. Wang, and S. Shang, "Mobilevlm: A vision- language model for better intra-and inter-ui understanding. arXiv preprint arXiv:2409. 14818, 2024.
[45] Z. Yuan, Z. Li, W. Huang, Y. Yc, and L. Sun, "Tinygpt-v: Efficient multimodal large language model via small backbones," arXiv preprint 2r11: 2023.
[46] B. Zhou, Y. Hu, X. Weng, J. Jia, J. Luo, X. Liu, J. Wu, and L. Huang, "Tinyllava: A framework of scall-scale large multimodal models," arXiv preprint arXiv:2402. 14289, 2024.
[47] W. Huang, Z. Zhai, Y. Shen, S. Cao, F. Zhao, X. Xu, Z. Ye, Y. Hu, and S. Lin, "Dynamic-llava: Efficient multimodal large language models via dynamic vision-language context sparsification," arXiv preprint arXiv:2412.00876 2024.
[48] S. Zhang. Q Fang. Z. Yang, and Y. Feng. "Llava-mini: Efficient image and video large multimodal models with one vision token," " arXiv preprint arXiv:2501.03m 003895, 2025.
[49] Z. Lin, M. Lin, Lin, and R. Ji. "Boosting multimodal large language models with visual tokens withdrawal for rapid inference," " in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 39, no. 5, 2025, pp. 5334-5342,
[50] Z. Ning, J. Zhao, Q. Jin, W. Ding, and M. Guo, "Inf-mllm: Efficient streaming inference of multimodal large language models on single gpu," arXiv preprint arXiv:2409.090M, 2024.
[51] 7. Liu, B. Liu, J. Wang, Y. Dong, G. Chen, Y. Rao, R. Krishna, and J. Lu, "Efficient inference of vision mnstruction-folloving models with elastic cache," European Conference on Computer Vision. Springer. 2024. pp. 54-69.
[52] G. Singh, X. Wang. Y. Hu, T. Yo, L. Xing, W. Jiang, Z. Wang. X. Bai, Y. Li, Y. Xiong et al., "Efficiently serving large multimodal models using epd disaggregation," arXiv preprint arXiv:5001.05460, 2024.
[53] R. Qin, Z. Li. W. He, M. Zhang, Y. Wu, W. Zheng, and X. Xu. "Mooncake: kveache-centric disaggregated architecture for Ilm serving," arXiv preprint arXiv:2407.00077 2024.
[54] T. Dao, Fu, S. Ermon, A. Rudra, and C. Ré, "Flashattention: Fast and memory-efficient exact attention with io-awareness," Advances in neural information processing syxtems, vol. 35, pp. 16 366444466300 1559, 2022.
[55] T. Dao, "Flashattention-2: Faster attention with better parallelism and work partitioning." arXiv preprint arXiv:2.307008m 2023.
[56] B. Sun, Z. Huang, H. Zhao, W. Xiao, X. Zhang, Y. Li, and W. Lin, "Llumnix: Dynamic scheduling for large language model serving," in 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), 2024, pp. 173-191.
[57] H. Qiu. W. Mao, A. Patke, S. Cui, S. Jha, C. Wang, H. Franke, Z. T. Kalbarczyk, T. Başar, and R. K. lyer. "Efficient interactive Ilm serving with proxy model-based sequence length prediction," arXiv preprint arXiv:2404.0 085099 2024.
13

---

[58] A. Palke, D. Reddy, S. Jha, H. Qiu. C. Pinto, C. Narayanaswami, Z. Kalbarczyk, and R. Iyer. "Queue management for slo-oriented large language model serving," in Proceedings of the 2024 ACM Symposium on Cloud Computing, 2024, pp. 18-35.
[59] Q. Su. W. Zhao, X. Li, M. Andoorveedu, C. Jiang. Z. Zhu, K. Song. C. Giannoula, and G. Pekhimenko. "Seesaw: High- thuuhhu Ilm inference via model re-sharding," arXiv preprint arXiv:2503.06433 2025.
[60] H. Zhao, Q. Tian, H. Li, and Z. Chen, "(FlexPipe): Maximizing training efficiency for transformer-based models with VVariabl--enngth inputs." in 2025 USENIX Annual Technical Conference (USENIX ATC 25), 2025, pp. 143 159.
14

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification:
The abstract and introduction accurately summarize our key contributions,
methodology, and experimental results. Section 1 explicitly outlines our claims and their scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: In Section 4.1, explicitly discuss the limitations of our system, noting that the current implementation has only been tested in a single-node environment, and we identify multi-node deployment and testing larger clusters as important directions for future work.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
15

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Our experiments use multiple mainstream open-source models and datasets, making them inherently reproducible. We will soon release code for full reproduction.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: Our experiments use multiple mainstream open-source models and datasets, making them inherently reproducible. We will soon release our code for full reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
16

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
We have provided links to all open-source data and models used our experiments, and our code will be made publicly available shortly.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https: btttps://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: /nips.
gwwcc/gg///e//eeeeeeeeecee /aaeee///ooebbiiiiecc aaiooiiccyy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Core experimental settings are described in section 4.1, with additional details provided in the appendix 6.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide detailed reporting of error bars and their calculation methods in the supplementary materials.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Section 4.1 provides detailed information about the computational resources used in our experiments.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't i make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https /ttp///eeeuuppeeeeeeeeeeee https.cc/publle cc/public// http/uullc//Ehiisseeeeeeeeeeeeee csGuidelinees
Answer: [Yes]
Justification:
Our research fully complies with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: ElasticMM provides positive societal impact by introducing paradigm for multimodal large model serving and deployment, contributing to better and faster AI service implementation.
Guidelines:
18

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate point that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e-g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: This paper focuses on inference processes and does not include model training, so these risks are not applicable.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We respect the intellectual property rights of the original authors of the data and models used in this paper, citing them appropriately and using them in full compliance with their respective licenses.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
19

---

The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.839
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing or research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This paper does not involve research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https htt///eerrppp//peee cc/ hwt////oonecee///eeeeeeeeeeoeeee wtw.eenncee//2//111/// for what should or should not be described.
Appendix A: Implementation Details
We implement ElasticMM using 7,000 lines of code based Python, C++. We build upon vLLMM [16] (v0.6.6), a stateof-the-art generative model inference platform. To ensure efficient GPU-to-GPU memory transfer of KV cache, and GPU Direct RDMA.
Py PyTorch's distributed communication with the NCCL backend
The frontend of ElasticMM uses the OpenAl API format, identical to vLLM, allowing users who have previously used vLLM to send requests to ElasticMM without any modifications. ElasticMMM's Modality-leve manager and Stage-level manager are primarily implemented in Python. However, some core logic, such as Stage Allocation, is implemented loop functions. The Modality-leve manager uses Ray for communication between elastic instances, while the Stage-Level manager assigns each batch to a Python coroutine to manage multiple batches simultaneously. Similar vL.LM, when tensor parallelism is enabled, the Stage-Level manager sends information to a single rank in the elastic instance, which then broadcasts the information to other ranks using NCCL.
C++ to improve the efficiency of
For each elastic instance, ElasticMM manages the KV cache pool using PagedAttention at the granularity of single token. Referencing previous work [18], employ an LRU strategy for dynamic release in the cache pool. Specifically, each KV cache node in the prefix tree maintains a user count, and when this count drops to zero, it becomes eligible for eviction. As GPU memory quickly fills with KV cache, the eviction manager releases KV nodes based on least-recently-usee order when the cache pool reaches its limit.
Communication between elastic instances based on NCCL, using dedicated CUDA streams, with hardware support from NVLINK offering 400GB/s bandwidth. To support multiple dynamic parallel groups at the iteration level, we use NCCL group functions to merge multiple point-to-poine operations, forming collective operations in selected NCCL ranks.
Appendix B: Mathematical Proof of Inference Equivalence in Elastic Multimodal Parallelism
In this appendix, we provide formal proof that the Elastic Multimodal Parallelism (EMP) framework preserves inference equivalence with the standard inference process, ensuring that our system optimizations do not affect model accuracy.
21

---

B.1 Preliminaries and Notation
We begin by formalizing the inference process Multimodal Large Language Models (MLLMs).
Let us define:
.M A multimodal large language model
I: Set of image inputs
T: Set of text inputs
. O: Set of text outputs
I > O: The inference function of model MM
For standard inference process, the computation can be decomposed into three sequential stages:
gE I + V: Encoding function that maps images visual tokens V
gp : V T + H x K: Prefill function that produces hidden states Ht and KV cache KC gn : H x - O: Decoding function that generates output tokens
B.2 Inference Equivalence Theorem
Theorem (Inference Equivalence). For any multimodal model M with inference function fM, the Elastic Multimodal Parallelism framework produces outputs identical to the standard sequential execution, iee.
fM(I, = (0(gp(ge(I),))) fM FRMP (I,T)
where TEMP represents the inference function under the EMP framework.
Proof. We prove this by examining each component of our EMP framework and demonstrating that they maintain computational equivalence.
B.3 Modality-Level Equivalence
Lemma (Modality-Level Equivalence). The separation of requests into modality groups preserves inference equivalence.
Proof. In EMP, we partition requests into text-only (Rr) and multimodal (RM) groups. For any request Rr, the computation involves only gp and go text inputs, while for E RMs involves the complete pipeline gø, gP, and gD-
For any request (i, () where E I and E T:
1. If 0 (text-only request), then fm(itt) / t)).
2. If * (multimodal request), then fM(it (0)
Our modality- aware aware load balancer ensures requests are routed appropriate groups without altering their computation path. Therefore, for any request r, the output remains identical regardless of group assignment.
B.4 Stage-Level Equivalence
Lemma 2 (Inference Stage Separation). The decoupling of encoding, prefill, and decoding stages across separate compulational resources preserves compulational equivalence.
Proof. In the standard sequential execution, a multimodal request r = (i, t) undergoes:
(i)))
EMP disaggregates this pipeline into independently scheduled stages that may execute on different hardware instances:
22

---

9 (i) (h, k) gr (. 9D h, k)
(Encoding stage instance E)
(6)
(Prefill stage on instance P) (Decode stage instance D)
(7)
(8)
For this disaggregation to preserve equivalence, must ensure:
Lossless intermediate representation: The visual tokens v generated by g must be identical when transferred from instance E' to instance P. This is guaranteed by use of deterministic scrialization and descrialization protocols with checksum verification.
2. Computational state preservation: The hidden states h and KV cache k generated by gp must be identical when transferred from instance P to instance D. Our implementation uses exact memory copying with NCCL primitives that ensure bit-level accuracy during transfers.
3. Execution determinism: Each function gE, 9P, and gD must produce identical outputs for identical inputs regardless of hardware allocation. This is ensured by:
Using deterministic CUDA operations
Fixing random seeds across all instances
Employing identical doating-poing computation settings
Therefore, by induction on the stages:
VEMP = Ustandard
(9)
(hEMP,KENS (hstandords (standard)
(10)
OEMP Ostandard
(11)
Thus, the decoupled execution preserves computational equivalence with the standard sequential exccution.
Lemma 3 (Dynamic Parallelism Invariance). Changes in the degree of parallelism within each inference stage do not affect output correctness
Proof. ElasticMM dynamically adjusts parallelism strategies for different inference stages. Let IIg, 1r. and Il represent different parallelism configurations for the encoding, prefill, and decoding stages respectively.
For the encoding stage gø executed under parallelism strategy IIg, we denote the execution as Te We must prove that:
n 9F
(1) g m (i)
(12)
for any image input i and any two valid parallelism strategies IIIs and IT2
Our implementation primarily uses data parallelism, where computation is partitioned across multiple devices and results are gathered using deterministic reduction operations. For data parallelism with devices:
Pr (i) ((ther((g g
(i.)))
(13)
where i,, i.,.... in represent partitions of input and g'z represents the computation on device j. Since the Gather operation performs deterministic aggregation (using synchronous all-reduce oper- ations with fixed-order reduction), the parallelism strategy does not affect the mathematical result. The same property holds for gp and 9n-
For encoder-decodes architectures with cross-attention mechanisms, ensure that the cross-attention patterns remain identical regardless of parallelism strategy by maintaining consistent attention mask distributions across devices.
23

---

Therefore, for any valid parallelism strategies:
9E (i) 98 (i)
It's
11
(14)
11
9p (u,t) 9p (v,t)
(15)
lbb (h, k) 9D (h,
11:
9D
(16)
Thus, the output of the entire inference process remains invariant changes in parallelism strategy.
B.5 Data Integrity During Migration
Lemma 4 (KV Cache Migration Fidelity). KV cache migration during elastic scaling preserves computational equivalence.
Proof. When ElasticMM performs instance scaling, it migrates KV cache entries k E K. between GPUs. Let K,, represent the KV cache on source instance 8 and K. represent the same cache after migration to destination instance d.
We must show that K. Ka after migration. Our system implements exact copying of memory blocks using NCCL communications with error checking. For each tensor T E K.,, the migration process performs:
Ta (T.)
(17)
Since modern GPU interconnects (NVLink) support lossless data transfer and our implementation verifies integrity through checksums, we ensure T, Ta for all tensors, thus K. = K.a-
To formalize this further, let IC (T1, .2.... Tm be the set of tensors in the KV cache. We define a migration function Kc, + K.a. For each tensor T; E K,,:
(T,) T, + G
(18)
where E; represents any potential error introduced during migration. Our implementation guarantees that:
||elloo = 0 Vi (1, m]
(19)
Therefore, (TT;) = T, for all i, ensuring K., = K.d-
After migration, the decoding process continues with the exact same KV cache state, thus producing identical outputs to the non-migrated scenario.
B.6 Non-blocking Encoding Equivalence
Lemma (Non-blocking Encoding Correctness). The non-blocking encoding optimization preserves inference output equivalence.
Proof. In standard sequential execution, the encoding process blocks further computation until visual tokens are generated. In ElasticMM's non-blocking encoding implementation, the encoding process executes asynchronously in parallel with other computations.
Let (i,t) be multimodal request. In standard exccution:
(i())(ge(i)tt)
(20)
With non-blocking encoding, we have:
g (executes asynchronously)
(21)
(h, k) = gp (waits for to be available)
(22)
o 9n (h, k)
(23)
24

---

Because our implementation ensures proper synchronization before the prefill stage accesses the encoded visual tokens, the data dependencies are preserved. Specifically, the prefill stage gp will begin execution until the encoding result gæ(i) is complete and available.
The non-blocking optimization affects only the scheduling of operations across compute resources, not the mathematical computations themselves. All data dependencies in the computational graph are preserved through synchronization barriers that ensure the prefill stage has access the complete and correctly encoded visual tokens.
Therefore, non- non-blocking encoding preserves inference equivalence while improving computational efficiency.
B.7 Unified Multimodal Prefix Caching Correctness
Lemma 6 (Prefix Cache Correctness). The unified multimodal prefix caching mechanism preserves inference equivalence.
Proof. Our unified multimodal prefix caching mechanism stores and reuses computed results for both visual encodings and KV cache prefixes. For any request r = (7, t), we maintain cache that maps inputs to their computed representations:
Cv : + (Visual token cache)
(24)
Cg V V - K. (KV cache prefix)
(25)
For cached visual tokens, we must show that:
V C = - gr(()
(26)
This is guaranteed by our deterministic image preprocessing and encoding pipeline, which ensures that identical inputs produce identical encoded outputs. We use cryptographic hashing to verify input identity.
For cached KV prefixes, we must show that:
v(u, (prenx) E T CK(U, (prett)) = 9p (v, tpretix)
partial
(27)
where ofr
partial computes KV cache entries for the prefix portion of the text.
When cache hit occurs, ElasticMM reuses the cached computations follows:
Cv(1) if is in cache gE(()
(28)
otherwise
Ck (v, tpretix) partial (v, (prefix)
if (v, tpreni)) is in cache otherwise
kppefix
(29)
Tp
(h, Rian) = 9p gn (h, kna))
remaining (v, i, lipoelis)
(30)
(31)
Since cached values are exact duplicates of what would be computed from scratch, and our cache invalidation logic ensures stale entries are never used, the output remains identical to non-cached execution for any given input.
B.8 Analysis of Numerical Stability
While the mathematical equivalence is guaranteed in theory, practical implementations may introduce minor numerical differences due floating-pon ppint operations. We now analyze these potential sources of error.
Proposition (Numerical Stability). Any numerical differences introduced by EMP are bounded and negligible.
25

---

Proof. The primary sources of potential numerical differences in EMP are:
1. Parallel computation order: When using data parallelism, the order of reduction operations could theoretically affect dloatingpooin summation due to non-associativity. However, modern frameworks use deterministic reduction algorithms that ensure consistent results regardless of partition count.
2. Tensor partitioning boundaries: In some parallel strategies, tensor partitioning might introduce different computation patterns. ElasticMM prioritizes data parallelism which preserves tensor integrity.
3. Mixed precision operations: When using mixed precision, the accumulation of partial results might vary slightly. However, these differences are typically on the order of 10 for fp16 operations, which is far below the threshold of affecting logical model outputs.
For token generation specifically, let ofwy/we be the probability of generating token we given previous tokens. The maximum variation in these probabilities due to numerical differences is bounded by:
Ap(w)/wee) <max
(32)
where Emax 10-7 for fp16 operations.
Since token selection uses argmax operations, these minute differences do not affect the final output unless two candidate tokens have probability differences smaller than emax, which statistically negligible.
B.9 Empirical Validation
To empirically verify theoretical guarantees, conducted experiment comparing outputs from standard sequential inference and EMP-based inference across 1,000 diverse prompts from our evaluation datasets.
Table 2: Output Consistency between Standard and EMP Inference
Model
Identical Outputs (%) Avg. Token Probability Diff.
Qwen2.5-VI. 7B
100%
< 10-8
Llama3. 2- Vision 1IB
100%
10-8
The outputs were bit-identical in 100% of cases, confirming that our EMP framework preserves inference equivalence in practice, including when stages are separated across different computational
resources.
B.10 Conclusion
We have formally proven that Elastic Multimodal Parallelism maintains exact inference equivalence with standard sequential execution. Our proof demonstrates that:
1. Separation into modality groups preserves computational paths
2. Decoupling of inference stages across different resources maintains output equivalence
3. Dynamic adjustment of parallelism strategies does not affect results
4. KV cache migration during clastic scaling preserves state fidelity
5. Non-blocking encoding and unified prefix caching optimizations maintain correctness
This mathematical guarantee ensures that all performance improvements reported in our experi- mental evaluation come without any sacrifice in model accuracy or output quality. ElasticMM therefore achieves superior efficiency while maintaining the exact same inference results as traditional sequential execution.
26


Reviews:
[{"review_id": "2QA70vdRYN", "reviewer": "Reviewer_kbpM", "summary": "The authors introduced ElasticMM, a multi-model LLM serving framework designed to optimize time-to-first-token (TTFT) and throughput. The authors noticed the importance of separating text-only and multi-modal requests and the need to have an elastic serving system to handle dynamic multi-modal requests. Modality-aware load balancing and elastic instance allocation and elastic auto-scaling mechanism are proposed to achieves these goals. Extensive experiments on real-world work loads validated the effectiveness of the proposed method.", "questions": "* In Figure 2, the average time per output token is shown. For the proposed approach, it increases at a very low rate as request rate increases. While for vllm, the average time per output token increases sharply. I am wondering what is the reason for the sharp increase of average time per output token when vllm is used?\n* If model parallel is needed in order to serve the model, will things change?\nFor example, if a very large vision encoder is used and a GPU only holds the vision encoder (pipeline parallel rank 0 only holds the vision encoder). Will the performance gain of the proposed method become less obvious?", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n* This paper studies an important problem, i.e., multi-modal LLM serving, which differs from text-only LLM serving but does not receive much attention. The paper is novel and well motivated. The proposed elastic instance allocation and elastic auto-scaling mechanism do work.\n* This paper is well-written and easy to follow.\n\nWeaknesses:\n* As the authors discussed in the Limitations section, the proposed method is only implemented and tested in a single-node environment. While multi-node model serving is not studied, the contribution of this work is non-trival. It would be great if how the proposed method could be extended to work in a multi-node model serving environment could be briefly discussed in a short future work section.", "was_revised": true, "final_justification": "Although MM model serving is not a very \"hot\" topic in the academia, it has high practical value. The proposed method works well and has high practical value. The authors addressed my questions, e.g., generalization to a multi-node setting. Hence, my recommendation is \"accept\".", "rebuttal": "### Weaknesses\n\n**W1:** As the authors discussed in the Limitations section, the proposed method is only implemented and tested in a single-node environment. While multi-node model serving is not studied, the contribution of this work is non-trival. It would be great if how the proposed method could be extended to work in a multi-node model serving environment could be briefly discussed in a short future work section.\n\n**A:** Thank you for the suggestion.  **We supplemented a preliminary experiment to demonstrate the capability of our proposed method in multi-node environment.** We deployed two instances of the Llama3.2-Vision 11B model on two machines connected via 200 GB/s InfiniBand, and migrated requests with varying sequence lengths between them. We measured the downtime during migration and compared it with both recomputation latency and single-step decode latency, as shown in Table 1.\n\n***Table 1.  Downtime and overhead of migration on multi-node***\n| Overhead        | 1K   | 2K   | 4K   | 8K    | 10K   |\n|-----------------|------|------|------|-------|-------|\n| **Migration**       | 21ms | 23ms | 28ms | 30ms  | 33ms  |\n| **Recompute**       | 255ms| 476ms| 812ms| 1750ms| 2136ms|\n| **Decode Latency**  | 46ms | 42ms | 41ms | 44ms  | 45ms  |\n\nThese results show that even in inter-node scenarios, the overhead of KV cache migration remains low.  It is not only significantly less than recomputation, but also lower than a single decode step. **This preliminary experiment demonstrates that ElasticMM maintains its performance benefits under common multi-node deployment settings, validating the scalability of our elastic scheduling strategy.**\n\n**As suggested, we will further add a discussion on multi-node scalability in the future work section.** Specifically, we would like to briefly discuss the **challenges and potential solutions** related to extending our system to multi-node environments.\nUnder our EMP paradigm, the architecture involves relatively frequent KV cache migrations, which may make inter-node network latency a potential bottleneck. To mitigate this, we plan to use **high-throughput communication technologies** such as InfiniBand to maximize the benefits of elastic scheduling. Additionally, the scheduler can incorporate placement constraints to reduce latency. For example, **by co-locating GPU instances handling the same inference stage on the same node.** Furthermore, techniques such as **asynchronous KV cache migration** can help minimize the impact of migration on the inference process.\n\n### Quesiton\n\n**Q1:** In Figure 2, the average time per output token is shown. For the proposed approach, it increases at a very low rate as request rate increases. While for vllm, the average time per output token increases sharply. I am wondering what is the reason for the sharp increase of average time per output token when vllm is used?\n\n**A:** The significant difference in performance between vLLM and ElasticMM stems from their distinct architectural designs and scheduling strategies. Specifically,\n\n1. vLLM adopts a **tightly coupled architecture**, where the inference pipeline stages—encode, prefill, and decode—are executed serially on the same hardware. This design leads to contention for compute resources and, more critically, causes the time-consuming encode stage to block subsequent inference steps. **As a result of its coupled design, output latency increases sharply.**\n\n2. **In contrast, ElasticMM employs elastic partitioned scheduling, which dynamically adjusts resource allocation between the prefill and decode stages.** When the decode stage becomes resource-constrained due to large batch sizes, ElasticMM reallocates additional instances to this stage. Moreover, because different stages are isolated in execution, the decode stage is not interfered with by others. Consequently, as the request rate increases, output latency remains consistently low.\n\n**Q2:** If model parallel is needed in order to serve the model, will things change? For example, if a very large vision encoder is used and a GPU only holds the vision encoder (pipeline parallel rank 0 only holds the vision encoder). Will the performance gain of the proposed method become less obvious?\n\n**A:** As mentioned in your example, where the vision encoder is served on a dedicated GPU (or multiple GPUs), **the performance benefit of our method remains unaffected. In our system, we also isolate the encode stage onto separate GPU instances, even when it is relatively lightweight.** This is because the encode stage tends to be computationally intensive, and isolation helps prevent blocking the subsequent inference stages. This is a key optimizations proposed in our work. As shown in Figure 9, the ablation study confirms that this design significantly reduces input latency.\n\nHowever, larger models do introduce additional complexity to our elastic scheduling algorithm. When an LLM backend needs to be deployed across multiple GPUs using TP or other parallelization methods, it introduces some additional constraints. For example, we only be able to use an even number of GPUs to execute a inference stage. Furthermore, larger models often require multi-node deployment, which introduces new system-level challenges. Maintaining the performance gains of our elastic scheduling under large models and large-scale GPU clusters remains an important direction for our future work."}, {"review_id": "KRhBUB3BuE", "reviewer": "Reviewer_t62T", "summary": "The paper introduces Elastic Multimodal Parallelism, a new serving paradigm for efficiently serving multimodal large language models. The proposed system, ElasticMM, addresses key challenges in multimodal inference pipelines, including significant overhead due to additional model components and diverse computational requirements across different modalities. Experimental evaluations demonstrate that ElasticMM reduces TTFT by up to 4.2× and increases throughput by 3.2–4.5× compared to SOTA methods.", "questions": "Please see Weaknesses.\n\nIt should be noted that due to possible resource or permission limitations, the author may not be able to fully address the above issues. However, I hope the author can at least add relevant baselines and their comparisons. I will adjust my score based on the extent of the author's responses.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "# Strengths:\n- **Quality:** The system ElasticMM is well-designed, leveraging detailed insights about workload characteristics (burstiness, computational bottlenecks) to drive performance.\n- **Significance:** Substantial improvements demonstrated experimentally highlight the practical significance, addressing critical performance bottlenecks in multimodal LLM deployment.\n\n# Weaknesses:\n- **Lack of Baseline:** This system only compares with vLLM and self-implemented vLLM-Decouple. However, in December 2024, a work related to MLLM serving was posted on arXiv (*Efficiently Serving Large Multimodal Models Using EPD Disaggregation*), which also involves issues such as the separation of different components. I suggest that the author include this work in the baseline to verify the advancement of the system.\n- **Scalability Validation:** The paper's experiments are limited to a single-node environment, potentially constraining insights about performance in real-world large-scale deployments. Although the author stated that multi-machine experiments will be left for future work, as a serving task, I believe a single machine is not sufficient, especially since the author has also involved tech like elastic scaling that clearly require multi-machine validation. Considering that the author may have limitations in machine resources, different model, trace and machines can be used for small-scale experiments.\n- **Clarity:** Some statements in the paper are not very clear, for example, where does the workload in Fig. 2 come from? Is it an open-source trace or from the company's actual production trace? Please at least indicate the source.", "was_revised": true, "final_justification": "The author answered my questions well and also provided the corresponding experiments, so I raised my score.", "rebuttal": "### Weaknesses\n\n**W1: Lack of baseline.** This system only compares with vLLM and self-implemented vLLM-Decouple. However, in December 2024, a work related to MLLM serving was posted on arXiv (Efficiently Serving Large Multimodal Models Using EPD Disaggregation), which also involves issues such as the separation of different components. I suggest that the author include this work in the baseline to verify the advancement of the system.\n\n**A:** Thank you for your suggestion. We agree that the original baseline was limited. **We have incorporated two new baselines for a more comprehensive evaluation: EPDServe [1], as you suggested, and DistServe (OSDI 2024) [2].**\n\n* **EPDServe is a recent work** that was built on top of vLLM and introduces component separation and elastic scheduling optimizations specifically for multimodal LLMs. To ensure fairness in comparison, we have upgraded its vLLM version to match vLLM version.\n* **DistServe, another crucial baseline,** proposes prefill/decode disaggregation and is known for its high throughput on general LLM workloads. We adapted DistServe to support multimodal LLMs by integrating a vision encoder module.\n\nWe evaluated all these systems on Input Latency and Maximum throughput using Llama3.2-Vision 11B model and ShareGPT-4o dataset. The results are illustrated as follows:\n\n***Table1. Norm input latency (s/token) with different request rate***\n| System      | 2 req/sec | 4 req/sec | 6 req/sec | 8 req/sec | 10 req/sec |\n|-------------|---------|---------|---------|---------|----------|\n| vLLM        | 0.11    | 0.30    | 0.45    | 0.66    | 1.33     |\n| DistServe   | 0.08    | 0.21    | 0.39    | 0.53    | 1.24    |\n| EPDServe    | 0.04    | 0.07    | 0.23    | 0.46    | 0.67    |\n| *ElasticMM*   | 0.03    | 0.05    | 0.15    | 0.25    | 0.49  |\n\n***Table2. Maximum throughput (req/min) meeting SLO***\n| System      | 1 SLO scale    | 2 SLO scales   | 3 SLO scales   | 4  SLO scales  | 5 SLO scales   |\n|-------------|------|------|------|------|------|\n| vLLM        | 225  | 258  | 264  | 309  | 402  |\n| DistServe   | 434  | 489  | 532  | 587  | 694  |\n| EPDServe    | 597  | 628  | 731  | 845  | 963  |\n| *ElasticMM*   | 720  | 792  | 1030 | 1216 | 1640 |\n\nWe can observe that **ElasticMM significantly outperforms existing baselines by fundamentally addressing their architectural and scheduling limitations:**\n* **Compared to DistServe, ElasticMM achieves 2.6× lower input latency and 2.3× higher throughput.** DistServe adopts a decoupled architecture, but relies on static resource allocation, making it less adaptive to dynamic workloads and thus limiting throughput. In addition, it does not differentiate between text-only and multimodal requests, which further constrains efficiency under mixed-load scenarios. \n* **Compared to EPDServe, ElasticMM achieves 1.8× lower input latency and 1.4× higher throughput.** EPDServe introduces elastic scheduling on top of separated inference stages, but still does not differentiate between request types. Specifically, it mixes text-only and multimodal requests in the same batch for backend LLM inference. For encoder-decoder models such as Llama3.2-Vision, this mixed batching strategy significantly increases inference latency due to computation graph heterogeneity, especially from cross-attention layers. \n\nWe will add these results and discussions into the revised paper.\n\n**W2: Scalability Validation.** The paper's experiments are limited to a single-node environment, potentially constraining insights about performance in real-world large-scale deployments. Although the author stated that multi-machine experiments will be left for future work, as a serving task, I believe a single machine is not sufficient, especially since the author has also involved tech like elastic scaling that clearly require multi-machine validation. Considering that the author may have limitations in machine resources, different model, trace and machines can be used for small-scale experiments.\n\n**A:** Thank you for pointing this out this. To address this, **we conducted preliminary experiments to evaluate the scalability of our system in a multi-node environment below.** Under the EMP paradigm, our architecture involves relatively frequent KV cache migrations, making inter-node communication latency a potential bottleneck. To assess this, we deployed two inference instances of the LLaMA-Vision 11B model on two machines connected via 200 GB/s InfiniBand, and migrated requests with varying sequence lengths between them. We measured the downtime during migration and compared it with both recomputation latency and single-step decode latency, as shown as the below table.\n\n***Table 3.  Downtime and overhead of kv cache migration on multi-node***\n| Overhead        | 1K   | 2K   | 4K   | 8K    | 10K   |\n|-----------------|------|------|------|-------|-------|\n| **Migration**       | 21ms | 23ms | 28ms | 30ms  | 33ms  |\n| **Recompute**       | 255ms| 476ms| 812ms| 1750ms| 2136ms|\n| **Decode Latency**  | 46ms | 42ms | 41ms | 44ms  | 45ms  |\n\n**These results show that even in inter-node scenarios, the overhead of KV cache migration remains low. This is not only far less than recomputation, but also lower than a single decode step.** In contrast, the recomputation cost increases significantly with sequence length, reaching up to 64× the migration time. For instance, recomputing the KV cache of a 10k token sequence takes 2.1 seconds, equivalent to stalling 47 decode iterations.\n\n**This preliminary experiment demonstrates that ElasticMM maintains its performance benefits under common multi-node deployment settings, validating the scalability of our elastic scheduling strategy. Extending our system to a larger number of nodes will be a crucial focus of our future work.**\n\n\n**W3:** Some statements in the paper are not very clear, for example, where does the workload in Fig. 2 come from? Is it an open-source trace or from the company's actual production trace? Please at least indicate the source.\n\n**A:** The data for this figure is derived from the company's production traces, covering a 24-hour interval. We will supplement the source of this workload in the paper. \n\nReference：\n\n*[1]: Efficiently Serving Large Multimodal Models Using EPD Disaggregation. [arxiv link](https://arxiv.org/abs/2501.05460)*\n\n*[2]: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. [arxiv link](https://arxiv.org/abs/2401.09670)*"}, {"review_id": "eUhgAsxDcV", "reviewer": "Reviewer_3qBq", "summary": "The paper targets the high-latency, low-throughput pain points that arise when existing LLM serving systems are asked to handle *multimodal* workloads.  The authors observe that current pipelines run every stage—image preprocessing/encoding, LLM prefill, and decode—on the same GPU instances and batch both text-only and multimodal requests together, leading to severe resource contention and SLO violations.\n\nTo address this, they propose **Elastic Multimodal Parallelism (EMP)** and implement it in a system called **ElasticMM**.  The key ideas are:\n\n* **Modality-aware decoupling:** Route text-only and multimodal requests to separate *modality groups*, each with its own GPU pool, then let a load balancer proactively allocate—and reactively pre-empt—GPUs across groups as traffic shifts.\n* **Stage-level elasticity:** Within each group, further disaggregate the pipeline so that encoding, prefill, and decode can scale up or shrink independently via an elastic partition scheduler driven by a gain-cost model.\n* **MLLM-specific optimizations:** (i) a **unified multimodal prefix cache** that reuses both image and text prefixes, and (ii) **non-blocking encoding** that runs vision encoding asynchronously, eliminating its interference with prefill and decode.\n\nThe prototype is built atop vLLM and evaluated on an 8×A800 workstation with two representative MLLMs—LLaMA-3.2-Vision-11B (enc-dec) and Qwen-2.5-VL-7B (dec-only)—under two mixed workloads, VisualWebInstruct and ShareGPT-4o.  Compared with vLLM, ElasticMM cuts time-to-first-token by up to **4.2×** and boosts end-to-end throughput by **3.2–4.5×** while meeting service-level objectives; ablations show that each component of EMP contributes materially to these gains.", "questions": "1.Multi-node scalability.\nYour experiments are limited to a single 8 × A800 box. Could you either (a) add preliminary multi-node results or (b) provide an analytic model/trace-driven simulation that shows EMP still improves TTFT and throughput when network latency and cross-node KV migration are present?\n\n2.Baseline coverage.\nPlease justify excluding SGLang, DeepSpeed-FastGen, Splitwise, DistServe and Mooncake, or add at least one of them to Table 3. If ElasticMM still shows ≥2 × improvement under identical settings, that would materially strengthen the claim of significance.", "limitations": "Environmental impact. Running eight A800s at high utilization is energy intensive; explicit reporting or mitigation discussion is absent.\n\nSecurity / privacy risk. Unified prefix caching potentially exposes vision embeddings across requests. A short note on isolation mechanisms would be valuable.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "### Strengths\n\n1. **Robust quality and engineering rigor**\n   An end-to-end implementation atop vLLM is benchmarked on two real-world workloads and two representative MLLMs, showing up to $4.2\\times$ TTFT reduction and $3.2\\!-\\!4.5\\times$ higher throughput, with detailed ablations that isolate each component’s contribution.\n\n2. **Novel decoupled-and-elastic paradigm for MLLM serving (Originality & Significance).**\n   Elastic Multimodal Parallelism (EMP) introduces (i) modality-aware load balancing, (ii) stage-level elastic partition scheduling, plus (iii) unified multimodal prefix caching and non-blocking encoding—an architectural combination not present in prior systems.\n\n### Weaknesses\n\n1. **Single-node evaluation limits external validity**\n   All results come from one 8 × A800 workstation; multi-node or heterogeneous-cluster behavior is left to “future work,” so real-world scalability remains unproven.\n\n2. **Baseline coverage is too narrow**\n   Only vLLM and an in-house static variant are compared; other state-of-the-art disaggregated or high-throughput frameworks (SGLang, DeepSpeed-FastGen, Distserve etc.) are excluded, leaving the true performance gap uncertain. And the number of models are limited.", "was_revised": true, "final_justification": "This is a very solid paper on inference in the multimodal domain. I recommend that it be accepted.", "rebuttal": "### Question/Weakness\n\n**Q1: Multi-node scalability.** Your experiments are limited to a single 8 × A800 box. Could you either (a) add preliminary multi-node results or (b) provide an analytic model/trace-driven simulation that shows EMP still improves TTFT and throughput when network latency and cross-node KV migration are present?\n\n**A:** This is a good question regarding the scalability verification of ElasticMM.  Firstly, our core scheduling algorithm, EMP, operates at the granularity of individual GPU instances, meaning there is no fundamental difference at the algorithmic level whether it is deployed across multiple nodes. However, the primary challenge with multi-node deployment is the additional latency caused by inter-node network bandwidth.\n\n**Therefore, we added a preliminary multi-node experiment**: we deployed two instances running the Llama3.2-Vision 11B model on two machines interconnected with 200GB/s InfiniBand. We migrated requests of varying sequence lengths from one instance to another and recorded the downtime during the migration. We then compared this migration time with both recomputation time and single-decode latency.\n\n***Table 1. Downtime and overhead of KV cache migration on multi-node***\n| Overhead        | 1K   | 2K   | 4K   | 8K    | 10K   |\n|-----------------|------|------|------|-------|-------|\n| **Migration**       | 21ms | 23ms | 28ms | 30ms  | 33ms  |\n| **Recompute**       | 255ms| 476ms| 812ms| 1750ms| 2136ms|\n| **Decode Latency**  | 46ms | 42ms | 41ms | 44ms  | 45ms  |\n\n**The results show that even in inter-node scenarios, the overhead of KV cache migration remains low. This is not only far less than recomputation, but also lower than decode latency.** In contrast, the recomputation cost increases significantly with sequence length, reaching up to 64× the migration time. For instance, recomputing the KV cache of a 10k token sequence takes 2.1 seconds, equivalent to stalling 47 decode iterations. **This preliminary experiment demonstrates that ElasticMM maintains its performance benefits under common multi-node deployment settings, validating the scalability of our elastic scheduling strategy.**\n\n**Q2: Baseline coverage.** Please justify excluding SGLang, DeepSpeed-FastGen, Splitwise, DistServe and Mooncake, or add at least one of them to Table 3. If ElasticMM still shows ≥2 × improvement under identical settings, that would materially strengthen the claim of significance.\n\n**A:** Thank you for your suggestion. Our original baselines were indeed limited. **We have adopted your advice by adding a comparative experiment with DistServe** [1]. Additionally, we also include **EPDServe** [2] (ICML 2025) as another newer baseline for comparison.\n\n* **DistServe serves as a crucial baseline because it adopts prefill/decode disaggregation and is recognized for its high throughput in general LLM workloads.** To support evaluation on multimodal LLMs, we extended DistServe by adding a vision encoder module.\n* **EPDServe, a recent work, is built on top of vLLM and uniquely optimizes multimodal LLM inference by introducing component separation and elastic scheduling.** To ensure fairness in comparison, we have upgraded its vLLM version to match ours.\n\nWe evaluated all systems on Input Latency and Maximum throughput using LLaMA-Vision 11B model and ShareGPT-4o dataset. The results are as follows:\n\n***Table 2. Norm input latency (s/token) with increased request rates***\n| System      | 2 req/s | 4 req/s | 6 req/s | 8 req/s | 10 req/s |\n|-------------|---------|---------|---------|---------|----------|\n| vLLM        | 0.11    | 0.30    | 0.45    | 0.66    | 1.33     |\n| DistServe   | 0.08    | 0.21    | 0.39    | 0.53    | 1.24    |\n| EPDServe    | 0.04    | 0.07    | 0.23    | 0.46    | 0.67    |\n| *ElasticMM*   | 0.03    | 0.05    | 0.15    | 0.25    | 0.49   |\n\n***Table 3. Maximun throughput (req/min) meeting SLO scales***\n| System      | 1 SLO scale   | 2 SLO scales    | 3 SLO scales    | 4 SLO scales    | 5 SLO scales    |\n|-------------|------|------|------|------|------|\n| vLLM        | 225  | 258  | 264  | 309  | 402  |\n| DistServe   | 434  | 489  | 532  | 587  | 694  |\n| EPDServe    | 597  | 628  | 731  | 845  | 963  |\n| *ElasticMM*   | 720  | 792  | 1030 | 1216 | 1640 |\n\n**ElasticMM still outperforms the new baselines by addressing their architectural and scheduling limitations:**\n* **Compared to DistServe, ElasticMM achieves 2.6× lower input latency and 2.3× higher throughput.** DistServe adopts a decoupled architecture, but relies on static resource allocation, making it less adaptive to dynamic workloads and thus limiting throughput. Furthermore, it does not differentiate between text-only and multimodal requests, which further constrains efficiency under mixed-load scenarios. \n* **Compared to EPDServe, ElasticMM achieves 1.8× lower input latency and 1.4× higher throughput.** EPDServe introduces elastic scheduling on top of separated inference stages, but still does not differentiate between request types. Specifically, it mixes text-only and multimodal requests in the same batch for backend LLM inference. For encoder-decoder models such as LLaMA-Vision, this mixed batching strategy significantly increases inference latency due to computation graph heterogeneity, especially from cross-attention layers. \n\nThese results clearly demonstrate the superiority of our proposed method. We will include these results and the corresponding discussion in the revised paper.\n\nReference：\n\n*[1]: DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. [arxiv link](https://arxiv.org/abs/2401.09670)*\n\n*[2]: Efficiently Serving Large Multimodal Models Using EPD Disaggregation. [arxiv link](https://arxiv.org/abs/2501.05460)*"}, {"review_id": "CMXByzT7q2", "reviewer": "Reviewer_Cb71", "summary": "This paper proposes a serving framework for multimodal large language models (MLLMs) that elastically adapts to resource heterogeneity across different request types and inference stages. The proposed framework, ElasticMM, enables dynamic resource allocation via a modality-aware load balancer, parallelism adjustment and adaptive scaling through elastic partition scheduling, and inference optimization using unified multimodal prefix caching and non-blocking encoding. By evaluating the framework on the VisualWebInstruct and ShareGPT-4o datasets with LLaMA3.2-Vision-11B and Qwen2.5-VL-7B models, the authors demonstrate significant performance improvements, achieving a 4.2× reduction in time-to-first-token and approximately 4× higher throughput.", "questions": "- (Figure 1) What datasets are used in subfigures (b) and (c)?\n- (Figure 2) Why does the image-text model exhibit dynamic request distribution?\n- (Figure 4) What do the dashed lines within the rectangles signify? What does the GPU icon specifically represent? How could existing methods be visualized within this figure for comparison?\n- Would it be possible to conduct ablation studies for each method proposed in Sections 3.1 and 3.2?", "limitations": "The authors briefly mention future work in a single sentence. A more in-depth discussion on current limitations and potential future directions would enhance the clarity.", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strength \n- The paper achieves impressive improvements in inference efficiency, substantially reducing latency and enhancing throughput for MLLMs.\n- The paper provides a comprehensive review of existing literature, clearly outlining the motivation and necessity for the proposed approach.\n- The method is thoroughly analyzed, with detailed ablation studies validating the contributions of each component.\n\n\nWeakness\n- Some figures lack clear explanations, making it difficult to fully understand the presented concepts (see detailed questions below).\n- The architectures of the target inference models are described only in text, which makes it challenging to grasp the inference pipeline and model architecture. An overview diagram would significantly aid comprehension of Lines 72–94.\n- The evaluation focuses only on 7B and 11B models. It would be valuable to assess performance on larger model sizes for a more comprehensive understanding of scalability.\n\nMinor\n- In Figure 4, the letter \"s\" in “Images” appears to have a smaller font size.\n- Section 3.3 (Inference Optimization) may not logically belong under Section 3 (Elastic Multimodal Parallelism). I recommend either renaming Section 3 or moving Section 3.3 to a new standalone section.", "was_revised": true, "final_justification": "The paper addresses an important and useful topic. However, the current manuscript lacks comprehensive diagrams that would facilitate an overall understanding of the proposed approach. Additionally, practical serving experiments using real-world large-scale models are insufficiently demonstrated. For these reasons, I assigned a \"weak accept\". Nevertheless, I would like to note that I am not an expert in LLM serving.", "rebuttal": "### Weakness\n\n**W1:** The architectures of the target inference models are described only in text, which makes it challenging to grasp the inference pipeline and model architecture. An overview diagram would significantly aid comprehension of Lines 72–94.\n\n**A:** Thank you for your great suggestion. Adding an architectural diagram of the MLLM inference pipeline would indeed help readers better understand the processing workflow. We will include this figure in the revised version.\n\n**W2:** The evaluation focuses only on 7B and 11B models. It would be valuable to assess performance on larger model sizes for a more comprehensive understanding of scalability.\n\n**A:** Supporting larger models brings our system closer to real-world deployment scenarios and provides a stronger test of its scalability. Serving such models requires substantially more GPU resources. For example, a 70B model typically needs two or even four GPUs for inference. In our decoupled-stage inference design, effectively leveraging ElasticMM’s elastic scheduling for such models necessitates at least two nodes. **Therefore, deploying ElasticMM on larger clusters to support larger models is a key direction for our future work.**\n\n**However, to preliminarily validate the scalability of our system, we conducted a preliminary multi-node experiment.** we deployed two instances running the Llama3.2-Vision 11B model on two machines interconnected with 200GB/s InfiniBand. We migrated requests of varying sequence lengths from one instance to another and recorded the downtime during the migration, as shown in the below table.\n\n\n***Table 1. Downtime and overhead of KV cache migration on multi-node***\n| Overhead        | 1K   | 2K   | 4K   | 8K    | 10K   |\n|-----------------|------|------|------|-------|-------|\n| **Migration**       | 21ms | 23ms | 28ms | 30ms  | 33ms  |\n| **Recompute**       | 255ms| 476ms| 812ms| 1750ms| 2136ms|\n| **Decode Latency**  | 46ms | 42ms | 41ms | 44ms  | 45ms  |\n\nThe results show that even in inter-node scenarios, the overhead of KV cache migration remains low. This is not only far less than recomputation, but also lower than decode latency. **This preliminary experiment demonstrates that ElasticMM maintains its performance benefits under common multi-node deployment settings, validating the scalability of our elastic scheduling strategy.** \n\n**Minor:**\n* In Figure 4, the letter \"s\" in “Images” appears to have a smaller font size.\n* Section 3.3 (Inference Optimization) may not logically belong under Section 3 (Elastic Multimodal Parallelism). I recommend either renaming Section 3 or moving Section 3.3 to a new standalone section.\n\n**A:** Thank you for pointing this out. We acknowledge the inconsistency in the font in Figure 4 and will correct it in the revised version. Additionally, we agree that Section 3.3 does not logically align well with the rest of Section 3. To improve the structural  clarity of the paper, we will reorganize the content and move it into a separate section in the revision.\n\n### Question\n\n**Q1:** (Figure 1) What datasets are used in subfigures (b) and (c)?\n\n**A:** The data shown in the figures are sourced from the publicly available ShareGPT-4o dataset. ([dataset link](https://huggingface.co/datasets/OpenGVLab/ShareGPT-4o)) We will make sure to clearly indicate this in the revised version.\n\n**Q2:** (Figure 2) Why does the image-text model exhibit dynamic request distribution?\n\n**A:** This observation is based on traces from real-world production workloads. Today, many large model service providers use unified multimodal models to handle both text and multimodal requests. In practical user sessions, users often begin with pure text input and gradually introduce multimodal content such as images or PDFs, frequently followed by additional textual queries or descriptions. As a result, text-only requests tend to be more stable over time, whereas multimodal requests exhibit much greater temporal variability. This variability includes both clear periodic patterns (e.g., day-night cycles) and short-term spikes. Together, these behavior patterns lead to a highly dynamic request distribution over time for vision-language models.\n\n**Q3:** (Figure 4) What do the dashed lines within the rectangles signify? What does the GPU icon specifically represent? How could existing methods be visualized within this figure for comparison?\n\n**A:** In Figure 4, the dashed lines within each stage represent **relative batch sizes.** The decode stage shows more dashed lines than the prefill stage, indicating a larger batch size. This is because decode is primarily memory-bound, and increasing its batch size moderately does not significantly affect latency but improves overall throughput. Each GPU icon represents a single GPU instance. We agree that adding a comparison with existing methods would better highlight system differences. **Due to a new policy introduced by the committee, we are currently  unable to provide additional figures. However, we will include such a comparison diagram in the revised version of the paper.**\n\n**Q4:** Would it be possible to conduct ablation studies for each method proposed in Sections 3.1 and 3.2?\n\n**A:** It is difficult to conduct separate, standard ablation studies for the methods proposed in Sections 3.1 and 3.2, because **these methods are designed to function as a hierarchical and cooperative scheduling framework, rather than as independent components.** After resource adjustment across modality groups via the strategies in Section 3.1, intra-group resources must be further scheduled dynamically using the methods in Section 3.2. These two components are tightly coupled—for instance, intra-group resource scarcity may trigger inter-group migration or preemption of GPU instances. Therefore, the two techniques are executed jointly and influence each other during runtime, making it difficult to perform standard ablation studies by simply \"turning off\" one of them in isolation."}]

Author rebuttal:

Meta-review: In this paper, the authors introduce ElasticMM, which is a framework for serving multimodal models, to optimize for time-to-first-token (TTFT) and throughput. There are diverse computational requirements across different modalities resulting in several high-latency, low-throughput pain points. The authors observe that current pipelines run every stage—image preprocessing/encoding, LLM prefill, and decode—on the same GPU instances and batch both text-only and multimodal requests together, leading to severe resource contention and SLO violations. 

To address this, they introduce Elastic Multimodal Parallelism (EMP). Reviewer 3qBq has a nice summary of the main workings i.e.

"The key ideas are:
* Modality-aware decoupling: Route text-only and multimodal requests to separate modality groups, each with its own GPU pool, then let a load balancer proactively allocate—and reactively pre-empt—GPUs across groups as traffic shifts.
* Stage-level elasticity: Within each group, further disaggregate the pipeline so that encoding, prefill, and decode can scale up or shrink independently via an elastic partition scheduler driven by a gain-cost model.
* MLLM-specific optimizations: (i) a unified multimodal prefix cache that reuses both image and text prefixes, and (ii) non-blocking encoding that runs vision encoding asynchronously, eliminating its interference with prefill and decode.

The prototype is built atop vLLM and evaluated on an 8×A800 workstation with two representative MLLMs—LLaMA-3.2-Vision-11B (enc-dec) and Qwen-2.5-VL-7B (dec-only)—under two mixed workloads, VisualWebInstruct and ShareGPT-4o. Compared with vLLM, ElasticMM cuts time-to-first-token by up to 4.2× and boosts end-to-end throughput by 3.2–4.5× while meeting service-level objectives; ablations show that each component of EMP contributes materially to these gains."

The authors demonstrate results on real-world work loads.

Strengths

1. All reviewers think the paper is solidly executed. The proposed elastic instance allocation and elastic auto-scaling mechanism do work. well-designed, leveraging detailed insights about workload characteristics (burstiness, computational bottlenecks) to drive performance. Substantial improvements demonstrated experimentally highlight the practical significance, addressing critical performance bottlenecks in multimodal LLM deployment. The method is thoroughly analyzed, with detailed ablation studies validating the contributions of each component.
2. All reviewers also find the contributions extremely novel and also effective. They also appreciate the high quality of the manuscript that provides a comprehensive review of existing literature, clearly outlining the motivation and necessity for the proposed approach.

Weaknesses

- For this paper, all weaknesses have been favorably addressed by the authors in the rebuttal: For example:
    - Adding more recent extra baselines of EPDServe and DistServe
    - Showing initial results on multi-node deployments, which was earlier left to future work

All 4 reviewers have chosen to accept this paper. The paper represents a solid contribution to the area of serving multimodal LLMs, with all reviewer concerns addressed.
Decision: Accept (oral)