Efficiently Serving Large Multimodal Models Using EPD Disaggregation
Authors: Gursimran Singh
Xinglu Wang
Yifan Hu
Timothy Tin Long Yu
Linzi Xing
Wei Jiang
Zhefeng Wang
Bai Xiaolong
Yi Li
Ying Xiong
Yong Zhang
Zhenan Fan
Venue: ICML 2025 poster (2025)
Keywords: Efficient Inference
Large Multimodal Model Inference
Disaggregated Inference
Primary area: optimization->large_scale_parallel_and_distributed
TL;DR: An efficient inference system for serving large multimodal systems using disaggregation.

Abstract: Large Multimodal Models (LMMs) extend Large Language Models (LLMs) by handling diverse inputs such as images, audio, and video, but at the cost of adding a multimodal encoding stage that increases both computational and memory overhead. This step negatively affects key Service Level Objectives (SLOs), such as time to first token (TTFT) and time per output token (TPOT). We introduce Encode-Prefill-Decode (EPD) Disaggregation, a novel framework that separates the encoding, prefill, and decode stages onto dedicated resources. Unlike current systems, which bundle encoding and prefill together, our approach decouples these steps, unlocking new opportunities and optimizations. These include a mechanism to cache multimedia tokens for efficient transfer, a novel way to parallelize the encoding load within a request, a module for optimal resource allocation for disaggregated serving, and a novel role-switching method to handle changing workload characteristics. Experimental evaluations with popular LMMs show substantial gains in memory efficiency (up to 15× lower peak memory utilization), batch sizes (up to 22× larger), 10× more images per request, and 2.2× larger KV caches. Furthermore, it leads to significant improvements in SLO attainment (up to 90–100% improvement) and TTFT (up to 71% reduction), compared to systems that do not disaggregate. The code is available at [https://github.com/vbdi/epdserve](https://github.com/vbdi/epdserve).

Efficiently Serving Large Multimodal Models Using EPD Disaggregation
Gursimran Singh'
Xinglu Wang? Yifan Hu'
Timothy Yu' Linzi Xing'
Wei Jiang" Zhenan Fan
Zhefeng Wang'
Xiaolong Bai Yi
Ying Xiong'
Yong Zhang'
Abstract
Aggregated
LLM
LLM
Large Multimodal Models (LMMs) extend Large Language Models (LLMs) by handling diverse inputs such images, audio, and video, but the cost of adding multimodal encoding stage that increases both computational and memory overhead. This step negatively affects key Service Level Objectives (SLOs), such as time to first to- ken (TTFT) and time per output token (TPOT). We introduce Encode-Prefill- Decooee (EPD) Dis- aggregation, a novel framework that separates the encoding, prefill, and decode stages onto dedi- cated resources. Unlike current systems, which bundle encoding and prefill together, our approach decouples these steps, unlocking new opportuni- ties and optimizations. These include a mech- anism to cache multimedia tokens for efficient transfer, novel way to parallelize the encod- ing load within request, module for opti- mal resource allocation for disaggregated serv- ing, and novel role-switching method to handle changing workload characteristics. Experimen- tal evaluations with popular LMMs show sub- stantial gains in memory efficiency (up to 15x lower peak memory utilization), batch sizes (up 22x larger), 10x more images per request, and 2.2x larger KV caches. Furthermore, it leads to significant improvements in SLO attain- ment (up to 90-1009 improvement) and TTFT (up to 71% reduction), compared to systems that do not disaggregate. The code is available at http:///gghub..coo/bbbbbeeepeppp
LLM
LLM
LUM
Disaggregated
E
LUM!
LLM
3
LLMP
LLM
LLMP
Figure Aggregated (top) vs. disaggregated (bottom) sys- tem architectures. In the aggregated setup, the encoder (E) and LLM share the same GPUs. leading interference be- tween encode and prefill stages (e.g., L.MM-4 delays E5). Disaggregation isolates these stages across GPUs, reduc- ing contention and improving utilization. This separation enables better performance under multimodal workloads, addressing key limitations of existing systems.
1. Introduction
Large Language Models (LLMs) have revolutionized lan- guage understanding and reasoning, achieving superhuman performance on a variety of tasks (Achiam et al., 2023; Chang et al., 2024). Recently, the scope of these models has expanded to include multiple modalities, such as images, audio, and videos, leading to the emergence of Large Multi- modal Models (LMMs) (Yao et al., 2024; Liu al., 2023; interact with
Chen al., 2024). LMMs enable diverse data types, such posing questions about visual scenes analyzing audio clips, thereby unlocking novel ap- plications across fields like healthcare, autonomous systems, and creative industries (Luo et al., 2025).
However, serving LMMs an efficient manner presents unique challenges. Meeting strict Service Level Objectives (SLOs), such time first token (TTFT) and time per out- put token (TPOT), becomes increasingly difficult given the added computational and memory demands of processing multimodal data (Alvar et al., 2025; Liu et al., 2024). Unlike LLMs, where inference involves prefill and decoding stages, LMMs require an additional encoding stage to process raw multimodal inputs (e.g., images or videos) into tokenized
'Huawei Technologies Canada, BC, Canada SSion Fraser University. BC, Canada 'Huawei Cloud, China. Correspondence to: Zhenan Fan zhenanfan gww.l.ccm......... Gursirman Singh <gursimran.singg huawei.com>>
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Efficiently serving LMMs using EPD Disaggregation
Max Batch Size
images or supporting larger batch sizes. In preliminary investigations, removing the LLM and its associated KV cache significantly increases the maximum batch size and the number of images per request (Figure 2).
Disaggregated Aggregated
40
20
Thus, we propose
(EPD) Disaggre- gation, a framework that decouples the encode, prefill, and decode stages, assigning each stage dedicated resources to operate independently. This design enables customized strategies for batching, parallelization, and scheduling at each stage, optimizing resource utilization while reducing contention. As a result, EPD achieves better memory uti- lization, higher throughput, and improved compliance with critical SLOs like TTFT and TPOT.
1
3
5
15
20
30
40
#lmages/Request
Figure 2: Impact of disaggregation on supported batch size and number of images per request for the MiniCPM- 2.6 model. Removing the LLM from the GPU signifi- cantly increases capacity, enabling larger batches and higher- resolution inputs. This demonstrates the memory efficiency benefits of disaggregation.
The major contributions of this work are as follows:
We propose an efficient system for LMM inference that introduces the novel idea of disaggregating the encoding and prefill stages. Our approach addresses key challenges in inter- communication, efficient parallelization, re- source allocation, and performance optimization within this framework.
representations. This stage is computationally intensive, especially for high-resolution or complex multimodal in- puts, and often produces substantial number of additional tokens (Wu et al., 2023). The resulting token inflation in- creases resource consumption and leads to quadratic growth in prefill-stage compute demands, adversely impacting SL.O attainment.
We introduce intra-request parallelization (IRP), which shards request into independent encoding jobs that can be executed in parallel, significantly reducing first-token latency.
Disaggregating the prefill stage from the decode stage has emerged as a well-studied solution for improving LLM in-
ference efficiency (Zhong al., 2024; Qin et al., 2024: Patel et al., 2024; Jin et al., 2024; Hu et al., 2024). By assigning separate resources to each stage, prefill-dcoode disaggregation enables independent optimization of batch- ing, scheduling, and resource allocation strategies, signifi- cantly enhancing system throughput and memory utilization. However, these techniques fall short in addressing LMM- specific challenges, as the addition of an encoding stage fundamentally changes the resource dynamics. The encod- ing stage adds significant compute and memory overhead, inflates tokens, and creates dependencies that affect later stages, requiring a fresh look at disaggregation strategies for multimodal workloads.
We formulate the resource allocation problem as an op- timization over batch sizes. scheduling strategies, and parallelization approaches for each pipeline stage. Fur- thermore, we provide a simple black-box optimizer that uses workload samples to approximate the optimal con- figuration for a given workload.
We develop dynamic role-switching capability that mon- itors the system for bottlenecks and enables flexible re- allocation of resources across the E, P, and D stages by switching an instance's 's role between stages. This ensures the system can respond effectively to changes in workload requirements during online serving.
We conduct evaluations on several popular LMM models S.ss--MMniCMMVVV 2.6, InterVLL2-8B, and Intern 26B 26B-using both real-world and synthetic workloads. Re- sults demonstrate the superiority of our approach: up to 15x lower memory usage, 22x x larger batch sizes, 10x more images per request, up to a 90-100% improvement in SLO attainment, and up to 71 1 lower TTFT.
These challenges present optimization opportunities that cur- rent serving systems do not exploit. Presently, the encode and prefill stages are aggregated into single monolithic and synchronous step executed on the same set of GPUs. Sequential execution of encoding and prefilling introduces interference, as demonstrated in Figure 1. In the aggre- gated setup (top), the prefill step (LLM3) interferes with the encode-heavy request (E9). As a result, such aggrega- tion leads to suboptimal resource utilization and degraded SLO performance, revealing the inadequacy of current solu- tions for LMM workloads. Disaggregating encoding from prefill enables the system to reduce such interference under specific workloads. From another perspective, simultane- ously loading both the multimodal encoder (MME) and the LLM onto the same GPUs in aggregated setups restricts the available memory for processing multiple high-resolution
2. Related Work
In this section, we describe existing approaches in the litera- ture for multimodal model serving and their limitations, and then we review the use of disaggregation techniques.
Multimodal Model Serving. There are two ways to en- able a multimodal model serving system: (I) adopting and extending existing LLM serving systems, such vLLM
2

---

Efficiently serving LMMs using EPD Disaggregation
(Kwon al., 2023), SARATHI (Agrawal al., 2023), and Orca (Yu et al., 2022); and (2) leveraging open-source code from recent works on improving multimodal model infer- ence. The first approach fails to address the encoding bot- tleneck inherent in LMMs, limiting their applicability to multimodal workloads, particularly when handling high- volume and high-resolution multimedia data. In the second approach, recent advancements such as KV cache eviction (Li al., 2024; Ning et al., 2024) and compression (Kang et al., 2024) focus on specific features and limited scenar- ios. For instance, Inf-MLLMM (Ning et al., 2024) enables efficient streaming inference for LMMs on a single GPU, targeting resource-constrainee scenarios. While these im- plementations are feasible for serving LMMs, they fail to meet user SLOs in cloud serving scenarios. To the best of our knowledge, we are the first to propose an LMM serving system that leverages disaggregation and series of inte- grated techniques to enable better resource allocation and improved SLOs.
GPUs
P GPUs
D GPUs
Encoding Queue
Prefill Queue
Decode Queue
Encoding Stage
Prefill Stage
graitatn
Decode Stage
EP Bridge Quaue
PD Bridge Queue
Figure The inference pipeline of EPD Disaggregation.
Tinges-FF--igreee and 10-miggationnhhandee the trans- fer of data from encoding to prefill and from prefill to de- code, respectively. We denote the input text prompt as ip. multimodal data as im. and the output text as o. The steps are as follows:
Encoding (E): The multimodal input im is processed by the multimodal encoder E, converting it into multimodal tokens of = E(im), which form high-dimensional embedding for the next pipeline stage.
E---migratio: Once encoding completes, the generated tokens are transferred to the prefill stage P via the EP- migration function dEP, such that vi = ()F((Uf).
Disaggregated Serving. Disaggregated serving has emerged as a promising technique large model serving. By decoupling the prefill and decode stages, systems like SplitWise (Patel al., 2024), DistServe (Zhong et al., 2024), and Vu (Strati et al., 2024) mitigate interference be- tween these phases, enabling finer control over TTFT and TPOT. However, these systems primarily target LLMs and overlook the encoding step required for LMMs, which is tightly coupled with prefill. Recent work, such as Moon- cake (Qin et al., 2024) and PD-Serve (Jin et al., 2024), have extended disaggregation to include KV cache management and other system-level optimizations. Despite these ad- vancements, they remain limited in their applicability to LMMs. Disaggregating the encoding phase from prefill could unlock new opportunities for optimizing LMM serv- ing. For instance, when handling requests with long videos, the frames could be encoded in parallel, reducing head-of- line blocking and improving overall system efficiency. This approach would enable more flexible batching and schedul- ing strategies, achieving better latency and throughput for multimodal workloads.
Prefill (P): The prefill stage processes of and the text prompt ip to produce the initial KV cache and first output token: key of = P(r ip).
PD-migration: The KV cache and token are passed to the decode stage via the PD-migration function d/pD, where kuf, of OPD(kuf,
Decode (D): Decoding proceeds autoregressively, gen- erating the next token of41 and updating the cache: kuft of = D(kug of).
This autoregressive process continues until the output se- quence is fully generated.
3.2. System Design and Optimization
Figure 4 illustrates the architecture of the EPD Disaggre- gated inference system. Each pipeline stage (Encoding, Prefill, and Decoding) has independent instances that run the corresponding stage. These instances operate in data par- allel (DP) mode, enabling concurrent processing of multiple requests per stage and ensuring scalability and efficiency.
3. Method
In this section, we first describe the EPD disaggregation framework (Section 3.1), followed by detailed explanation of the system design and optimizations (Section 3.2), includ- ing asynchronous token transfer, intra-request parallelism, optimized resource allocation, and dynamic role switching.
Each instance comprises a scheduler, responsible for scheduling requests, block managers (responsible for man- aging cache(s)), and multiple workers. The workers operate in censor-paralle (TP) and/or appline-paralle (PP) mode, where each worker holds only a subset of the model weights and the corresponding caches required for the stage.
3.1. EPD Disaggregation
In the encoding stage, workers load only the encoder weights and initialize the Multimodal (MM) cache. In the prefill stage, the LLM weights are loaded, and both the MM and KV caches are required to efficiently manage all
As shown in Figure 3, disaggregating an LMM system involves dividing the inference process into three stages: encoding, prefill, and decode. Transitions between these
3

---

Efficiently serving LMMs using EPD Disaggregation
Multi-Media Request
Scheduler (Load Balancer) decides the assignment
Monitoring and Dynamic Role Switching (63.2.4) decides and controls instance migration
EP Bridge Queue
PD Bridge Queue
Migrate D instance to P
Encoding Instance
Prefill Instance
Decoding Instance
Scheduler
Scheduler
Scheduler
MM Block Manager
MM Block Manager
KV Block Manager
KV Block Manager
Encoding Worker
Prefill Worker
Decoding Worker
MM Cache
Async Transfer (33.2.1
MM Cache
KV Cache
Async Transfor
KV Cache
($3.2.1)
Encoder Weights
LLM Weights
LLM Weights
TP/PP
TP/PP
TP/PP
IRP ($3.2.2)
DP
DP
Figure 4: System architecture of the proposed EPI) Disaggregated Inference.
the data associated with the request. In the decoding stage, workers load the LLM weights for decoding tasks and use the KV cache.
the blocks are reassigned or de-allocated, ensuring flexible cache utilization even under heavy workloads.
3.2.2. INTRA-REQUER PARALLEL (IRP)
Cache transfers occur asynchronously when an instance pulls request from its queue, ensuring the transfer occurs once the downstream instance is ready. Further, our system incorporates several techniques to optimize the performance of the disaggregated LMM pipeline, with a primary focus on ensuring smooth token transfers between stages (Section 1), reducing latency (Section 3.2.2), managing resources effectively (Section 3. 3.2.3), and adapting to changing work- loads (Section 3.2.4). The ablation of these features is shown in Section 4.4.
Multimodal requests often include multiple high-resolution images in many practical scenarios like autonomous driving and video question answering. In modern LMMs, these images are further converted into large number of patches that need to be processed through a computationally heavy MME. This significantly increases the computation load, making the encoding process a major bottleneck.
To address this, we introduce Intra-Request Parallelism (IRP), which partitions single request's 's image patches across multiple encoding workers in a data-parallel fash- ion. Since patches are encoded independently, they can be processed and transferred concurrently. Specifically, each encoding worker concurrently processes a subset of patches, computes their token representations, and asynchronously transfers them to the prefill stage. Once all patch-level to- kens reach the prefill stage, they are aligned, projected, and merged to form the complete multimodal tokens.
3.2.1. ASYNCHRONOUS TOKEN TRANSFER
Disaggregating the system introduces an additional step of transferring vision tokens from the encoding to the prefill stage. To minimize latency during token transfers between stages, our system employs direct, asynchronous transfers via highbbhndwidth channels (NVLink, and InfiniBand). The asynchronous transfer allows the system to continue processing new requests without interruption. Both encod- ing and prefill workers maintain an MM cache to facilitate this process.
3.2.3. OPTIMIZED RESOURCE ALLOCATION
The EPD system requires tuning various configurations, including batch sizes, scheduling strategies, and paralleliza- tion methods for each pipeline stage. To determine the optimal setup, we collect historical workload samples and apply black-box optimizer.
When encoding is complete, tokens are stored in the encod- ing worker's MM cache, allowing it serve new requests immediately. An asynchronous event loop monitors com- pleted encoding tasks and initiates direct token transfers to the prefill worker's MM cache. Once the transfer is confirmed, the encoding cache entries are cleared to free memory. To manage these cache blocks effectively, we in- troduce the HTMBlockManggee which pre-allocates cache blocks based each request's needs. After token transfer,
Let f(-) represent the system's performance metric (c.g.. goodput, see Section 4). Since f(.) is treated as a black- box function with an unknown internal mechanism, we rely on a simulator-extendeer from DistServe (Zhong et al.,
4

---

Efficiently serving LMMs using EPD Disaggregation
2024) evaluate performance metrics efficiently. The objective is to maximize performance while minimizing GPU usage, which inherently reduces pipeline inefficiencies ge.g.. idle time) and improves resource utilization. Formally, we solve
switching between P and D stages, both the LLM and KV cache can be reused.
4. Experiments
In this section, we analyze and compare the performance of the proposed EPD disaggregation method against various baselines. We start with an end-to-end generation perfor- mance analysis in Section 4.1, followed by an examination of the first token latency in Section 4.2. We then evalu- ate the memory savings achieved by EPD in Section 4.3. Next, we present an ablation analysis of the key system components in Section 4.4. Finally, we present an exten- sion of our framework to Neural Processing Units (NPUs) in Section 4.5. Additional experiments are detailed in the Appendix, including 1) a throughput comparison for offline and heterogeneous settings in Appendix A.3, 2) additional analysis of the implementation on NPUs in Appendix F, 3) an experiment extending our framework to the audio modal- ity in Appendix A. and 4) additional SLO and memory experiments in Appendix A.
max f(p. b, s) - Scost(p)
(1)
(p.b,s)Ex'
Here, X' is the search space for system configs, including parallelization configs p. max batch size configs b, and scheduling configs s (See Appendix D for details). We use Bayesian optimization (Calvo et al., 2019) solve Problem 1.
3.2.4. DYNAMIC ROLE SWITCHING
The configuration optimizer described in the previous sec- tion can determine the optimal settings for a given workload. However, in an online environment, workload characteris- ties can change dynamically, requiring adjustments to the configuration. Re-initializing the entire system from scratch in response to these changes can be both difficult and costly. For instance, there may be ongoing requests in the encoding, context, or decoding phases with precomputed KV and/or VE caches stored in memory. A naive re- re-initialization would not only involve booting time but also force these requests to restart from the beginning, potentially causing a cascading impact on SLO metrics for subsequent requests.
Baselines: We compared the proposed EPD method against two popular baselines: DistServe (Zhong al., 2024) and vLLM (K woon et al., 2023). The DistServe baseline imple- ments the prefill-deoode (PD) disaggregation approach, in which the prefill and encoding phases are executed on one set of GPUs, while the decode phase is disaggregated on separate GPUs. Since DistServe was originally designed for LLMs, we extended it to support LMMs by enabling mul- timodal data processing and modifying its block manager to accommodate multimodal tokens. The vLLM baseline adopts monolithic architecture, where all three stages run on the same set of GPUs.
To overcome these challenges, we introduce dynamic role switching, which enables any instance in the E, P, or D stages to switch roles to any other stage (E, P, or D) with minimal overhead. At a high level, dynamic role switching continuously monitors the system's 's queuing statistics across all stages and reallocates workers to stages experiencing higher demand. When decision is made transfer instance from source stage S to destination stage T,, the migration process occurs in three key steps:
Models: We utilized three LMMs in our analysis: MiniCPM-V 2.6 (Yao et al., 2024), Intern VL2-8B, and Intem2L2-226 (Chen et al., 2024). These LMMs are renowned for their advanced capabilities in processing and understanding multimodal data. Detailed descriptions of the LMMs and their sizes can be found in Appendix F.2.
Offload: The instance in the S stage stops accepting new requests and redistributes its queued tasks to sib- ling instances in the same stage.
Datasets: To evaluate performance across diverse scenarios, we use three datasets: synthetic workload, NextQA, and Video-MME. The synthetic workload enables configurable parameters such as prompt length, number of images per request, image resolution, output length, and sampling set- tings. Unless otherwise noted, the input prompt length is set to 22 tokens. NexIQA (Xiao et al., 2021), ), benchmark video question-answent
Migration: The instance is reconfigured to meet the requirements of the T stage. This may involve switch- ing both the model and cache type. For example, if the E stage is involved, the instance may switch from an LLM to an MME model and from a KV cache to an MM cache.
Onload: The migrated instance resumes processing new requests, helping alleviate the queuing bottle- neck at the T stage.
dataset, features human-annotated ques- tions and answers, offering a more realistic reflection of real- world video request distributions compared to the synthetic workload. Video-MME (Fu et al., 2024) is multimodal evaluation dataset designed for assessing large LMMs on video understanding tasks. It contains multiple-choice video
This process typically takes less than 0.7 seconds. The duration is longer for migrations involving the E stage due to model and cache changes, but significantly shorter when
5

---

Efficiently serving LMMs using EPD Disaggregation
EPD
DistServe
vLLM
100
100
100
SLO (%6
50
50
50
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.8
1.0
100
100
100
SLO (%)
50
50
50
0
3
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Request Rate (req/s)
Request Rate (req/s)
Request Rate (req/s)
(a) MiniCPMv-2.6
(b) InterVVL.2-88
(c) Intern VI.2-26B
Figure 5: SLO attainment (1) for end-to-end inference across multiple models and image counts per request. Subfigures (a), (b), and (c) correspond to MiniCPM-V 2.6, Intern VL2-8B, and VL2-266, respectively. The top and bottom rows show results for 2 and 4 images per request. EPD consistently outperforms all baselines across configurations.
question answering items that span diverse video lengths, topics, and reasoning types. In contrast NextQA, which emphasizes open-ended video-text generation, Video-MME focuses strictly multiple-choice QA and covers broader spectrum of video durations and formats.
details of the corresponding SLO criteria are discussed in Appendix E.3. The results for three models (columns) are presented in Figure 5. The X-axis represents the overall request (A), and the Y-axis indicates the percentage of requests meeting both TTFT and TPOT requirements, with 90% threshold denoted by black dotted line.
We evaluate the system based on runtime performance and memory consumption. The per- formance metrics are as follows:
Evaluation Metrics:
EPD outperforms all baselines, achieving over 90% SLO attainment at lower request rates. This due to its ability to parallelize the computationally intensive image encod- ing step across multiple GPUs. DistServe and vLLM often maintain less than 10% SLO attainment due to interference. Meanwhile, comparing the first and second rows for vary- ing image counts per request (2 and 4), we observe that, while more images per request increase the workload, EPD maintains reasonable performance, whereas baselines' SLO attainment drops significantly. Lastly, InternVI. which prefill-heave experiences queuing delays due the higher number of image tokens, particularly higher request rates. This effect further exacerbates the performance degradation from the 8B to the 26B model. In contrast, MiniCPM-V, V, op- timized to generate fewer image tokens, avoids these delays and achieves lower overall latency. Further results with and 8 images are provided in Appendix A.4.
TTFT: The time from the submission of request to the system until the first token is received by the user. TPOT:
The average time interval between consecutive output tokens (excluding the first token).
SLO Attainment: The percentage of requests that meet predefined SLOs, such as TTFT and TPOT require- ments.
Goodput: The highest request rate at which 90% or more SLO attainment is achieved.
For memory benchmarking experiments, we analyze the baselines by evaluating the benefits of available free mem- ory in each approach. Additional free memory can facil- itate higher batch sizes, accommodate more images per request, or enable larger key-value (KV) cache sizes (see Section 4.3).
Next, repeat the experiment using the non-synthetic video question-answerring dataset, NextQA (Xiao al., 2021). To do so, randomly sampled 100 examples, with input text token lengths ranging from 4 to 21 (aver- age: 11.42) and output token lengths ranging from to 7 (average: 2.75). Each video request was represented by uniformly sampled frames, and we used MiniCPM-V 2.6 in the experiment, adhering the SLO criteria of TTFT 5.60 and TPOT 0.06. As illustrated in Figure 7, EPD is
4.1. SLO Attainment for End-to-End Generation
In this experiment, we evaluate the goodput of EPD and the baselines in an online setting where 100 multimodal requests arrive following a Poisson process with rate X. The length output tokens is fixed to 10. We test requests with 2 and images, each at a resolution of 4032 3024. The
6

---

Efficiently serving LMMs using EPD Disaggregation
EPD
DistServe
10.0
7.5
TTFT (5)
5.0
6
2.5
2
2
8 10 12 14 16
2
4
8
3
4
5
# #lmages/ Request
#lmages/ Request
#lmages/ Request
(a) MiniCPMv-2.6
(b) Intern VL2-8B
(c) Intern VL2-26B
Figure 6: Distribution TTFT (Y-axis) across varying numbers images per request (X-axis) for (a) MiniCPM-V 2.6, (b)
AnternVL--88 and (c) Intern VL2-26B. Each plot illustrates the latency behavior under increasing input sizes.
EPD
DistServe
VLLM
EPD
DistServe
vLLM
100
100
SLO (%)
SLO (%)
50
50
0
0.5
1.5
2.5
3.0
Request Rate (req/s)
Request Rate (req/s)
Figure 7: SLO attainment (f) versus request rate the NexIQA dataset using the MiniCPM-V 2.6 model. EPD consistently achieves higher SLO attainment compared to all baselines.
Figure 8: SLO attainment (f) versus request rate on the Video-MME dataset using the MiniCPM-V 2.6 model. EPD significantly outperforms competing baselines across all request rates.
the only framework achieving 90% SLO attainment low request rates, demonstrating its superior ability to handle real-world workloads compared to DistServe and vLLM.
4.2. First Token Generation Latency
Multimodal requests often impose a heavy load on both the E and P phases, making the first token generation key la- tency bottleneck. Therefore, in this experiment, we analyze the first token latency for various baselines. The results presented in Figure 6 show box plots of TTFT distributions for three different models. Note that, as the decoding phase is excluded, the vL.M baseline is equivalent to DistServe and thus omitted. Requests are generated according to Poisson distribution with fixed request rate A. Specifi- cally, 0.25 for MiniCPM-V 2.6 and 0.08 for both Inter- and Intern VL.2-26.. Thanks intra-request parallelization, EPD significantly outperforms both vLLM and DistServe. Specifically, TTFT is reduced by up to 71.9%, 32.8%, and 44.9% compared to the DistServe base- line for the MiniCPM-V 2.6, VL2-88, and InternVL22 26B models, respectively.
Finally, we conduct the same experiment using the Vidco- MME (Fu et al., 2024) dataset. We evaluate SLO attainment, defined TTFT < 3.1s and TPOT < 0.025s, on 100 ran- domly sampled Video-MME. examples using MiniCPM-V 2.6. Each video is represented by 64 uniformly sampled frames, following the MiniCPM frame configuration re- ported on the Video-MME leaderboard. The results are shown in Figure& 8.
As seen, EPD consistently outperforms vLLM and Dist- Serve across all rates, demonstrating a strong generalization temporal multimodal workloads, such videos.
#Frames
8
16
32
64
Method
vLLM
0.42
0.82
1.59
3.11
Additionally, conducted a TTFT comparison the Video-MME (Fu al., 2024) dataset for various baselines.
DistServe
0.42
0.81
1.54
3.08
EPD (ours)
0.24
0.30
0.49
1.00
The results are presented in Table 1. As seen, EPD achieves latency reduction of up to 38.2% over DistServe and up to 69.2% over vLLMM Moreover, the performance gap widens fnnth-forrat for instance, 8 frames, EPD reduces latency by 42.9% 9,, and 64 frames, the reduction
Table 1: Mean TTFT latency (in seconds) (4) for varying video lengths at fixed request rate of request/sec. Results averaged over 100 Video-MME samples. EPD achieves the lowest latency across all video lengths.
with increasing video length-

---

Efficiently serving LMMs using EPD Disaggregation
reaches 67.5% relative to DistServe. These results highlight EPI)'s superior scalability and robustness under increasingly demanding video processing workloads.
ferent settings of image resolution and models in Table 3. The number of images per request fixed 10, and the KV cache is allocated to utilize 80% of the available free mem- ory. As seen, EPD significantly outperforms DistServe for both E and P batch sizes. For example, with Intern V1.2-26B at 787 x x444 resolution, EPD supports a batch size of 22 for encoding vs. DistServe's I, achieving 22x improvement. Similarly, for MiniCPM-V 2.6 787 444, EPD achieves batch size of 29 vs. DistServe's 2 for prefill, 14.5x improvement.
4.3. Memory Savings through Stage Disaggregation
In this section, analyze the memory savings achieved by disaggregating the encoding and prefill stages. The E work- ers can save memory as they do not require LLM weights or the KV cache. Analyzing only the weight size indicates memory reduction of approximately 95%, 96.2%, and 78.3% for the MiniCPM-V 2.6, AnternVL2-88 and I2ternVL-226B models. Similarly, for the workers, memory savings of about 5%, 3.7%, and 21.6% are achieved, respectively. In practice, since KV cache is also not required at E workers, the memory saving can be even higher (93.3% saving, i.e.,
DistServe (E. P)
EPD
Model
Image Reso.
#Patch
E
313,234
1
7
49
86
MiniCPMv 2.6
787. 787.444
3
2
16
29
4032,3024
10
OOM
4
15x lower, according to profiling). These reductions in memory usage enable the EPD system to support higher numbers of images per request and batch sizes (discussed in the following experiments) and larger KV cache sizes (discussed in Appendix A.2).
313,234
13
15
Intern VL228B
787.444
3
67
4032,3024
13
15
313,234
13
OOM
6
Intern VL2-26B
787.444
3
1
22
4
4032, 3024
13
OOM
6
Model
Image Reso.
DistServe
EPD
Table 3: Comparison of the maximum supported batch sizes for E and P stages across different models and image reso- lutions. Higher values are better; italicized values indicate the best in each row. OOM denotes cases where the model ran out of memory.
313,234
77
490
MiniCPM-V 2.6
787 444
26
165
4032,3024
7
49
313,234
19
19
IntemVL2-8B
787.444
19
19
4032,3024
19
4.4. Ablation Study
313.234
10
Inter VL2-266
787.444
In this section, we analyze the impact of various components of the proposed system.
4032,3024
10
Table 2: Comparison of the maximum number of images supported per request for various image resolutions across different models. Higher values are better; best values in each row are italicized.
#I/R=2
#1/R=4
#1/R=6
#I/R=8
EPD w/o IRP
0.92
1.02
1.14
1.74
1.46 (1.6x)
2.47 (2.4x)
3.37
(2.9x)
4.27
(2.5x)
Table 4: Effect of ablating IRP feature from the proposed
EPD Supports Higher Number of Images per Request: We compare the maximum number of images per request supported by the disaggregated (EPD) and aggregated (Dist- Serve vLLM) systems across three multimodal models: MiniCPMv, internVL2-288 and internVL2-2611 The exper- iment was conducted at three different image resolutions, with a fixed batch size of 1 and with 80% of the available memory allocated for the KV cache. As shown in Table 2, EPD handles more images per request than DistServe, for example, at 6032x3024 resolution, 7x more for InterVL22 26B and 10x more for Intern VL2--B. For Intern VL2-8B, the limit of 19 images per request is due to its maximum context length. Without this constraint, a larger number of images per request could be supported.
system on TTFT (s). Disabling IRP negatively affects the TTFT (up to 2.9x worse) for various multiple images/ re- quest (#I/R). Results are averaged over 100 requests.
Effeet of IRP: We analyze the effect of ablating IRP, pre- sented in Section 3.2.2. The experimental settings are the same as the TTFT experiment in Section 4.2. As shown in Table 4, removing the IRP feature negatively affects TTFT across various images/ request. Moreover, the degradation exacerbates the number images/ request increases. This is because the IRP feature allows parallelization of encoding load within the same request across multiple GPUs.
Effect of Offline Optimizer: By default, the EPD system collects workload samples and finds the optimal configura- tion offline using the optimizer. To demonstrate its effect, we conduct an experiment without using the optimizer and
EPD Supports Higher Batch Sizes: We compare maxi- mum supported batch sizes for E and P stages across dif-
8

---

Efficiently serving LMMs using EPD Disaggregation
Goodput (u/s)*
TTFT (s) 4
TPOT (s) L
First, we compare EPD-NPU against the vLLM and Dist- Serve baselines on the Intern VL.2-8B LMM. We evaluate SLO attainment, defined as TTFT - 8.5s and TPOT - 0.12s on eight 4K images per request. We used the same settings as described in Section 4.1, except that we employed a heavy encoding workload of eight 4032> x 3024 images per request. The optimal configuration for this workload was found to be 5E2PID, and the corresponding results are presented in Figure 9. As shown, EPD is the only configuration that achieves the SLO requirements, while the other baselines fail to meet the SI.Os entirely, even at low request rates.
HDP
1.25
2.12
0.031
w/o Opt.
0.56 (2.2x)
4.48 12.1x)
0.025 (0.8x)
Table 5: Ablating the offline optimizer reduces goodput by 2.2x on average when configurations are selected randomly. 4 indicates lower is better; f indicates higher is better.
select configurations randomly. Specifically, we uniformly sample 10 configurations at random and report the expected value of the performance metric in the second row of Table 5. For fair comparison, when evaluating TTFT and TPOT, we maintain the same request rate of 1.25 r/s, which corre- sponds to the goodput of the EPD system (see Appendix E.4 for more details). As shown in Table 5, disabling the op- timizer causes significant performance degradation in terms of both goodput and TTFT. This highlights the importance of the optimizer, especially when the EPD system has many tunable configurations.
EPD
DistServe
VLLM
100
(90)
50
0
Effect of Dynamic Role Switching: To analyze the im- pact of dynamic role switching, we conduct a controlled experiment simulating a shift in workload characteristics. Specifically, we generate 100 requests following the same statie workload configuration as described in Section 3.2.4. However, to introduce artificial workload change, the first 10 requests generate 50 output tokens, while the remaining 90 requests generate 500 output tokens. The request arrival rate is fixed at 3 requests per second. The results are shown in Table 6.
0.01
0.02
0.03
Request Rate (req/s)
Figure 9: SLO attainment (f) on NPUs under varying re- quest rates for a synthetic workload using the Intern VL.2-8B model. EPD maintains positive SLO attainment under strict TTFT constraints, while baselines fail to meet the criteria.
Next, we compare the TTFT of EPD-NPU against vLLM. EPD-NPU achieves a 35.2% improvement over vLLM, sur- passing the 24.4% improvement observed in the GPU sce- nario. This additional ~10% improvement is attributed to the proportionally heavier encoding workload of LLMs on NPUs compared to GPUs. Please refer to Appendix F.I for more details. As result, disaggregating the encoding from the prefill phase yields greater benefits on NPUs than on GPUs.
I Latency (s)
TTFT (s) 4
TPOT (s) +
EPD w/o Switch
28.01
1.42
0.05
61.10 (2.2x)
1.33 (0.9x)
0.12 (2.4x)
Table 6: Ablating dynamic role-switching from EPD de- grades TPOT by 2.4x and increases end-to-end latency by 2.2x. Results are averaged over 100 requests with one 4K image each. 4 indicates lower is better.
5. Conclusion
In this paper, we proposed novel approach for optimizing LMM systems via disaggregation of key processing stages. By separating the encoding, prefill, and decoding phases into distinct stages, our system provides greater flexibility in resource allocation, enabling more efficient management of computational and memory resources. This disaggregation, together with dynamic resource allocation, asynchronous to- ken transfer, and advanced parallelization strategies, directly addresses several critical challenges in LMM deployment, including latency reduction, memory optimization, and ef- ficient computational resource usage. We validated the ef- fectiveness of our system through extensive experiments. Finally, we outline the system's 's limitations and future direc- tions in Appendix B and Appendix C, respectively.
Without dynamic worker migration, the system performs poorly because it remains fixed in the initial configuration (5EIP2D, optimized offline for 50 tokens) and is unable to adapt to the increased decoding demand. In contrast, the EPD system with migration dynamically reconfigures itself (2E1P5D) to handle the new workload (500 output tokens) by shifting three E instances to D, resulting in approximately 2x better performance.
4.5. Adaptation to Neural Processing Units (NPUs)
We present the results of adapting the proposed EPD frame- work to Huawei Ascend NPUs. For more details on the implementation and experiments, see Appendix F.

---

Efficiently serving LMMs using EPD Disaggregation
Impact Statement
ing. In International conference machine learning,
pp. 4507-4518. PMLR, 2020.
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which feel must be specifically highlighted here.
Jin, Y., Wang, T., Lin, H., Song, M., Li, P., Ma, Y. Shan, Y,, Yuan, Z., Li, C., Sun, Y., et al. P/d-serve: Serving disaggregated large language model at scale. arXiv:2408.06 1477, 2024.
References
Kang, H., Zhang, Q., Kundu, S., Jeong G., Liu, Z., Krishna, T., and Zhao, T. Gear: An efficient kv cache compres- sion recipefor near-lossless generative inference of Ilm. arXiv:4403.05527, 2024.
Achiam, J., Adler, S., Agarwal, S., Ahmad, L.,, Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Alt- man, S., Anadkat, S., et al. Gpt-4 technical report. arXiv:2303.08772 2023.
K Kwon, W., Li, Z., Zhuang. S., Sheng, Y., Zheng, L.,, Yu, C. H., Gonzalez, J. E., Zhang, H., and Stoica, I. Er- ficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Agrawal, A., Panwar, A., Mohan, J., Kwatra, N., Gula- vani, B. S., and Ramjee, R. Sarathi: Efficient Ilm in- ference by piggybacking decodes with chunked prefills. arXiv:2308. 16369, 2023.
Alvar, S. R., Singh, G., Akbari, M., and Zhang, Y. Divprune: Diversity- bbased visual token pruning for large multimodal models. arviv preprint 4Xiv:2503.02175. 2025.
Li, Y., Huang, Y., Yang, B., Venkitesh, B., Locatelli, A., Ye, H., Cai, T., Lewis, P., and Chen, D. Snapkv: LLM knows what you are looking for before generation. arXiv:2404.1446 2024.
Calvo, B., Shir, O. M., Ceberio, J., Doerr, C., Wang, H., Bäck, T., and Lozano, J. A. Bayesian perfor- mance analysis for black-box optimization benchmark- ing. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, pp. 1789-1797, 2019.
Liu, H., Li, C., Wu, Q. and Lee, Y. Visual instruction tuning. In Proceedings of NeurIPS, 2023.
Liu, H., Zhang, Y., Yang, X., Li, X., Chen, H., Zhu, J., and Gong, Y. A survey of resource-efficient 1lm and multimodal foundation models. arXiv:2401.080%) 2024.
Chang, Y. Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Transactions on Intelligent Systems and Technology, 15 (3):1-45, 2024.
Liu, Z., Wu, Z.,, Gan, C., Zhu, L.., and Han, S. Datamix: Efficient privacy-preserving edge-cloud in- ference. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XI 16, pp. 578-595. Springer, 2020.
Chen, 7., WU, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., Li, B., Luo, P., Lu, T., Qiao, Y., and Dai, J. Internvl: Scaling vision foundation models and aligning for generie visual- linguistic tasks. In Proceedings of CVPR, pp. 24185- 24198, June 2024.
Luo, J., Zhang, W., Yuan, Y., Zhao, Y., Yang. J., Gu, Y., Wu, B., Chen, B., Qiao, Z., Long, Q. Tu, R., Luo, X., Ju. W., Xiao, Z., Wang, Y., Xiao, M., Liu, C., Yuan, J., Zhang. S., Jin, Y., Zhang, F., Wu, X., Zhao, H., Tao, D., Yu, S., and Zhang, M. Large language model agent: A survey methodology, applications and challenges. MxXiv:2503221460 2025.
Fu, C., Dai, Y., Luo, Y., Li, L., Ren, S., Zhang, R., Wang, Z., Zhou, C., Shen, Y., Zhang, M., et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Ilms in video analysis. arXiv preprint AXiv:2405.2107. 2024.
Ning, Z., Zhao, J., Jin, Q., Ding, W., and Guo, M. Inf- miim: Efficient streaming inference of multimodal large language models single gpu. arXiv:2409.0908: 2024.
Hu, C., Huang, H., Xu, L.,, Chen, X., Xu, J., Chen, S., Feng, H., Wang, C., Wang, S., Bao, Y., et al. Inference with- out interference: Disaggregate 11m inference for mixed downstream workloads. arXiv:2401.1118.. 2024.
Patel, P., Choukse, E., Zhang, C., Shah, A., Goiri, I., Maleki, S., and Bianchini, Splitwise: Efficient gen- erative 11m inference using phase splitting. In 2024 ACM/IEEE 51st Annual International Symposium Computer Architecture (ISCA), pp. 118-132, 2024.
Huang, Y., Song, Z., Li, K. and Arora, S. Instahide: Instance-hiding schemes for private distributed learn-
10

---

Efficiently serving LMMs using EPD Disaggregation
Qin, R., Li, Z., He, W. Zhang, M., Wu, Y., Zheng, W., and Xu, X. Mooncake: A kvcache-centric disaggregated architecture for IIm serving. arXiv:2407.00073 2024.
actions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 9777-9786, June 2021.
Singh, G., Wang, C., Tazwar, A., Wang. I.,, and Zhang, Y. Ipprotect: protecting the intellectual property of visual datasets during data valuation. arXiv preprint arXiv:2212.1 1468, 2022.
Yao, Yu, T., Zhang, A., Wang, C., Cui, J., Zhu, H., Cai, T., Li, H., Zhao, W., He, Z., et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv:2408.0180 2024.
Yu, G.-L, Jeong, S., Kim, G.-W,, Kim. S., and Chun, B.-G. Orea: A distributed serving system for Transformer-Based generative models. In 16th USENIX Symposium Operating Systems Design and Implementation (OSDI 22), pp. 521-538, July 2022.
Strati, F. Mcallister, S., Phanishayee, A., Tarnawski, J., and Klimovic, A. Déjävu: Kv-cache streaming for fast, fault-tolerant generative IIm serving. In ICML., July 2024.
Wu, J., Gan, W., Chen, Z., Wan, S., and Philip. S. Y. Multimodal large language models: A survey. In 2023 IEEE International Conference on Big Data (BigData), pp. 2247-2256. IEEE, 2023.
Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X., and Zhang. H. Distserve: Disaggregating prefill and decoding for goodput-optimized large language model serving. In OSDI, pp. 193-210. USENIX Association, 2024.
Xiao, J., Shang, X., Yao, A., and Chua, T.S.. Next-qa: Next phase of question-answering to explaining temporal

---

Efficiently serving LMMs using EPD Disaggregation
A. Additional Experiments
Model
Images/Req.
DistServe
EPD
A.I. Adaptation to Audio modality
86%
99%
10
74%
97%
In this section, we extend EPD to the audio modality to eval- uate its effectiveness beyond vision tasks, highlighting the method's generalizability. Specifically, we evaluate EPD in an online, audio setting using the u travox-vo 3 model, based on LLaMA3. 1-8B. To simulate an encode-intensive workload, each request is configured to include 24 audio files. Evaluation metrics include SL.O attainment- as TTFT & 2.0 s and TPOT & 0.025 s- s-and goodput (re- quests per second, r/s). As shown in Table 7, EPD con- sistently outperforms vLLM and DistServe across all re- quest rates, demonstrating superior efficiency and reliability. Combined with prior results in the video domain, these find- ings reinforce EPD's robustness and generalizability across modalities.
MiniCPM-V 2.6
20
49%
95%
40
OOM
92%
80
OOM
OOCL
5
94%
95%
Intern VL2-8B
89%
91
20
OOCL
OOCI.
67%
89%
10
36%
80%
LL2-m2L2222B
20
OOM
63%
40
0OM
OOCI.
Table 8: Comparison maximum supported KV cache size (in terms of percentage of free memory) on prefill node for various #images/ request. Image resolution fixed to 4K. Higher (italicized) better.
Rate (r/s)
SLO Attainment Rates (t)
A.3. Throughput Comparison for Offline and Heterogeneous scenario
vLLM
DistServe
EPD
0.10
0.99
0.99
0.99
0.25
1.00
0.94
0.99
In this section, we compare the throughput of the EPD and DistServe methods under both offline and heterogencous settings. In the offline scenario, a batch of requests sub- mitted in advance, allowing the system to process them overnight with the goal of maximizing end-to-end (E2E) throughput. In the heterogeneous setting, we consider cluster composed of GPUs with varying computational and memory capacities.
0.50
0.99
0.89
1.00
1.00
0,91
0.72
0,96
1.10
0.87
0.69
0,93
1.15
0.87
0.68
0.93
Goodput (t/s) f
1.01
0.45
1.16
Table 7: SLO attainment results (1) for online audio bench- marking with ultravox-v0.3 (24 audio files per re- quest). All baselines use 4 GPUs: vLLM operates in data- parallel (DP) mode, DistServe uses 3P1D configuration, and EPD adopts a 2EIPID setup. EPD achieves consis- tently high SLO attainment and the highest goodput across all request rates.
Our motivation is as follows: in the DistServe system, both the encoding and prefill stages are executed on the same worker, requiring the corresponding memory to be allocated on a single GPU. However, GPU memory is limited, and this constraint becomes particularly problematic in hetero- geneous environments that include low-end GPUs. In such cases, the total memory demand of the encoding and pre- fill stages may approach or exceed the available memory, significantly restricting the batch size or even rendering the DistServe method infeasible on these devices. Thus, we consider a controlled scenario in our experiment where the memory usage of the prefill stage in DistServe remains within the capacity of low-end GPUs, but only permits minimal batch size of I for both encoding and prefill stages.
A.2. EPD Supports Larger KV Cache Sizes
In this section, we compare the maximum KV cache size that can be allocated across various baselines. The batch size fixed at 1, and the image resolution is set to 4032x3022 As shown in Table 8, EPD supports significantly larger KV cache sizes than DistServe. For instance, for the InterVL22
26B model at 10 images per request, EPD can support a KV cache size of 80% vs 36% for DistServe, representing 2.2x improvement. Notably, in certain scenarios-sun as with the InternVL2-26B model and 20 images per request- the DistServe system encounters an OOM error, indicating that it cannot process 20 images even with a KV cache size of 0. Additionally, some configurations encounter an Out of Context Limit (OOCL) error, where the large number of encoding tokens generated by high image counts exceeds the LLM's context limit during the prefill stage.
We conduct this experiment using 8 A800 GPUs. For the EPD system, we use the default 5E2P1D configuration, which allocates 5, 2, and GPUs to the encoding, prefill, and decoding stages, respectively. The corresponding max- imum batch sizes set 8 for encoding, 8 for prefill, and 128 for decoding. In the DistServe system, adopt the 7P1D setup, with maximum batch sizes of I and 128. Here, the encoding and prefill steps are conducted on the 7 P workers. The workload consists of 1,000 requests, each containing a single image, a simple prompt ("What the
12

---

Efficiently serving LMMs using EPD Disaggregation
EPD
DistServe
E2E Throughpo Suughhut (rea/s)
150
150
175
6030
150
100
125
50
100
2
4
28
128
#GPU
#image
64 Batch size
Figure 10: Left: Impact of varying the number of encoding workers in the EPD method. The notation xEy? denotes
a configuration with x encoder and prefill workers. The DistServe method uses fixed 7P configuration, assigning 7 workers to handle both encoding and prefill steps. Middle: Effect of the number of images per request on end-to-end throughput. Right: Sensitivity to encoding and prefill batch sizes, where these two numbers are set equal.
EPD
DistServe
vLLM
100
100
100
SLO (%)
0
2
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
100
100
100
SLO (%)
50
0
2 Request Rate (req/s)
3
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Request Rate (req/s)
Request Rate (req/s)
(a) MiniCPMv-2.6
(b) Intern VL.2-8B
(c) Intern VL.2-26B
Figure 11: SLO attainment (f) for end-to-end inference across multiple models and image counts per request. Subfigures (a), (b), and (c) correspond to MiniCPM-V 2.6, InternVL2-88 and Intern VL2-26B, respectively. The top and bottom rows show results for 6 and 8 images request. EPD consistently outperforms all baselines, demonstrating robust performance as image count increases..
content this image?"), and a maximum output to- kens. We perform an ablation study by tuning the system's 's hyperparameters as described in Sec. 3.2.3, and analyze their impact throughput. The results are presented in Fig. 10.
either manually choose a large batch size or rely on the al- gorithm in Sec. 3.2.3 automatically determine an optimal configuration.
A.4. SLO Attainment for Higher Number of Images
The left plot demonstrates the importance of selecting an appropriate GPU configuration. The algorithm detailed in Sec. 3.2.3 can automatically select the 5E2P config- uration the optimal setup for maximizing end-to-end (E2E) throughput. The middle plot shows that EPD achieves higher throughput when the number of images per request is small, indicating that its disaggregated architecture ef- fectively mitigates encoder-side compute bottlenecks. The right plot highlights that EPD is relatively insensitive to encoding and decoding batch sizes. This enables users to
In this section, we show additional results from the experi- ment in Section 4.1 pertaining and 8 images per request. The results presented in Figure 11. As seen, the SLO attainment for EPD begins decline as number of images rise, particularly at higher request rates. This decline re- flects the increasing computational demand associated with processing multiple high-resolution images within a single request. Nevertheless, EPD continues to outperform all basclines. which struggle even at low request rates and fail to scale effectively under increased workloads.
13

---

Efficiently serving LMMs using EPD Disaggregation
B. Limitations
cloud for prefill and decode stages, minimizing privacy risks without compromising model capability. This hybrid archi- tecture could form the foundation for privacy-preserving multimodal systems.
While disaggregated serving offers significant benefits in meeting strict service-level objectives, it is not universally optimal. In scenarios where throughput is the primary con- cern and SLOs can be relaxed, traditional aggregated serving can outperform disaggregated approaches due to superior GPU utilization. This is largely attributed to the absence of inter-stage communication overhead and pipeline ineffi- ciencies/ bubbles that can arise from resource imbalance in disaggregated setups.
D. Optimizer Details
We recapitulate the configuration optimization problem:
max fp. b, s) - 3 cost(p)
(2)
(p.b,s)EX'
From a cost-efficiency perspective, disaggregated serving is most beneficial when latency SLOs are negotaatii..... In contrast, when maximizing throughput per dollar is the primary goal and latency constraints are minimal, aggre- gated configurations are often more cost-effective. This can be attributed to the reduced communication cost and orchestration overhead that result in higher GPU utilization.
For the search space X, there could be implicit constraints during the configuration search. For example, the total number of GPUs must not exceed the available resources (e.g., 8 or 16). Alternatively, if the cloud service provider intends to fully utilize all available 8 GPUs, an implicit constraint could enforce the number of GPUs to be exactly 8. These constraints serve to reduce the search space. The system configuration involves parallelization configurations p. maximum batch size configurations b, and scheduling configurations These variables are vectors, where each element corresponds to the configuration of an individual instance. Each instance is capable of handling requests independently, including managing (sub-)workers for ten- sor parallelism and pipeline parallelism. Instances within a stage process different requests in parallel, a concept re- ferred to as data parallelism. Note that p. b, and s can have variable lengths, as the number of instances is itself configurable parameter. For the i-th instance, we denote its stage as Stage, E PE D}, where E represents the encoder stage, P represents the processing stage, and D represents the decoder stage.
Specific to the EPD disaggregation strategy, the encoding and prefill stages are predominantly compute-bound, As result, the primary performance gains from disaggregation stem from stage-specific finetuning of batching, schedul- ing, and parallelization strategies, especially IRP for paral- lelizing encoding load of a request across multiple GPUs. However, these gains are sensitive to pipeline inefficiencies, for instance pipeline bubbles can erode the latency benefits. Thus, careful tuning of resource allocation and responsive worker migration is required to ensure these stages are well- balanced for a given model and workload.
C. Future Work
Let p denote the vector of parallel con- figs for all instances. For the i-th instance, it is a prefill or decoding instance, then its config p; includes: p." the number of GPUs used for tensor parallelism; and p.P. the number of GPUs used for pipeline parallelism. If it is an encoding instance, considering IRP does not re- quire communication, which is better than TP, we only use IRP. Therefore for encode, we overload the symbol p = p.PP to denote the number of GPUs used for IRP. If the cost per GPU is a constant c, then the total cost is:
Parallelization:
Although the encode and prefill stages have similar compute characteristics, their memory usage profiles differ substan- tially. This asymmetry presents promising opportunity to explore heterogeneous hardware configurations. For in- stance, using high-memory high-compute GPUs for prefill and low-memory but high-compute units for encoding could further enhance efficiency and cost-effectiveness in disag- gregated setups.
Another area of exploration is reducing the overhead of visual token migration, which can be a bottleneck in mul- timodal disaggregated pipelines. Compression techniques, particularly those leveraging the inherent redundancy in vi- sual tokens as identified in token pruning literature (Alvar et al., 2025), may significantly reduce transfer latency and cost.
cost(p) = c > (D.T
(p. x "
Max Batch Size determines how many requests are pro- cessed simultaneously during the encoding, prefill, and decoding stages. Let b denote the batch size config for all instances. For i-th instance, by is its max batch size. This config involves a trade-off between latency and through- put. Larger batch sizes improve throughput by enabling parallel processing but may increase latency if the stages become compute-bound. bound.
Lastly, disaggregated serving introduces new possibilities in privacy-aware inference (Liu et al., 2020; Huang et al., 2020; Singh et al., 2022). especially in edge-cloud environ- ments. For example, encoding could be performed on edge devices, keeping raw, privacy-sensitive images local. Only the encoded representations would be transmitted to the
Scheduling involves two main decisions: First, which workers should each request be assigned to? Second, how should the order of requests be determined within a
14

---

Efficiently serving LMMs using EPD Disaggregation
worker queue? To solve these decisions, we adapt strate- gies from the DistServe framework (Zhong al., 2024). In the encoding stage, when a request arrives, it is as- signed and pushed to an instance queue. Between different stages, global queues are used, and each available engine pulls proactively from the queues. Possible assignment strategies in the encoding stage include hound-Robin or Least-Loaded First for assigning requests. Once request is assigned to a worker's queue, can apply ordering strategies like sest-come-first-sere or shortest-job-fing or more complex strategies that prioritize requests based on their Service Level Objectives (SLOs). For simplicity, we constrain that all instances within the same stage share the same scheduling strategy.
aggregation. For efficient data movement, the system em- ploys NCCL for inter-node GPU communication and asyn- chronous CudaMemepy for intra-node transfers, ensuring smooth operations without disrupting GPU computations across the disaggregated EPD framework.
E.1. parametrss in EPD system
We conducted our experiments using a cluster of 8 NVIDIA A100 GPUs (82GB). Each server was equipped with 128 CPUs and 1TB of RAM. The CUDA version was 12.2. Flash attention-2 was used for the attention implementation. We use FP16 precision for all experiments.
To ensure fair comparison, we standardized key performance-affecting settings of the inference engine across all baselines. Specifically, these include a block size of 16; a maximum of 2048 blocks per request; context tokens capped at 49,152, and decoding tokens at 81,920 per batch. The scheduling policy for all stages was set to Trist-Com--Firs--Soce
E. Implementation Details
EPD is a fully capable distributed serving system for LMMs, comprising several key components: a load estimation mod- ule resource allocation module, a RESTful API frontend, and multimodal-awar orchestration layer. The entire framework is implemented with a mix of Python and C+++ CUDA implementations, ensuring superior scalability and performance. To facilitate integration, we repurpose the distributed execution engine from vLLM, which supports numerous popular LLMs and LMMs, allowing easy adapta- tion of new models into our disaggregated framework with minimal effort.
(FCFS). Further, to allow enough resources for eeemory-heave multimodal requests to exe- cute, KV cache GPU utilization was set to 50%, and the maximum number of multimedia data of 32 was imposed per prompt. The size of the multimodal cache was fixed to 3000 across all models, and the vLLM inference engine was run in eager mode. Finally, for the vL.LM inference engine, we used the version 0.6. 0.6.t. which represents a stable version for multimodal inference.
The API interface adheres to OpenAl's 's multimodal specifi- cations, enabling users to specify parameters such as output length, temperature, and multimodal data inputs.
In our online experiments, requests were sent to the infer- ence engine using a Poisson arrival process with fixed X, representing the number of requests per second. Each trial was executed until 100 requests were completed, ensuring sufficient data for consistent performance analysis. Further, to optimize TTFT and TPOT in this latency-sensitive set- ting, we disabled batching by setting batch sizes to for the encoding, prefill, and decoding stages.
The scheduler is specifically designed for the disaggregated EPD framework, dynamically managing batch sizes and enabling asynchronous execution of the encoding, prefill, and decoding phases. The load estimation module ensures efficient GPU allocation across these phases, adapting to changing workload demands in real time.
E.2. LMM Details
Our repurposed distributed execution engine uses Ray actors to implement GPU workers, which manage multimodal and key-value caches, and coordinate the independent execution of the encoding, prefill, and decoding tasks. Furthermore, it supports 3D parallelism, incorporating Data Parallelism (DP), Tensor Parallelism (TP), and Pipeline Parallelism (PP) to maximize resource utilization and scalability.
MiniCPM-V V 2.6 integrates SigLip-400M vision encoder, comprising 400 million parameters, with Qwen2-7B lan- guage model, containing 7.6 billion parameters, culminat- ing in 8 billion parameters. It excels in tasks involving single-image, multi-image, and video comprehension, even surpassing GPT-4V in these domains (Yao et al., 2024).
The orchestration layer includes custom CUDA kernels op- timized for parallelism in the encoding and prefill phases. These kernels enable efficient management of paged mul- timodal caches and ensure seamless asynchronous transfer of multimodal tokens between encoding and prefill GPUs. The orchestration layer oversees the execution of encod- ing, prefill, and decoding instances, handling tasks such as request distribution, KV cache transmission, and result
vision encoder with 300 million parameters, paired with in- ternlm2.5-7b-chhe language model comprising 7.7 bil- lion parameters, totaling 8 billion parameters. Similarly, Intern V1.2-26B combines an BitemmiiT-B---Sp--VV sion encoder with billion parameters and an intern1m2- chat-20b language model containing 20 billion parameters, resulting in 26 billion parameters.
Intern VL2-8B features an Inter
vi-
15

---

Efficiently serving LMMs using EPD Disaggregation
Encode
Prefill
Latency Breakdown (9)
1.00
1.00
0.75
0.75
0.50
0.50
0.25
0.25
0.00
0.00
1
2
4
8
images/Request
Images/Reques:
(a) GPU
(b) NPU
Figure 12: Breakdown of latency for encode and prefill stages using the I1tern model across varying numbers of images per request. Subfigures (a) and (b) show results on GPU and NPU, respectively. Light green denotes encode latency and light blue indicates prefill latency. NPUs demonstrate distinct latency characteristics compared to GPUs as input size increases.
E.3. SLO Criteria
follows: hatch sizes for the E, P, and D stages are 2, 1, and 128, respectively; the number of workers in each stage 6, 1, and and IRP enabled. We uniformly random sample 10 configs from the search space. To maintain the same computational cost, all GPUs utilized across different configs. Consequently, the search space is constrained to leverage 8 GPUs, which can be enforced through rejection sampling. The sample mean of the performance metric (goodput, TTFT, and TPOT) is presented in Table 5. Recall that the goodput metric is defined as the maximum request rate while maintaining an SLO attainment of less than 90%. For fair comparison, When evaluating the TTFT and TPOT, we maintain the same request rate of 1.25 r/s, which is same the goodput of the EPD system.
Table 9 outlines the SLO criteria for TTFT and TPOT across various models and different numbers of images per request (#I/R). These criteria are empirically derived based on the characteristics of the underlying models, such as the compu- tational complexity of the MME and L.M.. We also consider what is realistically achievable by both our method and the baselines on fixed number (8 GPUs) used in experiments. Across models, as #I/R increases, there is an approximately linear increase in the TTFT criteria due to the higher encod- ing load and additional tokens generated during the prefill stage. In contrast, TPOT requires only minor adjustments since it not directly impacted by changes in #I/R.
#1/R
MiniCPM-V 2.6
InterVVL 8B
Intern VL 26B TTFT
TTFT
TPOT
TTFT
TPOT
TPOT
F. Extending EPD to NPU
2
1.40
0.04
1.20
0.05
3.50
0.07
We extend our EPD framework to a cluster of Neural Pro- cessing Units (NPUs) for two purposes: (1) investigate the impact of disaggregating (E) from (P) NPUs which tend to have higher encode-to-decode latency ratio and (2) to demonstrate the generalizability of EPD.
4
2,60
0.04
2.40
0.06
7.05
0.08
6
3,90
0.06
3.55
0.09
11.00
0.95
8
5.10
0.06
5.00
0.18
15.00
0.15
Table 9: TTFT and TPOT values (in seconds) used as SLO thresholds for different models and image counts per request (#I/R). Respective values are shown for MiniCPM-V 2.6, InternVL22881 and Intern VL2-26B models.
F.1. NPUs having higher encode-to-prefil latency ratios
When investigating architectures and potential benefits of an EPD-NPU framework, we first profiled the LMMs tested in this paper for their encoding and prefilling latencies to understand the impact of the characteristics described above. We calculated the average encode and prefill stage latencies of 10 requests. Across the different LMMs, NPUs spent more time encode than prefill across these models when compared to GPUs. Figure 12 shows the latency breakdown between encode and prefill for Intern VL2-8B across a range of #I/R. Across the models and different #I/R, we found a ~ 10-20% larger encode-to-prede latency ratios in NPU than in GPU. This trend was consistent across different large multimodal models (LMMss), indicating that NPUs tend
E.4. Details for the Ablation Experiment of Offline Optimizer
In this experiment, 100 User requests arrive in real-time with each request containing 6 images, and they all request for the MiniCPMy model. For simplicity, we limit the con- figuration space to a restricted search space, where each data-parallel (DP) worker uses the same batch size configu- ration, and both TP and PP are fixed to 1. We explore the batch size configurations for workers across different stages, the number of DP workers in each stage, and the decision enable IRP. The config identified by the optimizer is
16

---

Efficiently serving LMMs using EPD Disaggregation
spend greater proportion of time in the encoding phase compared to GPUs.
request-specific information (e.g., request ID, sampling parameters, block IDs pull) and the remaining imple- mentation mirrors that of EPD-GPU. This container-based deployment allows our framework to be easily scalable and portable to the cloud, providing intriguing future works regarding efficient utilization and scheduling of resources for users.
Since the encode in NPU is a larger portion of the TTFT when compared to GPU, we hypothesized that we would see more benefit from EPD and IRP towards NPU. Thus, we were motivated to adapt EPD for NPU.
E.2. Implementation Details for NPU
F.3. Experimental Settings for NPU
The EPI--NPU system was implemented similar man- ner to our EPD-GPU system as described in Appendix E. All EPD-NPU experiments utilize IRP. The EPD-NPU dis- tributed serving system developed and deployed on cluster of 910B3 NPUs, each with 64GB of high bandwidth memory. There are two major differences between the NPU and GPU implementations:
For EPD-NPU, We use eight 910B3 NPUs and evaluate the system by running the controlled experimental setup (as described in the experiments section of the main paper) an online setting. The number of input tokens was 22, length of output tokens was 10 with the end-of-sequence token ignored. The SLO requirements (8.5 and 0.12 for TTFT and TPOT, respectively) were also selected in similar manner as the GPU experiments. The maximum model length was set to 27000, and KV Cache utilization was set to 83%. EPD-NPU also used version of Ascend-vLLM (version 0.6.3.post1) with CANN version 7.6. Unless otherwise stated, the hyper-parametens are identical to those described in E.
AsLend-vLLM: EPD-NPU leverages Ascend-vLLM rather than vLLM which is an adaptation of vLLM Ascend NPU ware that utilizes the Compute Architec- ture for Neural Networks (CANN) software stack rather than CUDA.
Container-based deployment: Fach encode, prefill, and decode instance deployed separately an API for the scheduler engine call. These instances run in data paral- lel mode. The scheduler engine simply sends POST request to the corresponding API with some minimal
We evaluate EPD-NPU using Intern VL2-8B LMM against two baselines: (1) Ascend-vLLM and (2) Ascend-vLLM PD disaggregation for the heavy workload of 8 images (4032x3024 resolution) per request.
17


Reviews:
[{"review_id": "EfCVxIp5HO", "reviewer": "Reviewer_afT5", "summary": "After rebuttal: Thank the authors for the detailed comments. I'm keeping my recommendation of 3.\n\n----------------------------------\nThe authors propose a novel Encode-Prefill-Decode (EPD) disaggregation framework for Large Multimodal Model (LMM) inference. The proposed approach decouples encoding and prefill stages, so that GPU resources can be allocated more efficiently based on the jobs' encode, prefill, and decode stages. The EPD framework includes intra-request parallelization (IPR) to shard a request into independent sub-requests, an optimizer to handle resource allocation, and a dynamic role switching mechanism to monitor the bottlenecks and switch the role of a GPU if needed. Evaluations are conducted on MiniCPM-V 2.6, InternVL2-8B, and InternVL2-26B using synthetic workloads and NextQA benchmark dataset on a cluster of 8x NVIDIA A100 GPUs, and achieved 71% reduction in time to first token (TTFT) and 57% reduction in end-to-end throughput (E2ETP).", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful and constructive feedback. Below, we address the specific concerns.\n\n---\n\n### **Q1: Use of the Term “LMM” and Generalization Beyond Vision-Language Models**\n\nWe deliberately use the term **LMMs** instead of “VLMs” to reflect the **general applicability of EPD across modalities**, not just vision.\nEPD is designed for any model that consists of a modality-specific encoder followed by an LLM decoder. This includes **audio**, **video**, and other input types where encoders operate independently on local patches or chunks. \n\nEven for sequential modalities like audio or video, temporal structure is typically **reconstructed by the LLM using positional embeddings (e.g., RoPE)**—not by the encoder—making **intra-request parallelism and stage disaggregation effective across modalities**.\n\nTo support this, we have extended our experiments to the **audio domain**, using the `ultravox-v0_3` model (`LLaMA3.1-8B` backbone) in an **online, encode-heavy setting** with **24 audio files per request**. Table 1 shows that EPD **outperforms both vLLM and DistServe across request rates**, achieving **higher SLO attainment** and **higher goodput**.\n\n**Table 1. Online Audio Benchmarking: SLO Attainment Rate (↑)**  \n*SLO targets: TTFT ≤ 2.0s, TPOT ≤ 0.025s*\n\n| Method (Config) \\ Rate (req/s) | 0.1 | 0.25  | 0.5  | 1  | 1.1  | 1.15 | Goodput (r/s) |\n|------------------------|------|------|------|------|------|------|------|\n| **vLLM**  (4D)        | 0.99   | 1.00   | 0.99   | 0.91   | 0.87   | 0.87   |   1.01  |\n| **DistServe**  (3P1D)   | 0.99   | 0.94   | 0.89   | 0.72   | 0.69   | 0.68   |  0.45  |\n| **EPD (ours)**  (2E1P1D)    | **0.99**   | **0.99**   | **1.00**   | **0.96**   | **0.93**   | **0.93**   |  **1.16**  |\n\nThese results, along with our video experiments (see response to Reviewer QF6q), confirm that **EPD generalizes effectively beyond vision**, justifying the broader **“LMM”** terminology. We will make this clearer in the final version.\n\n---\n\n\n### **Q2: Perception as a Natural Extension of DistServe with added Encoding**\n\nOur system is inspired by DistServe’s disaggregation approach, but **disaggregating encoding from prefill** presents **unique challenges and opportunities** beyond prefill-decode separation.\n\n- First, we identify **Intra-Request Parallelism (IRP)** as crucial for reducing TTFT in encoding-heavy workloads. Otherwise, the benefits of disaggregation can be offset by communication overhead, since both E and P stages have similar characteristics (compute-bound).\n- Second, **optimized resource allocation** is essential, as different LMMs exhibit diverse compute profiles (e.g., LLaVA is prefill-heavy, while MiniCPM is encoding-heavy), requiring stage-specific tuning of resources.  \n- Third, in dynamic workloads, growing **pipeline imbalance can nullify IRP gains**, making **dynamic role switching** vital to maintain high throughput and efficiency.\n\nThese system-level insights are foundational to the design and performance gains demonstrated by EPD.\n\n---\n\nAdditionally, feel free to explore our released code at this anonymized [link](https://drive.google.com/drive/folders/1cEyGCPw54EkgBjZs73m-SZq51M2etD0X).\n\n---\n\n### **Q3: Formalism of Equation (2), concrete example, and Search Space**\n\nHere is an explanation and we will expand this in the appendix and include examples in the released code.\n\nEquation (2) is a **general formulation** for expressing optimization objectives in a disaggregated inference setting. It is intentionally designed to be flexible across deployment scenarios. \n\nIt captures the trade-off between system performance (e.g., throughput or SLO attainment) and deployment cost, and is designed to flexibly adapt to different deployment scenarios. Below, we explain two practical real-world use cases:\n\n**1. Offline Optimization:**  \nConsider a small business that wants to deploy with 8 GPUs, the user’s goal is to maximize **goodput** under a fixed hardware budget. In this case, the cost term in Equation (2) is constant (since all GPUs are active), and the objective simplifies to maximizing throughput. The user collects historical workload traces and feeds them to our resource allocation module. The optimizer searches over: (1) number of instances per stage (E/P/D), (2) degree of parallelism (IRP, TP, PP), (3) batch sizes, and (4) scheduling strategies. This yields the best static deployment setup for the given workload.\n\n**2. Online Serving with Autoscaling:**  \nIn dynamic cloud settings, Equation (2) incorporates both **performance (e.g., SLO attainment)** and **cost (e.g., number of active workers)**. The objective might be to balance **SLO attainment** (e.g., 95% of requests meeting latency targets) against **cost** (e.g., number of active GPU workers). The system collects request traces periodically and re-solves Equation (2) to adapt to changes in workload—for example, shifting from 3E2P3D to 2E1P5D when decoding becomes the bottleneck.\n\n---", "claims_and_evidence": "Overall I find the claims clear and convicing. The authors proprose the EPD framework to address the LMM inference efficiency issue brought by the additional encoding stage to process raw multimodal inputs (compared to LMMs), which could require substantial GPU resources to encode them into tokens. The experiments show that the proposed EPD framework can improve multiple memory and latency metrics. Ablation studies are also conducted to show the effectiveness of each component. \n\nI am not a big fan of the term Large Multimodal Model being used in the paper (I understand that InternVL calls themselves LLM). I think Vision Language Models would be more appropriate, as the advantage of the proposed EPD framework is largely based on the efficiency of the image encoding stage enabled by parallelism. We don't know if the same efficiency can be achieved for other modalities, e.g., audio.", "methods_and_evaluation_criteria": "The authors use both synthetic datasets and the NextQA benchmark dataset for evaluation, which make sense to me. Metrics such as the supported number of images per request, the batch size, TTFT, TPOT, SLO attainment rate, and goodput are used to evaluate the proposed EPD framework, which are relevant to the claim being made.", "theoretical_claims": "N/A. This is an application paper.", "experimental_designs_or_analyses": "The authors mainly compare the proposed approach with DistServe [Zhong et al., 2024], which implements the prefill decode (PD) disaggregation framework for LLMs (so no/little encoding compared to this paper). Overall the experimental designs are sound to me. In multiple experiments the authors use a resolution of 4032 x 3024, which arguably favors the proposed EPD approach due to the encoding cost.", "relation_to_broader_scientific_literature": "I'm not an expert in this area. The paper can feel like a natural extension of the DistServe framework (plus Intra-Request Parallelism), which is also a disaggregation framework for LMMs, except that the proposed EPD framework takes into account of the additional the encoding stage. To me the results are sound and decent but not surprising. A lot of these are based on the fact that image and video inputs can be patchified and parallelized efficiently, and I wonder if this still holds for more sequential type of modalities.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "The contributions of the paper are decent but not surprising. To be fair I feel VLM would be more appropriate in the title compared to LMM, as the advantage of the proposed EPD framework is largely based on the efficiency of the image encoding stage. We don't know if the same efficiency can be achieved for other modalities, e.g., audio.", "other_comments_or_suggestions": "N/A.", "questions_for_authors": "- Regarding Eq. (2), I think this falls into formalism and I cannot get much insight from it. I wonder if the authors can provide a concrete example. How should one use the formula in practice? How would the authors define the search space in their experiments?\n\n- Have the authors considered other modalities such as audio in the evaluation? If not, do you think the proposed EPD framework can be generalized to other modalities?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Oof4LqcXY0", "reviewer": "Reviewer_QF6q", "summary": "To address the negative impact of the multimodal encoding stage on key Service Level Objectives (SLOs), this paper proposes the Encode-Prefill-Decode (EPD) Disaggregation framework, which allocates the encoding, prefill, and decode stages to independent computing resources. Specifically, this work introduces: (1)A new mechanism for caching multimedia tokens to improve transmission efficiency. (2)A novel method for parallelizing encoding loads within the same request. (3)A resource allocation module optimized for disaggregated inference. (4)A dynamic role-switching approach to adapt to changing workload characteristics.\nExperiments on MiniCPMv-2.6, InternVL2-8B, and InternVL2-26B demonstrate that, compared to non-disaggregated methods, the proposed framework significantly improves memory efficiency, batch processing capacity, image handling capability, KV cache utilization, TTFT, and E2ETP. Overall, this paper presents a clear motivation, an intuitive and well-founded approach, and compelling experimental results, demonstrating strong practical applicability.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks you! Glad to hear that the **motivation, design, and experimental results were found to be compelling**.\n\nBelow we address the specific questions raised.\n\n---\n\n### **Q1: Why does TPOT perform better in the w/o Opt setting compared to w Opt in Table 4?**\n\nThanks for pointing it out, here is the clarification. The optimization strategy may sometimes trade off TPOT to improve the global objective -- goodput.  In the experiment from Table 4, for EPD with an optimizer, the optimizer's objective is solely to maximize the goodput metric. Therefore, the small decrease in average TPOT is due to the optimizer automatically finding a better tradeoff between average TTFT and TPOT, given the workload statistics of encoding, prefill, and decoding.\n\nSpecifically, the optimizer sacrifices average TPOT only slightly (from 0.025s to 0.031s) while significantly reducing average TTFT (from 4.48s to 2.12s). The final TTFT value of 2.12 s falls well below the 3.9 s requirement (recall that our experiment setup requires TTFT and TPOT to be 3.9s and 0.06s, respectively, as shown in Table 6). \n\n---\n\n### **Q2: Why is TTFT lower in the w/o Switch setting in Table 5?**\n\nThank you for this observation, we’re happy to clarify. The **lower TTFT in the w/o Switch setting** is a direct consequence of its **static resource allocation**: the system remains locked in the initial **5E1P2D** configuration, which was optimized offline for short outputs (50 tokens). This setup overprovisions the **encoding stage**, resulting in a slightly lower TTFT of **1.33s** vs **1.42s** for EPD.\n\nHowever, this comes at a significant cost. The static configuration fails to adapt when the workload shifts to **longer output requests (500 tokens)**, and decoding becomes the primary bottleneck. As a result, the **TPOT deteriorates sharply to 0.12s**, more than **2.4× worse** than EPD (**0.05s**).\n\nBy contrast, the **full EPD system**, equipped with **dynamic role switching**, detects this workload shift and **migrates 3 encoding workers to decoding**, transitioning to a **2E1P5D** configuration. While this reduces encoding capacity and leads to a slightly higher TTFT (**1.42s**), it **significantly improves decoding throughput**, yielding a much lower TPOT and overall faster response.\n\nUltimately, this leads to a **2.2× improvement in average end-to-end latency**: **28.01s for EPD vs. 61.10s for w/o Switch**. This tradeoff highlights the strength of our dynamic reconfiguration strategy: it **prioritizes end-to-end latency** by resolving the true bottleneck in real time, even if that means marginally a particular stage. \n\n--- \n\n\n### **Q3: How does the method perform on other datasets like Video-MME?**\n\nThank you for the thoughtful suggestion. To strengthen the paper, we conducted additional experiments on the **Video-MME benchmark**, which complements NextQA (open-ended questions) by focusing on multi-choice video QA with diverse video lengths.\n\nWe evaluated **SLO attainment** (TTFT ≤ 3.1s, TPOT ≤ 0.025s) on 100 randomly sampled examples using **MiniCPM-v2.6**, with 64 uniformly sampled frames per video, adhering to the MiniCPM frame setting reported on the Video-MME leaderboard. Results are shown below:\n\n**Table 1: SLO Attainment Rate (%) ↑ under Different Request Rates**\n\n| Method \\ Rate (req/s)  | 0.5  | 1.0  | 1.5  | 2.0  | 2.25 | 2.5  | 3.0  |\n|------------------------|------|------|------|------|------|------|------|\n| **vLLM**              | 70   | 52  | 46   | 29   | 32   | 21   | 3   |\n| **DistServe**         | 72   | 57   | 31   | 28   | 24   | 19   | 0   |\n| **EPD (ours)**        | **99**  | **100**   | **99**  | **87**   | **65**  | **43**   | **9**  |\n\n**EPD consistently outperforms vLLM and DistServe** across all rates, demonstrating strong generalization to temporal multimodal workloads. This highlights the effectiveness of our proposed EPD beyond the NextQA dataset for temporal multimodal inputs like video.\n\nWe also evaluated **TTFT latency** under varying video lengths (8–64 frames), at 1 req/s:\n\n**Table 2: TTFT (s) ↓ under Different Frame Counts**\n\n| Method \\ # Frames/Video | 8  | 16  | 32  | 64  |\n|------------------------|------|------|------|------|\n| **vLLM**              | 0.42   | 0.82  | 1.59  | 3.11   |\n| **DistServe**         | 0.42   | 0.81   | 1.54   | 3.08   |\n| **EPD (ours)**        | **0.24**  | **0.30**  | **0.49**  | **1.00**   |\n\nEPD **consistently achieves significantly lower TTFT latency** than both vLLM and DistServe across all frame counts. Moreover, the **performance gap widens with increasing video length**—for instance, at 8 frames, EPD reduces latency by **42.9%**, and at 64 frames, the reduction reaches **67.5%** compared to DistServe. These results highlight EPD’s **superior scalability and robustness** under increasingly demanding video processing workloads.\n\n---\n\nAdditionally, feel free to explore our released code at this [link](https://drive.google.com/drive/folders/1cEyGCPw54EkgBjZs73m-SZq51M2etD0X).\n\n---", "claims_and_evidence": "This paper claims three key contributions: Intra-Request Parallelization (IPR), Resource Allocation Optimization Modeling, and Dynamic Role Switching. After reading the full paper, I think all claims are clear and well-supported by evidence.", "methods_and_evaluation_criteria": "The proposed method is reasonable and its effectiveness is validated using mainstream models and datasets.", "theoretical_claims": "This paper primarily focuses on engineering-oriented inference optimization and lacks theoretical foundations. However, I believe this does not diminish its value.", "experimental_designs_or_analyses": "The experiments in this paper primarily focus on the following four aspects: (1) End-to-end SLO evaluation (EPD significantly outperforms the baselines) (2)Computational resource utilization efficiency (comparison of maximum supported images and batch sizes) (3)Ablation studies on key modules (IPR, DRS, and optimizer) (4)Analysis of TTFT and TPOT (key inference performance metrics comparison). However, some in-depth analyses are missing in this paper. For example, why does TPOT perform better in the w/o Opt setting compared to w Opt in Table 4? Similarly, why is TTFT lower in the w/o Switch setting in Table 5? I hope the authors can provide reasonable explanations for these phenomena. Additionally, the paper only conducts experiments on the NextQA dataset. How does the proposed method perform on other more popular datasets, such as Video-MME? More comprehensive evaluations would strengthen the validity of the results.", "relation_to_broader_scientific_literature": "N/A", "essential_references_not_discussed": "N/A", "other_strengths_and_weaknesses": "N/A", "other_comments_or_suggestions": "N/A", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ofG7ub8WKZ", "reviewer": "Reviewer_rZNR", "summary": "--- score updated from 3 (weak accept) to 4 (accept) after the rebuttal. ---  \n\n----  \n\nRecently Disaggregated Inference was proposed to use separate nodes for prefill and decoding whilst serving LLMs. This allows to more easily control SLO times like inter token latency or time to first token. \n\nThe paper generalizes this setting form LLMs to Multimodal models, proposing to additionally disaggregate the Encoding phase of images or videos. This idea is very relevant and timely, but also conceptually straightforward. The paper conducts many studies ablating the different aspects of the solution and convincingly making a case for the proposed solution.\n\nTheir key innovations are a/ an asynchronous token transfer between the workers, b/ Intra-request parallelism which allows to further speedup encoding by running independent image encodings of a single request across workers c/ an alogrithm to allocate the resources between encoding/prefill/decode and d/ an approach to dynamically switch the roles.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the thoughtful comments, recognition of the importance and timeliness of our work, and the positive assessment of our empirical validation. Below we address the main concerns raised:\n\n---\n\n### **Q1: Cost per served request and disaggregation tradeoffs**\n\n> _\"I would propose to also look at cost per served request. One criticism of Disaggregation is that it is not the most cost-efficient solution... it could be clearly discussed as a limitation with also data illustrating when this happens.\"_\n\n**Response:**  \nWe appreciate this insightful comment and fully agree that **disaggregation is not always the most cost-efficient approach**, particularly in scenarios where **tight Service-Level Objectives (SLOs) are not required** and **compute resources (e.g., GPUs) are limited**. For instance, in **offline batch processing**, where throughput is the sole objective, aggregated architectures can achieve **higher GPU utilization** and benefit from **lower inter-stage communication overhead**, making them more cost-effective.\n\nWe will **explicitly discuss this limitation in the revised version** and provide a more nuanced view of when disaggregation may or may not be cost-efficient.\n\nThat said, recent work such as [1] demonstrates that **disaggregation can sometimes yield even better throughput**, especially when **compute requirements of stages is significantly different**. By optimizing batching, parallelization and scheduling strateges, systems like DeepSeek-V3 (which disaggregates prefill and decode) are able to achieve better throughput. Our **Throughput in Offline Settings** analysis in the Supplementary Material supports this finding, showing that disaggregation can reduce computation bubbles and improve utilization when stages are well balanced.\n\nHowever, in **cloud-scale, interactive serving systems**, where **individual SLO guarantees (e.g., time-to-first-token, time-per-output-token)** become crucial for user experience, disaggregation is critical for cost efficiency. \n\nIn such cases, **ensuring SLOs with fully aggregated systems requires significant over-provisioning**, which can lead to **higher overall cost** than disaggregated setups. Disaggregation allows each stage to be optimized independently, reducing interference and tradeoffs between stages, that result in better control of per-request level statistics like TTFT and TPOT.\n\nThis is evident from experiments in Section 4.1. EPD has better SLO attainment than aggregated setups for same number of GPUs. Hence, for aggregated setups to match the SLOs of EPD, they need to overprovision, resulting in higher overall cost.\n\n[1] DeepSeek-AI. DeepSeek-V3 Technical Report. arXiv 2412.19437, 2025\n\n\n---\n\n\n### **Q2 & Q3: Lack of code release, especially for resource allocation and role switching**\n\n> _\"The major weakness is that no code is provided.  Especially for the Optimized resource allocation and the dynamic role switching.. I consider code and reproducibilty crucial for this work.\"_\n\n**Response:**  \nWe understand and agree with the reviewer that **code and reproducibility are vital**, especially for software systems that propose architectural and scheduling innovations.\n\nWe had planned to release the code as part of the camera-ready version (post-acceptance), but in light of the reviewer’s concern, we have **made the code available during the review period** at the following anonymous link:\n\n> **Anonymous Code Repository:** [link](https://drive.google.com/drive/folders/1cEyGCPw54EkgBjZs73m-SZq51M2etD0X) (released under Apache license)\n\nThis release includes:\n- The **full implementation of EPD**, including stage clusters, worker orchestration, cache management, and bridges\n- Our **custom resource allocator** with multiple backends (runtime, simulation)\n- The **dynamic role switching mechanism**, including metrics collection and migration orchestration\n- Scripts for **reproducing main experimental results**\n- **Readme file** located at <project_root>/README.md\n\nWe hope this supports the reviewer’s confidence in the rigor and reproducibility of our work.\n\n---", "claims_and_evidence": "Yes the empirical claims are supported by experimental results.", "methods_and_evaluation_criteria": "Yes the Evaluation mostly makes sense.\n\nI would propose to also look at cost per served request. One criticism of Disaggregation is that it is not the most cost-efficient solution. This is fine, the authors are also not necessarily claiming this, but it could be clearly discussed as a limitation with also data illustrating when this happens.", "theoretical_claims": "There are no theoretical claims", "experimental_designs_or_analyses": "The experiments seem mostly adequate, see also above. However, the code is not provided. In particular for the optimized resource allocation it is left unclear how the algorithm works.", "relation_to_broader_scientific_literature": "Prior work is appropriately discussed. The current submission fits timely into this evolving field", "essential_references_not_discussed": "n/A", "other_strengths_and_weaknesses": "The major weakness is that no code is provided. Especially for the Optimized resource allocation and the dynamic role switching this is crucial IMO. There are not enough details provided to even reproduce the results of the paper. Since this paper mainly proposes an  ellaborate efficient software for a rather simple scientific problem, I consider code and reproducibilty crucial for this work.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "Can you please include the code in your rebuttal? Then I will support acceptance.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The submission initially got three Weak Accepts. After the rebuttal discussion, one reviewer increased his score from Weak Accept to Accept, while the other two reviewers maintained their Weak Accepts but remain positive about the submission. After reading the reviews, I think the proposed approach EPD can bring interest in the community and can have practical benefit in industry. Because of these reasons, I recommend acceptance of the paper. I would just encourage the authors to polish the final paper with the feedback provided by the reviewers.
Decision: Accept (poster)