SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs
Authors: Zhicheng Li
Shuoming Zhang
Jiacheng Zhao
Siqi Li
Xiyu Shi
Yangyu Zhang
Shuaijiang Li
Donglin Yu
Zheming Yang
YUAN WEN
Huimin Cui
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal large language models; Inference optimizations; Infrastructure
Primary area: infrastructure

Abstract: Recent multimodal large language models (MLLMs) marry modality-specific
vision or audio encoders with a shared text decoder. While the encoder is compute-
intensive but memory-light, the decoder is the opposite, yet state-of-the-art serving
stacks still time-multiplex these complementary kernels, idling SMs or HBM in
turn. We introduce SpaceServe, a serving system that space-multiplexes MLLMs:
it decouples all modality encoders from the decoder, and co-locates them on the
same GPU using fine-grained SM partitioning available in modern runtimes. A
cost-model-guided Space-Inference Scheduler (SIS) dynamically assigns SM slices,
while a Time-Windowed Shortest-Remaining-First (TWSRFT) policy batches en-
coder requests to minimise completion latency and smooth decoder arrivals. 
Evaluation shows that SpaceServe reduces time-per-output-token by 4.81×
on average and up to 28.9× on Nvidia A100 GPUs. SpaceServe is available at
https://github.com/gofreelee/SpaceServe

SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs
Zhicheng Li
Shuoming Zhang 1.2
Jiacheng Zhao 122 Siqi Li
Xiyu Shi
Yangyu Zhang!?
Shuaijiang Li
Donglin Yu Huimin Cui
Zheming Yang!
Yuan Wer
State Key Lab of Processors, Institute of Computing Technology, Chinese Academy of Sciences, China 2 University the Chinese Academy of Sciences
University of Aberdeen Beijing University of Technology
The University of Illinois Urbana-Champaign "CCORESIMM CO.LTD.
Abstract
Recent multimodal large language models (MLLMs) marry modality-specifff vision or audio encoders with shared text decoder. While the encoder is compute- intensive but memory-light, the decoder the opposite, yet state-of-the-an serving stacks still time-multipler these complementary kernels, idling SMs or HBM in turn. We introduce SpaceServe, a serving system that space-multiplexes MLLMs: it decouples all modality encoders from the decoder, and co-locates them on the same GPU using fine-grained SM partitioning available modern run- times. A cost-model-guided Space-Inference Scheduler (SIS) dynamically assigns SM slices, while a Time- Windowed Shortest-Remaining Remaining- -Fist (TWSRFT) policy batches encoder requests minimise completion latency and smooth decoder rivals. Evaluation shows that SpaceServe reduces tine-per-outpu-tee average and up to 28.9x Nvidia A100 GPUs. SpaceServe is available bettps//githhn com www./oooeeeee//eeeeeeoooooe Saaeleeee/pcceSeeee
by 4.81
Introduction
Multimodal large language models (MLLMs), for example (iee2-VL[B3) and other recent systems[30 3 33 27,. have graduated from lab demos to production services that field image- grounded questions. interpret charts, and reason over audio video snippets. Conceptually, MLLM marries the unprecedented linguistic capability of large text-only foundation model with a set of modality- speeiiicc encoders that translate pixels, waveforms, or frames into the token space the decoder already understands, This "encoder + shared-text-deooder blueprint is now standard across state-of-the. models B I8 36].
As foundation-model decoders continue to improve "for free" (e.g.. via larger pre-training corpora 21
17 34| 9 or better post-training 26 25 28)), attention shifting toward the encoders. These front-end components are becoming both more powerful and more complex [33 36 2, handling higher-resolution images, longer audio clips, and even variable-length video. Consequently, an MLLM is effectively two decoupled yet complementary neural networks: a compute-intensive, memory-light encoder paired with memory -hungry, compute-light decoder. This pronounced resource asymmetry remains largely unaddressed by today's serving stacks.
32
Serving MLLMs inherits every headache of text-only LLM serving tight tail-latency 15 targets, massive K V-cache footprints [35] 29 [3]. dynamic batching [38]. and adds a new one: modality- specific encoders. Most production stacks [13 41 TO remain tuned for single-modality workloads.
Optimizations such as prefilling decoding disaggregation
chunked prefill [T] boost through-
Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

put for text models assuming homogeneous stream of decoder kernels. That assumption breaks once a vision or audio encoder, with a completely different resource profile, enters the pipeline.
Lacking encoder-aware machinery, state-of-the-art systems such as vL..M [3] or TGI [0]] revert to rime-multiplexing when they serve MILMM: the GPU runs the vision/audio encoder first, then switches context and launches the text decoder, and so on. Encoder and decoder kernels therefore contend for the accelerator sequentially, losing the complementary compute-versus -ersuss-roosss balance that could otherwise be exploited.
Our key insight is that encoders and decoders have complementary resource footprints: the compute cycles a decoder leaves idle are exactly what an encoder can exploit, while the memory an encoder scarcely touches is what the decoder hungers for to house its KV-cache. We therefore advocate space-multiplexing instead of traditional time-multillexing- kernels simultaneously on the same GPU, each claiming just the compute and memory it truly needs.
encoder and decoder
We embody this idea in SpaceServe, a split-encoder serving that: 1) decouples all modality encoders from the shared text-decoder; 2) dynamically routes multimodal requests through modality- aware scheduler that adapts to workload mix; 3) achieves meaceeuuliilexinng by co-locating encoder and decoder kernels each GPU whenever their complementary compute-memory footprints align. Our contributions are as follows:
Split-encoder serving architecture: We introduce SpaceServe, the first system that decouples all modality-specific encoders from the shared text decoder and provides a GPU runtime that co-locates their kernels via ppace-multiplexing. A lightweight, cost-model-guides co-location policy packs encoder and decoder kernels whenever their resources footprints fit.
TWSRFT encoder scheduler: To tackle bursty multimodal workloads, design Time- Windowed
(TWSRFT) scheduler that orders encoder requests within each time window by their remaining work. This preemptive policy minimizes encoder completion latency and smooths the arrival pattern seen by the shared decoder, improving end-to-end tail latency without harming encoder performance.
Comprehensive evaluation: We evaluate SpaceServe with 4 size of state-of-the-an MLLMs [36 31 on three GPU combinations (1 A10,,, 40,, 8* 100) using mixed workloads sampled from two well-known benchmarks for image [40] and video 8 Empirical results demonstrate that SpaceServe cuts tine-per-outputttoeen (TPOT) by 4.81x on average and up 28.9x.
2
Motivation: From time-Multiplexing to space-Multiplxxing
2.1 Preliminary
The canonical MLLM stack is organized around two core modules: 1) Modality-specific encoders that translate raw visual, audio, or other non-text inputs into high-dimensional embeddings. 2) A shared text-I.MM decoder that performs autoregression generation over these embeddings. A lightweight pre-processing stage precedes the encoders, normalising each modality into the expected tensor format-tokenising text, slicing images into fixed-size patches, converting audio into waveforms, thus the encoders can operate on consistent representation. For instance, a vision encoder typically ViT-based [6] structure encode visual patches into visual tokens, which are aligned with text tokens in a unified representation space for integration and then processed by the LLM for reasoning services and coherent outputs generation. A classic MLLM architecture is depicted by FFigure la
2.2 Resource footprint complementary: A quantitative analysis
Our major insight is a clear resource-footprint complementarity between modality encoders and the shared text decoder. We substantiate this with a quantitative study of arithmetic intensity All--loating--oin operations per transferred byte (FLOPs/ / Shoeiin that encoders are compute-rich but megory-light, whereas decoders are memory-hungry yet compute-light. This complementarity underpins the space- space- multiplexing design that follows.
Due to space limit, the detailed computation of AI for MLLMs are put in Appendix D
Table I reports the arithmetic intensity (AI, FLOPs/byte) of Qwen 2-VL-7B at two image resolutions-511 512 x 512 and 2048 2048. Processing such dynamic input sizes is crucial for accuracy, yet it complicates serving. Two key take-aways emerge:

---

Time Multiplexing
E., E.
Ps. Ps
D.
Du D.
yellow
Chunked Prefill
LLM Decoder
F, yellow puppy
Space Multiplexing
F,
Modality Encoder
Encoder Decoder
Text tokens
D
D
Dg
D
User: The picture
Modality tokens
Output tokens
Timeline
GPUs
(a) Core components MLLM: encoder de- coder.
(b) From Mume-Muliillxiing space space- Maluce--uliiiexinn
Figure I: Architecture of MLLMs and how vllm and SpaceServe serves MLLMs
Table Arithmetic intensity (FLOPs byte) of each serving stage Qwen2-VL-7B on A100 GPU as image resolution increases from 512 x 512 to 2048 x 2048. Higher Al means the kernel is more compute-bound; lower Al implies more memory-boume
Stage
AI @ 512 x 512
AI @ 2048 2048
Encoder encoding
338.21
8826.97
Decoder prefilling
72.13
1212.43
Decoder decoding
0.887
0.887
Encoders and decoders are complementary. At both resolutions the encoder encoding remains increases from 338.21 to 8826.97 FLOPs/bye byte resolution increases, but even 338.21 21 still far exceeds the decoder's AI. By contrast, decoder decoding stage stays memory-bound at 0.887 FLOPs/ //yts./ytc. Consequently. an encoder's surplus memory bandwidth can satisfy a decoder's KV-cache needs, while a decoder's idle SM cycles can be reclaimed by encoder kernels precisell the synergy exploited by our space-multiplexigg scheduler.
Input dynamics reshape the resource footprint. Increasing resolution quadruples the number of 14 14 vision patches (e.g., 256 - 1024 patches), inflating memory traffic slower than compute and increasing encoder AT by roughly 4x, as Al of encoder is quadratically related to the number of patches, detailed in Appendix C The Encoder phase changes from 338.21 - 8826.97, and the decode phase is nearly flat, so the relative gap between encoder and decoder shrinks as images grow. Any serving stack that statically partitions GPU resources will mis-size one stage or the other once the workload shifts; a scheduler must adapt per request, co-locating kernels according their current AI rather than fixed worst-case estimate.
In short. dynamic, nigh-resolution inputs amplify the pecoder-decodee asymmetry and underscore the need for an adaptive, space-multiplexed serving strategy.
2.3 Toward DpulieeMl Serving
Existing serving stacks are mggality-agnostic, so their schedulers treat vision/audio encoders and text decoders unrelated jobs that simply take turns the GPU. The result is classic timemlxlipigxiin (Figure 1b] top): the system either runs a batch of encoders or a batch of decoders, never both, squandering the complementary SM-versus HBM footprints we documented earlier.
State-of-the-ar tweaks such chunked [refill[]] (Figure lb middle) do achieve
prace-mulipplexing-.. momutppleeinng-imeeeeaningg the prefill and decode phases of text requests- but they rely critical assumption: both phases operate on the same modality (tokens) with a shared architecture. Vision and audio encoders violate that assumption; they ingest image patches or spectrogram windows and execute completely different kernel mix, so chunked prefill cannot repurposed overlap them with decoder stages.

---

We therefore argue for true space-multiplexing across modalities ( (Figure 16 bottom). Through GPU-level partitioning runtime and cost-aware scheduler that allocates sub-GPU slices of SMs (streaming multiprocessors) to each request, our approach co-locates encoders and decoders concurrently, turning their complementary resource profiles into tangible throughput and latency gains.
3 SpaceServe: Design
Requests
Modallty Inputs
Testinguts
Cost Model
Co-Locating Stage: Space Inference Scheduler
Encoder/ Decoder
Resource Utility Curve
Schedule
Operator-Loval Resource Utility Curve
STokens, SeeeerssSSeggeeee SSegLen.
Allocate Encoder
Modality Encoder
LLM Decoder
Allocate Decoder
Frofllin
Typrrpammeters
Hardware Info
Modality Scheduler
GPUs
Text Schedule
(TWSRFT)
to
MLLMM Networks
menmul
cate
Allocate to Decoder
Offline
Online
Modality Token Transfer
Figure 2: SpaceServe System Overview
SpaceServe converts the traditional time- time-multipexed serving pipeline into space-multiplexed one by decoupling modality encoders from the shared text decoder and then co-locating their kernels on the same GPUs. By co-locating kernels, hardware resources are optimized, as different kernels demand distinct resource types. For example, compute-intensive kernel utilize more stream multiprocessors (SMs) but require less memory bandwidth, while memory- kernels use the remaining less SMs without losing bandwidth. Strategic sharing of these kernels on the same GPU enhances resource utilization and boosts performance. Figure sketches the high-level architecture of our approach, which decouples encoders from the LLM and co-locate them on the same GPU, SpaceServe achieves ppace-multiplexnng of GPU resources through two complementary design pillars that operate in a split-share fashion: Disaggregated architecture. SpaceServe cleanly decouples modality encoders from the shared text decoder, enabling each component to execute and be scheduled independently.22 GPU runtime with a Space-Inference Scheduler (SIS). A novel runtime characterises the resource footprints of encoder and decoder kernels per request and locates them on the same GPU set. As illustrated Figure 2] the dedicated encoder pool is fronted by time-window scheduler that minimises encoding latency under fluctuating loads. At the heart of the system, the SIS ( 3.3) dynamically partitions streaming multiprocessors (SMs) between encoders and decoders, guided by a cost model (8$ 3.I) trained offline from extensive profiling of representative encoder and LLM workloads. During runtime, the SIS continuously monitors GPU utilisation; whenever new request arrives, rebalances the SM allocation according the model's recommendation, sustaining high utilisation while respecting tail-latency constraints.
3.1 Hierarchical Operator-level Profiling for Resource-Utility Curves
Efficient sub- GPU sharing requires precise performance model that can predict how both encoders and decoders behave under (i) dynamic inputs and (ii) dynamic resource allocations. Two chal- lenges emerge: 1) Input-dependent behaviour: As shown in $ 2 and SSS 2.2, changes in image or video resolution alter the arithmetic intensity (AI) both encoders and decoders, shifting their compute-memont balance. 2) Non-linear scaling with SM slices. The latency or throughput of a kernel does not scale linearly with the fraction of streaming multiprocessors (SMs) receives, making naive proportional models inaccurate.

---

end for These realities motivate the notion of ResourceUUtlitty Curves: explicit mappings from the GPU resources granted component (e.g., SM percentage) to the utility it delivers (e.g.. latency, tokens). Such curves are the foundation of an adaptive serving stack. We construct them via a hierarchical, operator-level profiling workflow ($S 3.1 which samples the performance of individual operators under controlled SM partitions and then aggregates the results to produce end-to-end utility curves for each encoder and decoder variant. The details of controlling SMs allocation are in $$$ 3.3.2 Thanks to the convergence of operators, we mainly focus on two types of operators:
Algorithm 1 Compose Model-Level Sessource-Utllip Curve
Input: ModelComponentTYye MCT
Input:
ModeLArchitecture Arch (details Niayere, moodets etc.)
Input: PrimaryInput Pmn (a specific Lpatches or Sten)
Input: SMCountRange RsM
Input: OpPeriDB
Output: ModeICurve[Nssl] - TotalModelLatency for Pin
1: ModelCurve t 0
2: for Nsw E R&M do
3: TotalModellatenge <-
4: for 1 to Arch.Nioyers do
5: latency < (Latency counter for the ith layer)
6: for Optazer E Arch. BerrchhgeLLayrOOSsMMMMM
i) do DlayerOp Arrdh.gel0pDimeniion(((O latency + latency + OpPerIDB (OprrrDrDD(OPe---/// yppe) DDapeeoppoo end for
7:
M MCT, Pin,
8:
9:
10: Layeroverheads + Art.. Wrrrh.geLLayerOOeeeeeedMMCCCCC i,
Pan, NsM)
latency & latency + Latoverheads
12:
TotalModelLatenny.. < 'I otal Latenccy + latency end for
14:
ModelCurve[Nssat < MotlaModeeLatenoo
16:
17:
return ModelCurve
Profiling GEMM Resource Utility Curve (1) Encoder: Profile various GEMMs involved in FFN layers (e.g., matrices of size Lpatches dene by dene x 4dene) and attention projections (e.g.. Lpatches X denc by dene X dene). For each matrix size, a curve of Latency vs. N sm generated. Fo) Decoder: Profile GEMMs for FFN layers (e.g.. x dum by dum x 4dum), attention projections for the new token, and the LM head. Each results in a Latency vs. sm curve.
Profiling Attention Resrouce Utility Curve:(I)Fo Encoder: Profile the encoder's self-attention mechanism for different Lpatches. This also yields a of Latency vs. Nsm curves, one for each Lpatches. (2) For Decoder: Profile the attention mechanism computing new token's attention against a KV cache of length Slen. This generates Latency vs. Nsm curves for various Slen.
After the profile, we compose the Model-level resource-utility curves by Algorithm
3.2 Encoder Stage: TWSRFT Scheduler
In line with dynamic batching 38 for text only decoders, SpaceServe also incorporate a modality- aware scheduler for batching encoder requests. The choice requests to batch significantly affects latency metrics, such Time to First Token (TTFT) and Time per Output Token (TPOT). For instance, encoding a high-resolution image typically takes longer than processing lower-resolution image embeddings. To minimize latency and boost throughput, batching smaller requests is effective. However, this approach faces two constraints. First, GPU computational capacity limits batching, as encoders compute-ing -ntruueeeeteeivve...m. For example, the NVIDIA A100 has an optimal arithmetic intensity of approximately 161 FLOPs/Byte. The threshold varies across GPU models. Second, prioritizing small requests can starve larger requests, delaying their processing.
To address these challenges, SpaceServe introduces the Windowedd Shortest Remaining Time First (TWSRTF) scheduler, the modality scheduler shown in Figure 2) The TWSRTF scheduler uses
5

---

time-window approach to manage request batching. Requests are collected in input queue within fixed time window. Within this window, requests are batched by size, prioritizing the smallest ones first. The scheduler also monitors batch capacity to ensure its stay within the GPU's limits. The detailed algorithm implementation is presented in Algorithm
Algorithm 2 Windowddddd Shortest Remaining Time First Scheduling for Encoder
Input: Request queue Q, window size w, maximum patches seglengghhhatchhsiittl Output: Encoded batches of requests
1: while Q is not empty do
2: W +- Q.get(w) (Fetch a window of requests)
3:
Sort W by ascending patches
4:
i B - [ Wii.. patches
5:
i 1-1+1
6:
while i < | do
7:
if + Wiid_ < seglengghhaatchenitaeee then
8:
Append W to
9:
8 patchess patches
10:
i i+i+1
11:
else
12:
encoder( [) [Process batch with encoder Remove B from Q
14:
end if
end while
16:
end while
3.3 Co-Locating Stage: Space Inference Scheduler for Space Multiplexing
The Online Space Inference Scheduler is cornerstone of the Co-Locating Stage, dynamically managing request execution flows and the partitions Streaming Multiprocessors (SMs) between encoders and the LLM decoders based on runtime conditions. It oversees the scheduling of incoming requests and ensures seamless coordination with the Modality Encoder Scheduler. When the Modality Encoder Scheduler, using Time-Wine Windowed Shortest Remaining Time First (TWSRTF), dispatches new encoding request, it triggers an action in the Online Space-Multiplex Scheduler. The scheduler utilizes the request's 's metadata, e.g. input sequence length and image resolution, alongside the cost model derived from offline profiling (detailed in SSS 3.1) This enables dynamic allocation of GPU resources of the same processor, particularly SMs, between the encoder task and current LLM decoding operations. Such adaptive, request-aware resource allocation optimizes performance and maximizes hardware utilization for diverse multimodal workloads.
3.3.1 GPU Runtime: Fine-grained Resource Partition
Effective space-multiplxxing hinges on ability to partition a single GPU and allocate resources be- low the device boundary. Modern accelerators runtime, for example, supports CU masks [2] that assign disjoint CU sets to different streams, while NVIDIA GPUs offer comparable streaming multiprocessor control via libsmctrl [4] SpaceServe builds on these sub-GPU partitioning primitives to realise fine-grati nee-grained resource allocation.
expose precisely this functionality. The AMD HIP
3.3.2 Request-aware SM Partition
During the serving phase of the inference engine, design online allocation mechanism based on our profiler to dynamically optimize Streaming Multiprocessor (SM) partitioning.
SpaceServe decomposes the inference service of multimodal large models into three independent processes: CPU process, an Encoder process, and LLM process. The CPU process handles user requests and performs preprocessing of multimodal data. Request information is then dispatched separately to both the Encoder process and the LLM process. Given that Encoder and LLM have distinct resource preferences and requirements, SpaceServe applies different scheduling strategies tailored to each. SpaceServe performs dynamic and efficient GPU resource partitioning tailored to incoming requests. This adaptive allocation is underpinned by our comprehensive Model Resource

---

Utility Curves in SS 3.1, The underlying Streaming Multiprocessor (SM) partitioning algorithm. which is elaborated Appendix Eloperates on the core principle of finding the resource configuration that minimizes the sum of the Encoder's ss and Decoder's execution times.
As for the LLM Decoder, SpaceServe employs ChunkedPrefill [T] with vision cache scheduling strategy for the LLM processing, which is formally described in Algorithm
4 Evaluation
4.1 Experimental Setup
Hardware:
We evaluate SpaceServe on a server which equipped with 8 NVIDIA A100 SXM GPUS and Intel Xcon(R) Gold 6430 CPU.
Models and workloads: We select state-of-the-an MLLMs with varying model size, including Qwen2-VI-2BB (36,, Qwen2-VI.-7B [36], Qwen2.5-VL-3225 [3] and Qwen2-VI.-72B 36) These models were chosen for their proficiency in handling high-resolution images with arbitrary aspect ratios, understanding long contextual sequences. Due to GPU-memory constraints, we run Qwen2- VL-2B and Qwen2-VL-7B on a single GPU, Qwen2.5-VL-32B on four GPUs, and the largest Qwen2-VL--2B on eight GPUs. For the multi-GPU configurations we employ tensor parallelism.
Datasets: We build 1, 1.740-example evaluation set by sampling from MMMU-Pro [40] and Video- MME 8 preserving an ratio of image to video items.
Baseline Systems: We adopt vLLM [3 SOTA serving framework as our baseline. In particular, we evaluate against the latest vLLM v1 architecture, which integrates recent advances such as zero- overhead scheduling and a suite of MLL..-ppecffcc optimizations.
Metrics: LLM inference efficiency centers two critical latency metrics: Time To First Token (TTFT) and Time Per Output Token (TPOT), which we use as primary metrics. Lower values for both TTFT and TPOT indicate better performance.
4.2 End-to-End Evaluation Results
VLLM
SpaceServve(Orrs)
500
450
500
450
TPOT
300
TPOT (ms]
400
35.0
250
900
300
250
350
450
350
400
300
150
100
100
150
150
100
100
50
50
10
Request (rea/s) (a) Qwen2-VL-2B
Request (req/s)
Request rate (req/s)
Request rate (req/s)
[b] Qwen2-VL-7B
(c) Qwen2.5-VL-328
(d) Qwen2-VL-728
Figure 3: TPOT of SpaceServe and vLLM on various models, lower TPOT is better
In the experiment, we increased the request rate from 2 to 10 requests per second. Across various model sizes, ranging from 7B to 72B, SpaceServe consistently and significantly outperforms the vLLM baseline. For instance, while serving the Qwen2-V1.-22 model, vLLM's average TPOT increased from 101 ms at 2 RPS to 365ms at 10 RPS, indicating a significant drop in efficiency. In contrast, SpaceServe demonstrated robust performance, with TPOT increasing only slightly from 8.85ms (2 RPS) to 12.62ms (10 RPS). This represented a substantial improvement, achieving a 28.9, 88.9x reduction in TPOT compared to vLLM at 10 RPS and thereby enhancing service throughput and user experience.
This trend continued across other models 10 RPS. For Qwen2- VL-7B, SpaceServe recorded a TPOT of 40ms compared to vLLM v1's 489ms, delivering a speedup of 12.3x. For larger models with greater resource demands, such as (wwen.55VL-332) on four NVIDIA A100 GPUs, SpaceServe's TPOT was 144ms compared to LLLM's 522ms (10 RPS), a speedup of 3.62x. Similarly, when serving Qwen2-VI--72B on eight NVIDIA A100 GPUs RPS, SpaceServe maintained its advantage with TPOT of 5558mm versus vLLM vl's 489ms, achieving 3.14x speedup.
7

---

Why do we have such an excellent token generation speed: The inherent latency for generating a single token during the decoder phase is typically low, around 10 milliseconds for Qwen2VL-2B. However, vI..-v1's I's time-division multiplexing architecture causes decoding requests to be blocked when the GPU is occupied by an encoder task, which can take several hundred milliseconds, e-g. ms for Owen2VL.22B, This forces decoding operations into a wait state until the encoder task completes, significantly inflating the Time Per Output Token (TPOT) in high request-per-secon (RPS) scenarios due to frequent encoder invocations.
In stark contrast, SpaceServe employs a spatial multiplexing strategy (or a functionally equivalent mechanism that allows for concurrent or dedicated resource allocation for encoder and decoder stages). This architectural design effectively decouples the -hghtthrughhpu decoder operations from the more time inteeintessve encoder tasks. By mitigating these inter-dependencies, SpaceServe ensures that the token generation rate for the decoder remains stable and consistently high, achieving approximately 100 tokens per second, even during frequent encoder activity. This resilience of the decoding pipeline to encoder-induced stalls is the primary contributor to SpaceServe' 's superior performance under demanding, mixed encoder-decoder workloads when measured in TPOT.
Due to page constraints, we put the TTFT results on Appendix B Unlike the sharp TPOT gains, the sime-to-firsttooen (TTFT) remains virtually unchanged between SpaceServe and vLLM for all four models. This is expected: SpaceServe design focuses on harvesting the memory- bandwidth slack present while an encoder is running to accelerate the memory- memory-bbund decoder, boosting steady-state throughput rather than the initial-token latency captured by TTFT.
4.3 Ablation Study
The performance advantages of SpaceServe compared to vLLM are attributed to two primary innovations: (1) the efficient spuce-mulipllexng methodology. (2) the specialized Time- Windowed Shortest Remaining Time First (TW-SRTF) scheduler for the vision encoder. To delineate the individual contributions of these elements, we performed targeted ablation studies, adhering to the experimental setup described earlier.
4.3.1 Ablation Study: Space Inference Scheduler vs. MPS
We ablate our space-multiplxxing design against NVIDIA Multi-Process Service (MPS) 20 using Qwen2-VLL-BB as the test model.
TPOT Latency (ms)
TPOT Latency (ms)
150
SpaceServe
MP5
150
Serve
VLLMscheduler
100
100
50
50
8
10
2
6
10
Request rate (rea/s)
Request rate trea/s)
(a) TPOT compared with MPS
(b) TPOT compared with vLLM scheduler
Figure 4: Ablation study of Qwen-2-VL-7B with TPOT
As shown in Figure 4. the ablation study confirms our hypothesis: Compared vLLMM, the MPS (Multi-Process, Service) version of SpaceServe accelerates the TPOT metric while keeping a largely unaffected TTFT. Our ablation study further evaluates our SpaceServe model, which employs fine-grained allocation strategy, against this MPS version. At an input rate of 2 Requests Per Second (RPS), the MPS version of SpaceServe shows TPOT of 90 ms. This latency increases with rising RPS, reaching peak of 132 ms. In contrast, our SpaceServe (with the fine- fine-grained strategy) achieves a TPOT of 28.25 ms 2 RPS, and its TPOT only reaches a maximum of 40.68 ms RPS increases. This comparison highlights that our fine- ime-grained allocation strategy maintains significantly lower TPOT latency and greater stability under high concurrent requests, achieving a TPOT speedup of up to 3.3x relative to the MPS version. Moreover, as depicted in Figure Figure for both approaches nearly identical, indicating that these TPOT enhancements not compromise first-token latency.
Figure 5b) the TTFT
8

---

The substantial performance improvement stems from alleviating resource contention at the SM microarchitectural level. When Encoder and LLM processes are co-located using only NVIDIA MPS, their respective CUDA kernels, comprised of multiple warps, may be concurrently scheduled by the MPS server onto any available SM. Within single SM, these distinct workloads then vie for limited resources such as arithmetic units, register file space, L.1 cache, and shared memory bandwidth. The disparate nature of these tasks-the Encoder being typically compute-bound and the LLM decoding phase often exhibiting bursty memory access patterns and varied computational demands- can lead to suboptimal SM utilization. For example, high register usage by one task could limit the number of active warps (occupancy) for the other, or differing memory access patterns could lead L1 cache pollution, increasing effective memory latency.
By contrast, SpaceServe's 's SM partitioning dedicates distinct sets of SMs to the Encoder and LLM processes. This spatial isolation minimizes direct inter-process contention for intra-SM resources. Each process can therefore more effectively exploit the full capacity of its allocated SMs, leading to improved instruction issue rates and better sustained occupancy for specific workload characteris- tics. This dedicated execution environment allows for more streamlined processing within each SM partition, reducing stalls and ultimately translating to a lower TPOT and an enhanced throughput.
4.3.2 Ablation Study: TWSRFT Scheduler vs. vLLM Default Scheduler
We ablate our redesigned encoder scheduler -uill for split-encoder architectures using the Qwen2- VL-7B model.
The results demonstrate a clear advantage for our custom scheduling approach under increasing load. At low concurrency of 2 Requests Per Second (RPS), the PpaccseervevLLLMsccdduleee exhibited a TPOT of 30.73 ms, which was only slightly higher than our full SpaceServe system's TPOT of 28.25 ms. However, the concurrency increased to 4 RPS, the TPOT for SpaceServe- VLLMscheduler rose more sharply to 55.88 ms, whereas our SpaceServe system maintained a TPOT of 40.18 18 ms. This performance disparity became significantly more pronounced at 10 RPS: the TPOT for SpaceServe- MLLMschedullr surged to 114.22 ms, while our SpaceServe system impressively sustained low TPOT latency of 39.8 ms, showcasing its stability and efficiency at higher throughputs. The observed degradation in TPOT for the Spacssarve-LLLstthhhueee MMsshhedllrr configuration be attributed to the vLLMM scheduler's lack of an independent batching strategy for the encoder component. When the aggregated batch size of vision tokens processed by the encoder reaches a critical threshold, any further increase in the number of concurrent requests leads to a substantial rise in the overall execution time for the encoder stage. This, in turn, inflates the average processing time per request. Consequently, the LLM process does receive a sufficient timely stream of vision tokens, which acts as a bottleneck and significantly slows down the generation of subsequent output tokens, thereby increasing the TPOT.
4.4 Performance on Modern MoE-based Architectures
To demonstrate SpaceServe's 's effectiveness on the latest generation of MILMs, we evaluated its performance on cutting-edge models featuring sparse MoE layers: DeepSeck-VI. and Kimi-VL. The experiments were run under a high-concurrency load of 10 requests per second.
The results, presented in Table 2 show that SpaceServe delivers substantial performance gains, We achieve 4.08x TPOT speedup on DeepSeek- VL2 and remarkable 9.84x speedup on Kimi-VL.
This dramatic performance improvement, especially on Kimi-VL, VL, attributed to its exceptionally innpute-intensive
compute- intensive vision encoder, which creates significant imbalance with its memory- bound MoE decoder. This precisely the scenario where SpaceServe's 's spatial multiplexing provides the greatest benefit, by co-locating the complementary kernels to maximize GPU utilization.
4.5 Performance Scaling with Input Resolution
Modern MLLMs increasingly use high high-resolution inputs to improve understanding. This makes the vision encoder substantially more compute-intensive creating major bottleneck for traditional serving systems. We evaluated SpaceServe from 224x224 to 2Kx2K under high concurrency (10 RPS). As shown in Table vL..M's TPOT rises sharply with resolution because the long-running encoder blocks the decoder, whereas SpaceServe's TPOT stays low and stable.

---

Table 2: TPOT speedup on MoE-based MLLMs under a high-concurrency workload (10 RPS).
Model (MoE-based)
Framework
TTFT (ms)
TPOT (ms)
TPOT Speedup
vLLM-v1
5597
122.5
Baseline
DeepSeek-Vin V1.2
SpaceServe (Ours)
4495
30.0
4.08x
vLLM-v1
57900
482.0
Baseline 9.84x
Kimi-VI.
SpaceServe (Ours)
56323
49.0
Crucially, the performance advantage of SpaceServe scales with the computational load. As illustrated in Table the TPOT speedup escalates from 1.37x on low-resolution inputs to an impressive 12.39x at 2K resolution. This demonstrates that spatial multiplexing is most effective precisely where is most needed: mitigating the latency of compute-heavy tasks. Far from being a limitation, high-resolution inputs highlight SpaceServe's 's fundamental strength in harnessing resource heterogeneity deliver efficient performance for the next generation of high-fidelity MLLMs.
Table 3: Performance comparison across varying input resolutions at 10 RPS. As resolution increases, SpaceServe maintains stable, low TPOT, while the baseline's 's latency degrades significantly.
Resolution
Framework
TTFT (s)
TPOT (ms)
TPOT Speedup
vLLM-v1
0.09
23.6
Baseline
224 X 224
SpaceServe (Ours)
0.09
17.2
1.37x
vLLM-v1 SpaceServe (Ours)
0.21
38.6
Baseline 1.49x
512 X 512
0.21
25.9
vLLM-v1 SpaceServe (Ours)
15.31
212.6
Baseline 5.97x
IK IK
15.54
35.6
vLLM-v1
139.35
470.7
Baseline 12.39x
2K X 2K
SpaceServe (Ours)
135.76
38.0
Related Work
Disaggregated Serving. Promising for large models, disaggregated serving (e-g., SplitWise [22]
DistServe 42 DéjaVu (31) decouples prefill/decode stages, mitigating interference for better TTFT/TPOT control. These L.LM-fouueed systems often neglect the MLLM-speccfic encoding step.
Even recent extensions (Pensieve [39] Mooncake [23], PD-Serve
[II with advanced KV cache
management offer limited MLLM applicability.
Multi-modality Model Serving. Serving multimodal models (MLLMs) typically relies on adapting
LLM systems (e.g., vLLM [33. SGLang [11) using recent MLLM inference code [24]. Adapting
LLM systems hits LMM encoding bottlenecks with rich multimedia. Techniques like KV cache
eviction [4 19] and compression 12 are often model/scenarro-speciis (e.g.. Inf-MLLM's [19
single-GPU streaming) and may miss cloud SL.Os. Early disaggregation ideas (e.g., EPD 30 lack
SpaceServe's comprehensive scheduling and partitioning.
Complementarity with Sparse MILM Architectures. SpaceServe complements sparse multi- modal architectures. including Mixture-of-Experr designs (7 [6]. Sparsity can lower per-request compute, but also increases resource heterogeneity by leaving hardware assigned to inactive experts underutilized. SpaceServe exploits this opportunity by co-locating the compute-intensive encoder of one request with the sparse. memory-bound decoder of another. using spatial multiplexing to fill idle resources and raise overall utilization
Conclusion
This paper introduces SpaceServe, novel MLLM serving system featuring two key innovations: fine-grained GPU resource management and an advanced scheduling framework. This design enables concurrent low-latency inference and high-throughput for multimodal L.MMs, outperforming SOTA
systems by up to 28.9
10

---

Acknowledgments
This work was supported by the National Key R&D Program of China (Grant No. 2024YFB4505603) and by the National Natural Science Foundation of China (Grant Nos. U23B2020, 62090024, and 62302479).
References
[1] Amey Agrawal, Nitin Kedia, Ashish Panwar, Jayashree Mohan, Nipun Kwatra, Bhargav Gula- vani, Alexey Tumanov, and Ramachandran Ramjee. Taming Throughput-Latency tradeoff in LLM inference with Sarathi -ervee In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 1 117-134, Santa Clara, CA, July 2024. USENIX Association.
[2] AMD Corporation, Setting the number of compute units, https://rocm: dww.cmmddoc.... amd com/ em/Latest/hoo--o//sttiiii cus html Accessed: 2025-05-16.
[3] Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang. Jun Tang, Humen Zhong, Yuanzhi Zhu, Mingkun Yang. Zhaohai Li, Jianqiang Wan, Pengfei Wang, Wei Ding, Zheren Fu, Yiheng Xu, Jiabo Ye, Xi Zhang. Tianbao Xie, Zesen Cheng, Hang Zhang, Zhibo Yang, Haiyang Xu, and Junyang Lin. Qwen2.5-vl technical report, 2025.
[4] Joshua Bakita and James H. Anderson. Hardware compute partitioning on nvidia gpus. In 2023 IEEE 29th Real-Time Time and Embedded Technology and Applications Symposium (RTAS), pages 54-66, 2023.
[5] Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. Flashattention: Fast and memory-efficient exact attention with io-awareness. Advances in neural information processing systems, 15:16344-16359, 2022.
[6] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
[7] Jiarui Fang, Yang Yu, Chengduo Zhao, and Jie Zhou. Tranffrranssssmrs::: an efficient GPU serving system for transformer models. In Proceedings of the 26th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming, pages 389-402, 2021.
[8] Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal Iims in video analysis. arXiv preprint arXiv:2405.21005, 2024.
[9] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint a7X., 2024.
[10] Hugging Face. text-
Text generation inference. 2022. Accessed: 2025-05-06.
/ttps://githul http://gggttbb...cgggggggggaeeee com/huggingface/
[11] Yibo Jin, Tao Wang, Huimin Lin, Mingyang Song, Peiyang Li, Yipeng Ma, Yicheng Shan, Zhengfan Yuan, Cailong Li, Yajing Sun, Tiandeng Wu, Xing Chu, Ruizhi Huan, Li Ma, Xiao You, Wenting Zhou, Yunpeng Ye, Wen Liu, Xiangkun Xu, Yongsheng Zhang. Tiantian Dong, Jiawei Zhu, Zhe Wang, Xijian Ju, Jianxun Song. Haoliang Cheng, Xiaojing Li, Jiandong Ding. Hefei Guo, and Zhengyong Zhang. P/d-serve: Serving disaggregated large language model scale, 2024.
[12] Hao Kang, Qingru Zhang, Souvik Kundu, Geonhwa Jeong, Zaoxing Liu, Tushar Krishna, and Tuo Zhao. GEAR: An efficient error reduction framework for KV cache compression LLM inference. In Mchdi Rezagholizadch, Peyman Passban, Soheila Samiee, Vahid Partovi Nia.

---

Yu Cheng, Yue Deng, Qun Liu, and Boxing Chen, editors. Proceedings of The 4th NeurIPS Efficient Natural Language and Speech Processing Workshop. volume 262 of Proceedings of Machine Learning Research, pages 305-321. PMLR, 14 Dec 2024.
[13] Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph Gonzalez, Hao Zhang, and lon Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the 29th Symposium on Operating Systems Principles, SOSP '23, page 61 1-26,, New York, NY, USA, 2023. Association for Computing Machinery.
[14] Yuhong Li, Yingbing Huang, Bowen Yang, Bharat Venkitesh, Acyr Locatelli, Hanchen Ye, Tianle Cai, Patrick Lewis, and Deming Chen. Snapky: Llm knows what you are looking for before generation. In Globerson, L.. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomoczz,, and C. Zhang, editors, Advances in Neural Information Processing Systems, volume 37, pages 22947-22970. Curran Associates, Inc., 2024.
[15] Zhuohan Li, Lianmin Zheng, Yinmin Zhong, Vincent Liu, Ying Sheng. Xin Jin, Yanping Huang, Zhifeng Chen, Hao Zhang, Joseph E Gonzalez, et al. (AlpaServe): Statistical multiplexing with model parallelism for deep learning serving. In 17th USENIX Symposium on Operating Systems Design and Implementation (OSDI 23), pages 663-679, 2023.
[16] Weixin Liang, LILI YU, Liang Luo, Srini lyer, Ning Dong. Chunting Zhou, Gargi Ghosh, Mike Lewis, Wen tau Yih, Luke Zettlemoyer, and Xi Victoria Lin. Mfxture-of-traasfomme A sparse and scalable architecture for multi-modal foundation models. Transactions on Machine Learning Research, 2025.
[17] Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint 14149774 2024.
[18] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[19] Zhenyu Ning, Jieru Zhao, Qihao Jin, Wenchao Ding, and Minyi Guo. Inf-mllm: Efficient streaming inference of multimodal large language models on single gpu, 2024.
[20] NVIDIA Corporation. Multi-process service (mps).
mps/ hte/indexx.hhr html, Accessed: 2025-05-16.
[21] OpenAL. Gpt-4 technical report, 2024.
[22] Pratyush Patel, Esha Choukse, Chaojie Zhang, Aashaka Shah, fñigo Goiri, Saeed Maleki, and Ricardo Bianchini. Splitwise: Efficient generative I1m inference using phase splitting. In 2024 ACM/IEEE 51 Annual International Symposium on Computer Architecture (ISCA), pages 118-132, 2024.
[23] Ruoyu Qin, Zheming Li, Weiran He, Jialei Cui, Feng Ren, Mingxing Zhang, Yongwei Wu, Weimin Zheng, and Xinran Xu. Mooncake: Trading more storage for less computation KVCache-centric architecture for serving LLM chatbot. 23rd USENIX Conference on File and Storage Technologies (FAST 25), pages 155-170, Santa Clara, CA, February 2025. USENIX
Association.
[24] Haoran Qiu, Anish Biswas, Zihan Zhao, Jayashree Mohan, Alind Khare, Esha Choukse, fñigo Goiri, Zeyu Zhang, Haiying Shen, Chetan Bansal, Ramachandran Ramjee, and Rodrigo Fonseca. Modserve: Scalable and resource-efficient large multimodal model serving, 2025.
[25] Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. In Thirty-seventh Conference Neural Information Processing Systems, 2023.
[26] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms, 2017.
[27] ByteDance Seed. Seed1.5-vl technical report, 2025.
12

---

[28] Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. Deepseekmath: Pushing the limits of mathematical reasoning in open language models, 2024.
[29] Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism, 2019.
[30] Gursimran Singh, Xinglu Wang, Yifan Hu, Timothy Yu, Linzi Xing, Wei Jiang. Zhefeng Wang, Xiaolong Bai, Yi Li, Ying Xiong, Yong Zhang, and Zhenan Fan. Efficiently serving large multimodal models using epd disaggregation, 2025.
[31] Foteini Strati, Sara McAllister, Amar Phanishayee, Jakub Tarnawski, and Ana Klimovic. Déjävu: Kv-cache streaming for fast, fault-tolerant generative Ilm serving. In Proceedings of the 41 International Conference on Machine Learning. ICML'24. JMLR. 2024.
[32] Gemini Team. Gemini: A family of highly capable multimodal models, 2025.
[33] Kimi Team. Kimi-vl technical report, 2025.
[34] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 239.213222.9971, 2023.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Eukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
[36] Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. Qwen2-vl: Enhancing vision- language model's perception of the world at any resolution, 2024.
[37] Zhiyu Wu, Xiaokang Chen, Zizheng Pan, Xingchao Liu, Wen Liu, Damai Dai, Huazuo Gao, Yiyang Ma, Chengyue Wu, Bingxuan Wang, Zhenda Xie, Yu Wu, Kai Hu, Jiawei Wang. Yaofeng Sun, Yukun Li, Yishi Piao, Kang Guan, Aixin Liu, Xin Xie, Yuxiang You, Kai Dong. Xingkai Yu, Haowei Zhang, Liang Zhao, Yisong Wang, and Chong Ruan. Deepseek-v12: Mixture-of-experts vision-language models for advanced multimodal understanding. 2024.
[38] Gyeong-In Yu, Joo Seong Jeong, Geon-Woo Kim, Soojcong Kim, and Byung-Gon Chun. Orea: A distributed serving system for Transformer-Bass generative models. In 16th USENIX Symposium on Operating Systems Design and Implementation (OSDI 22), pages 521-538. Carlsbad, CA, July 2022. USENIX Association.
[39] Lingfan Yu, Jinkun Lin, and Jinyang Li. Stateful large language model serving with pensieve. In Proceedings of the Twentieth European Conference on Computer Systems, pages 144-158. 2025.
[40] Xiang Yue, Tianyu Zheng, Yuansheng Ni, Yubo Wang, Kai Zhang, Shengbang Tong, Yuxuan Sun, Botao Yu, Ge Zhang, Huan Sun, Yu Su, Wenhu Chen, and Graham Neubig. Mmmu-pro: A more robust multi-discipline multimodal understanding benchmark, 2024.
[41] Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu. Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient execution of structured language model programs. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[42] Yinmin Zhong, Shengyu Liu, Junda Chen, Jianbo Hu, Yibo Zhu, Xuanzhe Liu, Xin Jin, and Hao Zhang. DistServe: Disaggregating prefill and decoding for goodput-optimizet large language model serving. In 18th USENIX Symposium on Operating Systems Design and Implementation (OSDI 24), pages 193-210, Santa Clara, CA, July 2024. USENIX Association.
13

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have clearly stated the main claims made in the abstract and introduction, which accurately reflect the paper's contributions and scope. The claims are supported by theoretical and experimental results, and we have discussed the limitations of our work.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitation of SpaceServe in Appendix
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
14

---

Answer:
[NA]
We use well-known roofline model to guide the system design, no new theory proposed.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Our experimental results are reproducible, we conducted multiple experiments in an 8-card A100 server environment, and the code will be open-sourced later
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
15

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will open source the SpaceServe after paper acceptance.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //iissccc/pill wttp:ccppppcccceeeeeeeeee..c.ett httpa/////iieeeoeetuuuuiiiissss Coooeooomisssssseeeeeeee www.siiiccciiiiiccoooooooiisiiii for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification:
We detailed the settings in Section 4.1.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The experiments very computationally exhaustive, therefore impractical to create enough data for meaningful statistical tests.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
16

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We state it in S$ 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics http:////eerrrpp..cuuutt/tt//tta
Answer: [Yes]
Justification:
We follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work of inspirational significance to the reasoning acceleration of multimodal large models, see $ 6
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
17

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: the paper poses such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We have properly credited the creators original owners of assets used in the paper.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
18

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: Our work does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
We do not use crowdsourcing or human subjects in our research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification:
We do not use crowdsoureing or human subjects in our research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
19

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: core method development in this research does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
hutpe:///oorrps.cc////fffrrrrrrr
20

---

A Limitations and future work
While SpaceServe effectively addresses the challenge of extended Time Per Output Token (TPOT) in multimodal large language model inference, another critical performance indicator, Time To First Token (TTFT), remains significant consideration. TTFT directly reflects the system's 's initial responsiveness and is paramount for a positive user experience. In its current state, SpaceServe maintains TTFT performance comparable existing state-of-the-art frameworks, such as vLLM- vl. However, substantially reducing the execution latency of single encoder pass the primary determinant of TTFT continues be demanding task. Optimizing this single-encoder latency to further enhance TTFT represents an important avenue for our future research.
B Experiments for TTFT metric
Table 4: Time To First Token (TTFT) comparison across different models with varying Request Per Second (RPS), lower TTFT, better
Method
Time To First Token (TTFT) (s) L
RPS
2
4
Owen-22VL-22B
vLLMM
12.068
16.180
18.655
23.035
26.038
SpaceServe
13.324
17.515
21.114
24.726
27.437
Qwen-2-VL-7B
vLLM
19.492
22.033
30.306
34.745
395
SpaceServe
18.832
22.419
32.050
36.533
39.086
Qwen-2.5-VL-33BB
vLLM
16.553
32.631
41.849
46.440
47.547
SpaceServe
17.131
33.242
42.489
47.144
48.966
Qwen-2-VL
vLLM
14.872
26.647
35.717
39.780
42.067
SpaceServe
14.810
26.633
36.321
38.378
43.834
SpaceServe
SpaceServe
TTFT T Latency (s)
MPS
IFFT Latency (
vLLMscheduler
30
30
20
20
10
10
2
4
10
2
4
6
10
Request rate (req/s)
Request rate (req/s)
(a) TTFT compared with MPS
TTFT compared with vLLM scheduler
Figure 5: Ablation study of Qwen-2-VL-7B with TTFT
C
Arithmetic Intensity Modeling for Encoder
Taking vision encoders for example, the patch generation characteristics be formulated as:
H
W
H''
W'
H¹ W' =
Lpatches
x
(I)
5
L
21

---

where S denotes image padding strides, H denotes the original image height, H' denotes the nor- malized height, W denotes the original image width, and W" denotes the normalized width. And I. denotes the patch size, /patches denotes the total patches after preprocessing. During the encoder stage, the FLOPss can be formulated as:
Vvisinn
(2Lpatches Drision papptches P + 4L patches dvision
Lo de &erision) * Layers (2)
Embedding
Attention
FFN
The overall Arithmetic Intensity can be approximated by:
FLOPsvision
Memory weight
D Arithmetic Intensity Modeling for LLM
LLM Prefill Stage Before autoregressive token generation, the Large Language Model (LLM) processes the entire input prompt (with sequence length Sprompt) in a single, parallel forward pass. This 'prefill' stage is essential for computing the initial set of Key- (KV) states for all tokens in the prompt. To optimize this process, particularly for managing the memory footprint and computational cost of the attention mechanism, techniques like Grouped-Query Attention (GQA) are often employed. These initial KV states, potentially generated using GQA, are then cached and utilized by the subsequent decoding steps.
LLM Prefill: FLOPs. Let Lum the number of LLM layers, dum be the hidden dimension (consistent with the decoding stage notation), Sprompt be the input prompt length, and V be the vocabulary size. The FFN expansion factor is (typically e = 4). The FLOPs for the Transformer backbone during prefill are primarily from self-attention and FFN computations:
Self-Attention per layer: R (8S + 4S pr FLOPs. This encompasses Q.K.V projections (655 propp dimm). attention score calculations (QK?) and application to Values (4S prompt ditm), and the output projection (2S di..))
FFN per layer: pr FLOPs. For e 4, this 168 promp
The total FLOPs for the Transformer blocks during prefill (FIOPs Pref [ilTs//soorrnmreeeeee are:
FLOPsPrefil Tranfffrree = Litm ((8 4c) promptt di + 45 promp
(4)
Assuming c 4:
FLOPsPre Pttlisse_aaaaeaeeeaeeaee fro_Frrmeerrmrr Lum (24S prompt + 4S or
(5)
If output logits computed for the entire prompt (e.g., for training certain inference strategies), the LM head adds F F'LOPsLM_Her Meadd Heaf__rrfftl 2ppromprdum V.
LLM Prefill: Memory Access. Memory operations during prefill include:
Model Weights (Memorueiights LLM))
Accessing the model parameters pwwpppprraaaaaaa LLM X
Naota__ params params_LLMM LLM, where Veenoryweightt. LLM
syjes_per_parraettr
Input Data Activations: Reading input prompt embeddings and handling intermediate activations for the Sprompt length sequence across layers.
KV Cache Generation: Writing the KV pairs for all Sprompt tokens. The size of this cache Syromps x Limm x 2 x dum pytes_per__at element.
is Memoryyv _ww_ch__ppooneee
LLM Prefill: Arithmetic Intensity (AI). The Arithmetic Intensity for prefill is Al Equation 5) (Memory Access for Weights, Input, Activations, and KV Cache Write).
Lum ((8 + Sproomp + +45prompa ditm))
Alprefal
(6)
Memory A
22

---

The 4Lim prompt Sprampp dit term in the FLOPs (Equation 5 means that for long prompts (S (Sprompt) ) the prefill Al can substantial, potentially making this phase compute- intenssve. This contrasts with the characteristics of the subsequent decoding phase.
In contrast, the Arithmetic Intensity of the LLM component is significantly lower. The LLM decoding stage operates autoregressively, generating one token at a time. Its computational characteristics are modeled follows:
LLM Decoding: FLOPs (Per Token Generation) For the generation of a single token, let Lum be the number of LLM layers, dum be the hidden dimension, Sten be the current sequence length (context from previously generated tokens), and V be the vocabulary size. The primary FLOPs contributions are:
Transformer Decoder Layers (with KV Cache): For each token, each layer involves:
Self-Attention (new token Q, K, V projections; attention with cached K, V): Approxi- mately Sdiu + 4Stendumm FLOPs. This includes Q, K, V projections for the current token (=s 6d((.) ). interaction with the Sten cached tokens (zs 4Siendits for QK and score-V multiplication), and the output projection (Rs 22d(m.)
Feed-Forward Network (FFN): Typically 16d FLOPs (assuming a 4 x dum interme- diate expansion).
Thus, per layer FL.OPs are approximately 24d + 48tendum.
LM Head: Mapping the final hidden state to vocabulary logits: Rs 2d V FLOPs.
The total FLOPs for generating single token (FLOPsLLA, deceodeeeeeeeeeeeeeeeeeeee can be expressed as:
FLOPsLLMM M_d_odde token 2 Lum (24d 4St + 2d V
A common simplified approximation for FLOPs per token (especially when Sien is moderate) is 2 X N Narrems_ _oo_nonnneee embeddind where Nparams_nve non embedding are the non-embedding parameters of the LLM, primarily from FFN and attention projection matrices.
LLM Decoding: Memory Access (Per Token Generation) dominated by:
Memory access during LLM decoding
Model Weights (Meeoryweightt_LL)) Parameters of the Transformer layers and LM head.
Memoryuecighta_LLM LLM param_LL x bytes_ fatts_peer_patttee per_meeeeeter
(8)
KV Cache (MemoryyV_cndee canhe): Stores Key (K) and Value (V) vectors for Sten previous tokens across Tatm layers.
Memorykv _aache Sten X Lutm x 2 x dum x gyyte_ppr_eeement
The critical dynamic memory access per token involves reading from and writing to this KV cache, along with accessing the relevant model weights for the current token's 's computation.
LLM Decoding:
Arithmetic Intensity (AI) The Arithmetic Intensity for LLM decoding is
FLOPs decode tokkn// M emory BocooyyAccess Access per token- If we primarily consider the model weights for memory access (as their total size loaded and parts are accessed for each token generation), simplified Al is:
N Nporams_nne
ALLLM decode parrmms_tttee botol_ J.L.M X fotta_tooaaa LLMS, and FLOPs for core
(10)
Given that N Nparams_ cateesson__eeeeeeee _on_eneeddiing is large fraction of Nperams_ _otal, matrix multiplies are roughly 2x parameters involved, if we consider all parameters:
Nparams_toted fotal LLM
2
ALLLM Aleooed decode
(11)
pparams, totallsssstt_____eeoooooos total_ot___aa. LLM x
tites_per_parameer
For FP16/BF16 (2 bytes/parameter), A decode Rs FLOP/Byte. This low Al signifies that LLM decoding is predominantly hhnnorybaandwith bound. The substantial memory footprint of weights and the dynamically accessed KV cache, relative to the per-token computational work, contributess to this characteristic.
23

---

E
Chunked Prefill Scheduling with Vision Cache Check
Pre-Scheduling Check: Prior each scheduling iteration, the scheduler examines the Quedernender resuts to determine any requests have completed encoding. Completed encoding results from the Encoder process cached in the LLM process's vision token cache ChunkedPrefill Execution:
During scheduling of prefill-stage requests: The scheduler verifies whether the candidate tokens contain vision tokens If required vision tokens are not present in the vision cache: Only the preceding text prompt portion is scheduled Vision token processing is deferred until encoding completion
Scheduling Priority: Strict FCFS ordering maintains request sequence integrity Within the Chun- kedPrefill framework: Decoding-stage requests receive scheduling priority over prefill requests. This ensures optimal throughput while preserving fairness
Algorithm 3 Chunked Prefill Scheduling with Vision Cache Check
Input:
Waiting prefill requests queue Queefill
During decoding requests queue decodinng
Token budget B (max num batched tokens)
Chunk prefill size Batchehunkeize
Vision cache V
Chunking enabled flag C
Output: Scheduled batch Batch containing decode and prefill chunks Initialize empty batch: Batch +-
2: for each request in Qecoding do
3: Append d to Batch
4: Update B +- B- tokens needed by d
5: end for
6: while B > 0 and Qure/su not empty do
7:
r + Upos + position of first vision token in
(Next prefill request]
8:
9:
if Upos exists and vision tokens at Upos not in then
10:
& <- number of prompt tokens before Upos
11:
Schedule prefill chunk with tokens [startpos, Upos) in Batch
else
13:
1 < tokens needed for next chunk of
if / B and C is enabled then
15:
Split r into chunk of size exactly B (take next B tokens) Schedule this chunk for prefill Batch
17:
Remove these B tokens from r
t B
19:
break
else if < B then
21:
Schedule full prefill request in Batch
22:
Remove r from Qprefat
23:
else
24:
break (No budget left)
25:
end if
26: end if
27: B +- -
28:
end while
29:
Execute scheduled Batch (decode + chunked prefill)
24

---

SM Partition Algorithm
Algorithm 4 Find(OpiimalSPPartttionn
Input:
1: Mg (Encoder request metadata) ROg {Encoder Resource- Utility Curve]
2: Tata (Decoder context length) CD (Decoder chunk configuration) RCD (Decoder Resource- Utility Curve)
3: SMorra (Total allocatable SMs) SM alloe (Minimum SMs per component (e.g. 1))
Output: (smig. smb) [Optimal SM allocation for Encoder, Decoder
4: Let (55E) be the latency function for Encoder derived from RCE using ME.
5: Let (s)) be the latency function for Decoder derived from RCD using CD, Leta-
6: smi f- null
7: sm + null
8: Lmakespan f- 00
9: found_ _alid__lid__ae allocation false
Smiin 10: for from S S.Mminnatte aaloc to Stottal Mmmin_allo alloc do
SD SMotta SE
12: if sD < S.mim_Mmiaai alloc then
continue
14: end if
current Lmakespan max(Laag(sg)) Latp(sp))
16:
if current Lakkespann < L makespan then
17:
L makksppan & current Liakkespnn smg + 19: smb 8D
18:
Sg
20: found_valid valid cadlid_allocation +- true
21: end if
22: end for
if found_valle valid allocation then
24: return (smig, sm Lakkeppn
25: else
26: return (null, null), ; (No valid allocation found)
27: end if
G Implementation Details
System stack and versions. SpaceServe is implemented on top of vLLM 0.7.2. We integrate at the scheduler and engine layers, keeping the model execution interface unchanged so that existing MLLM checkpoints can be served without modification
Concurrency via CUDA MPS. We enable NVIDIA CUDA Multi-Process Service (MPS) so that the vision encoder and the text decoder run as separate worker processes on the same GPU and submit kernels concurrently..
SM partitioning primitive. To realize space multiplexing, SpaceServe uses libsmctrl as the low-level primitive partition GPU Streaming Multiprocessors (SMs) across worker processes. The Space-Intee Infereene Scheduler (SIS) chooses an SM quota for each worker according to resource-utility curve and the current TWSRFT admission state, then applies the quota through libsmtrrl.
?Code: https::/gitthu... http:///hhhtbbe.moooo/ooeeee//ea
25


Reviews:
[{"review_id": "RKIMrP2Spw", "reviewer": "Reviewer_Sa3P", "summary": "This paper proposes a novel SpaceServe framework for efficient inference on MLLMs. The key insight is that the resource asymmetry of small encoders and large decoders significantly limits the efficiency of existing frameworks. SpaceServe proposes a space-multiplexing strategy to parallelize the encoding and decoding processes and fully explore the potential of the hardware. Experiments on the Qwen-VL series models show that SpaceServe largely reduced the TPOT.", "questions": "1. Efficiency of different frameworks on different input resolutions would be helpful to have a more thorough understanding.", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1) The motivation and insight are sound, as the encoding and decoding process efficiency bottleneck discrepancy. The encoding process is computation-intensive while the decoding process is more memory-intensive.\n2) The proposed SpaceServe method exhibits high performance compared with the vLLM baseline by x3 to x10 times.\n3) The proposed method could be helpful for the application of MLLMs and further research in this field.\n\nWeakness\n\n1) The tested models are limited to only the Qwen-VL series, while omitting other MLLMs with different architectures, such as LLaVA and MiniCPM-V.\n2) The tested hardware configuration and platforms are limited.", "was_revised": true, "final_justification": "My concerns on the limitation of model tested and experimental setting are solved. The author's discussion on hardware and platform is reasonable and I raise my score to 5.", "rebuttal": "Dear Reviewer Sa3P,\nWe appreciate your thorough and constructive feedback on our paper.\nWe are pleased that the motivation, insights, and performance benefits of SpaceServe were well-received. Your comments have been invaluable in refining our work.\n\nIn response to your questions, we have conducted additional experiments and revised the manuscript. These updates resolve the raised concern and strengthen the paper’s contribution.\n\n\n### 1\\. On the Weakness of Limited Model Diversity (Generalizability of SpaceServe)\n\n\nWe agree that demonstrating generalizability is essential, and thank you for raising this point. Below, we clarify our initial choice of Qwen-VL series and present new experiments showcasing SpaceServe’s broad applicability.\n\n**Rationale for Qwen-VL Selection**\n\n\nWe selected the Qwen-VL series because of its top popularity in MLLM benchmarks at the time of our experiment. Its advanced features, such as dynamic resolution and long context support, make it a robust testbed for SpaceServe. Significant performance gains on such a demanding model validate our approach’s effectiveness. \n\n**Expanded Evaluation**\n\nTo address concerns about broader applicability, we conducted new experiments across diverse model architectures under a high-concurrency workload (10 requests/second) to mirror real-world conditions.\n\n* **Established Architectures**  \n  * LLaVA-1.6: This older MLLM with fixed-resolution architecture is less vision-intensive. SpaceServe achieved a 2.32x speedup.  \n  * MiniCPM-V2.6: A model that is primarily optimized for edge devices. SpaceServe received 2.45x speedup.\n\n\tThose results show that our method delivers strong performance, even in models with reduced compute-memory heterogeneity between the encoder and the LLM.\n\nThose results demonstrate our method performs reasonably well even on models where the compute-memory heterogeneity between the encoder and the LLM is less pronounced.\n\n* **Modern MoE-based Architectures**  \n  * DeepSeek-VL-v2 and Kimi-VL: These recent models with dynamic resolution and Mixture-of-Experts (MoE) layers are highly relevant and well-suited for optimization using our method. SpaceServe delivered impressive speedups of 4.08x and 9.84x, respectively, which prove its effectiveness and generalizability for diverse cutting-edge MLLM architectures.\nThe results, summarized below, demonstrate a clear and consistent advantage for SpaceServe across all tested architectures:\n\n| Model | Framework | TTFT (ms) | TPOT (ms) | TPOT Speedup |\n| :---- | :---- | :---- | :---- | :---- |\n| **LLaVA-1.6** | vLLM | 20351 | 102 | baseline |\n|  | **SpaceServe (Ours)** | 19885 | 44 | **2.32x** |\n| **MiniCPM-V2.6** | vLLM | 1876 | 100.11 | baseline |\n|  | **SpaceServe (Ours)** | 1486 | 40.80 | **2.45x** |\n| **DeepSeek-VL-v2** | vLLM | 5597 | 122.45 | baseline |\n|  | **SpaceServe (Ours)** | 4495 | 30 | **4.08x** |\n| **Kimi-VL** | vLLM | 57900 | 482 | baseline |\n|  | **SpaceServe (Ours)** | 56323 | 49 | **9.84x** |\n\nThe particularly high speedup on Kimi-VL (9.84x) is noteworthy and highlights a key strength of our system. Kimi-VL's ability to support exceptionally high-resolution inputs, while enabling superior multimodal understanding, also drastically increases the computational intensity of its vision encoder. This creates a more pronounced resource heterogeneity between the compute-intensive encoder and the memory-intensive LLM decoder—a scenario where SpaceServe's spatial multiplexing provides the greatest benefit. We observe that this architectural trend is becoming a standard for state-of-the-art models, as seen in other recent releases like Seed-VL and Keye-VL. This underscores the growing importance of our approach for efficiently serving the next generation of powerful MLLMs.\n\n### 2\\. On the Question of Efficiency with Different Input Resolutions (Efficiency Across Different Input Resolutions)\n\n\nWe appreciate the reviewer’s insightful question regarding SpaceServe’s performance under varying input resolutions. It is critical to understand its efficiency. To address this, we conducted new experiments evaluating SpaceServe across a range of input resolutions, from `224x224` to `2048*2048`, under a high-concurrency workload (10 requests per second).\n\nThe results exceeded our expectations and underscore a key strength of our approach:\n\n| Resolution | Framework | TTFT | TPOT (ms) | TPOT Speedup |\n| :---- | :---- | :---- | :---- | :---- |\n| **224 \\* 224** | vLLM | 90.63ms | 23.62 | baseline |\n|  | **SpaceServe (Ours)** | 93.00ms | **17.23** | **1.37x** |\n| **512 \\* 512** | vLLM | 210.50ms | 38.59 | baseline |\n|  | **SpaceServe (Ours)** | 205.42ms | **25.85** | **1.49x** |\n| **1k \\* 1k** | vLLM | 15.31s | 212.58 | baseline |\n|  | **SpaceServe (Ours)** | 15.54s | **35.59** | **5.97x** |\n| **2k \\* 2k** | vLLM | 139.35s | 470.66 | baseline |\n|  | **SpaceServe (Ours)** | 135.76s | **37.99** | **12.39x** |\n\nIn our experiments, we observe that as image resolution increases, the vision encoder’s computational demand significantly degrades vLLM’s performance in terms of increasing both TTFT and TPOT. In contrast, SpaceServe maintains a remarkably low and stable TPOT across all resolutions.\n\nNotable, SpaceServe’s TPOT speedup over vLLM scales dramatically with resolution, from 1.37x at 224x224 to an impressive 12.39x at 2048x2048. This demonstrates that our spatial multiplexing approach effectively mitigates the latency of compute-intensive tasks. Far from being a limitation, high-resolution inputs highlight SpaceServe’s great strength. These important findings and corresponding analysis are now a core component of our revised paper. \n\n### 3\\. On the Weakness of Limited Hardware Configurations\n\n\nWe acknowledge the reviewer’s point that evaluating SpaceServe on diverse hardware would further validate our claim. To clarify, SpaceServe’s design leverages fundamental features of modern GPUs, which are the separation of streaming multiprocessors (SMs) and memory controllers to co-locate compute-intensive (encoder) and memory-intensive (decoder) tasks. This architectural principle is ubiquitous across modern NVIDIA GPUs\n\nDue to time and hardware constraints during this rebuttal period, we were unable to conduct a comprehensive hardware evaluation. We have added this in the \"Limitations and Future Work\" section of our revised paper and highlighted diverse hardware testing as a key direction for future research\n\n---  \n\n\nIn summary, thanks to your insightful feedback, we have significantly strengthened our paper. The new experiments provide robust evidence of SpaceServe’'s **generalizability across diverse MLLM architectures** and its **superior performance under heavy computational loads, particularly with high-resolution images**.\n\n\nWe sincerely thank you for your time and valuable guidance. We hope our revisions and detailed responses have fully addressed your concerns."}, {"review_id": "mn2QUCwBP4", "reviewer": "Reviewer_tKdj", "summary": "This paper introduces SpaceServe, a method to more efficiently serve multimodal LLMs.\nSpaceServe takes advantage of compute/memory tradeoffs between encoders and decoder with spatial multiplexing instead of time multiplexing. SpaceServe also proposes a scheduler to order encoder requests.\nEmpirical results show that the proposed method is more efficient than the vLLM baseline.", "questions": "* Does this generalize to architectures with drastically different shapes and to MLLM with multiple modalities on the output?\n* Are there other baselines than vllm? If so, how does this work compare to those baselines?\n* Given that vllm is the go to library for serving LLMs, could this approach be integrated into vllm? I did not see any mention of releasing the method open source. Is there such a plan?\n* If I understood correctly, the experiments are conducted on a limited set of GPUs. Could there be a sanity check that the findings hold at scale, i.e. with more gpus to parallelize the requests but also larger number of requests?\n273: unaffected TTFT: TTFT is not reported in Figure 4\n\npresentation comments:\n\nI would suggest that the abstract mention 4.81x on average as well.\n62: Mutliplexing → Multiplexing\n74: complementary → complementarity\n111: streaming multiprocessors. SM is used before. The abbreviation should be introduced once on first occurrence.\n199: and performs preprocessing of multimodal data Request → and performs preprocessing of multimodal data. Request (missing period)", "limitations": "yes", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n* the community is increasingly focused on multimodal models. As a result, this work is critical for more efficient serving of such workloads\n* the proposed method is effective in reducing time per output token (up to 28.9x)\n* ablations are conducted on both spatial multiplexing and scheduler to demonstrate how each component of the proposed method contributes to efficiency\n\nWeaknesses:\nIn general, weaknesses relate to more details/comments below. To highlight the main one:\n* the method provides efficiency improvements but the tradeoff is complexity. This potential tradeoff could be discussed in detail. If there is a plan to integrate this method into vllm, this could go a long way in encouraging adoption for example.", "was_revised": true, "final_justification": "The rebuttal provides thorough answers + experiments, which alleviates most of my concerns so I'm raising the score from 4 to 5.", "rebuttal": "Dear Reviewer tKdj, Thank you for your thorough review and for recognizing the importance and effectiveness of SpaceServe. Your insightful questions regarding complexity, generalizability, and community adoption are critical, and we have incorporated them into our revisions to strengthen the paper.\n\nBelow, we address each of your concerns in detail: \n\n### 1\\. On the Tradeoff between Efficiency and Complexity & Open-Sourcing\n\n\nWe acknowledge the reviewer’s concern about the complexity introduced by SpaceServe and the need to justify it. While SpaceServe incorporates new scheduling logic to achieve its performance gains, this complexity is encapsulated within the framework. It remains transparent to end-users and does not impact usability. To facilitate community adoption and transparency, we are committed to **open-sourcing the complete code for SpaceServe.** We have added a statement to the conclusion of our revised paper to reflect this commitment. Furthermore, we recognize the value of integrating SpaceServe's principles into vLLM. Our open-source release will be structured to encourage such integration, and we are open to collaborating with the community to make this happen.\n\n### 2\\. On Generalization to Different Architectures\n\nWe appreciate the reviewer’s critical question regarding whether the generalizability to MLLMs with \"drastically different shapes\" and multiple output modalities.\n\nFirst, to demonstrate the generalizability of our method, we conducted extensive new experiments across a range of diverse MLLMs with different architectures, including **LLaVA-1.6, MiniCPM-V2.6, DeepSeek-VL-v2, and Kimi-VL,** under a high-concurrency setting (10 requests/sec). The results, detailed below, confirm SpaceServe’s robust performance across varied architectures:\n\n| Model | Framework | TTFT (ms) | TPOT (ms) | TPOT Speedup |\n| :---- | :---- | :---- | :---- | :---- |\n| **MiniCPM-V2.6** | vLLM | 1876 | 100.11 | baseline |\n|  | **SpaceServe (Ours)** | **1486** | **40.80** | **2.45x** |\n| **DeepSeek-VL-v2** | vLLM | 5597 | 122.45 | baseline |\n|  | **SpaceServe (Ours)** | **4495** | **30** | **4.08x** |\n| **Kimi-VL** | vLLM | 57900 | 482 | baseline |\n|  | **SpaceServe (Ours)** | **56323** | **49** | **9.84x** |\n| **LLaVA-1.6** | vLLM | 20351 | 102 | baseline |\n|  | **SpaceServe (Ours)** | 19885 | 44 | **2.32x** |\n\nSecond, regarding MLLMs with multiple output modalities (e.g., text and image generation), SpaceServe remains highly effective . The core of SpaceServe’s approach is to exploit resource heterogeneity. In MLLMs, the root of this heterogeneity lies in the contrast between the **compute-intensive vision encoding** and the **memory-intensive text decoding**. As long as a mode exhibits this resource asymmetry between its constituent tasks, regardless of output modalities, SpaceServe can leverage spatial multiplexing to optimize scheduling and improve throughput.  \n We have added a discussion to the revised paper to clarify this applicability to multimodal output models. \n\n\n\n### 3\\. On Other Baselines\n\nWe chose vLLM as our primary baseline because it represents the state-of-the-art for high-throughput MLLM serving.\n\nWe also considered other promising systems like SGLang. However, at the time we conducted our experiments, SGLang's support for MLLMs was still maturing and **lacked several important features necessary for robustly serving modern multimodal models.** Specifically, when serving newer models that support dynamic, high-resolution inputs (such as Kimi-VL, Qwen-VL, and DeepSeek-VL) under **high concurrency**, we found that this lack of feature support led to **out-of-memory (OOM) errors**, making a fair, apples-to-apples comparison under realistic load infeasible.\n\nIn contrast, **vLLM has a larger community and offers more comprehensive and stable support for these demanding workloads**, making it the most suitable and robust baseline for our experiments. We will clarify this justification in the related work section.\n\n### 4\\. On Scalability (More GPUs & Requests)\n\nThis is a great question. We have addressed it in two ways:\n\n* **Larger Number of Requests:** As mentioned, all our experiments were conducted under a **high load of 10 requests per second**. Our results, which show increasing benefits under load, directly confirm that our findings hold for a large number of requests.  \n    \n* **Scaling to More GPUs:** SpaceServe's intra-GPU optimization is **orthogonal and complementary** to inter-GPU scaling strategies. By making *each individual GPU* in a distributed system more efficient, SpaceServe amplifies the overall system throughput. \n\n### 5\\.  Clarification on TTFT Results\nDue to space constraints, the full Time to First Token (TTFT) results were placed in Appendix B. For your convenience, we present the key TTFT data below, which shows that SpaceServe's TTFT is on par with vLLM.\n\n| Model | Method | TTFT (s) @ 2 RPS | @ 4 RPS | @ 6 RPS | @ 8 RPS | @ 10 RPS |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| **Qwen-2-VL-2B** | vLLM | 12.068 | 16.180 | 18.655 | 23.035 | 26.038 |\n| | **SpaceServe** | 13.324 | 17.515 | 21.114 | 24.726 | 27.437 |\n| **Qwen-2-VL-7B** | vLLM | 19.492 | 22.033 | 30.306 | 34.745 | 38.795 |\n| | **SpaceServe** | 18.832 | 22.419 | 32.050 | 36.533 | 39.086 |\n| **Qwen-2.5-VL-32B** | vLLM | 16.553 | 32.631 | 41.849 | 46.440 | 47.547 |\n| | **SpaceServe** | 17.131 | 33.242 | 42.489 | 47.144 | 48.966 |\n| **Qwen-2-VL-72B** | vLLM | 14.872 | 26.647 | 35.717 | 39.780 | 42.067 |\n| | **SpaceServe** | 14.810 | 26.633 | 36.321 | 38.378 | 43.834 |\n**Time To First Token (TTFT) comparison (unit: seconds) across different models with varying Request Per Second (RPS). Lower TTFT is better.**\n\n\n### 6\\. On Presentation Comments\n\nThank you for these meticulous suggestions\\! We have made all the corrections you pointed out in the revised manuscript, including adding the average speedup to the abstract, fixing typos, and ensuring TTFT results are clearly reported in Figure 4 and the text.\n\n---\n\nOnce again, we thank you for your insightful feedback, which has helped us significantly improve the clarity, scope, and impact of our paper. We hope that our responses and the extensive new experimental data have addressed your concerns."}, {"review_id": "YnqZnxTmYr", "reviewer": "Reviewer_aTid", "summary": "This paper proposes SpaceServe, a novel serving system that improves the efficiency of multimodal large language model (MLLM) inference by exploiting the complementary resource profiles of modality-specific encoders and the shared text decoder. The central idea is to shift from time-multiplexed scheduling—where encoders and decoders execute sequentially on the GPU—to space-multiplexed execution, in which both stages run concurrently on the same device through fine-grained streaming multiprocessor (SM) partitioning. To enable this, the authors introduce two key components: (1) a Space-Inference Scheduler (SIS) that dynamically allocates GPU resources using a cost model derived from operator-level profiling, and (2) a Time-Windowed Shortest Remaining Time First (TWSRFT) encoder scheduler that minimizes latency and smooths request arrivals. Through comprehensive empirical evaluation, SpaceServe is shown to deliver substantial speedups—up to 28.9× in time-per-output-token (TPOT)—across multiple model scales and request rates.", "questions": "See above.", "limitations": "Yes. The authors acknowledge the primary limitation regarding Time To First Token performance.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The paper addresses a real and increasingly salient challenge in multimodal LLM serving: how to make efficient use of heterogeneous GPU workloads when modality-specific encoders and a shared autoregressive decoder exhibit asymmetric compute and memory demands. The authors build a compelling case for space-multiplexing, backed by careful arithmetic intensity analysis and a cost-aware design that balances encoder latency against decoder throughput. Their implementation demonstrates strong engineering merit and outperforms the current state-of-the-art (vLLM) by a wide margin, particularly in high-load scenarios. Conceptually, the disaggregation and co-location strategy is not entirely new—earlier work on prefill/decode disaggregation and chunked execution exists—but this paper is one of the first to apply these ideas to the multimodal setting with fine-grained intra-GPU partitioning. This added granularity, along with the encoder-specific scheduler, constitutes a meaningful advancement.\n\nSeveral aspects warrant careful consideration. \n\n(major) The relationship between SpaceServe's serving-time optimizations and recent developments in multimodal architecture design merits discussion. Specifically, the work could benefit from consideration of how spatial multiplexing strategies might interact with sparse architectural approaches, such as those explored in \"Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models\" (TMLR 2025). While SpaceServe optimizes resource allocation for existing dense models, sparse architectures inherently reduce computational requirements through structural design choices. Understanding whether the benefits of spatial multiplexing would be preserved or modified when applied to inherently sparse multimodal architectures could provide valuable insights for future system design considerations.\n\n\n(minor) The resource utility curve construction relies on offline profiling, which may not fully capture dynamic workload variations encountered in production environments. The evaluation scope, while comprehensive within its domain, focuses primarily on Qwen family models, potentially limiting the generalizability of findings to other multimodal architectures with different resource characteristics. The cost model's adaptability to emerging architectural patterns remains unclear, particularly as the field continues to evolve toward more diverse encoder designs and cross-modal interaction mechanisms.", "was_revised": true, "final_justification": "Thank you for the detailed rebuttal. The response addresses my concerns on sparse architectures and generalizability, and I appreciate the additional clarifications and experiments provided.", "rebuttal": "Dear Reviewer  aTid,\n\nThank you for your exceptionally thorough and insightful review. We are truly encouraged by your \"excellent\" ratings and your deep appreciation for SpaceServe's core contributions. Your summary accurately captures the essence of SpaceServe, and we are grateful for your recognition of its novelty and engineering merit.\n\nYour forward-looking questions, particularly regarding sparse architectures, are highly stimulating and have prompted us to think more deeply about the future trajectory of our work.Below, we address your specific concerns in detail:\n\n### 1\\. (Major) On the Relationship with Sparse Architectures\n\nWe greatly appreciate your insightful question about the interplay between SpaceServe and sparse architectures, such as the Mixture-of-Transformers (MoT) referenced in your review. We agree that exploring the synergy between system-level optimization and architectural innovations is critical. To address this, we have added a detailed discussion in the revised paper.\n\n\nOur position is that **SpaceServe's spatial multiplexing and architectural approaches like MoT are not only compatible but are in fact also deeply complementary, yielding significant synergistic benefits.**\n\n\n\n**A. Theoretical Synergy with Mixture-of-Transformers (MoT):**\n\nThe MoT architecture provides compelling architectural-level evidence for our system's core premise. By creating modality-specific parameters (e.g., separate FFNs for image vs. text), MoT validates that different modalities have distinct computational profiles that benefit from specialized processing.\n\nHowever, this very design introduces a new, fine-grained scheduling challenge during inference. When serving an MoT model, only one modality's \"path\" (e.g., the image FFN) is active at any given time for a single request, leaving other modality-specific components idle. This resource under-utilization is precisely the challenge SpaceServe is designed to solve. Our spatial scheduler can dynamically co-locate the execution of one request's compute-intensive image path with another request's memory-intensive text path on the same GPU. This maximizes hardware utilization and effectively translates MoT’s architectural efficiency into enhanced serving throughput.\n\n\n**B. Empirical Evidence with Modern MoE-based MLLMs:**\n\nSparse architectures can be found in the latest models. Many new MLLMs, such as **DeepSeek-VL-v2** and **Kimi-VL**, adopt a hybrid \"Vision Encoder \\+ MoE-LLM\" structure. To empirically validate our system's effectiveness on such architectures, we conducted new experiments on these models under a high-concurrency setting (10 requests/second).\n\n\nThe results are conclusive:\n\n| Model (MoE-based) | Framework | TTFT | TPOT (ms) | TPOT Speedup |\n| :--- | :--- | :--- | :--- | :--- |\n| **DeepSeek-VL-v2** | vLLM | 5597ms | 122.45 | baseline |\n| | **SpaceServe (Ours)** | 4495ms | 30 | **4.08x** |\n| **Kimi-VL** | vLLM | 57.9s | 482 | baseline |\n| | **SpaceServe (Ours)** | 56.3s | 49 | **9.84x** |\n\n\nThis \"dense-sparse\" hybrid architecture exhibits a pronounced resource heterogeneity: a dense, compute-intensive vision encoder followed by a sparse, memory-intensive MoE decoder. Our **4x-9.8x speedup** on these models provides powerful evidence that SpaceServe effectively manages this new form of workload asymmetry.\n\nIn summary, whether the sparsity is achieved by \"modality-specific paths\" (like MoT) or by \"component-specific sparsity\" (like MoE-LLMs), the fundamental result is increased resource heterogeneity. Our work provides a general, system-level solution to harness this heterogeneity, demonstrating SpaceServe's lasting value for efficiently serving the next generation of MLLMs.\n\n### 2\\. (Minor) On Generalizability and Offline Profiling\nWe appreciate your points regarding model generalizability and the limitations of offline profiling, which are critical for ensuring SpaceServe’s broad applicability and practicality.\n\n* **Model Generalizability:** As noted above, our new experiments now include a wide range of architectures (LLaVA, MiniCPM-V, and the MoE-based DeepSeek-VL and Kimi-VL). The results, summarized below, confirm SpaceServe’s robust performance across varied architectures:\n\n\n| Model | Framework | TTFT (ms) | TPOT (ms) | TPOT Speedup |\n| :---- | :---- | :---- | :---- | :---- |\n| **MiniCPM-V2.6** | vLLM | 1876 | 100.11 | baseline |\n|  | **SpaceServe (Ours)** | **1486** | **40.80** | **2.45x** |\n| **DeepSeek-VL-v2** | vLLM | 5597 | 122.45 | baseline |\n|  | **SpaceServe (Ours)** | **4495** | **30** | **4.08x** |\n| **Kimi-VL** | vLLM | 57900 | 482 | baseline |\n|  | **SpaceServe (Ours)** | **56323** | **49** | **9.84x** |\n| **LLaVA-1.6** | vLLM | 20351 | 102 | baseline |\n|  | **SpaceServe (Ours)** | 19885 | 44 | **2.32x** |\n\n\n* **Offline Profiling:** You raised a valid point about the limitations of offline profiling. Our choice was a pragmatic one, trading dynamic adaptation for the stability and zero-overhead benefits that are standard in high-performance systems. Importantly, our framework is modular. It can adapt to new architectures, including the specific paths of an MoT model, by simply running a new profiling script to generate an updated cost profile without changing the core scheduling logic. \n---\nThank you again for your insightful review in supporting our work. Your comments have strengthened our paper by connecting SpaceServe to cutting-edge architectural advancements, particularly sparse (sparsity and MoT), and by prompting us to clarify our approach to generalizability and profiling. We hope our revision, including new experimental data and detailed discussion, meet your high standards and fully address you concerns."}]

Author rebuttal:

Meta-review: This paper presents SpaceServe, a serving system for multimodal large language models that improves GPU utilization by space-multiplexing modality-specific encoders and the shared text decoder. It introduces a Space-Inference Scheduler (SIS) and a Time-Windowed Shortest-Remaining-First (TWSRFT) policy to optimize SM partitioning and request batching. Experiments on Nvidia A100 GPUs show that SpaceServe achieves up to 29.71× speedup over existing serving stacks.

The concerns raised in the reviews and that were discussed can be summarized as follows:
1. Insufficient baselines (i.e., adding other VLMs besides Qwen family) in the experiments and limited hardware variations (R-Sa3P, R-aTid);
2. Lacking discussion about efficiency vs complexity tradeoff (R-tKdj); and
3. Lacking discussion about the relationship of SpaceServe and sparse architectures (R-aTid).

During the discussion between reviewers and authors, the authors provided new experiments and thorough clarifications that satisfied all reviewers. Given this positive discussion, reviewers recommended to accept the paper unanimously. Thus, we encourage the authors to include any additional clarification and experiments that help the story of the paper.
Decision: Accept (poster)