Title: SpaceServe: Spatial Multiplexing of Complementary Encoders and Decoders for Multimodal LLMs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal large language models; Inference optimizations; Infrastructure
Abstract: Recent multimodal large language models (MLLMs) marry modality-specific
vision or audio encoders with a shared text decoder. While the encoder is compute-
intensive but memory-light, the decoder is the opposite, yet state-of-the-art serving
stacks still time-multiplex these complementary kernels, idling SMs or HBM in
turn. We introduce SpaceServe, a serving system that space-multiplexes MLLMs:
it decouples all modality encoders from the decoder, and co-locates them on the
same GPU using fine-grained SM partitioning available in modern runtimes. A
cost-model-guided Space-Inference Scheduler (SIS) dynamically assigns SM slices,
while a Time-Windowed Shortest-Remaining-First (TWSRFT) policy batches en-
coder requests to minimise completion latency and smooth decoder arrivals. 
Evaluation shows that SpaceServe reduces time-per-output-token by 4.81×
on average and up to 28.9× on Nvidia A100 GPUs. SpaceServe is available at
https://github.com/gofreelee/SpaceServe


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SpaceServe, a novel serving system that improves the efficiency of multimodal large language model (MLLM) inference by exploiting the complementary resource profiles of modality-specific encoders and the shared text decoder. The central idea is to shift from time-multiplexed scheduling—where encoders and decoders execute sequentially on the GPU—to space-multiplexed execution, in which both stages run concurrently on the same device through fine-grained streaming multiprocessor (SM) partitioning. To enable this, the authors introduce two key components: (1) a Space-Inference Scheduler (SIS) that dynamically allocates GPU resources using a cost model derived from operator-level profiling, and (2) a Time-Windowed Shortest Remaining Time First (TWSRFT) encoder scheduler that minimizes latency and smooths request arrivals. Through comprehensive empirical evaluation, SpaceServe is shown to deliver substantial speedups—up to 28.9× in time-per-output-token (TPOT)—across multiple model scales and request rates.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper addresses a real and increasingly salient challenge in multimodal LLM serving: how to make efficient use of heterogeneous GPU workloads when modality-specific encoders and a shared autoregressive decoder exhibit asymmetric compute and memory demands. The authors build a compelling case for space-multiplexing, backed by careful arithmetic intensity analysis and a cost-aware design that balances encoder latency against decoder throughput. Their implementation demonstrates strong engineering merit and outperforms the current state-of-the-art (vLLM) by a wide margin, particularly in high-load scenarios. Conceptually, the disaggregation and co-location strategy is not entirely new—earlier work on prefill/decode disaggregation and chunked execution exists—but this paper is one of the first to apply these ideas to the multimodal setting with fine-grained intra-GPU partitioning. This added granularity, along with the encoder-specific scheduler, constitutes a meaningful advancement.\n\nSeveral aspects warrant careful consideration. \n\n(major) The relationship between SpaceServe's serving-time optimizations and recent developments in multimodal architecture design merits discussion. Specifically, the work could benefit from consideration of how spatial multiplexing strategies might interact with sparse architectural approaches, such as those explored in \"Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models\" (TMLR 2025). While SpaceServe optimizes resource allocation for existing dense models, sparse architectures inherently reduce computational requirements through structural design choices. Understanding whether the benefits of spatial multiplexing would be preserved or modified when applied to inherently sparse multimodal architectures could provide valuable insights for future system design considerations.\n\n\n(minor) The resource utility curve construction relies on offline profiling, which may not fully capture dynamic workload variations encountered in production environments. The evaluation scope, while comprehensive within its domain, focuses primarily on Qwen family models, potentially limiting the generalizability of findings to other multimodal architectures with different resource characteristics. The cost model's adaptability to emerging architectural patterns remains unclear, particularly as the field continues to evolve toward more diverse encoder designs and cross-modal interaction mechanisms.\n\nQUESTIONS:\nSee above.\n\nLIMITATIONS:\nYes. The authors acknowledge the primary limitation regarding Time To First Token performance.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SpaceServe, a method to more efficiently serve multimodal LLMs.\nSpaceServe takes advantage of compute/memory tradeoffs between encoders and decoder with spatial multiplexing instead of time multiplexing. SpaceServe also proposes a scheduler to order encoder requests.\nEmpirical results show that the proposed method is more efficient than the vLLM baseline.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* the community is increasingly focused on multimodal models. As a result, this work is critical for more efficient serving of such workloads\n* the proposed method is effective in reducing time per output token (up to 28.9x)\n* ablations are conducted on both spatial multiplexing and scheduler to demonstrate how each component of the proposed method contributes to efficiency\n\nWeaknesses:\nIn general, weaknesses relate to more details/comments below. To highlight the main one:\n* the method provides efficiency improvements but the tradeoff is complexity. This potential tradeoff could be discussed in detail. If there is a plan to integrate this method into vllm, this could go a long way in encouraging adoption for example.\n\nQUESTIONS:\n* Does this generalize to architectures with drastically different shapes and to MLLM with multiple modalities on the output?\n* Are there other baselines than vllm? If so, how does this work compare to those baselines?\n* Given that vllm is the go to library for serving LLMs, could this approach be integrated into vllm? I did not see any mention of releasing the method open source. Is there such a plan?\n* If I understood correctly, the experiments are conducted on a limited set of GPUs. Could there be a sanity check that the findings hold at scale, i.e. with more gpus to parallelize the requests but also larger number of requests?\n273: unaffected TTFT: TTFT is not reported in Figure 4\n\npresentation comments:\n\nI would suggest that the abstract mention 4.81x on average as well.\n62: Mutliplexing → Multiplexing\n74: complementary → complementarity\n111: streaming multiprocessors. SM is used before. The abbreviation should be introduced once on first occurrence.\n199: and performs preprocessing of multimodal data Request → and performs preprocessing of multimodal data. Request (missing period)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

comments: SUMMARY:
This paper proposes a novel SpaceServe framework for efficient inference on MLLMs. The key insight is that the resource asymmetry of small encoders and large decoders significantly limits the efficiency of existing frameworks. SpaceServe proposes a space-multiplexing strategy to parallelize the encoding and decoding processes and fully explore the potential of the hardware. Experiments on the Qwen-VL series models show that SpaceServe largely reduced the TPOT.

STRENGTHS AND WEAKNESSES:
Strengths

1) The motivation and insight are sound, as the encoding and decoding process efficiency bottleneck discrepancy. The encoding process is computation-intensive while the decoding process is more memory-intensive.
2) The proposed SpaceServe method exhibits high performance compared with the vLLM baseline by x3 to x10 times.
3) The proposed method could be helpful for the application of MLLMs and further research in this field.

Weakness

1) The tested models are limited to only the Qwen-VL series, while omitting other MLLMs with different architectures, such as LLaVA and MiniCPM-V.
2) The tested hardware configuration and platforms are limited.

QUESTIONS:
1. Efficiency of different frameworks on different input resolutions would be helpful to have a more thorough understanding.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-05