Title: Efficient and Workload-Aware LLM Serving via Runtime Layer Swapping and KV Cache Resizing
Primary area: infrastructure, software libraries, hardware, systems, etc.
Keywords: Large Language Models, LLM Serving Systems, LLM Serving Optimization, Mixed-Precision Serving, Workload-Aware Systems

Abstract: Efficiently serving large language models (LLMs) under dynamic and bursty workloads remains a key challenge for real-world deployment. Existing serving frameworks and static model compression techniques fail to adapt to workload fluctuations, leading to either service-level objective (SLO) violations under full-precision serving or persistent accuracy degradation with static quantization. We present MorphServe, a dynamic, workload-aware LLM serving framework based on morphological adaptation. MorphServe introduces two asynchronous, token-level runtime mechanisms: quantized layer swapping, which selectively replaces less impactful layers with quantized alternatives during high-load periods, and pressure-aware KV cache resizing, which dynamically adjusts KV cache capacity in response to memory pressure. These mechanisms enable state-preserving transitions with minimum runtime overhead and are fully compatible with modern scheduling and attention techniques. Extensive experiments on Vicuna and Llama family models with real-world workloads demonstrate that MorphServe reduces average SLO violations by 92.45% and improves the P95 TTFT latency by 2.2x-3.9x compared to full-precision serving, without compromising generation quality. These results establish MorphServe as a practical and elastic solution for LLM deployment in dynamic environments.

TL;DR: We propose MorphServe, a dynamic and workload-aware LLM serving framework that combines runtime layer swapping and elastic KV cache resizing to adaptively balance latency, accuracy, and memory efficiency under bursty traffic.