DeepFlow: Serverless Large Language Model Serving at Scale
Authors: name: Junhao Hu, name: Jiang Xu, name: Zhixia Liu, name: Yulong He, name: Yuetao Chen, name: Hao Xu, name: Jiang Liu, name: Baoquan Zhang, name: Shining Wan, name: Gengyuan Dan, name: Zhiyu Dong, name: Zhihao Ren, name: Jie Meng, name: Chao He, name: Changhong Liu, name: Tao Xie, name: Dayun Lin, name: Qin Zhang, name: Yue Yu, name: Hao Feng, name: Xusheng Chen, name: Yizhou Shan

This paper introduces DeepFlow, a scalable and serverless AI platform
designed to efficiently serve large language models (LLMs) at scale in cloud
environments. DeepFlow addresses key challenges such as resource allocation,
serving efficiency, and cold start latencies through four main design
components. First, it uses a simple serverless abstraction called the
request-job-task model, which helps manage AI workloads across post-training
and model serving tasks. Second, it builds an in-house serving engine FlowServe
using a microkernel-inspired design, NPU-centric execution, and SPMD-based
parallelism to optimize LLM serving. The system also includes novel scheduling
policies tailored for both PD-disaggregated and PD-colocated configurations.
With optimizations like pre-warmed pods, DRAM pre-loading, and NPU-fork,
DeepFlow can scale up to 64 instances in seconds. DeepFlow has been in
production for over a year, operating on a large Ascend NPU cluster and
providing industrystandard APIs for fine-tuning, agent serving, and model
serving to our customers.