Reasoning Language Model Inference Serving Unveiled: An Empirical Study
Authors: name: Qi Li, name: Junpan Wu, name: Xiang Liu, name: Yuxin Wang, name: Zeyu Li, name: Zhenheng Tang, name: Yuhan Chen, name: Shaohuai Shi, name: Xiaowen Chu

The reasoning large language model (RLLM) has been proven competitive in
solving complex reasoning tasks such as mathematics, coding, compared to
general LLM. However, the serving performance and behavior of RLLM remains
unexplored, which may undermine the deployment and utilization of RLLM in
real-world scenario. To close this gap, in this paper, we conduct a
comprehensive study of RLLM service. We first perform a pilot study on
comparing the serving performance between RLLM and traditional LLM and reveal
that there are several distinct differences regarding serving behavior: (1)
significant memory usage and fluctuations; (2) straggler requests; (3) adaptive
running time; (4) domain preference. Then we further investigate whether
existing inference optimization techniques are valid for RLLM. Our main
takeaways are that model quantization methods and speculative decoding can
improve service system efficiency with small compromise to RLLM accuracy, while
prefix caching, KV cache quantization may even degrade accuracy or serving
performance for small RLLM. Lastly, we conduct evaluation under real world
workload modeled by Gamma distribution to verify our findings. Empirical
results of real world workload evaluation across different dataset are aligned
with our main findings regarding RLLM serving. We hope our work can provide the
research community and industry with insights to advance RLLM inference
serving.