The Impact of Hyperparameters on Large Language Model Inference
  Performance: An Evaluation of vLLM and HuggingFace Pipelines
Authors: name: Matias Martinez
Published: 2024-08-02 06:56:59

Abstract: The recent surge of open-source large language models (LLMs) enables
developers to create AI-based solutions while maintaining control over aspects
such as privacy and compliance, thereby providing governance and ownership of
the model deployment process. To utilize these LLMs, inference engines are
needed. These engines load the model's weights onto available resources, such
as GPUs, and process queries to generate responses. The speed of inference, or
performance, of the LLM, is critical for real-time applications, as it computes
millions or billions of floating point operations per inference. Recently,
advanced inference engines such as vLLM have emerged, incorporating novel
mechanisms such as efficient memory management to achieve state-of-the-art
performance. In this paper, we analyze the performance, particularly the
throughput (tokens generated per unit of time), of 20 LLMs using two inference
libraries: vLLM and HuggingFace's pipelines. We investigate how various
hyperparameters, which developers must configure, influence inference
performance. Our results reveal that throughput landscapes are irregular, with
distinct peaks, highlighting the importance of hyperparameter optimization to
achieve maximum performance. We also show that applying hyperparameter
optimization when upgrading or downgrading the GPU model used for inference can
improve throughput from HuggingFace pipelines by an average of 9.16% and 13.7%,
respectively.