Package: vllm
Easy, fast, and cheap LLM serving for everyone|Documentation|Blog|Paper|Discord|Latest News🔥[2024/01] We hostedthe second vLLM meetupin SF! Please find the meetup slideshere.[2024/01] Added ROCm 6.0 support to vLLM.[2023/12] Added ROCm 5.7 support to vLLM.[2023/10] We hostedthe first vLLM meetupin SF! Please find the meetup slideshere.[2023/09] We created ourDiscord server! Join us to discuss vLLM and LLM serving! We will also post the latest announcements and updates there.[2023/09] We released ourPagedAttention paperon arXiv![2023/08] We would like to express our sincere gratitude toAndreessen Horowitz(a16z) for providing a generous grant to support the open-source development and research of vLLM.[2023/07] Added support for LLaMA-2! You can run and serve 7B/13B/70B LLaMA-2s on vLLM with a single command![2023/06] Serving vLLM On any Cloud with SkyPilot. Check out a 1-clickexampleto start the vLLM demo, and theblog postfor the story behind vLLM development on the clouds.[2023/06] We officially released vLLM! FastChat-vLLM integration has poweredLMSYS Vicuna and Chatbot Arenasince mid-April. Check out ourblog post.AboutvLLM is a fast and easy-to-use library for LLM inference and serving.vLLM is fast with:State-of-the-art serving throughputEfficient management of attention key and value memory withPagedAttentionContinuous batching of incoming requestsFast model execution with CUDA/HIP graphQuantization:GPTQ,AWQ,SqueezeLLM, FP8 KV CacheOptimized CUDA kernelsvLLM is flexible and easy to use with:Seamless integration with popular Hugging Face modelsHigh-throughput serving with various decoding algorithms, includingparallel sampling,beam search, and moreTensor parallelism support for distributed inferenceStreaming outputsOpenAI-compatible API serverSupport NVIDIA GPUs and AMD GPUs(Experimental) Prefix caching support(Experimental) Multi-lora supportvLLM seamlessly supports many Hugging Face models, including the following architectures:Aquila & Aquila2 (BAAI/AquilaChat2-7B,BAAI/AquilaChat2-34B,BAAI/Aquila-7B,BAAI/AquilaChat-7B, etc.)Baichuan & Baichuan2 (baichuan-inc/Baichuan2-13B-Chat,baichuan-inc/Baichuan-7B, etc.)BLOOM (bigscience/bloom,bigscience/bloomz, etc.)ChatGLM (THUDM/chatglm2-6b,THUDM/chatglm3-6b, etc.)DeciLM (Deci/DeciLM-7B,Deci/DeciLM-7B-instruct, etc.)Falcon (tiiuae/falcon-7b,tiiuae/falcon-40b,tiiuae/falcon-rw-7b, etc.)Gemma (google/gemma-2b,google/gemma-7b, etc.)GPT-2 (gpt2,gpt2-xl, etc.)GPT BigCode (bigcode/starcoder,bigcode/gpt_bigcode-santacoder, etc.)GPT-J (EleutherAI/gpt-j-6b,nomic-ai/gpt4all-j, etc.)GPT-NeoX (EleutherAI/gpt-neox-20b,databricks/dolly-v2-12b,stabilityai/stablelm-tuned-alpha-7b, etc.)InternLM (internlm/internlm-7b,internlm/internlm-chat-7b, etc.)InternLM2 (internlm/internlm2-7b,internlm/internlm2-chat-7b, etc.)LLaMA & LLaMA-2 (meta-llama/Llama-2-70b-hf,lmsys/vicuna-13b-v1.3,young-geng/koala,openlm-research/open_llama_13b, etc.)Mistral (mistralai/Mistral-7B-v0.1,mistralai/Mistral-7B-Instruct-v0.1, etc.)Mixtral (mistralai/Mixtral-8x7B-v0.1,mistralai/Mixtral-8x7B-Instruct-v0.1, etc.)MPT (mosaicml/mpt-7b,mosaicml/mpt-30b, etc.)OLMo (allenai/OLMo-1B,allenai/OLMo-7B, etc.)OPT (facebook/opt-66b,facebook/opt-iml-max-30b, etc.)Phi (microsoft/phi-1_5,microsoft/phi-2, etc.)Qwen (Qwen/Qwen-7B,Qwen/Qwen-7B-Chat, etc.)Qwen2 (Qwen/Qwen2-7B-beta,Qwen/Qwen-7B-Chat-beta, etc.)StableLM(stabilityai/stablelm-3b-4e1t,stabilityai/stablelm-base-alpha-7b-v2, etc.)Yi (01-ai/Yi-6B,01-ai/Yi-34B, etc.)Install vLLM with pip orfrom source:pipinstallvllmGetting StartedVisit ourdocumentationto get started.InstallationQuickstartSupported ModelsContributingWe welcome and value any contributions and collaborations.
Please check outCONTRIBUTING.mdfor how to get involved.CitationIf you use vLLM for your research, please cite ourpaper:@inproceedings{kwon2023efficient,title={Efficient Memory Management for Large Language Model Serving with PagedAttention},author={Woosuk Kwon and Zhuohan Li and Siyuan Zhuang and Ying Sheng and Lianmin Zheng and Cody Hao Yu and Joseph E. Gonzalez and Hao Zhang and Ion Stoica},booktitle={Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles},year={2023}}