Package: langport
LangPortLangPort is a open-source large language model serving platform.
Our goal is to build a super fast LLM inference service.This project is inspired bylmsys/fastchat, we hope that the serving platform is lightweight and fast, but fastchat includes other features such as training and evaluation make it complicated.The core features include:Huggingface transformers support.ggml (llama.cpp) support.A distributed serving system for state-of-the-art models.Streaming generation support with various decoding strategies.Batch inference for higher throughput.Support for encoder-only, decoder-only and encoder-decoder models.OpenAI-compatible RESTful APIs.FauxPilot-compatible RESTful APIs.HuggingFace-compatible RESTful APIs.Tabby-compatible RESTful APIs.Support Model ArchitecturesLLaMa, LLaMa2, GLM, Bloom, OPT, GPT2, GPT Neo, GPT Big Code and so on.Tested ModelsNingYu, LLaMa, LLaMa2, Vicuna, ChatGLM, ChatGLM2, Falcon, Starcoder, WizardLM, InternLM, OpenBuddy, FireFly, CodeGen, Phoenix, RWKV, StableLM and so on.News[2023/08/04] Dynamic batch inference.[2023/07/16] Support int4 quantization.[2023/07/13] Support generation logprobs parameter.[2023/06/18] Add ggml (llama.cpp gpt.cpp starcoder.cpp etc.) worker support.[2023/06/09] Add LLama.cpp worker support.[2023/06/01] Add HuggingFace Bert embedding worker support.[2023/06/01] Add HuggingFace text generation API support.[2023/06/01] Add tabby API support.[2023/05/23] Add chat throughput test script.[2023/05/22] New distributed architecture.[2023/05/14] Batch inference supported.[2023/05/10] Langport project started.InstallMethod 1: With pippipinstalllangportor:pipinstallgit+https://github.com/vtuber-plan/langport.gitIf you need ggml generation worker, use this command:pipinstalllangport[ggml]If you want to use GPU:CT_CUBLAS=1pipinstalllangport[ggml]Method 2: From sourceClone this repositorygitclonehttps://github.com/vtuber-plan/langport.gitcdlangportInstall the Packagepipinstall--upgradepip
pipinstall-e.Quick startIt is simple to start a local chat API service:First, start a worker process in the terminal:python-mlangport.service.server.generation_worker--port21001--model-path<yourmodelpath>Then, start a API service in another terminal:python-mlangport.service.gateway.openai_apiNow, you can use the inference API by openai protocol.Start the serverIt is simple to start a single node chat API service:python-mlangport.service.server.generation_worker--port21001--model-path<yourmodelpath>
python-mlangport.service.gateway.openai_apiIf you need the embeddings API or other features, you can deploy a distributed inference cluster:python-mlangport.service.server.dummy_worker--port21001python-mlangport.service.server.generation_worker--model-path<yourmodelpath>--neighborshttp://localhost:21001
python-mlangport.service.server.embedding_worker--model-path<yourmodelpath>--neighborshttp://localhost:21001
python-mlangport.service.gateway.openai_api--controller-addresshttp://localhost:21001In practice, the gateway can connect to any node to distribute inference tasks:python-mlangport.service.server.dummy_worker--port21001python-mlangport.service.server.generation_worker--port21002--model-path<yourmodelpath>--neighborshttp://localhost:21001
python-mlangport.service.server.generation_worker--port21003--model-path<yourmodelpath>--neighborshttp://localhost:21001http://localhost:21002
python-mlangport.service.server.generation_worker--port21004--model-path<yourmodelpath>--neighborshttp://localhost:21001http://localhost:21003
python-mlangport.service.server.generation_worker--port21005--model-path<yourmodelpath>--neighborshttp://localhost:21001http://localhost:21004
python-mlangport.service.gateway.openai_api--controller-addresshttp://localhost:21003# 21003 is OK!python-mlangport.service.gateway.openai_api--controller-addresshttp://localhost:21002# Any worker is also OK!Run text generation with multi GPUs:python-mlangport.service.server.generation_worker--port21001--model-path<yourmodelpath>--gpus0,1--num-gpus2python-mlangport.service.gateway.openai_apiRun text generation with ggml worker:python-mlangport.service.server.ggml_generation_worker--port21001--model-path<yourmodelpath>--gpu-layers<numlayertogpu(resizethisforyourVRAM)>Run OpenAI forward server:python-mlangport.service.server.chatgpt_generation_worker--port21001--api-url<url>--api-key<key>Licenselangport is released under the Apache Software License.See alsolangport-docslangport-sourceStar History