Package: llama-cpp-python-binary
🦙 Python Bindings forllama.cppSimple Python bindings for@ggerganov'sllama.cpplibrary.
This package provides:Low-level access to C API viactypesinterface.High-level Python API for text completionOpenAI-like APILangChain compatibilityOpenAI compatible web serverLocal Copilot replacementFunction Calling supportVision API supportMultiple ModelsDocumentation is available athttps://llama-cpp-python.readthedocs.io/en/latest.Installationllama-cpp-pythoncan be installed directly from PyPI as a source distribution by running:pipinstallllama-cpp-pythonThis will buildllama.cppfrom source using cmake and your system's c compiler (required) and install the library alongside this python package.If you run into issues during installation add the--verboseflag to thepip installcommand to see the full cmake build log.Installation with Specific Hardware Acceleration (BLAS, CUDA, Metal, etc)The default pip install behaviour is to buildllama.cppfor CPU only on Linux and Windows and use Metal on MacOS.llama.cppsupports a number of hardware acceleration backends depending including OpenBLAS, cuBLAS, CLBlast, HIPBLAS, and Metal.
See thellama.cpp READMEfor a full list of supported backends.All of these backends are supported byllama-cpp-pythonand can be enabled by setting theCMAKE_ARGSenvironment variable before installing.On Linux and Mac you set theCMAKE_ARGSlike this:CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS"pipinstallllama-cpp-pythonOn Windows you can set theCMAKE_ARGSlike this:$env:CMAKE_ARGS = "-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS"
pip install llama-cpp-pythonOpenBLASTo install with OpenBLAS, set theLLAMA_BLAS and LLAMA_BLAS_VENDORenvironment variables before installing:CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS"pipinstallllama-cpp-pythoncuBLASTo install with cuBLAS, set theLLAMA_CUBLAS=1environment variable before installing:CMAKE_ARGS="-DLLAMA_CUBLAS=on"pipinstallllama-cpp-pythonMetalTo install with Metal (MPS), set theLLAMA_METAL=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_METAL=on"pipinstallllama-cpp-pythonCLBlastTo install with CLBlast, set theLLAMA_CLBLAST=1environment variable before installing:CMAKE_ARGS="-DLLAMA_CLBLAST=on"pipinstallllama-cpp-pythonhipBLASTo install with hipBLAS / ROCm support for AMD cards, set theLLAMA_HIPBLAS=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_HIPBLAS=on"pipinstallllama-cpp-pythonWindows NotesIf you run into issues where it complains it can't find'nmake''?'or CMAKE_C_COMPILER, you can extract w64devkit asmentioned in llama.cpp repoand add those manually to CMAKE_ARGS before runningpipinstall:$env:CMAKE_GENERATOR = "MinGW Makefiles"
$env:CMAKE_ARGS = "-DLLAMA_OPENBLAS=on -DCMAKE_C_COMPILER=C:/w64devkit/bin/gcc.exe -DCMAKE_CXX_COMPILER=C:/w64devkit/bin/g++.exe"See the above instructions and setCMAKE_ARGSto the BLAS backend you want to use.MacOS NotesNote: If you are using Apple Silicon (M1) Mac, make sure you have installed a version of Python that supports arm64 architecture. For example:wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.shOtherwise, while installing it will build the llama.cpp x86 version which will be 10x slower on Apple Silicon (M1) Mac.Detailed MacOS Metal GPU install documentation is available atdocs/install/macos.mdUpgrading and ReinstallingTo upgrade or rebuildllama-cpp-pythonadd the following flags to ensure that the package is rebuilt correctly:pipinstallllama-cpp-python--upgrade--force-reinstall--no-cache-dirThis will ensure that all source files are re-built with the most recently setCMAKE_ARGSflags.High-level APIAPI ReferenceThe high-level API provides a simple managed interface through theLlamaclass.Below is a short example demonstrating how to use the high-level API to for basic text completion:>>>fromllama_cppimportLlama>>>llm=Llama(model_path="./models/7B/llama-model.gguf")>>>output=llm("Q: Name the planets in the solar system? A: ",# Promptmax_tokens=32,# Generate up to 32 tokensstop=["Q:","\n"],# Stop generating just before the model would generate a new questionecho=True# Echo the prompt back in the output)# Generate a completion, can also call create_completion>>>print(output){"id":"cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","object":"text_completion","created":1679561337,"model":"./models/7B/llama-model.gguf","choices":[{"text":"Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.","index":0,"logprobs":None,"finish_reason":"stop"}],"usage":{"prompt_tokens":14,"completion_tokens":28,"total_tokens":42}}Text completion is available through the__call__andcreate_completionmethods of theLlamaclass.Chat CompletionThe high-level API also provides a simple interface for chat completion.Note thatchat_formatoption must be set for the particular model you are using.>>>fromllama_cppimportLlama>>>llm=Llama(model_path="path/to/llama-2/llama-model.gguf",chat_format="llama-2")>>>llm.create_chat_completion(messages=[{"role":"system","content":"You are an assistant who perfectly describes images."},{"role":"user","content":"Describe this image in detail please."}])Chat completion is available through thecreate_chat_completionmethod of theLlamaclass.Function CallingThe high-level API also provides a simple interface for function calling.Note that the only model that supports full function calling at this time is "functionary".
The gguf-converted files for this model can be found here:functionary-7b-v1>>>fromllama_cppimportLlama>>>llm=Llama(model_path="path/to/functionary/llama-model.gguf",chat_format="functionary")>>>llm.create_chat_completion(messages=[{"role":"system","content":"A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. The assistant calls functions with appropriate input when necessary"},{"role":"user","content":"Extract Jason is 25 years old"}],tools=[{"type":"function","function":{"name":"UserDetail","parameters":{"type":"object","title":"UserDetail","properties":{"name":{"title":"Name","type":"string"},"age":{"title":"Age","type":"integer"}},"required":["name","age"]}}}],tool_choices=[{"type":"function","function":{"name":"UserDetail"}}])Multi-modal Modelsllama-cpp-pythonsupports the llava1.5 family of multi-modal models which allow the language model to
read information from both text and images.You'll first need to download one of the available multi-modal models in GGUF format:llava-v1.5-7bllava-v1.5-13bbakllava-1-7bThen you'll need to use a custom chat handler to load the clip model and process the chat messages and images.>>>fromllama_cppimportLlama>>>fromllama_cpp.llama_chat_formatimportLlava15ChatHandler>>>chat_handler=Llava15ChatHandler(clip_model_path="path/to/llava/mmproj.bin")>>>llm=Llama(model_path="./path/to/llava/llama-model.gguf",chat_handler=chat_handler,n_ctx=2048,# n_ctx should be increased to accomodate the image embeddinglogits_all=True,# needed to make llava work)>>>llm.create_chat_completion(messages=[{"role":"system","content":"You are an assistant who perfectly describes images."},{"role":"user","content":[{"type":"image_url","image_url":{"url":"https://.../image.png"}},{"type":"text","text":"Describe this image in detail please."}]}])Adjusting the Context WindowThe context window of the Llama models determines the maximum number of tokens that can be processed at once. By default, this is set to 512 tokens, but can be adjusted based on your requirements.For instance, if you want to work with larger contexts, you can expand the context window by setting the n_ctx parameter when initializing the Llama object:llm=Llama(model_path="./models/7B/llama-model.gguf",n_ctx=2048)OpenAI Compatible Web Serverllama-cpp-pythonoffers a web server which aims to act as a drop-in replacement for the OpenAI API.
This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).To install the server package and get started:pipinstallllama-cpp-python[server]python3-mllama_cpp.server--modelmodels/7B/llama-model.ggufSimilar to Hardware Acceleration section above, you can also install with GPU (cuBLAS) support like this:CMAKE_ARGS="-DLLAMA_CUBLAS=on"FORCE_CMAKE=1pipinstallllama-cpp-python[server]python3-mllama_cpp.server--modelmodels/7B/llama-model.gguf--n_gpu_layers35Navigate tohttp://localhost:8000/docsto see the OpenAPI documentation.To bind to0.0.0.0to enable remote connections, usepython3 -m llama_cpp.server --host 0.0.0.0.
Similarly, to change the port (default is 8000), use--port.You probably also want to set the prompt format. For chatml, usepython3-mllama_cpp.server--modelmodels/7B/llama-model.gguf--chat_formatchatmlThat will format the prompt according to how model expects it. You can find the prompt format in the model card.
For possible options, seellama_cpp/llama_chat_format.pyand look for lines starting with "@register_chat_format".Web Server FeaturesLocal Copilot replacementFunction Calling supportVision API supportMultiple ModelsDocker imageA Docker image is available onGHCR. To run the server:dockerrun--rm-it-p8000:8000-v/path/to/models:/models-eMODEL=/models/llama-model.ggufghcr.io/abetlen/llama-cpp-python:latestDocker on termux (requires root)is currently the only known way to run this on phones, seetermux support issueLow-level APIAPI ReferenceThe low-level API is a directctypesbinding to the C API provided byllama.cpp.
The entire low-level API can be found inllama_cpp/llama_cpp.pyand directly mirrors the C API inllama.h.Below is a short example demonstrating how to use the low-level API to tokenize a prompt:>>>importllama_cpp>>>importctypes>>>llama_cpp.llama_backend_init(numa=False)# Must be called once at the start of each program>>>params=llama_cpp.llama_context_default_params()# use bytes for char * params>>>model=llama_cpp.llama_load_model_from_file(b"./models/7b/llama-model.gguf",params)>>>ctx=llama_cpp.llama_new_context_with_model(model,params)>>>max_tokens=params.n_ctx# use ctypes arrays for array params>>>tokens=(llama_cpp.llama_token*int(max_tokens))()>>>n_tokens=llama_cpp.llama_tokenize(ctx,b"Q: Name the planets in the solar system? A: ",tokens,max_tokens,add_bos=llama_cpp.c_bool(True))>>>llama_cpp.llama_free(ctx)Check out theexamples folderfor more examples of using the low-level API.DocumentationDocumentation is available viahttps://llama-cpp-python.readthedocs.io/.
If you find any issues with the documentation, please open an issue or submit a PR.DevelopmentThis package is under active development and I welcome any contributions.To get started, clone the repository and install the package in editable / development mode:gitclone--recurse-submoduleshttps://github.com/abetlen/llama-cpp-python.gitcdllama-cpp-python# Upgrade pip (required for editable mode)pipinstall--upgradepip# Install with pippipinstall-e.# if you want to use the fastapi / openapi serverpipinstall-e.[server]# to install all optional dependenciespipinstall-e.[all]# to clear the local build cachemakecleanFAQAre there pre-built binaries / binary wheels available?The recommended installation method is to install from source as described above.
The reason for this is thatllama.cppis built with compiler optimizations that are specific to your system.
Using pre-built binaries would require disabling these optimizations or supporting a large number of pre-built binaries for each platform.That being said there are some pre-built binaries available through the Releases as well as some community provided wheels.In the future, I would like to provide pre-built binaries and wheels for common platforms and I'm happy to accept any useful contributions in this area.
This is currently being tracked in#741How does this compare to other Python bindings ofllama.cpp?I originally wrote this package for my own use with two goals in mind:Provide a simple process to installllama.cppand access the full C API inllama.hfrom PythonProvide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to usellama.cppAny contributions and changes to this package will be made with these goals in mind.LicenseThis project is licensed under the terms of the MIT license.