Package: llama-cpp-python
🦙 Python Bindings forllama.cppSimple Python bindings for@ggerganov'sllama.cpplibrary.
This package provides:Low-level access to C API viactypesinterface.High-level Python API for text completionOpenAI-like APILangChain compatibilityLlamaIndex compatibilityOpenAI compatible web serverLocal Copilot replacementFunction Calling supportVision API supportMultiple ModelsDocumentation is available athttps://llama-cpp-python.readthedocs.io/en/latest.InstallationRequirements:Python 3.8+C compilerLinux: gcc or clangWindows: Visual Studio or MinGWMacOS: XcodeTo install the package, run:pipinstallllama-cpp-pythonThis will also buildllama.cppfrom source and install it alongside this python package.If this fails, add--verboseto thepip installsee the full cmake build log.Installation Configurationllama.cppsupports a number of hardware acceleration backends to speed up inference as well as backend specific options. See thellama.cpp READMEfor a full list.Allllama.cppcmake build options can be set via theCMAKE_ARGSenvironment variable or via the--config-settings / -Ccli flag during installation.Environment Variables# Linux and MacCMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS"\pipinstallllama-cpp-python# Windows$env:CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS"pipinstallllama-cpp-pythonCLI / requirements.txtThey can also be set viapip install -C / --config-settingscommand and saved to arequirements.txtfile:pipinstall--upgradepip# ensure pip is up to datepipinstallllama-cpp-python\-Ccmake.args="-DLLAMA_BLAS=ON;-DLLAMA_BLAS_VENDOR=OpenBLAS"# requirements.txt

llama-cpp-python -C cmake.args="-DLLAMA_BLAS=ON;-DLLAMA_BLAS_VENDOR=OpenBLAS"Supported BackendsBelow are some common backends, their build commands and any additional environment variables required.OpenBLAS (CPU)To install with OpenBLAS, set theLLAMA_BLASandLLAMA_BLAS_VENDORenvironment variables before installing:CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS"pipinstallllama-cpp-pythoncuBLAS (CUDA)To install with cuBLAS, set theLLAMA_CUBLAS=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_CUBLAS=on"pipinstallllama-cpp-pythonMetalTo install with Metal (MPS), set theLLAMA_METAL=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_METAL=on"pipinstallllama-cpp-pythonCLBlast (OpenCL)To install with CLBlast, set theLLAMA_CLBLAST=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_CLBLAST=on"pipinstallllama-cpp-pythonhipBLAS (ROCm)To install with hipBLAS / ROCm support for AMD cards, set theLLAMA_HIPBLAS=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_HIPBLAS=on"pipinstallllama-cpp-pythonVulkanTo install with Vulkan support, set theLLAMA_VULKAN=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_VULKAN=on"pipinstallllama-cpp-pythonKomputeTo install with Kompute support, set theLLAMA_KOMPUTE=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_KOMPUTE=on"pipinstallllama-cpp-pythonSYCLTo install with SYCL support, set theLLAMA_SYCL=onenvironment variable before installing:source/opt/intel/oneapi/setvars.shCMAKE_ARGS="-DLLAMA_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx"pipinstallllama-cpp-pythonWindows NotesError: Can't find 'nmake' or 'CMAKE_C_COMPILER'If you run into issues where it complains it can't find'nmake''?'or CMAKE_C_COMPILER, you can extract w64devkit asmentioned in llama.cpp repoand add those manually to CMAKE_ARGS before runningpipinstall:$env:CMAKE_GENERATOR = "MinGW Makefiles"
$env:CMAKE_ARGS = "-DLLAMA_OPENBLAS=on -DCMAKE_C_COMPILER=C:/w64devkit/bin/gcc.exe -DCMAKE_CXX_COMPILER=C:/w64devkit/bin/g++.exe"See the above instructions and setCMAKE_ARGSto the BLAS backend you want to use.MacOS NotesDetailed MacOS Metal GPU install documentation is available atdocs/install/macos.mdM1 Mac Performance IssueNote: If you are using Apple Silicon (M1) Mac, make sure you have installed a version of Python that supports arm64 architecture. For example:wgethttps://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bashMiniforge3-MacOSX-arm64.shOtherwise, while installing it will build the llama.cpp x86 version which will be 10x slower on Apple Silicon (M1) Mac.M Series Mac Error: `(mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))`Try installing withCMAKE_ARGS="-DCMAKE_OSX_ARCHITECTURES=arm64 -DCMAKE_APPLE_SILICON_PROCESSOR=arm64 -DLLAMA_METAL=on"pipinstall--upgrade--verbose--force-reinstall--no-cache-dirllama-cpp-pythonUpgrading and ReinstallingTo upgrade and rebuildllama-cpp-pythonadd--upgrade --force-reinstall --no-cache-dirflags to thepip installcommand to ensure the package is rebuilt from source.High-level APIAPI ReferenceThe high-level API provides a simple managed interface through theLlamaclass.Below is a short example demonstrating how to use the high-level API to for basic text completion:>>>fromllama_cppimportLlama>>>llm=Llama(model_path="./models/7B/llama-model.gguf",# n_gpu_layers=-1, # Uncomment to use GPU acceleration# seed=1337, # Uncomment to set a specific seed# n_ctx=2048, # Uncomment to increase the context window)>>>output=llm("Q: Name the planets in the solar system? A: ",# Promptmax_tokens=32,# Generate up to 32 tokens, set to None to generate up to the end of the context windowstop=["Q:","\n"],# Stop generating just before the model would generate a new questionecho=True# Echo the prompt back in the output)# Generate a completion, can also call create_completion>>>print(output){"id":"cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","object":"text_completion","created":1679561337,"model":"./models/7B/llama-model.gguf","choices":[{"text":"Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.","index":0,"logprobs":None,"finish_reason":"stop"}],"usage":{"prompt_tokens":14,"completion_tokens":28,"total_tokens":42}}Text completion is available through the__call__andcreate_completionmethods of theLlamaclass.Pulling models from Hugging Face HubYou can downloadLlamamodels inggufformat directly from Hugging Face using thefrom_pretrainedmethod.
You'll need to install thehuggingface-hubpackage to use this feature (pip install huggingface-hub).llm=Llama.from_pretrained(repo_id="Qwen/Qwen1.5-0.5B-Chat-GGUF",filename="*q8_0.gguf",verbose=False)By defaultfrom_pretrainedwill download the model to the huggingface cache directory, you can then manage installed model files with thehuggingface-clitool.Chat CompletionThe high-level API also provides a simple interface for chat completion.Note thatchat_formatoption must be set for the particular model you are using.>>>fromllama_cppimportLlama>>>llm=Llama(model_path="path/to/llama-2/llama-model.gguf",chat_format="llama-2")>>>llm.create_chat_completion(messages=[{"role":"system","content":"You are an assistant who perfectly describes images."},{"role":"user","content":"Describe this image in detail please."}])Chat completion is available through thecreate_chat_completionmethod of theLlamaclass.For OpenAI API v1 compatibility, you use thecreate_chat_completion_openai_v1method which will return pydantic models instead of dicts.JSON and JSON Schema ModeTo constrain chat responses to only valid JSON or a specific JSON Schema use theresponse_formatargument increate_chat_completion.JSON ModeThe following example will constrain the response to valid JSON strings only.>>>fromllama_cppimportLlama>>>llm=Llama(model_path="path/to/model.gguf",chat_format="chatml")>>>llm.create_chat_completion(messages=[{"role":"system","content":"You are a helpful assistant that outputs in JSON.",},{"role":"user","content":"Who won the world series in 2020"},],response_format={"type":"json_object",},temperature=0.7,)JSON Schema ModeTo constrain the response further to a specific JSON Schema add the schema to theschemaproperty of theresponse_formatargument.>>>fromllama_cppimportLlama>>>llm=Llama(model_path="path/to/model.gguf",chat_format="chatml")>>>llm.create_chat_completion(messages=[{"role":"system","content":"You are a helpful assistant that outputs in JSON.",},{"role":"user","content":"Who won the world series in 2020"},],response_format={"type":"json_object","schema":{"type":"object","properties":{"team_name":{"type":"string"}},"required":["team_name"],},},temperature=0.7,)Function CallingThe high-level API supports OpenAI compatible function and tool calling. This is possible through thefunctionarypre-trained models chat format or through the genericchatml-function-callingchat format.>>>fromllama_cppimportLlama>>>llm=Llama(model_path="path/to/chatml/llama-model.gguf",chat_format="chatml-function-calling")>>>llm.create_chat_completion(messages=[{"role":"system","content":"A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user's questions. The assistant calls functions with appropriate input when necessary"},{"role":"user","content":"Extract Jason is 25 years old"}],tools=[{"type":"function","function":{"name":"UserDetail","parameters":{"type":"object","title":"UserDetail","properties":{"name":{"title":"Name","type":"string"},"age":{"title":"Age","type":"integer"}},"required":["name","age"]}}}],tool_choice=[{"type":"function","function":{"name":"UserDetail"}}])Functionary v2The various gguf-converted files for this set of models can be foundhere. Functionary is able to intelligently call functions and also analyze any provided function outputs to generate coherent responses. All v2 models of functionary supportsparallel function calling. You can provide eitherfunctionary-v1orfunctionary-v2for thechat_formatwhen initializing the Llama class.Due to discrepancies between llama.cpp and HuggingFace's tokenizers, it is required to provide HF Tokenizer for functionary. TheLlamaHFTokenizerclass can be initialized and passed into the Llama class. This will override the default llama.cpp tokenizer used in Llama class. The tokenizer files are already included in the respective HF repositories hosting the gguf files.>>>fromllama_cppimportLlama>>>fromllama_cpp.llama_tokenizerimportLlamaHFTokenizer>>>llm=Llama.from_pretrained(repo_id="meetkai/functionary-small-v2.2-GGUF",filename="functionary-small-v2.2.q4_0.gguf",chat_format="functionary-v2",tokenizer=LlamaHFTokenizer.from_pretrained("meetkai/functionary-small-v2.2-GGUF"))Multi-modal Modelsllama-cpp-pythonsupports the llava1.5 family of multi-modal models which allow the language model to
read information from both text and images.You'll first need to download one of the available multi-modal models in GGUF format:llava-v1.5-7bllava-v1.5-13bbakllava-1-7bThen you'll need to use a custom chat handler to load the clip model and process the chat messages and images.>>>fromllama_cppimportLlama>>>fromllama_cpp.llama_chat_formatimportLlava15ChatHandler>>>chat_handler=Llava15ChatHandler(clip_model_path="path/to/llava/mmproj.bin")>>>llm=Llama(model_path="./path/to/llava/llama-model.gguf",chat_handler=chat_handler,n_ctx=2048,# n_ctx should be increased to accomodate the image embeddinglogits_all=True,# needed to make llava work)>>>llm.create_chat_completion(messages=[{"role":"system","content":"You are an assistant who perfectly describes images."},{"role":"user","content":[{"type":"image_url","image_url":{"url":"https://.../image.png"}},{"type":"text","text":"Describe this image in detail please."}]}])Loading a Local ImageImages can be passed as base64 encoded data URIs. The following example demonstrates how to do this.importbase64defimage_to_base64_data_uri(file_path):withopen(file_path,"rb")asimg_file:base64_data=base64.b64encode(img_file.read()).decode('utf-8')returnf"data:image/png;base64,{base64_data}"# Replace 'file_path.png' with the actual path to your PNG filefile_path='file_path.png'data_uri=image_to_base64_data_uri(file_path)messages=[{"role":"system","content":"You are an assistant who perfectly describes images."},{"role":"user","content":[{"type":"image_url","image_url":{"url":data_uri}},{"type":"text","text":"Describe this image in detail please."}]}]Speculative Decodingllama-cpp-pythonsupports speculative decoding which allows the model to generate completions based on a draft model.The fastest way to use speculative decoding is through theLlamaPromptLookupDecodingclass.Just pass this as a draft model to theLlamaclass during initialization.fromllama_cppimportLlamafromllama_cpp.llama_speculativeimportLlamaPromptLookupDecodingllama=Llama(model_path="path/to/model.gguf",draft_model=LlamaPromptLookupDecoding(num_pred_tokens=10)# num_pred_tokens is the number of tokens to predict 10 is the default and generally good for gpu, 2 performs better for cpu-only machines.)EmbeddingsTo generate text embeddings usecreate_embedding.importllama_cppllm=llama_cpp.Llama(model_path="path/to/model.gguf",embeddings=True)embeddings=llm.create_embedding("Hello, world!")# or create multiple embeddings at onceembeddings=llm.create_embedding(["Hello, world!","Goodbye, world!"])Adjusting the Context WindowThe context window of the Llama models determines the maximum number of tokens that can be processed at once. By default, this is set to 512 tokens, but can be adjusted based on your requirements.For instance, if you want to work with larger contexts, you can expand the context window by setting the n_ctx parameter when initializing the Llama object:llm=Llama(model_path="./models/7B/llama-model.gguf",n_ctx=2048)OpenAI Compatible Web Serverllama-cpp-pythonoffers a web server which aims to act as a drop-in replacement for the OpenAI API.
This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).To install the server package and get started:pipinstall'llama-cpp-python[server]'python3-mllama_cpp.server--modelmodels/7B/llama-model.ggufSimilar to Hardware Acceleration section above, you can also install with GPU (cuBLAS) support like this:CMAKE_ARGS="-DLLAMA_CUBLAS=on"FORCE_CMAKE=1pipinstall'llama-cpp-python[server]'python3-mllama_cpp.server--modelmodels/7B/llama-model.gguf--n_gpu_layers35Navigate tohttp://localhost:8000/docsto see the OpenAPI documentation.To bind to0.0.0.0to enable remote connections, usepython3 -m llama_cpp.server --host 0.0.0.0.
Similarly, to change the port (default is 8000), use--port.You probably also want to set the prompt format. For chatml, usepython3-mllama_cpp.server--modelmodels/7B/llama-model.gguf--chat_formatchatmlThat will format the prompt according to how model expects it. You can find the prompt format in the model card.
For possible options, seellama_cpp/llama_chat_format.pyand look for lines starting with "@register_chat_format".If you havehuggingface-hubinstalled, you can also use the--hf_model_repo_idflag to load a model from the Hugging Face Hub.python3-mllama_cpp.server--hf_model_repo_idQwen/Qwen1.5-0.5B-Chat-GGUF--model'*q8_0.gguf'Web Server FeaturesLocal Copilot replacementFunction Calling supportVision API supportMultiple ModelsDocker imageA Docker image is available onGHCR. To run the server:dockerrun--rm-it-p8000:8000-v/path/to/models:/models-eMODEL=/models/llama-model.ggufghcr.io/abetlen/llama-cpp-python:latestDocker on termux (requires root)is currently the only known way to run this on phones, seetermux support issueLow-level APIAPI ReferenceThe low-level API is a directctypesbinding to the C API provided byllama.cpp.
The entire low-level API can be found inllama_cpp/llama_cpp.pyand directly mirrors the C API inllama.h.Below is a short example demonstrating how to use the low-level API to tokenize a prompt:>>>importllama_cpp>>>importctypes>>>llama_cpp.llama_backend_init(False)# Must be called once at the start of each program>>>params=llama_cpp.llama_context_default_params()# use bytes for char * params>>>model=llama_cpp.llama_load_model_from_file(b"./models/7b/llama-model.gguf",params)>>>ctx=llama_cpp.llama_new_context_with_model(model,params)>>>max_tokens=params.n_ctx# use ctypes arrays for array params>>>tokens=(llama_cpp.llama_token*int(max_tokens))()>>>n_tokens=llama_cpp.llama_tokenize(ctx,b"Q: Name the planets in the solar system? A: ",tokens,max_tokens,llama_cpp.c_bool(True))>>>llama_cpp.llama_free(ctx)Check out theexamples folderfor more examples of using the low-level API.DocumentationDocumentation is available viahttps://llama-cpp-python.readthedocs.io/.
If you find any issues with the documentation, please open an issue or submit a PR.DevelopmentThis package is under active development and I welcome any contributions.To get started, clone the repository and install the package in editable / development mode:gitclone--recurse-submoduleshttps://github.com/abetlen/llama-cpp-python.gitcdllama-cpp-python# Upgrade pip (required for editable mode)pipinstall--upgradepip# Install with pippipinstall-e.# if you want to use the fastapi / openapi serverpipinstall-e.[server]# to install all optional dependenciespipinstall-e.[all]# to clear the local build cachemakecleanYou can also test out specific commits oflama.cppby checking out the desired commit in thevendor/llama.cppsubmodule and then runningmake cleanandpip install -e .again. Any changes in thellama.hAPI will require
changes to thellama_cpp/llama_cpp.pyfile to match the new API (additional changes may be required elsewhere).FAQAre there pre-built binaries / binary wheels available?The recommended installation method is to install from source as described above.
The reason for this is thatllama.cppis built with compiler optimizations that are specific to your system.
Using pre-built binaries would require disabling these optimizations or supporting a large number of pre-built binaries for each platform.That being said there are some pre-built binaries available through the Releases as well as some community provided wheels.In the future, I would like to provide pre-built binaries and wheels for common platforms and I'm happy to accept any useful contributions in this area.
This is currently being tracked in#741How does this compare to other Python bindings ofllama.cpp?I originally wrote this package for my own use with two goals in mind:Provide a simple process to installllama.cppand access the full C API inllama.hfrom PythonProvide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to usellama.cppAny contributions and changes to this package will be made with these goals in mind.LicenseThis project is licensed under the terms of the MIT license.