Package: llama-cpp-python-kirp
🦙 Python Bindings forllama.cppSimple Python bindings for@ggerganov'sllama.cpplibrary.
This package provides:Low-level access to C API viactypesinterface.High-level Python API for text completionOpenAI-like APILangChain compatibilityDocumentation is available athttps://llama-cpp-python.readthedocs.io/en/latest.[!WARNING]Starting with version 0.1.79 the model format has changed fromggmlv3togguf. Old model files can be converted using theconvert-llama-ggmlv3-to-gguf.pyscript inllama.cppInstallation from PyPIInstall from PyPI (requires a c compiler):pipinstallllama-cpp-pythonThe above command will attempt to install the package and buildllama.cppfrom source.
This is the recommended installation method as it ensures thatllama.cppis built with the available optimizations for your system.If you have previously installedllama-cpp-pythonthrough pip and want to upgrade your version or rebuild the package with different  compiler options, please add the following flags to ensure that the package is rebuilt correctly:pipinstallllama-cpp-python--force-reinstall--upgrade--no-cache-dirNote: If you are using Apple Silicon (M1) Mac, make sure you have installed a version of Python that supports arm64 architecture. For example:wget https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh
bash Miniforge3-MacOSX-arm64.shOtherwise, while installing it will build the llama.ccp x86 version which will be 10x slower on Apple Silicon (M1) Mac.Installation with Hardware Accelerationllama.cppsupports multiple BLAS backends for faster processing.To install with OpenBLAS, set theLLAMA_BLAS and LLAMA_BLAS_VENDORenvironment variables before installing:CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS"pipinstallllama-cpp-pythonTo install with cuBLAS, set theLLAMA_CUBLAS=1environment variable before installing:CMAKE_ARGS="-DLLAMA_CUBLAS=on"pipinstallllama-cpp-pythonTo install with CLBlast, set theLLAMA_CLBLAST=1environment variable before installing:CMAKE_ARGS="-DLLAMA_CLBLAST=on"pipinstallllama-cpp-pythonTo install with Metal (MPS), set theLLAMA_METAL=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_METAL=on"pipinstallllama-cpp-pythonTo install with hipBLAS / ROCm support for AMD cards, set theLLAMA_HIPBLAS=onenvironment variable before installing:CMAKE_ARGS="-DLLAMA_HIPBLAS=on"pipinstallllama-cpp-pythonWindows remarksTo set the variablesCMAKE_ARGSin PowerShell, follow the next steps (Example using, OpenBLAS):$env:CMAKE_ARGS = "-DLLAMA_OPENBLAS=on"Then, callpipafter setting the variables:pip install llama-cpp-pythonSee the above instructions and setCMAKE_ARGSto the BLAS backend you want to use.MacOS remarksDetailed MacOS Metal GPU install documentation is available atdocs/install/macos.mdHigh-level APIThe high-level API provides a simple managed interface through theLlamaclass.Below is a short example demonstrating how to use the high-level API to generate text:>>>fromllama_cppimportLlama>>>llm=Llama(model_path="./models/7B/llama-model.gguf")>>>output=llm("Q: Name the planets in the solar system? A: ",max_tokens=32,stop=["Q:","\n"],echo=True)>>>print(output){"id":"cmpl-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx","object":"text_completion","created":1679561337,"model":"./models/7B/llama-model.gguf","choices":[{"text":"Q: Name the planets in the solar system? A: Mercury, Venus, Earth, Mars, Jupiter, Saturn, Uranus, Neptune and Pluto.","index":0,"logprobs":None,"finish_reason":"stop"}],"usage":{"prompt_tokens":14,"completion_tokens":28,"total_tokens":42}}Adjusting the Context WindowThe context window of the Llama models determines the maximum number of tokens that can be processed at once. By default, this is set to 512 tokens, but can be adjusted based on your requirements.For instance, if you want to work with larger contexts, you can expand the context window by setting the n_ctx parameter when initializing the Llama object:llm=Llama(model_path="./models/7B/llama-model.gguf",n_ctx=2048)Loading llama-2 70bLlama2 70b must set then_gqaparameter (grouped-query attention factor) to 8 when loading:llm=Llama(model_path="./models/70B/llama-model.gguf",n_gqa=8)Web Serverllama-cpp-pythonoffers a web server which aims to act as a drop-in replacement for the OpenAI API.
This allows you to use llama.cpp compatible models with any OpenAI compatible client (language libraries, services, etc).To install the server package and get started:pipinstallllama-cpp-python[server]python3-mllama_cpp.server--modelmodels/7B/llama-model.ggufSimilar to Hardware Acceleration section above, you can also install with GPU (cuBLAS) support like this:CMAKE_ARGS="-DLLAMA_CUBLAS=on"FORCE_CMAKE=1pipinstallllama-cpp-python[server]python3-mllama_cpp.server--modelmodels/7B/llama-model.gguf--n_gpu_layers35Navigate tohttp://localhost:8000/docsto see the OpenAPI documentation.Docker imageA Docker image is available onGHCR. To run the server:dockerrun--rm-it-p8000:8000-v/path/to/models:/models-eMODEL=/models/llama-model.ggufghcr.io/abetlen/llama-cpp-python:latestDocker on termux (requires root)is currently the only known way to run this on phones, seetermux support issueLow-level APIThe low-level API is a directctypesbinding to the C API provided byllama.cpp.
The entire low-level API can be found inllama_cpp/llama_cpp.pyand directly mirrors the C API inllama.h.Below is a short example demonstrating how to use the low-level API to tokenize a prompt:>>>importllama_cpp>>>importctypes>>>llama_cpp.llama_backend_init(numa=False)# Must be called once at the start of each program>>>params=llama_cpp.llama_context_default_params()# use bytes for char * params>>>model=llama_cpp.llama_load_model_from_file(b"./models/7b/llama-model.gguf",params)>>>ctx=llama_cpp.llama_new_context_with_model(model,params)>>>max_tokens=params.n_ctx# use ctypes arrays for array params>>>tokens=(llama_cpp.llama_token*int(max_tokens))()>>>n_tokens=llama_cpp.llama_tokenize(ctx,b"Q: Name the planets in the solar system? A: ",tokens,max_tokens,add_bos=llama_cpp.c_bool(True))>>>llama_cpp.llama_free(ctx)Check out theexamples folderfor more examples of using the low-level API.DocumentationDocumentation is available athttps://abetlen.github.io/llama-cpp-python.
If you find any issues with the documentation, please open an issue or submit a PR.DevelopmentThis package is under active development and I welcome any contributions.To get started, clone the repository and install the package in editable / development mode:gitclone--recurse-submoduleshttps://github.com/abetlen/llama-cpp-python.gitcdllama-cpp-python# Upgrade pip (required for editable mode)pipinstall--upgradepip# Install with pippipinstall-e.# if you want to use the fastapi / openapi serverpipinstall-e.[server]# to install all optional dependenciespipinstall-e.[all]# to clear the local build cachemakecleanHow does this compare to other Python bindings ofllama.cpp?I originally wrote this package for my own use with two goals in mind:Provide a simple process to installllama.cppand access the full C API inllama.hfrom PythonProvide a high-level Python API that can be used as a drop-in replacement for the OpenAI API so existing apps can be easily ported to usellama.cppAny contributions and changes to this package will be made with these goals in mind.LicenseThis project is licensed under the terms of the MIT license.