Package: llm-llama-cpp
llm-llama-cppLLMplugin for running models usingllama.cppInstallationInstall this plugin in the same environment asllm.llminstallllm-llama-cppThe plugin has an additional dependency onllama-cpp-pythonwhich needs to be installed separately.If you have a C compiler available on your system you can install that like so:llminstallllama-cpp-pythonYou could also try installing one of the wheels made available in theirlatest releaseon GitHub. Find the URL to the wheel for your platform, if one exists, and run:llminstallhttps://...If you are on an Apple Silicon Mac you can try this command, which should compile the package with METAL support for running on your GPU:CMAKE_ARGS="-DLLAMA_METAL=on"FORCE_CMAKE=1llminstallllama-cpp-pythonRunning a GGUF model directlyThe quickest way to try this plugin out is to download a GGUF file and execute that using theggufmodel with the-o path PATHoption:For example, download theuna-cybertron-7b-v2-bf16.Q8_0.gguffile fromTheBloke/una-cybertron-7B-v2-GGUFand execute it like this:llm-mgguf\-opathuna-cybertron-7b-v2-bf16.Q8_0.gguf\'Instruction: Five reasons to get a pet walrusResponse:'The output starts like this:Walruses are fascinating animals that possess unique qualities that can captivate and entertain you for hours on end. Getting the chance to be around one regularly would ensure that you'll never run out of interesting things to learn about them, whether from an educational or personal standpoint.Pet walruses can help alleviate depression and anxiety, as they require constant care and attention. Nurturing a relationship with these intelligent creatures provides comfort and fulfillment, fostering a sense of purpose in your daily life. Moreover, their playful nature encourages laughter and joy, promoting overall happiness. [...]Adding modelsYou can also add or download models to execute them directly using the-moption.This tool should work with any model that works withllama.cpp.The plugin can download models for you. Try running this command:llmllama-cppdownload-model\https://huggingface.co/TheBloke/Llama-2-7b-Chat-GGUF/resolve/main/llama-2-7b-chat.Q6_K.gguf\--aliasllama2-chat--aliasl2c--llama2-chatThis will download the Llama 2 7B Chat GGUF model file (this one is 5.53GB), save it and register it with the plugin - with two aliases,llama2-chatandl2c.The--llama2-chatoption configures it to run using a special Llama 2 Chat prompt format. You should omit this for models that are not Llama 2 Chat models.If you have already downloaded allama.cppcompatible model you can tell the plugin to read it from its current location like this:llmllama-cppadd-modelpath/to/llama-2-7b-chat.Q6_K.gguf\--aliasl27c--llama2-chatThe model filename (minus the.ggufextension) will be registered as its ID for executing the model.You can also set one or more aliases using the--aliasoption.You can see a list of models you have registered in this way like this:llmllama-cppmodelsModels are registered in amodels.jsonfile. You can find the path to that file in order to edit it directly like so:llmllama-cppmodels-fileFor example, to edit that file in Vim:vim"$(llmllama-cppmodels-file)"To find the directory with downloaded models, run:llmllama-cppmodels-dirHere's how to change to that directory:cd"$(llmllama-cppmodels-dir)"Running a prompt through a modelOnce you have downloaded and added a model, you can run a prompt like this:llm-mllama-2-7b-chat.Q6_K'five names for a cute pet skunk'Or if you registered an alias you can use that instead:llm-mllama2-chat'five creative names for a pet hedgehog'More models to tryLlama 2 7BThis model is Llama 2 7B GGML without the chat training. You'll need to prompt it slightly differently:llmllama-cppdownload-model\https://huggingface.co/TheBloke/Llama-2-7B-GGUF/resolve/main/llama-2-7b.Q6_K.gguf\--aliasllama2Try prompts that expect to be completed by the model, for example:llm-mllama2'Three fancy names for a posh albatross are:'Llama 2 Chat 13BThis model is the Llama 2 13B Chat GGML model - a 10.7GB download:llmllama-cppdownload-model\'https://huggingface.co/TheBloke/Llama-2-13B-chat-GGUF/resolve/main/llama-2-13b-chat.Q6_K.gguf'\-allama2-chat-13b--llama2-chatLlama 2 Python 13BThis model is the Llama 2 13B Python GGML model - a 9.24GB download:llmllama-cppdownload-model\'https://huggingface.co/TheBloke/CodeLlama-13B-Python-GGUF/resolve/main/codellama-13b-python.Q5_K_M.gguf'\-allama2-python-13b--llama2-chatOptionsThe following options are available:-o verbose 1- output more verbose logging-o max_tokens 100- max tokens to return. Defaults to 4000.-o no_gpu 1- remove the default `n_gpu_layers=1`` argument, which should disable GPU usage-o n_gpu_layers 10- increase then_gpu_layersargument to a higher value (the default is1)-o n_ctx 1024- set then_ctxargument to1024(the default is4000)For example:llmchat-mllama2-chat-13b-on_ctx1024These are mainly provided to support experimenting with different ways of executing the underlying model.DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cdllm-llama-cpp
python3-mvenvvenvsourcevenv/bin/activateNow install the dependencies and test dependencies:pipinstall-e'.[test]'To run the tests:pytest