Package: llama2-wrapper
llama2-wrapperUsellama2-wrapperas your local llama2 backend for Generative Agents/Apps,colab example.Run OpenAI Compatible APIon Llama2 models.FeaturesSupporting models:Llama-2-7b/13b/70b,Llama-2-GPTQ,Llama-2-GGML,CodeLlama...Supporting model backends:tranformers,bitsandbytes(8-bit inference),AutoGPTQ(4-bit inference),llama.cppDemos:Run Llama2 on MacBook Air;Run Llama2 on Colab T4 GPUUsellama2-wrapperas your local llama2 backend for Generative Agents/Apps;colab example.Run OpenAI Compatible APIon Llama2 models.News,Benchmark,Issue Solutionsllama2-wrapperis the backend and part ofllama2-webui, which can run any Llama 2 locally with gradio UI on GPU or CPU from anywhere (Linux/Windows/Mac).Installpipinstallllama2-wrapperStart OpenAI Compatible  APIpython -m llama2_wrapper.serverit will usellama.cppas the backend by default to runllama-2-7b-chat.ggmlv3.q4_0.binmodel.Start Fast API forgptqbackend:python -m llama2_wrapper.server --backend_type gptqNavigate tohttp://localhost:8000/docsto see the OpenAPI documentation.API Usage__call____call__()is the function to generate text from a prompt.For example, run ggml llama2 model on CPU,colab example:fromllama2_wrapperimportLLAMA2_WRAPPER,get_promptllama2_wrapper=LLAMA2_WRAPPER()# Default running on backend llama.cpp.# Automatically downloading model to: ./models/llama-2-7b-chat.ggmlv3.q4_0.binprompt="Do you know Pytorch"# llama2_wrapper() will run __call__()answer=llama2_wrapper(get_prompt(prompt),temperature=0.9)Run gptq llama2 model on Nvidia GPU,colab example:fromllama2_wrapperimportLLAMA2_WRAPPERllama2_wrapper=LLAMA2_WRAPPER(backend_type="gptq")# Automatically downloading model to: ./models/Llama-2-7b-Chat-GPTQRun llama2 7b with bitsandbytes 8 bit with amodel_path:fromllama2_wrapperimportLLAMA2_WRAPPERllama2_wrapper=LLAMA2_WRAPPER(model_path="./models/Llama-2-7b-chat-hf",backend_type="transformers",load_in_8bit=True)completioncompletion()is the function to generate text from a prompt for OpenAI compatible API/v1/completions.llama2_wrapper=LLAMA2_WRAPPER()prompt=get_prompt("Hi do you know Pytorch?")print(llm.completion(prompt))chat_completionchat_completion()is the function to generate text from a dialog (chat history) for OpenAI compatible API/v1/chat/completions.llama2_wrapper=LLAMA2_WRAPPER()dialog=[{"role":"system","content":"You are a helpful, respectful and honest assistant. "},{"role":"user","content":"Hi do you know Pytorch?",},]print(llm.chat_completion(dialog))generategenerate()is the function to create a generator of response from a prompt.This is useful when you want to stream the output like typing in the chatbot.llama2_wrapper=LLAMA2_WRAPPER()prompt=get_prompt("Hi do you know Pytorch?")forresponseinllama2_wrapper.generate(prompt):print(response)The response will be like:Yes, 
Yes, I'm 
Yes, I'm familiar 
Yes, I'm familiar with 
Yes, I'm familiar with PyTorch! 
...runrun()is similar togenerate(), butrun()can also acceptchat_historyandsystem_promptfrom the users.It will process the input message to llama2 prompt template withchat_historyandsystem_promptfor a chatbot-like app.get_promptget_prompt()will process the input message to llama2 prompt withchat_historyandsystem_promptfor chatbot.By default,chat_historyandsystem_promptare empty andget_prompt()will add llama2 prompt template to your message:prompt=get_prompt("Hi do you know Pytorch?")prompt will be:
 





Hi do you know Pytorch? 
If useget_prompt("Hi do you know Pytorch?", system_prompt="You are a helpful..."):
 

You are a helpful, respectful and honest assistant. 



Hi do you know Pytorch? 
get_prompt_for_dialogget_prompt_for_dialog()will process dialog (chat history) to llama2 prompt for OpenAI compatible API/v1/chat/completions.dialog=[{"role":"system","content":"You are a helpful, respectful and honest assistant. "},{"role":"user","content":"Hi do you know Pytorch?",},]prompt=get_prompt_for_dialog("Hi do you know Pytorch?")# 
 
# You are a helpful, respectful and honest assistant.# 
## Hi do you know Pytorch?