Package: ooba-api-client
Python API Client for Ooba-Booga's Text Generation Web UIAn API client for the text generation UI, with sane defaults.Motivation: documentation isn't great, examples are gnarly, not seeing an existing library.Supported use cases:generate / instructchatstreaming instructstreaming chatmodel infomodel loadingInstallationpip install ooba-api-clientWhat model should I use?If you're new to LLMs, you may be unsure what model to use for your use case.In general, models tend to come in three flavors:a foundational modela chat modelan instruct modelThe foundational model typically is used for text prediction (typically suggestions), if its even good for that. You probably don't want this. Foundamational models often need behavior training to be useful.The chat model is used for conversation histories. This would be the preferred model if you're trying to create a chat bot who replies to the user.The instruct models are tuned towards following instructions. If your interest is in creating autonomous agents, this is probably what you want. Note that you can always roll up chat histories into a single prompt.Exampleimportloggingfromooba_apiimportOobaApiClient,Parameters,LlamaInstructPromptlogger=logging.getLogger("ooba_api")logger.setLevel(logging.DEBUG)logging.basicConfig(level=logging.DEBUG)client=OobaApiClient()# defaults to http://localhost:5000response=client.instruct(LlamaInstructPrompt(system_prompt=("Generate only the requested code from the user. Do not generate anything else. ""Be succint. Generate markdown of the code, and give the correct type. ""If the code is python use ```python for the markdown. Do not explain afterwards"),prompt="Generate a Python function to reverse the contents of a file",),parameters=Parameters(temperature=0.2,repetition_penalty=1.05),)print(response)```python
def reverse_file(file_path):
    with open(file_path, "r") as f:
        content = f.read()
    return content[::-1]
```Model Information and LoadingTo get the currently loaded model:fromooba_apiimportOobaApiClient,OobaModelInfo,OobaModelNotLoadedmodel_info:OobaModelInfo=client.model_info()print(model_info)# model_info will be OobaModelNotLoaded if no model is currently loadedassertnotisinstance(model_info,OobaModelNotLoaded)To load a model:load_model_response:OobaModelInfo=client.load_model("codellama-7b-instruct.Q4_K_M.gguf",args_dict={"loader":"ctransformers","n-gpu-layers":100,"n_ctx":2500,"threads":0,"n_batch":512,"model_type":"llama",},)AppendixSpecific Model Help# Code Llama Instructfromooba_api.promptsimportLlamaInstructPromptresponse=client.instruct(LlamaInstructPrompt(system_prompt=("Generate only the requested code from the user. Do not generate anything else. ""Be succint. Generate markdown of the code, and give the correct type. ""If the code is python use ```python for the markdown. Do not explain afterwards"),prompt="Generate a Python function to reverse the contents of a file",),parameters=Parameters(temperature=0.2,repetition_penalty=1.05),)# falcon instructfromooba_api.promptsimportInstructPromptresponse=client.instruct(InstructPrompt(prompt=("Generate only the requested code from the user. Do not generate anything else. ""Be succint. Generate markdown of the code, and give the correct type. ""If the code is python use ```python for the markdown. Do not explain afterwards.\n""Generate a Python function to reverse the contents of a file")),parameters=Parameters(temperature=0.2,repetition_penalty=1.05),)Running Ooba-BoogaThe Text Generation Web UI can be found here:https://github.com/oobabooga/text-generation-webui/This start-up config gives very good performance on a RTX 3060-TI with 8 GB of VRAM on an i5 system with 32 GB of DDR4. In many cases this produced over 20 tokens / sec. There is no delay before producing tokens.python server.py --model codellama-7b-instruct.Q4_K_M.gguf --api --listen --gpu-memory 8 --cpu-memory 1 --n-gpu-layers 1000000 --loader ctransformer --n_ctx 2500Increasing the context size (2500) will slow it down.