Package: gai-lib-gen
Gai/Gen: Universal LLM WrapperUniversal Multi-Modal Wrapper Library for LLM inferencing. The library provides a simplified and unified interface for seamless switching between multi-modal open source language models on a local machine and OpenAI APIs. This is intended for Developers who are targetting the use of multi-modal LLMs for both OpenAI API and local machine models.Table of ContentsGai/Gen: Universal LLM Wrapper1. Introduction2. Requirements3. Credits4. Using Gai as a LibraryConfigurationAPI KeyQuick StartExamples5. Using Gai as a ServiceEndpointsExamplesText-to-Text GenerationSpeech-to-Text GenerationText-to-Speech GenerationImage-to-Text GenerationRetrieval Augmented Generation1. IntroductionThe core object is calledGaigen- generative AI generator. The premise is for the code to run on as commonly available commodity hardware as possible. The main focus is on 7 billion parameters and below open source models. Gaigen is designed as singleton wrapper where only one model is loaded and cached into memory at any one time.To avoid dependency conflicts, the wrappers are organised under thegenfolder according to 5 mutually-exclusive categories:ttt: Text-to-Texttts: Text-to-Speechstt: Speech-to-Textitt: Image-to-Textrag: Retrieval-Augmented Generation2. RequirementsThe instructions are tested mainly on:Windows 11 (22H2 Build 22621.3007) with WSL2 (5.12.133.1-microsoft-standard-WSL2)Ubuntu 20.04.2 LTSNVIDIA RTX 2060 GPU with 8GB VRAM. Runnvidia-smito check if CUDA driver is installed.CUDA Toolkit 11.8 is required for the GPU accelerated models. Runnvcc --versionto check if CUDA Toolkit is installed. Refer herehttps://gist.github.com/kakkoii1337/8a8d4d0bc71fa9c099a683d1601f219eif you need guidance.3. CreditsThis library is made possible by the generosity and hardwork of the following open source projects. You are highly encouraged to check out the original source and documentations.TTTTheBlokefor all the quantized models in the demoturboderpfor ExLlamaMeta Team for theLLaMa2ModelHuggingFace team for theTransformerslibrary and open source modelsMistral AI Team forMistral7BModelGeorgi Gerganov forLLaMaCppITTLiu HaoTian for theLLaVaModel and LibraryTTSCoqui-AIfor the xTTS ModelSTTOpenAIfor Open Sourcing Whisper v3RAGchromadbfor AI-native open-source embedding databaseinstructoropen source embedding model4. DisclaimerMaintainers of this repo are not responsible for the actions of third parties who use the models. Please consult an attorney before using models for commercial purposes.5. LicenseThis project is licensed under theMITLicense - see the LICENSE file for details.6. Using Gai as a LibraryUsing Gai as a library requires you to install the right category of package but gives you more control over your interaction with Gaigen.
It is highly recommended that you install each category in separate virtual environments.# Install library for text-to-text generationpipinstall"gai-lib-gen[TTT]"# Install library for text-to-speech generationpipinstall"gai-lib-gen[TTS]"# Install library for speech-to-text generationpipinstall"gai-lib-gen[STT]"# Install library for image-to-text generation (install LLaVA separately)pipinstall"gai-lib-gen[ITT]"gitclonehttps://github.com/haotian-liu/LLaVA&&cdLLaVA&&pipinstall-e.# Install library for retrieval-augmented generationpipinstall"gai-lib[RAG]"6.1 ConfigurationStep 1.Create a.gaircfile in your home directory. This file contains the default configurations for Gai.{"app_dir":"~/gai"}Step 2.Create a/gaidirectory.mkdir~/gaiCopygai.jsonfrom this repository into~/gai. This file contains the configurations for models and their respective loaders.Step 3.Create/gai/modelsdirectory.mkdir~/gai/modelsThe final user directory structure looks like this:home
├──gai
│├──gai.json
│└──models
└──.gairc6.2 Downloading ModelsWhen downloading from huggingface model hub, it is recommended to use thehuggingface CLI.
You will need to install the CLI first.pipinstallhuggingface-hubTo download a model, run the following command:huggingface-clidownload<repo-name>/<model-name>--local-dir~/gai/models/<model-name>--local-dir-use-symlinksFalseExample:Downloading the main branchhuggingface-clidownloadTheBloke/Mistral-7B-Instruct-v0.1-GPTQ\--local-dir~/gai/models/Mistral-7B-Instruct-v0.1-GPTQ\--local-dir-use-symlinksFalseExample:Downloading 2 fileshuggingface-clidownloadTheBloke/Mistral-7B-Instruct-v0.1-GGUF\mistral-7b-instruct-v0.1.Q4_K_M.gguf\config.json\--local-dir~/gai/models/Mistral-7B-Instruct-v0.1-GGUF\--local-dir-use-symlinksFalse6.3 API KeyAll API keys should be stored in a.envfile in the root directory of the project.For example,OPENAI_API_KEY=<--replace-with-your-api-key-->
ANTHROPIC_API_KEY=<--replace-with-your-api-key-->6.4 Quick StartStep 1. Install virtal environment and GaiThe following example shows how to install the TTT category but the same steps are applicable to the other categories as well.condacreate-nTTTpython=3.10.10-y
condaactivateTTT
pipinstallgai-lib-gen[TTT]Step 2. Setup OpenAI API Key.Save your OpenAI API key in the.envfile in the root directory of your project.OPENAI_API_KEY=<--replace-with-your-api-key-->Step 3. Run Inferencing on GPT4.Run Text-to-Text generation using OpenAI by loadinggpt-4wrapper.fromgai.genimportGaigengen=Gaigen.GetInstance().load('gpt-4')response=gen.create(messages=[{'role':'USER','content':'Tell me a one paragraph short story.'},{'role':'ASSISTANT','content':''}])print(response)Step 4. Install Mistral7B.Download the modelMistral-7B-Instruct-v0.1-GPTQinto the~/gai/modelsfolder.huggingface-clidownloadTheBloke/Mistral-7B-Instruct-v0.1-GPTQ\--local-dir~/gai/models/Mistral-7B-Instruct-v0.1-GPTQ\--local-dir-use-symlinksFalseStep 5. Run Inferencing on MistralRun Text-to-Text generation using Mistral7B by replacinggpt-4withmistral7b-exllama.fromgai.genimportGaigengen=Gaigen.GetInstance().load('mistral7b-exllama')response=gen.create(messages=[{'role':'USER','content':'Tell me a one paragraph short story.'},{'role':'ASSISTANT','content':''}])print(response)7. Using Gai as a ServiceGai Service is meant to be a one-model-per-instance service. Unlike library, you cannot change the model during runtime.The easiest to run Gai Service is to use a Docker container. You will need to download the models into ~/gai/models and map the volume to the container. You can then start up a container and post REST API calls to following endpoints.EndpointsThe following endpoints are only available for the category of models that you have installed.- Text-to-Text (TTT)Endpoint:http://localhost:12031/gen/v1/chat/completionsMethod: POSTType: BodyNameTypeDescriptionDefaultmodelstrgenerator namemistral7b-exllamamessageslistSee belowstreamboolTrue, FalseTrue...Hyperparameters based on modelNote:messages[{"role":"system","content":systemmessage},{"role":"user","content":usermessage},{"role":"assistant","content":AImessage},...]- Text-to-Speech (TTS)Endpoint:http://localhost:12031/gen/v1/audio/speechMethod: POSTType: BodyNameTypeDescriptionDefaultmodelstrgenerator namextts-2inputstrtext to be spokenvoicestrvoice id (speaker)languagefilelanguage codeenstreamboolTrue, FalseTrue...Hyperparameters based on model- Speech-to-TextEndpoint:http://localhost:12031/gen/v1/audio/transcriptionsMethod: POSTType: Multipart Form-DataNameTypeDescriptionDefaultmodelstrgenerator namefilefileaudio file object- itt: Image-to-TextEndpoint:http://localhost:12031/gen/v1/vision/completionsMethod: POST
Type: Body
Parameters:NameTypeDescriptionDefaultmodelstrgenerator namemessageslistsee belowstreamboolTrue,False...Hyperparameters based on modelNote:messages format[{"role":"user","content":[{"type":"text","text":text},{"type":"image_url","image_url":{"url":'data:image/jpeg;base64,.....',},},],...}]- rag: Retrieval-Augmented Generationa) Endpoint:http://localhost:12031/gen/v1/rag/index_fileMethod: POSTType: Multipart Form-Data
Parameters:NameTypeDescriptionDefaultcollection_namestrcollection name in the storefilefilethe document to be indexedmetadatadictmetadata tied to the documentb) Endpoint:http://localhost:12031/gen/v1/rag/retrieveMethod: POSTType: Body
Parameters:NameTypeDescriptionDefaultcollection_namestrcollection name in the storequery_textsstrqueryn_resultsintno. of nearest result returned8. ExamplesText-to-Text Generation (OpenAI GPT4 vs Open-Source Mistra7B)Speech-to-Text Generation (OpenAI Whisper vs Open-Source Whisper)Text-to-Speech Generation (OpenAI Speech vs Open-Source xTTS)Image-to-Text Generation (OpenAI Vision vs Open-Source Llava)Retrieval Augmented Generation