Package: llmflex
LLMFlexLLMFlex is a python package that allows python developers to work with different large language models (LLM) and do prompt engineering with a simple interface. It favours free and local resources instead of using paid APIs to develop truly local and private AI-powered solutions.It provides classes to load LLM models, embedding models, and vector databases to create LLM powered applications with your own prompt engineering and RAG techniques. With a one-liner command, you can load a chatbot interface to chat with the LLM or serve a model as OpenAI API as well.Installing LLMFlexCreating a virtual environment before installing the package is highly recommended. Also make sure you have installedPytorchandllama-cpp-pythonwith the correct installation method according to your hardware configuration before installing LLMFlex. Please visit the links provided for the respective packages for more detailed installation guides.After you have done the above steps, you can easily install llmflex withpip.pip install llmflexYou can also installExLlamaV2,AutoAWQ, andAutoGPTQif you have CUDA devices. Please visit the links provided for the respective packages for more detailed installation guides.Features1. Multilple LLMs with different genration configurations from one modelUnlike Langchain, you can create multiple LLMs with different temperature, max new tokens, stop words etc. with the same underlying model without loading the model several times using theLlmFactoryclass. This can be useful when you create your own agent with different LLM tasks which requires different configurations.2. Langchain compatibility with enhanced performancesAll the LLMs created withLlmFactoryare langchain compatible, and can be seamlessly integrated in your existing Langchain code. All the LLM classes are re-implementations of some langchain LLM classes which support more efficient streaming and stop words management, all with a unified interface.3. Multiple model formats supportMultiple model formats are all supported, and the loading process are all handled in theLlmFactoryclass, so it is just plug and play.
Supported formats:PyTorch(using transformers)GGUF(using llama-cpp-python)AWQ(using transformers)GPTQ(using transformers)OpenAI API(Work with any local servers that serve models with OpenAI API format)EXL2(using exllamav2)4. Embedding ToolkitsBundled classes for using embedding models which contains the embedding model and a tokens-count-based text splitter using the embedding model.5. Vector databaseUtilising Embedding toolkits and FAISS, aVectorDatabaseclass can allow you to store and search texts for your RAG tasks.6. Chat memoriesChat memory classes for storing chat memory on disk.BaseChatMemoryMemory class without using any embedding models or vector databases.LongShortTermChatMemoryMemory class using an underlyingVectorDatabaseto maintain long term memory along with the most recent memory.7. Prompt templateAPromptTemplateclass is implemented to format your prompt with different prompt formats for models from different sources. Some presets likeLlama2,ChatML,Vicuna, and more are already implemented, but you can alway add your own prompt format template.8. Custom toolsA base classBaseToolfor creating llm powered tools. AWebSearchToolpowered byDuckDuckGois implemented as an example.9. Chatbot frontend interfaceIf you simply want to play with a model, there's a gradio frontend chatbot that allows you to chat with a model with different generation configurations. You can switch between chat histories and prompt format, and you can set your system prompt and other model text generation sampling configurations in the gradio webapp.Using LLMFlex1. Create LLMsThis is how you can start with any text generation model on HuggingFace with your machine.fromllmfleximportLlmFactory# Load the model from Huggingfacemodel=LlmFactory("TheBloke/OpenHermes-2.5-Mistral-7B-GGUF")# Create a llmllm=model(temperature=0.7,max_new_tokens=512,stop=['```'])# Use the LLM for your taskprompt="""Instruction:Write a python script to load a Pandas Dataframe from the csv file 'test.csv'Solution:```python"""script=llm.invoke(prompt)print(script)# Or if you prefer to generate the output with token streamming.fortokeninllm.stream(prompt):print(token,end="")2. Load embeddings toolkit and create vector databaseTo load an embedding model and use a vector database:fromllmfleximportHuggingfaceEmbeddingsToolkit,VectorDatabase# Loading the embedding model toolkitembeddings=HuggingfaceEmbeddingsToolkit(model_id="thenlper/gte-large")# Create a vector databasefood=["Apple","Banana","Pork"]vectordb=VectorDatabase.from_data(index=food,embeddings=embeddings)# Do semantic search on the vector databaseprint(vectordb.search("Beef"))3. Use memory and prompt template to create a chatbotUse the Memory classes and thePromptTemplateclass to create a chatbot.fromllmflex.MemoryimportLongShortTermChatMemory,create_long_short_promptfromllmflex.PromptsimportPromptTemplate# Create the memory with the embeddings toolkit created earliermemory=LongShortTermChatMemory(title="My first conversation with OpenHermes",embeddings=embeddings,from_exist=False)# Get a prompt templateprompt_template=PromptTemplate.from_preset("ChatML")# Create an llm from the model factorychat_llm=model(max_new_tokens=512,stop=prompt_template.stop+["###"])# Create the prompt for the llmuser_input="What is the meaning of life?"prompt=create_long_short_prompt(user_input=user_input,prompt_template=prompt_template,llm=chat_llm,memory=memory)# Generating the response from the modeloutput=""fortokeninchat_llm.stream(prompt):output+=tokenprint(token,end="")# Save the interaction between you and the chatbotmemory.save_interaction(user_input=user_input,assistant_output=output)4. Use toolsAWebSearchToolclass is implemented as an example to build a tool with LLMFlex. The tool is usingDuckDuckGoby default. Here is how you can use it:fromllmflex.ToolsimportWebSearchTool# Create a web search tool with the embeddings toolkit created earliertool=WebSearchTool(embeddings=embeddings,verbose=True)# Run the tool with the previsous chat memory for a more coherent responsetool_input="Can you give me some views of the meaning of life from some famous philosophers?"output_generator=tool.run(tool_input,llm=chat_llm,history=memory.get_recent_memory(last_k=2),prompt_template=prompt_template,return_type="response",stream=True)tool_output=""fortokeninoutput_generator:tool_output+=tokenprint(token,end="")# And save the response to the memorymemory=memory.save_interaction(user_input=tool_input,assistant_output=tool_output)5. Chat with the model in a Streamlit web appIf you just want a GUI to start chatting with your LLM model with both long term and short term memory, type this command in the terminal:llmflexinterface--model_idTheBloke/OpenHermes-2.5-Mistral-7B-GGUF--embeddingsthenlper/gte-smallIf you want to chat with the web search tool:# Use a model with longer context for web searchllmflexinterface--model_idTheBloke/OpenHermes-2.5-Mistral-7B-16k-GGUF--embeddingsthenlper/gte-small--web_search--extras"model_file='openhermes-2.5-mistral-7b-16k.Q6_K.gguf', context_length=16384"You will see a streamlit frontend, use it to chat with the LLM model.6. Serve an OpenAI API with a GGUF modelTo serve a GGUF model with OpenAI API:llmflexserve--model_idTheBloke/OpenHermes-2.5-Mistral-7B-GGUF--model_fileopenhermes-2.5-mistral-7b.Q6_K.gguf--context_size4096DocumentationsPython documentation for all the classes, methods, and functions is provided in the./docsdirectory in this repository.LicenseThis project is licensed under the terms of the MIT license.