Package: llmx
LLMX - An API for Chat Fine-Tuned Language ModelsA simple python package that provides a unified interface to several LLM providers of chat fine-tuned models [OpenAI, AzureOpenAI, PaLM, Cohere and local HuggingFace Models].Notellmx wraps multiple api providers and its interfacemaychange as the providers as well as the general field of LLMs evolve.There is nothing particularly special about this library, but some of the requirements I needed when I started building this (that other libraries did not have):Unified Model Interface: Single interface to create LLM text generators with support formultiple LLM providers.fromllmximportllmgen=llm(provider="openai")# support azureopenai models too.gen=llm(provider="palm")# or googlegen=llm(provider="cohere")# or palmgen=llm(provider="hf",model="HuggingFaceH4/zephyr-7b-beta",device_map="auto")# run huggingface model locallyUnified Messaging Interface. Standardizes on the OpenAI ChatML message format and is designed forchat finetunedmodels. For example, the standard prompt sent a model is formatted as an array of objects, where each object has a role (system,user, orassistant) and content (see below). A single request is list of only one message (e.g., write code to plot a cosine wave signal). A conversation is a list of messages e.g. write code for x, update the axis to y, etc. Same format for all models.messages=[{"role":"user","content":"You are a helpful assistant that can explain concepts clearly to a 6 year old child."},{"role":"user","content":"What is  gravity?"}]Good Utils (e.g., Caching etc): E.g. being able to use caching for faster responses. General policy is that cache is used if config (including messages) is the same. If you want to force a new response, setuse_cache=Falsein thegeneratecall.response=gen.generate(messages=messages,config=TextGeneratorConfig(n=1,use_cache=True))Output looks likeTextGenerationResponse(text=[Message(role='assistant',content="Gravity is like a magical force that pulls things towards each other. It's what keeps us on the ground and stops us from floating away into space. ... ")],config=TextGenerationConfig(n=1,temperature=0.1,max_tokens=8147,top_p=1.0,top_k=50,frequency_penalty=0.0,presence_penalty=0.0,provider='openai',model='gpt-4',stop=None),logprobs=[],usage={'prompt_tokens':34,'completion_tokens':69,'total_tokens':103})Are there other libraries that do things like this really well? Yes! I'd recommend looking atguidancewhich does a lot more. Interested in optimized inference? Try somthing likevllm.InstallationInstall from pypi. Please usepython3.10or higher.pipinstallllmxInstall in development modegitclonecdllmx
pipinstall-e.Note that you may want to use the latest version of pip to install this package.python3 -m pip install --upgrade pipUsageSet your api keys first for each service.# for openai and cohereexportOPENAI_API_KEY=<yourkey>exportCOHERE_API_KEY=<yourkey># for PALM via MakerSuiteexportPALM_API_KEY=<yourkey># for PaLM (Vertex AI), setup a gcp project, and get a service account key fileexportPALM_SERVICE_ACCOUNT_KEY_FILE=<pathtoyourserviceaccountkeyfile>exportPALM_PROJECT_ID=<yourgcpprojectid>exportPALM_PROJECT_LOCATION=<yourprojectlocation>You can also set the default provider and list of supported providers via a config file. Use the yaml format in thissampleconfig.default.ymlfileand set theLLMX_CONFIG_PATHto the path of the config file.fromllmximportllmfromllmx.datamodelimportTextGenerationConfigmessages=[{"role":"system","content":"You are a helpful assistant that can explain concepts clearly to a 6 year old child."},{"role":"user","content":"What is  gravity?"}]openai_gen=llm(provider="openai")openai_config=TextGenerationConfig(model="gpt-4",max_tokens=50)openai_response=openai_gen.generate(messages,config=openai_config,use_cache=True)print(openai_response.text[0].content)See thetutorialfor more examples.A Note on Using Local HuggingFace ModelsWhile llmx can use the huggingface transformers library to run inference with local models, you might get more mileage from using a well-optimized server endpoint likevllm, or FastChat. The general idea is that these tools let you provide an openai-compatible endpoint but also implement optimizations such as dynamic batching, quantization etc to improve throughput. The general steps are:install vllm, setup endpoint e.g., on port8000use openai as your provider to access that endpoint.fromllmximportllmhfgen_gen=llm(provider="openai",api_base="http://localhost:8000",api_key="EMPTY,)...Current WorkSupported modelsOpenAIPaLM (MakerSuite,Vertex AI)CohereHuggingFace (local)CaveatsPrompting. llmx makes some assumptions around how prompts are constructed e.g., how the chat message interface is assembled into a prompt for each model type. If your application or use case requires more control over the prompt, you may want to use a different library (ideally query the LLM models directly).Inference Optimization. For hosted models (GPT-4, PalM, Cohere) etc, this library provides an excellent unified interface as the hosted api already takes care of inference optimizations. However, if you are looking for a library that is optimized for inference withlocal models(e.g., huggingface)(tensor parrelization, distributed inference etc), I'd recommend looking atvllmortgi.CitationIf you use this library in your work, please cite:@software{victordibiallmx,author={Victor Dibia},license={MIT},month={10},title={LLMX - An API for Chat Fine-Tuned Language Models},url={https://github.com/victordibia/llmx},year={2023}}