Package: llm-vm
🤖 Anarchy LLM-VM 🤖An Open-Source AGI Server for Open-Source LLMsThis isAnarchy'seffort to build 🏗️ an open generalized artificial intelligence 🤖 through the LLM-VM: a way to give your LLMs superpowers 🦸 and superspeed 🚄.You can find detailed instructions to try it live here:anarchy.aiThis project is in BETA. Expect continuous improvement and development.Table of ContentsTable of ContentsAboutWhatWhyFeatures and RoadmapQuick Start and InstallationRequirementsInstallationGenerating CompletionsRunning LLMs LocallySupported ModelsPicking Different ModelsTool UsageContributing📚 About 📚💁 What is the Anarchy LLM-VM?The Anarchy LLM-VM is a highly optimized and opinionated backend for running LLMs with all the modern features we've come to expect from completion: tool usage, persistent stateful memory, live data augmentation, data and task fine-tuning, output templating, a web playground, API endpoints, student-teacher distillation, data synthesis, load-balancing and orchestration, large context-window mimicry.Formally, it is a virtual machine/interpreter for human language, coordinating between data, models (CPU), your prompts (code), and tools (IO).By doing all these things in one spot in an opinionated way, the LLM-VM can properly optimize batch calls that would be exorbitantly expensive with distributed endpoints.  It furthermore strives for both model and architecture agnosticism, properly optimizing the chosen model for the current architecture.🤌 Why use the Anarchy LLM-VM?In line with Anarchy's mission, the LLM-VM strives to support open-source models. By utilizing open-source models and running them locally you achieve several benefits:Speed up your AGI development 🚀:With AnarchyAI, one interface is all you need to interact with the latest LLMs available.Lower your costs 💸:Running models locally can reduce the pay-as-you-go costs of development and testing.Flexibility 🧘‍♀️:Anarchy allows you to rapidly switch between popular models so you can pinpoint the exact right tool for your project.Community Vibes 🫂:Join our active community of highly motivated developers and engineers working passionately to democratize AGIWYSIWYG 👀:Open source means nothing is hidden; we strive for transparency and efficiency so you can focus on building.🎁 Features and RoadmapImplicit Agents 🔧🕵️:The Anarchy LLM-VM can be set up to use external tools through our agents such asREBELjust by supplying tool descriptions!Inference Optimization 🚄:The Anarchy LLM-VM is optimized from the agent level all the way to assembly on known LLM architectures to get the most bang for your buck. With state-of-the-art batching, sparse inference and quantization, distillation, and multi-level colocation, we aim to provide the fastest framework available.Task Auto-Optimization 🚅:The Anarchy LLM-VM will analyze your use cases for repetitive tasks where it can activate student-teacher distillation to train a super-efficient small model from a larger more general model without losing accuracy.  It can furthermore take advantage of data-synthesis techniques to improve results.Library Callable 📚:We provide a library that can be used from any Python codebase directly.HTTP Endpoints 🕸️:We provide an HTTP standalone server to handle completion requests.Live Data Augmentation 📊:(ROADMAP)You will be able to provide a live updating data set and the Anarchy LLM-VM willfine-tuneyour models or work with avector DBto provide up-to-date information with citationsWeb Playground 🛝:(ROADMAP)You will be able to run the Anarchy LLM-VM and test its outputs from the browser.Load-Balancing and Orchestration ⚖️:(ROADMAP)If you have multiple LLMs or providers you'd like to utilize, you will be able to hand them to the Anarchy LLM-VM to automatically figure out which to work with and when to optimize your uptime or your costsOutput Templating 🤵:(ROADMAP)You can ensure that the LLM only outputs data in specific formats and fills in variables from a template with either regular expressions, LMQL, or OpenAI's template languagePersistent Stateful Memory 📝:(ROADMAP)The Anarchy LLM-VM can remember a user's conversation history and react accordingly🚀 Quickstart 🚀🥹 RequirementsInstallation RequirementsPython >=3.10 Supported. Older versions of Python are on a best-effort basis.Usebash > python3 --versionto check what version you are on.To upgrade your python, either create a new python env usingbash > conda create -n myenv python=3.10or go tohttps://www.python.org/downloads/to download the latest version.If you plan on running the setup steps below, a proper Python version will be installed for youSystem RequirementsDifferent models have different system requirements. Limiting factors on most systems will likely be RAM, but many functions will work at even 16 GB of RAM.That said, always lookup the information about the models you're using, they all have different sizes and requirements
in memory and compute resources.👨‍💻 InstallationThe quickest way to get started is to runpip install llm-vmin your Python environment.Another way to install the LLM-VM is to clone this repository and install it with pip like so:>gitclonehttps://github.com/anarchy-ai/LLM-VM.git
>cdLLM-VM
>./setup.shThe above bash scriptsetup.shonly works for MacOS and Linux.Alternatively you could do this:>gitclonehttps://github.com/anarchy-ai/LLM-VM.git
>cdLLM-VM
>python-mvenv<name>
>source<name>/bin/activate
>python-mpipinstall-e."[dev]"One Last Step, almost there!If you're using one of the OpenAI models, you will need to set theLLM_VM_OPENAI_API_KEYenvironment
variable with your API key.✅ Generating CompletionsOur LLM-VM gets you working directly with popular LLMs locally in just 3 lines. Once you've installed it (as above), just load your model and start generating!# import our clientfromllm_vm.clientimportClient# Select which LLM you want to use, here we have OpenAIclient=Client(big_model='chat_gpt')# Put in your prompt and go!response=client.complete(prompt='What is Anarchy?',context='',openai_key='ENTER_YOUR_API_KEY')print(response)# Anarchy is a political ideology that advocates for the absence of government...🏃‍♀ Running LLMs Locally# import our clientfromllm_vm.clientimportClient# Select the LlaMA 2 modelclient=Client(big_model='llama2')# Put in your prompt and go!response=client.complete(prompt='What is Anarchy?',context='')print(response)# Anarchy is a political philosophy that advocates no government...😎 Supported ModelsSelect from the following modelsSupported_Models=['chat_gpt','gpt','neo','llama2','bloom','opt','pythia']☯ Picking Different ModelsLLM-VM default model sizes for local models are intended to make experimentation
with LLMs accessible to everyone, but if you have the memory required, larger parameter models
will perform far better!for example, if you want to use a large and small neo model  for your teacher and student, and you
have enough RAM:# import our clientfromllm_vm.clientimportClient# Select the LlaMA modelclient=Client(big_model='neo',big_model_config={'model_uri':'EleutherAI/gpt-neox-20b'},small_model='neo',small_model_config={'model_uri':'EleutherAI/gpt-neox-125m'})# Put in your prompt and go!response=client.complete(prompt='What is Anarchy?',context='')print(response)# Anarchy is a political philosophy that advocates no government...Here are some default model's details:NameModel_UriModel paramsCheckpoint file sizeNeoEleutherAI/gpt-neo-1.3B1.3B5.31 GBBloombigscience/bloom-560m1.7B1.12 GBOPTfacebook/opt-350m350m622 MBFor some other choices of memory usage and parameter count in each model family, check out the
tablesmodel_uri_tables.🛠 Tool UsageThere are two agents: FLAT and REBEL.Run the agents separately by going into thesrc/llm_vm/agents/<AGENT_FOLDER>and running the file that is
titledagent.py.Alternatively, to run a simple interface and choose an agent to run from the CLI, run thesrc/llm_vm/agents/agent_interface.pyfile
and follow the command prompt instructions.🩷 Contributing 🩷We welcome contributors!  To get started is to join our activediscord community.  Otherwise here are some ways to contribute and get paid:JobsWe're always looking for serious hackers.  Prove that you can build and creatively solve hard problems and reach out!The easiest way to secure a job/internship with us is to submit pull requests that address or resolve open issues.Then, you can apply directly herehttps://forms.gle/bUWDKW3cwZ8n6qsU8BountyWe offer bounties for closing specific tickets! Look at the ticket labels to see how much the bounty is.  To get started,join the discord and read the guide🙏 Acknowledgements 🙏Matthew Mirman- CEOGitHub:@mmirmanLinkedIn:@matthewmirmanTwitter:@mmirmanWebsite:mirman.comVictor Odede- UndoomerGitHub:@VictorOdedeLinkedIn:@victor-odedeAbhigya Sodani- Research InternGitHub:@abhigya-sodaniLinkedIn:@abhigya-sodaniCarter Schonwald- Fearless ContributorGitHub:@cartazioLinkedIn:@carter-schonwaldKyle Wild- Fearless ContributorGitHub:@dorkitudeLinkedIn:@kylewildAarushi Banerjee- Fearless ContributorGitHub:@AB3000LinkedIn:@ab99Andrew Nelson- Fearless ContributorGitHub:@ajn2004LinkedIn:@ajnelsnycLicenseMIT License