Package: uniteai
Your AI Stack in your Editor: Voice-to-text, Local LLM, and GPT, +more.Requirements:Python 3Editor:VSCode(ium) or Emacs or Any Editor with LSP capabilities (most).The MissionThe future is about Humans Augmented with AIs.We need ourAI Stack(Online, or local models)Inside aconvenient interface(Text Editors > Web UIs)Friendly withany editor(The project is anLSPand therefore highly portable)Andclose to the code(It's easy to tweak and add features. All the logic happens in friendlypythoncode, not bespoke one-off editor code).Screencast DemoSome Core Featuresscreencast.webmDocument Chat (NEW)akaRetrieval Augmented Generationscreencast_document_chat.webmQuickstart, installing EverythingYou can install more granularly thaneverything, but we'll demoeverythingfirst.1.) Make sure Python 3 + Pip is installed.python--version
pip--version# orpython3--version
pip3--version2.) The only platform-dependent dependency right now isportaudio, and that is only needed if you want speech-to-text/transcription.# Macbrewinstallportaudio# Ubuntu/Debiansudoaptinstallportaudio19-dev3.) InstallUniteAI.pip3install--user"uniteai[all]"# install deps for all featuresYour editor will make use of the installed binary, it so it needs to be on yourPATH. Add your python installationsbin/to your systemPATHenvironment variable.# Linux and Mac~/.local/bin# WindowsC:\Users\USERNAME\AppData\Roaming\Python\PythonXX\Scripts\uniteai_lspshould now be callable from a fresh terminal (but you will never need to call it yourself).4.)Optional:Start the longlived LLM server which offers your editor a connection to your local large language model.The editor will not start up the LLM itself, so you can just run this from anywhere, and the editor will connect to it.uniteai_llm5.) Install in your editor:ForVSCodeget theuniteaiextension. Eg in VSCode,Ctrl-Pthenext install uniteai.uniteai.ForVSCodium, VSCode Marketplace files are not compatible, so you'll need to either:Download the prepackageduniteai.vsixextension, then:codium--install-extensionclients/vscode/uniteai.vsixDIY:npminstall-g@vscode/vsce
gitclonehttps://github.com/freckletonj/uniteaicduniteai/clients/vscode
vscepackage
codium--install-extensionuniteai-version.vsixForEmacs, copy thelsp-modeconfigto yourinit.el.For other editors with LSP support (most do), we just need to copy theemacs/vscode configuration, and translate it to your editor. Please submit a PR with new editor configs!6.)Config:When you first open a compatible file in your editor, the LSP will start.It will prompt you to create a config file in your local dir, or your home dir.You will need to edit this file, and then restart your editor.Granular installsIf you didpip install "uniteai[all]", ignore this section!Your config determines what modules/features are loaded.The following makes sure to get your dependencies for each feature. This will become more relevant when more community features are added.Transcription dependencies# Debian/Ubuntusudoaptinstallportaudio19-dev# needed by PyAudio# Macbrewinstallportaudio# needed by PyAudiopip3install"uniteai[transcription]"Local LLM dependenciespip3install"uniteai[local_llm]"OpenAI/ChatGPT dependenciespip3install"uniteai[openai]"KeycombosYour client configuration determines this, so if you are using the example client config examples in./clients:VSCodeEmacsEffectM-'Show Code Actions MenuCtrl-Alt-dC-c l dDo semantic search on a documentCtrl-Alt-gC-c l gSend region toGPT, stream output to text bufferCtrl-Alt-cC-c l cSame, butChatGPTCtrl-Alt-lC-c l lSame, butLocal (eg Falcon) modelCtrl-Alt-vC-c l vStartvoice-to-textCtrl-Alt-sC-c l sWhatevers streaming, stop itI'm still figuring out what's most ergonomic, so, I'm accepting feedback.Ctrl-Alt-don ubuntu means defaults to "minimize all windows". You candisablethat.Cmd-Alt-don Mac does something else too, so, we should revisit default bindings.Retreival Augmented Generation (RAG)For thedocumentfeature, you can reference one of multiple document types, and lookup passages with a similar "gist" to them (semantic similarity search).Check that your.uniteai.yamlconfig hasuniteai.documentenabled.You can use links to: YouTube (will read transcripts), Arxiv papers, PDFs, Git repos, or any HTML.To use this feature, write some YAML, highlight it, and hitC-c l d(emacs) orC-A-d(vscode).query:docs:-title:(optional)url:...-title:...url:...It will take a couple minutes for long documents to get an embedding for each chunk it finds in the document, but that then gets cached and goes fast afterward.More details.ContributionsWhy?Because there areso many cool toolsto yet be added:Image creation, eg:"Write a bulleted plan for a Hero's Journey story about X, and make an image for each scene."Contextualize the AI via reading my emails via POP3, and possibly responding, eg:"what was that thing my accountant told me not to forget?"Ask my database natural language questions, eg:"what were my top 10% customers' top 3 favorite products?"Write-ahead for tab-completion, eg:"Once upon a ____".Chat with a PDF document, eg:"what do the authors mean by X?"Do some searches, scrape the web, and upload it all into my db.Sky's the limit.How?A Key goal of this project is to beContributor-Friendly.Make anIssuewith your cool concept, or bug you found..todo/is a directory of community "tickets", eg.todo/042_my_cool_feature.md. Make a ticket or take a ticket, and make a PR with your changes!./todo/README.mdgives some overview of the library, and advice on building against this library.a./contribdirectory is where you can add your custom feature. See./uniteai/contrib/example.py..uniteai.ymlconfiguration chooses which modules to load/not load.The code iswell-documented,robust, andsimple, to reduce friction.Adding a feature is as simple as writing some python code, and making use ofuniteai's library to directly handle issues like concurrency and communicating/modifying the text editor.MiscNotes on Local LLMsThe file./llm_server.pylaunches a TCP server in which the LLM weights are booted up. Thelsp_serverwill make calls to thisllm_server.The reason is that thelsp_serverlifecycle is (generally*) managed by the text editor, and LLM models can be really slow to boot up. Especially if you're developing a feature, you do not want the LLM to keep being read into your GPU each time you restart thelsp_server.*you don't have to let the editor manage thelsp_server. For instance,eglotin emacs allows you to launch it yourself, and then the editor client can just bind to the port.Falcon LLM Issue:If Falcon runs on multiple threads, its cache has an issue. You need a separatemodelling_RW.pythat makes sure it never tries to cache.https://github.com/h2oai/h2ogpt/pull/297Replacingcos_simwith this seems to do the trick:defcos_sin(self,seq_len:int,device="cuda",dtype=torch.bfloat16,)->torch.Tensor:t=torch.arange(seq_len,device=device).type_as(self.inv_freq)freqs=torch.einsum("i,j->ij",t,self.inv_freq)emb=torch.cat((freqs,freqs),dim=-1).to(device)ifdtypein[torch.float16,torch.bfloat16]:emb=emb.float()cos_cached=emb.cos()[None,:,:]sin_cached=emb.sin()[None,:,:]cos_cached=cos_cached.type(dtype)sin_cached=sin_cached.type(dtype)returncos_cached,sin_cachedA separate bitsandbytes issue remains unresolved, but is less serious than the above.https://github.com/h2oai/h2ogpt/issues/104https://github.com/TimDettmers/bitsandbytes/issues/162LicenseCopyright (c) Josh Freckleton. All rights reserved.Licensed under theApache-2.0license.