Package: llmvm-cli
LLMVMLLMVM is a CLI based productivity tool that uses Large Language Models and local Python tools/helpers to reason about and execute your tasks. A CLI client (client.py) either connects directly to an LLM provider or will connect to a local server (server.py) that coordinates tool execution,Retrieval Agumented Generation, document search and more.It supportsOpenAIGPT 3.5/4.0/4 Turbo/Vision models from OpenAI, andClaude 2.1fromAnthropic.GeminiandMistralare currently experimental. It's best used with thekittyterminal as LLMVM will screenshot and render images and work directly with GPT 4.5 vision models.LLMVM's features are best explored through examples:Tool Use: Controlling Firefox Browser$ pip install llmvm-cli$ python -m llmvm.server$ python -m llmvm.clientquery>>Gotothehttps://ten13.vc/teamwebsiteandextractthelistofnamesThe LLMVM server is coordinating with the LLM to deconstruct the query into executable code calls various Python helpers that can be executed in the server process on behalf of the LLM. In this case, the server is using a headless Firefox instance to download the website url, screenshot and send progress back to the client, convert the website to Markdown, and hand the markdown to the LLM for name extraction. More on how this works later.Tool Use: Finance and Searchingquery>>Ihave5MSFTstocksand10NVDAstocks,whatismynetworthingramsofgold?...Here we're calling Yahoo Finance to get the latest prices of Microsoft and NVidia. We're also using Google Search functionality to find the latest price of gold.Tool Use: PDF Parsing and Understandingquery>>-pdocs/turnbull-speech.pdf"what Malcolm Turnbull advocating for?"LLMVM will parse and extract PDF's (including using OCR if the PDF doesn't extract text properly) and supply the LLM with the text as content for queries.As a Command Line UtilityI bash/fish/zsh alias llm:alias llm=LLMVM_EXECUTOR="openai" LLMVM_MODEL="gpt-4-vision-preview" LLMVM_PROFILING="true" python -m llmvm.clientor if you're using pyenv:functionllm(){localpyenv_ver=$(cat~/dev/llm/llmvm/.python-version)$PYENV_ROOT/versions/$pyenv_ver/bin/python-mllmvm.client"$@"}and then:catsomecode.py|llm-odirect"rewrite this code; make it cleaner and easier to read"Image understanding is supported via OpenAI's GPT 4.5 vision model:catdocs/beach.jpg|llm"describe this image for me"llm"generate cat names">cat_names.txtllm-pmeeting_notes.txt"correct spelling mistakes and extract action items"And some really nice Unix pipe foo:llm"download the latest news about Elon Musk as bullet points"|\llm"write a small blog post from the bullet points in the previous message"|\llm"create a nice html file to display the content">output.htmlAs a Client REPLIt integrates well withvimor your favorite editor to build multiline queries, or edit long message threads.You can even Ctrl-y + p to paste images into the Repl for upload and parsing by OpenAI's vision models.InstallYou'll need either an OpenAI API account (including access to the GPT 4.x API), anAnthropic API account, a GoogleGemini API accountor aMistral AI API account. It's highly recommended to sign up for a freeSerpAPIaccount to ensure that web searches (Google, News, Yelp and more) work. Asec-api.iois optional so LLMVM can download public company 10-K or 10-Q filings.Ensure you have the following environment variables set:OPENAPI_API_KEY# your Openai API key, or ...ANTHROPIC_API_KEY# your Anthropic API keyGOOGLE_API_KEY# your Gemini API keyMISTRAL_API_KEY# your Mistral API keyEDITOR# set this to your favorite terminal editor (vim or emacs or whatever) so you can /edit messages or /edit_ast the Starlark code before it gets executed etc.These are optional:SERPAPI_API_KEY# https://serpapi.com/ API key for web and news searches.SEC_API_KEY# if you want to use SEC's Edgar api to get 10K's and 10Q's etc, get an account at https://sec-api.io/Manual InstallationIf you don't want to dopip install llmvm-cliyou can do:Installpyenv:curl https://pyenv.run | bashInstall Python 3.11.6 using pyenv and set a virtualenv:pyenv install 3.11.7pyenv virtualenv 3.11.7 llmvmpyenv local llmvmInstallpoetry:curl -sSL https://install.python-poetry.org | python3 -poetry config virtualenvs.prefer-active-python trueInstall library dependencies:poetry installInstallPlaywrightFirefox automation:playwright install firefoxInstallPopplersudo apt install poppler-utilsbrew install popplerEdit and save config.yaml into the config directorycp llmvm/config.yaml ~/.config/llmvm/config.yamlRun the llmvm server and client:python -m llmvm.serverpython -m llmvm.client[Optional]Installviufor image rendering in macos/linux terminalscargo install viu(the kitty terminal renders images out of the box)Docker instructions:rundocker.sh -g(builds the image, deploys into a container and runs the container)python -m llmvm.server will automatically run on container port 8011. The host will open 8011 and forward to container port 8011.Use docker desktop to have a look at the running server logs; or you can ssh into the container, kill the server process, and restart from your own shell.With the docker container running, you can run client.py on your local machine:export LLMVM_ENDPOINT="http://localhost:8011"python -m llmvm.clientYou can ssh into the docker container: ssh llmvm@127.0.0.1 -p 2222Configuring Anthropic vs. OpenAIopen~/.config/llmvm/config.yamland change executor to 'anthropic' or 'openai':executor:'anthropic'# or 'openai'or, you can set environment variables that specify the execution backend and the model you'd like to use:exportLLMVM_EXECUTOR='openai'exportLLMVM_MODEL='gpt-4-vision-preview'python-mllmvm.client"hello, who are you?"Performance Profilingopen~/.config/llmvm/config.yamland change profiling to 'true' or 'false'.If the LLMVM server is running, profiling output will be emitted there, and if the server is not running, the LLMVM client will emit profiling information to the debug stream:exportLLMVM_PROFILING="true"...
DEBUGtotal_time:8.42prompt_time:0.55sample_time:7.87
DEBUGprompt_len:28sample_len:194DEBUGp_tok_sec50.52s_tok_sec:23.15ArchitectureYou can:Write arbitrary natural language queries that get translated into Starlark code and cooperatively executedUpload .pdf, .txt, .csv and .html and have them ingested by FAISS and searchable by the LLMVM.Add arbitrary Python helpers by modifying ~/.config/llmvm/config.yaml and adding your Python based helpers. Note: you may need to hook the helper instarlark_runtime.py. You may also need to show examples of its use inprompts/starlark/starlark_tool_execution.promptserver.pyvia /v1/chat/completions endpoint, mimics and forwards to OpenAI's /v1/chat/completions API.Useclient.pywithout running the server. Tools no longer work, but most other things do.TODO: build real time streaming, so you can wire up pipelines of llmvm execution to arbitrary streams of incoming data.TODO: search is weak. Make it better.TODO: all the threading and asyncio stuff is not great. Fix it. Might use RxPy.TODO: local llama can work, but doesn't work well.TODO: anthropic support [done]Advanced Architectural DetailsError CorrectionEach step of statement execution is carefully evaluated. Calls to user defined helper functions may throw exceptions, and code may be semantically incorrect (i.e. bindings may be incorrect, leading to the wrong data being returned etc). LLMVM has the ability to back-track up the statement execution list (todo: transactional rollback of variable assignment is probably the right call here but hasn't been implemented yet) and work with the LLM to re-write code, either partially or fully, to try and achieve the desired outcome.The code that performs error correction startshere, but there's still a bit more work to do here, including having the LLM engage in a "pdb" style debugging session, where locals in the Python runtime can be inspected for code-rewriting decisions.HelpersYou can define any arbitrary helper, and add it to the Starlark Runtime inStarlarkRuntime.setup(). It'll automatically generate the helper tool's one-shot prompt example for the LLM, and will appear in the LLM responses for Starlark generated code. The LLMVM runtime will sort out the binding and marshalling of arguments via llm_bind().Here are the list of helpers written so far:WebHelpers.search_linkedin_profile(first_name,last_name,company_name)# Searches for the LinkedIn profile of a given person name and optional company name and returns the profile textWebHelpers.get_linkedin_profile(linkedin_url)# Extracts the career information from a person's LinkedIn profile from a given LinkedIn urlEdgarHelpers.get_latest_form_text(symbol,form_type)# This is useful to get the latest financial information for a company,theircurrentstrategy,investmentsandrisks.PdfHelpers.parse_pdf(url_or_file)# You can only use either a url or a path to a pdf file.MarketHelpers.get_stock_price(symbol)# Get the current or latest price of the specified stock symbolMarketHelpers.get_market_capitalization(symbol)# Get the current market capitalization of the specified stock symbolDownloading web content (html, PDF's etc), and searching the web is done through special functions:download()andsearch()which are defined in the LLMVM runtimes base class libraries.download()as mentioned uses Firefox via Microsoft Playwright so that we can avoid web server blocking issues that tend to occur with requests.get().search()usesSerpAPI, which may require a paid subscription.Walkthrough of tool binding and executionFor our first example:query>>gotohttps://ten13.vc/teamandextractallthenamesLet's walk through each line of the generated Starlark:var1 = download("https://ten13.vc/team")Thedownload()function is part of a set of user definable base class libraries that the LLM knows about: download() llm_call() llm_loop_bind(), llm_bind(), answer() and so on. download() fires up an instance of Firefox viaPlaywrightto download web or PDF content and convert them to Markdown.var2 = llm_call([var1], "extract list of names")  # Step 2: Extract the list of namesllm_call(expression_list, instruction) -> strtakes an expression list, packages those expressions up into a stack of LLM User messages, and passes them back to the LLM to perform the instruction. If the stack of Messages is too big to fit in the context window,faissis used to chunk and rank message content via the following pseudocode:Chunk content via tiktoken, 256 token chunks.Take the original query "Go to thehttps://ten13.vc/teamwebsite ..." and the instruction query "extract list of names" and vector search and rank on message content.Take a random sample of chunks, ask the LLM to decide ifallcontent is required to achieve success in the task.If "YES", Map-Reduce the task over all Message chunks.If "NO", fit top (n) chunks sized to context window and perform instruction.Return instruction result.The map-reduce is done per-message, allowing for multiple expressions to be chunked and ranked independently, which is useful for queries like "download document 1, and document 2 and compare and contrast".for list_item in llm_loop_bind(var2, "list of names"):  # Step 3: Loop over the list of namesllm_loop_bind(expression, instruction) -> Listtakes an arbitrary expression, converts it to a string, then has an LLM translate that string into a Starlark list ["one", "two", "three", ...].In this particular case,var2has the following string, the response from GPT:Based on the provided data, here is a list of names:Steve BaxterStew GlynnAn VoAlexander CohenMargot McQueenSophie RobertsonMel HarrisonElise CannellSeamus CrawfordAlexander BarratAndllm_loop_bind()takes this arbitrary text and converts it to: ["Steve Baxter", "Stew Glynn", "An Vo", "Alexander Cohen", "Margot McQueen", "Sophie Robertson", "Mel Harrison", "Elise Cannell", "Seamus Crawford", "Alexander Barrat"]var3 = llm_bind(list_item, "WebHelpers.search_linkedin_profile(first_name, last_name, company_name)")llm_bind(expression, function_definition_str) -> Callableis one of the more interesting functions. It takes an expression and a string based function definition and tries to bind arbitrary data to the function arguments (turning the definition into a callsite). It performs these steps:For these scopes [expression, original query, dictionary of currently in scope locals()] ask the LLM to bind the arguments (in this case: first_name, last_name, company_name).If the LLM can't bind any of the arguments, specify "None", then add a comment string "#" to the end of the callsite with a natural language question that, if answered, would allow the callsite to be bound. e.g:WebHelpers.search_linkedin_profile("Steve", "Baxter", None)  # Which company does Steve Baxter work for?Move up a scope, and include the original binding ask, plus the LLM generated question, which combined greatly improves the odds of argument binding.(in this particular case, the company Steve Baxter works for is defined in the original webpage download() string, and when the LLM is passed the locals() dictionary, is able to self-answer the question of "which company Steve Baxter works for" and thus bind the callsite properly).-> WebHelpers.search_linkedin_profile("Steve", "Baxter", "Transition Level Investments")answer(answers)  # Step 7: Show the summaries of the LinkedIn profiles to the useranswer()is a collection of possible answers that either partially solve, or fully solve for the original query. Once code is finished executing, each answer found in answers() is handed to the LLM for guidance on how effective it is at solving/answering the query. The result is then shown to the user, and in this case, it's a career summary of each of the individuals fromTEN13extracted from LinkedIn.Debugging Firefox Automation IssuesThe Starlark runtime usesPlaywrightto automate Firefox on its behalf. By default, it runs Firefox in headless mode, but this can be changed in~/.config/llmvm/config.yaml:firefox_headless:trueYou can also copy your own browsers cookies file into Playwright's Firefox automation instance. This allows the Playwright instance to assume your logged in sessions to things like LinkedIn. Simply:runscripts/extract_cookies.sh > cookies.txtmove the cookies.txt file to a secure locationupdateconfig.yamlto point to the cookies file:firefox_cookies:'~/.local/share/llmvm/cookies.txt'The Problem this prototype solvesChatGPT supports 'function calling' by passing a query (e.g. "What's the weather in Boston") and a JSON blob with the signatures of supporting functions available to be called locally (i.e. def get_weather(location: str)...). Examples seenhere.However, this interaction is usually User Task -> LLM decides what helper function to call -> local host calls helper function -> work with result, and does not allow for arbitrary deconstruction of a task into a series of helper function calls that can be intermixed with both control flow, or cooperative sub-task execution.This prototype shows that LLM's are capable of taking a user task, reasoning about how to deconstruct the task into sub-tasks, understanding how to program, schedule and execute those sub-tasks on its own or via a virtual machine, and working with the VM to resolve error cases. We ask the LLM to useStarlarkexpressed asA-normal formas the programming language, and execute Starlark statement-by-statement on a local Python interpreter. When errors arise (syntax errors, exceptions, or semantic problems), we pause execution and work with the LLM to understand and resolve the error by exposing the locals dictionary, and allowing the LLM to "debug" the current execution state.Other cute stuffYou can use theactcommand, which will searchawesome promptsand set the System Prompt to "act" like the awesome prompt you select.A fun one is graphing "narrative extraction", which is useful for quickly summarizing news articles:Download two news articles and put them in the "Messages" stack:urlhttps://www.cnn.com/2023/08/27/football/what-happened-luis-rubiales-kiss-intl/index.html
urlhttps://www.bbc.com/sport/football/66645618actgraph"extract the narrative from the documents I sent you in the first two messages"Gives us a GraphVis visualization (cut off to fit screen):And related narrative extraction + code:Things to doError handling still needs a lot of work.Working on Source Code insights (mode 'code'). You'll be able to hand it a project directory and work with the LLM to understand what the code is doing. Check outsource.pyChatGPT and Claude doesn't conform to system prompts specifying how to shape the output of the response. Saying things like "Only reply with Starlark code" and "don't apologize" tends to work randomly, or not at all. Need to fix this.Integration with local LLM's viallama.cppetc. [started]More complicated natural language queries tends to make the LLM generate unwieldy code. Need statement by statement execution, passing the LLM the already executed code, asking it if it wishes to re-write the later part of the code (sort of like continuation passing style execution).Playwright integration with LLM -- it should be straight forward to have cooperative execution for the task of proceeding through web app flows (login, do stuff, extract info, logout).Fix bugs and refactor. The code is pretty hacky.Write some docs.