Saravanan666/pypi-org huggingface dataset:
Package: llm-openrouter
llm-openrouterLLMplugin for models hosted byOpenRouterInstallationFirst,install the LLM command-line utility.Now install this plugin in the same environment as LLM.llminstallllm-openrouterConfigurationYou will need an API key from OpenRouter. You canobtain one here.You can set that as an environment variable calledOPENROUTER_KEY, or add it to thellmset of saved keys using:llmkeyssetopenrouterEnter key: <paste key here>UsageTo list available models, run:llmmodelslistYou should see a list that looks something like this:OpenRouter: openrouter/openai/gpt-3.5-turbo
OpenRouter: openrouter/anthropic/claude-2
OpenRouter: openrouter/meta-llama/llama-2-70b-chat
...To run a prompt against a model, pass its full model ID to the-moption, like this:llm-mopenrouter/anthropic/claude-2"Five spooky names for a pet tarantula"You can set a shorter alias for a model using thellm aliasescommand like so:llmaliasessetclaudeopenrouter/anthropic/claude-2Now you can prompt Claude using:catllm_openrouter.py|llm-mclaude-s'write some pytest tests for this'DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cdllm-openrouter
python3-mvenvvenvsourcevenv/bin/activateNow install the dependencies and test dependencies:pipinstall-e'.[test]'To run the tests:pytest
Package: llm-os
chatgpt-tool-hubAn open-source chatgpt tool ecosystem where you can combine tools with chatgpt and use natural language to do anything.I will update this doc laterInstallationYou can install directly usingpipby doingchatgpt-tool-hubExamplefromchatgpt-tool-hub.appsimportload_appos.environ["PROXY"]="http://192.168.7.1:7890"app=load_app()reply=app.ask("北京今天的天气怎么样？")print(reply)Toolpython_replrequests(GET by default)terminalopen-meteo-apiwikipedianews-api(you need news_api_key fromhttps://newsapi.org/)wolfram-alpha(you need wolfram_alpha_appid fromhttps://developer.wolframalpha.com/)google-search(you need google_api_key and google_cse_id described herehttps://cloud.google.com/docs/authentication/api-keys?visit_id=638154342880239948-3750907574&rd=1&hl=zh-cn)
Package: llm-palm
llm-palmPlugin forLLMadding support for Google's PaLM 2 model.InstallationInstall this plugin in the same environment as LLM.llminstallllm-palmConfigurationYou will need an API key from Google. Instructions for obtaining one:https://developers.generativeai.google/tutorials/setupYou can set that as an environment variable calledPALM_API_KEY, or add it to thellmset of saved keys using:llmkeyssetpalmEnter key: <paste key here>UsageThis plugin adds a model calledpalm. You can execute it like this:llm-mpalm"Ten great names for a pet pelican"PaLM also supports system prompts:echo"I like pelicans a lot"|llm-mpalm--system"Translate to french"DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cd llm-palm
python3 -m venv venv
source venv/bin/activateNow install the dependencies and test dependencies:pip install -e '.[test]'To run the tests:pytest
Package: llm-pdf-chat
llm-pdfA CLI utility to index, summarize, and chat with PDF files.Installationpipxinstallllm-pdf-chatUsageUsage:llm-pdf[OPTIONS]COMMAND[ARGS]...

Options:--versionShowtheversionandexit.--helpShowthismessageandexit.

Commands:arxivDownloadandindexapaperfromarxiv.chatChatwithyourPDFs.deleteResetthecollection.indexChatwithyourPDFs.listResetthecollection.summarizeChatwithyourPDFs.ExamplesComing soon!
Package: llmplus
llmplusUsagefromllmplus.ChatGot.apiimportAPI,Modelsauthorization_key='your authorization key from website headers'api=API(authorization_key)api.model=Models.GPT4()answer=api.send('who is the manager of the real madrid?')asnwer2=api.send('who is the CEO?')ModelsModels.GPT4Models.GPT3Models.Claude2Acsess to the chat historyAPI.json_data['messages']Extracting code block
set extract_code to the type of code block to return just the code blockasnwer2  = api.send('who is the CEO?', extract_code = "json")LLM output:
#=====================
your output is here{"a":"ssdgs"}#=====================LLM output with extract_code:
#=====================
{"a": "ssdgs"}
#=====================
Package: llmpool
LLM-PoolThis simple project is to manage multiple LLM(Large Language Model)s in one place. Because there are too many fine-tuned LLMs, and it is hard to evaluate which one is bettern than others, it might be useful to test as many models as possible. Below is the two useful usecases that I had in mind when kicking off this project.compare generated text from different models side by sidecomplete conversation in collaboration of different modelsUsecasefromllmpoolimportLLModelPoolfromllmpoolimportLocalLoRAModelfromllmpoolimportRemoteTxtGenIfLLModelfromtransformersimportAutoModelForCausalLMmodel_pool=LLModelPool()model_pool.add_models(# alpaca-lora 13bLocalLoRALLModel("alpaca-lora-13b","elinas/llama-13b-hf-transformers-4.29","LLMs/Alpaca-LoRA-EvolInstruct-13B",model_cls=AutoModelForCausalLM),RemoteTxtGenIfLLModel("stable-vicuna-13b","https://...:8080"),)formodelinmodel_pool:result=model.batch_gen(["hello world"],GenerationConfig(...))print(result)_,stream_result=model.stream_gen("hello world",GenerationConfig(...))forretinstream_results:ifinstanceof(model,LocalLoRALLModel)or\instanceof(model,LocalLLModel):print(ret,end='')else:print(ret.token.text,end='')Alternatively, you can organize the model pool with yaml filefromllmpoolimportinstantiate_modelsmodel_pool=instantiate_models('...yaml')TodoAdd example notebooksYaml parser to add models to model pool
Package: llm_providers
llm-providersA common, minimal-dependency, async-first interface for interacting with Large Language Model provider APIs.Getting Startedpip install llm-providersfromllm_providers.openaiimportOpenAIProviderasyncdefmain():provider=OpenAIProvider(connection_str=os.environ.get("OPENAI_API_KEY"),)completion=awaitprovider.complete(prompt="hey i'm a robot who")assertcompletion.prompt=="hey i'm a robot who"asserttype(completion.completion_text)==strif__name__=="__main__":asyncio.run(main())LicenseThis project is licensed under the Apache LicenseAcknowledgmentsMuch code borrowed fromhttps://github.com/HazyResearch/manifest
Package: llm-python
llm-pythonRun a Python interpreter in the LLM virtual environmentInstallationInstall this plugin in the same environment asLLM.llminstallllm-pythonUsageThis plugin adds a newpythoncommand to LLM. This executes Python in the same virtual environment as LLM itself.You can use this to check the Python versionllmpython--version# Should output 'Python 3.10.10' or similarOr to start a Python shell. In that shell you can importllmand use it to interact with models:llmpythonPython 3.10.10 (main, Mar 21 2023, 13:41:05) [Clang 14.0.6 ] on darwinType "help", "copyright", "credits" or "license" for more information.>>>importllm>>>m=llm.get_model("mistral-7b-instruct-v0")>>>print(m.prompt("Three fun facts about pelicans"))1. Pelicans have a unique method of hunting for food. They fly high above the water and then fold their wings into a disc shape, creating a large scoop that they use to catch fish. This technique is called “plunge diving” and it allows them to catch up to six pounds of fish in one dive!2. Pelicans have an incredible memory when it comes to finding food. They can remember the location of every single fishing spot they’ve ever visited, even if it’s been years since they last went there. This is because they use a combination of visual cues and the earth’s magnetic field to navigate.3. Pelicans are incredibly social birds that form large flocks called “rookeries.” These rookeries can contain thousands of pelicans, and they are often found in areas with abundant food sources such as coastlines or offshore islands. In these groups, pelicans will engage in a variety of behaviors, including preening, grooming, and even playing with one another.DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cdllm-python
python3-mvenvvenvsourcevenv/bin/activateNow install the dependencies and test dependencies:llminstall-e'.[test]'To run the tests:pytest
Package: llmrails
LLMRails Python SDKThis package provides functionality developed to simplify interfacing with theLLMRails APIin Python 3.DocumentationFor more details on advanced parameters, you can also consult theAPI documentation.InstallationThe package can be installed withpip:pipinstall--upgradellmrailsInstall from source:pipinstall.RequirementsPython 3.7+Quick StartTo use this library, you must have an API key and specify it as a string when creating thellmrails.Clientobject. API keys can be created through theplatform. This is a basic example of the creating the client and using thegenerateendpoint.importllmrails# initialize the LLMRails Client with an API Keyrails=llmrails.Client('YOUR_API_KEY')# generate a prediction for a promptembeddings=rails.embed(model='embedding-english-v1',text=['Hey There'])# print the predicted textprint('embeddings:{}'.format(embeddings.data[0].embedding))
Package: llm-repl
LLM REPLWhat is this?The goal of this project is to create a simple, interactiveREPL(Read-Eval-Print-Loop) that allows users to interact with a variety of Large Language Models (LLMs). The project is mainly built on top of two Python libraries:langchain, which provides a convenient and flexible interface for working with LLMs, andrichwhich provides a user-friendly interface for the REPL.
The REPL can also be launched inheadledssmode and it can be interacted with usingwebsocket.Currently, the project is in development and only supports interaction with the ChatGPT but it has been structure to make it easy to extend it use any LLMs, including custom ones (by extendingBaseLLMin./src/llm_repl/llms/__init__.py).ChatGPT can be interacted by using the modelsgpt-3.5-turboandgpt4(For users who got GPT-4 API beta).FeaturesThe REPL supports the following features:Streaming ModeThe REPL won't wait for the model to finish generating the output, but it will start printing the output as soon as it is available.Conversation MemoryThe REPL supports conversation memory. This means that the model will remember the previous conversation and will use it to generate the next response.Pretty PrintingThe REPL supports Markdown rendering both of the input and the output.PS: In this initial version of the REPL, the full Markdown syntax is only when running the tool innon-streamingmode. Instreamingmode only code sections will be pretty printed.Headless ModeThe REPL can be run in headless mode. This means that it can be interacted with using a websocket. This is useful for integrating the REPL with other applications / other UIs.To launch the REPL in headless mode, run the following command:llm-repl--replwebsocket--port<PORT>Model Switching on the FlyCOMING SOON...Conversation HistoryCOMING SOON...Installationpipinstallllm-replUsageFirst export your OpenAI API key as an environment variable:exportOPENAI_API_KEY=<OPENAI_KEY>Then run the REPL:llm-replOr if you want to use a specific model:llm-repl--llmchatgpt4Run inside Dockerdockerrun-it--rm-eOPENAI_API_KEY=<OPENAI_KEY>phate/llm-replOr if you want to source the environment variables from a file, first create a file called.envwith the following content:OPENAI_API_KEY=<OPENAI_KEY>And then run the following command:dockerrun-it--rm--env-file.envphate/llm-replDevelopmentTo install the REPL in development modeThen install the package in development mode:pipinstall-e".[DEV]"Before contributing, please make sure to run the following commands:pre-commitinstall
Package: llm-router
LLM RouterA fast semantic router using Embeddings.LLM Router lets you define "routes" - sets of sentences or keywords with similar semantics. You can then use these routes to route a user's input to the appropriate LLM (or other model) for a response, leading to faster inference and better results. It's a faster and more efficient alternative to using a single LLM for all responses or letting a single LLM choose function calls.It can work entirely locally usingsentence_transformersor using OpenAI's API.Important: results WILL vary between openai and the sbert/sentence-transformer model you pick. You'll need to experiment to find the best model for your use case, as well as adjusting the threshold for your needs.But why?This code was created as part of Clio AI (my most recent startup) - the idea then was to allow selecting the best "agent" to handle a certain conversation based on the semantics of what the user is saying, which allowed us to use "lesser" models like GPT-3.5 instead of GPT-4 to respond with the appropriate set of system prompt, function calls, etc. In some cases, it allows you to skip LLMs altogether, saving time and $$$.UsageInstall with:pip install llm-routerTo use the OpenAI API, you'll need an API Key and theopenaipip package (pip install openai).To use a Sentence Transformers model, you'll need thesentence_transformerspip package (pip install sentence_transformers). No API key is required, model weights will be downloaded from huggingface on first use.By default, the Router will always match one of the routes. If you'd like to allow the user to say something that doesn't match any of the routes, you can set thethresholdvalue when initializing the Router. This will returnNoneif no routes match with a certain percentage (0 to 1.0).Define routes in code and use the router like so:fromllm_routerimportRouter,Routefromllm_router.chromaimportSentenceTransformerrouter=Router(engine=SentenceTransformersEngine(model_name='all-distilroberta-v1',threshold=0.3,),routes=[Route(name='upscale',sentences=['upscale the image','increase resolution','I want a larger image','increase the pixel count','increase the size of the image','increase the resolution of the image','I think this is too small',]),Route(name='edit',sentences=['edit image','rotate image','flip image','resize image','adjust contrast','adjust saturation','adjust brightness','change colors','color balance','change image format','change dimensions','change size','crop image',])])user_message='I want to increase the resolution of the image, is that possible?'ifrouter.match(user_message)=='upscale':handle_upscale()elifrouter.match(user_message)=='edit':handle_edit()else:print('Sorry, I don\'t understand.')ReferencesPretrained Sentence Transformer models-all-mpnet-base-v2is recommended for most use casesm but you can also useall-distilroberta-v1for a smaller model that's faster. I haven't tested other models yet.Future enhancements (PRs welcome!)Support for other embedding APIs (cohere, vertexai, etc)Storing weights to prevent reprocessing every timeAdd a finetuning script to create routes from a dataset and refine sentences
Package: llm-sampler
LLM SamplerInstall with:pipinstallllm_samplerllm_samplerallows you to sample from any LLM by given logits.It is a collection of various sampling techniques found online.For now, the methods are:sample_huggingface- the one used intransformerssample_gpt_fast- the one used ingpt-fastIt also allows you get probability scores for sequences given by the user.For example, if you supply the input:
Input:The sentiment of the sentence 'I loved it' isOption 0:positiveOption 1:negativeOption 2:unknownThis lib will help youcreate_batch_with_continuations
Package: llm-sentence-transformers
llm-sentence-transformersLLMplugin for embedding models usingsentence-transformersFurther reading:LLM now provides tools for working with embeddingsEmbedding paragraphs from my blog with E5-large-v2InstallationInstall this plugin in the same environment as LLM.llminstallllm-sentence-transformersConfigurationAfter installing the plugin you need to register one or more models in order to use it. Theall-MiniLM-L6-v2model is registered by default, and will be downloaded the first time you use it.You can try that model out like this:llmembed-mmini-l6-c'hello'This will return a JSON array of floating point numbers.You can add more models using thellm sentence-transformers registercommand. Here is alist of available models.Two good models to start experimenting with areall-MiniLM-L12-v2- a 120MB download - andall-mpnet-base-v2, which is 420MB.To install thatall-mpnet-base-v2model, run:llmsentence-transformersregister\all-mpnet-base-v2\--aliasmpnetThe--aliasis optional, but can be used to configure one or more shorter aliases for the model.You can runllm aliasesto confirm which aliases you have configured, andllm aliases setto configure further aliases.UsageOnce you have installed an embedding model you can use it like this:llmembed-msentence-transformers/all-mpnet-base-v2\-c"Hello world"Or use its alias:llmembed-mmpnet-c"Hello world"Embeddings are more useful if you store them in a database - seethe LLM documentationfor instructions on doing that.Be sure to review the documentation for the model you are using. Many models will silently truncate content beyond a certain number of tokens.all-mpnet-base-v2says that "input text longer than 384 word pieces is truncated", for example.DevelopmentTo set up this plugin locally, first checkout the code. Then create a new virtual environment:cdllm-sentence-transformers
python3-mvenvvenvsourcevenv/bin/activateNow install the dependencies and test dependencies:pipinstall-e'.[test]'To run the tests:pytest
Package: llm-sepweight
llm_sepweightThellm_sepweightmicrolib is designed to manage the weights of large language models (LLMs) by organizing them into
a unified format calledsepweight.Every LLM has roughly the same three parts:begin- the part of the model which computes the embeddings before the layersmid- a number of (most commonly transformer) layersend- the part of the model which converts the hidden state into a prediction for the next tokensepweightessentially mirrors the state dict of the LLM into the filesystems, meaning that you will (roughly) have one
file per each of these components of the LLM.This format enables the distributed execution of LLMs by separating the model weights into distinct segments that can
be individually managed and accessed as needed.This microlib provides methods for:dump-ing from different formats tosepweightload-ing the state dict needed to run a part of the LLMThe only dependency istorch.Microlib docs are available onhttps://microlib.org/llm_sepweight.htmlInstallationpipinstallllm_sepweightQuick ExampleTo convert an existing state dict intosepweight, you need to provide:decideris a function which will be called for each key in the state dict, and has to decide whether that key should
be part of thebegin,mid, orendsection and the the new name of the key.Examplestate_dict- is just your usual PyTorch state dictout_pathis the directory, in which you want the result to be stored.importllm_sepweightllm_sepweight.dump(decider=decider,state_dict=state_dict,out_path=out_path)You could have multiple state dicts (for example coming from multiple files), it's ok to calldump_to_directorywith
each of them. The result will be combined state dict of all the state dicts provided for a givenout_path.Goal formatllm_sepweightallows you to convert different formats to its own directory format, which is very simple.
Let's have a look at an example:├──begin.pth
├──end.pth
├──mid.00000.pth
├──mid.00001.pth
├──mid.00002.pth
├──mid.00003.pth
├──mid.00004.pth
└──mid.00005.pthAll the weights are stored in a directory in usual.pthfiles.This format is very simple and allows great flexibility. For example, a node running layers 0 to 3 would only need to
download thebegin,mid.00000,mid.00001,mid.00000files.Why do we need it?There are all sorts of different formats for storing the weights of an LLM -.pthfiles,safetensors,H5,arrow,GGUF, etc.Moreover, there is a lot of difference in the naming of the transformer layers, of the start embedding, and of the final head.llm_sepweightaims to provide functions, through which you can convert different formats into asepweightformat.
Thesepweightformat is a unified, simple format that allows you to treat the weights of all LLMs in the same way
when running nodes in distributed way.
Package: llm-space
LLM ToolsThis package wraps the OpenAI API and other LLM models that share the completion endpoint feature. It provides a simple interface to generate text from a prompt.Installationpipinstallllm-spaceUsagefromllm_space.openaiimportpromptfrompathlibimportPath# Create a prompt object that will be logged to the `my-task` directory under the# current working directory.# The PromptZeros class will use arguments that will maximize the determinism of the outputs.pz=prompt.PromptZeros(Path("./"),"my-task")# Send a prompt to the API and return the response.w=pz.send_prompt("What is the WDI code for GDP?",return_data=True)# Print the response.print(w)# Output:# {'message': [{'role': 'system', 'content': ''}, {'role': 'user', 'content': 'What is the WDI code for GDP?'}], 'response': {'id': 'chatcmpl-8V0IN0VbeYTrsBAohs309DHJQLJL8', 'choices': [{'finish_reason': 'stop', 'index': 0, 'message': {'content': 'The WDI code for GDP (Gross Domestic Product) is "NY.GDP.MKTP.CD".', 'role': 'assistant', 'function_call': None, 'tool_calls': None}}], 'created': 1702399995, 'model': 'gpt-3.5-turbo-0613', 'object': 'chat.completion', 'system_fingerprint': None, 'usage': {'completion_tokens': 23, 'prompt_tokens': 20, 'total_tokens': 43}}, 'content': 'The WDI code for GDP (Gross Domestic Product) is "NY.GDP.MKTP.CD".', 'api_kwargs': {'temperature': 0, 'top_p': 0, 'n': 1, 'stream': False, 'frequency_penalty': 0, 'presence_penalty': 0, 'model': 'gpt-3.5-turbo', 'max_tokens': 4029}, 'metadata': {}, 'message_hash': '1a5816095c232f29428d4cec70498391'}NOTE: The output is also logged to themy-taskdirectory under the current working directory.
Package: llm-sql-prompt
ChatGPT Prompt for SQL WritingGenerate a prompt for writing SQL queries with LLMs like ChatGPT. Drop your database URL and table name into the script and it will generate a prompt for you to copy and paste into your favorite LLM.What this doesSnapshot of Table Structure: Understand the columns, types, and organization of your table at a glance.Sample Rows: Includes INSERT statements to describe the data in your table.UsageInstall the package:pipinstallllm-sql-promptHere's how to use it:Usage:llm-sql-prompt[OPTIONS]DATABASE_URL[TABLE_NAME]Options:--helpShowthismessageandexit.Generate a prompt from a postgres database:llm-sql-promptpostgresql://postgres:postgres@localhost:5555/database_nametable_name|pbcopy
llm-sql-prompt$DATABASE_URLTunneling to a remote portIf you find yourself wanting to tunnel into a remote box and work with a production database, here's some helpful commands so you don't need to remember the weird SSH tunneling syntax:functionfind_random_open_port(){localstart_port=${1:-1024}localmax_attempts=100localattempt=0localport=$start_portwhile((attempt<max_attempts));doif!nc-zlocalhost$port2>/dev/null;thenecho$portreturnfiport=$((port+1))attempt=$((attempt+1))doneecho"No open port found after$max_attemptsattempts, starting from$start_port.">/dev/stderrreturn1}functionssh-tunnel(){if[$#-lt2];thenecho"Usage: ssh-tunnel remote_host remote_port [local_port]"echo"This function sets up SSH port forwarding."return1filocalremote_host=$1localremote_port=$2locallocal_port=${3:-$(find-random-open-port$remote_port)}if[[-z$local_port]];thenecho"Failed to find an open local port."return1fiecho"Forwarding local port$local_portto remote port$remote_porton$remote_host..."set-xssh$remote_host-L${local_port}:localhost:${remote_port}}TODOSuper basic script, needs a lot of workpg supportone entrypointmultiple tablesprompt tweakingunderstand prompt size limits and sample records until one fits