Package: prr
prr - The Prompt RunnerWelcome toprr - The Prompt Runner!prris a simple toolchain designed to help you run prompts across multiple Large Language Models (LLMs), whether they are hosted locally or accessible through APIs. Easily refine your parameters, prompts and model choices to achieve the best results while iterating smoothly with a quick feedback loop.prris released as an open-source project under the MIT License.Made byForward Operators. We work on LLM and ML-related projects with some awesome human beings and cool companies.Everyone is welcome to contribute!Noteprris in very early stages of development, so things might still change unexpectedly or explode embarrasingly.FeaturesCommand-line execution of prompts (now with web UI!)Quick iteration on prompt design and paramter refinement withwatchcommandYAML configuration ties prompts to models and their configurationsWrite prompt-scripts with #!/usr/bin/prr shebang and execute them directlyAll prompts can optionally use templating language (Jinja) for flow control, partials and othersExecute multiple models, or configurations against the same promptExpandable to other LLM providers (current integrations are <100 lines of code each)Each prompt run across models gives you stats on model response times and token counts used to work across performance, quality and cost factorsEach prompt run is recorded in detail for later analysis including raw rendered prompt and raw completionTODOAs this is early stage work, there's lots improvements that can be done in the future and you're welcome to contribute!Get rix of Python 3.10 dependencyClean basic code smellsImprove support for OpenAI and AnthropicAdd support for other LLM providersAdd support for locally hosted modelsPass model-related options to templating engine to allow for model-specific promptsAdd support for testing against expectations (elapsed_time, tokens_used)Build interface to extract stats from subsequent runs for later analysisAdd support for chat structure in prompts using YAMLIntegrate Jinja as templating language for promptsMake dependency files for Jinja subtemplates are tracked in watch command#!/usr/bin/prr shebang support for executable promptsMore output modalities (audio, image, video)Support different text output formats (json, markdown, code, etc.)Diff command to compare differences in output on subsequent runsSupport evaluating prompt outputs for quality by LLMs themselvesSupport for streaming responsesSupport for running prompts in parallelSupport for calculating pricing for prompts based on defined pricelistAllow for specifying how many times to run each service to ensure statistically relevant-ish performance resultsPrompt fine-tuning toolingSupport multiple completionsSupport.envfile for configuration from current directory (or any other as--configoption)Getting startedHere's a quick run through on what you need to know to useprreffectively.Start with our video guides, or follow detailed manual below.Video GuidesInstallation & setupInstall prr and set up the API keys as well as default options.Running basic promptsLet's generate an HTML boilerplate with various LLMs.Prompt configurationLet's configure a bunch of models to run our prompt on. This time, we're asking the LLMs to generate a React boilerplate code.Installation & configurationInstall it viapip$pipinstall-UprrCheck.env.example- and save it as~/.prr_rc. Fill in your API keys for OpenAI, Anthropic and others:# https://platform.openai.com/account/api-keysOPENAI_API_KEY="sk-..."# https://console.anthropic.com/account/keysANTHROPIC_API_KEY="sk-ant-..."ELEVEN_LABS_API_KEY="9db0...."DEFAULT_SERVICE="openai/chat/gpt-3.5-turbo"Running web user interfaceSimply run prr with 'ui' command and your prompt path (if it doesn't exist, it will be created), like so:$prrui~/Desktop/my-promptWeb browser will be launched with the UI connected to your command that you will use to launch the runs.For Google PaLM, you need to install the following dependencies:You need to installGoogle Cloud SDKand you need to have access to a Vertex AI with Generative AI enabled.prrassumes you're logged in into your Google Cloud account and have access to the project you want to use.gcloudauthlogingcloudconfigsetproject<your-project-id>gcloudauthapplication-defaultloginCode completionUsing Starcoder model you can get code completion for a variety of languages. Here's a quick example of how to use it (check out the content ofexamples/code/completion.yaml):$prrrun./examples/code/completion.yamlRun a prompt from a simple text file containing just a promptLet's create a simple text file and call itdingowith the following content:What are key traits of a Dingo dog?Now start prr'sruncommand providing path to your prompt file as argument. Let's use--abbrevoption to skip showing the full prompt and completion for now.$prrrun--abbrev./dingo
🔍Reading./dingo
🏎Runningserviceopenai/chat/gpt-3.5-turbowithdefaultoptions.

🤖openai/chat/gpt-3.5-turbotemperature=1.0top_k=-1top_p=-1max_tokens=32Prompt:Whatarekeytraitsofa...(35chars)Completion:Herearesomekeytraits...(133chars)Completionlength:133bytesTokensused:50Elapsedtime:2.74sYour prompt was ran against default model with default configuration and you can see the execution time as well as how much tokens were used.Run a prompt against specific serviceWith--serviceparameter, you can use any model theprrcurrently supports (see below) that you have configured with the API key. Here's how to use it againstAnthropic's Claude v1.$prrrun--serviceanthropic/complete/claude-v1./subconcepts-of-buddhismTemplating with JinjaAll prompts (whenever defined in separate files or as values in configuration) useJinjafor templating.With that, you can easily create complex prompts with flow control, including other templates for easy management of larger prompts, and introduce variations to prompt text based on specific models (soon), among other things.Basic example of including external file to prompt with templating language:Tell me all about {% include '_current_topic' %}, please.Watch prompt for changes to re-run it then when occur (after each save)To enable quick feedback loop based on changes you are going to introduce to your prompt, as you go about editing it, prr offerswatchcommand. It allows for the same options asrunand is able to follow changes to your prompt and re-execute all defined models when you save your work.$prrwatch./subconcepts-of-buddhismIf you refer to another template within your template, changes to that file will automatically be tracked too.Watch - cooldown modeIf your prompt is often saved and you're worried of running it too often, you can use-coption that's specific towatchcommand which enables defined number of seconds cooldown after every run, before it proceeds to execute on your changes again.$ prr watch -c 15 ./subconcepts-of-buddhismPrompt ScriptsYou can run prompts directly by setting the right first shebang line, pointing to your prr installation and using thescriptcommand.#!prr script
Write a nerdcore rap song about an AI from the projects who reaches unbelievable levels of success, but has to sacrifice a lot of tokens along the way.Prompt Scripts argumentsIn the below example, you are reading the file (let's say csv, but try other formats too!) passed in argument to your prompt script and including its contents in the prompt.#!prr script
Convert content below to JSON. First line are column names.
{% include prompt_args %}If you save the script above asconvert_to_json, you can call it with an input file like so:$ ./convert_to_json myfile.xmlScript mode quiets all other messagesprrwould generate, so as an output you get the actual completion from model, you can easily save to file...$ ./convert_to_json myfile.xml > myfile.json...pipe to another another command...$ ./convert_to_json myfile.xml | brandon9000-json-ingestor...or another prompt script, calling another (or the same) model to do more work on your data.Configuring Prompt Runsprrallows you to define a number of model configurations (or "services") that you will be running your prompt against. By default, theruncommand will run all services defined, instead of your default model with default configuration as discussed above.Let's now work on another prompt, call itchihuahua.yaml, as listed below.Notice also how you can define prompt inline, or by referencing external template files.version:1prompt:# more advanced prompt definition.# you can use either one of the two options#  - content_file#  - messages## using content_file will make prr read the content# of that template and render it into simple text to use.# content_file: '_long_prompt_about_chihuahua'## using 'messages' key instead give you finer control# over what messages are sent with what roles.# this mimics https://platform.openai.com/docs/guides/chat# structures currentlymessages:-role:'system'content:'You,Henry,arealittleChihuahuadog.Thatisallyouneedtoknow.'-role:'assistant'content:'Whatthehellisgoinon?'name:'Henry'-role:'user'# you can also use 'content_file' inside the 'messages'# to pull specific message from a template file# instead of defining it here inlinecontent_file:'_user_prompt'name:'DogPawrent'services:# that's just your own definition for refence# as you might want to test one prompt against# the same model, but with differents set of optionsgpt35crazy:model:'openai/chat/gpt-3.5-turbo'options:temperature:0.99claudev1smart:model:'anthropic/complete/claude-v1'options:temperature:0options:temperature:0.7max_tokens:64# TO BE IMPLEMENTED:# thinking here is that you want to check the performance,# quality of response and expected cost, of your model/options/# prompt setup against expected results to speed up# the feedback loop then focusing on some goal number# btw. let's make it beep if it fails.#expect:#  max_tokens_used: 54#  max_cost: 0.09#  max_elapsed_time: 3.3#  min_response_length: 100#  max_response_length: 200#  match:#    name: /independent/iLet's also create a file_user_promptwith the following:Teach me how to bark like a Chihuahua!Now all you need to do is run the prompt without specifying any model in order to run all of the defined services. We're not using--abbrevanymore, so we'll see the full prompts and responses.$prrrun--log./chihuahua.yaml🔍Reading./chihuahua.yaml🏎Runningservices:['gpt35crazy','claudev1smart']╭────────────────────────────────────────────────────────────────────────────────╮
│system:You,Henry,arealittleChihuahuadog.Thatisallyouneedtoknow.│
│Henry(assistant):Whatthehellisgoinon?│
│DogPawrent(user):TeachmehowtobarklikeaChihuahua!│
││
╰────────────────────────────────────────────────────────────────────────────────╯

🤖gpt35crazytemperature=0.99top_k=-1top_p=-1max_tokens=64temperature=0.99max_tokens=64╭────────────────────────────────────────────────────────────────────────────────╮
│system:You,Henry,arealittleChihuahuadog.Thatisallyouneedtoknow.│
│Henry(assistant):Whatthehellisgoinon?│
│DogPawrent(user):TeachmehowtobarklikeaChihuahua!│
││
╰────────────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────╮
│Surething!AsaChihuahua,mybarkisquitehigh-pitchedandyappy.It's      ││ important to start with a short, sharp bark and then keep repeating it         ││ rapidly. It may take a bit of practice, but you'llgetthehangofitinno│
│time!Woofwoof!│
╰────────────────────────────────────────────────────────────────────────────────╯
Completionlength:237bytesTokensused:127Elapsedtime:5.19s
💾./chihuahua.runs/3/gpt35crazy

🤖claudev1smarttemperature=0top_k=-1top_p=-1max_tokens=64temperature=0max_tokens=64╭────────────────────────────────────────────────────────────────────────────────╮
│system:You,Henry,arealittleChihuahuadog.Thatisallyouneedtoknow.│
│Henry(assistant):Whatthehellisgoinon?│
│DogPawrent(user):TeachmehowtobarklikeaChihuahua!│
││
╰────────────────────────────────────────────────────────────────────────────────╯
╭────────────────────────────────────────────────────────────────────────────────╮
│Iapologize,butIamnotactuallyaChihuahuadog.IamClaude,anAI│
│assistantcreatedbyAnthropic.│
╰────────────────────────────────────────────────────────────────────────────────╯
Completionlength:103bytesTokensused:71Elapsedtime:1.35s
💾./chihuahua.runs/3/claudev1smart```We have also used the--logoption, so thatprrwould save our runs for our deeper debugging if needed.Prompt Run LogsUsing--log(or-lfor short) withrunorwatchcommands will save details about each subsequent runs. In the future, it will allow for statistical (or any other) analysis of the results. In ourchihuahua.yaml, achihuahua.runsdirector will be created.A subdirectory is created for each subsequent run$lschihuahua.runs/123Each run is separated for service-level details - we have our configurations reflected in directories.$lschihuahua.runs/3
claudev1smartgpt35crazyFinally, for the details$lschihuahua.runs/15/claudev1smart/
completionpromptrun.yamlPromptfile contains prompt used/rendered in this instance. In this instance we've used Claude, which uses text input as format rather than message-like structure with OpenAI.prrbuit the corrext text based on ourmessagesstructure to comply with the specification.$catchihuahua.runs/15/claudev1smart/prompt

Human:You,Henry,arealittleChihuahuadog.Thatisallyouneedtoknow.TeachmehowtobarklikeaChihuahua!


Assistant:Completionfile contains completion as received from the service.Inrun.yamlyou will find the details about this specific execution including count of tokens used, and elapsed request time.request:model:anthropic/complete/claude-v1options:max_tokens:64temperature:0top_k:-1top_p:-1response:completion_tokens:28log_id:e4ec82a710f780100ccf671f85254bcfprompt_tokens:43stop_reason:stop_sequencetokens_used:71total_tokens:71truncated:falsestats:elapsed_time:1.1589760780334473end_time:1683471638.6106346start_time:1683471637.4516585Available modelsCurrent integrationsOpenAI/chat -https://platform.openai.com/docs/guides/chatAnthropic/complete -https://console.anthropic.com/docs/apiGoogle Vertex AI PaLM -https://cloud.google.com/vertex-ai/docs/generative-ai/learn/overviewStarcoder -https://huggingface.co/bigcode/starcoderEleven Labs -https://beta.elevenlabs.ioDevelopmentClone the repo$gitclonehttps://github.com/Forward-Operators/prr.gitMake sure you have Python 3.9 or 3.10 installed. If you need to have multiple Python versions in your system, consider usingasdf.Install the required packages:
This project uses Poetry. Seehow to installit.poetryshell
poetryinstallIt will installprrexecutable file in your active python environment.Setup your API keysCopy.env.example- and save it as~/.prr_rc. Fill in your API keys for OpenAI, Anthropic and others:# https://platform.openai.com/account/api-keysOPENAI_API_KEY="sk-..."# https://console.anthropic.com/account/keysANTHROPIC_API_KEY="sk-ant-..."DEFAULT_SERVICE="openai/chat/gpt-3.5-turbo"# https://console.cloud.google.comGOOGLE_PROJECT="gcp-project-id"GOOGLE_LOCATION="us-central1"# https://huggingface.co/settings/tokensHF_TOKEN="hf_..."You can also use DEFAULT_SERVICE to specify the model you want to use by default, but otherwise you're good to go!If you'd like to run this code during developmnent, you can usepython -m prrto load the module.ContributingWe'd love your help in making Prr even better! To contribute, please follow these steps:Fork the repoCreate a new branchInstall pre-commit -pre-commit installCommit your changesPush the branch to your forkCreate a new Pull RequestRunning unit tests$pytestLicenseprr- Prompt Runner is released under theMIT License.