Package: apollo-sdk
apollo-sdk: model management toolApollo is an open-source validation framework to deploy guardrails for improved user experience and responsiveness of your AI systems..Docs »Join the waitlist·Report Bug·Community DiscordWhy Apollo API?Before Apollo, testing model quality and automating workloads was time-consuming, with Apollo, you can simplify, accelerate and backtest the entire process. This makes it easier to train classifiers, handle real-time changes and make data driven decisions.🚀 Interesting, how can I try it?Lets install the SDK first...pipinstallapollo-sdkRegression tests vs Automated pipelinesapollo-sdkhelps you tune LLM prompts systematically across many relevant test cases. By evaluating and comparing LLM outputs to build decision making workflows. Users can test prompt quality and catch regressions faster.Evaluating prompt qualityWith Apollo python library and CLI toolkit, you can:Detecting real-time changesin your dataAutomating tasks againstimage, video, audio or textSimplifying the process ofback-testing qualityfor your AI modelsMaking sure your integration is robust, so younever again have to worry about stuck/stale data or false-positivesTest multiple promptsagainst predefined test casesEvaluate quality and catch regressionsby comparing LLM outputs side-by-sideSpeed up evaluationswith caching and concurrent testsFlag bad outputs automaticallyby setting "expectations"Use as acommand line tool, or integrate into your workflow with our libraryUse anyAI provider, API or database under one APIapollo-sdkproduces table views that allow you to quickly review prompt outputs across many inputs. The goal: tune prompts systematically across all relevant test cases, instead of testing prompts by trial and error.Usage (command line)View full documentation »Support for user interface coming soonIt works on the command line, you can output to [json,csv,yaml]:To get started, run the following command:apollo-sdk initThis will create some templates in your current directory:prompts.txt,vars.csv, andconfig.json.After editing the prompts and variables to your desired state,apollo-sdkcommand to kick off an prompt evaluation test:apollo-sdk -p ./prompts.txt -v ./vars.csv -r openai:completionIf you're looking to customize your usage, you have a wide set of parameters at your disposal. See theConfiguration docsfor more detail:OptionDescription-p, --prompts <paths...>Paths to prompt files, directory, or glob-r, --providers <name or path...>One of: openai:chat, openai:completion, openai:model-name, hive:hate, google:safety, etc. SeeAI Providers-o, --output <path>Path to output file (csv, json, yaml, html)-v, --vars <path>Path to file with prompt variables (csv, json, yaml)-c, --config <path>Path to configuration file.config.jsonis automatically loaded if present-j, --max-concurrency <number>coming soonMaximum number of concurrent API calls--table-cell-max-length <number>coming soonTruncate console table cells to this length--gradercoming soonProvider that will grade outputs, if you are usingExamplesView full documentation »Prompt qualityIn this example, we evaluate whether adding adjectives to the personality of an chat bot affects the responses:apollo-sdk-pprompts.txt-vvars.csv-ropenai:gpt-3.5-turboThis command will evaluate the prompts inprompts.txt, substituing the variable values fromvars.csv, and output results in your terminal.Have a look at the setup and full output in another format:apollo-sdk -p prompts.txt -v vars.csv -r openai:gpt-3.5-turbo -o output.jsonYou can also output a nicespreadsheet, JSON, or YAMLfile:{"results":[{"prompt":{"raw":"Rephrase this in French: Hello world","display":"Rephrase this in French: {{body}}"},"vars":{"body":"Hello world"},"response":{"output":"Bonjour le monde","tokenUsage":{"total":19,"prompt":16,"completion":3}}}// ...],"stats":{"successes":4,"failures":0,"tokenUsage":{"total":120,"prompt":72,"completion":48}}}Here's an example of a side-by-side comparison of multiple prompts and inputs:Model qualityYou can evaluate the difference between safety outputs for a specific context:Model quality tests & python package for model testing is a beta feature at the moment, open an issue and tag us to setupapollo-sdk-pprompts.txt-rhiveai:hategoogle:safety-ooutput.jsonConfigurationSetting up an model test: Learn more about how to set up prompt files, vars file, output, etc.Building Automated Pipelines in the User Interface or ProgrammaticallyView full documentation »Let's setup your first Integration!It will pull from your local database (and keep it in sync).# import the packagefromapollo.clientimportApollo# sync data from your database instance# (we support supabase at the current moment or postgresql via uri format)Apollo.connect("postgres://username:password@hostname:port/database_name")# If you want to test out operation on your external connectionApollo.fetch_tables()Apollo.query("desc","table","column")...and create a workflow with a simple command:Note: you can use our sandbox api and skip providing a token or obtain a Auth tokenhere, sign up today on ourSite# import the packagefromapollo.clientimportApollo# Use any providerApollo.use("google_perspective:<model name>",secret="YOUR_API_TOKEN_HERE")# Lets check to see if a phrase contains threatsApollo.detectText(prompt="Phrase1",content_id="content-id",community_id="user-id")Example response:{"attributeScores":{"THREAT":{"spanScores":[{"begin":0,"end":12,"score":{"value":0.008090926,"type":"PROBABILITY"}}],"summaryScore":{"value":0.008090926,"type":"PROBABILITY"}},"INSULT":{"spanScores":[{"begin":0,"end":12,"score":{"value":0.008804884,"type":"PROBABILITY"}}],"summaryScore":{"value":0.008804884,"type":"PROBABILITY"}},"SPAM"// ...},"languages":["en"],"clientToken":"content_123","detectedLanguages":["en","fil"]}Experimental inputs:# Create custom rules which creates a task!Apollo.rule('Phrase1','>=','0.8')# https://docs.apolloapi.io/docs/featuresApollo.detectImage('Image1','contains','VERY_LIKELY')# Image Analysis/OCRApollo.detectSpeech('Audio1','contains','UNLIKELY')# Audio ProcessingApollo.detectVideo('Video1','contains','POSSIBLE')# Video AnalysisApollo.detectText('Phrase1','contains','UNKNOWN')# Text AnalysisApollo.test('prompt','expected_output')# ML ValidationThat's all it takes!In practice, you probably want to use one of our native SDKs to interact with Apollo's API or use our custom browser client so you dont have to write code. If so, sign up atApollo API!Cool, what can I build with it?Apollo can help youquickly automate tasksfor model management, performance, labeling, object detection and more.Teams can use Apollo tobuild native in-app connectionsrelated to active response, content moderation, risk management, fraud detection, etc.Someautomate their personal liveswith Apollo by integrating against discord communities or their personal livesDevelopmentContributions are welcome! Please feel free to submit a pull request or open an issue.📦 pre-commit configAs an open source project, Apollo welcomes contributions from the community at large. This isn’t an exhaustive reference and is a living document subject to change as needed when the project formalizes any practice or pattern.Clone the repo and start Apollo locally...gitclonehttps://github.com/apolloapi/apolloapi.gitcdapolloapi&&python3-mvenvenv&&sourceenv/bin/activate&&pipinstall-rrequirements.txtAfter installing system dependencies be sure to install pre-commit for lint checkspipinstallpre-commit

pre-commitinstall

pre-commitrun--all-filesApollo uses commit messages for automated generation of project changelog. For every pull request we request contributors to be compliant with the following commit message notation.<type>: <summary>

<body>Accepted<type>values:new = newly implemented user-facing featureschg = changes in existing user-facing featuresfix = user-facing bugfixesoth = other changes which users should know aboutdev = any developer-facing changes, regardless of new/chg/fix statusSummary (The first line)The first line should not be longer than 75 characters, the second line is always blank and other lines should be wrapped at 80 characters.🔍 Neat, I would like to learn more⭐ Follow our development by starring us here on GitHub ⭐Share feedback or ask questions on theDiscord communityChat with a member of the team👋Check ourblog on Trust & SafetyLook at our docs on how to get startedhere!AI ProvidersWe support OpenAI as well as a number of models. It's also possible to set up your own custom AI provider.See Provider documentationfor more details.