Package: athina
Documentation|Watch Demo Video →OverviewAthina is an open-source library with plug-and-play preset evals designed to help engineers systematically improve their LLM reliability and performance through eval-driven-development.Quick LinksDocumentationQuick StartPreset EvalsRun an evalRun an eval suiteCustomize an evalView Results on Athina DashboardLoading Data for EvalsCookbooksProduction MonitoringWhy you need evalsEvaluations (evals) play a crucial role in assessing the performance of LLM responses, especially when scaling from prototyping to production.They are akin to unit tests for LLM applications, allowing developers to:Catch and prevent hallucinations and bad outputsMeasure the performance of modelRun quantifiable experiments against ambiguous, unstructured text dataA/B test different models and prompts rapidlyDetect regressions before they get to productionMonitor production data with confidence🔴 Problem: Flaws with Current LLM Developer WorkflowsThe journey from a demo AI to a reliable production application is not easy.Developers usually start iterating on performance by manually inspecting the outputs. Eventually they progress to using spreadsheets, CSVs, or evaluating against a golden dataset.Each method has drawbacks, requires different tooling, and evaluation methods.See moreA lot of manual effort is required to set up a good infrastructure for running evals - creating a dataset, reviewing the responses, creating evals, and internal tooling / dashboard, tracking experiment parameters and metrics for historical record.Eventually every LLM developer realizes the indispensable need for evals and an infrastructure to consistently run and track iterations to improve performance and reliability systematically.🟢 Solution: Athina EvalsGithub|Watch Demo Video|DocsAthina is an open-source library that offers a system for eval-driven development, overcoming the limitations of traditional workflows.Our solution allows for rapid experimentation, and customizable evaluators with consistent metrics.Here’s why this is better than building in-house eval infrastructure:Plug-and-PlayPreset Evals: Ready-to-use evals for immediate applicationIntegrated Dashboard: For tracking experiments and inspecting the results in a web UI.Custom Evaluators: A flexible framework to craft tailored evals.Consistent Metrics: Uniform evaluation standards across all stages. Evaluate your model in dev and prod using a consistent set of metrics.Historical Record: Automatic tracking of every prompt iteration.Quick Start: Easy 5-min set up.Here’s ademo video.Quick StartThe easiest way to get started is to use one of ourExample Notebooksas a starting point.To get started with Athina Evals:1. Install theathinapackagepip install athina2. Set your API keysIf you are using the python SDK, then can set the API keys like this:fromathina.keysimportAthinaApiKey,OpenAiApiKeyOpenAiApiKey.set_key(os.getenv('OPENAI_API_KEY'))AthinaApiKey.set_key(os.getenv('ATHINA_API_KEY'))If you are using the CLI, then runathina init, and enter the API keys when prompted.3. Load your dataset like this:You can alsoload datausing a CSV or Python Dictionaryfromathina.loadersimportRagLoaderdataset=RagLoader().load_json(json_filepath)4. Now you can run evals like this.fromathina.evalsimportDoesResponseAnswerQueryDoesResponseAnswerQuery().run_batch(data=dataset)For more detailed guides, you can follow the links below to get started running evals using Athina.Quick Start GuideRun an evalRun an eval suiteCustomize an evalView Results on Athina DashboardLoading Data for EvalsPreset EvalsYou can use our preset evaluators to add evaluation to your dev stack rapidly.Here are the preset evaluators in this library:RAG EvalsThese evalsare useful for evaluating LLM applications with Retrieval Augmented Generation (RAG).Context Contains Enough InformationContext RelevanceAnswer RelevanceDoes Response Answer QueryResponse FaithfulnessWe have also built other evaluators that are not yet a part of this library (but will soon be)You can find more information about these in our documentation.Summarization Accuracy Evals:These evals are useful for evaluating LLM-powered summarization performance.Summarization HallucinationSummarization InformativenessMore EvalsOther EvalsCustom EvalsSee this page for more information, on how to write your owncustom evals.Why should I use Athina's Evals instead of writing my own?You could build your own eval system from scratch, but here's why Athina might be better for you:Athina provides you with plug-and-play preset evals that have been well-testedAthina evals can run on both development and production, giving you consistent metrics for evaluating model performance and drift.Athina removes the need for your team to write boilerplate loaders, implement LLMs, normalize data formats, etcAthina offers a modular, extensible framework for writing and running evalsAthina calculate analytics like pass rate and flakiness, and allows you to batch run evals against live production data or dev datasetsNeed Production Monitoring and Evals? We've got you covered...Athina eval runs automatically write into Athina Dashboard, so you can view results and analytics in a beautiful UI.Athina track your experiments automatically, so you can view a historical record of previous eval runs.Athina calculates analytics segmented at every level possible, so you can view and compare your model performance at very granular levels.About AthinaAthina is building an end-to-end LLM monitoring and evaluation platform.Website|Demo VideoContact us athello@athina.aifor any questions about the eval library.