Package: benchllm
🏋️‍♂️ BenchLLM 🏋️‍♀️🦾 Continuous Integration for LLM powered applications 🦙🦅🤖BenchLLMis a Python-based open-source library that streamlines the testing of Large Language Models (LLMs) and AI-powered applications. It measures the accuracy of your model, agents, or chains by validating responses on any number of tests via LLMs.BenchLLM is actively used atV7for improving our LLM applications and is now Open Sourced under MIT License to share with the wider community💡 Get help onDiscordorTweet at usUse BenchLLM to:Test the responses of your LLM across any number of prompts.Continuous integration for chains likeLangchain, agents likeAutoGPT, or LLM models likeLlamaor GPT-4.Eliminate flaky chains and create confidence in your code.Spot inaccurate responses and hallucinations in your application at every version.⚠️NOTE:BenchLLM is in the early stage of development and will be subject to rapid changes.For bug reporting, feature requests, or contributions, please open an issue or submit a pull request (PR) on our GitHub page.🧪 BenchLLM Testing MethodologyBenchLLM implements a distinct two-step methodology for validating your machine learning models:Testing: This stage involves running your code against any number of expected responses and capturing the predictions produced by your model without immediate judgment or comparison.Evaluation: The recorded predictions are compared against the expected output using LLMs to verify factual similarity (or optionally manually). Detailed comparison reports, including pass/fail status and other metrics, are generated.This methodical separation offers a comprehensive view of your model's performance and allows for better control and refinement of each step.🚀 InstallTo install BenchLLM we use pippip install benchllm💻 UsageStart by importing the library and use the @benchllm.test decorator to mark the function you'd like to test:importbenchllm# Your custom model implementationdefrun_my_model(input):# Your model's logic goes here.returnsome_result@benchllm.test(suite="/path/to/test/suite")# If the tests are in the same directory, just use @benchllm.test.definvoke_model(input:str):returnrun_my_model(input)Next, prepare your tests. These are YAML/JSON files structured as follows:input: What's 1+1? Be very terse, only numeric output
expected:
  - 2
  - 2.0In the above example, theinputis the query or instruction that your model will process, andexpectedcontains the potential responses that your model should return. It's important to note thatinputcan be a simplestror a more complex nested dictionary; BenchLLM will extract the type of theinputargument in the Python code and load theinputfield from the YAML file accordingly.By default, BenchLLM uses OpenAI's GPT-3 model for thesemanticevaluator. This requires setting theOPENAI_API_KEYenvironment variable. If you do not want to use this default evaluator, you can specify an alternative one (discussed in further detail below):exportOPENAI_API_KEY='your-api-key'Replace 'your-api-key' with your actual OpenAI API key.To initiate testing, use thebench runcommand:$benchrunBy default, the bench run command looks for Python files implementing the @test decorator in the current directory. To target a specific file or folder, specify it directly:$benchrunpath/to/my/file.pyor/path/to/folder/with/filesThe--retry-countparameter allows BenchLLM to run a test multiple times, useful for models that may have variability in their outputs:$benchrun--retry-count5BenchLLM offers multiple evaluation methods to determine if the prediction matches the test case's expected values. You can use the--evaluatorparameter to specify the evaluation method:There are multiple ways to evaluate if the test functions prediction matches the test cases expected values.
By default GPT-3 is used to compare the output. You can use--evaluatorto use a different methodsemantic, checks semantic similarity using language models like GPT-3, GPT-3.5, or GPT-4 (--modelparameter). Please note, for this evaluator, you need to set theOPENAI_API_KEYenvironment variable.embedding, uses cosine distance between embedded vectors. Please note, for this evaluator, you need to set theOPENAI_API_KEYenvironment variable.string-match, checks if the strings are matching (case insensitive)interactive, user manually accepts or fails tests in the terminalweb, uses pywebio fora simple local web interfaceThe non interactive evaluators also supports--workers Nto run in the evaluations in parallel$benchrun--evaluatorstring-match--workers5To accelerate the evaluation process, BenchLLM uses a cache. If a (prediction, expected) pair has been evaluated in the past and a cache was used, the evaluation output will be saved for future evaluations. There are several types of caches:memory, only caches output values during the current run. This is particularly useful when running with--retry-count Nfile, stores the cache at the end of the run as a JSON file in output/cache.json. This is the default behavior.none, does not use any cache.$benchrunexamples--cachememoryWhen working on developing chains or training agent models, there may be instances where these models need to interact with external functions — for instance, querying a weather forecast or executing an SQL query. In such scenarios, BenchLLM facilitates the ability to mock these functions. This helps you make your tests more predictable and enables the discovery of unexpected function calls.input: I live in London, can I expect rain today?
expected: ["no"]
calls:
  - name: forecast.get_n_day_weather_forecast
    returns: It's sunny in London.
    arguments:
      location: London
      num_days: 1In the example above, the functionget_n_day_weather_forecastin theforecastmodule is mocked. In other words, every time this function is invoked, the model will receive"It's sunny in London". BenchLLM also provides warnings if the function is invoked with argument values different fromget_n_day_weather_forecast(location=London, num_days=1). Please note, the provision of these argument parameters is optional.🧮 EvalWhilebench runruns each test function and then evaluates their output, it can often be beneficial to separate these into two steps. For example, if you want a person to manually do the evaluation or if you want to try multiple evaluation methods on the same function.$benchrun--no-evalThis will generate json files inoutput/latest/predictionsThen later you can evaluate them with$benchevaloutput/latest/predictions🔌 APIFor more detailed control, BenchLLM provides an API.
You are not required to add YML/JSON tests to be able to evaluate your model.
You can instead:InstantiateTestobjectsUse aTesterobject to generate predictionsUse anEvaluatorobject to evaluate your modelfrombenchllmimportStringMatchEvaluator,Test,Tester# Instantiate your Test objectstests=[Test(input="What's 1+1?",expected=["2","It's 2"]),Test(input="First rule of fight club?",expected=["Do not talk about fight club"]),]# Use a Tester object to generate predictions using any test functionstester=Tester(my_test_function)tester.add_tests(tests)predictions=tester.run()# Use an Evaluator object to evaluate your modelevaluator=StringMatchEvaluator()evaluator.load(predictions)results=evaluator.run()print(results)If you want to incorporate caching and run multiple parallel evaluation jobs, you can modify your evaluator as follows:frombenchllm.cacheimportFileCache...evaluator=FileCache(StringMatchEvaluator(workers=2),Path("path/to/cache.json"))evaluator.load(predictions)results=evaluator.run()In this example,FileCacheis used to enable caching, and theworkersparameter ofStringMatchEvaluatoris set to2to allow for parallel evaluations. The cache results are saved in a file specified byPath("path/to/cache.json").☕️ Commandsbench add: Add a new test to a suite.bench tests: List all tests in a suite.bench run: Run all or target test suites.bench eval: Runs the evaluation of an existing test run.🙌 ContributeBenchLLM is developed for Python 3.10, although it may work with other Python versions as well. We recommend using a Python 3.10 environment and pip >= 23. You can use conda or any other environment manager to set up the environment:$condacreate--namebenchllmpython=3.10
$condaactivatebenchllm
$pipinstall-e".[dev]"To run all the examples first install the examples extra dependencies$pipinstall-e".[examples]"Contribution steps:Fork the repository.Create a new branch for your changes.Make your changes.Test your changes.Submit a pull request.We adhere to the PEP8 style guide. Please follow this guide when contributing.If you need any support, feel free to open an issue on our GitHub page.