Package: autoevals
AutoevalsAutoevals is a tool to quickly and easily evaluate AI model outputs.It bundles together a variety of automatic evaluation methods including:Heuristic (e.g. Levenshtein distance)Statistical (e.g. BLEU)Model-based (using LLMs)Autoevals is developed by the team atBrainTrust.Autoevals uses model-graded evaluation for a variety of subjective tasks including fact checking,
safety, and more. Many of these evaluations are adapted from OpenAI's excellentevalsproject but are implemented so you can flexibly run them on individual examples, tweak the prompts, and debug
their outputs.You can also create your own model-graded evaluations with Autoevals. It's easy to add custom prompts, parse outputs,
and manage exceptions.InstallationAutoevals is distributed as aPython library on PyPIandNode.js library on NPM.PythonpipinstallautoevalsNode.jsnpminstallautoevalsExampleUse Autoevals to model-grade an example LLM completion using thefactuality prompt.
By default, Autoevals uses yourOPENAI_API_KEYenvironment variable to authenticate with OpenAI's API.Pythonfromautoevals.llmimport*# Create a new LLM-based evaluatorevaluator=Factuality()# Evaluate an example LLM completioninput="Which country has the highest population?"output="People's Republic of China"expected="China"result=evaluator(output,expected,input=input)# The evaluator returns a score from [0,1] and includes the raw outputs from the evaluatorprint(f"Factuality score:{result.score}")print(f"Factuality metadata:{result.metadata['rationale']}")Node.jsimport{Factuality}from"autoevals";(async()=>{constinput="Which country has the highest population?";constoutput="People's Republic of China";constexpected="China";constresult=awaitFactuality({output,expected,input});console.log(`Factuality score:${result.score}`);console.log(`Factuality metadata:${result.metadata.rationale}`);})();Using Braintrust with AutoevalsOnce you grade an output using Autoevals, it's convenient to useBrainTrustto log and compare your evaluation results.Pythonfromautoevals.llmimport*importbraintrust# Create a new LLM-based evaluatorevaluator=Factuality()# Evaluate an example LLM completioninput="Which country has the highest population?"output="People's Republic of China"expected="China"result=evaluator(output,expected,input=input)# The evaluator returns a score from [0,1] and includes the raw outputs from the evaluatorprint(f"Factuality score:{result.score}")print(f"Factuality metadata:{result.metadata['rationale']}")# Log the evaluation results to BrainTrustexperiment=braintrust.init(project="Autoevals",api_key="YOUR_BRAINTRUST_API_KEY")experiment.log(inputs={"query":input},output=output,expected=expected,scores={"factuality":result.score,},metadata={"factuality":result.metadata,},)print(experiment.summarize())Node.jsCreate a file namedexample.eval.js(it must end with.eval.jsor.eval.js):import{Eval}from"braintrust";import{Factuality}from"autoevals";Eval("Autoevals",{data:()=>[{input:"Which country has the highest population?",expected:"China",},],task:()=>"People's Republic of China",scores:[Factuality],});Then, runnpxbraintrustrunexample.eval.jsSupported Evaluation MethodsModel-Based ClassificationBattleClosedQAHumorFactualitySecuritySummarizationSQLTranslationFine-tuned binary classifiersEmbeddingsEmbedding similarityBERTScoreHeuristicLevenshtein distanceNumeric differenceJSON diffJaccard distanceStatisticalBLEUROUGEMETEORCustom Evaluation PromptsAutoevals supports custom evaluation prompts for model-graded evaluation. To use them, simply pass in a prompt and scoring mechanism:PythonfromautoevalsimportLLMClassifier# Define a prompt prefix for a LLMClassifier (returns just one answer)prompt_prefix="""You are a technical project manager who helps software engineers generate better titles for their GitHub issues.You will look at the issue description, and pick which of two titles better describes it.I'm going to provide you with the issue description, and two possible titles.Issue Description: {{input}}1: {{output}}2: {{expected}}"""# Define the scoring mechanism# 1 if the generated answer is better than the expected answer# 0 otherwiseoutput_scores={"1":1,"2":0}evaluator=LLMClassifier(name="TitleQuality",prompt_template=prompt_prefix,choice_scores=output_scores,use_cot=True,)# Evaluate an example LLM completionpage_content="""As suggested by Nicolo, we should standardize the error responses coming from GoTrue, postgres, and realtime (and any other/future APIs) so that it's better DX when writing a client,We can make this change on the servers themselves, but since postgrest and gotrue are fully/partially external may be harder to change, it might be an option to transform the errors within the client libraries/supabase-js, could be messy?Nicolo also dropped this as a reference: http://spec.openapis.org/oas/v3.0.3#openapi-specification"""output=("Standardize error responses from GoTrue, Postgres, and Realtime APIs for better DX")expected="Standardize Error Responses across APIs"response=evaluator(output,expected,input=page_content)print(f"Score:{response.score}")print(f"Metadata:{response.metadata}")Node.jsimport{LLMClassifierFromTemplate}from"autoevals";(async()=>{constpromptTemplate=`You are a technical project manager who helps software engineers generate better titles for their GitHub issues.You will look at the issue description, and pick which of two titles better describes it.I'm going to provide you with the issue description, and two possible titles.Issue Description: {{input}}1: {{output}}2: {{expected}}`;constchoiceScores={1:1,2:0};constevaluator=LLMClassifierFromTemplate({name:"TitleQuality",promptTemplate,choiceScores,useCoT:true,});constinput=`As suggested by Nicolo, we should standardize the error responses coming from GoTrue, postgres, and realtime (and any other/future APIs) so that it's better DX when writing a client,We can make this change on the servers themselves, but since postgrest and gotrue are fully/partially external may be harder to change, it might be an option to transform the errors within the client libraries/supabase-js, could be messy?Nicolo also dropped this as a reference: http://spec.openapis.org/oas/v3.0.3#openapi-specification`;constoutput=`Standardize error responses from GoTrue, Postgres, and Realtime APIs for better DX`;constexpected=`Standardize Error Responses across APIs`;constresponse=awaitevaluator({input,output,expected});console.log("Score",response.score);console.log("Metadata",response.metadata);})();Creating custom scorersYou can also create your own scoring functions that do not use LLMs. For example, to test whether the word'banana'is in the output, you can use the following:PythonfromautoevalsimportScoredefbanana_scorer(output,expected,input):returnScore(name="banana_scorer",score=1if"banana"inoutputelse0)input="What is 1 banana + 2 bananas?"output="3"expected="3 bananas"result=banana_scorer(output,expected,input)print(f"Banana score:{result.score}")Node.jsimport{Score}from"autoevals";constbananaScorer=({output,expected,input}):Score=>{return{name:"banana_scorer",score:output.includes("banana")?1:0};};(async()=>{constinput="What is 1 banana + 2 bananas?";constoutput="3";constexpected="3 bananas";constresult=awaitbananaScorer({output,expected,input});console.log(`Banana score:${result.score}`);})();Why does this library exist?There is nothing particularly novel about the evaluation methods in this library. They are all well-known and well-documented. However, there are a few things that are particularly difficult when evaluating in practice:Normalizing metrics between 0 and 1 is tough. For example, check out the calculation innumber.pyto see how it's done for numeric differences.Parsing the outputs on model-graded evaluations is also challenging. There are frameworks that do this, but it's hard to
debug one output at a time, propagate errors, and tweak the prompts. Autoevals makes these tasks easy.Collecting metrics behind a uniform interface makes it easy to swap out evaluation methods and compare them. Prior to Autoevals, we couldn't find an open source library where you can simply pass ininput,output, andexpectedvalues through a bunch of different evaluation methods.DocumentationThe full docs are availablehere.