Package: eval-to-go
Eval To GoA no-fuss eval tool for zapping through RAG and LLM evaluations!
It is a loose version of correctness evaluation in RAGAS framework.
Written for the busy folks like myself who are just too swamped to deal with the setup of professional eval tools.
As a resulteval_to_gois by no means a comprehensive, complete evaluation tool.
Instead, it's a simple way to get evaluations done in just a few lines of code, like when you need an eval in a PoC that was due yesterday.In other words, you may well encounter cases that are not covered.
However, it provides a foundation for evals based on another LLM acting as a judge, by modifying the prompt template or LLM output format, among other things.For serious projects, consider using more appropriate tools such as the RAGAS framework or enterprise implementations like H2O.ai's Eval Studio.Getting StartedInstallationGet up and running withEval To Go:pipinstalleval_to_goUsageEval To Gosupports both command-line and script-based interactions. Below are examples to kickstart your evaluation process:Input File FormatYour input dataset must be structured in a CSV format with specific columns:question: The question intended to be answered by your RAG or LLM.ground_truth: The main essence of the expected response, which could be a single word or a short phrase.answer: The actual response provided by your RAG or LLM to the question.Below is an example of how your input CSV file should be structured:questionground_truthanswerWhat was the actual value of the customer satisfaction in 2022?0.75The actual value of the customer satisfaction metric in 2022 was 0.83.What was the target value for the customer satisfaction metric in 2022?0.7The target value for the customer satisfaction metric in 2022 was 0.70.What is Firm GmbH's number of customer accounts as of 31 December 2022?97945As of 31 December 2022, Firm GmbH had a total of 97,945 customer accounts.What are Firm GmbH's values?Integrity, Transparency, Customer-Centricity, Sustainability, Social ResponsibilityFirm GmbH values Customer-Centricity, Integrity, Transparency, Sustainability, and Social Responsibility.Using OpenAICommand Linepython-meval_to_go.mainopenai--modelgpt-3.5-turbo-0125./examples/sample_data.csvoutput_data.csv--verboseIn a Python ScriptimportosimportpandasaspdfromtypingimportDict,Anyfromeval_to_go.modelsimportOpenAILLMfromeval_to_go.evaluationimportscore_correctness# Load your datadf=pd.read_csv("./sample_data.csv")# Define a callback to process resultsdefmy_callback(result:Dict[str,Any]):print(json.dumps(result,indent=4))# Configure the LLMopenai_llm=OpenAILLM(api_key=os.environ.get("OPENAI_API_KEY"),model="gpt-3.5-turbo-0125",)# Perform evaluationscore_correctness(df,openai_llm,inplace=True,callback=my_callback)# Review resultsprint(df)Evaluating with h2oGPT and Enterprise h2oGPT ModelsSimilar to the OpenAI example, you can useEval To Gofor evaluation using h2oGPT models as the judges. See theexamplesdir orrun_example.shfor the additional examples.Expected OutputAt completion of the evaluation, the input dataset will have two additional columns to provide insights into the evaluation results:eval_result: A Boolean value indicating whether the LLM's answer  matches the ground truth (True) or not (False).eval_reason: A brief rationale behind theeval_resultvalue.Here's how the output would look like for the sample dataset provided:questionground_truthanswereval_resulteval_reasonWhat was the actual value of the customer satisfaction in 2022?0.75The actual value of the customer satisfaction metric in 2022 was 0.83.FalseThe GPT model's response (0.83) does not match the Ground Truth (0.75).What was the target value for the customer satisfaction metric in 2022?0.7The target value for customer satisfaction metric in 2022 was 0.70.TrueAccurate and relevant.What is Firm GmbH's number of customer accounts as of 31 December 2022?9794597,945TrueAccurate and relevant.What are Firm GmbH's values?• Integrity • Transparency • Customer-Centricity • Sustainability • Social ResponsibilityFirm GmbH's values are Customer-Centricity, Integrity, Transparency, Sustainability, Social ResponsibilityTrueAccurate and relevant.ContributingContributions toEval To Goare welcomed, whether it's through bug reports, code enhancements, documentation improvements, or feature requests.LicenseEval To Gois made available under theMIT License. For more details, see the LICENSE file in the project repository.