Package: tonic-validate
Tonic Validate Evaluation FrameworkTonic Validate is a framework for the evaluation of LLM outputs, such as Retrieval Augmented Generation (RAG) pipelines. Validate makes it easy to evaluate, track, and monitor your LLM and RAG applications.  Install via pip and quickly embed Validate into your existing code.  More detailed documentation is availablehere. But the below quickstart guide is likely all you need to get started.QuickstartInstall Tonic Validatepip install tonic-validateUse the following code snippet to get started.fromtonic_validateimportValidateScorer,Benchmarkimportosos.environ["OPENAI_API_KEY"]="your-openai-key"# Function to simulate getting a response and context from your LLM# Replace this with your actual function calldefget_llm_response(question):return{"llm_answer":"Paris","llm_context_list":["Paris is the capital of France."]}benchmark=Benchmark(questions=["What is the capital of France?"],answers=["Paris"])# Score the responses for each question and answer pairscorer=ValidateScorer()run=scorer.score(benchmark,get_llm_response)This code snippet, creates a benchmark with one question and reference answer and then scores the answer.  Providing a reference answer is not required for most metrics (see below Metrics table).CI/CDMany users find value in running evaluations during the code review/pull request process.  You can create your own automation here using the snippet above and knowledge found in our documentation and this readme OR you can take advantage of our absolutely free Github Action in the Github Marketplace.  The listing ishere.  It's easy to setup but if you have any questions, just create an issue in the corresponding repository.Tonic Validate MetricsMetrics are used to score your LLM's performance.  Validate ships with 6 metrics which are applicable to most RAG systems.  You can create your own metrics as well by providing your own implementation ofmetric.py.  To compute a metric, you must provide it data from your RAG application.  The table below describes the available default metrics.Metric NameInputsScore RangeWhat does it measure?Answer similarity scoreQuestionReference answerLLM answer0 to 5How well the reference answer matches the LLM answer.Retrieval precisionQuestionRetrieved Context0 to 1Whether the context retrieved is relevant to answer the given question.Augmentation precisionQuestionRetrieved ContextLLM answer0 to 1Whether the relevant context is in the LLM answer.Augmentation accuracyRetrieved ContextLLM answer0 to 1Whether all the context is in the LLM answer.Answer consistencyRetrieved ContextLLM answer0 to 1Whether there is information in the LLM answer that does not come from the context.Answer consistency binaryRetrieved ContextLLM answer0 to 1Whether there is information in the LLM answer that does not come from the context.Note: More details on these metrics can be found in ourdocumentation.Metric InputsMetric inputs in Tonic Validate are used to provide the metrics with the information they need to calculate performance. Below, we explain each input type and how to pass them into Tonic Validate's SDK.QuestionWhat is it: The question askedHow to use: You can provide the questions by passing them into theBenchmarkvia thequestionsargument.fromtonic_validateimportBenchmarkbenchmark=Benchmark(questions=["What is the capital of France?","What is the capital of Germany?"])Reference AnswerWhat is it: A prewritten answer that serves as the ground truth for how the RAG application should answer the question.How to use: You can provide the reference answers by passing it into theBenchmarkvia theanswersargument. Each reference answer must correspond to a given question. So if the reference answer is for the third question in thequestionslist, then the reference answer must also be the third item in theanswerslist.  The only metric that requires a reference answer is the Answer Similarity Scorefromtonic_validateimportBenchmarkbenchmark=Benchmark(questions=["What is the capital of France?","What is the capital of Germany?"]answers=["Paris","Berlin"])LLM AnswerWhat is it: The answer the RAG application / LLM gives to the question.How to use: You can provide the LLM answer via the callback you provide to the Validate scorer. The answer is the first item in the tuple response.# Function to simulate getting a response and context from your LLM# Replace this with your actual function calldefget_rag_response(question):return{"llm_answer":"Paris","llm_context_list":["Paris is the capital of France."]}# Score the responsesscorer=ValidateScorer()run=scorer.score(benchmark,ask_rag)If you are manually logging the answers without using the callback, then you can provide the LLM answer viallm_answerwhen creating theLLMResponse.fromtonic_validateimportLLMResponse# Save the responses into an array for scoringresponses=[]foriteminbenchmark:# llm_answer is the answer that LLM givesllm_response=LLMResponse(llm_answer="Paris",benchmark_item=item)responses.append(llm_response)# Score the responsesscorer=ValidateScorer()run=scorer.score_responses(responses)Retrieved ContextWhat is it: The context that your RAG application retrieves when answering a given question. This context is what's put in the prompt by the RAG application to help the LLM answer the question.How to use: You can provide the LLM's retrieved context list via the callback you provide to the Validate scorer. The answer is the second item in the tuple response. The retrieved context is always a list# Function to simulate getting a response and context from your LLM# Replace this with your actual function calldefget_rag_response(question):return{"llm_answer":"Paris","llm_context_list":["Paris is the capital of France."]}# Score the responsesscorer=ValidateScorer()run=scorer.score(benchmark,ask_rag)If you are manually logging the answers, then you can provide the LLM context viallm_context_listwhen creating theLLMResponse.fromtonic_validateimportLLMResponse# Save the responses into an array for scoringresponses=[]foriteminbenchmark:# llm_answer is the answer that LLM gives# llm_context_list is a list of the context that the LLM used to answer the questionllm_response=LLMResponse(llm_answer="Paris",llm_context_list=["Paris is the capital of France."],benchmark_item=item)responses.append(llm_response)# Score the responsesscorer=ValidateScorer()run=scorer.score_responses(responses)Scoring With MetricsMost metrics are scored with the assistance of a LLM.  Validate supports OpenAI and Azure OpenAI but other LLMs can easily be integrated (just file an github issue against this repository).Important: Setting up OpenAI Key for ScoringIn order to use OpenAI you must provide an OpenAI API Key.importosos.environ["OPENAI_API_KEY"]="put-your-openai-api-key-here"If you already have theOPENAI_API_KEYset in your system's environment variables then you can skip this step. Otherwise, please set the environment variable before proceeding.Using AzureIf you are using Azure, instead of setting theOPENAI_API_KEYenvironment variable, you instead need to setAZURE_OPENAI_KEYandAZURE_OPENAI_ENDPOINT.AZURE_OPENAI_ENDPOINTis the endpoint url for your Azure OpenAI deployment andAZURE_OPENAI_KEYis your API key.importosos.environ["AZURE_OPENAI_KEY"]="put-your-azure-openai-api-key-here"os.environ["AZURE_OPENAI_ENDPOINT"]="put-your-azure-endpoint-here"Setting up the Tonic Validate ScorerTo use metrics, instantiate an instance of ValidateScorer.fromtonic_validateimportValidateScorerscorer=ValidateScorer()The default model used for scoring metrics is GPT 4 Turbo. To change the OpenAI model, pass the OpenAI model name into themodel_evaluatorargument forValidateScorer. You can also pass in custom metrics via an array of metrics.fromtonic_validateimportValidateScorerfromtonic_validate.metricsimportAnswerConsistencyMetric,AnswerSimilarityMetricscorer=ValidateScorer([AnswerConsistencyMetric(),AugmentationAccuracyMetric()],model_evaluator="gpt-3.5-turbo")If an error occurs while scoring an item's metric, the score for that metric will be set toNone. If you instead wish to have Tonic Validate throw an exception when there's an error scoring, then setfail_on_errortoTruein the constructorscorer=ValidateScorer(fail_on_error=True)Important: Using the scorer on AzureIf you are using Azure, you MUST set themodel_evaluatorargument to your deployment name like soscorer=ValidateScorer(model_evaluator="your-deployment-name")Running the ScorerAfter you instantiate theValidateScorerwith your desired metrics, you can then score the metrics using the callback you defined earlier.fromtonic_validateimportValidateScorer,ValidateApi# Function to simulate getting a response and context from your LLM# Replace this with your actual function calldefget_rag_response(question):return{"llm_answer":"Paris","llm_context_list":["Paris is the capital of France."]}# Score the responsesscorer=ValidateScorer()run=scorer.score(benchmark,ask_rag)Running the Scorer with manual loggingIf you don't want to use the callback, you can instead log your answers manually by iterating over the benchmark and then score the answers.fromtonic_validateimportValidateScorer,LLMResponse# Function to simulate getting a response and context from your LLM# Replace this with your actual function calldefget_rag_response(question):return{"llm_answer":"Paris","llm_context_list":["Paris is the capital of France."]}# Save the responses into an array for scoringresponses=[]foriteminbenchmark:rag_response=get_rag_response(item.question)llm_response=LLMResponse(llm_answer=rag_response["llm_answer"],llm_context_list=rag_response["llm_context_list"],benchmark_item=item)responses.append(llm_response)# Score the responsesscorer=ValidateScorer()run=scorer.score_responses(responses)Viewing the ResultsThere are two ways to view the results of a run.Option 1: Print Out the ResultsYou can manually print out the results via python like soprint("Overall Scores")print(run.overall_scores)print("------")foriteminrun.run_data:print("Question: ",item.reference_question)print("Answer: ",item.reference_answer)print("LLM Answer: ",item.llm_answer)print("LLM Context: ",item.llm_context)print("Scores: ",item.scores)print("------")which outputs the followingOverall Scores
{'answer_consistency': 1.0, 'augmentation_accuracy': 1.0}
------
Question:  What is the capital of France?
Answer:  Paris
LLM Answer:  Paris
LLM Context:  ['Paris is the capital of France.']
Scores:  {'answer_consistency': 1.0, 'augmentation_accuracy': 1.0}
------
Question:  What is the capital of Spain?
Answer:  Madrid
LLM Answer:  Paris
LLM Context:  ['Paris is the capital of France.']
Scores:  {'answer_consistency': 1.0, 'augmentation_accuracy': 1.0}
------Use the Tonic Validate UI (Recommended, Free to Use)You can easily view your run results by uploading them to ourfree to use UI. The main advantage of this method is the Tonic Validate UI provides graphing for your results along with additional visualization features. To sign up for the UI, go tohere.Once you sign up for the UI, you will go through an onboarding to create an API Key and Project.Copy both the API Key and Project ID from the onboarding and insert it into the following codefrom tonic_validate import ValidateApi
validate_api = ValidateApi("your-api-key")
validate_api.upload_run("your-project-id", run)This will upload your run to the Tonic Validate UI where you can view the results. On the home page (as seen below) you can view the change in scores across runs over time.You can also view the results of an individual run in the UI as well.TelemetryTonic Validate collects minimal telemetry to help us figure out what users want and how they're using the product. We do not use any existing telemetry framework and instead created our own privacy focused setup. Only the following information is trackedWhat metrics were used for a runNumber of questions in a runNumber of questions in a benchmarkWe doNOTtrack things such as the contents of the questions / answers, your scores, or any other sensitive information. For detecting CI/CD, we only check for common environment variables in different CI/CD environments. We do not log the values of these environment variables.We also generate a random UUID to help us figure out how many users are using the product. This UUID is linked to your Validate account only to help track who is using the SDK and UI at once and to get user counts. If you want to see how we implemented telemetry, you can do so in thetonic_validate/utils/telemetry.pyfileIf you wish to opt out of telemetry, you only need to set theTONIC_VALIDATE_DO_NOT_TRACKenvironment variable toTrue.FAQWhat models can I use an LLM evaluator?We currently allow the family of chat completion models from Open AI.This restriction makes it easy to follow the logic for the definition of the metrics in this package. It also ensures that this package does not depend on langchain, which also makes the logic of the package easier to follow.We'd like to add more models as choices for the LLM evaluator without adding to the complexity of the package too much.The default model used for scoring metrics is GPT 4 Turbo. To change the OpenAI model, pass the OpenAI model name into themodelargument forValidateScorerscorer=ValidateScorer([AnswerConsistencyMetric(),AugmentationAccuracyMetric()],model_evaluator="gpt-3.5-turbo")