Package: continuous-eval
Open-Source Evaluation for GenAI Application PipelinesOverviewcontinuous-evalis an open-source package created for granular and holistic evaluation of GenAI application pipelines.How is continuous-eval different?Modularized Evaluation: Measure each module in the pipeline with tailored metrics.Comprehensive Metric Library: Covers Retrieval-Augmented Generation (RAG), Code Generation, Agent Tool Use, Classification and a variety of other LLM use cases. Mix and match Deterministic, Semantic and LLM-based metrics.Leverage User Feedback in Evaluation: Easily build a close-to-human ensemble evaluation pipeline with mathematical guarantees.Synthetic Dataset Generation: Generate large-scale synthetic dataset to test your pipeline.Getting StartedThis code is provided as a PyPi package. To install it, run the following command:python3-mpipinstallcontinuous-evalif you want to install from source:gitclonehttps://github.com/relari-ai/continuous-eval.git&&cdcontinuous-eval
poetryinstall--all-extrasTo run LLM-based metrics, the code requires at least one of the LLM API keys in.env. Take a look at the example env file.env.example.Run a single metricHere's how you run a single metric on a datum.
Check all available metrics here:linkfromcontinuous_eval.metrics.retrievalimportPrecisionRecallF1datum={"question":"What is the capital of France?","retrieved_context":["Paris is the capital of France and its largest city.","Lyon is a major city in France.",],"ground_truth_context":["Paris is the capital of France."],"answer":"Paris","ground_truths":["Paris"],}metric=PrecisionRecallF1()print(metric(**datum))Off-the-shelf MetricsModuleCategoryMetricsRetrievalDeterministicPrecisionRecallF1, RankedRetrievalMetricsLLM-basedLLMBasedContextPrecision, LLMBasedContextCoverageText GenerationDeterministicDeterministicAnswerCorrectness, DeterministicFaithfulness, FleschKincaidReadabilitySemanticDebertaAnswerScores, BertAnswerRelevance, BertAnswerSimilarityLLM-basedLLMBasedFaithfulness, LLMBasedAnswerCorrectness, LLMBasedAnswerRelevance, LLMBasedStyleConsistencyClassificationDeterministicClassificationAccuracyCode GenerationDeterministicCodeStringMatch, PythonASTSimilarityLLM-basedLLMBasedCodeGenerationAgent ToolsDeterministicToolSelectionAccuracyCustomDefine your own metricsTo define your own metrics, you only need to extend theMetricclass implementing the__call__method.
Optional methods arebatch(if it is possible to implement optimizations for batch processing) andaggregate(to aggregate metrics results over multiple samples_).Run evaluation on pipeline modulesDefine modules in your pipeline and select corresponding metrics.fromcontinuous_eval.evalimportModule,ModuleOutput,Pipeline,Datasetfromcontinuous_eval.metrics.retrievalimportPrecisionRecallF1,RankedRetrievalMetricsfromcontinuous_eval.metrics.generation.textimportDeterministicAnswerCorrectnessfromtypingimportList,Dictdataset=Dataset("dataset_folder")# Simple 3-step RAG pipeline with Retriever->Reranker->Generationretriever=Module(name="Retriever",input=dataset.question,output=List[str],eval=[PrecisionRecallF1().use(retrieved_context=ModuleOutput(),ground_truth_context=dataset.ground_truth_context,),],)reranker=Module(name="reranker",input=retriever,output=List[Dict[str,str]],eval=[RankedRetrievalMetrics().use(retrieved_context=ModuleOutput(),ground_truth_context=dataset.ground_truth_context,),],)llm=Module(name="answer_generator",input=reranker,output=str,eval=[FleschKincaidReadability().use(answer=ModuleOutput()),DeterministicAnswerCorrectness().use(answer=ModuleOutput(),ground_truth_answers=dataset.ground_truths),],)pipeline=Pipeline([retriever,reranker,llm],dataset=dataset)print(pipeline.graph_repr())# optional: visualize the pipelineNow you can run the evaluation on your pipelineeval_manager.start_run()whileeval_manager.is_running():ifeval_manager.curr_sampleisNone:breakq=eval_manager.curr_sample["question"]# get the question or any other field# run your pipeline ...eval_manager.next_sample()Tologthe results you just need to call theeval_manager.logmethod with the module name and the output, for example:eval_manager.log("answer_generator",response)The evaluator manager also offerseval_manager.run_metrics()to run all the metrics defined in the pipelineeval_manager.run_tests()to run the tests defined in the pipeline (see the documentationdocsfor more details)ResourcesDocs:linkExamples Repo:end-to-end example repoBlog Posts:Practical Guide to RAG Pipeline Evaluation:Part 1: RetrievalPractical Guide to RAG Pipeline Evaluation:Part 2: GenerationHow important is a Golden Dataset for LLM evaluation?linkDiscord:Join our community of LLM developersDiscordReach out to founders:EmailorSchedule a chatLicenseThis project is licensed under the Apache 2.0 - see theLICENSEfile for details.Open AnalyticsWe monitor basic anonymous usage statistics to understand our users' preferences, inform new features, and identify areas that might need improvement.
You can take a look at exactly what we track in thetelemetry codeTo disable usage-tracking you set theCONTINUOUS_EVAL_DO_NOT_TRACKflag totrue.