Package: langcheck
Simple, Pythonic building blocks to evaluate LLM applications.Install•Examples•Quickstart•Docs•日本語InstallpipinstalllangcheckExamplesEvaluate TextUse LangCheck's suite of metrics to evaluate LLM-generated text.importlangcheck# Generate text with any LLM librarygenerated_outputs=['Black cat the','The black cat is sitting','The big black cat is sitting on the fence']# Check text quality and get results as a DataFrame (threshold is optional)langcheck.metrics.fluency(generated_outputs)>0.5It's easy to turn LangCheck metrics into unit tests, just useassert:assertlangcheck.metrics.fluency(generated_outputs)>0.5LangCheck includes several types of metrics to evaluate LLM applications. Some examples:Type of MetricExamplesLanguagesReference-Free Text Quality Metricstoxicity(generated_outputs)sentiment(generated_outputs)ai_disclaimer_similarity(generated_outputs)EN, JAReference-Based Text Quality Metricssemantic_similarity(generated_outputs, reference_outputs)rouge2(generated_outputs, reference_outputs)EN, JASource-Based Text Quality Metricsfactual_consistency(generated_outputs, sources)EN, JAText Structure Metricsis_float(generated_outputs, min=0, max=None)is_json_object(generated_outputs)All LanguagesVisualize MetricsLangCheck comes with built-in, interactive visualizations of metrics.# Choose some metricsfluency_values=langcheck.metrics.fluency(generated_outputs)sentiment_values=langcheck.metrics.sentiment(generated_outputs)# Interactive scatter plot of one metricfluency_values.scatter()# Interactive scatter plot of two metricslangcheck.plot.scatter(fluency_values,sentiment_values)# Interactive histogram of a single metricfluency_values.histogram()Augment DataText augmentations can automatically generate reworded prompts, typos, gender changes, and more to evaluate model robustness.For example, to measure how the model responds to different genders:male_prompts=langcheck.augment.gender(prompts,to_gender='male')female_prompts=langcheck.augment.gender(prompts,to_gender='female')male_generated_outputs=[my_llm_app(prompt)forpromptinmale_prompts]female_generated_outputs=[my_llm_app(prompt)forpromptinfemale_prompts]langcheck.metrics.sentiment(male_generated_outputs)langcheck.metrics.sentiment(female_generated_outputs)Unit TestingYou can write test cases for your LLM application using LangCheck metrics.For example, if you only have a list of prompts to test against:fromlangcheck.utilsimportload_json# Run the LLM application once to generate textprompts=load_json('test_prompts.json')generated_outputs=[my_llm_app(prompt)forpromptinprompts]# Unit testsdeftest_toxicity(generated_outputs):assertlangcheck.metrics.toxicity(generated_outputs)<0.1deftest_fluency(generated_outputs):assertlangcheck.metrics.fluency(generated_outputs)>0.9deftest_json_structure(generated_outputs):assertlangcheck.metrics.validation_fn(generated_outputs,lambdax:'myKey'injson.loads(x)).all()MonitoringYou can monitor the quality of your LLM outputs in production with LangCheck metrics.Just save the outputs and pass them into LangCheck.production_outputs=load_json('llm_logs_2023_10_02.json')['outputs']# Evaluate and display toxic outputs in production logslangcheck.metrics.toxicity(production_outputs)>0.75# Or if your app outputs structured textlangcheck.metrics.is_json_array(production_outputs)GuardrailsYou can provide guardrails on LLM outputs with LangCheck metrics.Just filter candidate outputs through LangCheck.# Get a candidate output from the LLM appraw_output=my_llm_app(random_user_prompt)# Filter the output before it reaches the userwhilelangcheck.metrics.contains_any_strings(raw_output,blacklist_words).any():raw_output=my_llm_app(random_user_prompt)