Package: judyeval
JudyJudy is a python library and framework to evaluate the text-generation capabilities of Large Language Models (LLM) using a Judge LLM.Judy allows users to evaluate LLMs using a competent Judge LLM (such as GPT-4). Users can choose from a set of predefined scenarios sourced from recent research, or design their own. A scenario is a specific test designed to evaluate a particular aspect of an LLM. A scenario consists of:Dataset: A source dataset to generate prompts to evaluate models against.Task: A task to evaluate models on. Tasks for judge evaluations have been carefully designed by researchers to assess certain aspects of LLMs.Metric: The metric(s) to use when evaluating the responses from a task. For example - accuracy, level of detail etc.Judy has been inspired by techniques used in research including HELM [1] and LLM-as-a-judge [2].[1] Holistic Evaluation of Language Models -https://arxiv.org/abs/2211.09110[2] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena -https://arxiv.org/abs/2306.05685InstallationUse the package managerpipto install Judy. Note: Judy requires python >= 3.10.pipinstalljudyevalAlternate InstallationYou can also install Judy directly from this git repo:pipinstallgit+https://github.com/TNT-Hoopsnake/judyGetting StartedSetup configsJudy uses 3 configuration files during evaluation. Only the run config is strictly necessary to begin with:Dataset Config: Defines all of the datasets available to use in the evaluation run, how to download them and which class to use to format them.You don't have to worry about specifying this config unless you plan on adding new datasets.Judywill automatically use the example dataset confighereunless you specify an alternate one using--dataset-config.Evaluation Config: Defines all of the tasks and the metrics used to evaluate them. It also restricts which datasets and metrics can be used for each task.You don't have to worry about specifying this config unless you plan on adding new tasks or metrics.Judywill automatically use the example eval confighereunless you specify an alternate one using--eval-config.Run Config: Defines all of the settings to use for your evaluation run. The evaluation results for your run will store a copy (with sensitive details redacted) of these settings as metadata. An example run config is providedhereSetup model(s) to evaluateEnsure you have API access to the models you wish to evaluate. We currently support two types of API formats:OPENAI: The OpenAI API ChatCompletion endpoint (ref)HUGGINGFACE: The HuggingFace Hosted Inference API (ref)If you are hosting models locally you can use a package likeLocalAIto get an OpenAI compatible REST API which can be used byJudy.Judy CommandsA CLI interface is provided for viewing and editing Judy config files.judyconfigRun an evaluation as follows:judyrun--run-configrun_config.yml--namedisinfo-test--output./resultsAfter running an evaluation, you can serve a web app for viewing the results:judyserve-r./resultsWeb App ScreenshotsThe web app allows you to view your evaluation results.RoadmapFeaturesCore frameworkWeb app - to view evaluation resultsAdd perturbations - the ability to modify input datasets - with typos, synonymns etc.Add adaptations - the ability to use different prompting techniques - such as Chain of Thought etc.ScenariosFLASKCode ComprehensionEmotional IntelligenceRED-INSTRUCTContributingPull requests are welcome. For major changes, please open an issue first to discuss what you would like to change. Please make sure to update tests as appropriate. Check out thecontribution guidefor more details.Citation - BibTeX@software{Hutchinson_Judy_-_LLM_2024,
  author = {Hutchinson, Linden and Raghavan, Rahul},
  month = feb,
  title = {{Judy - LLM Evaluator}},
  url = {https://github.com/TNT-Hoopsnake/judy},
  version = {2.0.0},
  year = {2024}
}