Package: crfm-helm
Holistic Evaluation of Language ModelsWelcome! Thecrfm-helmPython package contains code used in theHolistic Evaluation of Language Modelsproject (paper,website) byStanford CRFM. This package includes the following features:Collection of datasets in a standard format (e.g., NaturalQuestions)Collection of models accessible via a unified API (e.g., GPT-3, MT-NLG, OPT, BLOOM)Collection of metrics beyond accuracy (efficiency, bias, toxicity, etc.)Collection of perturbations for evaluating robustness and fairness (e.g., typos, dialect)Modular framework for constructing prompts from datasetsProxy server for managing accounts and providing unified interface to access modelsTo get started, refer tothe documentation on Read the Docsfor how to install and run the package.Directory StructureThe directory structure for this repo is as follows├── docs # MD used to generate readthedocs
│
├── scripts # Python utility scripts for HELM
│ ├── cache
│ ├── data_overlap # Calculate train test overlap
│ │ ├── common
│ │ ├── scenarios
│ │ └── test
│ ├── efficiency
│ ├── fact_completion
│ ├── offline_eval
│ └── scale
└── src
├── helm # Benchmarking Scripts for HELM
│ │
│ ├── benchmark # Main Python code for running HELM
│ │ │
│ │ └── static # Current JS (Jquery) code for rendering front-end
│ │ │
│ │ └── ...
│ │
│ ├── common # Additional Python code for running HELM
│ │
│ └── proxy # Python code for external web requests
│
└── helm-frontend # New React Front-endTutorialThis tutorial will explain how to use the HELM command line tools to run benchmarks, aggregate statistics, and visualize results.We will run two runs using themmluscenario on theopenai/gpt2model. Themmluscenario implements theMassive Multitask Language (MMLU)benchmark fromthis paper, and consists of a Question Answering (QA) task using a dataset with questions from 57 subjects such as elementary mathematics, US history, computer science, law, and more. Note that GPT-2 performs poorly on MMLU, so this is just a proof of concept. We will run two runs: the first using questions about anatomy, and the second using questions about philosophy.Usinghelm-runhelm-runis a command line tool for running benchmarks.To run this benchmark using the HELM command-line tools, we need to specifyrun spec descriptionsthat describes the desired runs. For this example, the run spec descriptions aremmlu:subject=anatomy,model=openai/gpt2(for anatomy) andmmlu:subject=philosophy,model=openai/gpt2(for philosophy).Next, we need to create arun spec configuration filecontining these run spec descriptions. A run spec configuration file is a text file containingRunEntriesserialized to JSON, where each entry inRunEntriescontains a run spec description. Thedescriptionfield of each entry should be arun spec description. Create a text file namedrun_specs.confwith the following contents:entries: [
  {description: "mmlu:subject=anatomy,model=openai/gpt2", priority: 1},
  {description: "mmlu:subject=philosophy,model=openai/gpt2", priority: 1},
]We will now usehelm-runto execute the runs that have been specified in this run spec configuration file. Run this command:helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 10The meaning of the additional arguments are as follows:--suitespecifies a subdirectory under the output directory in which all the output will be placed.--max-eval-instanceslimits evaluation to only the firstNinputs (i.e. instances) from the benchmark.helm-runcreates an environment directory environment and an output directory by default.The environment directory isprod_env/by default and can be set using--local-path. Credentials for making API calls should be added to acredentials.conffile in this directory.The output directory isbenchmark_output/by default and can be set using--output-path.After running this command, navigate to thebenchmark_output/runs/v1/directory. This should contain a two sub-directories namedmmlu:subject=anatomy,model=openai_gpt2andmmlu:subject=philosophy,model=openai_gpt2. Note that the names of these sub-directories is based on the run spec descriptions we used earlier, but with/replaced with_.Each output sub-directory will contain several JSON files that were generated during the corresponding run:run_spec.jsoncontains theRunSpec, which specifies the scenario, adapter and metrics for the run.scenario.jsoncontains a serializedScenario, which contains the scenario for the run and specifies the instances (i.e. inputs) used.scenario_state.jsoncontains a serializedScenarioState, which contains every request to and response from the model.per_instance_stats.jsoncontains a serialized list ofPerInstanceStats, which contains the statistics produced for the metrics for each instance (i.e. input).stats.jsoncontains a serialized list ofPerInstanceStats, which contains the statistics produced for the metrics, aggregated across all instances (i.e. inputs).helm-runprovides additional arguments that can be used to filter out--models-to-run,--groups-to-runand--priority. It can be convenient to create a largerun_specs.conffile containing every run spec description of interest, and then use these flags to filter down the RunSpecs to actually run. As an example, the mainrun_specs.conffile used for the HELM benchmarking paper can be foundhere.Usinghelm-summarizeThehelm-summarizereads the output files ofhelm-runand computes aggregate statistics across runs. Run the following:helm-summarize --suite v1This reads the pre-existing files inbenchmark_output/runs/v1/that were written byhelm-runpreviously, and writes the following new files back tobenchmark_output/runs/v1/:summary.jsoncontains a serializedExecutiveSummarywith a date and suite name.run_specs.jsoncontains the run spec descriptions for all the runs.runs.jsoncontains serialized list ofRun, which contains the run path, run spec and adapter spec and statistics for each run.groups.jsoncontains a serialized list ofTable, each containing information about groups in a group category.groups_metadata.jsoncontains a list of all the groups along with a human-readable description and a taxonomy.Additionally, for each group and group-relavent metric, it will output a pair of files:benchmark_output/runs/v1/groups/latex/<group_name>_<metric_name>.texandbenchmark_output/runs/v1/groups/json/<group_name>_<metric_name>.json. These files contain the statistics for that metric from each run within the group.Usinghelm-serverFinally, thehelm-servercommand launches a web server to visualize the output files ofhelm-runandhelm-benchmark. Run:helm-serverOpen a browser and go tohttp://localhost:8000/to view the visualization. You should see a similar view aslive website for the paper, but for the data from your benchmark runs. The website has three main sections:Modelscontains a list of available models.Scenarioscontains a list of available scenarios.Resultscontains results from the runs, organized into groups and categories of groups.Raw Runscontains a searchable list of runs.Other TipsThe suite name can be used as a versioning mechanism to separate runs using different versions of scenarios or models.Tools such asjqare useful for examining the JSON output files on the command line.