Package: lm-eval
Language Model Evaluation HarnessAnnouncementA new v0.4.0 release of lm-evaluation-harness is available!New updates and features include:Internal refactoringConfig-based task creation and configurationEasier import and sharing of externally-defined task config YAMLsSupport for Jinja2 prompt design, easy modification of prompts + prompt imports from PromptsourceMore advanced configuration options, including output post-processing, answer extraction, and multiple LM generations per document, configurable fewshot settings, and moreSpeedups and new modeling libraries supported, including: faster data-parallel HF model usage, vLLM support, MPS support with HuggingFace, and moreLogging and usability changesNew tasks including CoT BIG-Bench-Hard, Belebele, user-defined task groupings, and morePlease see our updated documentation pages indocs/for more details.Development will be continuing on themainbranch, and we encourage you to give us feedback on what features are desired and how to improve the library further, or ask questions, either in issues or PRs on GitHub, or in theEleutherAI discord!OverviewThis project provides a unified framework to test generative language models on a large number of different evaluation tasks.Features:Over 60 standard academic benchmarks for LLMs, with hundreds of subtasks and variants implemented.Support for models loaded viatransformers(including quantization viaAutoGPTQ),GPT-NeoX, andMegatron-DeepSpeed, with a flexible tokenization-agnostic interface.Support for fast and memory-efficient inference withvLLM.Support for commercial APIs includingOpenAI, andTextSynth.Support for evaluation on adapters (e.g. LoRA) supported inHuggingFace's PEFT library.Support for local models and benchmarks.Evaluation with publicly available prompts ensures reproducibility and comparability between papers.Easy support for custom prompts and evaluation metrics.The Language Model Evaluation Harness is the backend for 🤗 Hugging Face's popularOpen LLM Leaderboard, has been used inhundreds of papers, and is used internally by dozens of organizations including NVIDIA, Cohere, BigScience, BigCode, Nous Research, and Mosaic ML.InstallTo install thelm-evalpackage from the github repository, run:gitclonehttps://github.com/EleutherAI/lm-evaluation-harnesscdlm-evaluation-harness
pipinstall-e.We also provide a number of optional dependencies for extended functionality. Extras can be installed viapip install -e ".[NAME]"NameUseanthropicFor using Anthropic's modelsdevFor linting PRs and contributionsgptqFor loading models with GPTQifevalFor running the IFEval taskmambaFor loading Mamba SSM modelsmathFor running math task answer checkingmultilingualFor multilingual tokenizersopenaiFor using OpenAI's modelsoptimumFor running Intel OpenVINO modelspromptsourceFor using PromptSource promptssentencepieceFor using the sentencepiece tokenizertestingFor running library test suitevllmFor loading models with vLLMzenoFor visualizing results with Zeno------------------------------------------------------allLoads all extras (not recommended)Basic UsageHugging FacetransformersTo evaluate a model hosted on theHuggingFace Hub(e.g. GPT-J-6B) onhellaswagyou can use the following command (this assumes you are using a CUDA-compatible GPU):lm_eval--modelhf\--model_argspretrained=EleutherAI/gpt-j-6B\--taskshellaswag\--devicecuda:0\--batch_size8Additional arguments can be provided to the model constructor using the--model_argsflag. Most notably, this supports the common practice of using therevisionsfeature on the Hub to store partially trained checkpoints, or to specify the datatype for running a model:lm_eval--modelhf\--model_argspretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float"\--taskslambada_openai,hellaswag\--devicecuda:0\--batch_size8Models that are loaded via bothtransformers.AutoModelForCausalLM(autoregressive, decoder-only GPT style models) andtransformers.AutoModelForSeq2SeqLM(such as encoder-decoder models like T5) in Huggingface are supported.Batch size selection can be automated by setting the--batch_sizeflag toauto. This will perform automatic detection of the largest batch size that will fit on your device. On tasks where there is a large difference between the longest and shortest example, it can be helpful to periodically recompute the largest batch size, to gain a further speedup. To do this, append:Nto above flag to automatically recompute the largest batch sizeNtimes. For example, to recompute the batch size 4 times, the command would be:lm_eval--modelhf\--model_argspretrained=EleutherAI/pythia-160m,revision=step100000,dtype="float"\--taskslambada_openai,hellaswag\--devicecuda:0\--batch_sizeauto:4The full list of supported arguments are providedhere, and on the terminal by callinglm_eval -h. Alternatively, you can uselm-evalinstead oflm_eval.[!Note]
Just like you can provide a local path totransformers.AutoModel, you can also provide a local path tolm_evalvia--model_args pretrained=/path/to/modelMulti-GPU Evaluation with Hugging FaceaccelerateWe support two main ways of using Hugging Face'saccelerate 🚀library for multi-GPU evaluation.To performdata-parallel evaluation(where each GPU loads aseparate full copyof the model), we leverage theacceleratelauncher as follows:accelerate launch -m lm_eval --model hf \
    --tasks lambada_openai,arc_easy \
    --batch_size 16(or viaaccelerate launch --no-python lm_eval).For cases where your model can fit on a single GPU, this allows you to evaluate on K GPUs K times faster than on one.WARNING: This setup does not work with FSDP model sharding, so inaccelerate configFSDP must be disabled, or the NO_SHARD FSDP option must be used.The second way of usingacceleratefor multi-GPU evaluation is when your model istoo large to fit on a single GPU.In this setting, run the libraryoutside of theacceleratelauncher, but passingparallelize=Trueto--model_argsas follows:lm_eval --model hf \
    --tasks lambada_openai,arc_easy \
    --model_args parallelize=True \
    --batch_size 16This means that your model's weights will be split across all available GPUs.For more advanced users or even larger models, we allow for the following arguments whenparallelize=Trueas well:device_map_option: How to split model weights across available GPUs. defaults to "auto".max_memory_per_gpu: the max GPU memory to use per GPU in loading the model.max_cpu_memory: the max amount of CPU memory to use when offloading the model weights to RAM.offload_folder: a folder where model weights will be offloaded to disk if needed.These two options (accelerate launchandparallelize=True) are mutually exclusive.Tensor + Data Parallel and Optimized Inference withvLLMWe also support vLLM for faster inference onsupported model types, especially faster when splitting a model across multiple GPUs. For single-GPU or multi-GPU — tensor parallel, data parallel, or a combination of both — inference, for example:lm_eval--modelvllm\--model_argspretrained={model_name},tensor_parallel_size={GPUs_per_model},dtype=auto,gpu_memory_utilization=0.8,data_parallel_size={model_replicas}\--taskslambada_openai\--batch_sizeautoFor a full list of supported vLLM configurations, please reference our vLLM integration and the vLLM documentation.vLLM occasionally differs in output from Huggingface. We treat Huggingface as the reference implementation, and provide ascriptfor checking the validity of vllm results against HF.Model APIs and Inference ServersOur library also supports the evaluation of models served via several commercial APIs, and we hope to implement support for the most commonly used performant local/self-hosted inference servers.To call a hosted model, use:exportOPENAI_API_KEY=YOUR_KEY_HERE
lm_eval--modelopenai-completions\--model_argsmodel=davinci\--taskslambada_openai,hellaswagWe also support using your own local inference server with servers that mirror the OpenAI Completions and ChatCompletions APIs.lm_eval--modellocal-chat-completions--tasksgsm8k--model_argsmodel=facebook/opt-125m,base_url=http://{yourip}:8000/v1Note that for externally hosted models, configs such as--deviceand--batch_sizeshould not be used and do not function. Just like you can use--model_argsto pass arbitrary arguments to the model constructor for local models, you can use it to pass arbitrary arguments to the model API for hosted models. See the documentation of the hosting service for information on what arguments they support.API or Inference ServerImplemented?--model <xxx>nameModels supported:Request Types:OpenAI Completions:heavy_check_mark:openai-completions,local-completionsAll OpenAI Completions API modelsgenerate_until,loglikelihood,loglikelihood_rollingOpenAI ChatCompletions:heavy_check_mark:openai-chat-completions,local-chat-completionsAll ChatCompletions API modelsgenerate_until(no logprobs)Anthropic:heavy_check_mark:anthropicSupported Anthropic Enginesgenerate_until(no logprobs)Textsynth:heavy_check_mark:textsynthAll supported enginesgenerate_until,loglikelihood,loglikelihood_rollingCohere:hourglass: - blocked on Cohere API bugN/AAllcohere.generate()enginesgenerate_until,loglikelihood,loglikelihood_rollingLlama.cpp(viallama-cpp-python):heavy_check_mark:gguf,ggmlAll models supported by llama.cppgenerate_until,loglikelihood, (perplexity evaluation not yet implemented)vLLM:heavy_check_mark:vllmMost HF Causal Language Modelsgenerate_until,loglikelihood,loglikelihood_rollingMamba:heavy_check_mark:mamba_ssmMamba architecture Language Models via themamba_ssmpackagegenerate_until,loglikelihood,loglikelihood_rollingHuggingface Optimum (Causal LMs)✔️openvinoAny decoder-only AutoModelForCausalLM converted with Huggingface Optimum into OpenVINO™ Intermediate Representation (IR) formatgenerate_until,loglikelihood,loglikelihood_rollingYour local inference server!:heavy_check_mark:local-completionsorlocal-chat-completions(usingopenai-chat-completionsmodel type)Any server address that accepts GET requests using HF models and mirror's OpenAI's Completions or ChatCompletions interfacegenerate_untilModels which do not supply logits or logprobs can be used with tasks of typegenerate_untilonly, while local models, or APIs that supply logprobs/logits of their prompts, can be run on all task types:generate_until,loglikelihood,loglikelihood_rolling, andmultiple_choice.For more information on the different taskoutput_typesand model request types, seeour documentation.Other FrameworksA number of other libraries contain scripts for calling the eval harness through their library. These includeGPT-NeoX,Megatron-DeepSpeed, andmesh-transformer-jax.To create your own custom integration you can follow instructions fromthis tutorial.Additional FeaturesIf you have a Metal compatible Mac, you can run the eval harness using the MPS back-end by replacing--device cuda:0with--device mps(requires PyTorch version 2.1 or higher).[!Note]
You can inspect what the LM inputs look like by running the following command:pythonwrite_out.py\--tasksall_tasks\--num_fewshot5\--num_examples10\--output_base_path/path/to/output/folderThis will write out one text file for each task.To verify the data integrity of the tasks you're performing in addition to running the tasks themselves, you can use the--check_integrityflag:lm_eval--modelopenai\--model_argsengine=davinci\--taskslambada_openai,hellaswag\--check_integrityAdvanced Usage TipsFor models loaded with the HuggingFacetransformerslibrary, any arguments provided via--model_argsget passed to the relevant constructor directly. This means that anything you can do withAutoModelcan be done with our library. For example, you can pass a local path viapretrained=or use models finetuned withPEFTby taking the call you would run to evaluate the base model and add,peft=PATHto themodel_argsargument:lm_eval--modelhf\--model_argspretrained=EleutherAI/gpt-j-6b,parallelize=True,load_in_4bit=True,peft=nomic-ai/gpt4all-j-lora\--tasksopenbookqa,arc_easy,winogrande,hellaswag,arc_challenge,piqa,boolq\--devicecuda:0GPTQquantized models can be loaded by specifying their file names in,autogptq=NAME(or,autogptq=Truefor default names) in themodel_argsargument:lm_eval--modelhf\--model_argspretrained=model-name-or-path,autogptq=model.safetensors,gptq_use_triton=True\--taskshellaswagWe support wildcards in task names, for example you can run all of the machine-translated lambada tasks via--task lambada_openai_mt_*.To save evaluation results provide an--output_path. We also support logging model responses with the--log_samplesflag for post-hoc analysis.Additionally, one can provide a directory with--use_cacheto cache the results of prior runs. This allows you to avoid repeated execution of the same (model, task) pairs for re-scoring.For a full list of supported arguments, check out theinterfaceguide in our documentation![!Tip]
Running lm-evaluation-harness as an external library and can't find (almost) any tasks available? runlm_eval.tasks.initialize_tasks()to load the library's stock tasks before callinglm_eval.evaluate()orlm_eval.simple_evaluate()!Visualizing ResultsYou can useZenoto visualize the results of your eval harness runs.First, head tohub.zenoml.comto create an account and get an API keyon your account page.
Add this key as an environment variable:exportZENO_API_KEY=[yourapikey]You'll also need to install thelm_eval[zeno]package extra.To visualize the results, run the eval harness with thelog_samplesandoutput_pathflags.
We expectoutput_pathto contain multiple folders that represent individual model names.
You can thus run your evaluation on any number of tasks and models and upload all of the results as projects on Zeno.lm_eval\--modelhf\--model_argspretrained=EleutherAI/gpt-j-6B\--taskshellaswag\--devicecuda:0\--batch_size8\--log_samples\--output_pathoutput/gpt-j-6BThen, you can upload the resulting data using thezeno_visualizescript:pythonscripts/zeno_visualize.py\--data_pathoutput\--project_name"Eleuther Project"This will use all subfolders indata_pathas different models and upload all tasks within these model folders to Zeno.
If you run the eval harness on multiple tasks, theproject_namewill be used as a prefix and one project will be created per task.You can find an example of this workflow inexamples/visualize-zeno.ipynb.How to Contribute or Learn More?For more information on the library and how everything fits together, check out all of ourdocumentation pages! We plan to post a larger roadmap of desired + planned library improvements soon, with more information on how contributors can help.Implementing new tasksTo implement a new task in the eval harness, seethis guide.In general, we follow this priority list for addressing concerns about prompting and other eval details:If there is widespread agreement among people who train LLMs, use the agreed upon procedure.If there is a clear and unambiguous official implementation, use that procedure.If there is widespread agreement among people who evaluate LLMs, use the agreed upon procedure.If there are multiple common implementations but not universal or widespread agreement, use our preferred option among the common implementations. As before, prioritize choosing from among the implementations found in LLM training papers.These are guidelines and not rules, and can be overruled in special circumstances.We try to prioritize agreement with the procedures used by other groups to decrease the harm when people inevitably compare runs across different papers despite our discouragement of the practice. Historically, we also prioritized the implementation fromLanguage Models are Few Shot Learnersas our original goal was specifically to compare results with that paper.SupportThe best way to get support is to open an issue on this repo or join theEleutherAI Discord server. The#lm-thunderdomechannel is dedicated to developing this project and the#release-discussionchannel is for receiving support for our releases. If you've used the library and have had a positive (or negative) experience, we'd love to hear from you!Cite as@misc{eval-harness,
  author       = {Gao, Leo and Tow, Jonathan and Abbasi, Baber and Biderman, Stella and Black, Sid and DiPofi, Anthony and Foster, Charles and Golding, Laurence and Hsu, Jeffrey and Le Noac'h, Alain and Li, Haonan and McDonell, Kyle and Muennighoff, Niklas and Ociepa, Chris and Phang, Jason and Reynolds, Laria and Schoelkopf, Hailey and Skowron, Aviya and Sutawika, Lintang and Tang, Eric and Thite, Anish and Wang, Ben and Wang, Kevin and Zou, Andy},
  title        = {A framework for few-shot language model evaluation},
  month        = 12,
  year         = 2023,
  publisher    = {Zenodo},
  version      = {v0.4.0},
  doi          = {10.5281/zenodo.10256836},
  url          = {https://zenodo.org/records/10256836}
}