explain the following like i am a newbie (layperson: ""Skip to content
tatsu-lab
/
alpaca_eval

Type / to search

Code
Issues
1
Pull requests
1
Actions
Projects
Security
Insights
Owner avatar
alpaca_eval
Public
tatsu-lab/alpaca_eval
Go to file
t
Add file
Folders and files
Name		
Latest commit
actions-user
actions-user
update __version__
f5046ae
 · 
13 hours ago
History
.github/workflows
feat: add test for updating leaderboard
5 months ago
client_configs
[DOC] clarify azure
2 months ago
docs
minor test fix
13 hours ago
example
minor
8 months ago
figures
Yann/length correction (#258)
13 hours ago
notebooks
Yann/length correction (#258)
13 hours ago
results
Add Mistral-ORPO-Beta to AlpacaEval (#257)
4 days ago
scripts
Yann/length correction (#258)
13 hours ago
src/alpaca_eval
update __version__
13 hours ago
tests
Yann/length correction (#258)
13 hours ago
.gitignore
[TEST] ensure that the baseline is always correct
last month
.pre-commit-config.yaml
[CLEAN] setting up precommit (#61)
8 months ago
CITATION.cff
add citation.cff
4 months ago
LICENSE
gitignore
9 months ago
MANIFEST.in
gitignore
9 months ago
README.md
minor bibtex
13 hours ago
pytest.ini
[ENH] update to allow AF to use AE (#241)
last month
requirements.txt
Yann/length correction (#258)
13 hours ago
setup.py
Yann/length correction (#258)
13 hours ago
Repository files navigation
README
Apache-2.0 license
 AlpacaEval : An Automatic Evaluator for Instruction-following Language Models
Code License Data License Python 3.10+ discord

AlpacaEval 2.0 with length-controlled win-rates has a spearman correlation of 0.98 with ChatBot Arena while costing less than $10 of OpenAI credits run and running in less than 3 minutes. Our goal is to have a benchmark for chat LLMs that is: fast (< 5min), cheap (< $10), and highly correlated with humans (0.98). Here's a comparison with other benchmarks:

LC AlpacaEval is the most highly correlated benchmark with Chat Arena.

Updates:

🎉 Length-controlled Win Rates are out and used by default! This increases the correlation with ChatBot Arena from 0.93 to 0.98, while significantly decreasing length gameability. The raw win rates are still shown on the website and the CLI. More details here.

🎉 AlpacaEval 2.0 is out and used by default! We improved the auto-annotator (better and cheaper) and use GPT-4 preview as baseline. More details here. For the old version, set your environment variable IS_ALPACA_EVAL_2=False.

Table of Contents
Overview
Quick Start
Leaderboards and how to interpret them
Models
Evaluators
Use-cases
Evaluating a model
Making a new leaderboard
Making a new evaluator
Contributing
Contributing a model
Contributing an evaluator
Contributing an eval set
Contributing a completion function
Limitations
Analysis
Analyzing an evaluator
Analyzing an eval set
Citation
Additional information
Length-controlled win rates
AlpacaEval 2.0
Data Release
Differences with AlpacaFarm
Related work
Interpreting annotations
Major updates
Overview
Evaluation of instruction-following models (e.g., ChatGPT) typically requires human interactions. This is time-consuming, expensive, and hard to replicate. AlpacaEval in an LLM-based automatic evaluation that is fast, cheap, replicable, and validated against 20K human annotations. It is particularly useful for model development. Although we improved over prior automatic evaluation pipelines, there are still fundamental limitations like the preference for longer outputs. AlpacaEval provides the following:

Leaderboard: a leaderboard of common models on the AlpacaEval evaluation set. Caution: Automatic evaluators (e.g. GPT-4) may be biased towards models that generate longer outputs and/or that were fine-tuned on the model underlying the evaluator (e.g. GPT-4).
Automatic evaluator: an automatic evaluator that has high agreement with humans (validated on 20K annotations). We evaluate a model by measuring the fraction of times a powerful LLM (e.g. GPT-4) prefers the outputs from that model over outputs from a reference model. Our evaluators enable caching and output randomization by default.
Toolkit for building automatic evaluators: a simple interface for building advanced automatic evaluators (e.g. with caching, batching, or multi-annotators) and analyzing them (quality, price, speed, statistical power, bias, variance etc).
Human evaluation data: 20K human preferences between a given and reference model on the AlpacaFarm evaluation set. 2.5K of these are cross-annotations (4 humans annotating the same 650 examples).
AlpacaEval dataset: a simplification of AlpacaFarm's evaluation set, where "instructions" and "inputs" are merged into one field, and reference outputs are longer. Details here.
When to use and not use AlpacaEval?
Quick Start
To install the stable release, run

pip install alpaca-eval
To install the nightly version, run

pip install git+https://github.com/tatsu-lab/alpaca_eval
Then you can use it as follows:

export OPENAI_API_KEY=<your_api_key> # for more complex configs, e.g. using Azure or switching clients see client_configs/README.md 
alpaca_eval --model_outputs 'example/outputs.json' 
This will print the leaderboard to the console, and save both the leaderboard and the annotations to the same directory as the model_outputs file. Important parameters are the following:

model_outputs : A path to a json file for the outputs of the model to add to the leaderboard. Each dictionary should contain the keys instruction and output.
annotators_config: This is the annotator to use. We recommend using weighted_alpaca_eval_gpt4_turbo ( default for AlpacaEval 2.0), which has a high agreement rate with our human annotation data, large context size, and is pretty cheap. For a comparison of all annotators see here.
reference_outputs: The outputs of the reference model. Same format as model_outputs. By default, this is gpt4_turbo for AlpacaEval 2.0.
output_path: Path for saving annotations and leaderboard.
If you don't have the model outputs, you can use evaluate_from_model and pass a local path or a name of a HuggingFace model, or a model from a standard API (OpenAI, Anthropic, Cohere, google, ...). Other commands:

>>> alpaca_eval -- --help
SYNOPSIS
    alpaca_eval COMMAND

COMMANDS
    COMMAND is one of the following:

     evaluate
       Evaluate a model based on its outputs. This is the default entrypoint if no command is specified.

     evaluate_from_model
       Evaluate a model from HuggingFace or an API provider. This is a wrapper around `evaluate` which includes generating from a desired model.

     make_leaderboard
       Precompute and save an entire leaderboard for a given dataset / evaluator / set of models generations.

     analyze_evaluators
       Analyze an evaluator and populates the evaluators leaderboard (agreement with human, speed, price,...).
For more information about each function use alpaca_eval <command> -- --help.

Leaderboards and how to interpret them
Models
Our leaderboards are computed on the AlpacaEval dataset. We precomputed the leaderboard for important models using different baseline models and autoannotators. Our two main leaderboards ("AlpacaEval 2.0" and "AlpacaEval") can be found on this page. "AlpacaEval 2.0" uses weighted_alpaca_eval_gpt4_turbo for the annotator and gpt4_turbo for the baseline. "AlpacaEval" uses alpaca_eval_gpt4 for the annotator and text_davinci_003 for the baseline. For all precomputed leaderboards see here. Later we also show how to add your model to the leaderboard and how to make a new leaderboard for your evaluator/dataset. See here for the configs of all models that are available out of the box.

AlpacaEval minimal leaderboard:

Win Rate	Std Error
gpt4	95.3	0.7
claude	88.4	1.1
chatgpt	86.1	1.2
guanaco-65b	71.8	1.6
vicuna-13b	70.4	1.6
text_davinci_003	50.0	0.0
alpaca-farm-ppo-human	41.2	1.7
alpaca-7b	26.5	1.5
text_davinci_001	15.2	1.2
How exactly are those metrics computed?
Details about our auto-annotator: alpaca_eval_gpt4
Evaluators
We evaluate different automatic annotators on the AlpacaEval set by comparing to 2.5K human annotations we collected (~650 instructions each with 4 human annotations). Below we show metrics for our suggested evaluators (weighted_alpaca_eval_gpt4_turbo,alpaca_eval_gpt4), for prior automatic evaluators (alpaca_farm_greedy_gpt4,aviary_gpt4,lmsys_gpt4), for humans (humans), and for different base models with essentially the same prompt (gpt4,claude,text_davinci_003,chatgpt_fn,guanaco_33b, chatgpt). See here for the configs of all evaluators that are available out of the box and their associated metrics.

Human agreement	Price [$/1000 examples]	Time [seconds/1000 examples]	Spearman corr.	Pearson corr.	Bias	Variance	Proba. prefer longer
alpaca_eval_gpt4	69.2	13.6	1455	0.97	0.93	28.4	14.6	0.68
alpaca_eval_cot_gpt4_turbo_fn	68.6	6.3	1989	0.97	0.90	29.3	18.4	0.67
alpaca_eval_gpt4_turbo_fn	68.1	5.5	864	0.93	0.82	30.2	15.6	0.65
gpt4	66.9	12.5	1037	0.88	0.87	31.5	14.6	0.65
alpaca_farm_greedy_gpt4	66.4	15.3	878	0.85	0.75	30.2	19.3	0.60
alpaca_eval_cot_gpt4_turbo_fn	65.7	4.3	228	0.78	0.77	33.9	23.7	0.61
humans	65.7	300.0	36800	1.00	1.00	0.0	34.3	0.64
claude	65.3	3.3	173	0.93	0.90	32.4	18.5	0.66
lmsys_gpt4	65.3	13.9	17982	0.98	0.97	31.6	15.9	0.74
text_davinci_003	64.1	8.7	121	0.85	0.83	33.8	22.7	0.70
longest	62.2	0.0	0	0.27	0.56	37.8	0.0	1.00
chatgpt	57.3	0.8	285	0.72	0.71	39.4	34.1	0.59
How exactly are those metrics computed?
Tips for choosing evaluators
The above metrics are computed with respect to annotations from crowd-workers. Although useful, those annotations are not perfect, e.g., crowd-workers often favor style over factuality. We thus recommend users to validate automatic evaluators on their own instructions and human annotations. Details in limitations.

Use-cases
Evaluating a model
>>> alpaca_eval evaluate -- --help
>>> alpaca_eval evaluate_from_model -- --help
To evaluate a model you need to:

Choose an evaluation set and compute outputs specified as model_outputs. By default, we use the 805 examples from AlpacaEval. To compute outputs on AlpacaEval use:
import datasets

eval_set = datasets.load_dataset("tatsu-lab/alpaca_eval", "alpaca_eval")["eval"]
for example in eval_set:
    # generate here is a placeholder for your models generations
    example["output"] = generate(example["instruction"])
if your model is a HuggingFace model or from a standard API provider (OpenAI, Anthropic, Cohere). Then you can directly use alpaca_eval evaluate_from_model to also take care of generating outputs.

Compute the reference outputs reference_outputs. By default, we use precomputed outputs of gpt4_turbo on AlpacaEval. If you want to use a different model or a different dataset follow the same steps as (1.).
Choose an evaluator specified via annotators_config. We recommend using alpaca_eval_gpt4_turbo_fn. For other options and comparisons see this table. Depending on the evaluator you might need to set the appropriate API_KEY in your environment or int the client_configs.
Running all together:

alpaca_eval --model_outputs 'example/outputs.json' \
  --annotators_config 'alpaca_eval_gpt4_turbo_fn'
If you don't have decoded outputs, you can use evaluate_from_model which takes care of decoding (model and reference) for you. Here's an example:

# need a GPU for local models
alpaca_eval evaluate_from_model \
  --model_configs 'oasst_pythia_12b' \
  --annotators_config 'alpaca_eval_gpt4_turbo_fn'      
Here the model_configs and reference_model_configs (optional) are paths to a directory that specifies the prompt, the model provider (here HuggingFace) and decoding parameters. See this directory for examples. For all model providers that are available out-of-the-box see here.

Information about annotators
Making a new leaderboard
Making a new evaluator
Contributing
We are accepting PRs for new models, evaluators, and eval sets, in addition to bug fixes. We will update the leaderboard website regularly with new community contributions. We have also created a support discord for AlpacaEval in case you run into any issues and wish to ask help from the community.

To get started, please first fork the repo, and install the package from source pip install -e .

Contributing a model
First, you'll need to add a model config definition in the models_configs folder. As an example, you can look at the falcon-7b-instruct yaml. Please make sure the folder name and key name in the yaml match exactly.

Then, please follow the steps in Evaluating a model to run inference on the model to produce outputs on the eval set and score the model according to one of the evaluators. An example command may look like:

alpaca_eval evaluate_from_model \
  --model_configs 'falcon-7b-instruct'
After running this command, you should have generated an outputs json and a new entry in the corresponding leaderboard file. Please make a PR with the config, outputs file, and updated leaderboard.

Concretely you should do something like:

Fork the repository in github
Clone the forked repository git clone <URL>
Make a model config at src/alpaca_eval/models_configs/<model_name> and evaluate it evaluate_from_model --model_configs '<model_name>'
Add the model configs, output, and leaderboard entry to the forked repository
git add src/alpaca_eval/models_configs/<model_name>
git add src/alpaca_eval/leaderboards/ 
git add -f results/<model_name>/model_outputs.json
git add -f results/<model_name>/*/annotations.json
git commit -m "Add <model_name> to AlpacaEval"
git push
Create a pull request on AlpacaEval
Note: if you are generating outputs outside of AlpacaEval you should still add a model config but with fn_completions: null. See this config for an example.

Getting your model verified
Contributing an evaluator
Contributing an eval set
Contributing a completion function
Limitations
The AlpacaEval evaluation pipeline, like other current evaluators have important limitations and should therefore not be used as replacement for human evaluation in important settings, such as to decide whether a model is ready to be deployed. Those can broadly be clustered into 3 categories:

Instructions might not be representative of real-usage: the AlpacaEval set contains examples from a variety of datasets (self-instruct, open-assistant, vicuna, koala, hh-rlhf) which might not be representative of real-usage and advanced applications of better models like GPT4. This likely makes the best closed models (GPT4 / Claude / ChatGPT / ...) seem more similar to the open models than what they are. Indeed, those closed models seem to be pretrained/finetuned on much more diverse data. See for example this blog for preliminary results on more complex instructions. Note, however, that in AlpacaFarm we showed that win-rates on our evaluation set are highly correlated (0.97 R2) with win-rates on instructions from user interactions with the Alpaca Demo. Furthermore, the AlpacaEval leaderboard shows larger gap between the open models and OpenAI models than other leaderboards ( e.g. lmsys).

Biases of automatic annotators: the raw automatic annotators seem to have implicit biases. In particular, we found that they tend to prefer longer outputs and outputs that contain lists (e.g. 0.68 / 0.69 for alpaca_eval_gpt4 and 0.62 / 0.58 for claude). Although we found that humans have similar biases (0.64 / 0.61), we believe that this could be more of a limitation of human annotation pipeline we used rather than a true human bias. More generally, through qualitative analysis, we found that automatic annotators give more importance to the style of the output than its content (e.g. factuality). Finally, we found that automatic evaluators tend to prefer outputs from models that are similar (likely trained on the same data) as suggested by the big difference between ChatGPT/GPT4 on claude's and alpaca_eval_gpt4's leaderboard. Note that the length bias is partially mitigated in our length-controlled win-rates.

Lack of safety evaluation: importantly, AlpacaEval only evaluates the instruction-following capabilities of models rather than the harm that they could cause (e.g. toxic behavior or bias). As a result the small gap between current ChatGPT and the best open source models should not be interpreted as if that the latter are ready to be deployed.

Beyond those limitations about the evaluation pipelines, there are also limitations about our validation of the evaluators and our proposed approach to selecting evaluation sets.

Limitations about our validation pipeline
Additional analysis and plots
Caution: all the following results are about AlpacaEval 1.0 and have not been updated since

Length-controlled AlpacaEval (LCAE)
Length-controlled AlpacaEval Visualizations: analyzing an evaluator

Length-controlled AlpacaEval Development: analyzing an evaluator

The notebook shows different options that we considered for mitigating the length bias of automatic annotators.

Here we briefly summarize the main results. Namely:

LCAE increases the correlation with Chat Arena to 0.98 from 0.94 for AlpacaEval 2.0. This makes LCAE the most highly correlated benchmark with Chat Arena as seen in the plot below.
LC AlpacaEval is the most highly correlated benchmark with Chat Arena.

LCAE decreases length gameability one of the major issues of AlpacaEval is that you can increase your win-rate by increasing the length of your outputs. For example, in AlpacaEval 2.0 the win-rate for the baseline (50%) increases to 64% when prompted to “give as much detail as possible” and decreases to 23% when prompted to “be as concise as possible while still providing all the necessary information to answer the question”. More generally the relative length gameability was ~21% for AlpacaEval and decreases to ~6% for LCAE, so it's 3x less gameable through prompt length. This is shown in the plot below.
LC AlpacaEval decreases length gameability of the benchmark.

We can predict performance for different baselines One other benefit of using a GLM for controlling for length bias. Is that we now have a model that can predict the win-rate of a model for different baselines. In particular, our GLM has many nice properties, for example win_rate(m,b) = 1 - win_rate(b,m) \in [0,1] and win_rate(m,m) = 0.5. This is shown in the plot below.
Predicted win rate for different baselines

Finally, note that we are only controlling for length bias. There are other known biases that we are not controlling for, such as the fact that auto-annotators prefer outputs similar to their model. Although we could control for that, in practice we have found that to be less of an issue than length bias. For two reasons (1) this mostly a single model in the leaderboard because fine-tuning on outputs from the auto-annotator doesn't seem to have doesn't seem to impact the win-rate as much, and (2) the bias is actually less strong that what one could think. For example we show below a subset of the leaderboards auto-annotated by three different models, and we see that the ranking of models is exactly the same. In particular, claude-3-opus prefers gpt4_preview, and mistral-large prefers the former two.

Leaderboard by different auto-annotators

Analyzing an evaluator
Analyzing an eval set
Citation
Please consider citing the following depending on what you are using and referring to:

Code, results, and general benchmark: alpaca_eval (this repo). Specify whether you are using AlpacaEval or AlpacaEval 2.0. For length-controlled win-rates see below.
Length-controlled (LC) win rates: alpaca_eval_length.
Human annotations: dubois2023alpacafarm (AlpacaFarm)
AlpacaEval evaluation set: alpaca_eval and self-instruct, open-assistant, vicuna, koala, hh-rlhf.
Here are the bibtex entries:

@misc{alpaca_eval,
  author = {Xuechen Li and Tianyi Zhang and Yann Dubois and Rohan Taori and Ishaan Gulrajani and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto },
  title = {AlpacaEval: An Automatic Evaluator of Instruction-following Models},
  year = {2023},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/alpaca_eval}}
}
@misc{alpaca_eval_length,
  author = {Yann Dubois and Balazs Galambosi and Percy Liang and Tatsunori B. Hashimoto },
  title = {Length-Corrected AlpacaEval: A Simple Debiasing of Automatic Evaluators},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/tatsu-lab/alpaca_eval}}
}
@misc{dubois2023alpacafarm,
  title={AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback}, 
  author={Yann Dubois and Xuechen Li and Rohan Taori and Tianyi Zhang and Ishaan Gulrajani and Jimmy Ba and Carlos Guestrin and Percy Liang and Tatsunori B. Hashimoto},
  year={2023},
  eprint={2305.14387},
  archivePrefix={arXiv},
  primaryClass={cs.LG}
}
More information
Length-Controlled Win Rates
AlpacaEval 2.0
Data Release
Differences with AlpacaFarm
Related work
Interpreting annotations
Major updates
About
An automatic evaluator for instruction-following language models. Human-validated, high-quality, cheap, and fast.

tatsu-lab.github.io/alpaca_eval/
Topics
nlp deep-learning leaderboard evaluation instruction-following foundation-models large-language-models rlhf
Resources
 Readme
License
 Apache-2.0 license
Citation
 Activity
 Custom properties
Stars
 995 stars
Watchers
 9 watching
Forks
 133 forks
Report repository
Releases 29
Release v0.6
Latest
13 hours ago
+ 28 releases
Packages
No packages published
Contributors
46
@YannDubs
@actions-user
@rtaori
@lxuechen
@victorsungo
@jondurbin
@sanderland
@inferLLM
@44670
@gblazex
@winglian
@JetRunner
@nbl97
@imoneoi
+ 32 contributors
Deployments
392
 github-pages 13 hours ago
+ 391 deployments
Languages
Jupyter Notebook
91.4%
 
Python
8.6%
Footer
© 2024 GitHub, Inc.
Footer navigation
Terms
Privacy
Security
Status
Docs
Contact
Manage cookies
Do not share my personal information
""