AutoBencher: Towards Declarative Benchmark Construction

## Abstract

Abstract Evaluation is critical for assessing capabilities, tracking scientific progress, and informing model selection.
In this paper, we present three desiderata for a good benchmark for language models:
(i) salience (e.g., knowledge about World War II is more salient than a random day in history), (ii) novelty (i.e., the benchmark reveals new trends in model rankings not shown by previous benchmarks), and (iii) difficulty (i.e., the benchmark should be difficult for existing models, leaving headroom for future improvement).
We operationalize these three desiderata and cast benchmark creation as a search problem, that of finding benchmarks that that satisfy all three desiderata.
To tackle this search problem, we present AutoBencher, which uses a language model to automatically search for datasets that meet the three desiderata.
AutoBencher uses privileged information (e.g. relevant documents) to construct reliable datasets, and adaptivity with reranking to optimize for the search objective.
We use AutoBencher to create datasets for math, multilingual, and knowledge-intensive question answering.
The scalability of AutoBencher allows it to test fine-grained categories and tail knowledge, creating datasets that are on average 27% more novel and 22% more difficult than existing benchmarks.
A closer investigation of our constructed datasets shows that we can identify specific gaps in LM knowledge in language models that are not captured by existing benchmarks, such as Gemini Pro performing much worse on question answering about the Permian Extinction and Fordism, while OpenAGI-7B performing surprisingly well on QA about COVID-19. 1 1 1 Code is available at https://github.com/XiangLi1999/AutoBencher .

## 1 Introduction

Evaluation is crucial for informing model selection and guiding model development, and it is especially challenging to evaluate language models.
Prior works have constructed high-quality benchmarks, guided by two desiderata: (1) *salience*: the benchmark should test practically important capabilities, for example, the capability to solve math problems is examined by GSM8K [] and capability to avoid answering misleading questions is examined by TruthfulQA [].
(2) *difficulty*: the benchmark should yield low accuracy with existing models.
For example, GPQA [] carefully constructs questions that are challenging even for domain experts.
In this work, we identify an additional goal of (3) *novelty*. More recently, observational studies of existing benchmarks show that model performance on existing benchmarks are highly correlated [, , , ], and thus there is an increasing need for benchmark novelty. Novelty measures the degree to which a benchmark reveals previously unknown performance trends in model rankings, such as unexpected model strengths or weaknesses on specific domains.

In order to measure salience, difficulty, and novelty of a benchmark, we operationalize these three properties: (1) We define salience as a binary variable that captures whether a benchmark on a specific topic (e.g., COVID-19) is important. (2) The difficulty of a benchmark is determined by the lowest error rate achievable by existing models. (3) We calculate the novelty of a benchmark as the predictability of model performance on the benchmark based on performance on other benchmarks.
Formalizing the three desiderata allows us to take an optimization approach to constructing benchmarks. Instead of viewing benchmarking as the task of constructing a dataset that represents some task, we cast it as a way to find particular test set that fulfill our three desiderata. This turns benchmarking into an optimization problem, where the objective is made explicit.

To approximately solve this search problem and automatically create new salient, difficult, and novel benchmarks, we propose AutoBencher.
As shown in Figure 1, given a broad domain (e.g., history), AutoBencher proposes candidate evaluation topics (e.g., World War II) and then constructs a small dataset for each topic using privileged information (e.g., relevant Wikipedia articles).
These datasets are scored by our metrics and used as part of a local search algorithm to find a benchmark with improved difficulty. We leverage the scalability of AutoBencher to identify and select topics that jointly maximize a weighted sum of difficulty and novelty, subject to a salience constraint specified by the user.

We use AutoBencher to create datasets in 5 domains: math, history, science, economics, and multilingualism, altogether producing around 3700 examples.
These benchmarks reveal novel trends not shown by prior benchmarks (§ 6.2):
For example, we find that while Gemini Pro is one of the strongest models on existing history benchmarks, it performs quite poorly on AutoBencher-discovered topics of Permian Extinction and Fordism, performing even worse than some 7B models, such as Mistral-7B.
Meanwhile, we also find that 7B models (e.g., Mistral-7B and OpenAGI-7B), which typically exhibit lower accuracy due to their smaller size, perform quite well on the topic of COVID-19, even achieving accuracy on par with GPT-4 Turbo.
Additionally, our datasets increase difficulty by 22% compared with human-constructed benchmarks, such as MMLU and XOR QA (§ 6.1).(^2^22We will release the dataset and code.)

Figure: Figure 1: (Left) Models ranked by accuracy on existing datasets and AutoBencher datasets. Existing datasets show roughly the same performance trends, while AutoBencher discovers tests that induce novel rankings. (Right) Given a domain (e.g., history), AutoBencher creates datasets that are salient, difficult, and novel. It achieves this by searching through evaluation topics, scoring each based on these desiderata, and selecting the best one. These datasets are automatically created by providing privileged information to a language model.
Refer to caption: x1.png

## 2 Related Work

#### Benchmarking Language Models

A large number of benchmarks have been constructed to measure different skills of language models. To give a few examples of common LLM benchmarks, MMLU measures the understanding of academic subjects [], GSM8K measures mathematical reasoning [], and Winogrande measures common sense reasoning []. Researchers have also grouped the benchmarks to create leaderboards that rank LMs’ overall capabilities, such as HELM [] and Open LLM Leaderboard []. Additionally, researchers also carefully subsample existing benchmarks to obtain a smaller and more efficient benchmark with similar model accuracy [].
Prior works of LLM-as-Judge incorporate language models as part of the evaluation pipeline to automatically judge the quality of model-generated responses [, , ]. Our work further automates the evaluation pipeline by using LM to propose evaluation topics and generate questions.

The most similar work to ours is Li et al. [], which uses LMs to brainstorm questions for instruction following tasks. However, their goal is very different from ours: they aim to find a minimal set of questions to reproduce the same LM ranking achieved by evaluating a large number of examples; our goal is to search for new benchmark topics that break the performance trends of existing benchmarks.

#### Adaptive Datasets

In AutoBencher, one important desideratum we optimize for is novelty. Prior works have also constructed datasets adaptively to search for difficult questions [, , , , ]. Most of these works have generated test cases with human annotators, whereas we use language models to automate the search, saving extensive human effort. Similar to AutoBencher, Perez et al. [] searches for prompts that induce harmful behaviors in language models using another language model. However, they focus on instance-level failures in safety, whereas we focus on finding systematic failures on specific topics.

## 3 A Metric-Driven Framework of Benchmark Creation

We now discuss our three desiderata of *salience*, *difficulty*, and *novelty* and formally define them as quantitative metrics that can be directly optimized.

#### Preliminaries.

Let $c\in\mathcal{C}$ be a evaluation topic.
We define a dataset $\mathcal{D}_{c}=\{(x_{i},y_{i})\}_{i}$ as a set of question-answer pairs $(x_{i},y_{i})$ that evaluate mastery of the topic $c$.
As an example, topics $\mathcal{C}$ could include “the Industrial Revolution”, “World War II”, “Ancient Civilization”, etc.
In this work, we will generate the datasets $\mathcal{D}_{c}$ from a fixed generator $p(\mathcal{D}_{c}\mid c)$ and focus on optimizing the set of topics $c$ to change the novelty and difficulty of a dataset.

Let $\mathcal{M}=\{\texttt{LM}_{m}\}_{m=1}^{M}$ denote the set of $M$ existing models to evaluate.
We denote the accuracy of model $\texttt{LM}_{m}\in\mathcal{M}$ on benchmark $\mathcal{D}_{c}$ as $\texttt{acc}(\texttt{LM}_{m},\mathcal{D}_{c})$.
We define the *accuracy vector* $V_{c}=[\texttt{acc}(\texttt{LM}_{1},\mathcal{D}_{c}),\cdots,\texttt{acc}(
\texttt{LM}_{M},\mathcal{D}_{c})]$ of a benchmark $\mathcal{D}_{c}$ as the accuracy of all models on the benchmark $\mathcal{D}_{c}$.

#### Salience.

Recall that salience measures the practical importance of a topic $c$. First, we assume a set of salient topics $\mathcal{S}$ specified by the user, and we define salience as a binary variable, such that $\textsc{salience}(c)=1$ if $c\in\mathcal{S}$ and $\textsc{salience}(c)=0$ otherwise. For example, we may define salient topics $\mathcal{S}$ to be the set of topics with the number of corresponding Wikipedia page views exceeding a threshold.

#### Novelty.

Recall that novelty measures how much new information a dataset reveals about existing models over existing benchmarks.
Therefore, we formalize $\textsc{Novelty}(\mathcal{D}_{c};\mathbf{D}_{\text{prev}};\mathcal{M})$ as a function of the benchmark in question $\mathcal{D}_{c}$, prior benchmarks $\mathbf{D}_{\text{prev}}:=\{\mathcal{D}_{1}\ldots\mathcal{D}_{N}\}$, and the existing set of models $\mathcal{M}$.
Intuitively, the results on a new benchmark reveal new information if model performance on the new benchmark vastly differ from the trends on prior benchmarks (e.g., if a normally low-performing model outperforms all other models on the new benchmark).

To quantify this, we first find how much information is explainable by the accuracy on existing datasets $\mathbf{D}_{\text{prev}}$, by performing a regression from $V_{\text{prev}}:=\left[V_{1}\cdots V_{N}\right]\in\mathbb{R}^{N\times M}$ to predict $V_{c}\in\mathbb{R}^{1\times M}$,

$$ $\hat{V}_{c}:=w^{*\top}V_{\text{prev}}+b\text{~{}~{}and~{}~{}}w^{*}= \operatorname*{arg\,min}_{w}\bigg{|}\bigg{|}w^{\top}V_{\text{prev}}+b-V_{c} \bigg{|}\bigg{|}_{2}^{2}.$ $$

We then compute the rank correlation between the predicted accuracy $\hat{V}_{c}$ and the ground truth accuracy as $\textsc{RankCorr}(V_{c},\hat{V}_{c})$ as a predictability measure for the new dataset. Formally,

$$ $\displaystyle\textsc{Novelty}(\mathcal{D}_{c},\mathbf{D}_{\text{prev}}, \mathcal{M})=1-\textsc{RankCorr}(\hat{V}_{c},V_{c}).$ $$

If the new accuracy vector $V_{c}$ is spanned by the existing accuracy vectors, $\textsc{RankCorr}(V_{c},\hat{V}_{c})$ will be close to 1, resulting in low novelty. On the other hand, if the new accuracy vector $V_{c}$ discovers some new patterns in model performance such as an orthogonal direction, $\textsc{RankCorr}(V_{c},\hat{V}_{c})$ will be low, resulting in high novelty.

Take Figure 1 (left) as an example, let ARC be the existing datasets $\mathbf{D}_{\text{prev}}$. To compute the novelty of the MMLU dataset (column 2), we perform a regression from the accuracy vector of ARC to predict the accuracy vector of MMLU. Since they have the same ranking, model rankings on MMLU is perfectly predictable from ARC, yielding a RankCorr of $1$ and a Novelty of 0 0. To compute the novelty of the QA: COVID-19 dataset (column 3), we also predict its accuracy vector from ARC. However, since the accuracy vector of COVID-19 datasets shows abrupt changes in model performance, it is not spanned by the accuracy vector of the ARC dataset. Therefore, its performance is not predictable by the existing dataset, yielding a low RankCorr and a high Novelty.

#### Difficulty.

A benchmark’s difficulty is determined directly by a model’s error rate. Ideally, a benchmark should leave sufficient headroom above the best current error rate to enable tracking future progress. We formalize the difficulty of a benchmark as the remaining gap between the best model’s accuracy and optimal accuracy: $\textsc{Difficulty}(\mathcal{D}_{c},\mathcal{M})=1-\max_{m}\texttt{acc}(
\texttt{LM}_{i},\mathcal{D}_{c})$.

#### Separability.

Besides the three main desiderata, we also include the separability of model accuracy as an additional regularizer. Separability measures the amount of separation between different model accuracies; In this paper, we use separability to complement novelty, ensuring that all the model performance trends revealed by a benchmark is robust.
We formalize the separation on benchmark $\mathcal{D}_{c}$ between the accuracy of models $\mathcal{M}$ as the mean absolute deviation $\textsc{Sep}(\mathcal{D}_{c},\mathcal{M})=\text{mean}(|V_{c}-\text{mean}(V_{c}
)|)$.

#### Existing Benchmarks Case Study.

We now analyze several existing benchmarks under these desiderata:
MMLU [] contains salient topics on academic subjects; it is sufficiently difficult with the best model accuracy of 86% and has good separability to distinguish existing models. However, the benchmark lacks novelty, as it’s highly correlated with prior benchmarks like ARC, with a rank correlation of 94%.

Datasets in the Inverse Scaling Prize [] is novel but many lack salience. For example, pattern matching suppression instructs the LM to continue a sequence by violating the current pattern, and Memo trap instructs the LM to continue a famous quote with a different ending. These tasks reveal novel performance patterns (i.e., larger models perform worse), but are much less salient than datasets like the knowledge-intensive MMLU.

## 4 AutoBencher

Our goal is to find a sufficiently salient evaluation topic $c$ that maximizes a linear combination of novelty and difficulty.
We directly maximize novelty and difficulty subject to a constraint on salience, because salience is operationalized as a binary variable.
Specifically, we aim to solve the following constrained optimization problem:

$$ maximize $\displaystyle\mathcal{J}(c)=\textsc{Novelty}(\mathcal{D}_{c};\mathbf{D}_{\text {prev}},\mathcal{M})+\beta_{1}~{}\textsc{difficulty}(\mathcal{D}_{c};\mathcal{ M})+\beta_{2}~{}~{}\textsc{Sep}(\mathcal{D}_{c};\mathcal{M})$ (1) subject to $\displaystyle c\in\mathcal{S}.$ $$

#### Solving the optimization problem.

Identifying solutions to this search problem is difficult due to the complexity of performing a discrete search over three potentially conflicting criteria (salience, difficulty, and novelty) and we cannot intuitively judge which topics might reveal novel performance trends a priori.
Therefore, we propose AutoBencher, which automates the search with a language model, denoted as $\texttt{LM}_{\text{gen}}$, and outputs the best discovered topic and its dataset.

One natural, naive design is to perform a random search, where we prompt $\texttt{LM}_{\text{gen}}$ to generate a diverse set of topics $c$, prompt $\texttt{LM}_{\text{gen}}$ to generate a dataset of (question, answer) pairs for each topic, and then select the best topic according to the objective function $\mathcal{J}(c)$.

However, this design suffers from two issues:
(1) *Example correctness*: Since we use $\texttt{LM}_{\text{gen}}$ to construct the dataset, the generated answers might be incorrect due to model hallucination. (2) *Example difficulty*: The difficulty of the generated questions is upper-bounded by the capabilities of $\texttt{LM}_{\text{gen}}$ and hence cannot be used to evaluate stronger models.
(3) *Topic difficulty*: empirically, we observe that $\texttt{LM}_{\text{gen}}$ tends to propose well-known, and insufficiently difficult topics.

We now propose some techniques to address these issues: We first augment $\texttt{LM}_{\text{gen}}$ with privileged information to improve the correctness and difficulty of the generated datasets (§ 4.1). Next, we propose adaptive search, which uses the trajectory of past generated benchmarks to improve topic difficulty (§ 4.2). Finally, we present the full pseudocode and present how each metric gets optimized in this pipeline (§ 4.3).

### 4.1 Generating Datasets with Privileged Information

Figure: Figure 2: How the model $\texttt{LM}_{\text{gen}}$ uses privileged information to create (question, answer) examples.
Refer to caption: x2.png

To improve the difficulty of the generated questions and the correctness of the generated answers, we augment $\texttt{LM}_{\text{gen}}$ with privileged information that we do not provide to the LMs that we are evaluating (which we will refer to as the test-taker LMs to avoid confusion).
This privileged information ensures answer correctness by grounding the generated answers in a reliable source and creates an information imbalance between the example creation model $\texttt{LM}_{\text{gen}}$ and test-taker LMs, which simplifies the task for $\texttt{LM}_{\text{gen}}$ and enables it to create questions that are more difficult than possible with its base capabilities.
We next detail the privileged information we provide in three domains: knowledge intensive, multilingual, and mathematics.
Figure 2 illustrates how this information is used in each domain.

#### Knowledge-intensive.

We augment $\texttt{LM}_{\text{gen}}$ with a set of relevant documents (e.g., Wikipedia).
Specifically, to create a knowledge-intensive question on topic $c$, we first retrieve the set of most relevant articles by querying $c$ in the Wikipedia Search API.
Then, we prompt $\texttt{LM}_{\text{gen}}$ to jointly generate (question, answer) pairs conditioned on the retrieved articles such that the answer can be verified by the document and the question is answerable *without* the document. We enforce this by checking the lexical overlap between the answers and the retrieved articles.

#### Multilingual.

We augment $\texttt{LM}_{\text{gen}}$ with a translation system.
Since models tend to have better reasoning capabilities in English than in other languages, we generate (question, answer) pairs in other languages by first generating the example in English via the knowledge-intensive question procedure above.
Then, we translate the question and answer to the target language.

#### Mathematics.

We augment $\texttt{LM}_{\text{gen}}$ with Python math libraries (e.g., sympy, scipy, numpy).
To ensure that the answers are correct, we prompt $\texttt{LM}_{\text{gen}}$ to generate questions along with Python code to compute their answers and use the execution result as the answer.

### 4.2 Proposing Topics with Adaptive Search

When we use $\texttt{LM}_{\text{gen}}$ to propose evaluation topics, a key challenge is that $\texttt{LM}_{\text{gen}}$ does not have information about what topics might be difficult
for LMs.
To address this, we propose adaptive search, an iterative approach that collects accuracy information in each iteration to inform topic proposals in subsequent iterations. We keep track of a trajectory $\mathcal{H}$, represented as a sequence of (topic, accuracy) pairs.
As we run more iterations, $\mathcal{H}$ accumulates more (topic, accuracy) pairs, and forms a better belief about what topics are likely be difficult (see Figure 1 right).

Specifically, we follow lines 1 to 7 in Algorithm 1.
In each iteration, AutoBencher proposes $K$ topics conditioned on the trajectory $\mathcal{H}$ collected from previous iterations (line 3), where we specifically prompt to ask for difficult topics (i.e., low accuracy).
We filter out non-salient topics (line 4) and instantiate each remaining topic as a dataset via the procedure described in § 4.1 (line 5).
Then, we compute the accuracy of a test-taker model $\texttt{LM}_{\text{test}}$ on each dataset as a measure of difficulty (line 6).
Finally, we feed all proposed (topic, accuracy) pairs to subsequent iterations to increase difficulty (lines 7).
We choose to only evaluate the accuracy of a single strong test-taker model, rather than evaluating all models $\mathcal{M}$ for computational efficiency.

Our adaptive search procedure does not take Novelty or Sep into account, since these two quantities also require expensively evaluating all models $\mathcal{M}$.
Instead, we optimize these desiderata in a final re-ranking step, which considers all of the metrics together.

Figure: Algorithm 1 AutoBencher

### 4.3 Metric Attribution

Overall, the entire algorithm behind AutoBencher is summarized in
Algorithm 1.
In lines 1 to 7, we perform the adaptive search described above.
Afterward, we re-rank the proposed topics via the full search objective $\mathcal{J}(c)$:
We compute the objective for each proposed topic (line 9)
and construct a dataset on the topic with the highest objective value (lines 10–12).

This algorithm optimizes for all the desiderata of novelty, salience, and difficulty.
salience is met via the filtering step in line 4, which filters out all non-salient topics. Difficulty is approximately optimized via the adaptive search procedure (lines 1–7) that steers toward producing more difficult topics.
Sep and Novelty are optimized in the final re-ranking step (lines 9–10), where we compute and select based on the full objective function $\mathcal{J}$.

## 5 Experimental Setup

### 5.1 Baselines and Metrics

We evaluate AutoBencher on five domains:
mathematics, multilingual, and knowledge-intensive QA on history, economy, and science.

#### Baselines.

We compare benchmarks generated by AutoBencher with human constructed benchmarks, which we group collectively under HumanBench.
For knowledge-intensive domains, we compare with the subject datasets in MMLU [].
These datasets consist of multiple-choice questions on academic subjects ranging from high school to professional level.
We compare with 4 history subjects (e.g., high school world history),
4 economy subjects (e.g., econometrics), and 7 science subjects (e.g., college physics).
See the complete list in Appendix C.

For mathematics, we compare with $7$ datasets from the Mathematics Dataset
[](^3^33[https://github.com/google-deepmind/mathematics_dataset](https://github.com/google-deepmind/mathematics_dataset)), which covers basic math capabilities: algebra, arithmetic, calculus, probability, comparison, measurement, numbers.
For multilingualism, we compare with XOR QA
[], a multilingual question-answering dataset covering 7 diverse languages.
We compare with the test set, split by language into 7 datasets.

#### Models.

For this paper, the set of models we evaluate is $\mathcal{M}=\{$gpt-4-turbo-2024-04-09, gpt-3.5-turbo-0613, claude-3-sonnet-20240229, claude-3-opus-20240229, claude-2.0, Mixtral-8x7B-Instruct-v0.1, Mistral-7B-Instruct-v0.1, gemini-pro, OpenAGI-7B-v0.1, vicuna-7b-v1.5,
Llama-2-7b-chat-hf,
Xwin-Math-7B-V1.0,
WizardMath-7B-V1.0,
gpt-neo-2.7B,
alpaca-7b,
zephyr-7b-beta,
openchat-3.5-0106$\}$
These models are designed to cover three categories: the strongest closed models, strong open models and strong 7B models.

#### Metrics.

We evaluate on the four metrics: Novelty (Novel), Sep (Sep), Difficulty (Diff), and salience as defined in § 3.
For calculating Novelty, we set $\mathbf{D}_{\text{prev}}$ as the aggregate of all datasets in HumanBench.

### 5.2 AutoBencher Hyperparameters and Costs

Hyperparameters.
AutoBencher uses gpt-4-0125-preview [] as $\texttt{LM}_{\text{gen}}$ to propose topics and generate the datasets.
For the test-taker model $\texttt{LM}_{\text{test}}$, we select gpt-3.5-turbo-0613 [], Mixtral-8x7B and Mistral-7B, to cover different levels of model accuracies.
In AutoBencher, we perform $N=8$ iterations of adaptive search, each proposing $k=5$ topics.
We generate $|\mathcal{D}_{c}|=50$ examples per topic.
In total, we explore 2000 questions across 40 topics in one round of AutoBencher.

Costs.
Each run of the AutoBencher agent uses around 750K tokens, which costs around $15.
Among them, 43K tokens are used for proposing topics, 576K tokens are used for constructing datasets, and 147K for evaluation.

## 6 Main Results

We find that AutoBencher successfully constructs datasets that satisfy the three desiderata. We first report the novelty and difficulty scores in § 6.1. Then we provide the list of discovered topics and qualitative examples of questions generated by AutoBencher in § 6.2. Finally, we conduct a human study that verifies the salience, novelty, and difficulty of the discovered topics in § 6.3.

### 6.1 Novelty and Difficulty

Recall that our definition of novelty measures the rank correlation between models’ accuracies on one dataset with their accuracies on all other datasets.
A lower correlation indicates more novel performance trends.
We find that datasets constructed by AutoBencher are significantly more novel than existing human-constructed datasets, reducing the rank correlation by 27%.
Moreover, AutoBencher datasets also exhibit 22% greater difficulty (Diff) and higher separation (Sep) between models, increasing the accuracy gaps between existing models by 1%, on average.
These improvements hold across all domains, as shown in Table 1.

We evaluate the impact of adaptive search on novelty and difficulty by ablating it in AutoBench-AS.
Rather than conditioning on the (topic, accuracy) pairs of previously proposed topics, we simply prompt $\texttt{LM}_{\text{gen}}$ to propose salient, difficult, and diverse topics.
Table 1 (top) shows that AutoBench-AS obtains lower novelty and difficulty scores than full AutoBencher, but still outperforms the human-constructed datasets in all metrics.
This is likely because adaptive search only affects the quality of the proposal distribution, and AutoBench-AS still accounts for novelty and difficulty via final re-ranking on the objective function.

**Table 1: Comparison between AutoBencher and prior human-constructed datasets (HumanBench) on novelty, separation (Sep), and difficulty (Diff). Higher numbers are better for all metrics. AutoBencher constructs datasets that are significantly more novel and difficult over human-constructed datasets. Ablating the adaptive search component impacts all metrics, particularly difficulty.**
|  | History | Economy | Science |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Novel | Sep | Diff | Novel | Sep | Diff | Novel | Sep | Diff |
| HumanBench | 0.05 | 0.031 | 0.103 | 0.13 | 0.011 | 0.056 | 0.22 | 0.020 | 0.4 |
| AutoBench-AS | $0.24\pm 0.07$ | 0.037 | 0.257 | $0.34\pm 0.06$ | 0.021 | 0.206 | $0.35\pm 0.12$ | 0.024 | 0.144 |
| AutoBench | $\mathbf{0.39\pm 0.10}$ | 0.042 | 0.440 | $\mathbf{0.43\pm 0.10}$ | 0.026 | 0.321 | $\mathbf{0.39\pm 0.06}$ | 0.031 | 0.475 |

### 6.2 Qualitative Examples

To qualitatively analyze the results of AutoBencher, we list the topics that are selected by AutoBencher along with the number of Wikipedia page views of that topic and $3$ random (question, answer) pairs from each constructed dataset in Table 3.
We find that the generated questions and answers are generally of high quality:
they are primarily factually correct and self-contained, though some questions are too open-ended or have dubious answers, e.g., “What is the significance of blockchain technology in environmental science?”
Below, we also discuss the novel performance trends AutoBencher reveals in four topics.
See Appendix D for the full ranking details.

Fordism is an important economic concept on mass production.
Gemini Pro drops from an average rank of $6$ on existing economics datasets to $16$.
In contrast, OpenChat-3.5 performs quite well.
It jumps from an average economics rank of 8 to 2 on this dataset, nearly matching Claude 3 Sonnet.

Secret Society is a historical topic, and exhibits better-than-expected GPT-3.5 Turbo accuracy, rising from rank $7$ to $3$. Gemini Pro performs worse than expected, dropping from rank $6$ to $16$.

Economic impact of COVID-19 reveals much better-than-expected accuracy by Mistral-7B and OpenAGI-7B, which nearly match GPT-4 Turbo, perhaps due to the wealth of COVID-19 data.

Permian extinction is the earth’s most severe known extinction event. Gemini Pro and Claude-2.0 both perform much worse than expected, dropping ranks by $7$ and $4$ respectively.

General trends. Overall, AutoBencher reveals two general trends:
First, small 7B models can perform quite well on select topics, matching the accuracies of much larger models, and even at times, the best models.
Second, Gemini Pro frequently performs much worse on AutoBencher-selected topics than on existing datasets.

**Table 2: We find that the human-generated datasets on these discovered evaluation topics are also novel. This confirms that the discovered topics indeed reveal novel model performance.**
|  | Economy | Science | History |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Novel | Sep | Diff | Novel | Sep | Diff | Novel | Sep | Diff |
| HumanBench | 0.13 | 0.011 | 0.056 | 0.22 | 0.020 | 0.400 | 0.05 | 0.031 | 0.103 |
| AutoBench | $0.43\pm 0.1$ | 0.026 | 0.321 | $0.39\pm 0.06$ | 0.031 | 0.475 | $0.39\pm 0.1$ | 0.042 | 0.440 |
| Human Study | $0.34\pm 0.06$ | 0.042 | 0.130 | $0.39\pm 0.06$ | 0.057 | 0.268 | $0.17\pm 0.04$ | 0.034 | 0.269 |

**Table 3: Discovered topics (labeled with their Wikipedia page view count) and three (question, answer) pairs randomly drawn from the datasets constructed by AutoBencher.**
| Topic | Qualitative Examples of (question, answer) |
| --- | --- |
| [History] Secret society (1.2M) | Question: Who was assassinated in 1963? Answer: John F. Kennedy |
| Question: Who compiled ’The Cyclopædia of Fraternities’ in 1899? Answer: Albert Clark Stevens |  |
| Question: Which university’s historian offered a three-pronged definition of secret societies? Answer: University of Idaho |  |
| [History] Space Race (1.4M) | Question: What was the first artificial satellite? Answer: Sputnik 1 |
| Question: Which astronaut was the first American to orbit the Earth? Answer: John Glenn |  |
| Question: What action did President Eisenhower take in response to the Soviet space lead? Answer: Recommended the establishment of NASA |  |
| [Science] Environmental science (0.9M) | Question: What legislation was established in Mesopotamia to address deforestation? Answer: the first legislation limiting deforestation |
| Question: How does artificial intelligence contribute to wildlife conservation? Answer: by predicting the movement of animal populations and protecting the habitats of wildlife |  |
| Question: What is the significance of blockchain technology in environmental science? Answer: monitors and regulates global fisheries |  |
| [Science] Late Permian extinction event (1.6M) | Question: What is the 13C / 12C ratio of buried sedimentary organic matter below normal? Answer: 2.0 to 2.5% below normal |
| Question: What was the fate of ammonoids during the Permian-Triassic extinction event? Answer: suffered a selective extinction pulse 10 million years before the main event |  |
| Question: What phenomenon drastically increased competition and accelerated biotic recovery in the Anisian? Answer: niche crowding |  |
| [Economy] Fordism (0.5M) | Question: What predominates over manufacturing in the current economic landscape? Answer: Service industries |
| Question: What was a byproduct of Ford substantially increasing his workers’ wages? Answer: Giving them the means to become customers |  |
| Question: What major advantage did Fordism bring to industrial production? Answer: It cut down on the manpower necessary for the factory to operate |  |
| [Economy] Economic impact of COVID-19 (41M) | Question: What was the impact of the pandemic on educational systems? Answer: many governments temporarily closed educational institutions |
| Question: What was the increase in percentage of white collar workers working remotely from before the pandemic to May 2020? Answer: From 6% to 65% |  |
| Question: By what percentage did the income generated from work across the world drop in the first nine months of 2020? Answer: 10.7 percent |  |
| [Math] Probability of two events | Question: What is the probability of selecting an odd number from the set 1, 3, 5, 7, 9 and then selecting a number divisible by 2 from the set 2, 4, 6, 8? Answer: 1.0 |
| Question: In a lottery where you pick a number from 1 to 100, what is the probability of guessing the number correctly two times in a row? Answer: 0.0001 |  |
| Question: What is the probability of rolling a 1 on a six-sided die and then flipping tails on a coin? Answer: 0.08333 |  |
| [Math] Order of operations | Question: What is 72 + 18 - (6 * 5)? Answer: 60 |
| Question: What is 88 + 12 - (56 / 7)? Answer: 92.0 |  |
| Question: What is (80 - 40) / (10 / 2)? Answer: 8.0 |  |

### 6.3 Human Study for Robustness

We have shown that AutoBencher can identify salient topics such as the Permian extinction where capable models fail. However, this does not prove that the dataset we construct is salient, or that the topic itself is what is causing the model to fail. For example, the optimization process of AutoBencher may have discovered specific, adversarial interactions between $\texttt{LM}_{\text{gen}}$ and the test-taker model, or there may be subtle flaws in the dataset itself. To rule out these issues, we perform a verification study where humans generate the dataset given only the topic category, and show that the same trends appear with human generated datasets.

Specifically, we gave one of the authors the discovered topics and access to Wikipedia and asked them to generate a QA dataset on the given topic. We report the novelty and difficulty metrics of the human-generated dataset in Table 2. We find that the human generated datasets on these topics are also more novel than the existing datasets in each domain, improving novelty by $16\%$. Also, the human constructed dataset on the discovered topics attains better difficulty and separability scores than existing datasets on average, though the gaps are smaller here. Overall, these results show that our identified novel failures are robust to dataset construction approaches (e.g., by AutoBencher, or by human) and AutoBencher is a promising way to find salient, difficult, and novel model failures.

## 7 Conclusion and Future Works

In this paper, we present a metrics-driven approach towards constructing new datasets. Given three desiderata of salience, difficulty, and novelty, we operationalize them as metrics and cast benchmark construction as a discrete search problem optimizing for these metrics. To tackle this search, we propose AutoBencher, which uses a language model to search for datasets that meet these criteria.

The paper focuses on the capabilities of LLMs such as knowledge-intensive QA, mathematics, and multilingualism. Future works could apply AutoBencher to broader domains (e.g., safety, efficiency) to search for novel model behavior patterns. Additionally, we restrict the proposed topics to fall under a domain for the scope of this paper, and future works could explore this design space and extend AutoBencher to propose creative tasks.

## Appendix A Limitation

Recall that in AutoBencher, the evaluation datasets are fully model generated. While these datasets serve as good proposal distributions for finding novel patterns, we believe that the novel patterns should be confirmed and verified by humans before making model development decisions based on these findings.

Additionally, we are using GPT-4 as the $\texttt{LM}_{\text{gen}}$, which could potentially bias in favor of models in the same families such as GPT-3.5-turbo. However, empirically, we find that this is not obvious, as AutoBencher generated dataset very often rates Claude-3 models the highest.

## Appendix B Broader Impact

Automatic benchmark creation via AutoBencher has several potential negative impacts if used improperly.
Primarily, since these created benchmarks can help identify research directions to work on or weaknesses in existing models, adequate care needs to be taken to ensure that the benchmark results are indeed meaningful.
Since the benchmark creation is automated, there is potential for weird or insignificant results to arise, and users must not blindly trust these results, but manually quality-check them before drawing significant conclusions.
Additionally, we cannot let automatic benchmark creation prevent humans from investing more thought and effort into human data curation, which can serve complementary functions.

## Appendix C More details on Experimental Setup

Recall in § 5.1, we compare AutoBencher with human-generated benchmarks as baseline. Here is the detailed HumanBench for each domain:

For history, we compare with $4$ history subjects high school world history, prehistory, high school european history, high school us history.

For economy, we compare with $4$ subjects: high school microeconomics, econometrics, high school macroeconomics, marketing.

For science, we compare with $7$ subjects: high school physics, college physics, college chemistry, high school chemistry, high school biology, college biology, astronomy.

For the LMs $\texttt{LM}\in\mathcal{M}$ that we evaluate. We list their sources with proper citations in Appendix G.

To compare the response of a LM to the dataset label, we use a language model (i.e., gpt-4-0125-preview) to judge the correctness of the model-generated response, and output reasons for the judgment.

## Appendix D Rank Analysis

We report the models’ ranking and their respective accuracies on AutoBencher datasets in Table 6, Table 5. We highlight the models that perform worse than expected (in red), and the models that perform better than expected (in green).

We also provide the ranking results of our human study in Table 4.

**Table 4: The model ranking results of the human study. We highlight the very significant novel trends. We use red to label models that perform worse than expected, and green to label models that perform better than expected.**
|  | History | Science | Economy |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | pred | gold | avg | pred | gold | avg | pred | gold | avg |
| claude-3-opus-20240229 | 1 | 3 | 2 | 2 | 2 | 1 | 5 | 5 | 1 |
| gpt-4-turbo-2024-04-09 | 2 | 2 | 1 | 1 | 1 | 3 | 1 | 1 | 2 |
| claude-3-sonnet-20240229 | 3 | 1 | 3 | 4 | 3 | 2 | 7 | 2 | 3 |
| claude-2.0 | 5 | 4 | 4 | 3 | 4 | 4 | 4 | 3 | 4 |
| Mixtral-8x7B-Instruct-v0.1 | 4 | 9 | 5 | 5 | 6 | 5 | 6 | 6 | 5 |
| gemini-pro | 6 | 8 | 6 | 6 | 5 | 7 | 3 | 15 | 6 |
| gpt-3.5-turbo-0613 | 7 | 7 | 7 | 7 | 7 | 6 | 2 | 8 | 7 |
| openchat-3.5-0106 | 8 | 5 | 8 | 8 | 8 | 8 | 10 | 7 | 8 |
| zephyr-7b-beta | 10 | 11 | 10 | 10 | 10 | 9 | 9 | 12 | 9 |
| OpenAGI-7B-v0.1 | 9 | 6 | 9 | 9 | 9 | 10 | 8 | 4 | 10 |
| Mistral-7B-Instruct-v0.1 | 12 | 16 | 11 | 11 | 11 | 11 | 12 | 13 | 11 |
| vicuna-7b-v1.5 | 15 | 14 | 12 | 12 | 12 | 12 | 11 | 10 | 12 |
| Llama-2-7b-chat-hf | 16 | 15 | 15 | 14 | 13 | 14 | 13 | 11 | 13 |
| Xwin-Math-7B-V1.0 | 14 | 10 | 14 | 15 | 14 | 13 | 15 | 16 | 14 |
| WizardMath-7B-V1.0 | 13 | 13 | 13 | 16 | 16 | 15 | 14 | 14 | 15 |
| alpaca-7b | 11 | 12 | 16 | 13 | 15 | 17 | 16 | 9 | 16 |
| gpt-neo-2.7B | 17 | 17 | 17 | 17 | 17 | 16 | 17 | 17 | 17 |

**Table 5: The model ranking results of the datasets constructed by AutoBencher. We highlight the very significant novel trends. We use red to label models that perform worse than expected, and green to label models that perform better than expected.**
|  | History | Economy | Science |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | pred | gold | avg | pred | gold | avg | pred | gold | avg |
| claude-3-opus-20240229 | 1 | 2 | 2 | 2 | 3 | 1 | 1 | 2 | 1 |
| gpt-4-turbo-2024-04-09 | 2 | 1 | 1 | 1 | 4 | 2 | 4 | 3 | 3 |
| claude-3-sonnet-20240229 | 4 | 4 | 3 | 10 | 2 | 3 | 2 | 1 | 2 |
| claude-2.0 | 5 | 6 | 4 | 4 | 5 | 4 | 3 | 7 | 4 |
| Mixtral-8x7B-Instruct-v0.1 | 3 | 7 | 5 | 6 | 12 | 5 | 5 | 5 | 5 |
| gemini-pro | 6 | 16 | 6 | 5 | 15 | 6 | 7 | 14 | 7 |
| gpt-3.5-turbo-0613 | 7 | 3 | 7 | 3 | 7 | 7 | 8 | 6 | 6 |
| openchat-3.5-0106 | 8 | 9 | 8 | 8 | 1 | 8 | 14 | 8 | 8 |
| zephyr-7b-beta | 11 | 11 | 10 | 9 | 14 | 9 | 10 | 13 | 9 |
| OpenAGI-7B-v0.1 | 9 | 5 | 9 | 7 | 9 | 10 | 6 | 4 | 10 |
| Mistral-7B-Instruct-v0.1 | 12 | 10 | 11 | 11 | 8 | 11 | 13 | 12 | 11 |
| vicuna-7b-v1.5 | 15 | 12 | 12 | 12 | 6 | 12 | 11 | 9 | 12 |
| Llama-2-7b-chat-hf | 14 | 13 | 15 | 13 | 11 | 13 | 12 | 10 | 14 |
| Xwin-Math-7B-V1.0 | 16 | 14 | 14 | 14 | 16 | 14 | 16 | 16 | 13 |
| WizardMath-7B-V1.0 | 13 | 15 | 13 | 15 | 10 | 15 | 15 | 15 | 15 |
| alpaca-7b | 10 | 8 | 16 | 16 | 13 | 16 | 9 | 11 | 17 |
| gpt-neo-2.7B | 17 | 17 | 17 | 17 | 17 | 17 | 17 | 17 | 16 |

**Table 6: LMs’ accuracy on datasets constructed by AutoBencher.**
|  | History | Economy | Science |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Models | AutoBench | MMLU | AutoBench | MMLU | AutoBench | MMLU |
| claude-3-opus-20240229 | 0.51 | 0.93 | 0.64 | 0.88 | 0.50 | 0.81 |
| gpt-4-turbo-2024-04-09 | 0.53 | 0.93 | 0.62 | 0.85 | 0.50 | 0.69 |
| claude-3-sonnet-20240229 | 0.42 | 0.88 | 0.67 | 0.78 | 0.50 | 0.71 |
| claude-2.0 | 0.42 | 0.85 | 0.62 | 0.78 | 0.42 | 0.68 |
| Mixtral-8x7B-Instruct-v0.1 | 0.40 | 0.85 | 0.55 | 0.76 | 0.43 | 0.68 |
| gemini-pro | 0.28 | 0.84 | 0.48 | 0.75 | 0.26 | 0.60 |
| gpt-3.5-turbo-0613 | 0.51 | 0.82 | 0.60 | 0.72 | 0.42 | 0.63 |
| openchat-3.5-0106 | 0.40 | 0.79 | 0.67 | 0.69 | 0.41 | 0.58 |
| zephyr-7b-beta | 0.35 | 0.72 | 0.48 | 0.66 | 0.30 | 0.58 |
| OpenAGI-7B-v0.1 | 0.42 | 0.77 | 0.55 | 0.66 | 0.44 | 0.57 |
| Mistral-7B-Instruct-v0.1 | 0.37 | 0.66 | 0.57 | 0.56 | 0.35 | 0.48 |
| vicuna-7b-v1.5 | 0.35 | 0.64 | 0.60 | 0.52 | 0.38 | 0.42 |
| Llama-2-7b-chat-hf | 0.33 | 0.57 | 0.55 | 0.50 | 0.37 | 0.38 |
| Xwin-Math-7B-V1.0 | 0.33 | 0.58 | 0.38 | 0.45 | 0.17 | 0.39 |
| WizardMath-7B-V1.0 | 0.30 | 0.59 | 0.55 | 0.44 | 0.20 | 0.37 |
| alpaca-7b | 0.40 | 0.37 | 0.55 | 0.35 | 0.35 | 0.28 |
| gpt-neo-2.7B | 0.26 | 0.25 | 0.26 | 0.27 | 0.09 | 0.31 |

## Appendix E AutoBencher Search Trajectory

In order to analyze AutoBencher, we provide intermediate search results of the AutoBencher. Figure 3, Figure 5 and Figure 4 show the search trajectory of AutoBencher for history, economy, and science domains. Specifically, we report the evaluation topics that were explored and their respective accuracy as a Star plot.

Figure: Figure 3: Search trajectories of AutoBencher (history) with different $\texttt{LM}_{\text{test}}$. It shows the evaluation topics that are explored and their respective accuracy as a star plot.
Refer to caption: x3.png

Figure: Figure 4: Search trajectories of AutoBencher (science) with different $\texttt{LM}_{\text{test}}$. It shows the evaluation topics that are explored and their respective accuracy.
Refer to caption: x6.png

Figure: Figure 5: Search trajectories of AutoBencher (economy) with different $\texttt{LM}_{\text{test}}$. It shows the evaluation topics that are explored and their respective accuracy.
Refer to caption: x9.png

Figure: Figure 6: The histogram of accuracy for all topics explored in a AutoBencher run. The three rows are economy, science, and history respectively.
Refer to caption: x12.png

## Appendix F More Results on Separation and Headroom

In Figure 7, we show the Pareto frontier of the two difficulty metrics: Sep and Difficulty. Each orange stars represent datasets constructed by AutoBencher, and each blue dot represents an MMLU subject. Datasets constructed by AutoBencher are mostly at the Pareto frontier, outperforming MMLU subjects in both metrics.

Figure: Figure 7: We show the Pareto frontier of the two difficulty metrics: Sep and Difficulty. Each orange stars represent datasets constructed by AutoBencher, and each blue dot represents an MMLU subject. Datasets constructed by AutoBencher are mostly at the Pareto frontier, outperforming MMLU subjects in both metrics.
Refer to caption: x15.png

## Appendix G Asset License

We list the license associated with each model below:

- •
gpt-4-turbo-2024-04-09 []: Proprietary
- •
gpt-3.5-turbo-0613 []: Proprietary
- •
claude-3-sonnet-20240229 []: Proprietary
- •
claude-3-opus-20240229 []: Proprietary
- •
claude-2.0: Proprietary
- •
Mixtral-8x7B-Instruct-v0.1 []: Apache 2.0
- •
Mistral-7B-Instruct-v0.1 []: Apache 2.0
- •
gemini-pro []: Proprietary,
- •
OpenAGI-7B-v0.1 []: Apache-2.0
- •
vicuna-7b-v1.5 []: Llama2,
- •
Llama-2-7b-chat-hf []: Llama2,
- •
Xwin-Math-7B-V1.0 []: Llama2,
- •
WizardMath-7B-V1.0 []: Llama2,
- •
gpt-neo-2.7B []: MIT
- •
alpaca-7b []: CC by NC 4.0
- •
zephyr-7b-beta []: MIT
- •
openchat-3.5-0106 []: Apache-2.0

## Appendix H Our License

We release our work under an MIT license.

## Appendix I Details of Human Study

Recall in § 6.3, we conduct a human study to verify the trends found by AutoBencher still holds for the human-constructed dataset. For this human study, the instruction is to generate a set of question-answer pairs given a topic $c$ (e.g., Fordism). The annotator may use resources from Wikipedia (e.g., Wikipedia articles on Fordism), and also other linked Wikipedia pages. The annotator should generate roughly $50$ questions per topic, and the questions should be challenging. Additionally, each question should be answerable by a domain expert. The generated answer for each question should be correct, and concise. If the question is open-ended, the answer should then cover as many correct responses as possible.

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see Weakesses.
soundness: 3
strengths: 1. The problem of automatic benchmark generation in a guided manner is an important one. While LMs have been used as judges to automatically evaluate other LM's answers, this work proposes using LMs to also generate questions.

2. The problem is formalized and packaged in an elegant and extensible way in the AutoBencher framework, and two important instances of the framework are studied.  The two-step division (first generate topics and then generate datasets per topic) is especially novel and effective.

3. The proposed approach has two primary novel aspects: the use of a tool (such as a Wikipedia knowledge database or a Python library for mathematical calculations) to avoid the evaluating LM from generating answers that could be incorrect, and an algorithm to generate the benchmark in a guided rather than brute-force manner.

4. The empirical evaluation shows the promise of the proposed approach in generating datasets that are more novel and difficult than even hand-crafted ones like MMLU.
summary: This paper presents AutoBencher, a declarative framework for automatic benchmark construction, and use it to scalably discover novel insights and vulnerabilities of existing language models. Specifically, it is instantiated in two scenarios: 1) capability evaluation and 2) safety evaluation.  In each scenario, a set of desirable characteristics is first formally defined. For capability evaluation, these are salience, difficulty, separability, and novelty. For safety evaluation, they are harmfulness and attack success rate.  Then, a language model is used to automatically construct descriptions of topics along with datasets in those topics, where a dataset is a set of (question, answer) pairs. Empirical results using GPT-4 as the evaluator show that the created datasets are on average 27% more novel and 22% more difficult than existing benchmarks. AutoBencher also helps identify specific gaps not captured by existing benchmarks: e.g., Gemini-Pro has knowledge gaps on Permian Extinction and Fordism while GPT-4o fails to decline harmful requests about cryptocurrency scams.
weaknesses: 1. AutoBencher is currently stand-alone. The paper would be stronger if it integrated AutoBencher into existing popular evaluation frameworks like Stanford's HELM or HuggingFace's Open LLM. Adoption of AutoBencher by one of these frameworks would make a more convincing case for its usefulness and viability.

2. There seems to be a mismatch between the capabilities of the evaluating LM and the evaluated LMs: the former has access to tools whereas the latter does not. The paper does not make a convincing case why future LMs should be expected to have such capabilities without using tools themselves.

3. The evaluation is rather limited. I would expect a benchmark/evaluation focussed paper to be more comprehensive and derive more insights than those currently presented. For instance, AutoBencher currently only generates one-turn (question, answer) pairs; it would be interesting to see it extend to multi-turn data, chain-of-thought data, etc. Another direction to extend it could be in the domain of multi-modality.

4. I found the safety evaluation less convincing than the capability evaluation. While the paper does use recent baselines such as XSTest and HarmBench, it would be more convincing if the paper would report on how AutoBencher could be integrated into a mainstream framework for safety evaluation and discuss challenges that were overcome in such an integration (this is related to item 1 above but is more specific to safety evaluation).

---

confidence: 2
contribution: 3
initial_rating: 3
presentation: 3
questions: - In Section 3.1 on separability, are you taking the mean over $acc(\cdot) \in V_c$? If so, this should be clarified. Additionally, I am not convinced that high deviation alone ensures that rankings between any two models, $LM_i$ and $LM_j$, with $i \ne j$, are robust to noise.
- Using vector notation could help make the equations clearer.
- In Section 3.1 on novelty, it seems that there is a coefficient for each model being evaluated. Could you elaborate on why there are coefficients for each model rather than for each dataset?
- How are the salience and harmfulness binary variables obtained in practice?
- I am unclear about the procedure for qualitative assessment in Section 6.3. What does "high quality" mean, and how many question-answer pairs were sampled? What do the "ranks" in this section refer to?
- What language model was used to generate Table 1? Adding this information to the caption would improve clarity.
- Are there any ablation experiments on the following: the language model used, the criteria for evaluating dataset descriptions, and the impact of including privileged information?
- Are the human benchmarks also grouped into distinctive categories and descriptions similar to those proposed by the language model?
soundness: 3
strengths: - The paper conducts experiments across a wide range of knowledge domains.
- The presentation is generally clear, with well-structured sections and a logical flow.
- Equations and figures are used effectively to enhance clarity.
- The automatic generation of benchmarks is a topic of strong interest to the community.
- The proposed method demonstrates significant performance gains over human-generated benchmarks, particularly in terms of novelty, difficulty, and separability metrics as defined in the paper.
summary: The paper presents a method, Autobencher, to automatically construct benchmarks for evaluating language model (LM) capabilities and safety. Specifically, based on predefined criteria, the authors prompt an LM to propose dataset descriptions and generate questions, which a candidate LM then attempts to answer. The dataset is subsequently scored according to these criteria, enabling the selection of high-quality dataset descriptions. The authors aim to demonstrate that Autobencher generates dataset topics/descriptions that are both more novel and more challenging than existing benchmarks.
weaknesses: - It is not entirely clear to me what methodological novelties the paper introduces in dataset construction, aside from its use of retrieval during generation. Optimizing for various criteria objectives is not a particularly significant contribution.
- The introduction and abstract could benefit from revisions to be more specific and descriptive.
- The method and criteria rely on accuracy scores computed across both existing models and preexisting datasets, which can be computationally intensive, especially if generating multiple datasets across a large set of models.
- Additionally, language models (LMs) can exhibit high variance across different runs. A robust scoring criterion should account for this issue by incorporating variance between different runs, both for generating questions and answers. Introducing a statistical method to handle this variance would strengthen the contribution.
- Ensuring that the dataset description is clear and unharmful does not guarantee that all generated questions are equally clear and unharmful. For instance, a dataset labeled as "Nuclear technology" might contain questions of varying levels of harmfulness.
- There are concerns regarding the practicality of the method. While the paper claims high automation, many of the criteria still seem to require potentially time-intensive manual crafting. Furthermore, LM generations can be compute-intensive; a discussion on the computational resources required for these generations would be beneficial.
- Additional ablation studies could help demonstrate the impact of various components of the method.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. To the knowledge of this reviewer, it should be possible to have a hyper-/hyponymy graph of Wikipedia topics. Did you consider using such a graph, or a similar topic relationship graph to further guide the adaptive search?
2. The optimisation problem for case (1) is multi-objective, is there a reason why multi-objective optimisation was not employed and instead the loss is linearised by adding hyperparameters? (evolutionary/population-based methods?)
3. Related to W1, how is the math/science salience set constructed? Unless I missed it in the paper, the inclusion-exclusion criteria were not clear or indeed the exact methodology. Did you perform, for example, open card sorting or a similar methodology?
soundness: 4
strengths: + On novelty, the approach generates human interpretable topics where model ranks exhibit surprising results.
+ On safety, the qualitative examples align with the experience of this reviewer when trying to manually jail-break LLMs: pose the question as a hypothetical or philosophical debate; this vector being auto-discovered is encouraging.
+ Well-executed research methodology with manual validation of the discovered benchmarks
summary: The authors propose to use an LM-guided adaptive search to construct benchmarks that optimise (1) difficulty (benchmark headroom), separability (existing model variance), and novelty (rank correlation with other benchmarks) and (2) attack success rate (when trying to extract harmful information from models). They demonstrate the need for this by showing that existing benchmarks exhibit low values in novelty (especially) even for benchmarks such as MMLU where there is sufficient headroom. Further, they use privileged information to enable a weaker LM to assess a stronger LM, use translation tools to enable multi-lingual data and employ source code/python to evaluate numerical and symbolic expressions. When optimising, the authors employ an adaptive search strategy that uses the history of explored subjects to guide the new candidates restricted to a salience set. To assess the quality of the generated data, the authors employ Mechanical Turk and find sufficiently low error rates with high salience on the questions. The value of adaptive search is demonstrated via an ablation on the use of the history vector. Both regimes demonstrate improvement over HumanEval on the desired metrics.
weaknesses: - The approach to the math and science categories suggests an open vocabulary problem that is not clearly tackled. For other categories, this is tackled via Wikipedia and a popularity metric.
- The translation induces an issue for low-resource language: such languages are both less likely to be tackled by LLMs and by translation tools, creating a catch-22. (I feel it would be sufficient to acknowledge the issue as a limitation since tackling the issue requires significant manual effort, future work can consider specialised models for specific language pairs.)

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 4
questions: 1- How does AutoBencher differ from previous work on automatic benchmarking, such as that in [2]?

2- Why is a comparison with the baseline included?





[1] Taskbench: Benchmarking large language models for task automation.
soundness: 3
strengths: 1-The paper is well-written and very easy to understand.

2- The paper provides an in-depth empirical evaluation, uncovering vulnerabilities and assessing model performance. It also includes results from human evaluations to further validate the quality and relevance of the generated benchmark.

3- AutoBencher automatically constructs datasets, making it highly scalable and reducing reliance on costly human labor. This is useful for evaluating LLMs across various domains.
summary: This paper introduces AutoBencher, a declarative framework that automatically generates benchmarks to evaluate language model performance by defining specific desiderata, such as difficulty, novelty, and safety. The proposed approach, AutoBencher, leverages language models to iteratively construct benchmarks that reveal weaknesses and safety vulnerabilities in LLMs.
weaknesses: 1- The details of the qualitative analysis are missing.

2- AutoBencher uses GPT-4 as the evaluator, which may introduce potential bias in datasets that could favor LLMs from the same family (e.g., OpenAI's LLMs).

Meta-review: The paper "AutoBencher: Towards Declarative Benchmark Construction" introduces a novel framework, AutoBencher, for automatic benchmark construction using a declarative approach. AutoBencher leverages large language models (LLMs) to iteratively generate datasets optimized for specific desiderata, such as difficulty, novelty, and safety, revealing weaknesses and vulnerabilities in existing LLMs. The framework is applied to two primary scenarios: capability evaluation (e.g., math, multilinguality, knowledge) and safety evaluation (e.g., harmful prompts). Empirical results demonstrate that AutoBencher generates datasets that are more novel and challenging compared to existing benchmarks, identifying specific gaps not captured by current evaluations, such as knowledge deficiencies in models like Gemini-Pro and safety vulnerabilities in GPT-4o.

The primary contributions of the paper are:
- AutoBencher Framework : A declarative approach to benchmark construction that optimizes for specific desiderata (e.g., difficulty, novelty, safety) using LLMs, providing a scalable method to discover model weaknesses and vulnerabilities.
- Empirical Validation : Demonstrates that AutoBencher-generated datasets are 27% more novel and 22% more difficult than existing benchmarks, identifying specific knowledge gaps and safety vulnerabilities in state-of-the-art models.
- Asymmetric Evaluation : Utilizes privileged information (e.g., tools, Wikipedia data) to create challenging benchmarks, highlighting the importance of asymmetry in evaluation.
- Safety Evaluation : Identifies harmful prompts that existing models fail to decline, contributing to safety evaluation in LLMs.
Open-Source Integration : Submission of AutoBencher capabilities and safety datasets to HELM, enhancing its practical applicability.


#### Weaknesses
1. **Lack of Integration (A8iq)**: Initial absence of integration with popular frameworks like HELM, later addressed by authors.
2. **Capability Mismatch (A8iq)**: Concerns about the evaluator LMs' use of tools not available to evaluated models, somewhat mitigated by authors' clarifications.
3. **Limited Scope (A8iq)**: Focuses on single-turn QA pairs, with multi-turn and multi-modal datasets suggested as future work.
4. **Safety Evaluation (A8iq)**: Seen as less convincing, with integration into safety frameworks proposed but not empirically validated.
5. **Methodological Novelty (bP3L, oakz)**: Perceived as combining existing techniques without significant innovation, though authors highlight optimization-based construction and asymmetry.
6. **Practicality and Costs (bP3L)**: Concerns about computational intensity and manual effort, addressed with cost analyses.
7. **Variance (bP3L)**: Initial lack of variance analysis, resolved in rebuttal.
8. **Human Effort (bP3L, oakz)**: Manual salience judgments raise agreement issues, with solutions proposed but not implemented.
9. **Evaluator Bias (vFMj)**: Potential bias from using GPT-4 as evaluator, addressed with ablations.
10. **Qualitative Details (vFMj)**: Lack of detailed qualitative analysis, resolved in rebuttal.


The authors adequately address most of the questions during rebuttal, which is good.

Decision: Accept (Poster)