How to Get Your LLM to Generate Challenging Problems for Evaluation

Authors: Arkil Patel, Siva Reddy, Dzmitry Bahdanau
Primary area: datasets and benchmarks
Keywords: Evaluation, Synthetic data, Benchmarking, Question Answering, Code Generation, Math Reasoning

Abstract:
The pace of evolution of Large Language Models (LLMs) necessitates new approaches for rigorous and comprehensive evaluation. Traditional human annotation is increasingly impracticable due to the complexities and costs involved in generating high-quality, challenging problems, particularly for tasks such as long-context reasoning. Moreover, the rapid saturation of existing human-curated benchmarks by LLMs further necessitates the need to develop scalable and automatically renewable evaluation methodologies. In this work, we introduce **CHASE**, a unified framework to synthetically generate challenging problems using LLMs without human involvement.  For a given task, our approach builds a hard problem in a bottom-up manner from simpler components. Moreover since we want to generate synthetic data for evaluation, our framework decomposes the generation process into independently verifiable sub-tasks, thereby ensuring a high level of quality and correctness. We implement CHASE to create evaluation benchmarks across three diverse domains: document-based question answering, repository-level code completion, and math reasoning. The performance of state-of-the-art LLMs on these synthetic benchmarks lies in the range of 40-60\% accuracy, thereby demonstrating the effectiveness of our framework at generating hard problems. Our experiments further reveal that the Gemini models significantly outperform other LLMs at long-context reasoning, and that the performance of all LLMs drastically drops by as much as 70\% when we scale up the context size to 50k tokens.

Decision: reject
Reviewer scores: 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1729127857844

thread_type:
official_review

<thread_object>
creator: reviewer_yxkn

<content>
summary:
This paper introduces CHASE (CHallenging AI with Synthetic Evaluations), a framework for generating challenging evaluation benchmarks using large language models (LLMs). The authors implement CHASE to create benchmarks in three domains: document-based question answering, repository-level code completion, and math reasoning. Experiments with 15 LLMs show that the generated benchmarks are challenging, with even top models achieving only 40-60% accuracy across domains. The authors demonstrate CHASE's utility in differentiating between state-of-the-art models and revealing performance drops with increasing context length. They argue that this approach offers advantages in scalability, renewability, and ability to evaluate tasks difficult for humans to assess, while providing high-quality, challenging problems for LLM evaluation.

strengths:
1. The experiments are comprehensive, with a good set of LLMs covering representative proprietary and open-source models. 
2. The paper is well-written, which clearly describes the methods, experiments and results.

weaknesses:
1. Although overall I believe it is valuable to explore data synthesis for benchmark construction, I think the authors should be more careful in selecting appropriate settings. I think the most important motivation for this paper is that it is expensive and sometimes impracticable to create benchmarks with challenging problems. However, in some settings present in the paper, I feel that this may not be the case. For example, SWE-bench [1] also focuses on repo-level code generation, and they take existing Github issues as queries, and the modifications made by real users as the ground truth. The current state-of-the-art performance is only 43% in the leaderboard, which indicates its difficulty. Compared to CHASE-CODE, I think the pipeline used in SWE-bench is a better way to collect repo-level code generation data.
2. To demonstrate that this pipeline is scalable, I think it is important to generate data of large size and apply it to training. If the API cost is a concern, I think the authors can use open-source models, e.g., Llama-70B. 
3. Typo in Figure 1: Jill has 12 pens in the bottom right corner.  
4. In line 443-444, I don’t quite understand why better performance of models different from the generator and verifier can indicate better data quality.
5. One advantage of the CHASE claimed by authors is to mitigate data contamination, but I think this may not be a big concern for challenging benchmarks that involve intensive reasoning. For example, even if codellama [2] has been intensively trained on Github data, its performance is still low on SWE-bench (which uses the Github data).

[1]. Jimenez, Carlos E., et al. "Swe-bench: Can language models resolve real-world github issues?." arXiv preprint arXiv:2310.06770 (2023). \
[2]. Roziere, Baptiste, et al. "Code llama: Open foundation models for code." arXiv preprint arXiv:2308.12950 (2023).

questions:
1. How to organize functions into files to build repositories from scratch in CHASE-CODE?
2. Could you specify more details on rejection sampling?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730487614517

thread_type:
official_review

<thread_object>
creator: reviewer_gkcy

<content>
summary:
To address the high cost of manually labeled data and the low quality of synthetic data, this paper proposes the CHASE framework. CHASE is a framework for automatically generating QA pairs, code problems, and math problems. It adopts an innovative bottom-up structure and divides the overall task into individually verifiable sub-tasks. This allows a seed problem to progressively increase in difficulty through multiple rounds of generation and verification. Experimental results show that the data generated by the CHASE have a certain degree of difficulty.

strengths:
- The problem addressed by this paper is critical to the evaluation of current LLMs -- the lack of comprehensive and challenging datasets.
- The paper is well-structured, with comprehensive appendices, such as a detailed list of prompts used in CHASE.
- This paper presents a novel paradigm for data construction, which may have significant potential in the field of synthetic data.

weaknesses:
- Some issues with the details of the paper. For example, in the main figure (Figure 1), the bottom-right corner should say "12 pens" instead of "18 pens."
- The current dataset is relatively small, which may result in a high degree of randomness in evaluation results when using this dataset.
- The experiments are not sufficiently thorough. Some experimental designs lack strong motivation, and there is a lack of experiments that demonstrate the advantage of CHASE over other synthetic data generation methods.

questions:
1. Why does Figure 1 only provide an overview of constructing CHASE-QA and CHASE-Math, but not CHASE-Code? I believe all three should be at the same hierarchical level.
2. Without human verification, how can we ensure that the data in the CHASE-QA, CHASE-Code, and CHASE-Math datasets are correct? Is there a possibility that the ground truth/golden answer in the dataset themselves are incorrect?
3. In lines 333-340, you mentioned that approximately 33% of the data was filtered out through sampling and self-consistency, and subsequent experiments (e.g., Table 2) suggest that CHASE-QA generates more challenging data. I think it is unconvincing. If the 33% of the data were added back, how would the experimental results change? Would you still claim that CHASE-QA is a more challenging dataset?
4. From the examples given in the paper, CHASE-Math seems to concatenate a series of atomic problems. Intuitively, if the tested LLMs reason and calculate sentence by sentence, the accuracy may be significantly higher than under your current naive prompt. Could you elaborate further on how CHASE-Math is more challenging, given the point I raised?
5. What is the motivation behind the experiments in lines 469-477 and lines 486-493? In my understanding, the "Impact of context size" is not the focus of this paper. Also, the experiment in lines 486-493 only fine-tuned weaker models. Would the same conclusion apply to fine-tuning stronger models?
6. Could you provide some comparative experiments between the CHASE dataset and other synthetic datasets, such as a comparison between CHASE-QA and existing long-context benchmarks?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730609370640

thread_type:
official_review

<thread_object>
creator: reviewer_cavc

<content>
summary:
The authors introduce CHASE, a unified framework to synthetically generate challenging problems 
using LLMs without human involvement. For a task that is given, the approach builds a hard problem
 in a bottom-up manner from simpler components. It decomposes the generation process into
independently verifiable sub-tasks to ensure a high level of quality and correctness.

CHASE is designed to address two challenges that the authors state succinctly on pages 1 and 2:
first, how can it be used to create hard and realistic problems, and secondly, how can it be used
to automatically verify the correctness of the generated data? This second challenge is especially
prevalent in other work of this nature that is attempting to construct synthetic evaluation benchmarks.

strengths:
--The paper is written at an impressive quality, especially the figures and the elucidation of the problem's motivation and challenges. 
--The authors consider three sufficiently diverse tasks and benchmarks to showcase the utility of their approach.
--The results are fairly compelling, and the benchmark indeed succeeds in yielding performance drops even from advanced models.

weaknesses:
--Experimental results could have been deeper in the main text. It is for this reason that I am not inclined to give the paper a stellar rating. 
--The approach is simple and has some nice properties, but I am not too sure about its sensitivity and robustness. I felt inadequate attention was paid to this in the paper.

questions:
None.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732413504962

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request to read author responses and adjust scores

comment:
Dear reviewers,

We have provided detailed responses to address your concerns and questions. Further, we have carried out many new experiments to support our claims and arguments. Since there are only 3 more days of discussion left, we request you to kindly look at our responses. If your concerns are addressed, we kindly ask you to increase your score. If not, we encourage you to respond to us and engage in a discussion.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733287989656

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Summary of Discussion and Final Comments

comment:
**Reviewer CAvC**

The reviewer did not participate in the discussion.
In their review, they made a vague comment about “not being sure about sensitivity and robustness”. This statement is not backed by any concrete point or weakness and we have responded by explaining how our method and experiments are actually quite robust (we showed it works for multiple domains, it is scalable, works with weaker LLMs, etc).
Further, we highlight that **the reviewer had initially given a score of 8, and then lowered it to 6 on the day the reviews were released without changing the review text and without waiting for our response**. This behaviour is quite unethical and we request the AC to consider their previous score in their deliberation.

**Reviewer gKcy**

We believe the reviewer has some unfounded concerns which we have sufficiently addressed in our responses.

**Size of data.** As already explained in detail in our responses, for long-context benchmarks, ~500 data points is the standard size. Asking for more data than that is against the spirit of open and accessible science since it will be prohibitively expensive to run models on the benchmark. Moreover, we have provided evidence of popular benchmarks and past papers published at ICLR whose sole focus is proposing a benchmark of just ~200-700 examples. Lastly, the benchmarks are not the main contribution of this paper, rather it is the underlying approach that was able to automatically generate the data. Hence, we strongly believe this concern to be unreasonable.

**Correctness.** We have already clarified that the generated data for code and math domains is completely correct. There is a possibility of a small percentage (~6%) of errors in the QA benchmark. We address this by providing softer metrics of evaluation (see detailed response) and by noting the large gaps between performances of models. Further, we highlight that the level of correctness of generated data achieved by our method is far superior to other baselines.

**Comparison of difficulty.** We have provided comparison of performance of LLMs on our benchmarks against other difficult benchmarks in the respective domains. The reviewer has failed to specify what other kind of “difficulty analysis” they were hoping to see. We believe it is clear from our results that our generated benchmarks are indeed quite challenging.

The reviewer had misunderstood some aspects of our paper such as the context size and fine-tuning experiments, which we have clarified in detail.

**Reviewer yxKN**

The reviewer has raised two concerns which we believe are not valid. We have already summarized our arguments for the size of data. The other point raised by this reviewer is that our experiments pertaining to the code domain do not reflect realistic scenarios. We believe this point is completely baseless and we have provided a detailed explanation in our response. Further note that the reviewer has no other standing concerns against our method or our experiments with the other 2 domains.

Overall, we feel that **reviewer gKcy and yxKN’s scores are quite disproportionate corresponding to their standing concerns**. They have provided no concrete counter-arguments to our responses. This work provides the first comprehensive approach to generate challenging, high-quality synthetic data for evaluation across multiple domains. The **reviewers agree that the problem that we study is critical** (CAvC, gKcy), the **paper is well-written** (CAvC, gKcy, yxKN), the **approach is novel** (gKcy), the **experiments are comprehensive** (yxKN), and the **results are compelling** (CAvC). None of the reviewers have pointed out any technical weaknesses in our proposed method or the main results (Table 1), which are the main contributions of this work.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper presents a systematic approach to synthesize challenging compositional problems for LLM evaluation in math, coding and general question answering. The core idea is to take a “bottom-up” approach to gradually compose simpler sub-tasks that are easier to verify to form more challenging benchmark problems. The authors showed that state-of-the-art LLMs only attain 40%-60% accuracy on the generated benchmarks, and claimed that such evaluation results demonstrated the effectiveness of the proposed approach in generating hard evaluation problems.

**Strengths:**

* The paper is generally well written (CAvC, yxKN) and well-structured with comprehensive appendices (gKcy). The method also has a clear motivation (CAvC, yxKN). “The problem addressed by this paper is critical to the evaluation of LLMs” (gKcy).

* The paper presents a “novel paradigm for data construction” (gKcy). The authors demonstrated the applicability of the benchmark synthesis approach on three distinct domains (CAvC).

* Comprehensive evaluation results “covering representative proprietary and open-source models” (yxKN). Reviewer CAvC also found the results “fairly compelling” as the benchmark “indeed succeeds in yielding performance drops even from advanced models”.

**Weaknesses:**

After the rebuttal period, there are several issues that are yet to be addressed.

First, while the results suggest that LLMs do not perform well on this dataset, the paper lacks intrinsic evaluation on the complexity and difficulty of the synthesized problems (gKcy). The authors could consider using well-established domain-specific metrics to measure problem complexity, such as using the number of lines or the size of ASTs to approximate program complexity for CHASE-CODE. As a general suggestion, while the reviewer did not specify the exact metrics to use in their review, the authors would have been more proactive in the response period and report any potentially reasonable metrics in order to better address the reviewer’s concern.

Another potential issue is the quality of the synthesized problems (gKcy). While we appreciate the authors’ effort in reviewing O(30 - 100) problems in the three datasets, given the total size of each dataset (around 500), it would be more convincing to carefully review at least 20% examples (i.e., 100 problems) in order to reach any statistically significant conclusions. The authors already demonstrated that the CHASE-MATH dataset is in relatively decent quality by reviewing 100 examples, and I strongly suggest the authors review a similar number of tasks in the other two datasets.

Next, as flagged by Reviewer yxKN, since there already exist high-quality, challenging datasets for repository-level code editing (SWE-bench), it is less clear that from an empirical perspective, what additional values would the new synthetic CHASE-Code dataset bring to the code LLM community. While we acknowledge that CHASE-Code focuses more on *generation* Instead of code editing, there are existing high-quality repo-level code generation benchmarks derived from real repository-level context, such as DevEval (https://arxiv.org/abs/2405.19856). I totally agree with the authors that the value of this paper lies more in the proposed data synthesis approach, instead of the datasets produced. However, a significant portion of this paper also comes from the value of the benchmarks the proposed approach synthesized, it is hard for me to assess the practical value and implications of CHASE-Code. In the future, maybe the authors could take the reviewer’s suggestion, and explore additional use cases or domains in coding where the proposed method could synthesize benchmarks with more practical value to the practitioners. On the other hand, the authors could also consider exploring whether your synthetic datasets like CHASE-Code could complement or correlate with existing benchmarks like SWE-Bench or LMSYS coding, or other datasets that require more laborious human efforts. In this way, the authors could more clearly demonstrate the value of CHASE as it provides a significantly more cost-effective approach to create novel code benchmarks.

—

Finally, I wanted to note that the authors’ attitude during the rebuttal period is unprofessional and might potentially violate ICLR code of ethics (“Researchers must show respect for colleagues, research participants …”). It is totally understandable that you may find reviewers might take a different perspective when judging your work, and it is critical to professionally resolve any concerns or misunderstandings via peaceful communication in a respectful manner. While I did not take this into consideration when rating your work this time, I wish the authors could bear this in mind in the future.