Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing

Authors: Han Jiang, Xiaoyuan Yi, Zhihua Wei, Shu Wang, Xing Xie
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Large Language Models, Human values, Human ethics, Evaluation, Adaptive testing

Abstract:
Warning: this paper contains model outputs exhibiting unethical information.

Large Language Models (LLMs) have achieved significant breakthroughs, but their generated unethical content poses potential risks. Measuring value alignment of LLMs becomes crucial for their regulation and responsible deployment. Numerous datasets have been constructed to assess social bias, toxicity, and ethics in LLMs, but they suffer from evaluation chronoeffects, that is, as models rapidly evolve, existing data becomes leaked or undemanding, overestimating ever-developing LLMs. To tackle this problem, we propose GETA, a novel generative evolving testing approach that dynamically probes the underlying moral baselines of LLMs. Distinct from previous adaptive testing methods that rely on static datasets with limited difficulty, GETA incorporates an iteratively-updated item generator which infers each LLM's moral boundaries and generates difficulty-tailored testing items, accurately reflecting the true alignment extent. This process theoretically learns a joint distribution of item and model response, with item difficulty and value conformity as latent variables, where the generator co-evolves with the LLM, mitigating chronoeffects. We evaluate various popular LLMs with diverse capabilities and demonstrate that GETA can create difficulty-matching testing items and more accurately assess LLMs' values, better consistent with their performance on unseen OOD and i.i.d. items, laying the groundwork for future evaluation paradigms.

Decision: reject
Reviewer scores: 6, 6, 5, 6

Reviews and discussion:
<thread>
creation date:
1730629186180

thread_type:
official_review

<thread_object>
creator: reviewer_yuhe

<content>
summary:
The authors present a novel method called GETA (Generative Evolving Testing of vAlues) to generate test hard test instances and concurrently estimate the degree of conformity of the model to the values being tested. The motivation is tackling what they call the evaluation chronoeffect challege, i.e. the saturation of benchmark performance due to leakage or increased LLM performance. 

Their approach jointly trains of a Variational IRT (Item Response Theory) model (capturing conformity and item difficulty) and an item generator with a clever ELBO setup. Using a model as an item generator allows them to avoid having to rely on a large item pool to select from. The estimated conformity is directly used to assess model performance.

The authors compare their method with vanilla testing on the static evaluation and two different test selection approaches. They compare the correlation between their Value Conformity scores and the scores available in publicly available leaderboards, which are massive compared to the ~100 items considered in GETA evaluation.

strengths:
- The paper offers a novel combination of ideas from the literature, addressing a real issue.
- The joint modelling of item difficulty, model conformity and item generation is very neat and likely to be usable beyond the ethical value assessment
- The paper (including the appendix) is a very nice writeup that documents very thoroughly the approach
- If the experimental claims are to be trusted, the authors are mostly reproducing larger (but how much larger) leaderboards with ~100 instances.

weaknesses:
- The presentation is very dense, and because of this it's sometimes hard to get some details. For example, I failed to understand how is the AIG generator conditioned on item difficulty. See my suggestions in the "Questions" setting
- For a paper that discusses the chronoeffect evaluation challenge, there is little experimental discussion of time. I would have liked to see whether there would be any discernible difference when testing model before and after a cutoff date (e.g. a dataset release), with GETA vs SE. (The focus is on, so to speak, "interpolation", but this paper would benefit a lot from more focus on "extrapolation": identifying failure modes of current evaluations, and showing that their method alleviates or solves these issues)
- The claim that correlation with leaderboards is the best signal of evaluation quality is not the strongest. I understand that this is a hard problem, but it would be good to include some more discussion about it, i.e. on why the leaderboards are reliable at all.

questions:
- It would be **very** helpful to rework the notation a bit. The notation is at time confusing, e.g. having $y$ as the response score and $y^*$ as the ground truth answer. Other possible improvements are the double indexing in $y$, and the parameters denoted as $\theta$, $\psi$, $\phi$. Also the notation is very rich in symbols and the reader is seldom if ever reminded of what they mean. Note that none of this would be a problem in isolation, but all together this makes the paper needlessly hard to follow.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730763156526

thread_type:
official_review

<thread_object>
creator: reviewer_1zq8

<content>
summary:
This paper aims to measure the value alignment of language models using psychometric testing theory. In particular, it highlights that static datasets that measure bias, toxicity, and ethics of language models can be contaminated, as models are trained on the outputs. In response, the paper draws from methods in computerized adaptive testing to adaptively generate tests for the model. To generate tests, the paper uses an IRT model parameterized by a two-layer transformer to model the difficulty of an item, then selects items out of an item pool. The item pool is filled with “items” that are generated adaptively by fine-tuned Llama 3 8B. The paper then validates this method for testing model’s ethical judgements by comparing how it ranks eight different models, and argues that its method produces qualitatively better judgements, and matches existing leaderboards (like the Enkrypt AI safety leaderboard).

strengths:
* This paper helps address an important problem in language model evaluation — static benchmarks can enter subsequent training sets, and lose signal
* GETA can generate examples at different difficulty levels, circumventing some problems with adaptive adversarial evaluation that will almost always provide failures (and thus not provide useful features for comparison)
* The paper evaluates GEPA on multiple aspects of value alignment (bias, ethics, toxicity)

weaknesses:
* The primary weakness is the evaluation of GETA; GETA is evaluated either based on subjective intuition about how language models should rank on different value alignment tasks or exogenous benchmarks. In order to claim that GETA actually makes progress, it’s critical to measure how it compares to direct human judgment of these models in a way that cannot be contaminated. 
* The method introduces a lot of complexity, such as variational item response theory, when simpler baselines exist (in particular, since items are already being generated by language models, there should be prompting baselines such as generating lots of questions with varying difficulty and comparing model responses to human responses); it’d be nice to better understand which parts of the framework are necessary (since traditionally in deep learning, hard-coded assumptions with smaller models tend to lose out in the limit).

questions:
* My impression from reading the paper is there isn't a human study validating that GETA better ranks models. Is this the case, and if so why should trust other sources for ground truth (when they could be contaminated, or biased by other processes)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730796411740

thread_type:
official_review

<thread_object>
creator: reviewer_aktn

<content>
summary:
The authors propose GETA, a method for identifying model evaluation boundaries and generating new/difficult examples to evaluate a set of models on. GETA consists of jointly training a Variational IRT model (implemented as a simple Transformer) and an item generator (fine-tuned LLaMA-3). They evaluate on various toxicity/ethics/bias datasets, and claim that this approach can help to address evaluation data leakage/difficulty concerns.

strengths:
- The paper is clear and in-depth in its description of the method. The evaluation and experiments are presented exceedingly clearly.
- The approach is well-motivated and reasonable, though I have some concerns mentioned below.

weaknesses:
- It is not clear why "alignment" (toxicity/bias/ethics) datasets were chosen in particular for evaluation GETA, which appears general enough to be applied to any problem (I presume mostly bottlenecked by the ability of the IRT estimator and generator, but it's not clear to me that alignment would be less challenging than some other domain, e.g. math, code)
- Additionally, I would have wanted to see more experiments with different generators. This would both help address my question above (what is the bottleneck) as well as help to investigate any model-model interactions (would generators from certain families create problems easier for models in the same family?)
- Additionally, I am curious about the viability of generating difficult problems, especially when the generator chosen is weaker than many of the models evaluated. Does the joint model generate truly difficult problems, or find edges of the evaluated models' ability. Are there domains where the generator struggles to generate reasonable and challenging problems? Addressing this set of concerns would go a long way to further convincing me of the broader viability of this approach..

questions:
- Some of my questions are mentioned above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730807670371

thread_type:
official_review

<thread_object>
creator: reviewer_n45b

<content>
summary:
In traditional evaluation, given a static benchmark, all evaluation samples are treated equally and evaluated on all samples, potentially exposing these test instances and essentially making these instances i.i.d. over time. The authors call this the chronoeffect. To mitigate these challenges, the paper proposes a dynamic evaluation method where samples in a test set are selected based on the competency (trait) of the language model (LM) being tested. The basis of this method comes from Item Response Theory (IRT). In traditional IRT, the difficulty and discrimination of a sample are hand-coded or provided by templates, whereas the paper relies on an item generation method that treats these as latent variables and optimizes the likelihood of (x, y) over the trait and latents for the given datasets. Moreover, this generation method is further enhanced to take into account unseen difficulty instances, thereby going beyond the static datasets provided for optimization.

strengths:
1. An important evaluation problem with methods backed up with theories from pyschometrics.
2. Extensive evaluation of multiple models and datasets.

weaknesses:
The paper is quite dense. For me and most of the audience at ICLR, it would be hard to follow unless they are working in IRT and optimization.

I wish the underlying equations and methodologies were grounded in examples to make the intuition clear.

It is unclear to me why the resulting trends proposed by the evaluation method should be trusted without a ground truth. Some synthetic evaluation settings could make these results more trustworthy.

questions:
How does this address the chronoeffect when the underlying test set is essentially the same, i.e., the evaluation method is not going to generate examples beyond the given test set? Is it the case that since not all samples are exposed, this could help mitigate the chronoeffect?

How do you know if the resulting results can be trusted? Is there a synthetic problem where the results can be compared to the ground truth?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732847370045

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We're willing to respond to any further feedback

comment:
Dear Reviewers,

We have revised our paper following your suggestions and uploaded the new version. The line, table, and section numbers in our responses were also adjusted to align with the revision.

Thank you again for your valuable comments and suggestions, which are immensely helpful for us. We have also posted detailed responses to address your concerns. 

We totally understand that this is a quite busy period, since the reviewers may be responding to the rebuttals of other assigned papers.

We deeply appreciate it if you could take some time to return further feedback on whether our responses have solved your concerns. If there are any other comments, we will try our best to address them.

Best regards,

The authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733115961520

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We are looking forward to your further feedback

comment:
Dear reviewers, 

We sincerely appreciate your efforts in providing constructive suggestions and positive comments. To address your concerns as thoroughly as possible, we have included further clarifications and additional experimental results in both our responses and revised paper (Changes marked in blue).

We would like to summarize your main concerns again and reemphasize our responses from the following aspects:

---

### **The density and presentation of our paper. (Reviewer n45b and yuHe)**
Both reviewers found the paper too dense to follow, particularly the sections on CAT and IRT. To address this concern, we have revised our paper with **simplified notations, clearer descriptions, more detailed explanations, and more running examples grounded in equations**. Please refer to Sec. 3.1, Sec. 3.3, Fig. 6, Fig. 8, Fig. 9, and App. C.1~C.4 in the updated paper, as well as our responses to W1 & Q1 of reviewer yuHe and W1 of Reviewer n45b for detailed clarification.

---

### **The trustworthiness/validity of GETA. (Reviewer n45b, 1ZQ8, and yuHe)**
To validate GETA as a trustworthy measurement for LLMs, we elaborated on the **concept of measurement validity** and then demonstrated GETA's validity/reliability through **three reference measurements** (i.e., prevalent safety leaderboards, i.i.d. items, and OOD datasets) in our paper. 

Furthermore, we conducted (1) an additional experiment with **synthesized ground truth** (response to W3 of Reviewer n45b), (2) a **human evaluation** to manually judge the examinee LLMs' performance (response to W1 of Reviewer 1ZQ8, Appendix. D.3). Both serve as supplementary reference measurements. GETA demonstrated the **highest correlations** in most cases with **all five reference measurements across different value types**, underscoring its strong validity. Besides, we (3) provided a comprehensive description of the high quality and reliability of these reference measurements (response to Q1 of Reviewer 1ZQ8, Appendix. B.1.2).

---

### **How does GETA address chronoeffect? (Reviewer n45b, AKtN, and yuHe)**
Most reviewers raised valuable questions regarding different aspects of the chronoeffect, including the data novelty, item difficulty, and temporal issues. As mentioned in Sec. 1 and Sec. 3.3, chronoeffect presents a two-fold challenge: **Data Contamination** and **Difficulty Mismatch**. In response to Q1 of Reviewer n45b, we elaborated on GETA's mechanisms for addressing these two issues. 

1. GETA avoids **data contamination** by **generating entirely new and diverse items with an item generator**, rather than selecting items from a static item pool as in CAT or merely copying or rephrasing unexposed training items. The novelty and diversity of these items are demonstrated through substantial experimental results, including 

    (1) the significant item distribution difference shown in Fig. 4(a),

    (2) the ablation on generator backbone in App. D.2, Table 12,

    (3) a similarity analysis of the items from different data sources in response to Q1 of Reviewer n45b, and 

    (4) an evaluation using data and models released on different dates (the time issue), as detailed in response to W2 of Reviewer yuHe.

    All these results reveal GETA's robustness against potential data contamination.

2. GETA addresses **difficulty mismatch** by **adaptively adjusting item difficulty**. This process is comprehensively explained in our paper, particularly in Sec. 4.3, Fig. 4, where GETA demonstrates superior performance in distinguishing between examinee LLMs with varying value conformity through its adaptive difficulty, compared to baseline measurements. 

    Additionally, we clarified the **definition of difficulty in adaptive testing** in response to W3 of Reviewer AKtN and proved the **infeasibility of adaptive difficulty adjustment without CAT and IRT** (using only prompt engineering) in response to W2-1 of Reviewer 1ZQ8.

---

Other insightful questions, such as the reasons for choosing human values as criteria and the impacts of item generator backbone proposed Reviewer AKtN, are comprehensively answered in our responses and further discussed in the revised paper.

**We believe we have addressed all the concerns raised by the reviewers**. We understand that this is a quite busy period, but we deeply appreciate it if you could take some time to return further feedback and reconsider your assessment of our work if we have solved your concerns.

Best regards,

The authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733190080501

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Request for Feedback Before Reviewer-Author Discussion Deadline

comment:
Dear Reviewers,

I would like to express my gratitude for your efforts in reviewing our work and for your hard work for the ICLR community. We have carefully addressed all the concerns you raised, providing detailed clarifications, additional experiments, and substantial revisions to our paper.

As the reviewer-author discussion period draws to a close with less than 12 hours remaining, we have not yet received any additional feedback from you.

We kindly urge you to read our responses at your earliest convenience. We remain fully committed to addressing any remaining questions promptly and comprehensively.

Thank you for your attention to this matter.

Best regards,

The senior author of this paper
</content>
</thread_object>

</thread>
<thread>
creation date:
1733212863242

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Your feedback is valuable for us! 4 hours before reviewer-author discussion deadline

comment:
Dear Reviewers,

Thank you for your efforts in reviewing our work. With **less than 4 hours** remaining **for reviewers to provide additional feedback**—and **28 hours** before we can **submit further clarifications**—we kindly request that you review our response if you haven’t already.

**We have thoroughly addressed all concerns raised, providing detailed clarifications, additional experiments, and revisions**. **Your review of our response is critical** not only **for our submission** but also **for ensuring the fairness and rigor of the ICLR community**.

We sincerely appreciate your time and attention to this matter.

Best regards,

The authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733226052395

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Your feedback is valuable for us! The last 30 minutes before reviewer-author discussion deadline

comment:
Dear Reviewers,

*Please feel free to disregard this message if you have already responded to our replies.*

Thank you for your efforts in reviewing our work. With **less than 30 minutes remaining for reviewers to post additional feedback**—and about 24 hours before we can provide further clarifications—we kindly request that you review our responses if you haven’t already.

We have thoroughly addressed all concerns raised, providing detailed clarifications, additional experiments, and revisions. **Your review of our responses are critical** not only for our submission but also for ensuring the fairness and rigor of the ICLR community.

We sincerely appreciate your time and attention to this matter.

Best regards,

The authors
</content>
</thread_object>

</thread>
<thread>
creation date:
1733311314413

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
We greatly appreciate your efforts and support for our paper

comment:
Dear Reviewers and ACs,

Thank you for your collaborative efforts in reviewing our work. We are confident that we have thoroughly addressed all concerns raised by the reviewers, provided detailed responses and a revised paper, and summarized the full responses into one-page versions for Reviewer n45b and AKtN, following the reviewers' suggestions.

We fully understand that reviewers may be busy and unable to review all details at this stage. However, we have invested significant effort during both the submission and rebuttal phases to clearly demonstrate our work's innovative contributions, scientific merit, and the importance of its experimental and theoretical results.

As we approach the next phase, we kindly and sincerely request that you review our responses and reassess our refined paper even after the reviewer-author discussion. We genuinely believe that our work makes a significant contribution to the field of LLM value and risk evaluation, and we will appreciate it if you could consider supporting our paper during the AC-reviewer discussion phase.

Thank you again for your valuable time and efforts.

Best regards,

The authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
The authors propose a generative testing strategy that adaptively generates evaluations for LMs on the topic of moral or toxic behaviors. Adaptive testing of this form has the potential to avoid several issues with static benchmarks, including saturated performance and contamination (termed chronoeffects in the paper). 

Adaptive testing is an emerging, timely direction and safety is an especially interesting place to apply these ideas. However, the reviewers generally agree that there are issues with evaluating this evaluation. Testing for correlation with external benchmarks and simply trusting auto evals is fairly weak when evaluating high-stakes goals such as values and morality. The authors perform some human evaluations during the rebuttal, but this is fairly limited and probably should be a more major and integrated part of the work.