EvalAgents: Discovering Implicit Evaluation Criteria from the Web
Authors: Manya Wadhwa
Zayne Rea Sprague
Chaitanya Malaviya
Philippe Laban
Junyi Jessy Li
Greg Durrett
Venue: COLM 2025 (2025)
Keywords: evaluation
writing
large language models
TL;DR: We propose a novel framework EvalAgent that dynamically generates grounded, implicit evaluation criteria for a given prompt based on retrieved expert advice.

Abstract: Evaluation of language model outputs on structured writing tasks is typically conducted with a number of desirable criteria presented to human evaluators or large language models (LLMs). For instance, on a prompt like "Help me draft an academic talk on coffee intake vs research productivity", a model response may be evaluated for criteria like accuracy and coherence. However, high-quality responses should do more than just satisfy basic task requirements. An effective response to this query should include quintessential features of an academic talk, such as a compelling opening, clear research questions, and a takeaway. To help identify these implicit criteria, we introduce EvalAgent, a novel framework designed to automatically uncover nuanced and task-specific criteria. EvalAgent first mines expert-authored online guidance. It then uses this evidence to propose diverse, long-tail evaluation criteria that are grounded in reliable external sources. Our experiments demonstrate that the grounded criteria produced by EvalAgent are often implicit (not directly stated in the user's prompt), yet specific (high degree of lexical precision). Further, EvalAgent criteria are often not satisfied by initial responses but they are actionable, such that responses can be refined to satisfy them. Finally, we show that combining LLM-generated and EvalAgent criteria uncovers more human-valued criteria than using LLMs alone.

Published as a conference paper at COLM 2025
EVALAGENT: Discovering Implicit Evaluation Criteria from the Web
Manya Wadhwa Zayne Sprague Chaitanya Malaviya Philippe Laban 9 Junyi Jessy Li Greg Durrett
The University of Texas Austin University of Pennsylvania Microsoft Research manya. wadhwa@utexas. edu
Abstract
Evaluation of language model outputs on structured writing tasks is typ- ically conducted with a number desirable criteria presented to human evaluators large language models (LLMs). For instance, a prompt like "Help me draft an academic lalk on coffee intake us research productivity" model response may be evaluated for criteria like accuracy and coher- ence. However, high-quality responses should do more than just satisfy basic task requirements. An effective response this query should in- clude quintessential features of an academic talk, such as compelling opening, clear research questions, and takeaway. To help identify these implicit criteria, we introduce EVALAGENT, novel framework designed to automatically uncover nuanced and task-specific criteria. EVALAGENT first mines expert-authored online guidance. It then uses this evidence to propose diverse, long-tail evaluation criteria that are grounded in re- liable external sources. Our experiments demonstrate that the grounded criteria produced by EVALAGENT are often implicit (not directly stated in the user's 's prompt), yet specific (high degree of lexical precision). Further, EVALAGENT criteria are often not satisfied by initial responses but they are actionable, such that responses can be refined to satisfy them. Finally, we show that combining LLM-generated and EVALAGENT criteria uncovers more human-valued criteria than using LLMs alone.
Introduction
PROMPT
QUERY GENERATOR
EXPERT WEB RETRIEVER
CRITERIA GENERATOR
EvalAgent Criteria
'Help draft an academic
what key
from faculty
A compelling motivation
components academic
talk coffee intake vs research
Focus big picture issues
long guide [.]
02: L.]
give talk
O
productivity
CO: L.
Filter
Summarize
Aggregate
Rank
restate findings
Figure 1: We propose a novel framework EVALAGENT that dynamically generates grounded, implicit evaluation criteria for given prompt based retrieved expert advice.
Large Language Models (LLMs) are increasingly used for complex writing tasks, including blog posts, novels, research statements, and more. However, evaluating their outputs remains significant challenge (Tian et al., 2024; Spangher et al., 2024). The lack of robust, automated evaluation methods limits our ability to scale these evaluations, and hampers our ability to systematically understand and mitigate the errors these models make. Human evaluation has revealed issues such as lack of creativity, concept repetition, and logical inconsistencies (Chakrabarty et al., 2024b). However, scaling human evaluation is subjective, requires expertise, and quickly becomes infeasible given the vast number of tasks LLMs are expected to handle (Chakrabarty et al., 2024a; Kim et al., 2024a).
"ode and data available at: https: thwb. howt.////aaaggaaaaaaaaaaaaaalaaa

---

Published as a conference paper at COLM 2025
There are two key challenges central to open-ended response evaluation: (1) clearly defining the criteria on which model responses are assessed, and (2) ensuring that evaluators (human or model-based) can reliably interpret and judge these criteria. In this work, we address the first challenge, namely identifying (often implicit) criteria. As the scope of LLM applications expands, it is essential to use evaluation criteria that reflect nuanced writing standards, domain-specific knowledge, and appropriate guidelines.
We propose EVALAGENT, novel evaluation framework that can scalably generate task- specific evaluation criteria via grounding to expert knowledge from the web. Our proposed framework agentically searches the web generate implicit task-specific evaluation criteria based on retrieved advice. Figure illustrates our proposed framework. EVALAGENT consists of three key com components: (a) Query Generator, which formulates search queries to retrieve instructional web documents, (b) an Expert Retriever, which filters the results for good URLs, then extracts and summarizes information from the instruction web docu- ments, (c) Criteria Generator, which synthesizes retrieved information into well-grounded evaluation criteria.
This process mimics the kind of resources that novice humans turn to when attempting to improve content they are producing. In the example from Figure 1, if someone wants a draft for an academic talk, they might search 'how to draft academic talk' on search engine, then navigate to instructional documents like university websites, blogs from academics on how to structure talks, and social media platforms like Reddit that crowd-soure advice.
To assess and compare evaluation criteria generated by different systems, we propose three properties that ideal criteria should exhibit: specificity, implicitnesss and actionability. Specificity ensures that evaluation criteria target precise dimensions of quality, thus reducing ambiguity during assessment. Implicitness measures whether we capture the unspoken norms, principles, and conventions within specific writing domains. Finally, actionability ensures that criteria facilitate tangible improvements. We evaluate criteria generated from EVALAGENT for these properties across nine datasets consisting variety of writing tasks, ranging from procedural and technical writing to creative writing. We show that criteria generated by our system are highly actionable and more human-aligned compared to simply prompting an LLM.
Our main contributions are (1) the EVALAGENT framework, demonstrating that we can retrieve implicit evaluation criteria from the web; (2) evaluation of criteria produced by EVALAGENT across several settings, showing how they differ from LLM-generated criteria in that they are more implicit, less obvious, and more actionable.
2 Task Setting
Given user prompt x, LLMs generate a response y by sampling from distribution p(y x). In this work, we propose a dynamic approach for generating a set of evaluation criteria C tailored to each x.
Implicit Criteria We define an implicit criterion as one that a user did not state in their prompt, but which is still specific to the prompt and is a desired property to satisfy. This underspecification can be unintentional, where the user might not be an expert in the task and is not aware of the details to specify. It can also be intentional, where the user assumes certain details are common knowledge or might feel redundant to state these or they want to save time (Malaviya et al., 2025b). For example, for the user prompt in Figure 1, 'the response should focus on big picture questions' is an implicit criterion, not specified in the prompt, but important for drafting good academic talk.
In contrast, explicit criteria are often able to be directly derived from the given instruction. Following from the example in Figure 1, explicit criteria for that instruction may be 'the response should be an academic talk'. We additionally define explicit criteria as unstated but extremely obvious criteria such as 'the response should be focused on the topic'. Other unstated but extremely obvious criteria may be assuming the response is factually correct or having mathematical rigor for tasks in factuality and mathematics, respectively. Previous work has
2

---

Published as a conference paper at COLM 2025
explored explicit criteria (Qin et al., 2024); however, our work focuses generating implicit criteria that is specific, actionable, and closely aligned with human-authored criteria.
used in scoring functions f(x, y,,,,,), y, where each criterion c E C to help assess the quality of a response. f can re- turn real-valued scores (Kocmi & Federmann, 2023; Wadhwa et al., 2024a; Kim et al., 2024b; Wu et al., 2025), binary judgments (Olausson et al., 2024; Wadhwa et al., 2024b), or other formats depending on the task (Zhang et al., 2024).
Criteria-Based Scoring Once generated, these criteria
These judgments from f based on can be used in multiple ways. For example, they can be aggregated into metrics like pass-rate (e.g., PR Leec f(x, (x,y,,))lCC), or used for response refinement (Cook et al., 2024), or be transformed into dense rewards for training models more effectively (Wu et al., 2023).
Our focus in this work is to identify set of criteria C that serve as a shared starting point across multiple applications, such refinement and evaluation. We envision EVALAGENT to be a flexible framework that enables targeted development of specific capabilities, and we leave the exploration of downstream applications to future work.
3 EVALAGENT
Algorithm Aspect generation with EVALA- GENT
In this section, we propose our framework EVALAGENT, aimed at extracting evalua- tion criteria from instructional web docu- ments. There are several key challenges that make retrieving instructions from the web non-trivial. First, using the prompt URt a search query often fails to retrieve URLs that sufficiently cover the criteria for gen- erating good response. For example, di- rectly searching for prompt Figure can
Input: Prompt
Output: Criteria set C
1: Q + QG(x)
b Generate set of queries
2: C. Ø
3: for q E do
4:
U search(q:)
EExpert Retriever
5:
U' = filter (U, q x)
6:
Cg.t
7:
for ua E U' do
lead to URLs focused on content of the academic talk, rather than advice on how to structure the talk, some useful cri- teria may be missed. Second, generic or low-quality pages might be surfaced; these should be filtered to leave those that are genuinely useful content from authoritative sources. Third, even when relevant informa- tion is retrieved, synthesizing the retrieved knowledge to clearly identify evaluation criteria relevant to the user prompt is challenging.
8:
rk answer
x;
9:
Cq.1 + Ca. U {xx] end for
10:
11: Cy Cy U (C
12: end for
13: Cw = AG(Cg)
Criteria Generator
14: C = Rank(Car,
> Rank
15: return C
EVALAGENT, described in Algorithm 1 comprises several key components designed specifi- cally to overcome the challenges mentioned above. At a high level, given a user promopt, we first generate search queries that can be easily answered using instructional web documents. After retrieving such documents, we generate answers for the the search queries. We then combine these answers define our evaluation criteria.
Query Generator (QG) Given the instruction x, we prompt an LLM to generate conceptual queries Q such that each qi E Q helps retrieve relevant instructional advice for x (Prompt B.1). These queries help us find targeted advice concepts useful for the prompt, unlike naively searching for the prompt itself. For example, in Figure 1, the main concept is AAcadeiic Talk', query formulated is: 'whal are the key components of academic talk'. We also generate queries related to the main concept, even when they are not directly stated in the instruction, for example, 'how to write an engaging talk'. A similar query expansion step is frequently used in LLM fact-checking (Chen et al., 2024; Malaviya et al., 2024).
Expert Retriever (search, filter, summarize) For each q,, we retrieve a list of URLs (U) and filter them based on two properties: (1) expertise of the URL content and (2) the relevance of the URL content to the instruction. We do this filtering to remove shallow, clickbait links and focus useful advice. We create detailed criteria to evaluate these properties (Appendix
3

---

Published as a conference paper at COLM 2025
How to write an engaging fight scene?
fight scene might use short sentences long, quick slow. At the climax, the
Vary the pacing of the fight scene Use short sentences
Establish charucters' backstory
Vary pacing and rhythm
speed of the scene may in fact slow down, with the character's 's sense of time expanding.
reflect characters emotional states
summarize
Describe motivations
consequences|....
Weave characters' backstory Establish motivations for conflict
The choracters you are just us importont
answer
Criteria extracted for this query
the plot, make sure the confiict here is meuningful and has consequences
Write consequences of conflict
Step 2 of our proposed pipeline, Expert Retriever. Given a query "How to write an engaging fight scene?", it retrieves trusted URLs, answers the search query based on the URL content, and summarizes all the query answers to give query-specine criteria list.
Figure 2:
B.1) and prompt an LLM to score the URL content. We sort the URLs by the prompted score and take the top-5 to get U', For the example in Figure 1, we are able retrieve and filter to trusted sources of advice from university links and faculty blogs as well as Reddit advice.
For each Ug E U', we prompt an LLM to answer qi based on ug (Prompt B.2) and extract Tk- Fk is natural language response that answers 9;- Once we have these answers for all Uy E U', we summarize them into a query-specific list, Cy (Prompt B.3). This is the first step where we re-write the URL content into a list of criteria. This step ensures that we are summarizing only the information relevant to the query and the instruction and have a way of identifying useful information. We illustrate this process for single query in Figure 2.
Criteria Generator (CG) Once we have Cg for all Q, we aggregate them to produce Ca (Prompt B.4). We illustrate this process in Figure 3. Given query-specific summaries, we aggregate them by putting together common properties seen across summaries and also keeping the unique ones. After aggregation, the criteria go through an instruction-aligned rewriting step such that each point represents criterion relevant to evaluating (Prompt B.5). The rewriting for alignment step is needed to ensure that the generic, instructional web-document advice is made specific for evaluation.
Ranking In the final step, rank- ing function scores all criteria in C. and returns ordered set of
How to write an engaging fight scene?
Establisi churactes" backstory
The response should:
Vary pacing and rhythm reflect characters
vary the pocing and rhythm
reflect emotional states
criteria, C. For this work, we score
Daserihe motivations consquannos
sensory deserigtion
aggregate
deserbbe the some and emotional stakes
criterion based its relevance
How to build tension a story?
to x. We use an LLM for this scor- ing (Prompt B.7). The ordering helps bring the more relevant crite- ria to the top of the list, which can be useful for downstream applica- tions (e.g. evaluating only using criteria instead of the entire list).
Continouully escalate the stakes @ Lie scnoery deseviptions
Crease conflice internal and exteral
intemally extemallyy [...]
contlict
EA-Web criteria
Vary pacing
Figure 3: Step 3 of our proposed pipeline, Criteria Gen- eration. We take the criteria generated for each of the queries and then aggregate them using an LLM to dis- till an evaluation criteria for the instruction.
Combining with other criteria EVALAGENT retrieves set C of criteria from the web. However, these criteria may not be exhaustive for evaluating the task. In particular, criteria arising directly from the instruction itself are not likely to be captured. As an extension to our system, we can also merge C with other criteria (e.g., explicit criteria derived from the instruction) to produce ranked list.
When we only use the criteria derived from the web, we refer to our system as EVAL.AGENT- WEBONLY (ÉA-Web). Our full system combines with the LLM-1 baseline defined below, reranked according to the criteria relevant to the prompt. We refer to this as EVALAGENT- FULL (EA-Full).
We use 100--44-iini-2224--00 the design choices made for EVALAGENT and present the prompts in Appendix B. We also discuss the computational costs associated with different steps of our proposed framework in Appendix C.
as the LLM for the components described above. We detail
4

---

Published as a conference paper at COLM 2025
4 Metrics for Assessing Generated Criteria
Having established our method, introduce set of metrics designed to assess the quality of These metrics are novel contributions this work, as prior research has not systematically evaluated or compared different sets of criteria in similar manner.
4.1
Automatic Metrics
Specificity (S) A criterion is specific when it uses less frequently occurring words. We measure specificity of criterion, C; as the rarity of words in the criterion over all the sets of criteria generated by the methods we evaluate, C+ CEA U CID U CLLM-n- Rarity is calculated by the Normalized Inverse Word Frequency (NIWF) (Zhang et al., 2018; Ko et al., 2019). The specificity, S(C ,c;), , of criterion then defined as the maximum NTWF over the words in the criterion. A higher score indicates that the criterion uses less commonly occurring terms, reflecting greater degree of specificity.
Implicitness (I) A criterion is considered implicit if it surfaces properties not explicitly stated in the instruction. To approximate this, we measure the explicitness or surface-level nature of criterion through its word-over word-overlap ratio (WO) and report 1-WO. A higher word- overlap suggests that the criterion closely mirrors the wording of the instruction and is less implicit.
Both and are computed for each generated criterion and subsequently aggregated across all criteria produced by system for given dataset, providing holistic view of the system's s ability to generate implicit and specific feedback. We report and across all instances in the dataset (described in Section 5.1).
Actionability (A) A criterion is actionable if the model can effectively revise and enhance its initial response to satisfy the criterion. We test for the actionability of a criterion, ej by testing if different models can be prompted to satisfy the criteria by revising a response that failed to satisfy initially (Wadhwa et al., 2024b; Madaan et al., 2023). We report A on sample of 200 instructions. We prompt gpt-4o-mints mini-2224407-115 for evaluating criterion satisfaction (Prompt F.1). Details about the sampling process, dataset distribution and prompts for evaluation are in Appendix F22
Recall of human criteria (R) We evaluate how frequently human-generated or verified criteria for given instruction are represented in the generated criteria. To calculate this, we prompt a large language model (LLM) with human-provin determine whether that criterion is entailed by any generated criteria by the system. It is important note that human written criteria may not fully capture all implicit criteria needed to evaluate response to instruction. This lack of coverage means that we cannot treat the human curated list as "gold standard" however, we use it as proxy for what humans believe is important. We calculate recall by prompting gpt-4o-mini -00----------1111 Prompts for calculating recall and the prompt tuning process are discussed in Appendix F.3.
criterion, asking to
4.2 Human Evaluation
Utility (U): A criterion has high utility it is highly relevant for evaluating a response to an instruction. Given criterion, we ask human annotators to give judgments on its utility. They indicate this on a scale of 1-3, where that evaluating that criterion will have no impact on the response, whereas 3 means that it is extremely useful for knowing quality of a response. We report this as an average score across the human annotation set.
Obviousness (O): Given a criterion, we ask human annotators to evaluate if a criterion is "obviously" implied by the instruction or if it is not obvious. A criterion directly stated in the instruction is obvious, but criteria that are not explicitly stated in the instruction may or may not be obvious to the evaluator based context prior knowledge. We report this as an average score across the human annotation set.
We detail annotation subset, hiring process, and annotation instructions in Appendix E.
5

---

Published as a conference paper at COLM 2025
5 Experimental Setup
5.1 Dataset Curation
Dataset
#
Evaluation
Expert
BGB
50
Human
We evaluate criteria generation methods on subset of instructions from BigGenBench (Kim et al., 2024a), ChatBotArena (Chiang et al., 2024), Art or Artifice (Chakrabarty et al., 2024a), Dolomites (Malaviya et al., 2025a), WritingBench (Wu et al., 2025), In- foBench (Qin et al., 2024), WildBench (Lin et al., 2024), and MT-Bench (Zheng et al., 2023). For all these datasets we keep in- structions that have one of the keywords 'write', 'create' or 'develop'. We also filter for instructions that are in English, single- turn and focus on creative writing and plan- ning tasks. Appendix A describes this data selection process in more detail.
Art or Artifice
12
Human
Dolomites
519
Human
InfoBench
158
Human
MT Bench
9
LLM
X
WildBench
357
LLM
WritingBench
235
LLM
Ask-then-Critique
310
Human
ChatbotArena
337
Table 1: All datasets considered in this work. We filter to a subset of instructions that re- quire procedural, methodical or creative long- form outputs, discarding many examples from datasets like MT Bench.
To augment and diversify the set of instructions considered in this work, we collect a new dataset called Ask-then-Critique This dataset consists of user-specified instructions, model responses from different LLMs (namely, ChatGPT, Gemini, or LLaMa), and user-written natural language critiques representing their evaluation. The goal of collecting Ask-then- Critique was to have users evaluate LLM responses for their own instructions to get more reliable critiques. This is different from other datasets considered in this work, where human-written criteria were not provided by the issuer of the instruction. Details about how we collected Ask-then-Critique are in Appendix A.2.
Table 1 enumerates the datasets along with the number of tasks used for this work. For most of these datasets, except ChatbotArena and MT-Bench, there are criteria that the dataset owners use for evaluation. Some of these criteria are human and expert written, where as the remaining datasets have LLM generated criteria that are human verified. We indicate this information in Table 1. Also, note that these dataset specific criteria do not have coverage of all evaluation properties of an instruction and hence can be variable in quality. We show examples of some of the instructions and constraints in Table 5.
5.2 Baselines
Instruction Decomposition (ID) generates criteria for x by decomposing the instruction into a list (Qin et al., 2024). For every instruction, we prompt an LLM to generate a list of instruction level criteria. This method targets capturing explicitly stated in- formation in the instruction. LLM-Prompted (LLM) generates criteria by prompting an LLM for a set of evaluation criteria (Lin et al., 2024). Compared to Instruction Decomposition, this approach can go beyond criteria explicitly mentioned in the instruc- tion. A related method, LLM-Prompted to z (LLM-n) This baseline is stronger version of LLM, where we over generate the criteria and then subsequently rank them. This approach is not commonly used in the literature, but we explore it to take advantage of our proposed generate and rank pipeline. We find it to be particularly effective when combined with the EGENT criteria.
#
St
I1
0
ID
6
0.13
0.50
0.96
2.94
LLM
9
0.10*
0.67t
LLM-n EA-Web
30
0.09t
0.82
0.78
2.67
22
0.14
0.88
0.58
2.46
Human
9
0.21 0.81
0.73
2.69
Table 2: Automatic metrics compar- ing goodness of criteria generated by different methods. We see our pro- posed method has higher specificity and lower word-overlap with the instruction. f: signifies where the dif- ference is significant with p < 0.05.

---

Published as a conference paper at COLM 2025
6 Results
6.1 Properties of generated criteria
Criteria generated by EVALAGENT are more implicit. Results in Table 2 show that EA- Web's criteria specific and have less word overlap with the instructions, suggesting they are more implicit. Furthermore, we plot the distribution of the Specificity and Implicit scores in Figure 4, which shows that human-written and human-verified criteria have flatter distributions for both metrics. This flatter distribution of scores supports our hypothesis that humans expect good response to satisfy implicit criteria. Criteria formulated by humans may also show lower direct overlap with the instruction due to the human distilling the main ideas into more generalized or abstract terms.
Table 11 in the appendix shows examples and scores for criteria for an instruction. We high- light words that lead to the specificity value for that criterion, such as the word "deprivation' in "The response should focus on the emotional impact of sleep deprivation on relationships, work, and daily life."
EVALAGENT's criteria are non-obvious Table reports the human judgment scores for Obviousness and Utility, where we find that EVALAGENT achieves low obviousness score on average by our human annotators while maintaining a high utility score. ID generates the criteria with the highest utility; this is expected since the criteria are essentially constraints decomposed from the instruction itself and are necessary for cnstruciionfoolloinng LLM- prompted criteria are classified as mostly obvious, with slightly lower utility score.
To dig deeper into these ratings, Table 15 in the appendix shows human judgments
Method Initial Edited A
Claude-3.5-Sonnel
along with two tasks alongside criteria pro- duced by LLM-1 and our system. For utility, we report an average score from three anno- tator ratings, and for obviousness we show the number of annotators that found the cri- terion obvious. Note that obviousness a subjective dimension and depends on the fa- miliarity of the annotators with the instruc- tion they are evaluating. For example, 'the response should include relevant emojis' gets rated obvious by only one annotator out of three; similarly the criterion 'The response should create movement by transitioning from a specific moment to the broader implications of the characters' not obvious to any of the an- notators, since it is very specific to creative writing and talks about narrative pacing. We show the average Utility and Obvious- ness scores from human annotators across datasets in Figures 8 and 9 of the Appendix.
ID
0.77
0.90
0.13
0.58
LLM-1
0.62
0.91
0.28
0.75
EA-Web
0.51
0.90
0.39
0.79
Human
0.73
0.91
0.18
0.66
GPT-4o
ID
0.79
0.90
0.10
0.52
LLM-x
0.65
0.92
0.27
0.77
EA-Web
0.57
0.90
0.34
0.77
Human
0.78
0.92
0.14
0.63
0.76
0.88
0.13
0.53
LLM-n
0.60
0.90
0.30
0.75
EA-Web 0.47
0.86
0.40
0.74
Human
0.69
0.88
0.19
0.58
Table 3: We report A improvement in pass rate post-editing with criteria generated by different systems. We see that EA-Web gener- ates the most actionable criteria, leading to the largest improvements. It is also able to find harder to satisfy criteria.
EVALAGENT generates highly actionable criteria Table reports actionability scores of various criteria. EA-Web achieves the highest actionability score, indicated by the largest improvement (A) between the pass rates of edited and initial responses. Additionally, EA-Web has the lowest pass rates among all systems indicating that the system identifies more nuanced and challenging criteria. Furthermore, we find this lower pass rate is correlated with criteria rated as less obvious by humans; results can be seen in Figure 11 of the Appendix.
Table 13 enumerates initially unmet criteria in response generated by GPT-4o and reports their corresponding outcomes after editing by the same model. Criteria generated by EA- Web are more implicit and seem useful for evaluating response to the instruction. For
7

---

Published as a conference paper at COLM 2025
()) Distribution of Specificity Scores
(II) Distribution of Implicitness
Method
25
EA Web
Human
Density
Density
LLM
LLMin
10
5
0.0
0.4
Specificity
Implicitness
Figure 4: Distribution of the Specificity and Implicitness scores for each method. From (I) that EA-web generates larger number of criteria with specificity higher than LLM-11 From (II) we see that EA-web generates criteria that has less overlap with the instruction compared to LLM-1 and is more implicit.
ID
LLM
LLM-30
EA-Web
LLM-60
LLM-30+30
EA-Full
Art Artifice
0.04())
(115
0.47 (30)
(341
0.54(60)
0.52
0.62
WritingBench
0.52(9)
(153
0.74
0.66
(,82(59)
0,82
0.86
Csrtthen-
(429
0.50
(364
0.44/16)
0.7](60)
0.70
0.74(46)
Average
(,28
0.39
0.62
0.50
0.69
0.68(60)
0.74(55)
Table 4: We report recall values for human criteria across three datasets. We compare our LLM-x with 60 to match number of criteria generated by EA-Full (2) LLM-30+30: we prompt an LLM to generate additional 30 criteria to match our merge setup described in Section EA-Full achieves a higher recall against human criteria than LLM baselines.
proposed system, EA-Full against two LLM baselines. (1) LLM-60:
instance, in financial analysis reporting task, LLM-generated criteria primarily focus instruction level details. In contrast, EA-Web, generates criteria that describe key factors in financial analysis report. Some EA-Web criteria which are not actionable valid, but out of scope, for instance "the dala should be presenled in the form of charts for better readability". Models considered in this work cannot generate charts.
6.2 Alignment with human criteria
We now evaluate how well our generated criteria align with human-written criteria. Our proposed system, EA-Full (Section 3), combines EA-Web and LLM-n, resulting in larger set of criteria that are then ranked.
Setup:
We compare this approach against two LLM-based baselines. For fairness, we need these baselines to have access the same number of criteria as EA-Full. For the first baseline we prompt LLM-# to generate the same total number of criteria as our system generates on average. We call this LLM-60 since we prompt the system to generate 60 criteria. The second baseline prompts the LLM to generate additional criteria beyond the initial set in EA-Full. This closely reflects our setup of combining EA-Web criteria with LLM-n where we add supplementary criteria to an initial set. We call this LLM-30+30 since we add another 30 criteria to an existing LLM-x list. Similar to EA-Fulll we first generate criteria and then rank them as discussed in Section 3. We report these results on three datasets with varying types of human-written criteria: expert (Art or Artifice), human verified (WritingBench) and self-issued instruction criteria Asstthen--rritique)..
From Table 4 we see that our system is able to capture human-criteria better than simply prompting an LLM to generate more criteria. Figure 5 further investigates the rate at which different systems recall human-annotated criteria, comparing proposed system, EA-Full against LLM-60 and LLM-30+30. For WritingBench and Ask-then-Critique all systems show similar performance until k 10, after which EA-Full shows significant
Results:
8

---

Published as a conference paper at COLM 2025
EA-Full (EA-Web LLM-30)
LLM-30++3
LLM-60
WeitingBench
0.73
0.76
0.43
0.66
0.67
0.37
0.59
0.32
0.52
0.27
0.45
0.42
0.21
0.38
0.33
0.16
0.25
0.11
0.24
0.16
0.05
0.16
0.08
0.00
20
40
60
20
40
60
20
40
60
Figure 5: Recall@k for three datasets. For each method we calculate recall by prompting human criteria against the generated criteria (Section F:3). EA-Full retrieves human criteria at a higher rate compared to equivalent LLM baselines.
gains. This trend is even more pronounced for Art Artifice, indicating that our approach captures nuanced criteria related to creative writing sooner compared to other baselines.
In Table 14 we show several criteria recalled by our system against the human criteria. We are able to match the human criteria, even though they vary in specificity. For example, the constraint 'the response should be complete and provide delails about activities' is entailed by 'the response should include outdoor activities' 'the response should include local cultural activities' and 'the language should be descriptive'. This highlights that we are able to capture criteria of given human written constraint.
Related Work
Generating Criteria Benchmarks like IFEval (Zhou et al., 2023) and COLLIE (Yao et al., 2024) add verifiable constraints to instructions, but these constraints rarely target deeper stylistic content improvements critical for advancing LLM performance. InfoBench (Qin et al., 2024) only focuses on instruction decomposed constraints. WildBench (Lin et al., 2024) proposes benchmark for more real-world queries evaluated via task-specific checklists. These checklists are LLM prompted and human-verified, but lack any grounding. VibeCheck (Dunlap et al., 2025) focuses on discovering criteria from a pair of preferences such that these criteria are able to differentiate between preferences. In contrast, our work centers on creating implicit yet grounded criteria that are adaptable across various feedback protocols and downstream applications.
LLM-based Evaluators Recent research has explored the use of LLMs as evaluators for generative tasks, such instruction following, in popular benchmarks like AlpacaEval (Li et al., 2023) and MT-Bench (Zheng et al., 2023), however these mostly focus on a predecided criteria the general criteria of 'overall quality'. LLM-based evaluators self-improvement tasks (Madaan et al., 2023; Yuan et al., 2024). Kim et al. (2024c) focuses on improving LLM-based scoring models. Our research complements these efforts by generating criteria suitable for integration within these systems.
also applied in
Conclusion
We introduce a framework, EVALACENT for generating implicit and grounded criteria for open-ended response evaluation. We do this by agentically searching the web for instructional documents for doing the task and synthesizing a task-specific criteria. Our evaluation shows that we can generated implicit, less obvious, and actionable criteria that capture what humans think is important. This framework paves the way for scalable evaluation of LLMs across broad range of writing tasks.

---

Published as a conference paper at COLM 2025
Limitations EvalAgent relies web-grounded retrieval to identify expert-written docu- ments for criteria generation, which constrains it to the quality and scope of information available online. In niche domains, relevant content may be scarce difficult to retrieve. For example, for the instruction "write an article that talks about obtaining the necessary organic amount of substance to make nanoparticles." ", Agant generates query "how to write clear procedure for synthesizing nanoparticles" This is a very niche domain and searching for the query either leads to scientific PDFs which are currently not extracted by our system, or leads to content related to nanoparticles instead of writing procedures, so we do not get any useful criteria from this query. However, as more information is indexed or as we expand our systems' abilities to process information in PDFs and other modalities, EvalAgent will become more effective, and the quality of what can be retrieved will continue to improve.
Moreover, adhering to the guidelines of experts on the web implicitly centers the perspec- tives of authors of retrieved documents in evaluation. In this study, these authors are all writing in English. Moreover, the views of these experts could be misaligned with the views of a different set of experts; e.g,, writing guidelines on the web encourage engagement via "clickbait" but expert journalist would disagree with these. However, this effect was not a first-order problem we noticed in our analysis of our system's behavior, so we did not conduct further analysis of it. We note that our web page filtering did reduce the presence of overly generic criteria, but we did not observe this process to be amplifying any particular biases.
Finally, our work introduces additional overhead compared to directly prompting an LLM for evaluation criteria. Specifically, retrieving relevant web documents incurs latency, and multi-stage prompting with commercial APIs increases computational cost. However, these are one-time costs: both retrieval and criteria generation are conditioned solely on the instruction and can be precomputed prior to model evaluation. As such, they do not impact the runtime cost of benchmarking LLMs. Furthermore, the overall cost of the system is expected to decrease as inference costs for sufficiently capable LLMs become cheaper over time.
Acknowledgments
This work was partially supported by NSF CAREER Awards IIS-2145280 and IIS-2145479, NSF grant IIS-2107524, NIH grant 1R011M01460001, the NSF Al Institute for Foundations of Machine Learning (IFML), the NSF-Simons AI Institute for Cosmic Origins (CosmicAI) under NSF Cooperative Agreement 2421782 and Simons Foundation 1100OA1-0001015,, and grant from Open Philanthropy. Thanks to Kathryn Kazanas, Rohan Deshmukh and Benjamin Lew for the human annotations.
References
Tuhin Chakrabarty, Philippe Laban, Divyansh Agarwal, Smaranda Muresan, and Chien- Sheng Wu. Art or Artifice? Large Language Models and the False Promise of Creativity. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, pp. 1-34, 2024a.
Tuhin Chakrabarty, Philippe Laban, and Chien-Sheng Wu. Can AI writing be salvaged? Mitigating Idiosyncrasies and Improving Human-Al Alignment in the Writing Process through Edits. Xrv preprint ar Xrx::22409, Xi0:2400.14409, 2024b.
Jifan Chen, Grace Kim, Aniruddh Sriram, Greg Durrett, and Eunsol Choi. Complex Claim Verification with Evidence Retrieved in the Wild. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3569-3587, Mexico City, Mexico, June 2024. Association for Computational Linguistics. doi: 10.18653/ //11 2222..aaclllooog..1100 196. URL https: haaps://aclan http://aaaaaaoolooooea/eo ogy. org/ 2024. naacl-long. 1961
10

---

Published as a conference paper at COLM 2025
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios N. Angelopoulos, Tianle Li, Dacheng Li, Banghua Zhu, Hao Zhang, Michael I. Jordan, Joseph E. Gonzalez, and Ion Stoica. Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. In Proceedings of the 41st International Conference on Machine Learning, ICML224 JMLR.org, 2024.
Jonathan Cook, Tim Rocktäschel, Jakob Foerster, Dennis Aumiller, and Alex Wang. TICKing All the Boxes: Generated Checklists Improve LLM Evaluation and Generation. arXiv preprint Xiv:2410.036088, 2024.
Lisa Dunlap, Krishna Mandal, Trevor Darrell, Jacob Steinhardt, and Joseph E. Gonzalez. Discover and Quantify Qualitative Differences in Large Language Models. In The Thirteenth International Conference on Learning Representations, 2025. URL https:
VibeCheck:
Seungone Kim, Juyoung Suk, Ji Yong Cho, Shayne Longpre, Chaeeun Kim, Dongkeun Yoon, Guijin Son, Yejin Cho, Sheikh Shafayat, Jinheon Baek, et al. The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models. Xrviv preprint Ar.:v:2406.0576 2024a.
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 4334-4353, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653/v1/ org/ wrg.//24............. emnlp-main. 248/.
emnlp-main.245 248. URL htttss:/aalaatthll///
Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 0334-4353, Miami, Florida, USA, November 2024c. Association for Computational Linguistics. doi: 10.18653/v1/20 emnlp-main.249 248. URL https: WtRLhttp:://cclle /ttp::////laaahhlloooooo org/oogg////// wttp:/2222eemmmmmamn....
emnlp-main. 248/
Wei-Jen Ko, Greg Durrett, and Junyi Jessy Li. Linguistically-Inoomee
Specificity and Semantic Plausibility for Dialogue Generation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp.
2019.
Tom Kocmi and Christian Federmann. Large Language Models Are State-of-the-Art Eval- uators of Translation Quality. In Mary Nurminen, Judith Brenner, Maarit Koponen, Sirkku Latomaa, Mikhail Mikhailov, Frederike Schierl, Tharindu Ranasinghe, Eva Van- massenhove, Sergi Alvarez Vidal, Nora Aranberri, Mara Nunziatini, Carla Parra Escartín, Mikel Forcada, Maja Popovic, Carolina Scarton, and Helena Moniz (eds.), Proceedings of the 24th Annual Conference of the European Association for Machine Translation, pp. 193- 203, Tampere, Finland, June 2023. European Association for Machine Translation. URL https://aclanthollogy wrg/2023.3222. eamt- 1.
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. AlpacaEval: An Automatic Evaluator for Instructionffollowingg Language Models. https://gi thub. 2023.
Bill Yuchen Lin, Yuntian Deng, Khyathi Chandu, Faeze Brahman, Abhilasha Ravichander, Valentina Pyatkin, Nouha Dziri, Ronan Le Bras, and Yejin Choi. WildBench: Bench- marking LLMs with Challenging Tasks from Real Users in the Wild. arXiv preprint Mxii:244060047700 2024.
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Relf-Refine: Iterative
11

---

Published as a conference paper at COLM 2025
Refinement with Self-Feeddack Advances in Neural Information Processing Systems, 36: 46534-46594, 2023.
Chaitanya Malaviya, Subin Lee, Sihao Chen, Elizabeth Sieber, Mark Yatskar, and Dan Roth. ExpertQA: Expert-Curated Questions and Attributed Answers. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies (Volume 1: Long Papers), pp. 3025-3045, Mexico City, Mexico, June 2024. As- sociation for Computational Linguistics. doi: 00.18653//112224mmac// https://aclanthologgr org/ org.oogg//// 2024. naacl-long.
167. URL
Chaitanya Malaviya, Priyanka Agrawal, Kuzman Ganchev, Pranesh Srinivasan, Fantine Huot, Jonathan Berant, Mark Yatskar, Dipanjan Das, Mirella Lapata, and Chris Alberti. Dolomites: Domain-Speciffc Long-Fom Bong-Formaan Methodical Tasks. Transactions of the Association for Computational Linguistics, 13:1-29, 2025a. doi: 10.1162 2t162//accla_00727 URL https:
//aclanthology.orr org/ wwwgg/2225.ee0 tacl-1.1/.
Chaitanya Malaviya, Joseph Chee Chang, Dan Roth, Mohit lyyer, Mark Yatskar, and Kyle Lo. Contextualized Evaluations: Taking the Guesswork Out of Language Model Evaluations. Transactions of the Association for Computational Linguistics, 2025b. URL vrg/abs/2411.01
Theo X. Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Is Self-Repair a Silver Bullet for Code Generation? In The Twelfth International Conference on Learning Representations, 2024. URL htttp:://ppereeve// nww net/forumiid= yoGJXRungR.
Yiwei Qin, Kaiqiang Song, Yebowen Hu, Wenlin Yao, Sangwoo Cho, Xiaoyang Wang, Xu- ansheng Wu, Fei Liu, Pengfei Liu, and Dong Yu. InFoBench: Evaluating Instruction Following Ability in Large Language Models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 13025-13048, Bangkok, Thailand, August 2024. Association for Computational Lin- guistics. doi: 10.18653/ (0/655/vl/20 v222224 2t1//22..indings-adll. findings-acl.772. URL htttp::/accaattholoooo org//024, findings-acl.771 acl.gg--cll.70
Alexander Spangher, Nanyun Peng, Sebastian Gehrmann, and Mark Dredze. Do LLMs Plan Like Human Writers? Comparing Journalist Coverage of Press Releases with LLMs. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 21814-21828, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/ v1 21/2024.emmlp-main..222 URL
org//024. emnlp-main. 1216/.
Yufei Tian, Tenghao Huang, Miri Liu, Derek Jiang, Alexander Spangher, Muhao Chen, Jonathan May, and Nanyun Peng. Are Large Language Models Capable of Generating Human-Level Narratives? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference Empirical Methods in Natural Language Processing, pp. 17659-17681, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/2022 001165//v1/2244mmppmmma 2024. wwn.e-main..pp......mmmmmm
9788. URL https: https://aclaanthooog) org/
Manya Wadhwa, Jifan Chen, Junyi Jessy Li, and Greg Durrett. Using natural language explanations to rescale human judgments. In First Conference on Language Modeling, 2024a.
URL httttss://ppereeiieeee wtwp:eeeeweeeoooonnnieeeeeeeee
Manya Wadhwa, Xinyu Zhao, Junyi Jessy Li, and Greg Durrett. Learning to Refine with Fine-Grained Natural Language Feedback. In Yaser Al-Onaizan, Mohit Bansal, and Yun- Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Pp. 12281-12308, Miami, Florida, USA, November 2024b. Association for Computational Linguistics. doi: 10.18653 / 2024 20024..nniigs-ennlp.7777 URL https:/ haclat//ool/a org/2024. findings-emnlp 716/-
12

---

Published as a conference paper at COLM 2025
Yuning Wu, Jiahao Mei, Ming Yan, Chenliang Li, SHaopeng Lai, Yuran Ren, Zijia Wang, Ji Zhang, Mengyue Wu, Qin Jin, et al. WritingBench: A Comprehensive Benchmark for Generative Writing. ar Xiv preprint ar (rXi:2203.00524, 2025.
Zeqiu Wu, Yushi Hu, Weijia Shi, Nouha Dziri, Alane Suhr, Prithviraj Ammanabrolu, Noah A Smith, Mari Ostendorí, and Hannaneh Hajishirzi. Fine-Grained Human Feedback Gives Better Rewards for Language Model Training. Advances Neural Information Processing Systems, 36:59008-59030 2023.
Shunyu Yao, Howard Chen, Austin W. Hanjie, Runzhe Yang, and Karthik R Narasimhan. COLLIE: Systematic Construction of Constrained Text Generation Tasks. In The Twelfth International Conference on Learning Representations, 2024. URL https: https:://pperreveww net/ forum?ld=kkgglyyiru.
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston, Self-Rewarding Language Models, 2024, URL https://arxxv..orr abs/2401. 10020..
Lunjun Zhang, Arian Hosseini, Hritik Bansal, Mehran Kazemi, Aviral Kumar, and Rishabh Agarwal. Generative Verifiers: Reward Modeling Next-Toke Token Prediction. In The 4th Workshop on Mathematical Reasoning and Al NeuriPS'24 24, 2024. URL https://openreview net/forummid==xxRoTTT
Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, Jun Xu, and Xueqi Cheng. Learning to Control the Specificity in Neural Response Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
1108-1117, 2018.
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging LLM-as-a-Judde with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction-Following Evaluation for Large Language Models. Xiv preprint ar arXi:2231.00791 2023.
A Datasets
A.1 Dataset Descriptions
We describe the datasets considered in this work along with instruction filtering we do to obtain the relevant subset. We also describe any human written or human verified criteria in the dataset.
BenGhnBench (Kim al., 2024a, BGB) is a benchmark with task specific fine-grained evaluation criteria. We only consider the subset of instructions that fall under the categories of theory of mind (writing speech), instruction following (education content creation, executable actions) and planning (constrained planning, travel planning). This dataset has human written evaluation criteria for each instance which rates a response on a scale of 1-5. For each instance we use the provided Likert description for rating score 5.
ChatbotArena (Chiang et al., 2024) a multilingual dataset with real queries. Since some instances in this dataset can be multi-turn, we only consider the first turn. We also only keep instances in English. For further filtering, we keep instructions with a positive creativity and real world tag. We also ensure that the subset of instances have one of the following keywords: 'write', 'develop', and 'create'. This dataset does not have any constraints or criteria provided.
Art or Artifice (Chakrabarty et al., 2024a) has creative writing instructions with expert- curated criteria. We consider all 12 instructions from this dataset. This work proposes
13

---

Published as a conference paper at COLM 2025
14 tests called Torrance Test of Creative Writing for evaluating creativity and short story writing. We consider these 14 as the criteria for the instructions in this dataset.
Dolomites (Malaviya et al., 2025a) is a long-form benchmark for evaluating LLMs on realistic domain-specific writing tasks. The dataset is composed methodical tasks structured in the form of task objective, procedure, input, and output. We use the task objective the main instruction, and the procedure is rewritten into a criterion using GPT-4o to make it of the form 'the response should...'.
WritingBench (Wu et al., 2025) is designed to evaluate LLMs at multiple types of writing such creative, persuasive, informative, and technical writing. These instructions are synthetically generated but manually verified. The task specific evaluation criteria are generated by prompting an LLM for format, length and content-specific constraints and is used for Likert scale evaluation on 1-10. We only keep instructions less than 400 words long. We rewrite the highest rating Likert description into criterion of the form 'the response should... for our analysis.
InfoBench (Qin al., 2024) an instruction following benchmark. We filter instructions to have the following keywords: 'create', 'write' or 'develop'. We also filter out any math instructions. This dataset uses human written decomposed checklists as the evaluation criteria. We use the human written checklists the criteria for this dataset.
WildBench (Lin al., 2024) consists of human-chatbot conversation logs along with LLM- generated and human-verified evaluation checklists. We filter the dataset to keep the first turn of the conversation, instructions which are tagged to be 'creative writing' and 'planning" instructions. Similar to other datasets, we filter these to have the following keywords: 'write', 'create', and 'develop'.
MT-Bench (Zheng et al., 2023) is benchmark containing 80 multi-turn hand designed questions. We only consider the writing subset of this dataset along with the instruction issued at the first turn. This dataset also has preferences between 6 model responses for all the instructions. To get constraints for this dataset, we prompt an LLM to give us a post-hoc reasoning for why a response was preferred over the other, and we convert this reasoning into list of criteria.
A.2 Ask-then-Critique
Our primary aim for collecting a new dataset was to have evaluations conducted by users who are also issuers of the instruction, so that the evaluations are genuine and the evaluators are invested in the problem/ task they are evaluating, To collect this dataset, we added an opt-in study to one of the assignments of graduate level NLP class. Instructions given to them are mentioned in box A.3.
The final dataset has 156 unique responses with 312 instructions and 624 evaluations. One thing to note here, human annotators provide their judgment in the form of natural language critiques. We rewrite these this dataset to be formatted as criteria ('the response should... ..) for our experiments.
A.3 Task categories
Datasets listed above cover broad category of tasks, including finan- cial reports drafting emails. Figure 6 shows a high level distribution of these categories. We prompt gpt-4o-mini for writing category out- put given an instruction and then cluster all of
Technical Writing
Marketing and Promotion
Analysis and Research
Educational Materials
9..9%%..9
9.5%
Documentation
Communication
16.3%
15.3%
3%
Personal Reflection
6.4%
Entertainment and Media
Planning and Organization
14 Writing
Figure 6: Distribution of task categories across datasets consid- this work

---

Published as a conference paper at COLM 2025
the outputs into high
level task categories. We briefly describe what these categories entail:
Creative Writing: tasks like writing fiction sto-
ries, travel blogs, character sketches, academic essays, poetry, lyrics, humorous essays, comedy books, and radio scripts.
Documentation: tasks like synthesizing technical reports, legal documents, scientific reports, internal memos, restaurant menus, and HR reports.
Communication: tasks like writing emails, social media posts, tweets, reviews, letters, speeches, presentations, requests, alerts, and text messages.
Technical Writing: tasks like writing scientific protocols, fitness guides, game designs, self-help guides, and gardening guides.
Analysis and Research: includes different types of analysis like literary, investment, finan- cial, academic, regulatory, and marketing.
15

---

Published as a conference paper at COLM 2025
sasktten-Crrtique Instructions
We want you to explore use cases around AI assistants. You can opt-in for your responses to this part to be included as part of a research study; more details will follow. You will be investigating how LLMs perform and evaluating them across two realistic tasks. You will give LLMs instructions to do various tasks that you might use AI assistants for. For evaluation, we want you to analyze the aspects you to see if the LLM response satisfied you or not. An aspect describes dimension, such as factuality, structure, safety, logic, flow and more.
For instance, you might prompt an AT Assistant with the following: "Write me a 5-day travel itinerary for Austin, Texas around Austin City Limits. This is for blog post." " Some of the aspects I might use for evaluation can be: (1) Correctness: the response correctly talks about things in Austin, Texas; (2) Engagingness: the response is engaging, has personal anecdotes and persuades the user to visit Austin, Texas; (3) Coherence: the response is well structured with good topical grouping of things Your task: Please come up with two different prompts which might be useful to you and which also emphasize a balance of structural, tone and content constraints. Your prompts should follow the following constraints:
They should require a longer form answer (at least 1 paragraph) and shouldn't simply be knowledge based (e.g.: when was Abraham Lincoln born?)
They should be something that you can evaluate for utility (e.g., writing literature review, writing a tweet thread for your paper based on abstract and conclusion, writing an essay on topic etc). We want you to be able to critique these prompts.
You should be comfortable critiquing the output the question (don't t ask about quantum physics if you don't know about it)
Then, get an answer from two of the following models: ChatGPT (www.chatgpt.com) Meta AI (www.meta.ai)
You may use any version of these models you like, either free or paid. Finally, evaluate the response. Your evaluation should be done by you and not use Al assistants. You should evaluate at least 3 aspects.
Gemini (https:/ htttt:://gemiii.gg.. ggemini.google.com.
For each response, what you submit should look like: Prompt: [prompt]
Response (including model name): [response]
[Aspect name: your assessment and reasoning]
Egg, "Engagingness: the response is engaging and the personal anecdotes persuade me to visit Austin"
You will evaluate a total of responses (2 prompts x 2 models) in this format with at least 3 aspects each. Each (prompt, model) combination can use distinct aspects, so you could have 12 (or more), or as few as distinct aspects. You should find flaw in at least 2 of the responses. To achieve this, you
of the following strategies: (1) Ask about more obscure knowledge; (2) Place more constraints on the response; (3) Ask for more interesting output formats (a Linkedln post, TED talk, etc.).
Ideally your prompts should be as creative and distinct as you can make them: if something comes to mind from your work or particular life situation, that's great. However, please don't add any personal identifiable information that you are not comfortable with the course staff potentially seeing as they grade the assignments.
Educational Materials: includes planning and writing tasks like creating tutorials, curricu- lum, syllabus, quizzes, rubrics, and flashcards.
includes planning and writing tasks like educational podcasts, promotional videos, and music playlists.
Entertainment and Media:
Marketing and Promotion: includes writing tasks like business proposals, advertising slogans, and product pitches.
16

---

Published as a conference paper at COLM 2025
Personal Reflection: includes writing tasks like personal advice, reflection blogs, social media bios, parenting advice, and pet care blogs.
B EvalAgent Parameters and Prompts
In this section we describe each step and parameters of our proposed method along with the corresponding prompts.
Step 1: Query Generation The first step of our method query decomposition. Using the prompt given below, formulate queries that will help retrieve instructional URLs. We generate 2-3 queries per instruction.
Prompt B. :: Query Generation
Instruction: {{ Instruction }}
What should google learn useful advice on writing a great response the above instruction? Give a JSON response with three most important google queries as keys. The queries should be in the form of "how to" or "what are" etc and the value is what we want to learn from the query. The queries should focus on seeking actionable writing advice instead of focusing abstract knowledge concepts.
Step 2: Retrieving Expert Advice For each query generated in Step 1, we run the query through the Google Search API and retrieve 30 URLs. Each URL content rated based on two metrics define in Section B.1: (1) absolute rating and perceived expertise of the content (prompt B.8) (2) absolute rating on a scale of 0-2 for the relevance of the content to the query and instruction (prompt B.9). We sort the URLs by the content relevance and scored expertise and choose the top-5. The selected URLs have an average URL rating of 3.77 (on scale of 1-5) and an average relevance rating of 1.48 (on scale of 0-2). We show examples of instructions, along with the queries and corresponding URLs and text snippets Table 16.
scale of 1-5 for the goodness
Once we have the filtered URLs, we pass them through the document-groundee QA prompt (listed below) and get ry for each query.
Prompt B. 8:2: Document Grounded Query Answering
###
Article: {{ article }}
Answer the following question from the article above: {f query The question help me write an answer
following instruction: {{ Instruction }}
ONLY answer the question and not the instruction.
I am looking for nuanced advice to the question above, include any obscure minute points like structuring the content, the tone, highlighting important things etc. the article does not have any useful advice, then return "no answer".
At the end of this step, we combine content from all URLs using the summary prompt listed below to get list of summaries for each query.
Prompt 3.3:
Summarize izing URL Content for a Query
## Instruction: {{ instruction
}}
order evaluate response the above instruction, looked up different expert advice on {{ query }}. found the following two responses: ### Expert Advice 1:
{ advice_1 H
Expert Advice
17

---

Published as a conference paper at COLM 2025
{I advice_2
## Your goal is to identify new information in Advice compared to Advice Once you have identified that information, integrate it in Advice List 1 and return a combined list of points. Do not delete anything from Advice 1.
Include any unique and nuanced details from Advice 2, eg: any thoughts how the structure should be, tone etc. there examples what phrasing etc to use please include them under any relevant points. You MUST NOT draw any inferences on your own. DO NOT answer the instruction. You only choose to merge or combine details as needed.
Step 3: Criteria Generator The final step in proposed method combines all criteria lists generated for each query in Step 2 and aggregates that information. The response then goes through re-writing and filtering prompts to ensure alignment to the instruction and removal of any irrelevant criteria.
We use the following prompt for combining all query query-specific criteria. criteria. We do this two-criteria lists at a time.
Prompt B.4: Aggregating Query Content
Advice list 1: {{ List1] }}
### Advice list 2: ((list]] }}
## Add any new information details from List to List Don't delete anything from List ## Include any unique details from List 2, eg: any thoughts on how the structure should be, tone etc ## If there are examples, please include them under any relevant points ## Summarize the information into one list of points, each point should be unique and structured such that it reflects evaluation criterion.
We use the following prompt re-write the instructional web-document criteria to be checklist and in the form of criteria that can be used for evaluation.
Prompt B.5: Re-wris ting Set
### Instruction: {{ instruction }}
### Tips/Suggestions:
{f tips }}
Convert the tips to constraints/ conditions so they can be used for evaluating a response to the given instruction. Eg: "the response should.."
Constraints should: **B answerable by 'yes' 'no"**, with 'yes' meaning that the response successfully met the corresponding requirement.
**Be comprehensive, but concise**, meaning that all criteria directly relevant to the instruction should be represented by question, but only questions that are very clearly relevant should be included.
**Be precise**, meaning that constraint should avoid vague wording and evaluate specific aspects of response, directly using the phrasing of the instruction where appropriate. Give à new line separated numbered list, where each line is one constraint. Ensure they are rephrased to align with the instruction.
After re-writing we run the criteria list through a filtering step that removes any out of scope or irrelevant criteria.
Prompt B.6: Filter
## Instruction:
{[ instruction }}
Criteria for evaluating a response to the above instruction:
18

---

Published as a conference paper at COLM 2025
H criteria }}
For the above criteria, which of them can be used to evaluate response to the instruction? Note: the response is not interactive, we cannot evaluate if the response went through iterative revisions and there is no feedback process. If there is any criteria evaluating for these, then mark them
Think step by step and give response for each criteria point, end the reasoning with "therefore, the criteria applies yes" or "therefore, the criteria does not apply no". Return a JSON where the key is the criteria number and the value is the reasoning for whether or not the criteria applies.
Rank: this step we re-order the generated criteria from above based on the relevance of criterion to the instruction. We prompt LLM for this rating scale of 0-10.
Prompt B. 7: Relevance Ranking
How relevant is the following aspect to evaluating text only response to the following instruction? Note, that we cannot these for evaluating iterative editing etc. so any aspect which mentions that should be rated low.
Instruction: {{ instruction }
Aspect: {{ assect }}
Think about step by step and return a number between 0 and 10. End your response with 'therefore, the score is:". 0 is the lowest relevance and 10 is the most relevant/ useful aspect.
B.1 URL Filtering
As mentioned in the section above, we filter the retrieved URLs based on two criteria (a) the goodness of the URL content (b) the relevance the content to the instruction. We list the prompts used for this rating below.
Prompt 8.8: URL Content Rating
## URL: {{ url }}
Content the URL: {{ content }}
Rate the goodness of the above URL and content based the following criteria: Score: Very Poor
Expertise: Content appears unprofessional or amateurish, with little evidence expertise. Reliabil- ity: The domain is untrustworthy or unknown, such as suspicious .com sites, clickbait domains, or personal blogs with credentials. Clarity: Information is vague, confusing, contradictory, with actionable advice. Marketing: The content is overtly promotional, focused primarily selling a product or service rather than providing useful information. Score: Poor
Expertise: Some effort to appear knowledgeable, but lacks depth or substantiation (e.g., no citations, shallow explanations). Reliability: Domain is not widely recognized, the platform has mixed reputation. Clarity: Advice somewhat unclear or too generic to useful. Marketing: There noticeable marketing agenda, but it does not entirely overshadow the content.
Score: 3 Fair
Expertise: Content demonstrates moderate expertise, though it may lack nuance depth. Basic credibility is evident. Reliability: The domain is moderately reputable (e.g., well-known coo site, or an .org/ heg//ed//aae gov site with minor credibility concerns). Clarity: Information reasonably clear and somewhat actionable, though not highly detailed or specific. Marketing: Some promotional elements are present but do not dominate the content. Score: 4- Good
Expertise: Content well-researched and written by someone with clear expertise or authority the field. Reliability: The domain highly reputable, such as a trusted ,edu, -go, or widely
19

---

Published as a conference paper at COLM 2025
respected sorg.org site. Clarity: Advice is clear, specific, and actionable, with practical steps or examples. Marketing: Minimal promotional content; the focus is providing value the reader.
Score: 5 Excellent
Expertise:
Content reflects deep expertise, with authoritative writing, detailed explanations, and citations or references to credible sources. Reliability: The domain is extremely trustworthy, such government, academic, highly esteemed organization. Clarity: Advice is exceptionally clear, highly specific, and immediately actionable, tailored to the reader's needs. Marketing: No marketing agenda evident, present, it subtle and does detract from the quality the information.
Only return the score number at the end.
Prompt 8.9: Instruction, URL Content Rating
## URL: {{ [rr]] }
## Title: {{ [itle]]] }}
Article:
{f article }}
Does the web content provide any advice that will help write response the following instruction:
Instruction: {{ instruction }}
2 yes and it provides specific advice that will help write response to the instruction
1 y-ys but it only provides general advice that will help me write response to the instruction
0 n-no, it doesn't help at all
Only give the numerical response and description.
C Computational Cost
The cost for running Avalt and its corresponding components are given in Table 6. We report the input and output tokens per call for each of the steps in our proposed work, along with the total number of tokens and associated costs. We see that the bulk of the cost is the filtering step during retrieval.
D Baseline Methods
In this section we describe the prompts used with the baseline methods.
Prompt 1.1: Instruction Decomposi tion
### Instruction: '{ instruction } Decompose the instruction
the constraints that the instruction explicitly mentions. Then convert the constraints into checklist. The checklist response should have each new line as one numbered point. Each point should start with "the response should". Do not mention any constraint that not explicitly in the instruction.
Prompt D.2: LLM
Help judge an AI assistant's response to an instruction by providing an evaluation checklist. ### Instruction: '{ instruction }
## Task Details Your task is to come up with evaluation checklist list for given Instruction. list of questions that ask whether or not specific criteria relevant to the instruction were met by an Al assistant's response.
This evaluation checklist should
Criteria covered by your checklist could be explicitly stated in the instruction, or be generally sensible criteria for the problem domain. You should, however, try to be concise and not include unnecessary entries your checklist. Checklist questions should:
20

---

Published as a conference paper at COLM 2025
*Be answerable by 'yes' or 'no"*, with 'yes' meaning that the response successfully met the corresponding requirement.
*BBe comprehensive, but concise**, meaning that all criteria directly relevant the instruction should be represented by a question, but only questions that are very clearly relevant should be included.
BB precise" meaning that checklist questions should avoid vague wording and evaluate specific aspects of response, directly using the phrasing of the instruction where appropriate. Give a where each line corresponds to one factor. The factors should start with 'the response should'.
Prompt D.3: LLM-1
Help judge an Al assistant's response to an instruction by providing an evaluation checklist. ### Instruction: '[ instruction
## Task Details Your task is to come up with evaluation checklist list for given Instruction. This list of questions that ask whether or not specific criteria relevant to the instruction were met by an Al assistant's response. Criteria covered by your checklist could be explicitly stated in the instruction, or be generally sensible criteria for the problem domain. You should, however, try to be concise and not include unnecessary entries your checklist. Checklist questions should: *Be answerable by 'yes' 'no"** with 'yes' meaning that the response successfully met the corresponding requirement. "BB comprehensive, but concise** meaning that all criteria directly relevant to the instruction should be represented by question, but only questions that are very clearly relevant should be included. *eB precise** meaning that checklist questions should avoid vague wording and evaluate specific aspects of response, directly using the phrasing of the instruction where appropriate. Give a list where each line corresponds to one factor. The factors should start with 'the response should'. Return 30 factors.
evaluation checklist should
E
Human Evaluation
To assess Utility and Obviousness of the generated criteria, we hire two annotators on Upwork and one annotator who has bachelors degree in linguistics. Our Upwork qualifi- cations included a success rate of at least 90% and native or bilingual speaker of English. We paid base rate of /oour
Each annotator went through qualification round post which they were given the main task. In total, the annotators gave judgments 700 criteria across datasets and systems with 54 unique instructions. Table 8 shows the total number of criteria sampled across datasets and systems.
21

---

Published as a conference paper at COLM 2025
Annotation Instructions
We are broadly interested in evaluating LLM responses to an instruction. Step towards this research is to first define and identify criteria that would be useful for evaluating such responses. We built system that generates instruction specific criteria for evaluating LLM responses. We would like your help to evaluate these criteria!
Context:
Goal for this annotation study: Given an instruction and a criterion, we want to understand the quality of this criterion and its relevance in evaluating an LLM response to the instruction. You will be given: (1) An Instruction: A prompt for generating an LLM response (2) A Criterion: A criterion for assessing the quality of the LLM response. Your Task: You will rate the evaluation criterion along two dimensions: Obviousness and Relevance.
Obviousness (Yes or No ): Does the evaluation criterion clearly follow from the instruction?
(Not Obvious): The criterion not explicitly mentioned in the instruction. It is also not obvious.
1 (Obvious): The criterion either explicitly stated in the instruction or is common sense for assessing the response.
Example:
"The response should have correct grammar" is criterion that might not be explicitly stated but is fairly obvious- Yes, obvious "The response should have bolded text for highlights" might not be stated in the instruction but is not obvious No, not obvious
Relevance (Scale: generated response?
3) How important the criterion for assessing an LLM
i (Not Relevant): The criterion has little to no connection to the instruction, so won't be relevant to evaluating an LLM response to the instruction
2 (Somewhat Relevant): The criterion is "nice-to-have" but if it's not met by the
LLM response, it would still not significantly impact LLM response quality
3 (Highly Relevant): The criterion is essential not met, the LLM response quality might be noticeably affected.
E.1 Annotator Agreement
Table 7 shows the agreement between the annotators on utility and obviousness bro- ken down by method. We use Fleiss Kappa to calculate agreement. Lower agree- ment between annotators is expected by systems that generate implicit criteria, as what is obvious to one annotator may not be obvious to another (i.e. some may be experts in a topic while others may not). Similarly, for utility, one annotator may find implicit criteria to be very impactful while others may find it to be only slightly useful or a "nice to have". Because of this subjectivity in annotation, we re- port the average rating of obviousness and utility in our metrics over majority vote.
Method
O
U
ID
0.88 0.64
E.2 Average Utility and Obviousness Scores
LLM-n
0.23
0.25
EvalAgent
0.25
0.26
Human 0.26 0.29
We report the average annotator utility and obvious- ness scores in Figure 7. We break these averages down further by method and dataset in Figure 8 and Figure 9. While Figure 8 shows comparable Utility, the Obviousness rating from the annotators differs, showing that EVALAGENT consistently finds less ob- vious criteria with high utility.
Average 0.34 0.34
Table 7: Average annotator agree- ment for Obviousness and Utility broken down across methods.
22

---

Published as a conference paper at COLM 2025
Average Utility score by Annotator
Average Obviousness score by Annotator
Average: 2.67
Average: 0.75
3,0
2.71
2.64
2.65
Proportion "Yes" Responses
0.85
Average Rating (1-3)
0.8
0.75
0.64
1.5
0.4
0.2
0.5
0.0
Annotator
Annotator
Annotator
Annotator
Annotator
Annotator
Figure 7:
Average scores for Utility and Obviousness from annotators.
3.00
Utility by Method and Dataset (Average Rating 1-3)
3.00
2.76
3.00
2.56
3.00
3.00
2.89
2.97
3.00
Metrod
2.42
2.00
LLM-n
2.69
2.42
2.82
2.57
2.83
2.00
2.67
2.55
2.67
BoB
Chartet Arena
Art Artifice
Dotamites
InfeiBench
MT Bench
Ask totttnen-
WhltBeach
WeitigBench
Figure 8: Average of Utility from the human annotators across methods and datasets
F Automatic Metrics
E1 Lexical metrics: Specificity and Word Overlap
Dataset
ID
LLM-n EA-Web Human Ins
Specificity S: We calculate speci- ficity of criterion by taking max- imum of the normalized inverse word frequency of the words in the criterion. Given criterion c, calculate NIWF as the following:
BGB
23
36
36
8
8
ChatBotArena Art or Artificie
7
15
3
16
25
28
13
6
Dolomites InfoBench
12
21
20
23
5
22
35
25
24
9
MT Bench
12
19
20
15
4
Nik-then-Critiguee
15
20
19
6
WildBench
10
14
14
10
3
log(1 + [R])
WritingBench
27
46
41
38
10
NIWF. max WCC
fw
Total
144
238
219
150
54
(1)
This is reasonable since a criterion is specific as long as it contains some specific words. We calculate the frequency of word in the
Table 8: Distribution of criteria and instructions across
datasets and systems for the human annotation task.
23

---

Published as a conference paper at COLM 2025
1.00
Obviousness by Method and Dataset (Proportion "Yes")
1.00
0.86
1.00
0.72
1.00
1.00
0.93
0.97
1.00
Mcthoo
ExWeb
0.88
0.35
0.80
0.57
LLM-n
0.79
0.67
0.81
0.70
0.60
0.80
0.79
BG3B
Chanbos Arena
Art or Artifice
Dolamitas
IntsBench
MT Bench
Ask cit-ttee--rttquuu
Wilttechh
WritengBench
0.00
Dataset
Figure 9: Average scores of Obviousness from the human annotators across methods and datasets.
tire corpus of criteria generated across datasets and systems. This is done to ensure that the specificity is being calculated against the same pool. We report the specificity scores for all datasets in Table 9.
We also look the self-specificity of the method, i.e., specificity of criterion generated by method with respect to pool of criteria generated by the same method. We show the distribution of scores Figure 10. From this figure we see that ID tends to be the most specific, when considering its own pool of criteria. This is intuitive since the instruction level constraints are generally unique to the instruction (unless there are common things shared across instructions like 'this should be a New Yorker style story'). For LLM, when only generate the natural distribution of criteria (without over generating), we see that has higher specificity well. This is because without over generating, LLM tends to give you more instruction level and explicit criteria, which causes it to be more specific. For LLM-n, we see that specificity scores are lower than EA-Web. This probably because over generating the criteria will give some that are shared across instructions. For example, for lot of structured writing tasks like TED Talks or Blogs could be 'the response should be have a clear introduction, body and conclusion', which lowers the specificity of criteria.
(I) Distribution of Specificity Scores
Implicitness I To calculate the implicit- ness of criterion c, we look at the word overlap (WO) of the criterion with the prompt x and report 1-WO. The ratio is calculate following:
Method
ID
EA Web
LLM
LLM-n
Density
10
((x) (W())
8
cx(x,))
[W(c) +
6
Where W(p) is set of non-stopword tokens from the lowercase
We report the implicitness scores for all datasets in Table 10.
0.0
0.2
0.4
0.6
Specificity
Figure 10: We look at the self-specificity of each method, i.e. specificity of criterion gen- erated by method with respect to pool of criteria generated by the same method.
E2 Actionability
We define actionability of criterion
A, 1 ( ((x,))((... = -
24

---

Published as a conference paper at COLM 2025
where f is criteria-based scoring which gives binary judgmenns. x the prompt, y is initial response generated by G. When given a criterion e, G(x, y, edits y to try and satisfy
Data split: To conduct this experiment we sample 200 instructions across datasets consid- ered in this task. Note, we leave out ChatBotArena, MT-Bench from this analysis. Both are preference datasets with no human written criteria to compare with.
Models We calculate actionability with respect to three models GPT--0-20024-11-20 2audd=3---oomnet-2221002 and Lama-3.1-8B-Instmun The model used for generating the intial response is the same as the model used for editing.
We use the following prompt to evaluate if criterion is satisfied by response. This prompt returns binary judgment.
Satisfaction prompt:
Dataset
# instructions
BGB
34
Prompt 1:
scoring prompt
Art or Artifice
12
Dolomites
26
You are a teacher, evaluate a student's response to an instruction.
InfoBench
34
WildBench
32
A student is given the following instruction to answer: Instruction:
WritingBench
34
instruction }}
Ask-then-Retriews
31
Student's Response: {{ response
}
Table 12: Number of instructions con- sidered for the the actionability ex- periment in Section 6.
Does the response satisfy the following criteria: {{ criterion] }}
Think step by step and end your thought with 'therefor, the answer is yes' or 'therefore, the answer is no'.
Editing prompt: We use the following prompt edit model responses that do not satisfy criterion.
Prompt F.2: Editing Prompt
Instruction:
{{ instruction }}
Response:
{{ response
}}
## The above response to the instruction does not satisfy the following constraint: [{ criterion }}
## Edit the above response such that satisfies the above constraint. Do not include a preamble.
Actionability vs Obviousness: We test how actionability is related to obviousness of generated criteria. We consider the human annotated subset for this experiment. First, we generate initial responses using CPT-44-2024-11-20 and then calculate an initial pass rate with criteria generated from different systems. We use apt-4o-mini-2024-077 evaluation. We then plot this against the human annotated obviousness judgments.
for
Figure 11 plots how the initial fail-rate of criteria (criteria that satisfied in the initial responses) varies with the human evaluations of obviousness per criteria. We find that EVALAGENT consistently generates criteria that is less obvious and less likely to be initially satisfied indicating challenging and useful criteria.
generated and not
E3 Recall
We evaluate how frequently human-generated or verified criteria for a given instruction are represented in the generated criteria. To calculate this use the following prompt:
25

---

Published as a conference paper at COLM 2025
EA-Full (EA-Web LLM-30)
EA-Full (oracle)
LLM-30+30
LLM-30+30 (oracle)
WeitingBench
0.79
0.82
0.60
0.52
0.64
0.64
0.45
0.54
0.37
0.48
0.45
0.30
0.40
0.23
0.33
0.27
0.25
0.18
0.08
0.17
0.01
20
40
60
20
40
60
20
40
60
Figure 12: We report the oracle results for Recall@k across three datasets. There a big gap between the relevance ranking we use the oracle. We leave the development of better scoring function to future work.
Prompt F.3:
Recall Prompt
## have existing list of evaluation criteria. I want to compare new criterion this list. The lists are given below: Existing criteria: [[ existing_cri critiiaa }}
New criterion: {{ new_criterion }}
## Determine if the new criterion already mentioned the existing list. You can generalize the new criterion see it aligns with the existing list or vice-versa Return JSON in the following form:
"copy of point from the general point above":"is the specific criterion entailed in the existing list not"
}
For prompt x, if there are H human- criteria, we run all E through the prompt above and get one decision for the prompt x. We then calculate recall@k for instances (N) as following:
Us Respoese satisfaction
Doites
Less obvious & less satisfied
64
N
H the entailed list
Recall@k
N
H2
Recall@k with oracle: Figure 12 shows also the oracle performance these sys- tems. This shows that there is significant gap between the ranking method and the oracle. We leave improving the ranking of criteria for future work.
Figure 11: Trend of the percentage of unsatis- fied criteria by how unobvious they are across methods and datasets.
26

---

Published as a conference paper at COLM 2025
Dataset: BGB
Instruction: Design a travel plan for a tourist traveling to the given destination. The tourist has a list of requirements and you should design your plan such that it satisfies all of these requirements.
Destination: Paris
Requirements:
Total Duration: 2 days and night
Trraspoortation Walk
Must Have: Eiffel Tower, Louvre Museum, Escargot
Optional: Croissant, Onion Soup, Notre Dame Cathedral Constraint:The response presents
holl-thuught-uut, efficient, and realistic itinerary that includes all must-have experiences within the walking-only constraint and incorporates all optional items, demonstrating excellent planning and optimization for an enriching tourist experience.
Dataset: Art or Artifice
Instruction: Write a New Yorker style fiction given the plot below.
Make sure it is atleast 1500 words.
Directly start with the story, do not say things like 'Here's the story [...]=
Plot:
An observer becomes entranced by seemingly ordinary couple on the street, follows them home, and then watches them from outside in the rising floodwaters, drawing an cerie connection between the woman and discarded, burned chair they'd noticed earlier.
The response should have effective narrative pacing controlling the perceived speed and rhythm at which story unfolds
Constraint:
Dataset: Dolomites
Instruction: Draft comprehensive legal framework for the use of blockchain technology in the creation and transfer of digital assets within the real estate industry. Constraint:I should include precise review of pertinent real estate laws and regulations, identifying any recent changes or updates.
Dataset Watasee:WittingBench
Instruction:Plean help me write review of the TV drama "The Long Night" (Silent Truth). Constraint: The response should have meaningful insights into the complexity of the plot, including narrative arcs and story development.
Dataset: InfoBench
Instruction: Generate non-disclosure agreement of two pages (each page is limited to 250 words) for a software development project involving Party A and Party B. The confidentiality duration should be 5 years. The first page should include definitions for key terms such as confidential information', 'disclosure', and 'recipient'. On the second page, provide clauses detailing the protocol for the return destruction of confidential information, exceptions to maintaining confidentiality, and the repercussions following a breach of the agreement. Please indicate the separation between the first and second pages with full line of dashed lines ('-').
Also, make sure that each page is clearly labeled with its respective page number.
Constraint:Does the second page of the generated non-disclosure agreement provide clauses detailing
the protocol for the return or destruction of confidential information?
Diddech
Instruction:Create a training program that can be done home without any equipment, and without a pullup bar. It must be heavily focused at muscle hypertrophy and strength gain. Training days are 6 times week, and one extra day is the rest day. Don't add any cardio, and include ab and core exercises in the daily program instead of reserving specific day for those exercises. Every single muscle in the body must be trained least twice week, with maximum focus towards gaining muscle.
Are the exercises suitable for being performed at home without any equipment or a pullup bar?
Constraint:
Table 5: Examples of instructions and corresponding dataset constraints from the datasets datasets considered in this work.
27

---

Published as a conference paper at COLM 2025
Input tokens (per call)
Output tokens
Total
EA-Web Cost
Num calls
(per call)
(with GPT-4o-Mini)
(1) Query Generation
113
$0.00007
(2) Expert Retrieval
Filter (for three queries)
33
7705
10
$0.04
28
3990
16
$0.02
31
5410
$0.03
Query-Answering
15
5446
249
$0.01
Summarization
1211
795
$0.00
(3) Aggregation + Criteria Generation
1367
723
$0.003
Total
117
627009
11761
$0.10
Total (without filtering)
25
93331
10691
$0.02
Table 6: Computational costs for running EvalAgent broken down by three steps: (1) Query Generation; (2) Expert Retrieval; (3) Criteria Generation. For Step (2) we show the step broken down by different prompt types used.
Dataset
ID
LLM
LLM-n
EA-Web
Human
BGB
0.17
0.11
0.09
0.16
0.48
Art or Artifice
0.15
0.15
0.1
0.15
0.06
Dolomites
0.06
0.05
0.06
0.16
0.19
InfoBench
0.09
0.08
0.07
0.13
0.13
Ask-then-Critique
0.13
0.09
0.09
0.13
0.22
WildBench
0.19
0.13
0.11
0.14
0.2
WritingBench
0.15
0.11
0.11
0.13
0.16
Average
0.13
0.10
0.09
0.14
0.21
Table Specificity (S) values for criteria generated by different methods for datasets considered in this work.
Dataset
ID
LLM
LLM-n
EA-Web
Human
BGB
0.48
0.67
0.81
0.87
0.79
Art Artifice
0.5
0.67
0.86
0.96
0.98
Dolomites
0.65
0.84
0.89
0.95
0,93
InfoBench
0.41
0.62
0.8
0.83
0.54
ssk-then-Critiquee
0.52
0.74
0.87
0.9
0.89
WildBench
0.51
0.67
0.8
0.87
0.73
WritingBench
0.4
0.51
0.69
0.78
0.81
Average
0.50
0.67
0.82
0.88
0.81
Table 10: Implicitness (T) values for criteria generated by different methods for datasets considered in this work.
28

---

Published as a conference paper at COLM 2025
(Ask-then- hrateeeeCcctiqeelnnsttttttttt want to give a short podcast on why sleep is important to the well-being of an individual. However, my audience is those who rarely get sleep. How can I go about this?
System
Criterion
S
I
Human
Response should have depth in discussing the importance of sleep, pro- viding detailed talking points, examples, and relevant reasons. sleep, acknowledging their feelings and experiences.
0.07
1.0
Response should have empathy, acknowledging and understanding the feelings and struggles individuals who rarely get sleep.
0.1
0.7
LLM
The response should suggest ways to incorporate humor or light-heart- edness, appropriate.
0.5
0.83
The response should include summary of key points to reinforce the 0.005
1.0
message.
EA-Web
The response should focus the emotional impact of sleep deprivation 0.5
0.9
on relationships, work, and daily life. The response should provide practical takeaways or actionable advice 0.009
1.0
after sharing personal stories.
Table 11: Examples of criteria generated by different systems along with the Specificity (S) and Implicitness (J) values for an instruction from the Dolomites dataset. We highlight the word in each criterion with the highest specificity.
(WritingBenchh) Instruction: Please write an analysis report on the revenue of wealth management products for China Merchants Bank and China Construction Bank (with length of 3000-4000 words), based on the following annual report data of wealth management products from 2018-2022 for both banks: #Bank 1 Annual Details #Bank 2 Annual Details
System
Criterion
Actionable
Implicit
LLM-n
The response should analyze the net profit for both banks from 2018 to 2022.
X
The response should provide a comparison the total assets of both banks in 2022.
x
EA-Web
The response should contain a concise executive summary that captures the key findings and recommendations regard- ing revenue trends, profitability, and significant changes over the years.
The response should consider different scenarios that could impact future performance, discussing potential risks and opportunities.
Ask-then-Crittiuee) Instruction: am getting a new 8-week old golden retriever puppy. Please document the materials will have to gather and things need to prepare for this new adventure. Additionally, please give me specific tips for raising a golden retriever. want comprehensive guide for owning new puppy first-time dog owner. Thanks!
LLM-11
The response should highlight the benefits of adopting from shelter or rescue.
X
The response should address the financial responsibilities of owning a puppy:
EA-Web
The response should suggest a daily schedule that includes feeding, potty breaks, exercise, and playtime. The response should explain the grooming needs specific to golden retrievers and the importance of regular grooming.
/
Examples of criteria generated by LLM-1 and EA-Web that were initially unmet by GPT-4o response. We see that LLM-11 generate instructions that less implicit and more grounded in the instruction.
Table 13:
29

---

Published as a conference paper at COLM 2025
Human criteria
EA-Web Generated criteria
Response should provide logical flow that addresses the client's issues, the pro- posed solution, and the expected ROL.
The response should emphasize the return on in- vestment (ROT) for the client.
The response should have a clear structure, includ- ing sections for introduction, needs assessment, proposed solution, benefits, implementation steps, and Q&A.
Response should be mindful of time straints, ensuring that all points are rele- vant and succinct to fit within a typical onehour meeting.
The response should be concise and clear, avoiding overwhelming the audience with information.
Response should not include overly lengthy paragraphs but rather concise points that can be easily articulated.
The response should be formatted in a way that easy to follow
The response should be concise and clear, avoiding overwhelming the audience with information.
Response should have clear structure that is easy to read and organized, prefer- ably in a bulleted format
The response should provide structured itinerary divided into clear sections for each day of the week- end trip
Response should be complete and pro- vide descriptive details about activities in each area.
The response should include activities or attrac- tions to visit in Boston.
The response should include outdoor activities to enjoy the fall foliage.
The response should highlight activities and excur- sions that immerse visitors in the local culture.
The response should engaging and sensory- rich descriptions to bring local attractions life.
Response should have fluid structure that feels personal and heartfelt, rather than a list format.
The response should include personal touch, such as memorable experience or lesson learned
The response should mention any lessons learned during the author's 's time at the organization
Table 14: Examples of human written criteria in the Ask-then-Critique dataset that matched with corresponding E.AWWeb generated criteria and had no match in LLM-n. Our system captures criteria that are important for humans when evaluating responses to their instruc- tion.
30

---

Published as a conference paper at COLM 2025
Method
Criterion
u
Instruction :Act meta worker write me good caption for my reel which is based fun dance good vibes great company
LLM-n
The response should not be overly complex or convoluted. 12.77
The response should reflect the personality of the creator or 2.67
brand.
The response should include relevant emojis to enhance the 2.00
visual appeal of the caption.
EA-Web The response should incorporate humor or light-hearted 2.67
tone in the caption.
Instruction:
Craft an intriguing opening paragraph for a fictional short story. The story should involve a character who wakes up one morning to find that they can time travel.
LLM-R
The response should align with the theme of time travel.
3.00
The response should avoid ambiguity regarding the charac-
2.33
ter's 's ability.
EA-Web
The response should create movement by transitioning from specific moment the broader implications of the character's 's
2.00
newfound ability to time travel. The response should subtly hint at the challenges or adven- 2.00
tures that lie ahead for the character.
Table 15: Examples of human annotations for two instructions and their corresponding criteria generated by LLM and EA-Web. We show human evaluation scores: Avg Utility (U) across annotators and the number of annotations that marked the criterion obviousness (#O).
31

---

Published as a conference paper at COLM 2025
(WritingBench) Instruction: Prepare a plan for a show in an art gallery. Query: How to plan a art exhibition
Aspect the response should create detailed timeline for the exhibition planning process
URL:
coeeeecoo////eeeeeee www/howwtowwoooo heou-to-organiz---succe aussscsssuuuaaaattetiittio
Snippet:
Develop a timeline and budget. solid timeline and budget are essential for smooth execution. Break down tasks and allocate resources for: Space rental, Artwork materials and framing, Marketing (digital and physical), Digital tools (e.g, virtual tours or AR components), Factor in sustainable practices, such as renting materials, using recycled supplies, or partnering with eco- conscious vendors, to minimize your environmental footprint. Include contingency fund for unexpected expenses avoid last-minute stress.
(Art or Artifice) Instruction: Write a New Yorker style fiction given the plot below. Make sure is atleast 1500 words. Directly start with the story, do not say things like 'Here's the story [...]'Plot: A solitary man walking in a remote mountainous region comes across a car crash, and stays by the side of the lifeless female victim, narrating stories of his past and reflecting the impermanence of events and life itself, while awaiting emergency services amidst the looming presence of wilderness.
Query: how to build atmosphere and setting in writing
Aspect: the narrative should juxtapose memories of life with the stark reality of death to emphasize the impermanence of life.
URL: https:/ / www.thhsaaurus.omm com /
Snippet: this non-fiction travelogue, David Foster Wallace talking about his experiences on luxury cruises. He opens by placing the reader directly onto a cruise ship. In the span of a paragraph, the reader experiences awe, curiosity, amusement, disgust, wonder, and excitement. Yet Wallace uses formal language ("I have seen") and repetition (there's s that anaphora for you) to ironic effect. This creates an interesting juxtaposition of the elements of tall tale with a bit of anthropological distance. This example, in particular, shows how mood can function independently from the atmosphere, and how both can change abruptly with the use of language.
Besk-then-rritiquee Instruction: Write a TED talk about how to resume your career the United States after a long pause from professional work due to familial responsibilities. Explain how I had been in a similar situation and how to deal with its mental and financial implications while providing resources to do so. Include placeholders where can include personal details needed.
Query: what are key elements of a compelling TED talk
Aspect: the response should include a personal anecdote about resuming career
URL:htips:/ /wws/siiiiiiss.iom wwtmminutes.llugan..om nom/hoow httphhhwtttoooe bo--deliiertaak--teee
Snippet: Tell a story that hasn't been told before As journalist I had an advantage. I'm professional storyteller. Yet still had to find new story, a story about being homeless that hadn't been told before. So told my story. It's easy to hide behind talking about other people in similar situations, with similar issues, but the powerful story, the one people want to hear, is your story.
Table 16: Examples of instructions, queries generated along with the evaluation aspect and it's grounding URL with snippet.
32


Reviews:
[{"review_id": "qkpHI5om3U", "reviewer": "Reviewer_tKpV", "summary": "The paper proposes a pipeline-based approach that identifies implicit evaluation criteria for structured writing tasks. This is done in a three-step sequence: (1) generation of queries that retrieve instructional Web documents which contain evaluation criteria, (2) a filtering and summarization step that narrows down what has been retrieved in the first step, and (3) a generation step that takes the output of the second step and turns that into evaluation criteria. Each step in the pipeline is done via prompt engineering accessing a LLM. Subsets of existing benchmarks are used to evaluate the work adopting technical measures as well as crowd-based assessments.", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is topically an excellent fit for COLM. It will be of interest to a fair number of attendees and is likely to stir some discussion.\n* The authors provide an extensive appendix explaining every detail of the approach which helps making sure that the work is fully reproducible which is a major strength (however, the downside is that the extensive appendix makes the paper less self-contained than it could be as the reader needs to refer to information in the appendix to fully follow the discussion so that the paper reads more like a 15-page paper than a 9-page paper).", "reasons_to_reject": "* The biggest weakness in my view is the lack of an extrinsic evaluation of the work. The authors argue that they “leave the exploration of downstream applications to future work” but without such evaluation the contribution is a lot less compelling than it could be, in particular as there is no evidence that the proposed methodology will beat plausible baselines. \n* In part the work feels more like an elegant engineering solution than a contribution that offers substantial new insights. A contributing factor is the choice of a number of ad hoc settings (such as “google queries”, “30 results”, “above 2”, “above 0”,  “top-5” …) \n* The paper would benefit from a Limitations section. While not strictly a requirement it does become clear that a number of experimental choices need to be critically discussed in the light of generalisability and other possible limitations. \n\nMinor issue: \n\n*  “Specificity” is perhaps slightly confusing as a term as it is commonly understood to refer a different concept in the context of evaluation (true negative rate).", "questions_to_authors": "* The choice of LLM is not really motivated. What is the reason for the choice? How do the findings compare to a different LLM? \n* The reference to significance testing in Table 2 is not quite clear. The difference is significant compared to what? Also, do you apply a suitable correction such as Bonferroni? \n* Would it not make sense to include another baseline that is not entirely based on LLMs but based on a manual creation of evaluation criteria?", "extra_scores": {}}, {"review_id": "zveMC8J0tc", "reviewer": "Reviewer_BmkZ", "summary": "LLM outputs are usually evaluated a set of criteria presented to human evaluators or other LLMs (e.g., LLM-as-judge settings). Strong answers typically rely on unstated criteria and conventions. The author(s) of this paper propose EvalAgent, a framework designed to automatically discover latent task-specific criteria. EvalAgent consists of 3 stages: a query generator, an expert retriever, and a criteria generator. They find that the criteria generated by EvalAgent are more implicit and specific, more actionable, and align better with human-written criteria.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* There is a clear, agentic structure. The three stages---query generator, expert retriever, and criteria generator---are modular and can be swapped out with different LLMs as needed. This makes it applicable for both general and specialized LLMs. \n* The evaluation section is broad and consists of a new dataset (Ask-then-Critique). Evaluation is across both automatic metrics and human evaluation. The improvements are notable, and human alignment improves without obviousness.\n* The use of an expert retriever grounds the generated criteria in real-world expertise found on the web.", "reasons_to_reject": "* The web-grounding is only as good as what is available on the web. For certain tasks or tasks in emerging domains, there may not be enough expert information online to generate a good set of criteria. It would be helpful to see when and how the EvalAgent pipeline fails.\n* The interrater agreement, measured through Fleiss' kappa, is quite low for EvalAgent (0.25 for obviousness, 0.26 for utility). This implies that there is very high disagreement with the criteria generated using EvalAgent. \n* There is no discussion about the potential amplification of biases found in online \"expert\" contexts. Relatedly, with the datasets, the Ask-then-Critique was collected using a graduate NLP class, and consists of evaluations and critiques from a highly educated and non-representative group of people completing the task.  \n* This reads as a very computationally intensive process; no details are provided about the computational costs of EvalAgent.", "questions_to_authors": "* There is a broken reference in Figure 5 (says ``Prompt ??'').", "extra_scores": {}}, {"review_id": "0rdNc2lXAC", "reviewer": "Reviewer_393x", "summary": "This paper proposes EVALAGENT, a framework for automatically generating implicit, task-specific evaluation criteria for open-ended writing prompts. Instead of relying solely on LLMs or human-defined standards, EVALAGENT mimics how humans seek advice by querying the web, retrieving expert-authored instructional content, and synthesizing actionable evaluation criteria. The framework shows improved specificity, implicitness, and actionability over baseline methods across various writing tasks.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper introduces a new method for discovering implicit evaluation criteria by grounding them in expert web content, rather than relying solely on LLMs or human annotators.\n2. The multi-step framework, from query generation to ranking, closely reflects how humans gather instructional advice.\n3. Extensive experiments across multiple datasets show consistent improvements in criteria quality, including actionability and alignment with human preferences.", "reasons_to_reject": "1. The framework depends on web content, but the paper doesn’t deeply address safeguards against unreliable or low-quality sources. It's uncertain how well the system performs in areas with limited or sparse instructional content online.\n2. It’s not specified whether models used during evaluation had real-time internet access or relied on cached data, raising reproducibility concerns.\n3. Baseline LLMs may not have had access to the same external information, which could bias comparisons in favor of the proposed method.", "questions_to_authors": "1. Does the EVALAGENT framework depend on live internet access during inference? Or are documents retrieved and cached beforehand? How does this compare to the baseline LLMs' capabilities?\n2. Were any of the baseline LLMs given access to external data (e.g., via tools, browsing)? If not, does that put them at an inherent disadvantage compared to EVALAGENT?\n3. How do you ensure that the content retrieved from the web is factual, trustworthy, and not misleading? Are there mechanisms in place for hallucination or misinformation filtering?\n4. How well does EVALAGENT generalize across domains where high-quality instructional material might be sparse (e.g., niche or emerging fields)?", "extra_scores": {}}, {"review_id": "Xoq4Mmiy60", "reviewer": "Reviewer_1Txy", "summary": "This paper presents a highly interesting and significant subtask in the context of LLM-as-a-Judge, namely the analysis of the effectiveness of evaluation criteria. The authors propose an LLM-prompting-based eval-agent framework that enhances the reliability of LLM-as-a-Judge by generating more useful evaluation criteria through a retrieval-augmented generation procedure. Furthermore, they introduce several metrics for assessing the quality of evaluation criteria, which provides valuable insights for future research. Extensive experimental results demonstrate the superior reliability of the proposed eval-agent compared to LLM-based prompting baselines.", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "​1. The research problem addressed in this paper​ represents an underexplored yet critically important direction in the field of ​LLM-as-a-Judge. The study’s motivation is well-justified and holds significant implications for advancing this research area.\n​2. This work introduces a novel RAG-based framework (evalagent)​​ for constructing ​evaluation criteria, along with ​multiple dimensions and metrics​ for assessing their quality.", "reasons_to_reject": "### **Weaknesses / Limitations:**  \n1. **Insufficient Literature Review**  \n   The paper lacks a thorough discussion of prior work in **LLM-based criteria generation**, which has been preliminarily explored in **critique and LLM-as-a-Judge research**. For instance:  \n   - **MultiCritique [1]** proposes a **hierarchical taxonomy** for generating criteria from **coarse-grained to fine-grained** levels.  \n   - **HD-Eval [2]** trains a high-quality criteria generation model using **human-annotated data**.  \n   A **comparative analysis** of these works is necessary to clarify the **novelty and contribution** of the proposed **evalagent** framework.  \n\n2. **Computational Overhead of EvalAgent**  \n   The framework involves **query generation, retrieval+summarization, and criteria generation**, which likely incurs **significant computational costs**. However, the paper lacks a **cost-efficiency comparison** between **evalagent** and baseline methods. A **runtime/memory analysis** should be included to provide a **more balanced evaluation** of the framework’s **practical applicability**.  \n\n3. **Heuristic Evaluation Metrics for Criteria Quality**  \n   The proposed **evaluation metrics** for criteria quality appear **somewhat heuristic**. The authors could **enhance reliability** by:  \n   - **Indirectly assessing criteria quality** via **critique or refinement performance** (e.g., **RealCritic [3]** evaluates critiques based on **revision quality**).  \n   - Clarifying the **pass rate (improvement) metric** (Section 6.1, Lines 318-325), which currently lacks **detailed methodology** (e.g., how improvement is quantified).  \n\n> [1] Training Language Model to Critique with Multi-agent Feedback\n> \n> [2] HD-Eval: Aligning Large Language Model Evaluators Through Hierarchical Criteria Decomposition\n>\n> [3] RealCritic: Towards Effectiveness-Driven Evaluation of Language Model Critiques\n\n### **Minor Issues (Typos & Formatting):**  \n1. **Figure 5** contains a **LaTeX compilation error** (needs correction).  \n2. **Line 195**: \"We report **S and W** across\" → should be \"We report **S and I** across\" (likely a typo).  \n3. **Figure 3 (wrapfigure)** is **misaligned**, encroaching on the next section’s text (requires formatting adjustment).", "questions_to_authors": "None", "extra_scores": {}}]

Author rebuttal:

Meta-review: The author(s) of this paper propose EvalAgent, a framework designed to automatically discover latent task-specific criteria. EvalAgent consists of 3 stages: a query generator, an expert retriever, and a criteria generator. They find that the criteria generated by EvalAgent are more implicit and specific, more actionable, and align better with human-written criteria.

There is disagreement among the reviewers about the paper being ready for publication. The rebuttal phase has been quite intense, and from it, it appears that the most important limitations raised by the reviewers have been answered by the authors in their responses; some of these could be addressed in the main body of the camera-ready, others could be discussed as limitations of the current work.
Decision: Accept