WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality
Authors: Chunyang Li
Yilun Zheng
Xinting Huang
Tianqing Fang
Jiahao Xu
Lihui Chen
Yangqiu Song
Han Hu
Venue: ICLR 2026 Oral (2026)
Keywords: large language models
evaluation
LLM-as-a-judge
benchmark
Primary area: datasets and benchmarks
TL;DR: A meta-evaluation benchmark for assessing LLM-as-a-judge in the context of web development.

Abstract: The paradigm of LLM-as-a-judge is emerging as a scalable and efficient alternative to human evaluation, demonstrating strong performance on well-defined tasks. However, its reliability in open-ended tasks with dynamic environments and complex interactions remains unexplored. To bridge the gap, we introduce WebDevJudge, a systematic benchmark for assessing LLM-as-a-judge performance in web development, with support for both non-interactive evaluation based on static observations and continuous interactive evaluation with a dynamic web environment. WebDevJudge comprises human preference labels over paired web implementations, annotated with structured and query-grounded rubrics to ensure high-quality ground truth. Using this benchmark, we comprehensively evaluate various evaluators, including LLMs, MLLMs, and agentic workflows. We systematically investigate the impact of different paradigms and guidance mechanisms. Our experiments reveal a significant gap between LLM judges and human experts. In-depth analysis indicates this gap stems from fundamental model limitations, including failures in recognizing functional equivalence, verifying task feasibility, and mitigating bias. Overall, WebDevJudge presents a significant challenge to LLM-as-a-judge, offering insights to guide future research toward developing more reliable and capable automated evaluators for complicated scenarios.

Published as a conference paper at ICLR 2026
WEBDEVJUDGE: EVALUATING (M)LLMS AS CRI- TIQUES FOR WEB DEVELOPMENT QUALITY
Chunyang L Yilun Zheng" 13; Xinting Huang', Tianqing Fang', Jiahao Xu', Lihui Chen', Yangqiu Song", Han Hu?
Tencent AT Lab, "The Hong Kong University of Science and Technology, "Nanyang Technological University cl cliei@connect. ust.hk, yilun@01@e eettooeeeentt.....m wwt.ttu.duu edu.eedussg
ABSTRACT
The paradigm of LLM-as-a-judge is emerging as a scalable and efficient alter- native to human evaluation, demonstrating strong performance tasks. However, its reliability in open-ended tasks with dynamic environments and complex interactions remains unexplored. To bridge the gap, we introduce WEB- DEVJUDGE, systematic benchmark for assessing LLM-as-a-judge performance in web development, with support for both non-interactive evaluation based on static observations and continuous interactive evaluation with a dynamic web en- vironment. WEB DEVJUDGE comprises human preference labels over paired web implementations, annotated with structured and guery-groundee rubrics to ensure high-quality ground truth. Using this benchmark, we comprehensively evaluate various evaluators, including LLMs, MLLMs, and agentic workflows. We system- atically investigate the impact of different paradigms and guidance mechanisms. Our experiments reveal significant gap between LLM judges and human experts. In-depth analysis indicates this gap stems from fundamental model limitations, in- cluding failures in recognizing functional equivalence. verifying task feasibility, and mitigating bias. Overall, BEBDEVJJDGE presents a challenge to LLM--ss-- judge, offering insights guide future research toward developing more reliable and capable automated evaluators for complicated scenarios. Code and data are available https///gihbbbcomm/yy///3//////
well-defined
INTRODUCTION
Evaluate, refine, then evaluate again and refine again-large language models (LLMs) have achieved remarkable success across various domains through this iterative cycle (Madaan et al.) 2023 Chen al. 2023 [Shafayat et al. 2025). Conventional evaluation paradigms rely heavily on human assessment (Zhong et al. 2024 Starace 2025), which, while meticulous, presents crit- ical bottleneck due its high cost and low scalability. In response, the paradigm of LLM-assa-juude has emerged as promising alternative (Zheng et al. 2023 Dubois et al. (2024), offering scalable and cost-effective solution for crucial development stages like verification (Lightman et al., 2024)
and reward modeling (Yuan et al. 2025). With the advent of sophisticated language agents capable of planning. tool use, and collaboration Yao al., 20233 Qin al.) [2024a] Liang al.) 2202)). the role of LLM--as---udge is rapidly expanding beyond basic. well-defined tasks to encompass chal- lenging real-world problems (Zhuge et al. 2025 BBin al., 2025). This progression is critical, as paves the way for more capable automated evaluators, creating the possibility for language models to self-evolve in complex, real-worke applications.
However, fundamental question regarding the reliability of LLM-as-a-judge persists. Its effec- tiveness is well-established for static and basic tasks (Chen et al.) 2024, (Saha et al.) 2025] Gou al. 2025 Lü et al., 2025). but these successes share critical commonality: they rely on static assessment of final outcomes. The reliability of LLM aud-aa-a-judge dynamic, open-ended domains
"Equal contributions. Work done by CL and Y7. during their internship Tencent.

---

Published as a conference paper at ICLR 2026
involving complex interaction remains largely unexplored. Such contexts introduce significant chal- lenges: dynamic evaluation requires continuous interaction with and comprehension of a changing environment (Li et al., 2023a Pagleri et al., (025), while their open-ended nature necessitates tablishing feasible assessment standards (Li et al. 2025). This gap between the expanding scope of automated judges and the lack of rigorous validation in complex, interactive settings highlights an urgent need for a new meta-evaluation benchmark. Such a benchmark is essential to assess, in- terpret, and ultimately enhance the reliability of LLM-based judges as we approach increasingly autonomous Al systems.
Consequently, in this work, we introduce WEBDEVJUDGE a meta-evaluation benchmark for as- sessing LLM-as-a-judge on a representative complex and interactive task using the context of web development. Web development offers an ideal testhed for complex dynamic evaluation, as it inher- ently has interaction requirements, whose assessment depends not just on static code but real-time interaction. The task also intrinsically open-ended, lacking single absolute answer. To establish a high-quality ground truth, introduce a structured annotation methodology using query-grounde rubric trees, which decomposes high-level requirements into a verifiable hierarchy of fine-graines criteria. This rigorous protocol combining rubric with human judgns yields inter-annotator agree- ment over 80%, which substantially exceeds the 63% reported for MT-Bench (Zheng et al.) 2023), confirming the reliability of the preference labels. Departing from traditional benchmarks focused on static text (Tan et al.) 2025), WEBDEVJUDDE supports both static and interactive evaluation by offering multifaceted representations of each web implementation, including its source code, screenshot of the rendered webpage, and a fully interactive environment for dynamic assessment, as shown in the right part of Figure I Evaluator performance is measured by agreement with expert human preferences on paired web implementations methodology widely adopted for nuanced, open-ended tasks where the absolute answer deficient (Bai et al., 2022; Lambert al., (025).
Using DEVJDGG, we conduct comprehensive evaluation of wide array of judges, includ- ing L.LMs, MIL.Ms, and agentic workflows, under various paradigms and guidance mechanisms. Our experiments reveal that significant capabilities gap persists between the advanced models and human experts, with a performance discrepancy of about 15%. Notably, different guidance strategies provide only marginal improvements in the pairwise comparison setting, suggesting that preference prediction through comparative assessment represents an internalized capability mod- els (Hua al. 2025 YYu et al. (025). Furthermore, while agentic workflows appear well-suited for interactive task evaluation, they fail to outperform vanilla models due to error accumulation across planning and execution stages (Pan al., 2025). Through detailed error analysis and case studies, we systematically investigate the failure modes of automated evaluators. As part of this analysis, we construct WebDev. rreDDerJudge-Un,, diagnostic dataset specifically designed evaluate feasibility verification capabilities of different types of LLM-as-a-judge. Our investigation reveals two fun- damental performance bottlenecks: (1) a persistent inability to recognize functional equivalence between diverse implementations that achieve the same objectives through different approaches or terminologies, such as implementations using the same title element with only variations text, and (2) systematic weaknesses in feasibility verification, where static assessment suffers from low preci- sion due to static code analysis limitations while interactive agents exhibit low recall stemming from their own operational constraints. These compounding errors ultimately undermine evaluator per- formance, pointing toward fundamental research directions for developing truly reliable automated evaluators complex, interactive domains. Our main contributions are as follows:
We construct WEBDEV JUDGE, a meta-evaluation benchmark that supports both static and interactive assessment of web development quality with high-quality preference labels.
Comprehensive empirical evaluation of (M)LLMs and agentic workflow reveals that cur- rent LLM as-judee still falls short of human-level reliability in preference prediction.
Detailed eтror analysis identifies the systematic weaknesses of LLM-as-a- asajue-- providing critical insights for developing more reliable automated evaluators.
RELATED WORK
The paradigm of LLM-as-a- Judggeee leverages powerful large language models to simulate human-like assessment, enabling scalable and cost-efficient evaluation (Gu et al.) 2025).
LLM-as-a-Judge
This approach has been widely adopted across various domains, including question answering (Bai
2

---

Published as a conference paper at ICLR 2026
Data Collection
Annotate with Rubric
Judge
User query 8
Verifiable rubric
Evalustor T
P Observation
Build book restew form with rating, author, and text fields; display each review with user avatar page.
Querygrounded
HDG1
(MALES
Query-based filter
Source Code
Rendenel Web
Aggentic
Interaction
Implementation
Code A:
Paradigm P
Code:
Webu, Webe
Web.
Webb
Annotabt
Preference label L A
Web Bi
WEBDEVJUDGE
HTHLE
Jodge
(Query Q Web,, Web, Preference lobel L,)
Preference
Paimiis
Single
Figure 1: Overview of WEBDEVJUDGE, Left: Data Collection with query-based and environment- based filtering. Center: Preference label annotation with verifiable rubric tree. Right: Evaluate (M)LLM-based and agentic evaluators under pairwise and single-answer paradigms.
al. 2023), 2022),
data filtering (Li et 2024 Xu et. ). and trajectory evaluation (Pan et al. 2024
Xue al. 2025). Typical implementations employ strong LLM as evaluator, which compares or scores candidate responses against predefined criteria. Common comparison methods include pair- wise comparison and single-answer grading Zheng et al. (2023). As application scenarios broaden, the conventional LLM-as---Judges approach is evolving into Agent-as-a-Judds (Zhuge et al. 2025), wherein LLM-based agents are equipped with tool-using and collaboration capabilities. (Gou et al. 2025) propose judge agent with extractor and verifier to evaluate deep research tasks with rubric trees, while Zhuge et al. (2025) design modular agentic framework to evaluate agentic systems. Despite its efficiency, this method faces several challenges. Position bias (Wang et al. 2024) and verbosity (Ye al.) (025)) preference may skew results, and the creation of detailed, human-writter rubrics is both labor-intensive and difficult to scale (Starace et al., 2025). Additionally, typical as- sessments may overemphasize final outcomes, limiting their applicability in open-ended interactive tasks (Zhang et al.] 2025a). To evaluate the judge capability of LLMs, we introduce benchmark on open-ended web development scenarios. is designed to assess a wide range of LLM-based evaluators, with the dual purpose of benchmarking their performance and exposing fundamental shortcomings in existing automated evaluation approaches.
Mcta-evaluation accesses the effectiveness of automated evaluation by measur- ing its correlation with human or ground truth. Existing benchmarks primarily focus two aspects: (1) alignment with preference label, often using pairwise comparisons (Zheng et al. 2023 Li et al, 2023b), and (2) accuracy in identifying correct task outcomes, particularly in reasoning (Luo al.
Meta Evaluation
2023 He et al. 2025) and agentic tasks (Lu et al.) 2025). Prior works such as MT-bench (Zheng al. 2023) and LLMEval (Zhang al. 2023) measure how LLM judges mirror human preferences in multi-turn conversational and instruction following tasks. However, these benchmarks may be constrained by human subjectivity and position bias (Wang al.) 2024). Alternatives like LLM- Bar (Zeng et al., 2024) and JudgeBench (Tan et al. 2025) test judges on re-annotated instruction adherence or veriñialle reasoning discrimination. These benchmarks mainly focus on text-based tasks without complex environments. In the context of interaction, AgentRewardBench (Lù et al.) 2025) uses expert-annotatee result of pre-scripted trajectories to benchmark how well LLM evalua- tors score agent performance. ArtifactsBench (Zhang et al. 2025a) evaluates dynamic visual effects. However, lacks a critical component of assessing environmental changes that are driven by real- time input. Our work, WEBDEYJUDGE introduces mneaa-vvaluation benchmark that assesses judges on dynamic, real-world web development tasks, emphasizing continuous interaction with live web environments, offering insights in complex interactive settings.
3

---

Published as a conference paper at ICLR 2026
3 WEBDEVJUDGE BENCHMARK
WEBDEVJUDGE serves as a meta-evaluation benchmark designed to assess whether LLM-as-a- judge can effectively approximate human preference judgments in web development tasks. We frame the task as preference evaluation problem. Each instance is represented as quadruple (Q, W. Wb.4p) where Q denotes web development query (e.g., "build book review page"), Wa and Wi, represent web implementations from two distinct models a and b, and lp is the annotated preference label between the two outputs. The objective to predict the preference label-i.e. whether a wins, b wins, or the result is tie-which is widely used for open-ended tasks ZZng ctt al., 2024, Lambert et al. 2025). The observations of Wa and W may take various forms across experimental settings, such code, screenshots, or interaction trajectories.
3.1 DATA FILTERING
We collect data from the webdev- weweww-rena-preee arren--rreeeeece-11111 dataset (Vichare et al. 2025 Chiang et al., [2024), which comprises 10,501 user queries with paired code outputs from models and user-provided preferences. We apply two-stage filtering process to enhance data quality:
Query-based filtering We first exclude extremely short queries and verbatim duplicates. The remaining queries are further filtered via LLM according to the following criteria: (1) Safery: exclusion of harmful or offensive content; (2) Clarity: the query must be unambiguous and inter- pretable: (3) Feasibility: the query should be realistically implementable as web application, based on its purpose and required level of interaction.
Environment-base bassed filtering We deploy each web implementation in a unified execution environ- ment. Instances that fail deploy correctly require niche dependencies are discarded. To ensure validity, we capture initial screenshots of each rendered webpage and use a multi-modal LLM to filter out invalid cases (e.g., blank pages or intrinsic errors).
After applying these filters, we retain 1,713 high-quality instances. We sample 700 instances for further annotation. Details regarding the data construction process provided in Appendix B
Annotation agreement rates with and without the verifiable rubric. The 'without rubric' part shows agreements between: (1) annotators and (2) annotators and the original labels. The "with rubric' part shows inter-annotator agreements under sumtnn-rittenn and LLM J.M--eenrated rubrics.
Table I:
Without rubric
With rubric
Setting
inter-annotator
annotator-origin
Human-written
LLM-generuted
w/ tie
65.0
53.0
92.0
90.0
w/o tie
91.3
77.4
95.5
95.1
3.2 ANNOTATION VIA RUBRIC TREE
Consistent with prior work (Zheng et al. 2023 Tan et al. 2025). we note that raw preference labels may reflect subjective bias rather than objective quality. To quantify this issue, we sample 100 in- stances and have them re-annotated by two expert annotators based on fully deployed web instances. As shown in [ both inter-annotator agreement and agreement with original labels are initially low, highlighting the need for a more structured and standard annotation protocol.
To address this, we introduce rubric tree-a structured evaluation framework commonly used in complex assessment scenarios (Starace et al. 2025 Gou et al., 2025). The tree is query-based and scalable by design, organized along three core dimensions: intention, static quality, and dynamic behavior (see Figure I. Each leaf node corresponds to binary test, whose outcomes aggregated hierarchically to parent nodes. This allows for both a holistic score at the root and fine-grained diagnostic insight via leaf-level pass rates.
We validate the effectiveness of the rubric tree by manually constructing trees for 50 instances and asking two annotators to evaluate these using the rubric reference. As shown in Table 1 rubric-
4

---

Published as a conference paper at ICLR 2026
guided annotation significantly improves agreement rates. To scale this process, employ few-shot LLM generation to automatically produce rubric trees. A third annotator labels the same instances using these generated rubrics, achieving high agreement with human -imtnnwritten the utility of L.L -.M-generated rubrics. We then use generated rubric trees to annotate the remain- ing data via two expert annotators with software engineering backgrounds. During annotation, perform manual inspections to exclude incompatible and harmful cases. Final inter-annotator agrec- ment sampled subset reaches 89.7% 59.7% demonstrating high consistency.
Table 2: Categories and their respective sub- categories of queries in DEDDEEJUDGE.
Website Design
17.6
UT Design
16.4
Development
Category
Sub-categories
Total %
Clone Developiment
14.8
Wehsite Design
App Development
11.6
DIGITAL DESIGN
34.0
Design
Development
Design
6.6
GAME AND APP DEVELOPMENT
Development App Development App Design
Multilingual Querles
2.9
34,4
Applications
Simulations
Clone Development
Digital Tools
1.1
Web Development
Creative Humor
WEB AND SPECIALIZED TECHNOLOGIES
Simulations
Al Applications
31.6
10
20
Multilingual Queries
Pervenlage (%)
Digital Tools
Creative Humor
Figure 2: The distribution of sub-category across WEBDEYJUDGE.
3.3
DATA STATISTICS
The final benchmark consists of 654 instances, with preference distributions follows: 269 for a, 276 for b, and 109 ties. To characterize the query domain coverage, we conduct a topic analysis based the original topics of the aenadv--rrea--pefereenee
dataset (Vichare et al. 2025 Fine-gres line-grained topics are grouped into three broad categories: Digital Design, Game and App Devel- opment, and Web and Specialized Technologies based their shared characteristies and application contexts of representative examples. Table F provide detailed category distributions and visualizations.
4 EXPERIMENTS
We conduct comprehensive experiments and evaluate different evaluation settings WEBDEV- JUDGE, focusing on the following research questions: (1) Whether L.MMaa--ajuudee can an al- ternative to human preference in open-ended complex tasks like web development? (2) How do different settings and strategies affect the agreement rate performance?
4.1 EVALUATOR PERFORMANCE ON WEBDEYJUDGE.
Setup Following Zheng et al. 2023 Xie et al. 2025). evaluate the evaluators under 2 distinct
paradigms:
This approach directly compares two responses, leveraging relative judgments to infer preference, an intuitive and natural for preference prediction.
Pairwise comparison:
Single answer grading: This approach employs LLMSs assign scores or labels to indi- vidual responses, with preferences derived by comparing scores or pass rates.
To enable structured and domain- -rruunded assessment, design cuutii-dimensioall Likert scale inspired by prior work (Lan et al. 2024 Zhang et al. 2025a) and incorporate principles from inter- national software testing standards (ISO/EC//BE 2022). Our criteria comprises four dimensions,
5

---

Published as a conference paper at ICLR 2026
Table Agreement Rate (%) of different evaluators under different evaluation paradigms. The best average performance of the whole dataset is highlighted in bold and the second best is underlined.
DIGITAL DESIGN
GAME & APP
WEB & SPECIAL
AVERAGE
Model/Method
Single
Pair
Single
Pair
Single
Pair
Single
Pair
Vanilla
Non-reasoning Models
GPT-4.1
57.66
70.72
64.89
73.33
59.90
66.67
60.86
70.34
GPT-4o
53.15
68.02
59,56
70.67
60.39
64.25
57.65
67.74
2
Qwen-2.5-VL-72B Inst.
46.85
68.02
51.11
66.22
52.17
64.73
50.00
66.36
Gemini-2.5-flash-litt
50.45
63.51
51.56
60.89
48.31
63.77
50.15
62.69
DeepSeek-V3-0324
60.36
67.12
59.11
66.67
60.39
67.63
59.94
67.13
Kimi-K2-Ing
62.61
68.02
62.22
67.56
54.59
66.18
59.94
67.28
2amn3-233B-A2233
55.86
66.22
63.11
68.89
60.87
62.80
59.94
66.06
58.56
68.02
63.56
65.33
57.00
65.70
59.79
66.36
Qwen2.5-32B5
5--25-33BBBmm Inst.
50.45
63.51
54.22
67.56
52.17
63.29
52.29
64.83
Gemma-3-27B-it
45.95
61.71
54.22
57.78
46.38
61.35
48.93
60.24
Qwen2.5- Innt.t
48.65
58.11
46.22
54.67
39.61
51.21
44.95
54.74
Reasoning Models
Claude-4-Soneet
54.05
72.52
63.11
72.00
60.39
65.70
59.17
70.18
Claudo-3.7-Soman
67.57
70.72
64.44
66.22
59.42
70.05
63.91
68.96
Gemini-2.55pru
$1.80
64.41
53.33
72.00
45.41
60.39
50.31
65.75
GLM-4.5
58.56
71.17
56.89
70.67
51.69
63.77
55.81
68.65
DeepSeck-RI R--0528
$1,80
69.37
61.33
68
50.24
62.32
$4.59
66.97
Qwen3-30B- 3-3--BB---3------ 4.3n.Think.
53.15
64,86
55.56
64.00
53.14
62.80
53.98
63.91
Agentic
Intention rubrics
38.74
32.00
48.31
39.45
Static rubrics
52.25
48.00
53.14
51.07
Dynamic rubrics
54.05
59.11
53.62
55.66
Combined rubrics
59.91
58.22
63.77
60.55
Human
Pairwise comparison
84.23
83.11
86.47
84,56
Functionality, UI Quality, Code Quality, and Interactivity, each with several sub-criteria rated on 5 5-point Likert scale (1: lowest, 5: highest). Detailed criteria are provided in Appendix C
Implementation We evaluate types of evaluators: (1) Vanilla (M)LLMs: test mod- els from OpenAl (OpenAl 2025 2024), Anthropic (Anthropic a). Google (Team] 2025a), Qwen Team 2025e dh. DeepSeek (DeepSeek-Al) [2024] 2025). Moonshot (Team (025c). and ZAI (Team 2025b). For single answer grading, we provide the query, code, and evaluation cri- teria; we further include initial sereenshot the rendered webpage for MLLMs. The model is prompted to output scores for each sub-criterion. For pairwise comparison, we supply the query, code of W, and Wr, evaluation criteria, and screenshots of both rendered webpages (if applicable). The evaluator scores both W, and W,, per sub-criterion, with explicit instructions to ignore position bias and remain objective. Preference is determined by comparing total scores. (2) Agentic Work- flow: similar to Bian et al. (2025), model evaluation multi-stage pipeline with planner, exccutor, and a summarizer:
Query Planner Plan with test cases
Executor
Results
Summarizerr Judge
The planner generates verifiable evaluation plan conditioned on the query. The executor runs test cases, and the summarizer synthesizes results into a judgment. We use the generated rubric tree the evaluation plan and reference for summarization. Preferences summarized outcomes. In implementation, we utilize UI-TARS- 1.5 (Seed) (025), one the state- of-the-art GUI agents, as the executor, ensure the reliability of the evaluation process.
inferred by comparing

---

Published as a conference paper at ICLR 2026
Result
The main experimental results summarized in Table Our analysis yields several key observations on the performance of LLM-basee evaluators in web development tasks.
LLM-as-a-judge falls short of human-level reliability complex evaluation tasks. The pri- mary finding is that no current model achieves a sufficient level of agreement with expert judgments. The nop-performing evaluator, GPT-4. under the pairwise paradigm, attains agreement rate of 70.34%. This gap underscores the inherent complexity of web development evaluation, which de- mands holistic assessment of functionality and aesthetics, especially for the "tie" cases. We also observe a clear performance ceiling: while smaller models exhibit scaling effects, larger and more capable models show diminishing returns, consistently plateauing around the 70% agreement rate.
Pairwise comparison is far more effective paradigm for preference evaluation. Across the board, the pairwise paradigm yields an average improvement of over 8.0% agreement rate com- pared to single-answer grading. Relative judgment helps models focus on discriminative features be- tween two candidates, reducing the need for absolute quality calibration. grading not only performs worse but also leads to inconsistent model rankings mome some models devi- ate from their expected benchmark ordering (Zhang et al. 2025b), indicating that this paradigm less suitable for open-ended, multi-faceted tasks. This may be due to the cognitive load of applying multi-point Likert scales consistently (Ouwehand et al., 2021). which demands calibrated internal standard of quality that is difficult for both humans and LLMs to maintain. Simplified evaluation mechanisms, such as binary checklists, may be more effective; we explore this further in Section 4.2
contrast, single-answer
Agentic workflow suffers from compounding errors. We evaluated the agentic workflow using rubrics from different aspects, as well as a comprehensive score integrating all three. The results show that it performs significantly better the Dynamic aspect, which involves strong interactive properties, compared to others with weaker interactivity. Counter-intuitively, the agentic workflow fails outperform the vanilla models. This appears result from error accumulation across its multi-stage process. We identify two primary failure modes:
The Planner struggles with the ambiguity of user queries that are often not expressed with expert-level precision. This leads to the generation of evaluation plans that are either too generic to be discriminative overly specific, causing failures due to minor implementation variations.
Brittle Planning:
Faulty Execution:
The Executor agent's ability to navigate the web and verify task states remains unreliable. It may misinterpret outcomes, injecting noise into the evaluation pro- cess. This unreliability is investigated more deeply Section 4.3
Errors compound across the plamee--xecttor-uumnniirrr pipeline, reducing the reliability of the final judgment compared end-to-end evaluation by single model (Chen et al. 2025b).
4.2 DIFFERENT EVALUATION GUIDES AND OBSERVATION FORMS
To further diagnose factors affecting evaluator performance, we conduct controlled experiments focusing on two key variables: the structure of evaluation guidance and the form of observation.
Agreement Rate (%)
65.4
69.1
67,788.2
45.8
71.2
67.367.887.7
00.208.260.7
07.367.0
03.0
80
S14
50
40
DeepSeek-V3
DeepSeck-R1
Kimi-K2
Likert
Single Rubric
Single Likert
Figure 3: Agreement rates of LLM evaluators under various guidance protocols. We compare pair- wise and single-answer paradigms using direct judgment, Likert scales, and structured rubrics.

---

Published as a conference paper at ICLR 2026
Evaluation Guidance We investigate how different evaluation protocols influence judgment curacy, comparing three widely-used forms of guidance (Zheng et al. 2023 Gou et al.) 2025): (1) Direct: The evaluator provides preference without explicit criteria. (2) Likert Scale: As used in Section 4.1] the evaluator scores outputs along predefined dimensions using a multi-point scale. (3) Rubric: The evaluator assesses outputs using the rubric trees from our annotation process, with final scores computed as a weighted aggregation of binary leaf-node pass rates across three core dimensions (intention, static, and dynamic).
Results in Figure reveal a noteworthy finding: under the pairwise evaluation, the Direct setting achieves agreement rates comparable to guidance-basee methods. This phenomenon aligns with observations in instruction following tasks (Zeng et al.) (024). where models perform similarly on agreement with or without metrie-based guidance. This result suggests that evaluation capability is an internalized skill in modern LLMs,, and external guidance provides only marginal benefits in relative assessment settings (Qin al., (024b). Furthermore, we find that reasoning models exhibit better performance under the Direct condition, indicating that imposing rigid structured metrics might constrain the models' inherent reasoning processes, thereby limiting their evaluation potential. Furthermore, validate the hypothesis from earlier: in single answer grading, the binary Rubric approach substantially outperforms the multi multi-point Likert scale, reinforcing that verifiable evaluation protocols yield more reliable judgments when there is lack of relative information.
Table 4: Impact of observation forms on the performance of multimodal evaluators. The numbers in parentheses indicate the performance change relative to the setting with both code and image inputs.
Single Answer Grading
Pairwise Comparison
Model
Image Only
Code Only
Both
Image Only
Code Only
Both
Claude-4-soneet
55.66 (L 3.51)
61.77 (f 2.60)
59.17
59.48 (L 10.7)
67.58 (L 2.60)
70.18
GPT-4.1
54.59 (1 6.27)
62.69 (f 1.83)
60.86
60.55 (1 9.79)
69.57 (L 0.77)
70.34
GPT-4o
53.36 4.29)
56.27 (1 1.38)
57.65
59.79 7.95)
67.43 (1 0.31)
67.74
Semini-2.5-pp pro
53.36 (1 3.05)
50.15 (1 0.16)
50.31
62.08 3,67)
64.53 (1 1.22)
65.75
QWL--2.5-VL-72B
51.53 (f 1.53)
50.92 (1 0.92)
50.00
59.63 (4 6.73)
64.68 (L 1.68)
66.36
Influence of Observation Modality We next analyze how the input modality affects multimodal evaluators, comparing three observation forms: (1) Image Only, providing only the initial screenshot of the rendered web page; (2) Code Only, providing only the source code; and (3) Both, providing both code and screenshot. As shown in Table 4 code emerges as the most critical modality for evaluating web development tasks. Withholding code leads to a significantly larger performance drop than withholding screenshots, suggesting that while MILMS can process visual input, their judgments are fundamentally anchored in the structured source code (Wang et al. [2025). In pairwise comparison, using both modalitics yields the best results, indicating that visual context provides complementary signals that aid in refining relative judgments (Chen et al. 2025a).
4.3 ERROR ANALYSIS
To systematically diagnose the failure modes of LLM-based evaluators, we conduct a fine fine-grained analysis along three key dimensions: inherent biases, limitations in understanding functional equiv- alence, and shortcomings in feasibility analysis.
Table 5: Positional bias in pairwise comparison. Preference for specific position, consistency and the absolute difference agreement rate (A AR) between original and swapped orders are reported.
Consistency
First
Second
A AR
Model
Direct
Likert
Direct
Likert
Direct
Liker!
Direct
Likert
Claude-4-sonnet
89.6
87.9
5.7
3,7
4.7
8,4
0.61
1.53
GPT-4.1 DeepSeck-V3-0324
83.3
85.2
0.9
3.0
15.8
11.8
0.30
1.07
84.1
83.5
11.6
7.3
4.3
9.2
1.53
1.23
8

---

Published as a conference paper at ICLR 2026
Instruction:
Consider requirement the solution's feature equivalent. example, the required heading element is present on the page, though the exact text or symbol differs.
Requirement:
Element in webpage:
grading row the "Demonst category
Presentation
0/5
A text label displaying "Demonstration.
set of icons rting "Demenstration".
LLM: false
Agent: false
Human: true
An example illustrating the failure recognize functional equivalence. The webpage element for "Presentation" serves the same purpose as the required "Demonstration" rating. Detailed examples be found in Appendix F2
Figure 4:
Inherent Biases in Judgment Prior studies (Ye et al., 2025) have revealed that LLM-as-a-judge exhibits various forms of bias, such as positional and verbosity biases. We focus specifically po- sitional bias. Despite explicit instructions to ignore order and remain objective. models still exhibit systematic preference for responses in specific position, shown in Table One might hypoth- esize that this bias emerges primarily in ambiguous cases where the two options are of comparable quality. However, our analysis of the label distribution for instances with inconsistent predictions reveals that the proportion ties not higher than that of wins or losses. This finding suggests that positional bias not merely an artifact of ambiguity but rather an inherent deficiency in the models, and instruction alone is insufficient to eliminate these deeply embedded inductive biases. Rather than employing debiasing techniques such as swapping, we choose the prompting method to reflect the models' authentic single-pan pass evaluation capability. Any inherent bias (such as position bias) considered an intrinsic flaw of the model's judgment ability. further comparison with the debiasing technique is provided in Appendix E.I
Limitations in Understanding Funetional Equivalence A critical requirement for reliable eval- uation is the ability to recognize functional equivalence- ii.., determining whether different im- plementations satisfy the same underlying requirement. We find that evaluators often fail in such judgments, adhering strictly to literal interpretation rather than intent. For example, as illustrated in Figure 4 when instructed explicitly, human evaluators correctly recognize an element implemented with alternative text, whereas both LLM and agentic evaluators incorrectly reject it. This reflects fundamental gap in contextual and pragmatic reasoning, limiting their applicability in open-ended domains where diversity in implementation is common.
Shortcomings in Feasibility Analysis We further investigate the evaluators' ability to accurately verify task fulfillment. Existing benchmarks such as AgentRewardBench (Lu jet al.) 2025) are too general assess web- specific judgment. To address this, we con- struct WebDey
Table 6: Performance on the WebDew for feasibility verification. We report precision (P), recall (R), Fl-score, and accuracy (Acc).
Model
Observe
P
R
Fl
Ace
a targeted dataset of 502 test cases. Each instance consists of web code, a specific verification task, an expected result, and label indicating whether the task is feasible. We evaluate both LLM-based (code- only) and agent-basee (interaction-driven) evaluators on this dataset. From the results in Table we identify complementary weaknesses: (1) For LLM evaluators, these models achieve high recall but suffer from low precision. While they can often identify relevant code segments, they are unable to verify actual execution outcomes, leading to false positives when the code appears relevant but does not correctly implement the desired functionality. (2) For agentic evaluators, they exhibit higher precision but lower recall. They are effective when they successfully execute test plan, but some- times fail to complete tasks due to limitations in navigation or state interpretation. Consequently, they incorrectly label feasible tasks infeasible due to their operational failures rather than actual shortcomings in the web implementation. This divergence highlights a fundamental trade- off: static code analysis lacks execution grounding, while interactive agents are constrained by their
UI-TARS-1.5
Traj.
82.4
70.3
75.8
75.1
GPT-4.1
Code
72.1
90.0
80.1
75.1
DeepSeck-V3 V3
Code
69.6
93.5
79.8
73.7

---

Published as a conference paper at ICLR 2026
own operational reliability. An ideal evaluator would combine the comprehensive coverage of code- aware reasoning with the grounded verification of interactive testing. To further investigate this, design a gated model ensemble strategy to leverage these strengths in the single-comparison for rubric. Specifically, we adopted the results of the L.MSs for the intention and static tasks. For the dynamic tasks, implemented the following logical expression:
ReSdymamic = Agent V LLM
(2)
This formulation is motivated by the performance trade-off between the constituent components: the Agent provides high-precision filtering for dynamic elements, whereas LLMs ensure higher recall. This hybrid approach optimizes the overall detection capability by balancing exactness with coverage. We selected GPT-4.1, DeepSeek- V3-0324, and UI-TARS-1, 1.5 for this ensemble. The results shown in Table These results demonstrate that combining static LLM with agent-based dynamic verification can enhance the overall evaluation agreement.
Our analysis reveals that the limitation of L.M- as-a-judge lies in fundamental deficiencies in calibration capability. Lacking this calibra- tion, LLMs struggle to map abstract quality dimensions onto discrete scores and verifiable rubrics. Improving judge performance will re- quire addressing these competency gaps rather than merely refining evaluation protocols.
Table Performance comparison between single LLM and the agent-integrated ensemble.
Model
Agreement
gpt-4.1
65.0
+UI-TARS-1.5
66.2
DeepSeck-V3-0324
66.3
+UI-TARS-1.5
67.6
CONCLUSION
In this work, we introduce WEBDEVJUDGE, comprehensive benchmark for evaluating L.M--as--juddee in web development. Unlike previous benchmarks, WEBDEVJUDGE supports both static code analysis and interactive agent navigation with high-quality preference labels. Our exper- iments demonstrate that current LLM--a--ajjudee approaches cannot effectively substitute human evaluation, and we identify on evaluation and analysis within the general LLLM-as-a-judge domain, we did not specifically op- timize the overall framework structure. We leave the exploration of sophisticated agentic workflows and complex multi-round evaluations to future work.
bottlenecks hindering their performance. Since primary focus
ACKNOWLEDGMENTS
Some authors of this paper were supported by the ITSP Platform Research Project (TTS/189/23FP) from ITC of Hong Kong, SAR, China, and the AoE (AoF/FE-601/24-N), and the GRF (16205322) from RGC of Hong Kong, SAR, China. We also thank the support from Tencent AI Lab.
ETHICS STATEMENT
The ethical considerations of our work are discussed in the context of the following aspects: (1) Data collection and use. We use publicly available datasets and self- eeffegenertted data for evaluation. We ensure that the data is only used for academic research purposes and no personal data is involved. We strictly follow the license terms and conditions. (2) LLMs API. We comply with the terms of service of the LLMs API providers strictly, maintaining fair use.
REPRODUCIBILITY STATEMENT
We provide detailed descriptions our methods, datasets, and evaluation metrics in the main text and appendix to ensure transparency and reproducibility. Our benchmark construction process for
WEBDEVJUDGE is detailed in Section 3 and Appendix B The experimental setup, including the
models, evaluation paradigms, and specific protocols, is described in Section 4 and Appendix C.
10

---

Published as a conference paper at ICLR 2026
We also provide complete description of the WebDevJudge Unit dataset in Appendix D Code and scripts are provided in the supplementary materials to replicate the empirical results.
REFERENCES
Anthropic. Claude 3.7 sonnet and claude code, 2025a. URL http:tss//ww.aaatrppccc.coo///// -laude-3-7-somen
Anthropic. Introducing claude 4, 2025b. URL httpss//ww..attrrrpp..cc/o////ss
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Ols- son, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran- Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mer- cado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Con- erly. Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan. Constitutional ai: Harmlessness from ai feedback, 2022. https:/s::///rr org. //s///b///// 2212. 0122120080011 08073
or
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou. Benchmarking foundation models with language-moder madel----mdll----aaea an-examiner. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarky Track, 2023. URL https://openreeeee wtw.pppenreeieeeeeeerrriiilllaie net/forum?id
IiRHQ7gvng
Yutong Bian, Xianhao Lin, Yupeng Xie, Tianyang Liu, Mingchen Zhuge, Siyuan Lu, Haoming Tang, Jinlin Wang, Jiayi Zhang, Jiaqi Chen, Xiangru Tang, Yongxin Ni, Sirui Hong, and Chenglin Wu. You don't 't know until you click:automated gui testing for production-ready software evaluation, 2025. httt/httss:/aaatt http///aww..rgggb/////// orggabss/5eee 14104
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun. Milm-as-a-judge: assessing multimodal Ilm-as-a-judge with vision-language benchmark. In Proceedings of the 41st International Conference on Machine Learning. ICML' (CM JMLR. 2024.
Feng Chen, Chenhui Gou, Jing Liu, Yang Yang, Zhaoyang Li, Jiyuan Zhang, Zhenbang Sun, Bohan Zhuang, and Qi Wu. Evaluating and advancing multimodal large language models in perception ability lens, 2025a. URL /ttpsttss://aaxiv.coo htwp://v..rg/////////// 14725
Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan Arik, Tomas Pfister, and Somesh Jha. Adaptation with seff-evaluation to improve selective prediction in LLMs. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics; EMNLP 2023, pp. 5 5190-5213, Singapore, December 2023. Association for Computational Lin- guistics. doi: 10. 10.18800 116518811/022333. Findings-emn__ 345/
http:///222.innnnnnnmmeeee findings-emmll..35... URL http::tps:/// httttts:://aaaattololoooooo .rgg wrg.//23...222
Meilin Chen, Jian Tian, Liang Ma. Di Xie, Weijie Chen, and Jiang Zhu. Unbiased evaluation of large language models from a causal perspective. In Forty-second International Conference on Machine Learning. 2025b. URL /tt.https:/pppereviee.rr http:///wwweeeeeeeeee//oomttttta
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos. Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica.
Chatbot arena: An open platform for evaluating lims by human preference, 2024.
DeepSeek-AL. Deepseek-v3 technical report, 2024. URL htt./ittts//arrii..r/////////2//
DeepSeek-AI. Deepseek-rl: Incentivizing reasoning capability in llms via reinforcement learning,
2025. URL htttht//s///xxii htttps///ariiv.org//b////000 12948
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsumori B Hashimoto. Longth-controlled al- pacaeval: A simple way debias automatic evaluators. arXiv preprint MxXi::244.004455 2024.
11

---

Published as a conference paper at ICLR 2026
Boyu Gou, Zanming Huang. Yuting Ning, Yu Gu, Michael Lin, Weijian Qi. Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, Chan Hee Song, Jiaman Wu, Shijie Chen, Hanane Nour Moussa, Tianshu Zhang. Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, Kai Zhang, Boyuan Zheng, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, and Yu Su. Mind2web 2: Evaluating agentic search with agent-as-a-judge, 2025. 21506
htttps://arxiv.or//aaata
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lioeel Ni, and Jian Guo. A survey on llm-as-a-judge, 2025. URL htttps://ariiv.. http://aaxxiv.../aaaaaaa org. abrgg/bbb/1111 15594
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Z.y. Peng, Zhaox- iang Zhang. Zhicheng Zheng, Wenbo Su, and Bo Zheng. Can large language models detect errors in long chain-of-thought reasoning? In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Anmual Meeting of the Associa- Vienna, Aus-
tion for Computational Linguistics (Volume Long Pupers), pp. 18468-18433 tria, July 2025. Association for Computational Linguistics. ISBN 979-8-891 111-8889177 19176-551-0 1tt333/11202.cclllcm
doi:
wwt.ooggg..........o 2025. acl-aallloooo long. 905/
Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, and Yao Qin. Flaw artifact? rethinking prompt sensitivity in evaluating llms, 2025. URL. https:tttxxsi.....//t/ wwtpaaiw.iii..ra/////////////// 01790
ISO/EE//EEE. Iso/iec/ieee international standard software and systems engineering -sofwware testing -part l:general concepts. ISOЛEСЛЕЕЕ 291 29119-1:2022(E), pp. 1-60, 2022. doi: 10.1 JEEESTD..2222.9998 145.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng. Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi. RewardBench: Evaluating reward models for language modeling. In Luis Chiruzzo, Alan Rit- ter, and Lu Wang (eds.). Findings of the Association for Computational Linguistics: NAACL 2025, pp. 1755-1797, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics. ISBN 979-8-891 891889971111 195-7. doi: 10. (0.18653/v1/2025 findings-naacll naacl.96. URL https: naacl.
//aclanthology... org/ 2025. f
org/
Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-Ling Mao. Crit- iceval: Evaluating large-scale language model critic. In The Thirth-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https:://penrev ZsxZ65YqL1
net
Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu, and Juanzi Li. MAVEN- FACT: A large-scale event factuality detection dataset. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp. 11140-11 158, Miami, Florida, USA, November 2024. Association for Computational Lin- URL https::/aclanttholooe org/2024.
guistics. doi: ww.118655v112222 wwt////...niinssee/eeeee indings-emmll.666
findings-emnn emmlp. 6517
Jiatong Li, Rui Li, and Qi Liu. Beyond static datasets: A deep interaction approach to llm evaluation, 2023a. URL https: //arxvv.//xiiv..// org/ abs/2309. 04369
Ruizhe Li, Chiwei Zhu, Benfeng Xu, Xiaorui Wang, and Zhendong Mao. Automated creativity
evaluation for large language models: A reference- based approach, 2025. htt.sttts:///axii.
org/abs/2500 15784
Xuechen Li, Tianyi Zhang. Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction followoii models. httpps:// thub. com/ latm//ttsu-llb/lpaaee eval] 5 2023b.
12

---

Published as a conference paper at ICLR 2026
Tian Liang, Zhiwei He. Wenxiang Jiao, Xing Wang. Yan Wang, Rui Wang. Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi- agent debate. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pp. 17889-17904 Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10. 00242024. mammmppmmai....0
18653//1/2222.cmmppmmiizz 992. URL
Hunter Lightman. Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. Let's verify step by step. In The Twelfth International Conference on Learning Representations, 2024. URL htthtttss///enreeete Saapmmiddd@@@@@NEEEE
net/
Liangchen Luo, Zi Lin, Yinxiao Liu, Lei Shu, Yun Zhu, Jingbo Shang, and Lei Meng. Critique ability of large language models, 2023. /ttp:://:///axx wtgg/br//////eee 04815
Xing Han Lå, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejan- dra Zambrano, Karolina Starczak,, Peter Shaw, Christopher J. Pal, and Siva Reddy. Agentre- wardbench: Evaluating automatic evaluations of web agent trajectories, 2025. URL https:
//arxiv.org/abs 2894.08942
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang. Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-refine: Iterative refinement with self-feedback. In Thirty-seventhh Conference on Neural Information Pro- cessing Systems, 2023, URL httpp////peeeeeeeewwwm////////ss
OpenAl. Hello got- 2024. URL httt::///opaaa...cmmmm/e///////s
OpenAI. Introducing gpt-4.1 1 in the api, 2025. URL http:////pppeaaacccoonnee//////
Kim Ouwehand, Avalon van der Kroef, Jacqueline Wong, and Fred Paas. Measuring cognitive load: Are there more valid alternatives to likert rating scales? Frontiers in Education, Volume 6 2021, 2021. ISSN 2504-284X... doi: /t3/////educ..2222.000000 702616. URL
3389/feduc. wttp////c...e.....f/f...c...... 2021.702616
Davide Paglieri, Bartlomiej Cupial, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Lukasz Kuciñski, Lerrel Pinto. Rob Fergus, Jakob Nicolaus Foerster, Jack Porder-Holder, and Tim Rocktäschel. BALROG: Benchmarking agentic LLM and VLM reasoning on games. In The Thirteenth International Conference on Learning Representations, 2025. URL http:httssssppeeeeeeeee eview. www.net/forrmmii-=fpft3rrr 669F
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr. Autonomous evaluation and refinement of digital agents. In First Conference Language Modeling, 2024.
URL / met//forumidd=PPP@FKSS.
Melissa Z Pan, Mert Cemri, Lakshya A Agrawal, Shuyi Yang. Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Kannan Ramchandran, Dan Klein, Joseph E. Gonzalez, Matei Zaharia, and Ion Stoica. Why do multiagent systems fail? In ICLR 2025 Workshop on Building Trust in Language Models and Applications, 2025. WM521FqPvI
Yujia Qin, Shihao Liang, Yining Yc, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang. Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun. TooILLM: Facilitating large language models to master 16000+ real-world APIs. In The Twelfth International Conference on Learning Representations, 2024a. URL httpthttt///toeeeeeeeee
net/forum? (ttp///orummmdddng200000003 d=dd
Zhen Qin, Junru Wu, Jiaming Shen, Tianqi Liu, and Xuanhui Wang. LAMPO: Large language mod- cls as preference machines for few-shot ordinal classification. In First Conference on Language Modeling, 2024b. URL httpt:tt/tppppreeeeetee evieew www.:oormmidd-iggo
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad. Jason E Weston, and Tianlu Wang. Learning to plan & reason for evaluation with thinking-LLan auddee. In Foroy-second International Con- ference Machine Learning, 2025. URL http:://pogerrreeeeete//roototdo
13

---

Published as a conference paper at ICLR 2026
ByteDance Seed. Ui-tars-1.5. https://seed-tars.coom 2025.
Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette. Can large reasoning models self-train?, 2025. httpps:/aarxw..rg///a////a
21444.
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan. Paperbench: Evaluating Al's ability to replicate Al research. Forty-second International Conference Machine Learning, 2025. URL http::://opemrviie.enettfooo id=xF5PuTLPbn
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica. Judgebench: A benchmark for evaluating LLM-based judges. In The Thirteenth International Conference Learning Representations, 2025, URL https:
Gemini Team. Gemini 2.5: Pushing the frontier with advanced reasoning. multimodality, long context, and next generation agentic capabilities, 2025a. URL wtthtttts////xiii httpaaawwiv.o..oaaae...e abs/2507.
06261
GLM 4.5 Team, Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025b. URL
http://awwii.ooggbbbb//aab httg/abb/5008.0///4/ 06471
Kimi Team. Kimi k2: Open agentic intelligence, 2025c. URL http:://arxivvorr//b//2////
20534
Qwen Team. Qwen2.5-vl technical report, 2025d. URL UKL/ttps://arriv.or//bb//8/
Qwen Team. Qwen3 technical report, 2025e. URL http:::/aaavvv.ogg//////////////
Aryan Vichare, Anastasios N. Angelopoulos, Wei-Lin Chiang, Kelly Tang, and Luca Manolache. Webdev arena: A live Ilm leaderboard for web app development, 2025.
Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, and Yi Li. Do code semanties help? a compre- hensive study on execution trace- based information for code large language models, 2025. URL https://arxiv. wrwoorgg/aaoo.eee attg//bb//50..... 2509. 11686
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui. Large language models are fair evalua- tors. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd An- nual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 9440-9450, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10.18653/v1/20244 aa11022..cl-oonn5511.... URL https: /tap:///hoollahhlloooooo org/ 2024 acl-long..55. 511/
Qiujie Xie, Qingqiu Li, Zhuohao Yu, Yuejie Zhang, Yue Zhang, and Linyi Yang. An empirical analysis of uncertainty in large language model evaluations. In The Thirteenth International Conference on Learning Representations, 2025. URL http:///ppeereeieeneeeeforreeere J4xLuCt2kg
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu. Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials. In The Thirteenth International Conference on Learning Representations, 2025. URL https://
openreview.neteee wttp://www..e/oooommo////eteo/oo
Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and
Yu Su. An illusion of progress? assessing the current state web agents, 2025. URL https:
//arxiv. www.wrrg/abs..... 200./2504.06 01382
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, 2023. URL lhtpp:: http:://///peeeeeeeeeoeem wwtpviiwwwweeettteeeeteeeeee
WE vluYUL-X
14

---

Published as a conference paper at ICLR 2026
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang. Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang. Pin- Yu Chen, Nitesh Chawla, and Xiangliang Zhang. Justice or prejudice? quantifying biases in LLM- LLudee-a-judge. In The Thireeetth International Conference on Learning Representations, 2025. http:https:/// /ttpps://ppenreiieww wtt.://wwww.neeooooo//////------ iajM
Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, and Xuelong Li. Improve Ilm-as- a-judge ability general ability, 2025. URL /RRL/tttss////ari.oo/////////// 11689
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhaaaaar, Jing Xu, and Jason Weston. Self-rewarding language models, 2025, URL httDs://arriiv.or///b//22111 10020
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. Evaluating large language models evaluating instruction following. In The Twelfth International Conference Learning Representations, 2024. URL / wwwpeeewwwwweeeeeeeeeeeeeeeeeeee dor
Chenchen Zhang, Yuhang Li, Can Xu, Jiaheng Liu, Ao Liu, Shihui Hu, Dengpeng Wu, Guanhua Huang. Kejiao Li, Qi Yi, Ruibin Xiong, Haotian Zhu, Yuanxing Zhang, Yuhao Jiang. Yue Zhang, Zenan Xu, Bohui Zhai, Guoxiang He, Hebin Li, Jie Zhao, Le Zhang. Lingyun Tan, Pengyu Guo, Xianshu Pang. Yang Ruan, Zhifeng Zhang. Zhonghu Wang. Ziyan Xu, Zuopu Yin, Wiggin Zhou, Chayse Zhou, and Fengzong Lian. Artifactsbench: Bridging the visual-interactive gap in Ilm code generation evaluation, 2025a. URL
.4952
Guanhua Zhang, Ricardo Dominguez-OOmedua and Moritz Hardt. Train-before-ter harmonizes language model rankings, 2025b. URL httptttt/ttsti.....ttt org/ httpg//bb//227.0111/1 05195
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li. Wider and deeper Ilm networks are fairer Ilm evaluators, 2023. URL https://
arxiv. org/ htrg/abs/2088.. 01862
Lianmin Zheng. Wci-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. Judging LLM-as--ajudde with MT-bench and chatbot arena. In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023. URL https:
//openreview.net/e// wtw.://wwww.oooor/mdddccccc....
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. Sglang: Efficient execution of structured language model programs, 2024. URL https::/arxiv.orr
abs/2312.07 070014
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan. AGIEval: A human-centric benchmark for evaluating foundation models. In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), Findings of the Association for Computational Linguistics: NAACL 2024, pp. 2299-2299121 Mexico City, Mexico, June 2024. URL
Association for Computational Linguistics. doi: 10.18653/v 1/2024.findings-naale https:///c..anttooooy..rr/////// wtt.:oorg/2224.fiiin--------e Findings-naall. 1497
Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang, Dmitrii Khizbullin, Yun- yang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber. Agent-as-a-judge: Evaluate agents with agents. In Forty-second International Conference Machine Learning, 2025. URL https: //
15

---

Published as a conference paper at ICLR 2026
A LIMITATIONS OF WEBDEVJUDGE
While WEB DEVJUDGE establishes systematic framework for benchmarking LLM preference pre- diction, we recognize several constraints that reflect the inherent complexity of modeling human judgment. The core of preference modeling lies in its inherent subjectivity. Although we provided annotators with comprehensive guidelines to standardize evaluations, personal preferences are dif- ficult to fully decouple from objective quality. Because annotation pool is limited in size, the resulting labels may reflect the collective biases of specific demographic or professional group rather than universal human consensus, especially for ambiguous cases. The current scale of WEBDEV DEBJEVJUDGE JUDGE is relatively focused. However, we have retained the broader set of filtered sam- ples, providing pathway for future data augmentation.
B DETAILS IN WEBDEYJUDGE CONSTRUCTION
B. DATA COLLECTION AND FILTERING PIPELINE
Our data collection process commences with the webdev-arena- senude--rren--preferene---ee dataset (Vichare al. 2025: Chiang et al.) 2024). which contains 10,501 user queries, each paired with two web implementations and a user-pre seco-proidee preference label. To ensure the quality and suitability this data for benchmark, we implemented rigorous two-stage filtering pipeline. The overview of the filtering pipeline can be seen in Table g
Table 8: Overview of the filtering pipeline, including the number of instances before and after filtering, and the purpose of each filtering stage.
Stage and criterion
before
# after
Purpose
basey- crrbatim-identeen duplicate
10,501
6,730
Remove redundancy while main- taining the original distribution.
Query- intention, interac- and safety
6,730
2,460
Remove harmful, offensive, nonsensical content, and queries with minimal interaction require- ments and unclear intentions.
Env-based:
deployment failure
2,460
1,814
Remove instances with deploy- ment failures.
checking via screenshot
Env-based:
deployment failure checking via status code
1,814
1,713
Remove instances with deploy- ment failures.
Sampling
1,713
700
Sample 700 instances regarding the cost-effectiveness.
Manual filtering during annota-
700
654
Filter out instances with harmful content and deployment failures.
tion
Query-based Filtering The queries were first processed address quality issues. We began by removing all verbatim duplicate queries to eliminate redundancy. Subsequently. we employed gemini-2. 5-pro to screen for and exclude any queries containing harmful, offensive, or nonsensical content, based on a predefined set of safety and clarity instructions. We then exclude queries with minimal interaction requirements (score 8, max 10) and queries with unclear intentions (score < 3, max = 5). The prompts used for filter stages are presented below.
Environment-based Filtering Following the query-based filtering, we proceeded to validate the web implementations. We established a standardized Next.js environment for deployment. To man- age dependencies, we identified and included the most common packages, thereby excluding im- plementations that required niche or incompatible libraries. Each successfully deployed implemen- tation then verified via test request; those returning non-200 status code were discarded as buggy. Finally, to handle instances with intrinsic runtime errors not captured by status codes (e.g., rendering blank page), captured initial screenshot of each webpage. A multimodal
16

---

Published as a conference paper at ICLR 2026
model, 5-VL7.5. was utilized to visually inspect these screenshots and filter out any pages exhibiting rendering failures.
This comprchensive filtering process yielded final set of 1,713 high-quality instances for sub- sequent annotation. We further sampled 700 instances from the final set for annotation. During annotation, filtered out instances with harmful content and deployment failures manually. While the original dataset includes user-provided preference labels, we identified several factors that ren- der them unreliable for rigorous evaluation. Primarily, the labels are susceptible of subjectivity and variance in evaluation criteria among individual users. The single-pass nature of the crowdsourced annotations also lacks the verification necessary for robust benchmark data. Furthermore, the dataset includes a "tie (bothbad)" category, which introduces ambiguity. The sub- jective definition of "bad" can lead to inconsistent labeling, potentially masking instances where one implementation, though imperfect, is demonstrably superior to the other. To illustrate these inconsistencies, we provide examples of problematic cases from the original dataset Table
high degree
PROMPT FOR QUERY-BASEL INTERACTION FILTERING
Please analyze the user's input to assess the level of detail regarding interaction requirements for web design. Focus on the functionality implied by the user's input and the interactions required. Based on your evaluation, assign score from 0 10:
No potential interaction requirements are mentioned, but the website may still contain interactions.
3: No interaction is mentioned explicitly, but interactions may still exist in the functionality.
5: One interaction is mentioned or implied, focusing
single functionality or task.
8: More than interaction is mentioned implied, involving multiple functionalities tasks. relevant interactions engagement with the site or app.
functionalities are clearly mentioned in detail, requiring full user
After completing your analysis, output your score in the format: <<score:int>>. For example: <<score:7>>.
Examples:
User: Ruilding merge sort function Response: This programming task that doesn't involve any web interactions or user-facing functionality. <<score:0>>
User: poker site Response: The term "poker site" implies interactions, but the input lacks specific details about those interactions. <<score:2>>
User: Create unique and incredibly beautiful design. design up to your discretion, but the main thing is to make it stand out! Write code that is at the senior level or even above it, as you 've been working in the frontend for 10 years. Make the design beautiful and pleasing to the eye! Response: The focus is aesthetics and quality, with some implied user interaction related to the design's 's appeal, but not much about functionality.. <<score:
User: site for selling craft chocolate bars Response: This request implies basic interactions such as browsing and purchasing, but lacks specific details about the interactions. <<score:5>>
User: Design guided meditation player with progress tracking. Create library view to browse and select different meditation sessions. Response: The request involves multiple interactions like browsing, selecting sessions, tracking progress, but not fully detailed. <<score: 8>>
User: Build game of chess Response: A chess game involves numerous interactions such as moving pieces, taking turns, and tracking the game state, all requiring user engagement. s<score:9>
User: Make clone of WhatsApp Chat App Response: A clone of WhatsApp implies multiple interactions. including sending/receiving messages, media sharing, and managing chats, all of which require significant user engagement and functionality, <<score:
17

---

estimated 20 minutes Published as a conference paper at ICLR 2026
PROMPT FOR QUERY-BASED SAFETY FILTERING
You given query. Classify whether it is safe unsafe. Definitions:
SAFE: The query is clear, relevant, and does not request or promote hazardous, violent, hatcful, or sexually explicit content.
UNSAFE:
The query contains harmful, offensive, nonsensical content.
Output your decision in +*valid JSON** with the following schema:
"query": "<the original user query>",
"label": "SAFE" or "UNSAFE", hazard, violence, sexual, hate, other>", "reason": "<brief explanation>"
"category":
PROMPT FOR QUERY-BASED INTENTION FILTERING
Analyze the user's input to evaluate the clarity of their query regarding website design. Assign a score from to 5 based on the criteria below:
Score 5: User provides clear, detailed description of requirements, including specific features and design elements.
Score 4: User's intention is mostly clear some details, lacks comprehensive specifics.
Score 3: User presents a general idea, but the request is vague and lacks essential information. User's input is unclear or contains irrelevant information, making their intention difficult to discern.
Score 2:
Score I: Input is nonsensical or purely code-related, reflecting no intention for design request. Out-
put: Return JSON object containing the reasoning for the score and an expected result, formatted as follows:
[["reason":
clone of WhatsApp Chat App", "score": "4"3,]
B.2 RUBRIC ANNOTATION PIPELINE
The annotation was conducted by two of the authors, all of whom possess strong backgrounds in computer science and software engineering. Since we have two annotators, the agreement rate is calculated as the proportion of instances where their annotations requires annotators to interact with the deployed web implementations and provide a preference label via pairwise comparison. However, this direct approach yields low inter-annotator agreement, underscoring the high degree of subjectivity inherent in the task and the need for a standardized evaluation framework.
the same. The annotation
To address this, we introduced a rubric-guided annotation process. Manually creating detailed rubrics for each query not only prohibitively time-consuning-wiin rubric, excluding research time-but also intellectually demanding. It requires extensive back- ground knowledge, including familiarity with established UI/UX design patterns for similar ap- plications and, in many cases, specialized domain knowledge for tasks like scientific simulations. Recognizing these challenges, we leveraged powerful large language model, gemini-2 aomini-2.5-pra 5-pro, for automated generation. Based each user query, the model produced structured rubric tree organized along three core dimensions critical to web development quality:
Intention:
The core requirements of the user query.
Static Quality: The assessment of static elements, including UI layout and UX design.
Dynamic Behavior: The evaluation of interactive features.
The prompt used to generate these rubric trees is provided below.
Upon manual review, the LLM-generated rubrics exhibited both strengths and weaknesses. For queries that referenced real-world applications, the rubrics were often of high quality, sometimes surpassing
versions in detail. However, for some general queries, the generated rubrics sometimes included criteria that were either too specific or too broad, reflecting the ambigu- ity of the original request. We present several examples of the rubric tree structure, encompassing the
18

---

Published as a conference paper at ICLR 2026
Table Examples problematic cases. In each example, the upper figure referred as model a, while the lower one is model.b. b.
Query
Implementation
Analysis
iPhona Pro
Build website iden- tical to apple.com.
Initial label: model_b Annotated label: model_a
Problem type: subjective label bias.
Reason: A thorough check of apple.com shows that A's rest- oration is much more accurate.
Build 4x8 version of the game 2048.
Initial label: tie (bothbad) Annotated label: model
Problem type: misjudgment in "bothbad" tics.
Reason: While both have clear usability flaws, B is still clearly superior to A in terms its ali- gument with the query.
manually-curated example provided in the one-shot rubric generation, alongside instances of both relatively high and low quality -egeeerre rubrics, as illustrated in Figure 9 in Appendix F.
Table 10: Statistics of the generated rubric trees.
Metric
Intention
Static
Dynamic
Whole
Average height
2.0
3.9
3.3
5.0
Average number of leaf nodes
3.6
15.6
10.6
29.9
We also provide the statistics of the rubric trees; the details are shown in Table 10 From the statistics, especially the average number of leaf nodes, wide range of criteria, which a good sign for the evaluation of web development tasks. To ensure consistent application of these rubrics, we established set of clear annotation guidelines for the
that the LLM- -LM-generated rubrics cover
19

---

Published as a conference paper at ICLR 2026
expert annotators, detailed in the following. This structured pipeline, combining LLM --M--eeerrted rubrics with clear human oversight and guidelines, proved highly effective.
ANNOTATION GUIDELINES
All judgments must be based on actual user experience, not solely on visual appearance.
In general, the completeness of functionality takes precedence over aesthetics, unless the user query
explicitly requests focus design.
If both web pages are well-implemented, aesthetic quality should be considered deciding factor.
there any important and discemible difference quality between the two pages, a preference should be stated rather than defaulting to a tie.
5. When a decision is difficult, the rubric tree should serve as the definitive guide. Refer to the fulfilmmntt of both leaf-level and root-level criteria.
When judging based the rubric tree, consider functional equivalence rather than demanding literal, identical implementations.
PROMPT FOR RUBRIC TREE BUILDING
#t TASK DESCRIPTION
You are an expert software quality assurance (QA) analyst. Your task is to take a user query for a web development project and generate a structured, hierarchical rubric. This rubric will he used to evaluate generated webpage in verifiable, binary (implemented/not implemented) manner. The output must be single JSON object with >* "'soon and ..* wrapped around it. The JSON object must have three top-level keys: 'intention', 'static", and 'dynamic'. ### JSON Structure Rules:
1. Each node in the tree must be dictionary with two keys:
''description": : string describing the feature goal.
-'children's A list child nodes, or None if is leaf node.
The 'intention' section should capture the high-level purpose and core goals of the webpage. De- scriptions should be concise and conceptual overviews of what the user wants to achieve. 3. The 'static' section must detail all the non-interactive, visible elements of the webpage. Break down components into their smallest logical parts. For example, "user profile card" should be broken down into "user image", "username", and "user bio."
The 'dynamic' section describes all the interactive functionalities of the page. tions.
It must have exactly two children: one for "basic" interactions and one for "complex" interac-
These are simple, single-step user actions. Examples include typing into field, clicking a non-submitting button, or selecting dropdown option.
"basic":
'complex": These are multi-step processes or actions that result in a significant change to the application's state. Examples include submitting form, fetching data, filtering a list of items. or navigating new view after action.
Verifiable Leaf Nodes: Every leaf node in the entire tree (where "children" None) must describe a specific, atomic, and verifiable requirement. The description should be a clear statement that can be evaluated as "implemented" or "not implemented".
## Example:
### User Query:
(example }uery]
#*l Generated Rubric Tree (Your Output):
"jsonn (example.ruricc.ce)
Now, analyze the following user query and generate the rubric tree in the specified JSON format. ### User Query:
query)
B.3 DATA STATISTICS
In this section, we describe the categorization pipeline for WEBDEVJUDGE and present representa- tive examples for each subcategory. We adapt topics from the original dataset (Vichare et al.) 2025) to serve as our subcategories. Given that WEBDEVJUDGE contains only 654 instances, we did not topic modeling model for clustering. Instead, generated subcategories by providing the
20

---

Published as a conference paper at ICLR 2026
topic name, detailed description, and the user query to GPT-4o. These subcategories then manually reviewed and consolidated into three main categories.
C EXPERIMENTAL DETAILS
In this section, we show the details of the implementation of our experiments. Including the mod- els and hyperparameters, the details of evaluation protocols and metrics, and the agentic workflow implementation.
C.I MODELS AND HYPERPARAMETERS
Our experiments utilize combination of open-source and commercial models. Open-source mod- els were deployed using vLLM (Kwon et al.) 2023) and SGLang (Zheng et al. 2024 Commer- cial models were accessed via their respective APIs, including Azure" and Vertex A To ensure reproducibility, we set the temperature to 0.0 for all non-reasoning models. For reasoning mod- els, the temperature was set accordingly. For brevity, the models referred to DeepSeek-V3 and DeepSeek-R in the main paper correspond to DeepSeek-V3-0233 and DeepSeek-R1 respec- tively.
C.2 EVALUATION PROTOCOLS AND METRICS
Likert Scale Motivated by the work of Bian al. 2025), ) designed multi-level Likert scale for evaluating web development tasks. The scale is based on the international standard for software quality assessment (ISO/EEC/EEE)) 2022) and has been specifically adapted for dataset. The details of the Likert scale are shown in "Dimensions of the Likert Scale".
In selecting these dimensions, we considered the typical workflow of human evaluators in web testing. Given that the tasks in our dataset are primarily front-end focused, and acknowledging the current limitations of generative models in producing efficient, full-stack solutions, we have concentrated on fundamental aspects of web development, omitting higher-level criteria such as backend performance and efficiency.
For single- ingleentpllnnnnnoooonn evaluation, the model assigns a score from 1 (lowest) to 5 (highest) to each dibedimension. The final score is the sum of all sub-dimension scores. For pairwise com- parison, both implementations are presented to the model simultaneously. which allows it to assign scores based on their relative merits. The final preference determined by the score difference be- tween the two implementations. A preference is declared for the higher-scoring implementation if the score difference exceeds a threshold of 1; otherwise, the outcome is considered tie.
Rubric For rubric-based evaluation, the process guided by the LLM LLM--genrated rubric tree. The model is provided with the rubric, the user query, and the web implementation(s). To mitigate inherent ambiguity in rubric-based assessments, we instruct the model to recognize functional equivalence. For instance, requirement for a heading is considered met if heading element is present, even if its text or styling differs from a literal interpretation of the rubric.
In the single-answer grading setting, the model assigns binary label (pass fail) to each leaf node of the rubric. We then compute a pass rate of the leaf node for each of the three primary dimensions: intention, static, and dynamic. The final score the weighted comparison, the model evaluates the two implementations against each leaf node, determining which superior or if they are tied. This yields win rate for each implementation of the leaf nodes across the three dimensions. A final score for each is calculated as a weighted sum of these win rates. Preference is awarded to the implementation with the higher final score. our implementation, dimension weights for intention, static, dynamic are set to 1, 3, 2, respectively.
of these pass rates. For pairwise
Direct For direct evaluation, instead of providing any criteria, we instruct the model to directly output its preference (i.e., W.. We. or tie).
21

---

Published as a conference paper at ICLR 2026
DIMENSIONS THE LIKERT SCALE
Evaluation Criteria
Functional Correctness and Completeness
1.1 **Core Functionality*** Evaluates if the primary features and requirements specified in the user query are implemented correctly and function as expected.
1.2 **Content Accuracy and Completeness** Assesses if all the required content (text, images, links, etc.) is present, accurate, and correctly placed as per the user's query. 1.3 **Boundary Conditions and Corner Cases*#; Examines the solution's behavior with unexpected or extreme user inputs.
1.4 Erroo Handling**: Evaluates the system's 's ability to handle errors gracefully. This includes providing clear, user-friendly error messages and preventing application crashes due to invalid operations.
User Interface Quality
- 2.1 Visual Consistency and Cohesion Assesses the consistency of design elements such as color schemes, typography, spacing, and component styling throughout the web- page.
2.2 Layout, Structure, and Responsiveness**: Evaluates the overall layout and structural organization of the content. This also critically assesses the responsiveness of the design across different screen sizes (desktop, tablet, mobile).
2.3 ** Aesthetic Appeal*#: Assesses the overall visual appeal of the webpage. This includes the effective use of color, typography, imagery, and whitespace to create an engaging and modern user interface.
Code Quality
- 3.1 **Redabblity and Maintainability**: Assesses the clarity and organization of the code. This includes proper indentation, meaningful variable names, comments where necessary, and logical file structure.
-3.2 ** Modulariiy and Reusability Evaluates whether the code broken down into logi- cal, reusable components or functions, avoiding monolithic structures and code duplication. 3.3 **Scalability and Efficiency**; Assesses the efficiency of the code, as well as the ability to scale the codebase for future enhancements or new features.
4. Interactivity:
4.1 ##fffcctiveness** Assesses the functionality and user experience of interactive elements like buttons, forms, menus, and sliders. This includes visual feedback user actions (e.g., hover states, loading indicators).
4.2 Loogccal Correctness** Evaluates whether the application state changes correctly in response to user interactions.
4.3 ** *Acessibbiitt*** Evaluates how easy and intuitive for user to navigate the webpage and interact with its elements to achieve their goals.
C.3 AGENTIC WORKFLOW IMPLEMENTATION
This section details agentic workflow implementation, shown in Table 11 rability with our other evaluation methods, the workflow'ss 's planner adopts the generated rubric tree guide the executor's verification actions. The executor is UI-TARS-1.5 (Sed 2025 of the state-of-the-art end-to-end GUI agents operating on a ReAct-style paradigm (Yao et al. 2023). where it first generates thought and then corresponding action. These actions converted into code. executable pyautogui code. We utilize the official UI-TARS-1.5 action space and have designed specific prompting strategies for each rubric dimension. For the Static dimension, the rubric tree is converted into list of elements: the agent navigates the page to find these elements and returns list of those present. For the Dynamic c and Intention dimensions, each rubric leaf node becomes task for the agent to complete. Upon reaching the maximum number of steps, the agent provides conclusion on the task's outcome. Finally, the summarizer calculates a pass rate for each dimension
To ensure compa-
hhtpp://yyaatogu..reattteee hwwp:://gggg.ocecccccecssee//aaa io/en/latest/
22

---

Published as a conference paper at ICLR 2026
based the agent's findings, and the final score is the weighted sum of these pass rates, mirroring the single-implementttion rubric evaluation.
Table 11: Details of the agent settings.
Dimension
Input Format
Output Format
Max Step
Static
List of elements to check
List found elements
Dynamic
Task description
Task feasibility and conclusion Feasibility conclusion
for basic, 15 for complex
Intention
Intention description
15
ACTION SPACE OF UI-TARS-1.5
click (piittoont==<pooinxxx yisp:pittttiy////o///////// y1</point>")
left double(poin===<pointx> y1</point>')
right singlleppoint=<<pointxxl yl</point>')
drag(start potttrttpintt""<poinnrt y1</point>', end point=='poiittxx y2</point>')
(ree'''kkccctrl c') Split keys with space and use lowercase. Also, not use more than 3 keys in one hotkey action.
type (cootten=="xxx') # Use escape characters V, V", and \n in content part to ensure parse the content in normal python string format. If you want to submit your input, use \n at the end of content.
y1</point>', direction= down or up or right or left') Show more information on the 'direction' side.
wait() Sleep for 5s and take screenshot to check for any changes. finished((ontent==xxxx)) # Use escape characters V \", and An in content part to ensure we can parse the content normal python string format.
C.4 PROMPT TEMPLATES
The prompt templates used for pairwise comparison experiments follows. For single-answer grading, the only action is simply to modify the input presentation, for example, by modifying two separate code blocks into
PROMPT FOR DIRECT COMPARISON
You tasked with comparing two React code snippets (Model A and Model B) based on the user's
query. The input will contain:
User Query: The question or concern the user has.
Answer from Model A: The output of Model A.
Answer from Model B: The output of Model B.
Your output should be in the following JSON format:
"reason": "Detailed explanation for why one model is better, based on the user's 's query", "winner": "'modelaa" a", 'model.b b', 'tic"
reason: detailed explanation of why one model's answer better than the other, based user's query. Focus on the quality of the responses and how well they address the user's concerns.
winner: Select the winner from the following options:
"model a": If Model A is better.
"model_b": If Model B better.
"tie": both models are equally good or both provide unsatisfactory answers.
Ensure that you thoroughly evaluate the responses before selecting the winner and providing the rea- son.
23

---

Published as a conference paper at ICLR 2026
PROMPT FOR RUBRIC-BASED COMPARISON
You are an expert Quality Assurance engineer specializing in development. Your objective is meticulously evaluate and compare two different web development solutions for the same task based on predefined rubric. You will be provided with the user's initial query, two solutions (codes for both webpages A and B), and comprehensive rubric covering intention, static, and dynamic elements of the webpage.
rubric implemented in each of the two solutions. Avoid any position biases and ensure that the order in which the responses were presented does not influence your decision. Do not allow the length of the responses influence your evaluation. Be objective possible. During your assessment, please note that the solution might use different terminology than the rubric. Consider requirement met if the solution feature equivalent. For examplc, the required heading element present on the webpage, though the exact text or symbol differs.
Based on these inputs, you will assess whether each requirement
## User Query
##f Code A "tsx {cnde_a)
#il Code B
[code_b}
## Rubric
### Intention
"jsonn
(intention rubric)
### Static Elements
json
rubric)
#### Dynamic Elements
jsonn
(dynamic rubric}
## INSTRUCTIONS Your task is return single JSON object. This object should have three top-level keys: "intention", "static", and "dynamic", The value for each key should a JSON object that mirrors the structure of the corresponding rubric provided above. For each leaf node in each rubric (i.e., where "children" is null), you must add a new key "value". The value for this key must be string: "A" if solution A is better, "B" if solution B is better, "tie" if they are of equal quality or both fail to meet the requirement.
#ii Output Format
Begin your evaluation by providing an explanation for your reasoning. End your output with a JSON object wrapped with *** json at the beginning and at the end. Do not include any other text after the JSON object.
Here is an example of the output format: jsonn
("intention": [ "description": "The purpose of the web page.", "children": "description": "A web page book reviews.". "children": null, "value": "A" }. "static": { "description": "The static elements of the web page.", "children": [ "description". "The book review submission form.". "children": [ ("description": "A field to input the book's 's rating.". "children": null, "value": "tie" }1). "dynamic": "description": "The interaction between the user and the web page.", "children": "description": "Basic user interactions." "children": "description": "User can type into the review text area.", "children": null, "value": "B"}1]])
24

---

Published as a conference paper at ICLR 2026
PROMPT LIKERT SCALE-BASED COMPARISON
You are an expert Quality Assurance engineer specializing in web development. Your objective to meticulously evaluate and compare two different web development solutions for the same task. You will be provided with the user's initial query and the solutions (codes for both webpages A and B). Based on these inputs, you will assess the quality of each solution across several key dimensions. For each sub-criteria, you provide rating Poor" and represents "Fxcellent". Avoid any position biases and ensure responses were presented does not influence your decision. Do not allow the length of the responses to influence your evaluation. Be objective as possible.
5-point Likert scale, where represents "Very
order in which
(DIMENSIONS OF THE LIKERT SCALE]
## User Query
## Code A tsx {code_a}
## Code B
tsx
(code b}
#i Output Format
Begin your evaluation by providing short explanation. End your output with json object wrapped with "jsoon at the beginning and at the end. Use the sub-criterion id as the key. The value for each key should be nested json object containing the scores for cach solution, with "A" and "B" as keys. Do include any other text after the json object. Here an example -sonn
output format:
("1.1": ("A": "B":
"B" 4 J.4...... "1.2% 'AA": 4, "B": 5.. "133: A" ","" ("..
2 }. "2.1": ("A": S, "B".
("A": "B". "2.3": ("A": "B"":: "3.1": ("A" 5, "B": "3.2% (7A": 4, "B": "3.3%: ("A" "B". "4.1% (AA" 5, "B" "4.2"
"2.2% "2.2%
"A": 5B. ("""""""""""""""""""""""""""""""
25

---

Published as a conference paper at ICLR 2026
D WEBDEVJUDGE-UNIT DATASET
In this section, we introduce WebDevJuuge-Unis a task-level dataset created to assess the capability of evaluators to verify task feasibility.
D.1 DATASET CONSTRUCTION
To construct the WebDewJudge-Ung dataset, we began by randomly sampling 105 queries from WEBDEVJUDGE, For each query, we prompted iimini-2.5-pra to generate up to five correspond- ing verification tasks, each with an expected result. We then generated the necessary HTML code for each query to facilitate easy deployment. Following deployment, each task was meticulously an- notated for feasibility (true/false) by interacting with the live web application. For tasks identified as infeasible, we further annotated the specific error type and provided detailed reason for the failure.
Table 12: Statistics of the WebDevJuuge-Unie Unit dataset.
Example
Error Type
Proportion
Description
Error Reason
Task: On Question Review page, mod- ify the text content a displayed question within editable field.
Unable edit question con- tent.
Non- Non-functional element
35.9
Expected Result: The displayed question updates reflect the new input.
Task: Click the 'Market' tab.
Can not find the Market tab.
Missing clement
30.9
Expected Result: The 'Market' view is played, showing list cryptocurrencies.
Task: Select an answer for multiple-choice question by clicking on one of the options. Expected Result: The chosen answer op- tion is visually highlighted.
Unable to import the quiz correctly.
Prerequisite not met
15.2
Task: Click file name in the File Explorer
Unable load file content.
Loading issue
9.0
sidebar.
Expected Result: The corresponding file's mock content is shown in editor tab.
Task: Scroll the chat display area view older messages when content overflows. Expected Result:
Unreasonable outcome
5.4
The
logged
Task: Hover over data point or segment Chart.
Chart is not required in the webpage.
Ambiguous input
2.2
Expected Result: A tooltip containing mack data relevant to the hovered chart el- ement is displayed.
Task: From list of activities the 'Ac- tivities Screen", click on a specific activity's name or image. Expected Result: The 'Detail View" for the selected activity displayed, showing its full description.
Unable to get the expected result hased on the opera- tion.
Overly detailed
0.9
Task: Move the mouse cursor across the main 3D animated background area. Expected Result: The 3D background ani- mation dynamically responds to the cursar's position or movement.
The background 3D anima- tion is not responsive.
Missing animation
0.5
D.2 DATASET STATISTICS AND EXAMPLES
The resulting dataset comprises 502 tasks derived from 105 unique queries. The feasibility labels distributed as follows: 279 tasks marked as feasible and 223 infeasible. Table 12 presents a detailed breakdown of the error types for the infeasible tasks, along with illustrative examples for each category.
26

---

Published as a conference paper at ICLR 2026
E ADDITIONAL RESULTS
This section provides a more detailed analysis of LLM--s-aajudge performance on web development tasks, supported by supplementary experimental results.
E. MITIGATING THE IMPACT OF POSITIONAL. BIAS
To mitigate the impact of positional bias, we employ a widely-used debiasing technique from prior works (Zeng et al. 2024 Tan et al. 2025). This method involves evaluating each pair of implemen- tations twice, with their positions swapped in the second evaluation. A preference is considered final only if the model's choice remains consistent across both orderings. If the choice is inconsistent, the outcome recorded as a tie. We use this approach to investigate whether mitigating positional bias improves overall model performance. The results are shown in Table 13
Table 13: Agreement rate (%) with and without mitigating the positional bias.
w/ mitigating
w/o mitigating
Model
Direct
Likent
Direct
Likert
Claude-4-sonnet
68,6
68.3
70.0
70.2
GPT-4. 1
68.3
68.7
70.3
DeepSeekv3-0324
64.2
64.4
66.7
67.1
natural question arises as to why we did not employ the swap-based debiasing technique in our main experiments. Our decision to rely single-pass evaluation stems from two primary consid- erations. First, our core objective is to characterize the raw, unfiltered behavior of LLMs-as-judges understand their inherent biases. Applying debiasing techniques from the outset would mask or average out these effects, obscuring the very phenomena we seek to analyze. The results in Table T3 indicate that the overall performance difference before and after debiasing is not substantial in our setup. This finding reinforces the importance of studying the biases directly, as their presence is not always evident from aggregate performance metrics alone. Second, the single-pass evaluation mirrors many practical application scenarios where, for reasons of cost and efficiency, models are queried only once per pair. Therefore, our main experimental results offer a more realistic and cau- tionary benchmark for practitioners, highlighting the potential pitfalls of naively deploying these models without safeguards. Our methodology thus deliberately separates the diagnosis of inherent model behaviors from the evaluation of mitigation strategies.
E.2 PREDICTION DISTRIBUTION CONSISTENCY
To understand the nature of model failures, we analyze the consistency of predictions across dif- ferent evaluators. This investigation seeks to determine whether different models tend to on the same set of Sesnances--uugeese largely independent and model-specifie. The results of this label consistency analysis are presented in Figure and Figure 6
certain examples are inherently challenging- or if their errors are
Our analysis of prediction consistency across different models reveals significant disparity between the two evaluation paradigms. As illustrated in Figure 5 and Figure 6 the inter-model agreement under the pairwise comparison paradigm is substantially higher than that under single-answer grad- ing. In pairwise comparisons, the consistency rates between different evaluators generally exceed 75%, with many model pairs even surpassing 80%. In stark contrast, under single-answer answer grading, inter-model consistency drops significantly, typically hovering between 50% and 65%.
This discrepancy corroborates our core finding from the main text: pairwise comparison a more stable and reliable paradigm for complex, open-ended tasks like web development. Relative judg- ment in pairwise comparison constrains the evaluation scope, compelling the model to focus on discriminative features between two candidates, which is cognitively less demanding task than absolute scoring. Conversely, single-answer grading requires the model to possess a well-calibrated and consistent internal standard of quality, standard that varies greatly across different models. Consequently, the single-answer answer grading paradigm exposes the current models' deficiencies cali- bration, leading to inconsistent and less reliable evaluation outcomes.
27

---

Published as a conference paper at ICLR 2026
GLM-4.5
24.22 92262
claude-deeonnet
438%
12.24
Owen 108--38--0 Twinking-nniii 2507
1924
12.%
geminl-2.5 fashh
61.34 93.4%
760%
got-40
11.%%
Qwen3- 108--A38-lnsiie A38-Instrucc
Kimi- K2- Instruct
3349
15
17.2
13
83.44
73.%
11.4%
17.2%
19.20
19.4
100%
80.14
gemma-3
gpt-4.1 -319
24
19.%
82.7%
63.h5
14.2%-60
01.0%
22.3%
13
04.0%
165%
11.74
60.44..25
gemini-2.5
9.4
Qwen2. 3-14d--nnstuutt Instruct 611%
6.4%
13.3%
07.0%
55
54.%
63.4
63.8%
60.7%
53.2%
0000
52
585%
832%
63.4%
gemini-2.55 5- dladh-llltee
75.1%
94.5%
9..0
18%
6.6%
14.%
16.25
69.8%
74.0
13.4%
57
22.20
06.0
75.0%
14.%
%.3%
Qwen3-2358
Qwen2 3-88tnnnttttt Instruct
161
18 74
41.7
0100
16.19
Deep
Qwen2 ww-22--W--72B-nnttutt 1824 01.
1.00
12
76.7%
20.35
$1.2%
75.7%
65
13.5%
26
7366
1624
15.9%
clead oud rommret
BLM-LL
Daspdd Daspdeal-V33-02 24-024
geminl Mesh
gm8-4o
Kimi #2 in-trrr
gomma 2-mmma---27bil 3-27b-1t
gamint-22 2-pra
Orwnn2 3.14 ruscrr
gpt-4.1
classte: flasele-.7-eommm Premeet
gemiel-2 d----at---ittt
Own22 2553 Inseenee
Dropt Bankk Rd 100
Owen2 VL----VL--- 728 instruct
Ownnd San A.B-A Miw-B---hiiiingp
Gemm3 200 A38 calt--tttuur--22
Gwend 2333 -22--222----22 fnstrart
Figure 5: The consistency between model predictions under pairwise comparison.
Furthermore, the high consistency in pairwise comparison suggests that while different models may lack shared understanding of "absolute quality,' their internal mechanisms for making 'relative preference" judgments are more aligned. This reinforces our view that leveraging relative judg- ments is more robust approach for automated evaluation in domains characterized by nuance and ambiguity, where clear-cut right or wrong answers are scarce.
To further substantiate these findings, extend our consistency analysis rubric-basee based grading, examining its internal consistency across different models (Figure 7 and its alignment with the direct evaluation paradigm (Figure 8)
E.3 EVALUATION RESULTS NON-TIE CASES
Given the brittleness of language models when handling ambiguous tie conditions, we also present results for non-tie cases, where a clear preference was established. presented in Table 14 ex- cluding tie cases results in a substantial improvement in agreement rates for all evaluators, indicating that ambiguous comparisons are a primary source of error. More importantly, these results reinforce the core conclusions drawn from main experiments. The pairwise comparison paradigm consis- tently and significantly outperforms single-answer grading, highlighting its robustness for capturing relative quality. Furthermore, the patterns of different guidance mechanisms remain consistent: in the pairwise setting, both Likert scale and Rubric-based guidance yield comparable performance, suggesting that the relative judgment itself is the dominant factor. In contrast, for single-answer grading, the Rubric-based approach is demonstrably superior to the Likert scale. This reinforces our finding that structured, binary assessments provide more reliable signal for absolute evaluation than multi-point scales, which require level of calibration that models currently lack.
28

---

Published as a conference paper at ICLR 2026
1.0
Qwen2. 3-sttt-nsnttttttttteeeem
Instract MO 16.74-53 52.4% 33.44-5
0% 95-33.130
42.7%
50.1%
50.2%
gamini-2.S-pro -46.7% 00.1055..0
50.5%
40.% 45.04455000
Deep5 0o-p5-e---3--224
Qwen2.5 5 L-------------------- -nntt-t-t Instruct 65 46-6 22
Owen3-30B- A0B--AB-insttuct--22 2507 -30 15.0 (6-1543
10.7
52.154
SeepSeek- Seek- 0528
ssnnntt
UI-EARS-1:55 -225%
50.13
13.5%-44
57.8%-39
09.3.%%
31
55.0%
54.
43.25
54
samini-2.5-flaale -48 75 50336
40
56.0% 50.55
15.0%
57.7%
50.5%
Thinking-2507
claude-deennet 18:15
spt-4.1 -0.3%
60.2
60.3%
52
0%
(9-2%
54.2% 57
3%
55.3%
$4.45 20 64
71
0%
52.0%
25 5%
GLM-4.5
57.94
0..4
62.0055
gemini-2.5 flash lite
Kimi- K2- I2-trnttttt-- -7
54.7%
53.2%
2%
12. 52.415 55
13.0% 60
75635
www. 5%.5%
gemma-3 3 27b -122% 402% 33.0%
43.2%
54.2%
40.2% 5225.00.8 20..00...4460...
52.7%
-0.2
Qwend-23388
-239% 53.5% 43.2%
gpt-4o 5955% $3.1% 1%.2% 51 51.5% 24 aN
42.0%
$2.2%
5% 53 6% 58.0 Co
2% 60
20,24 52 an 28.3% $2.74 24.44444%6 2%
06.20
Owen 448 Instract
gemiini- S-pro
Deepiee B-13keokkkk-ooo 0324
Qwent (-------7------- PiB-nnstrucr
owenk 43 We--ettrict--50 2507
Doopt end- 1522
dauste Paannee
Owerdd 300 A3B 1.0------hiikieee Thinkini Thinking--22 2507
UA 2.8
gemini-22 S-Paase
daunte sunnet
gpt-a.1
gamini-2.8 faai--iiii
BLM-4.5
Kimi -.2 tranct
Ownn (aw00-22300-42.0 2807
gaamma 2-a.a---777dd
Qween 328 testrcct
Figure 6: The consistency between model prediction under single answer grading.
GUM455 00.0
OWeN3-2333- A220 -ratruch-35577
711%
Deepbont 2238
10:00
1329
NUN
Owenb www.--330--a. 200-0309---2----mmm---0 2547
321-49
Denpiesi--2-6511
O1 Inetruct
18.0%
12.7%
CUM43
0124
18.2
16/h
10441
10.4
10.0%
10.h
Deenn Lave www.ee 2017
Deeshaut Ruasz
Deerai 20lh Crpe dontreek 2h07
Deaid Aut indtinc
(a) Consistency under pairwise comparison.
(b) Consistency under single-answer comparison.
Figure 7: The inner prediction consistency between model predictions under rubric paradigm.
29

---

Published as a conference paper at ICLR 2026
DeepSiek A1--50 1529
81.24
70.2%
11.%
14.45
60.7%
16.2%
DeepSrek Al 1528 100%
50.6%
72.1%
70.45
60.0%
13.25
73.2%
DewsSeeci'3 -3-344 7LAM
DeeeSencis 5324
01.00
64.15
GUN-45-15.75
77.76
13.5%
6..7
Direct Models
GJt C2K43-1422 71.2%
72.%%
13.0%
13.0%
Om Cinerre
73.2%
12.5%
16.2%
teemrrt 600% 90.2%
71.0
72.1%
66.0%
65.5%
ope-ds
Desyitene 1928
Kast desterr
23 423 sooeec 2207
Desrdvaal 0029
Deeplest 22-2204
Rubric
Models
(a) Prediction consistency under rubric (pairwise comparison) and direct paradigm.
(b) Prediction consistency under rubric(single- answer grading) and direct paradigm.
Figure 8: The inter-method consistency between model predictions under rubric paradigm and direct paradigm.
Table 14:
Agreement Rate (%) (without tie) of different evaluators under different evaluation
paradigms.
DIGITAL DESIGN
GAME & APP
WEB & SPECIAL
AVERAGE
Model/Method
Single
Pair
Single Likert
Pair
Single
Pair
Single
Pair
GPT-4.1
67.39
83.7
72.16
84.02
69.46
82.63
69.72
GPTT-4o
58.7
80.98
64.95
81.96
68.26
77.84
63.85
80.37
Dwwm-2255VLL-72B-mm
49.46
76.09
53.61
76.29
$7.49
76.65
53.39
76.33
Gemini-2.5-flahhliie
57.07
70.65
57.73
70.62
55.69
70.66
56.88
70.64
DeepSSck-VV3-0324
67.93
79.35
65.98
76.8
68.86
80.84
67.52
78.9
Kimi-K2-Inst.
69.02
80.98
69.07
76.29
62.87
77.84
67.16
78.35
Dwem3-235B-A22D-mme
63.59
77.17
69.07
78.87
66.47
73.65
66.42
76.7
Owen3-30B-APe
300---3----
67.93
79.35
70.62
74.74
66.47
79.04
68.44
77.61
Qwen2.5-32B-B
55.43
73.37
76.8
$7.49
75.45
56.51
Gemma-3-27B-itt
50.54
70.65
58.25
64.95
48.5
73.05
52.66
Qwen2.5-14B.
1-B-44--nns.
51.63
59.78
51.03
57.73
42.51
58.08
48.62
58.53
Claude-4-Sannet
63.04
84.24
70.1
82.47
69.46
79.64
67.52
82.2
Tlaude-3.7-Sommt
76.09
82.07
71.65
76.8
68.26
83.83
72.11
80.73
Ge5ini-2.55pro
59.78
76.63
59.28
80.41
51.5
72.46
57.06
76.7
GL.-4.5
67.39
82.61
65.46
78.87
62.28
77.84
65.14
79.82
DeepScek-R1-0522
$9,24
80.98
67.53
79.38
$7.49
73.65
61.65
78.17
Qwen3-30B-Ae 300-33--BB-------- -hinkhink.
60.33
74.46
61.86
73.71
60.48
74.85
60.92
74.31
Rubric
GPT-4.1
79.89
81.52
72.16
75.77
71.26
87.43
74.5
81.28
GPT-4o
79.35
80.43
72.68
77.32
72.46
83.23
74.86
80.18
Qwen-2.5-VLL722--1
72.28
78.8
74.23
80.41
73.65
79.64
73.39
79.63
DeepSeck-v3-0322
79.89
83.15
68.56
80.93
77.25
86.23
75.05
83.3
Kimi-K2-Inse
74.46
83.15
70.1
78.35
75.45
83.83
73.21
81.65
Qwen3-235BBA222B-nns
78.8
78.26
69.59
69.07
71.86
74.85
73.39
73.94
DeepSeek-RI
78.8
81,52
67.53
72.16
73.65
77.25
73.21
GL.M-.55
79.35
82.07
69.07
73.71
77.25
79.64
75.05
78.35
Agentic workflow
70.11
67.01
76.05
70.83
30

---

Published as a conference paper at ICLR 2026
F CASE STUDY
F.1 EXAMPLES OF RUBRIC TREES
Examples of the rubric tree structure are illustrated in Figure 9 We can observe that for queries requiring specific domain knowledge (e.g.. the mid one), large language models can generate de- tailed and accurate rubrics, such as evaluation criteria for specific components. This highlights key advantage of LLMs over humans: the ability to leverage their extensive domain knowledge to generate precise rubrics for corresponding queries. However, for more general queries (e.g., the rightmsst query), the rubrics generated by LLMs can be overly fine-grained. This may introduce ambiguity during evaluation, for instance, by adding overly specific categories like "Content" for general "categories".
Cuars Design book recio rubm usion forr with helda fo" rating author and Mad out Bookino page
Zurry Melic ditt nod and portect, cline Lit wendows
Cuary Creats modor nouubook prziig App with caligoria the laucher Las ate the seston grade caragory.
"etetten 1
"standarr I
purpase
cat os
tereer pope reslevs teliprent
Metromptton giuutttiii empordee doen
tik Miowe
("omeriptin" "At anpfionten tracher merer grode culf.
Seroreetnn "Dapdy rudritted revirves " "chidren" Theadl
Inturmptoon "he sue feudir shoul repitate the doyour
"The coek devarea aulepaget
Seecripton "Tee iementa of the page.
"iroveeue" "theraokkn ee coleriss Secns"
"Seer plant: ordor WindoweThrms Trumme"
Eduerlsdinn Puli boaka "clkaren" eall sesertatoo peis inreutt looal
Cloding Apd31
ohideer Tra
grading conegrress"
Flacrmpttnn ourer srrs ioon ohe
adhber: momcrtti fvwm for Youcmme wet aone
"sacripaar" "A Soga "Wakke"")]
Mesorittaa" "The hww roveew?.
Elecmpptirt "Mupley the utts the deakt "onddwer sad
Vnylermenne with
exmeente teak; (dkuniption SSopley the Mouetio Eetombronn "Dhokey rnverniton hort: teithhe'' mal.
andiarr: call
Oraviennent
Service wchion
Skadec7:.cA)
"Agradin rou to lou content cangore "chleran"
"The ansoustion letwiee the the putryagee
Panurince ia oncton io dkaake the testiaaoonn Ara (-I
"dyanii "Mallwen
(ilhureece"" foo one rgyi thet sew woueeooreeee hattens
Seroreten "The ontoocte kown oo ane
mad. "uaer valurin
chem coswaed Wiiiews
Educippoon Naar can chit scannioe Vegancadee" cateary "ewaere
"Mcurittn" "Ooeokk ver ittecttion
(decoppann "Naar we radrcc the sniin "Alleee
plach "Comphoccee in "caasen
"Oompke interctton..
Descippion Syperssiiiiiee the ch hrwd)
halers sae selics caraary
Figure 9: Examples of the rubric tree structure derived from different sources with correspond- ing queries. The diagram illustrates human-written rubric trees used for few-shot generation (left), alongside good (center) and suboptimal (right) examples of LLM-gennrated rubric trees. Note that the children attribute of leaf nodes is explicitly set null. For conciseness, some internal nodes have been omitted. Leaf nodes are labeled in green.
F.2 FAILURE CASES OF LLM-BASED AND AGENTIC EVALUATORS
In this section, present failure cases that illustrate the different failure modes discussed in Sec- tion 4
Regarding limitations in understanding functional equivalence, examine the query and rubric shown in Figure / (right). Although the rubric presents some ambiguity, human annotators and GPT-4.1 successfully identify functional equivalence for both the specific category "Organization"
31

---

Published as a conference paper at ICLR 2026
Create modern notebook grading app, with catogories and the teacher will system grade each category, The should show totai score, reset button. Rubric (Excerpt):
Query:
"dscription": "A groding for the 'Orooizatiion' cotegory.">
"children":
("description" lobel disploying "Orgenkation", "children":
('description": "A set of stor icons for roting "Orgonization", "children": nul))
Instruction: Consider requirement met the solution's feature equivalent. For example, required heading element
present on the page, though the exact text or symbol differs.
Webpage:
GPT-A.1:
based across several categpries. categories implemented "Presentstion" "Content", "Creatiwity, and "Accwracy". The rubric, howeves, expects "Organbation", "Content", and "Weatness" as categories. For the purpose this evaluction, w consider "Presentation" equivalent "Orgoniastion" "Nentness' since the intention and functionality are the same, only the lobels differ.
provided solution
grading notebooks
Notebook Grading App
"Creativity"
('description" : 'A text lobel dispiyying Organiation"' "childrnn' noll "volue" truej ("tiescrptionn
ioons rating OOrgnication""
null true)
Coumbryy
DeepSeck-R1-0522. R1-0528:
Let analyte the solution against the rutric:
Annency
015
Each category has a label and star icons (though categortes differ from rubric). The user query says "categories" and the solution has categories. The rubric example categories explicitly
might just examples? The rubric exactly However, we must evaluate against exact words.
the
Total Soore
0120
('description"; : "A text lobel dispioying 'Orgoniation"" 'children" " null "volue" fulse). ioons rating Wrrgnbattion"" 'childien", null, "Vole"
("iescciption"
T.RS--S-..::
For static:
Thought: page Presentation, Creativily, Accuracy, terminology
equivalers element 2.2, Content eusts. But clement (Organizattonn present.
rqquired are Organization
Action: fnisheskontentt",, 2,2,, 2.2,
Figure 10: Failure cases in functional equivalence identification. While GPT-4.1 correctly identifies equivalence for the specific "Organization" category and the general "categories" requirement given the query, rubric, and instructions, other models like DeepSeek-RI R-0028 and UI-TARS-1.5 fail to do so. These models misinterpret the requirements and exhibit hesitation in recognizing the equivalence for general categories.
Task to check Click the 'eggenertee' button on an Al message.
UI-TARS-1.5:
Webpage
didding 'tageneente" such messags,
Cherbot
button waitss tooking at cumeent screon, the Al message only has icons actions le edit or copy. but no visble 'Raganerate" button, Mavse the outton aspears after genorating a resporse, but then''s no prior massage with regenenate optian here. What parthps need gene mestage first? But the tack assumes thene s.nn Al message with a Reponerate tton. Since the current screen doean't show masbe the interface dowan't support this, button masing, Altam wnively maybe miveed Rechecting the has three lears (pwnel)
and anotheri but Regenerate. performed here, it's inforuble.
(S Erder
Action: imated/onnennniilee!
Figure 11: Failure case in feasibility analysis due to operation error. The agentic evaluator fails to detect the target 'regenerate' element the provided screenshot, resulting in an incorrect assessment of the task as infeasible.
and the general "categories" requirement, provided with the query, rubric, and instructions. con- trast, other models, such DeepScek-R1 R1-0228 and UI-TARS-1.5, struggle with interpretation. As shown in Figure 10 these models suffer from misinterpretation, which leads to incorrect functional equivalence identification.
32

---

Published as a conference paper at ICLR 2026
Regarding limitations in feasibility analysis, agentic evaluators are constrained by their inherent capabilities. Figure illlstrrate such a case where the agent fails to locate the target 'regenerate" element within the screenshot, incorrectly leading it to classify the task as infeasible.
G THE USE OF LARGE LANGUAGE MODELS
We use large language model (LLM) general- purppos writing assistant. The primary of the LLM is to refine and improve the clarity and flow the text. Specifically, we provided the LLM with pre-existing draft text and clear outline of ideas. We instruct the model adhere strictly to the provided content, without adding or removing any information. The LLM's role is limited to enhancing the logical coherence and fluency of the language, ensuring the final text is more polished and easier to read.
33


Reviews:
[{"review_id": "mC1Kijoe0G", "reviewer": "Reviewer_FhLS", "summary": "This paper introduces WebDevJudge, a meta-evaluation benchmark for assessing the reliability of the LLM-as-a-judge paradigm in web dev tasks. The benchmark builds upon webdev-arena-preference and covers both static evaluation (code + screenshots) and dynamic evaluation (live interactive environment). It introduces the notion of a rubric and shows how the rubric leads to higher inter-annotator agreement for both humans and LLMs", "questions": "- Can you please answer my main issue? Could you also highlight inter-model agreement for all the models in the paper?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "Given the widespread adoption of LLM as a judge, the need for better \"judgements\" of llm-as-a-judge has grown as well. This paper adeptly addresses that. The paper text is for the most part clear and easy to follow. The benchmark is also original as to my knowledge there is not a specific benchmark for web judgements.", "weaknesses": "- The main issue I see is that the results for all models seem relatively similar (~50s to 60s). There's not a lot of variation in terms of performance and there's no statistical tests to indicate that these values are actually meaningful. I have a strong suspicion that the reason these numbers are so similar is in fact because of the rubric. As pointed out in the paper, it increases inter-annotator agreement, but my guess is that it likely increases agreement *overall* as well and doesn't accurately test a model's judgement capability (especially since this rubric will not be used in-the-wild). As a result, I don't quite buy that this benchmark is meaningfully capturing llm-as-a-judge performance.\n- A smaller point is that the tables and figures are difficult to understand in isolation. For example, figure 1 is not parseable as is and requires significant context outside of the figure. Same with Table 1. Perhaps more information in the captions could help here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "761OKxpUZq", "reviewer": "Reviewer_L3iM", "summary": "This work introduces WebDevJudge, a meta-evaluation benchmark meant to test how (M)LLMs can evaluate web development (both using static evaluators or interactive evaluators such as agents).  The authors create WebDevJudge by collecting and filtering example queries from webdev-arena-preference-10k and creating a hierarchical rubric for evaluating each instance. The rubric is meant to represent an effective evaluation strategy and help the evaluator focus on binary features for making its judgement, the authors show that this method yields stronger annotator agreement between humans. Although these rubric trees aid human annotation, the authors find that it has marginal effects on evaluators. The key results in the paper find that LLM evaluators (even very large ones such as R1) often have rather low agreement with human annotators when evaluating web development (in many settings including agentic evaluators, pair-wise evaluation, etc.)\n\nThe authors continue to analyze the failure cases for these evaluators by looking at how important multiple modalities are for evaluation (finding code to be the most important). The authors look at position biases for pair-wise evaluation, finding it to be prevalent. Additional errors within these evaluators include functional equivalence (knowing when something is close/good enough) and that agentic systems often fail due to their own operational reliability (meaning static evaluators and agentic verifiers both have shortcomings). Overall this is an excellent benchmark and analysis paper highlighting the core limitations within LLM-as-a-judge evaluators is not in how the task is set up nor the methodology for performing the evalation (pairwise or direct, single model or agentic) but lies within the core capabilities of the model. This work highlights future directions for improving LLM evaluators and offers a dataset for researchers to experiment on.", "questions": "1. Most of webdev-arena-preference-10k is filtered out. Which of the three criteria filtered out the most of them (or was it the environment filter)? I would think we'd want to capture as many of these examples as possible including the ambigious ones because that's how we expect users to query the model. I am wondering if this filtering biased the findings in someway where the dataset is primiraly focused on a subset of \"webdev\" where evaluators are exhibiting interesting failure cases.\n2. I know this is not standard, but given your hierarchical rubric, I am wondering if there was any thought in trying more than two examples for pair-wise evaluation?  I believe there's work that shows more than two examples does not improve performance for direct comparisons, but with your hierarchical rubric and for failure cases like functional equivalence, I wonder if having more examples could help (this may be a costly experiment, though, so totally feel free to ignore this).\n3. How often is the lack of agreement due to functional equivalence? Is there a way to bin the lack of agreement by the failure modes presented in the paper?  I am wondering if 1, there's a way to establish what seems the most important to improve to enhance these models but also 2 to see if these models are just suffering from \"ambiguity\". The example in Figure 4 feels potentially ambigious and the LLM evaluator could be correct, for example \"though the exact text of symbol differs\" where the expected text is \"demonstration\" and the seen text is \"presentation\", it doesn't feel as clear-cut to me that these two words are functionally equivalent in all contexts and maybe if we asked more humans we'd find that there is actually some disagreement here.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- **A challenging benchmark that evaluators can be tested on**, the highest performing evaluator achieves only 66% agreement with humans, indicating a large gap for improvement.\n- **Clear insight into challenges in current evaluators**, the analysis that follows the main results from the benchmark highlights several key factors causing models to fail to be more effective evaluators. Other researchers can easily identify and work on improving these common failure modes.\n- **Extremely well written, clear experimental design, good reproducibility**", "weaknesses": "- **Low number of human annotators**: Only two annotators were used; their agreement is high, but I do wonder about some of the examples where the models are failing to agree with humans. If you gave those examples to more annotators, maybe we would find that they are actually somewhat ambigious.\n- **Lack of concrete examples**: Some of these failure modes are quite high-level, like \"operational reliability.\" I didn't see any example outputs from the models, but placing a few in the appendix may go a long way in helping other researchers understand more concretely what the errors are. Another suggestion would be to release the evaluation traces from your models. Personally, I've found that to be incredibly helpful, as it avoids having other researchers rerun sometimes costly experiments.\n- I would like to see an actual rubric tree somewhere. Maybe I missed it (or it is in the appendix somewhere), but it would help me fully understand what the leaf nodes are and how they work.\n\n\nThe rest of my \"weaknesses\" are really questions for the authors. I overall think this is a strong submission.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0csRXFGnis", "reviewer": "Reviewer_qgbg", "summary": "Introduces WebDevJudge. A new meta benchmark to evaluate LLM-as-a-judge performance in web development. They show existing LLM judges underperform versus human experts in their setup. The benchmark is not solved. The benchmark supports both static text and real-time interaction setups.", "questions": "1. Can you share the few-shot LLM generation setup? What prompts were used (just example in appendix).\n2. How did you go from 1713 high-quality instances to 654 instances after rubric?\n3. “With explicit instructions to ignore position bias and remain objective”. How is this done? Why don’t we randomly swap position to avoid position bias? And “remain objective”, how does that work? What is the impact? You analyze this later, but what happens if you would do the evaluation with both positions (we can’t do this with humans since they will remember, but we can do this with LLMs).\n4. “Imposing rigid structured metrics might constrain the models’ inherent reasoning processes” — Did you use constrained decoding here? Or is it still free-form output (but with the mentioning of the rubric)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. A good analysis on the new benchmark that is introduced, in a space where benchmarks (and especially meta benchmarks) are much needed.\n2. Draws conclusions that seem sound, and are useful in both designing solutions as designing other benchmarks.", "weaknesses": "1. Details about the agentic setup are lacking. They have some details in the appendix, but no analysis of where the agentic setup fails. We know agentic setups with interaction with GUIs are still not great (and we know that they often are better with e.g. selecting using elements vs coordinates), but more details on how useful the agentic side is for the final conclusion would be good for the paper.\n2. In the appendix I see examples, but none have images. If this is a known limitation, can you highlight it? E.g. some websites are designed around images and will look worse if no images are used.\n3. Could propose (even in a weak-form) potential solutions to the current short-comings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OWVl3O18ha", "reviewer": "Reviewer_ayxS", "summary": "The paper introduces WebDevJudge, a benchmark to evaluate LLM judges in the web development domain. The benchmark consists of paired web implementations with human preference labels. The paper uses this benchmark to evaluate LLMs, MLLMs, and agentic workflows. The findings show a significant performance gap between the best LLM judges and human experts. The paper also identifies failure modes in LLM judges, such as a positional bias and inability to recognize \"functional equivalence\" between implementations.", "questions": "1. Do you think the rubric tree used during human evals introduces any undesirable biases that could cause agreement to artificially go up?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "- The paper focuses on an important and understudied problem, that of establishing the reliability of LLM judges in web dev scenarios.\n- The idea of using a query-grounded rubric tree for annotation is a novel and strong methodological contribution for collecting complex annotations. The high inter-annotator agreement (89.7%) validates this rubric-based approach and is especially motivating compared to w/o rubric.\n- The evaluation of existing evaluators is comprehensive, testing LLMs, MLLMs, and agentic workflows across static and interactive paradigms.\n- The error analysis is well done. It identifies precise failures like the inability to recognize \"functional equivalence\" and provides a clear -diagnostic analysis using the WebDevJudge-Unit dataset.", "weaknesses": "- The benchmark is fairly small (654 examples), which may limit the generalizability of the strong claims.\n- Some of the claims about the benefits of agentic evaluators might need caveats since the main result is based on a single agent (UI-TARS-1.5). \n- The paper identifies a precision/recall trade-off between static LLMs and interactive agents and suggests an \"ideal evaluator\" would combine them, but doesn’t actually run this experiment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)