WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Quality

## Abstract

Abstract The paradigm of LLM-as-a-judge is emerging as a scalable and efficient alternative to human evaluation, demonstrating strong performance on well-defined tasks. However, its reliability in open-ended tasks with dynamic environments and complex interactions remains unexplored. To bridge the gap, we introduce WebDevJudge , a systematic benchmark for assessing LLM-as-a-judge performance in web development, with support for both non-interactive evaluation based on static observations and continuous interactive evaluation with a dynamic web environment. WebDevJudge comprises human preference labels over paired web implementations, annotated with structured and query-grounded rubrics to ensure high-quality ground truth. Using this benchmark, we comprehensively evaluate various evaluators, including LLMs, MLLMs, and agentic workflows. We systematically investigate the impact of different paradigms and guidance mechanisms. Our experiments reveal a significant gap between LLM judges and human experts. In-depth analysis indicates this gap stems from fundamental model limitations, including failures in recognizing functional equivalence, verifying task feasibility, and mitigating bias. Overall, WebDevJudge presents a significant challenge to LLM-as-a-judge, offering insights to guide future research toward developing more reliable and capable automated evaluators for complicated scenarios. Code and data are available at https://github.com/lcy2723/WebDevJudge .

## 1 Introduction

Evaluate, refine, then evaluate again and refine again—large language models (LLMs) have achieved remarkable success across various domains through this iterative cycle (Madaan et al., 2023; Chen et al., 2023; Shafayat et al., 2025; Li et al., 2025a). Conventional evaluation paradigms rely heavily on human assessment (Zhong et al., 2024; Starace et al., 2025), which, while meticulous, presents a critical bottleneck due to its high cost and low scalability. In response, the paradigm of LLM-as-a-judge has emerged as a promising alternative (Zheng et al., 2023; Dubois et al., 2024), offering a scalable and cost-effective solution for crucial development stages like verification (Lightman et al., 2024) and reward modeling (Yuan et al., 2025). With the advent of sophisticated language agents capable of planning, tool use, and collaboration (Yao et al., 2023; Qin et al., 2024a; Liang et al., 2024), the role of LLM-as-a-judge is rapidly expanding beyond basic, well-defined tasks to encompass challenging real-world problems (Zhuge et al., 2025; Bian et al., 2025). This progression is critical, as it paves the way for more capable automated evaluators, creating the possibility for language models to self-evolve in complex, real-world applications.

However, a fundamental question regarding the reliability of LLM-as-a-judge persists.
Its effectiveness is well-established for
static and basic tasks (Chen et al., 2024; Saha et al., 2025; Gou et al., 2025; Lù et al., 2025),
but these successes share a critical commonality: they rely on static assessment of final outcomes. The reliability of LLM-as-a-judge in dynamic, open-ended domains involving complex interaction remains largely unexplored. Such contexts introduce significant challenges: dynamic evaluation requires continuous interaction with and comprehension of a changing environment (Li et al., 2023a; Paglieri et al., 2025), while their open-ended nature necessitates establishing feasible assessment standards (Li et al., 2025b). This gap between the expanding scope of automated judges and the lack of rigorous validation in complex, interactive settings highlights an urgent need for a new meta-evaluation benchmark. Such a benchmark is essential to assess, interpret, and ultimately enhance the reliability of LLM-based judges as we approach increasingly autonomous AI systems.

Consequently, in this work, we introduce WebDevJudge, a meta-evaluation benchmark for assessing LLM-as-a-judge on a representative complex and interactive task using the context of web development.
Web development offers an ideal testbed for complex dynamic evaluation, as it inherently has interaction requirements, whose assessment depends not just on static code but on real-time interaction. The task is also intrinsically open-ended, lacking a single absolute answer.
To establish a high-quality ground truth, we introduce a structured annotation methodology using query-grounded rubric trees, which decomposes high-level requirements into a verifiable hierarchy of fine-grained criteria. This rigorous protocol combining rubric with human judges yields an inter-annotator agreement of 89.7%, which substantially exceeds the 63% reported for MT-Bench (Zheng et al., 2023), confirming the reliability of the preference labels. Departing from traditional benchmarks focused on static text (Tan et al., 2025), WebDevJudge supports both static and interactive evaluation by offering multifaceted representations of each web implementation, including its source code, screenshot of the rendered webpage, and a fully interactive environment for dynamic assessment, as shown in the right part of Figure [1](https://arxiv.org/html/2510.18560v1#S2.F1). Evaluator performance is measured by agreement with expert human preferences on paired web implementations—a methodology widely adopted for nuanced, open-ended tasks where the absolute answer is deficient (Bai et al., 2022; Lambert et al., 2025).

Using WebDevJudge, we conduct a comprehensive evaluation of a wide array of judges, including LLMs, MLLMs, and agentic workflows, under various paradigms and guidance mechanisms. Our experiments reveal that a significant capabilities gap persists between the most advanced models and human experts, with a performance discrepancy exceeding 15%. Notably, different guidance strategies provide only marginal improvements in the pairwise comparison setting, suggesting that preference prediction through comparative assessment represents an internalized capability in models (Hua et al., 2025; Yu et al., 2025). Furthermore, while agentic workflows appear well-suited for interactive task evaluation, they fail to outperform vanilla models due to error accumulation across planning and execution stages (Pan et al., 2025). Through detailed error analysis and case studies, we systematically investigate the failure modes of automated evaluators. As part of this analysis, we construct WebDevJudge-Unit, a diagnostic dataset specifically designed to evaluate feasibility verification capabilities of different types of LLM-as-a-judge. Our investigation reveals two fundamental performance bottlenecks: (1) a persistent inability to recognize functional equivalence between diverse implementations that achieve the same objectives through different approaches or terminologies, such as implementations using the same title element with only variations in text, and (2) systematic weaknesses in feasibility verification, where static assessment suffers from low precision due to static code analysis limitations while interactive agents exhibit low recall stemming from their own operational constraints. These compounding errors ultimately undermine evaluator performance, pointing toward fundamental research directions for developing truly reliable automated evaluators in complex, interactive domains. Our main contributions are as follows:

- •
We construct WebDevJudge, a meta-evaluation benchmark that supports both static and interactive assessment of web development quality with high-quality preference labels.
- •
Comprehensive empirical evaluation of (M)LLMs and agentic workflow reveals that current LLM-as-a-judge approaches still fall significantly short of human-level reliability.
- •
Detailed error analysis identifies the systematic weaknesses of LLM-as-a-judge, providing critical insights for developing more reliable automated evaluators.

## 2 Related work

#### LLM-as-a-Judge

The paradigm of LLM-as-a-Judge leverages powerful large language models to simulate human-like assessment, enabling scalable and cost-efficient evaluation (Gu et al., 2025). This approach has been widely adopted across various domains, including question generation and answering (Bai et al., 2023), data filtering (Li et al., 2024; Xu et al., 2025), and trajectory evaluation (Pan et al., 2024; Xue et al., 2025).
Typical implementations employ a strong LLM as an evaluator, which compares or scores candidate responses against predefined criteria. Common comparison methods include pairwise comparison and single-answer grading Zheng et al. (2023). As application scenarios broaden, the conventional LLM-as-a-Judge approach is evolving into Agent-as-a-Judge (Zhuge et al., 2025), wherein LLM-based agents are equipped with tool-using and collaboration capabilities. Gou et al. (2025) propose a judge agent with extractor and verifier to evaluate deep research tasks with rubric trees, while Zhuge et al. (2025) design a modular agentic framework to evaluate agentic systems.
Despite its efficiency, this method faces several challenges. Position bias (Wang et al., 2024) and verbosity (Ye et al., 2025) preference may skew results, and the creation of detailed, human-written rubrics is both labor-intensive and difficult to scale (Starace et al., 2025). Additionally, typical assessments may overemphasize final outcomes, limiting their applicability in open-ended interactive tasks (Zhang et al., 2025a).
To evaluate the judge capability of LLMs, we introduce a benchmark on open-ended web development scenarios. It is designed to assess a wide range of LLM-based evaluators, with the dual purpose of benchmarking their performance and exposing fundamental shortcomings in existing automated evaluation approaches.

#### Meta Evaluation

Meta-evaluation accesses the effectiveness of automated evaluation by measuring its correlation with human or ground truth. Existing benchmarks primarily focus on two aspects: (1) alignment with preference label, often using pairwise comparisons (Zheng et al., 2023; Li et al., 2023b), and (2) accuracy in identifying correct task outcomes, particularly in reasoning (Luo et al., 2023; He et al., 2025) and agentic tasks (Lù et al., 2025).
Prior works such as MT-bench (Zheng et al., 2023) and LLMEval (Zhang et al., 2023) measure how LLM judges mirror human preferences in multi-turn conversational and instruction following tasks. However, these benchmarks may be constrained by human subjectivity and position bias (Wang et al., 2024). Alternatives like LLMBar (Zeng et al., 2024) and JudgeBench (Tan et al., 2025) test judges on re-annotated instruction adherence or verifiable reasoning discrimination. These benchmarks mainly focus on text-based tasks without complex environments.
In the context of interaction, AgentRewardBench (Lù et al., 2025) uses expert-annotated result of pre-scripted trajectories to benchmark how well LLM evaluators score agent performance. ArtifactsBench (Zhang et al., 2025a) evaluates dynamic visual effects. However, it lacks a critical component of assessing environmental changes that are driven by real-time user input.
Our work, WebDevJudge, introduces a meta-evaluation benchmark that assesses judges on dynamic, real-world web development tasks, emphasizing continuous interaction with live web environments, offering insights in complex interactive settings.

Figure: Figure 1: Overview of WebDevJudge. Left: Data Collection with filtering. Center: Annotation with Rubric. Right: Evaluate various evaluators based on different evaluation paradigms.
Refer to caption: x1.png

## 3 WebDevJudge Benchmark

WebDevJudge serve as a meta-evaluation benchmark designed to assess whether LLM-as-a-judge can effectively approximate human preference judgments in web development tasks. We frame the task as a preference evaluation problem. Each instance is represented as a quadruple $(Q,W_{\text{a}},W_{\text{b}},l_{\text{p}})$, where $Q$ denotes a web development query (e.g., “build a book review page”), $W_{\text{a}}$ and $W_{\text{b}}$ represent web implementations from two distinct models $a$ and $b$, and $l_{\text{p}}$ is the annotated preference label between the two outputs. The objective is to predict the preference label—i.e., whether $a$ wins, $b$ wins, or the result is a tie—which is widely used for open-ended tasks (Zeng et al., 2024; Lambert et al., 2025). The observations of $W_{\text{a}}$ and $W_{\text{b}}$ may take various forms across experimental settings, such as code, screenshots, or interaction trajectories.

### 3.1 Data Filtering

We collect data from the webdev-arena-preference-10k dataset (Vichare et al., 2025; Chiang et al., 2024), which comprises 10,501 user queries with paired code outputs from 2 models and user-provided preferences. We apply a two-stage filtering process to enhance data quality:

#### Query-based filtering

We first exclude extremely short queries and verbatim duplicates. The remaining queries are further filtered via an LLM according to the following criteria: (1) Safety: exclusion of harmful or offensive content; (2) Clarity: the query must be unambiguous and interpretable; (3) Feasibility: the query should be realistically implementable as a web application, based on its purpose and required level of interaction.

#### Environment-based filtering

We deploy each web implementation in a unified execution environment. Instances that fail to deploy correctly or require niche dependencies are discarded. To ensure validity, we capture initial screenshots of each rendered webpage and use a multi-modal LLM to filter out invalid cases (e.g., blank pages or intrinsic errors).

After applying these filters, we retain 1,713 high-quality instances. Further details regarding the data construction process are provided in Appendix [A](https://arxiv.org/html/2510.18560v1#A1).

**Table 1: Agreement rates with and without the rubric-based annotation. The ‘without rubric’ part shows agreements between: (1) annotators and (2) annotators and the original labels. The ‘with rubric’ part shows inter-annotator agreements under rubrics from different sources.**
| Setting | Without rubric | With rubric |  |  |
| --- | --- | --- | --- | --- |
| inter-annotator | annotator-origin | Human-written | LLM-generated |  |
| w/ tie | 65.0 | 53.0 | 92.0 | 90.0 |
| w/o tie | 90.0 | 77.9 | 95.5 | 96.9 |

### 3.2 Annotation via Rubric tree

Consistent with prior work (Zheng et al., 2023; Tan et al., 2025), we note that raw preference labels may reflect subjective bias rather than objective quality. To quantify this issue, we sample 100 instances and have them re-annotated by two expert annotators based on fully deployed web instances. As shown in Table [1](https://arxiv.org/html/2510.18560v1#S3.T1), both inter-annotator agreement and agreement with original labels are initially low, highlighting the need for a more structured and standard annotation protocol.

To address this, we introduce rubric tree—a structured evaluation framework commonly used in complex assessment scenarios (Starace et al., 2025; Gou et al., 2025). The tree is query-based and scalable by design, organized along three core dimensions: intention, static quality, and dynamic behavior (see Figure [1](https://arxiv.org/html/2510.18560v1#S2.F1)). Each leaf node corresponds to a binary test, whose outcomes are aggregated hierarchically to parent nodes. This allows for both a holistic score at the root and fine-grained diagnostic insight via leaf-level pass rates.

We validate the effectiveness of the rubric tree by manually constructing trees for 50 instances and asking two annotators to evaluate these using the rubric as a reference. As shown in Table [1](https://arxiv.org/html/2510.18560v1#S3.T1), rubric-guided annotation significantly improves agreement rates. To scale this process, we employ few-shot LLM generation to automatically produce rubric trees. A third annotator labels the same instances using these generated rubrics, achieving high agreement with human-written rubrics—confirming the utility of LLM-generated rubrics. We then use generated rubric trees to annotate the remaining data via two expert annotators with software engineering backgrounds. During annotation, we perform manual inspections to exclude incompatible and harmful cases. Final inter-annotator agreement reaches 89.7% (with ties) and 94.0% (without ties), demonstrating high consistency.

Table: Table 2: Categories and their respective sub-categories of queries in WebDevJudge.

### 3.3 Data Statistics

The final benchmark consists of 654 instances, with preference distributions as follows: 258 for $a$, 258 for $b$, and 138 ties. To characterize the query domain coverage, we conduct a topic analysis based on the original topics webdev-arena-preference-10k dataset (Vichare et al., 2025). Fine-grained topics are grouped into three broad categories: Digital Design, Game and App Development, and Web and Specialized Technologies based on their shared characteristics and application contexts of representative examples. Table [2](https://arxiv.org/html/2510.18560v1#S3.T2) and Figure [2](https://arxiv.org/html/2510.18560v1#S3.F2) provide detailed category distributions and visualizations.

## 4 Experiments

We conduct comprehensive experiments and evaluate different evaluation settings on WebDevJudge, focusing on the following research questions: (1) Whether LLM-as-a-judge can be an alternative to human preference in open-ended complex tasks like web development? (2) How do different settings and strategies affect the agreement rate performance?

### 4.1 Evaluator Performance on WebDevJudge

#### Setup

Following (Zheng et al., 2023; Xie et al., 2025), we evaluate the evaluators under 2 distinct paradigms:

- •
Pairwise comparison: This approach directly compares two responses, leveraging relative judgments to infer preference, an intuitive and natural fit for preference prediction.
- •
Single answer grading: This approach employs LLMs to assign scores or labels to individual responses, with preferences derived by comparing scores or pass rates.

To enable structured and domain-grounded assessment, we design a multi-dimensional Likert scale inspired by prior work (Lan et al., 2024; Zhang et al., 2025a) and incorporate principles from international software testing standards (ISO/IEC/IEEE, 2022). Our criteria comprises four dimensions, Functionality, UI Quality, Code Quality, and Interactivity, each with several sub-criteria rated on a 5-point Likert scale (1: lowest, 5: highest). Detailed criteria are provided in Appendix [B](https://arxiv.org/html/2510.18560v1#A2).

#### Implementation

We evaluate two types of evaluators: (1) Vanilla (M)LLMs: we test models from OpenAI (OpenAI, 2025; 2024), Anthropic (Anthropic, 2025b; a), Google (Team, 2025a), Qwen (Team, 2025e; d), DeepSeek (DeepSeek-AI, 2024; 2025), Moonshot (Team, 2025c), and ZAI (Team, 2025b). For single answer grading, we provide the query, code, and evaluation criteria; we further include an initial screenshot of the rendered webpage for MLLMs. The model is prompted to output scores for each sub-criterion. For pairwise comparison, we supply the query, code of $W_{\text{a}}$ and $W_{\text{b}}$, evaluation criteria, and screenshots of both rendered webpages (if applicable). The evaluator scores both $W_{\text{a}}$ and $W_{\text{b}}$ per sub-criterion, with explicit instructions to ignore position bias and remain objective. Preference is determined by comparing total scores. (2) Agentic Workflow: similar to Bian et al. (2025), we model evaluation as a multi-stage pipeline with a planner, an executor, and a summarizer:

$$ $\displaystyle\text{Query}\xrightarrow{\text{{Planner}}}\text{Plan with test cases}\xrightarrow{\text{{Executor}}}\text{Results}\xrightarrow{\text{{Summarizer}}}\text{Judge}$ (1) $$

The planner generates a verifiable evaluation plan conditioned on the query. The executor runs test cases, and the summarizer synthesizes results into a judgment. We use the generated rubric tree as the evaluation plan and reference for summarization. Preferences are inferred by comparing summarized outcomes. In implementation, we utilize UI-TARS-1.5 (Seed, 2025), one of the state-of-the-art GUI agents, as the executor, to ensure the reliability of the evaluation process.

Table: Table 3: Agreement Rate (%) of different evaluators under different evaluation paradigms. The best average performance of the whole dataset is highlighted in bold and the second best is underlined.

#### Result

The main experimental results are summarized in Table [3](https://arxiv.org/html/2510.18560v1#S4.T3). Our analysis yields several key observations on the performance of LLM-based evaluators in web development tasks.

LLM-as-a-judge falls short of human-level reliability on complex evaluation tasks.
The primary finding is that no current model achieves a sufficient level of agreement with expert judgments. The top-performing evaluator, Claude-4-Sonnet under the pairwise paradigm, attains an agreement rate of only 66.06%. This substantial gap underscores the inherent complexity of web development evaluation, which demands a holistic assessment of functionality and aesthetics. We also observe a clear performance ceiling: while smaller models exhibit scaling effects, larger and more capable models show diminishing returns, consistently plateauing below the mid-70% agreement rate.

Pairwise comparison is a far more effective paradigm for preference evaluation.
Across the board, the pairwise paradigm yields an average improvement of over 8.0% in agreement rate compared to single-answer grading. Relative judgment helps models focus on discriminative features between two candidates, reducing the need for absolute quality calibration. In contrast, single-answer grading not only performs worse but also leads to inconsistent model rankings—some models deviate from their expected benchmark ordering (Zhang et al., 2025b), indicating that this paradigm is less suitable for open-ended, multi-faceted tasks. This may be due to the cognitive load of applying multi-point Likert scales consistently (Ouwehand et al., 2021), which demands a calibrated internal standard of quality that is difficult for both humans and LLMs to maintain. Simplified evaluation mechanisms, such as binary checklists, may be more effective; we explore this further in Section [4.2](https://arxiv.org/html/2510.18560v1#S4.SS2).

Agentic workflow suffers from compounding errors. We evaluated the agentic workflow using rubrics from different aspects, as well as a comprehensive score integrating all three. The results show that it performs significantly better on the Dynamic aspect, which involves strong interactive properties, compared to others with weaker interactivity. Counter-intuitively, the agentic workflow fails to outperform the vanilla models. This appears to result from error accumulation across its multi-stage process. We identify two primary failure modes:

- •
Brittle Planning: The Planner struggles with the ambiguity of user queries that are often not expressed with expert-level precision. This leads to the generation of evaluation plans that are either too generic to be discriminative or overly specific, causing failures due to minor implementation variations.
- •
Faulty Execution: The Executor agent’s ability to navigate the web and verify task states remains unreliable. It may misinterprets outcomes, injecting noise into the evaluation process. This unreliability is investigated more deeply in Section [4.3](https://arxiv.org/html/2510.18560v1#S4.SS3).

Errors compound across the planner-executor-summarizer pipeline, reducing the reliability of the final judgment compared to an end-to-end evaluation by a single model (Chen et al., 2025b).

### 4.2 Different Evaluation Guides and Observation Forms

To further diagnose factors affecting evaluator performance, we conduct controlled experiments focusing on two key variables: the structure of evaluation guidance and the form of observation.

Figure: Figure 3: Agreement rates of LLM evaluators under various guidance protocols. We compare pairwise and single-answer paradigms using direct judgment, Likert scales, and structured rubrics.
Refer to caption: x3.png

#### Evaluation Guidance

We investigate how different evaluation protocols influence judgment accuracy, comparing three widely-used forms of guidance (Zheng et al., 2023; Gou et al., 2025): (1) Direct: The evaluator provides a preference without explicit criteria. (2) Likert Scale: As used in Section [4.1](https://arxiv.org/html/2510.18560v1#S4.SS1), the evaluator scores outputs along predefined dimensions using a multi-point scale. (3) Rubric: The evaluator assesses outputs using the rubric trees from our annotation process, with final scores computed as a weighted aggregation of binary leaf-node pass rates across three core dimensions (intention, static, and dynamic).

Results in Figure [3](https://arxiv.org/html/2510.18560v1#S4.F3) reveal a noteworthy finding: under the pairwise evaluation, the Direct setting achieves agreement rates comparable to guidance-based methods. This phenomenon aligns with observations in instruction following tasks (Zeng et al., 2024), where models perform similarly on agreement with or without metric-based guidance. This result suggests that evaluation capability is an internalized skill in modern LLMs, and external guidance provides only marginal benefits in relative assessment settings (Qin et al., 2024b). Furthermore, we find that reasoning models exhibit better performance under the Direct condition, indicating that imposing rigid structured metrics might constrain the models’ inherent reasoning processes, thereby limiting their evaluation potential.
Furthermore, we validate the hypothesis from earlier: in single answer grading, the binary Rubric approach substantially outperforms the multi-point Likert scale, reinforcing that verifiable evaluation protocols yield more reliable judgments when there is a lack of relative information.

Table: Table 4: Impact of observation forms on the performance of multimodal evaluators. The numbers in parentheses indicate the performance change relative to the setting with both code and image inputs.

#### Influence of Observation Modality

We next analyze how the input modality affects multimodal evaluators, comparing three observation forms: (1) Image Only, providing only the initial screenshot of the rendered web page; (2) Code Only, providing only the source code; and (3) Both, providing both code and screenshot. As shown in Table [4](https://arxiv.org/html/2510.18560v1#S4.T4), code emerges as the most critical modality for evaluating web development tasks. Withholding code leads to a significantly larger performance drop than withholding screenshots, suggesting that while MLLMs can process visual input, their judgments are fundamentally anchored in the structured source code (Wang et al., 2025). In pairwise comparison, using both modalities yields the best results, indicating that visual context provides complementary signals that aid in refining relative judgments (Chen et al., 2025a).

### 4.3 Error Analysis

To systematically diagnose the failure modes of LLM-based evaluators, we conduct a fine-grained analysis along three key dimensions: inherent biases, limitations in understanding functional equivalence, and shortcomings in feasibility analysis.

**Table 5: Positional bias in pairwise comparison. Preference for specific position, consistency and the absolute difference in agreement rate ($\Delta$ AR) between original and swapped orders are reported.**
| Model | Consistency | First | Second | $\Delta$ AR |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Direct | Likert | Direct | Likert | Direct | Likert | Direct | Likert |  |
| Claude-4-sonnet | 89.6 | 87.9 | 5.7 | 3.7 | 4.7 | 8.4 | 0.9 | 1.38 |
| GPT-4.1 | 83.3 | 85.2 | 0.9 | 3.0 | 15.8 | 11.8 | 0.8 | 0.46 |
| DeepSeek-V3-0324 | 84.1 | 83.5 | 11.6 | 7.3 | 4.3 | 9.2 | 0.5 | 1.38 |

#### Inherent Biases in Judgment

Prior studies (Ye et al., 2025) have revealed that LLM-as-a-judge exhibits various forms of bias, such as positional and verbosity biases. We focus specifically on positional bias. Despite explicit instructions to ignore order and remain objective, models still exhibit a systematic preference for responses in a specific position, as shown in Table [5](https://arxiv.org/html/2510.18560v1#S4.T5). One might hypothesize that this bias emerges primarily in ambiguous cases where the two options are of comparable quality. However, our analysis of the label distribution for instances with inconsistent predictions reveals that the proportion of ties is not higher than that of wins or losses. This finding suggests that positional bias is not merely an artifact of ambiguity but rather an inherent deficiency in the models, and instruction alone is insufficient to eliminate these deeply embedded inductive biases.

Figure: Figure 4: An example illustrating the failure to recognize functional equivalence. The webpage element for ‘Presentation’ serves the same purpose as the required ‘Demonstration’ rating.
Refer to caption: x4.png

#### Limitations in Understanding Functional Equivalence

A critical requirement for reliable evaluation is the ability to recognize functional equivalence—i.e., determining whether different implementations satisfy the same underlying requirement. We find that evaluators often fail in such judgments, adhering strictly to literal interpretation rather than intent. For example, as illustrated in Figure [4](https://arxiv.org/html/2510.18560v1#S4.F4), when instructed explicitly, human evaluators correctly recognize an element implemented with alternative text, whereas both LLM and agentic evaluators incorrectly reject it. This reflects a fundamental gap in contextual and pragmatic reasoning, limiting their applicability in open-ended domains where diversity in implementation is common.

Figure: Table 6: Performance on the WebDevJudge-Unit for feasibility verification. We report precision (P), recall (R), F1-score, and accuracy (Acc).

#### Shortcomings in Feasibility Analysis

We further investigate the evaluators’ ability to accurately verify task fulfillment. Existing benchmarks such as AgentBench (Lù et al., 2025) are too general to assess web-specific judgment. To address this, we construct WebDevJudge-Unit, a targeted dataset of 502 test cases. Each instance consists of web code, a specific verification task, an expected result, and a label indicating whether the task is feasible. We evaluate both LLM-based (code-only) and agent-based (interaction-driven) evaluators on this dataset. From the results in Table [6](https://arxiv.org/html/2510.18560v1#S4.T6), we identify complementary weaknesses: (1) For LLM evaluators, these models achieve moderate recall but suffer from low precision. While they can often identify relevant code segments, they are unable to verify actual execution outcomes, leading to false positives when the code appears relevant but does not correctly implement the desired functionality. (2) For agentic evaluators, they exhibit higher precision but lower recall. They are effective when they successfully execute a test plan, but sometimes fail to complete tasks due to limitations in navigation or state interpretation. Consequently, they incorrectly label feasible tasks as infeasible due to their own operational failures rather than actual shortcomings in the web implementation. This divergence highlights a fundamental trade-off: static code analysis lacks execution grounding, while interactive agents are constrained by their own operational reliability. An ideal evaluator would combine the comprehensive coverage of code-aware reasoning with the grounded verification of interactive testing.

Our analysis reveals that the core limitation of LLM-as-a-judge lies in fundamental deficiencies in calibration capability. Lacking this calibration, LLMs struggle to map abstract quality dimensions onto discrete scores and verifiable rubrics. Improving judge performance will require addressing these core competency gaps rather than merely refining evaluation protocols.

## 5 Conclusion and Future Work

In this work, we introduce WebDevJudge, a comprehensive benchmark for evaluating LLM-as-a-judge in web development. Unlike previous benchmarks, WebDevJudge supports both static code analysis and interactive agent navigation with high-quality preference labels. Our experiments demonstrate that current LLM-as-a-judge approaches cannot effectively substitute human evaluation, and we identify core bottlenecks hindering their performance. Since our primary focus is on evaluation and analysis within the general LLM-as-a-judge domain, we did not specifically optimize the overall framework structure. We leave the exploration of sophisticated agentic workflows and complex multi-round evaluations to future work.

## Ethics Statement

The ethical considerations of our work are discussed in the context of the following aspects: (1) Data collection and use. We use publicly available datasets and self-generated data for evaluation. We ensure that the data is only used for academic research purposes and no personal data is involved. We strictly follow the license terms and conditions. (2) LLMs API. We comply with the terms of service of the LLMs API providers strictly, maintaining fair use.

## Reproducibility statement

We provide detailed descriptions of our methods, datasets, and evaluation metrics in the main text and appendix to ensure transparency and reproducibility. Our benchmark construction process for WebDevJudge  is detailed in Section [3](https://arxiv.org/html/2510.18560v1#S3) and Appendix [A](https://arxiv.org/html/2510.18560v1#A1). The experimental setup, including the models, evaluation paradigms, and specific protocols, is described in Section [4](https://arxiv.org/html/2510.18560v1#S4) and Appendix [B](https://arxiv.org/html/2510.18560v1#A2). We also provide a complete description of the WebDevJudge-Unit dataset in Appendix [C](https://arxiv.org/html/2510.18560v1#A3). Code and scripts are available to replicate the empirical results.

## References

- Anthropic (2025a)
Anthropic.
Claude 3.7 sonnet and claude code, 2025a.
URL [https://www.anthropic.com/news/claude-3-7-sonnet](https://www.anthropic.com/news/claude-3-7-sonnet).
- Anthropic (2025b)
Anthropic.
Introducing claude 4, 2025b.
URL [https://www.anthropic.com/news/claude-4](https://www.anthropic.com/news/claude-4).
- Bai et al. (2022)
Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.
Constitutional ai: Harmlessness from ai feedback, 2022.
URL [https://arxiv.org/abs/2212.08073](https://arxiv.org/abs/2212.08073).
- Bai et al. (2023)
Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou.
Benchmarking foundation models with language-model-as-an-examiner.
In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023.
URL [https://openreview.net/forum?id=IiRHQ7gvnq](https://openreview.net/forum?id=IiRHQ7gvnq).
- Bian et al. (2025)
Yutong Bian, Xianhao Lin, Yupeng Xie, Tianyang Liu, Mingchen Zhuge, Siyuan Lu, Haoming Tang, Jinlin Wang, Jiayi Zhang, Jiaqi Chen, Xiangru Tang, Yongxin Ni, Sirui Hong, and Chenglin Wu.
You don’t know until you click:automated gui testing for production-ready software evaluation, 2025.
URL [https://arxiv.org/abs/2508.14104](https://arxiv.org/abs/2508.14104).
- Chen et al. (2024)
Dongping Chen, Ruoxi Chen, Shilin Zhang, Yaochen Wang, Yinuo Liu, Huichi Zhou, Qihui Zhang, Yao Wan, Pan Zhou, and Lichao Sun.
Mllm-as-a-judge: assessing multimodal llm-as-a-judge with vision-language benchmark.
In *Proceedings of the 41st International Conference on Machine Learning*, ICML’24. JMLR.org, 2024.
- Chen et al. (2025a)
Feng Chen, Chenhui Gou, Jing Liu, Yang Yang, Zhaoyang Li, Jiyuan Zhang, Zhenbang Sun, Bohan Zhuang, and Qi Wu.
Evaluating and advancing multimodal large language models in perception ability lens, 2025a.
URL [https://arxiv.org/abs/2411.14725](https://arxiv.org/abs/2411.14725).
- Chen et al. (2023)
Jiefeng Chen, Jinsung Yoon, Sayna Ebrahimi, Sercan Arik, Tomas Pfister, and Somesh Jha.
Adaptation with self-evaluation to improve selective prediction in LLMs.
In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2023*, pp. 5190–5213, Singapore, December 2023. Association for Computational Linguistics.
doi: 10.18653/v1/2023.findings-emnlp.345.
URL [https://aclanthology.org/2023.findings-emnlp.345/](https://aclanthology.org/2023.findings-emnlp.345/).
- Chen et al. (2025b)
Meilin Chen, Jian Tian, Liang Ma, Di Xie, Weijie Chen, and Jiang Zhu.
Unbiased evaluation of large language models from a causal perspective.
In *Forty-second International Conference on Machine Learning*, 2025b.
URL [https://openreview.net/forum?id=ETIsFhZwhJ](https://openreview.net/forum?id=ETIsFhZwhJ).
- Chiang et al. (2024)
Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica.
Chatbot arena: An open platform for evaluating llms by human preference, 2024.
- DeepSeek-AI (2024)
DeepSeek-AI.
Deepseek-v3 technical report, 2024.
URL [https://arxiv.org/abs/2412.19437](https://arxiv.org/abs/2412.19437).
- DeepSeek-AI (2025)
DeepSeek-AI.
Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning, 2025.
URL [https://arxiv.org/abs/2501.12948](https://arxiv.org/abs/2501.12948).
- Dubois et al. (2024)
Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto.
Length-controlled alpacaeval: A simple way to debias automatic evaluators.
*arXiv preprint arXiv:2404.04475*, 2024.
- Gou et al. (2025)
Boyu Gou, Zanming Huang, Yuting Ning, Yu Gu, Michael Lin, Weijian Qi, Andrei Kopanev, Botao Yu, Bernal Jiménez Gutiérrez, Yiheng Shu, Chan Hee Song, Jiaman Wu, Shijie Chen, Hanane Nour Moussa, Tianshu Zhang, Jian Xie, Yifei Li, Tianci Xue, Zeyi Liao, Kai Zhang, Boyuan Zheng, Zhaowei Cai, Viktor Rozgic, Morteza Ziyadi, Huan Sun, and Yu Su.
Mind2web 2: Evaluating agentic search with agent-as-a-judge, 2025.
URL [https://arxiv.org/abs/2506.21506](https://arxiv.org/abs/2506.21506).
- Gu et al. (2025)
Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Wen Gao, Lionel Ni, and Jian Guo.
A survey on llm-as-a-judge, 2025.
URL [https://arxiv.org/abs/2411.15594](https://arxiv.org/abs/2411.15594).
- He et al. (2025)
Yancheng He, Shilong Li, Jiaheng Liu, Weixun Wang, Xingyuan Bu, Ge Zhang, Z.y. Peng, Zhaoxiang Zhang, Zhicheng Zheng, Wenbo Su, and Bo Zheng.
Can large language models detect errors in long chain-of-thought reasoning?
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 18468–18489, Vienna, Austria, July 2025. Association for Computational Linguistics.
ISBN 979-8-89176-251-0.
doi: 10.18653/v1/2025.acl-long.905.
URL [https://aclanthology.org/2025.acl-long.905/](https://aclanthology.org/2025.acl-long.905/).
- Hua et al. (2025)
Andong Hua, Kenan Tang, Chenhe Gu, Jindong Gu, Eric Wong, and Yao Qin.
Flaw or artifact? rethinking prompt sensitivity in evaluating llms, 2025.
URL [https://arxiv.org/abs/2509.01790](https://arxiv.org/abs/2509.01790).
- ISO/IEC/IEEE (2022)
ISO/IEC/IEEE.
Iso/iec/ieee international standard - software and systems engineering –software testing –part 1:general concepts.
*ISO/IEC/IEEE 29119-1:2022(E)*, pp. 1–60, 2022.
doi: 10.1109/IEEESTD.2022.9698145.
- Kwon et al. (2023)
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica.
Efficient memory management for large language model serving with pagedattention.
In *Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles*, 2023.
- Lambert et al. (2025)
Nathan Lambert, Valentina Pyatkin, Jacob Morrison, LJ Miranda, Bill Yuchen Lin, Khyathi Chandu, Nouha Dziri, Sachin Kumar, Tom Zick, Yejin Choi, Noah A. Smith, and Hannaneh Hajishirzi.
RewardBench: Evaluating reward models for language modeling.
In Luis Chiruzzo, Alan Ritter, and Lu Wang (eds.), *Findings of the Association for Computational Linguistics: NAACL 2025*, pp. 1755–1797, Albuquerque, New Mexico, April 2025. Association for Computational Linguistics.
ISBN 979-8-89176-195-7.
doi: 10.18653/v1/2025.findings-naacl.96.
URL [https://aclanthology.org/2025.findings-naacl.96/](https://aclanthology.org/2025.findings-naacl.96/).
- Lan et al. (2024)
Tian Lan, Wenwei Zhang, Chen Xu, Heyan Huang, Dahua Lin, Kai Chen, and Xian-Ling Mao.
Criticeval: Evaluating large-scale language model as critic.
In *The Thirty-eighth Annual Conference on Neural Information Processing Systems*, 2024.
URL [https://openreview.net/forum?id=ZsxZ65YqL1](https://openreview.net/forum?id=ZsxZ65YqL1).
- Li et al. (2024)
Chunyang Li, Hao Peng, Xiaozhi Wang, Yunjia Qi, Lei Hou, Bin Xu, and Juanzi Li.
MAVEN-FACT: A large-scale event factuality detection dataset.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Findings of the Association for Computational Linguistics: EMNLP 2024*, pp. 11140–11158, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-emnlp.651.
URL [https://aclanthology.org/2024.findings-emnlp.651/](https://aclanthology.org/2024.findings-emnlp.651/).
- Li et al. (2025a)
Chunyang Li, Weiqi Wang, Tianshi Zheng, and Yangqiu Song.
Patterns over principles: The fragility of inductive reasoning in LLMs under noisy observations.
In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), *Findings of the Association for Computational Linguistics: ACL 2025*, pp. 19608–19626, Vienna, Austria, July 2025a. Association for Computational Linguistics.
ISBN 979-8-89176-256-5.
doi: 10.18653/v1/2025.findings-acl.1006.
URL [https://aclanthology.org/2025.findings-acl.1006/](https://aclanthology.org/2025.findings-acl.1006/).
- Li et al. (2023a)
Jiatong Li, Rui Li, and Qi Liu.
Beyond static datasets: A deep interaction approach to llm evaluation, 2023a.
URL [https://arxiv.org/abs/2309.04369](https://arxiv.org/abs/2309.04369).
- Li et al. (2025b)
Ruizhe Li, Chiwei Zhu, Benfeng Xu, Xiaorui Wang, and Zhendong Mao.
Automated creativity evaluation for large language models: A reference-based approach, 2025b.
URL [https://arxiv.org/abs/2504.15784](https://arxiv.org/abs/2504.15784).
- Li et al. (2023b)
Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.
Alpacaeval: An automatic evaluator of instruction-following models.
[https://github.com/tatsu-lab/alpaca_eval](https://github.com/tatsu-lab/alpaca_eval), 5 2023b.
- Liang et al. (2024)
Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu.
Encouraging divergent thinking in large language models through multi-agent debate.
In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pp. 17889–17904, Miami, Florida, USA, November 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.emnlp-main.992.
URL [https://aclanthology.org/2024.emnlp-main.992/](https://aclanthology.org/2024.emnlp-main.992/).
- Lightman et al. (2024)
Hunter Lightman, Vineet Kosaraju, Yuri Burda, Harrison Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe.
Let’s verify step by step.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=v8L0pN6EOi](https://openreview.net/forum?id=v8L0pN6EOi).
- Luo et al. (2023)
Liangchen Luo, Zi Lin, Yinxiao Liu, Lei Shu, Yun Zhu, Jingbo Shang, and Lei Meng.
Critique ability of large language models, 2023.
URL [https://arxiv.org/abs/2310.04815](https://arxiv.org/abs/2310.04815).
- Lù et al. (2025)
Xing Han Lù, Amirhossein Kazemnejad, Nicholas Meade, Arkil Patel, Dongchan Shin, Alejandra Zambrano, Karolina Stańczak, Peter Shaw, Christopher J. Pal, and Siva Reddy.
Agentrewardbench: Evaluating automatic evaluations of web agent trajectories, 2025.
URL [https://arxiv.org/abs/2504.08942](https://arxiv.org/abs/2504.08942).
- Madaan et al. (2023)
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.
Self-refine: Iterative refinement with self-feedback.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
URL [https://openreview.net/forum?id=S37hOerQLB](https://openreview.net/forum?id=S37hOerQLB).
- OpenAI (2024)
OpenAI.
Hello gpt-4o, 2024.
URL [https://openai.com/index/hello-gpt-4o/](https://openai.com/index/hello-gpt-4o/).
- OpenAI (2025)
OpenAI.
Introducing gpt-4.1 in the api, 2025.
URL [https://openai.com/index/gpt-4-1/](https://openai.com/index/gpt-4-1/).
- Ouwehand et al. (2021)
Kim Ouwehand, Avalon van der Kroef, Jacqueline Wong, and Fred Paas.
Measuring cognitive load: Are there more valid alternatives to likert rating scales?
*Frontiers in Education*, Volume 6 - 2021, 2021.
ISSN 2504-284X.
doi: 10.3389/feduc.2021.702616.
URL [https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2021.702616](https://www.frontiersin.org/journals/education/articles/10.3389/feduc.2021.702616).
- Paglieri et al. (2025)
Davide Paglieri, Bartłomiej Cupiał, Samuel Coward, Ulyana Piterbarg, Maciej Wolczyk, Akbir Khan, Eduardo Pignatelli, Łukasz Kuciński, Lerrel Pinto, Rob Fergus, Jakob Nicolaus Foerster, Jack Parker-Holder, and Tim Rocktäschel.
BALROG: Benchmarking agentic LLM and VLM reasoning on games.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=fp6t3F669F](https://openreview.net/forum?id=fp6t3F669F).
- Pan et al. (2024)
Jiayi Pan, Yichi Zhang, Nicholas Tomlin, Yifei Zhou, Sergey Levine, and Alane Suhr.
Autonomous evaluation and refinement of digital agents.
In *First Conference on Language Modeling*, 2024.
URL [https://openreview.net/forum?id=NPAQ6FKSmK](https://openreview.net/forum?id=NPAQ6FKSmK).
- Pan et al. (2025)
Melissa Z Pan, Mert Cemri, Lakshya A Agrawal, Shuyi Yang, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya Parameswaran, Kannan Ramchandran, Dan Klein, Joseph E. Gonzalez, Matei Zaharia, and Ion Stoica.
Why do multiagent systems fail?
In *ICLR 2025 Workshop on Building Trust in Language Models and Applications*, 2025.
URL [https://openreview.net/forum?id=wM521FqPvI](https://openreview.net/forum?id=wM521FqPvI).
- Qin et al. (2024a)
Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, dahai li, Zhiyuan Liu, and Maosong Sun.
ToolLLM: Facilitating large language models to master 16000+ real-world APIs.
In *The Twelfth International Conference on Learning Representations*, 2024a.
URL [https://openreview.net/forum?id=dHng2O0Jjr](https://openreview.net/forum?id=dHng2O0Jjr).
- Qin et al. (2024b)
Zhen Qin, Junru Wu, Jiaming Shen, Tianqi Liu, and Xuanhui Wang.
LAMPO: Large language models as preference machines for few-shot ordinal classification.
In *First Conference on Language Modeling*, 2024b.
URL [https://openreview.net/forum?id=ig6NI9oPhD](https://openreview.net/forum?id=ig6NI9oPhD).
- Saha et al. (2025)
Swarnadeep Saha, Xian Li, Marjan Ghazvininejad, Jason E Weston, and Tianlu Wang.
Learning to plan & reason for evaluation with thinking-LLM-as-a-judge.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=PNRznmmWP7](https://openreview.net/forum?id=PNRznmmWP7).
- Seed (2025)
ByteDance Seed.
Ui-tars-1.5.
[https://seed-tars.com/1.5](https://seed-tars.com/1.5), 2025.
- Shafayat et al. (2025)
Sheikh Shafayat, Fahim Tajwar, Ruslan Salakhutdinov, Jeff Schneider, and Andrea Zanette.
Can large reasoning models self-train?, 2025.
URL [https://arxiv.org/abs/2505.21444](https://arxiv.org/abs/2505.21444).
- Starace et al. (2025)
Giulio Starace, Oliver Jaffe, Dane Sherburn, James Aung, Jun Shern Chan, Leon Maksin, Rachel Dias, Evan Mays, Benjamin Kinsella, Wyatt Thompson, Johannes Heidecke, Amelia Glaese, and Tejal Patwardhan.
Paperbench: Evaluating AI’s ability to replicate AI research.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=xF5PuTLPbn](https://openreview.net/forum?id=xF5PuTLPbn).
- Tan et al. (2025)
Sijun Tan, Siyuan Zhuang, Kyle Montgomery, William Yuan Tang, Alejandro Cuadron, Chenguang Wang, Raluca Popa, and Ion Stoica.
Judgebench: A benchmark for evaluating LLM-based judges.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=G0dksFayVq](https://openreview.net/forum?id=G0dksFayVq).
- Team (2025a)
Gemini Team.
Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities, 2025a.
URL [https://arxiv.org/abs/2507.06261](https://arxiv.org/abs/2507.06261).
- Team (2025b)
GLM 4.5 Team.
Glm-4.5: Agentic, reasoning, and coding (arc) foundation models, 2025b.
URL [https://arxiv.org/abs/2508.06471](https://arxiv.org/abs/2508.06471).
- Team (2025c)
Kimi Team.
Kimi k2: Open agentic intelligence, 2025c.
URL [https://arxiv.org/abs/2507.20534](https://arxiv.org/abs/2507.20534).
- Team (2025d)
Qwen Team.
Qwen2.5-vl technical report, 2025d.
URL [https://arxiv.org/abs/2502.13923](https://arxiv.org/abs/2502.13923).
- Team (2025e)
Qwen Team.
Qwen3 technical report, 2025e.
URL [https://arxiv.org/abs/2505.09388](https://arxiv.org/abs/2505.09388).
- Vichare et al. (2025)
Aryan Vichare, Anastasios N. Angelopoulos, Wei-Lin Chiang, Kelly Tang, and Luca Manolache.
Webdev arena: A live llm leaderboard for web app development, 2025.
- Wang et al. (2025)
Jian Wang, Xiaofei Xie, Qiang Hu, Shangqing Liu, and Yi Li.
Do code semantics help? a comprehensive study on execution trace-based information for code large language models, 2025.
URL [https://arxiv.org/abs/2509.11686](https://arxiv.org/abs/2509.11686).
- Wang et al. (2024)
Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Lingpeng Kong, Qi Liu, Tianyu Liu, and Zhifang Sui.
Large language models are not fair evaluators.
In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pp. 9440–9450, Bangkok, Thailand, August 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.acl-long.511.
URL [https://aclanthology.org/2024.acl-long.511/](https://aclanthology.org/2024.acl-long.511/).
- Xie et al. (2025)
Qiujie Xie, Qingqiu Li, Zhuohao Yu, Yuejie Zhang, Yue Zhang, and Linyi Yang.
An empirical analysis of uncertainty in large language model evaluations.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=J4xLuCt2kg](https://openreview.net/forum?id=J4xLuCt2kg).
- Xu et al. (2025)
Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, and Tao Yu.
Agenttrek: Agent trajectory synthesis via guiding replay with web tutorials.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=EEgYUccwsV](https://openreview.net/forum?id=EEgYUccwsV).
- Xue et al. (2025)
Tianci Xue, Weijian Qi, Tianneng Shi, Chan Hee Song, Boyu Gou, Dawn Song, Huan Sun, and Yu Su.
An illusion of progress? assessing the current state of web agents, 2025.
URL [https://arxiv.org/abs/2504.01382](https://arxiv.org/abs/2504.01382).
- Yao et al. (2023)
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao.
React: Synergizing reasoning and acting in language models.
In *The Eleventh International Conference on Learning Representations*, 2023.
URL [https://openreview.net/forum?id=WE_vluYUL-X](https://openreview.net/forum?id=WE_vluYUL-X).
- Ye et al. (2025)
Jiayi Ye, Yanbo Wang, Yue Huang, Dongping Chen, Qihui Zhang, Nuno Moniz, Tian Gao, Werner Geyer, Chao Huang, Pin-Yu Chen, Nitesh V Chawla, and Xiangliang Zhang.
Justice or prejudice? quantifying biases in LLM-as-a-judge.
In *The Thirteenth International Conference on Learning Representations*, 2025.
URL [https://openreview.net/forum?id=3GTtZFiajM](https://openreview.net/forum?id=3GTtZFiajM).
- Yu et al. (2025)
Jiachen Yu, Shaoning Sun, Xiaohui Hu, Jiaxu Yan, Kaidong Yu, and Xuelong Li.
Improve llm-as-a-judge ability as a general ability, 2025.
URL [https://arxiv.org/abs/2502.11689](https://arxiv.org/abs/2502.11689).
- Yuan et al. (2025)
Weizhe Yuan, Richard Yuanzhe Pang, Kyunghyun Cho, Xian Li, Sainbayar Sukhbaatar, Jing Xu, and Jason Weston.
Self-rewarding language models, 2025.
URL [https://arxiv.org/abs/2401.10020](https://arxiv.org/abs/2401.10020).
- Zeng et al. (2024)
Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen.
Evaluating large language models at evaluating instruction following.
In *The Twelfth International Conference on Learning Representations*, 2024.
URL [https://openreview.net/forum?id=tr0KidwPLc](https://openreview.net/forum?id=tr0KidwPLc).
- Zhang et al. (2025a)
Chenchen Zhang, Yuhang Li, Can Xu, Jiaheng Liu, Ao Liu, Shihui Hu, Dengpeng Wu, Guanhua Huang, Kejiao Li, Qi Yi, Ruibin Xiong, Haotian Zhu, Yuanxing Zhang, Yuhao Jiang, Yue Zhang, Zenan Xu, Bohui Zhai, Guoxiang He, Hebin Li, Jie Zhao, Le Zhang, Lingyun Tan, Pengyu Guo, Xianshu Pang, Yang Ruan, Zhifeng Zhang, Zhonghu Wang, Ziyan Xu, Zuopu Yin, Wiggin Zhou, Chayse Zhou, and Fengzong Lian.
Artifactsbench: Bridging the visual-interactive gap in llm code generation evaluation, 2025a.
URL [https://arxiv.org/abs/2507.04952](https://arxiv.org/abs/2507.04952).
- Zhang et al. (2025b)
Guanhua Zhang, Ricardo Dominguez-Olmedo, and Moritz Hardt.
Train-before-test harmonizes language model rankings, 2025b.
URL [https://arxiv.org/abs/2507.05195](https://arxiv.org/abs/2507.05195).
- Zhang et al. (2023)
Xinghua Zhang, Bowen Yu, Haiyang Yu, Yangyu Lv, Tingwen Liu, Fei Huang, Hongbo Xu, and Yongbin Li.
Wider and deeper llm networks are fairer llm evaluators, 2023.
URL [https://arxiv.org/abs/2308.01862](https://arxiv.org/abs/2308.01862).
- Zheng et al. (2023)
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.
Judging LLM-as-a-judge with MT-bench and chatbot arena.
In *Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track*, 2023.
URL [https://openreview.net/forum?id=uccHPGDlao](https://openreview.net/forum?id=uccHPGDlao).
- Zheng et al. (2024)
Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng.
Sglang: Efficient execution of structured language model programs, 2024.
URL [https://arxiv.org/abs/2312.07104](https://arxiv.org/abs/2312.07104).
- Zhong et al. (2024)
Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan.
AGIEval: A human-centric benchmark for evaluating foundation models.
In Kevin Duh, Helena Gomez, and Steven Bethard (eds.), *Findings of the Association for Computational Linguistics: NAACL 2024*, pp. 2299–2314, Mexico City, Mexico, June 2024. Association for Computational Linguistics.
doi: 10.18653/v1/2024.findings-naacl.149.
URL [https://aclanthology.org/2024.findings-naacl.149/](https://aclanthology.org/2024.findings-naacl.149/).
- Zhuge et al. (2025)
Mingchen Zhuge, Changsheng Zhao, Dylan R. Ashley, Wenyi Wang, Dmitrii Khizbullin, Yunyang Xiong, Zechun Liu, Ernie Chang, Raghuraman Krishnamoorthi, Yuandong Tian, Yangyang Shi, Vikas Chandra, and Jürgen Schmidhuber.
Agent-as-a-judge: Evaluate agents with agents.
In *Forty-second International Conference on Machine Learning*, 2025.
URL [https://openreview.net/forum?id=Nn9POI9Ekt](https://openreview.net/forum?id=Nn9POI9Ekt).

## Appendix A Details in WebDevJudge Construction

### A.1 Data Collection and Filtering Pipeline

Our data collection process commences with the webdev-arena-preference-10k dataset (Vichare et al., 2025; Chiang et al., 2024), which contains 10,501 user queries, each paired with two web implementations and a user-provided preference label. To ensure the quality and suitability of this data for our benchmark, we implemented a rigorous two-stage filtering pipeline.

#### Query-based Filtering

The raw queries were first processed to address quality issues. We began by removing all verbatim duplicate queries to eliminate redundancy. Subsequently, we employed gemini-2.5-pro to screen for and exclude any queries containing harmful, offensive, or nonsensical content, based on a predefined set of safety and clarity instructions. We then exclude queries with minimal interaction requirements (score $>$ 8, max = 10) and queries with unclear intentions (score $<$ 3, max = 5). The prompts used for the safety filter, interaction filter, and intention filter are presented below.

#### Environment-based Filtering

Following the query-based filtering, we proceeded to validate the web implementations. We established a standardized Next.js environment for deployment. To manage dependencies, we identified and included the most common packages, thereby excluding implementations that required niche or incompatible libraries. Each successfully deployed implementation was then verified via a test request; those returning a non-200 status code were discarded as buggy. Finally, to handle instances with intrinsic runtime errors not captured by status codes (e.g., rendering a blank page), we captured an initial screenshot of each webpage. A multimodal model, Qwen2.5-VL-72B, was utilized to visually inspect these screenshots and filter out any pages exhibiting rendering failures.

This comprehensive filtering process yielded a final set of 1,713 high-quality instances for subsequent annotation. While the original dataset includes user-provided preference labels, we identified several factors that render them unreliable for rigorous evaluation. Primarily, the labels are susceptible to a high degree of subjectivity and variance in evaluation criteria among individual users. The single-pass nature of the crowdsourced annotations also lacks the verification necessary for robust benchmark data. Furthermore, the dataset includes a “tie (bothbad)” category, which introduces ambiguity. The subjective definition of “bad” can lead to inconsistent labeling, potentially masking instances where one implementation, though imperfect, is demonstrably superior to the other. To illustrate these inconsistencies, we provide examples of problematic cases from the original dataset in Table [7](https://arxiv.org/html/2510.18560v1#A1.T7).

Table: Table 7: Examples of problematic cases. In each example, the upper figure is referred to as model_a, while the lower one is model_b.

### A.2 Rubric Annotation Pipeline

The annotation was conducted by two of the authors, all of whom possess strong backgrounds in computer science and software engineering. Since we have two annotators, the agreement rate is calculated as the proportion of instances where their annotations are the same. The annotation requires annotators to interact with the deployed web implementations and provide a preference label via pairwise comparison. However, this direct approach yields low inter-annotator agreement, underscoring the high degree of subjectivity inherent in the task and the need for a standardized evaluation framework.

To address this, we introduced a rubric-guided annotation process. Manually creating detailed rubrics for each query is not only prohibitively time-consuming—with an estimated 20 minutes per rubric, excluding research time—but also intellectually demanding. It requires extensive background knowledge, including familiarity with established UI/UX design patterns for similar applications and, in many cases, specialized domain knowledge for tasks like scientific simulations. Recognizing these challenges, we leveraged a powerful large language model, gemini-2.5-pro, for automated generation. Based on each user query, the model produced a structured rubric tree organized along three core dimensions critical to web development quality:

- •
Intention: The core requirements of the user query.
- •
Static Quality: The assessment of static elements, including UI layout and UX design.
- •
Dynamic Behavior: The evaluation of interactive features.

The prompt used to generate these rubric trees is provided below:

Upon manual review, the LLM-generated rubrics exhibited both strengths and weaknesses. For queries that referenced real-world applications, the rubrics were often of high quality, sometimes surpassing human-authored versions in detail. However, for vague or overly general queries, which are common on crowdsourcing platforms, the generated rubrics sometimes included criteria that were either too specific or too broad, reflecting the ambiguity of the original request. We also provide the statistics of the rubric trees; the details are shown in Table [8](https://arxiv.org/html/2510.18560v1#A1.T8). From the statistics, especially the average number of leaf nodes, we can see that the LLM-generated rubrics cover a wide range of criteria, which is a good sign for the evaluation of web development tasks.

**Table 8: Statistics of the generated rubric trees.**
| Metric | Intention | Static | Dynamic | Whole |
| --- | --- | --- | --- | --- |
| Average height | 2.0 | 3.9 | 3.3 | 5.0 |
| Average number of leaf nodes | 3.6 | 15.6 | 10.6 | 29.9 |

To ensure consistent application of these rubrics, we established a set of clear annotation guidelines for the expert annotators, as detailed in the following:

This structured pipeline, combining LLM-generated rubrics with clear human oversight and guidelines, proved highly effective. The final inter-annotator agreement rate reached 89.7%, confirming the consistency and objectivity of our annotation process.

### A.3 Data Statistics

In this section, we describe the categorization pipeline for WebDevJudge and present representative examples for each subcategory. We adapt topics from the original dataset (Vichare et al., 2025) to serve as our subcategories. Given that WebDevJudge contains only 654 instances, we did not use a topic modeling model for clustering. Instead, we generated subcategories by providing the topic name, a detailed description, and the user query to GPT-4o. These subcategories were then manually reviewed and consolidated into three main categories.

## Appendix B Experimental Details

In this section, we show the details of the implementation of our experiments. Including the models and hyperparameters, the details of evaluation protocols and metrics, and the agentic workflow implementation.

### B.1 Models And Hyperparameters

Our experiments utilize a combination of open-source and commercial models. Open-source models were deployed using vLLM (Kwon et al., 2023) and SGLang (Zheng et al., 2024). Commercial models were accessed via their respective APIs, including Azure(^1^11https://learn.microsoft.com/en-us/azure/cognitive-services/openai/reference) and Vertex AI(^2^22https://cloud.google.com/vertex-ai/generative-ai/docs/learn/overview). To ensure reproducibility, we set the temperature to 0.0 for all models, with the exception of claude-3.7-sonnet and claude-4-sonnet, for which the temperature was set to 1.0 to enable their “thinking” mode. For brevity, the models referred to as DeepSeek-V3 and DeepSeek-R1 in the main paper correspond to DeepSeek-V3-0324 and DeepSeek-R1-0528, respectively.

### B.2 Evaluation Protocols and Metrics

#### Likert Scale

Motivated by the work of Bian et al. (2025), we designed a multi-level Likert scale for evaluating web development tasks. The scale is based on the international standard for software quality assessment (ISO/IEC/IEEE, 2022) and has been specifically adapted for our dataset. The details of the Likert scale are shown below:

In selecting these dimensions, we considered the typical workflow of human evaluators in web testing. Given that the tasks in our dataset are primarily front-end focused, and acknowledging the current limitations of generative models in producing efficient, full-stack solutions, we have concentrated on fundamental aspects of web development, omitting higher-level criteria such as backend performance and efficiency.

For single-implementation evaluation, the model assigns a score from 1 (lowest) to 5 (highest) to each sub-dimension. The final score is the sum of all sub-dimension scores. For pairwise comparison, both implementations are presented to the model simultaneously, which allows it to assign scores based on their relative merits. The final preference is determined by the score difference between the two implementations. A preference is declared for the higher-scoring implementation if the score difference exceeds a threshold of 2; otherwise, the outcome is considered a tie.

#### Rubric

For rubric-based evaluation, the process is guided by the LLM-generated rubric tree. The model is provided with the rubric, the user query, and the web implementation(s). To mitigate the inherent ambiguity in rubric-based assessments, we instruct the model to recognize functional equivalence. For instance, a requirement for a heading is considered met if a heading element is present, even if its text or styling differs from a literal interpretation of the rubric.

In the single-answer grading setting, the model assigns a binary label (pass or fail) to each leaf node of the rubric. We then compute a pass rate of the leaf node for each of the three primary dimensions: intention, static, and dynamic. The final score is the weighted sum of these pass rates. For pairwise comparison, the model evaluates the two implementations against each leaf node, determining which is superior or if they are tied. This yields a win rate for each implementation of the leaf nodes across the three dimensions. A final score for each is calculated as a weighted sum of these win rates. Preference is awarded to the implementation with the higher final score. In our implementation, all dimension weights are set to 1.

#### Direct

For direct evaluation, instead of providing any criteria, we instruct the model to directly output its preference (i.e., Wa, Wb, or tie). The prompt we used is shown below.

### B.3 Agentic Workflow Implementation

This section details our agentic workflow implementation. To ensure comparability with our other evaluation methods, the workflow’s planner adopts the LLM-generated rubric tree to guide the executor’s verification actions. The executor is UI-TARS-1.5 (Seed, 2025), one of the state-of-the-art end-to-end GUI agents operating on a ReAct-style paradigm (Yao et al., 2023), where it first generates a thought and then a corresponding action. These actions are converted into executable pyautogui(^3^33https://pyautogui.readthedocs.io/en/latest/) code. We utilize the official UI-TARS-1.5 action space and have designed specific prompting strategies for each rubric dimension. For the Static dimension, the rubric tree is converted into a list of elements; the agent navigates the page to find these elements and returns a list of those present. For the Dynamic and Intention dimensions, each rubric leaf node becomes a task for the agent to complete. Upon reaching the maximum number of steps, as detailed in Table [9](https://arxiv.org/html/2510.18560v1#A2.T9), the agent provides a conclusion on the task’s outcome. Finally, the summarizer calculates a pass rate for each dimension based on the agent’s findings, and the final score is the weighted sum of these pass rates, mirroring the single-implementation rubric evaluation.

**Table 9: Details of the agent settings.**
| Dimension | Input Format | Output Format | Max Step |
| --- | --- | --- | --- |
| Static | List of elements to check | List of found elements | 6 |
| Dynamic | Task description | Task feasibility and conclusion | 5 for basic, 15 for complex |
| Intention | Intention description | Feasibility conclusion | 15 |

Table: Table 10: Statistics of the WebDevJudge-Unit dataset.

## Appendix C WebDevJudge-Unit Dataset

In this section, we introduce WebDevJudge-Unit, a task-level dataset created to assess the capability of evaluators to verify task feasibility.

### C.1 Dataset Construction

To construct the WebDevJudge-Unit dataset, we began by randomly sampling 105 queries from WebDevJudge. For each query, we prompted gemini-2.5-pro to generate up to five corresponding verification tasks, each with an expected result. We then generated the necessary HTML code for each query to facilitate easy deployment. Following deployment, each task was meticulously annotated for feasibility (true/false) by interacting with the live web application. For tasks identified as infeasible, we further annotated the specific error type and provided a detailed reason for the failure.

### C.2 Dataset Statistics and Examples

The resulting dataset comprises 502 tasks derived from 105 unique queries. The feasibility labels are distributed as follows: 279 tasks are marked as feasible, and 223 as infeasible. Table [10](https://arxiv.org/html/2510.18560v1#A2.T10) presents a detailed breakdown of the error types for the infeasible tasks, along with illustrative examples for each category.

Figure: Figure 5: The consistency between model predictions under pairwise comparison.
Refer to caption: x5.png

Figure: Figure 6: The consistency between model prediction under single answer grading.
Refer to caption: x6.png

## Appendix D Additional Results

This section provides a more detailed analysis of LLM-as-a-judge performance on web development tasks, supported by supplementary experimental results.

### D.1 Prediction Distribution Consistency

To understand the nature of model failures, we analyze the consistency of predictions across different evaluators. This investigation seeks to determine whether different models tend to err on the same set of instances—suggesting certain examples are inherently challenging—or if their errors are largely independent and model-specific. The results of this label consistency analysis are presented in Figure [5](https://arxiv.org/html/2510.18560v1#A3.F5) and Figure [6](https://arxiv.org/html/2510.18560v1#A3.F6).

Our analysis of prediction consistency across different models reveals a significant disparity between the two evaluation paradigms. As illustrated in Figure [5](https://arxiv.org/html/2510.18560v1#A3.F5) and Figure [6](https://arxiv.org/html/2510.18560v1#A3.F6), the inter-model agreement under the pairwise comparison paradigm is substantially higher than that under single-answer grading. In pairwise comparisons, the consistency rates between different evaluators generally exceed 75%, with many model pairs even surpassing 80%. In stark contrast, under single-answer grading, inter-model consistency drops significantly, typically hovering between 50% and 65%.

This discrepancy corroborates our core finding from the main text: pairwise comparison is a more stable and reliable paradigm for complex, open-ended tasks like web development. Relative judgment in pairwise comparison constrains the evaluation scope, compelling the model to focus on discriminative features between two candidates, which is a cognitively less demanding task than absolute scoring. Conversely, single-answer grading requires the model to possess a well-calibrated and consistent internal standard of quality, a standard that varies greatly across different models. Consequently, the single-answer grading paradigm exposes the current models’ deficiencies in calibration, leading to inconsistent and less reliable evaluation outcomes.

Furthermore, the high consistency in pairwise comparison suggests that while different models may lack a shared understanding of ’absolute quality,’ their internal mechanisms for making ‘relative preference’ judgments are more aligned. This reinforces our view that leveraging relative judgments is a more robust approach for automated evaluation in domains characterized by nuance and ambiguity, where clear-cut right or wrong answers are scarce.

### D.2 Mitigating the Impact of Positional Bias

To mitigate the impact of positional bias, we employ a widely-used debiasing technique from prior works (Zeng et al., 2024; Tan et al., 2025). This method involves evaluating each pair of implementations twice, with their positions swapped in the second evaluation. A preference is considered final only if the model’s choice remains consistent across both orderings. If the choice is inconsistent, the outcome is recorded as a tie. We use this approach to investigate whether mitigating positional bias improves overall model performance. The results are shown in Table [11](https://arxiv.org/html/2510.18560v1#A4.T11).

**Table 11: Agreement rate (%) with and without mitigating the positional bias.**
| Model | w/ mitigating | w/o mitigating |  |  |
| --- | --- | --- | --- | --- |
| Direct | Likert | Direct | Likert |  |
| Claude-4-sonnet | 65.0 | 65.1 | 66.1 | 66.1 |
| GPT-4.1 | 64.8 | 65.1 | 64.2 | 66.1 |
| DeepSeekV3-0324 | 60.8 | 62.2 | 62.5 | 63.6 |

A natural question arises as to why we did not employ the swap-based debiasing technique in our main experiments. Our decision to rely on single-pass evaluation stems from two primary considerations. First, our core objective is to characterize the raw, unfiltered behavior of LLMs-as-judges to understand their inherent biases. Applying debiasing techniques from the outset would mask or average out these effects, obscuring the very phenomena we seek to analyze. The results in Table [11](https://arxiv.org/html/2510.18560v1#A4.T11) indicate that the overall performance difference before and after debiasing is not substantial in our setup. This finding reinforces the importance of studying the biases directly, as their presence is not always evident from aggregate performance metrics alone. Second, the single-pass evaluation mirrors many practical application scenarios where, for reasons of cost and efficiency, models are queried only once per pair. Therefore, our main experimental results offer a more realistic and cautionary benchmark for practitioners, highlighting the potential pitfalls of naively deploying these models without safeguards. Our methodology thus deliberately separates the diagnosis of inherent model behaviors from the evaluation of mitigation strategies.

### D.3 Evaluation Results on Non-Tie Cases

Given the brittleness of language models when handling ambiguous tie conditions, we also present results for non-tie cases, where a clear preference was established. As presented in Table [12](https://arxiv.org/html/2510.18560v1#A4.T12), excluding tie cases results in a substantial improvement in agreement rates for all evaluators, indicating that ambiguous comparisons are a primary source of error. More importantly, these results reinforce the core conclusions drawn from our main experiments. The pairwise comparison paradigm consistently and significantly outperforms single-answer grading, highlighting its robustness for capturing relative quality. Furthermore, the patterns of different guidance mechanisms remain consistent: in the pairwise setting, both Likert scale and Rubric-based guidance yield comparable performance, suggesting that the relative judgment itself is the dominant factor. In contrast, for single-answer grading, the Rubric-based approach is demonstrably superior to the Likert scale. This reinforces our finding that structured, binary assessments provide a more reliable signal for absolute evaluation than multi-point scales, which require a level of calibration that models currently lack.

Table: Table 12: Agreement Rate (%) (without tie) of different evaluators under different evaluation paradigms.

## Appendix E The use of large language models

We use a large language model (LLM) as a general-purpose writing assistant. The primary use of the LLM is to refine and improve the clarity and flow of the text. Specifically, we provided the LLM with our pre-existing draft text and a clear outline of our ideas. We instruct the model to adhere strictly to the provided content, without adding or removing any information. The LLM’s role is limited to enhancing the logical coherence and fluency of the language, ensuring the final text is more polished and easier to read.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 4
presentation: 4
questions: - Can you please answer my main issue? Could you also highlight inter-model agreement for all the models in the paper?
soundness: 2
strengths: Given the widespread adoption of LLM as a judge, the need for better "judgements" of llm-as-a-judge has grown as well. This paper adeptly addresses that. The paper text is for the most part clear and easy to follow. The benchmark is also original as to my knowledge there is not a specific benchmark for web judgements.
summary: This paper introduces WebDevJudge, a meta-evaluation benchmark for assessing the reliability of the LLM-as-a-judge paradigm in web dev tasks. The benchmark builds upon webdev-arena-preference and covers both static evaluation (code + screenshots) and dynamic evaluation (live interactive environment). It introduces the notion of a rubric and shows how the rubric leads to higher inter-annotator agreement for both humans and LLMs
weaknesses: - The main issue I see is that the results for all models seem relatively similar (~50s to 60s). There's not a lot of variation in terms of performance and there's no statistical tests to indicate that these values are actually meaningful. I have a strong suspicion that the reason these numbers are so similar is in fact because of the rubric. As pointed out in the paper, it increases inter-annotator agreement, but my guess is that it likely increases agreement *overall* as well and doesn't accurately test a model's judgement capability (especially since this rubric will not be used in-the-wild). As a result, I don't quite buy that this benchmark is meaningfully capturing llm-as-a-judge performance.
- A smaller point is that the tables and figures are difficult to understand in isolation. For example, figure 1 is not parseable as is and requires significant context outside of the figure. Same with Table 1. Perhaps more information in the captions could help here.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 4
questions: 1. Most of webdev-arena-preference-10k is filtered out. Which of the three criteria filtered out the most of them (or was it the environment filter)? I would think we'd want to capture as many of these examples as possible including the ambigious ones because that's how we expect users to query the model. I am wondering if this filtering biased the findings in someway where the dataset is primiraly focused on a subset of "webdev" where evaluators are exhibiting interesting failure cases.
2. I know this is not standard, but given your hierarchical rubric, I am wondering if there was any thought in trying more than two examples for pair-wise evaluation?  I believe there's work that shows more than two examples does not improve performance for direct comparisons, but with your hierarchical rubric and for failure cases like functional equivalence, I wonder if having more examples could help (this may be a costly experiment, though, so totally feel free to ignore this).
3. How often is the lack of agreement due to functional equivalence? Is there a way to bin the lack of agreement by the failure modes presented in the paper?  I am wondering if 1, there's a way to establish what seems the most important to improve to enhance these models but also 2 to see if these models are just suffering from "ambiguity". The example in Figure 4 feels potentially ambigious and the LLM evaluator could be correct, for example "though the exact text of symbol differs" where the expected text is "demonstration" and the seen text is "presentation", it doesn't feel as clear-cut to me that these two words are functionally equivalent in all contexts and maybe if we asked more humans we'd find that there is actually some disagreement here.
soundness: 4
strengths: - **A challenging benchmark that evaluators can be tested on**, the highest performing evaluator achieves only 66% agreement with humans, indicating a large gap for improvement.
- **Clear insight into challenges in current evaluators**, the analysis that follows the main results from the benchmark highlights several key factors causing models to fail to be more effective evaluators. Other researchers can easily identify and work on improving these common failure modes.
- **Extremely well written, clear experimental design, good reproducibility**
summary: This work introduces WebDevJudge, a meta-evaluation benchmark meant to test how (M)LLMs can evaluate web development (both using static evaluators or interactive evaluators such as agents).  The authors create WebDevJudge by collecting and filtering example queries from webdev-arena-preference-10k and creating a hierarchical rubric for evaluating each instance. The rubric is meant to represent an effective evaluation strategy and help the evaluator focus on binary features for making its judgement, the authors show that this method yields stronger annotator agreement between humans. Although these rubric trees aid human annotation, the authors find that it has marginal effects on evaluators. The key results in the paper find that LLM evaluators (even very large ones such as R1) often have rather low agreement with human annotators when evaluating web development (in many settings including agentic evaluators, pair-wise evaluation, etc.)

The authors continue to analyze the failure cases for these evaluators by looking at how important multiple modalities are for evaluation (finding code to be the most important). The authors look at position biases for pair-wise evaluation, finding it to be prevalent. Additional errors within these evaluators include functional equivalence (knowing when something is close/good enough) and that agentic systems often fail due to their own operational reliability (meaning static evaluators and agentic verifiers both have shortcomings). Overall this is an excellent benchmark and analysis paper highlighting the core limitations within LLM-as-a-judge evaluators is not in how the task is set up nor the methodology for performing the evalation (pairwise or direct, single model or agentic) but lies within the core capabilities of the model. This work highlights future directions for improving LLM evaluators and offers a dataset for researchers to experiment on.
weaknesses: - **Low number of human annotators**: Only two annotators were used; their agreement is high, but I do wonder about some of the examples where the models are failing to agree with humans. If you gave those examples to more annotators, maybe we would find that they are actually somewhat ambigious.
- **Lack of concrete examples**: Some of these failure modes are quite high-level, like "operational reliability." I didn't see any example outputs from the models, but placing a few in the appendix may go a long way in helping other researchers understand more concretely what the errors are. Another suggestion would be to release the evaluation traces from your models. Personally, I've found that to be incredibly helpful, as it avoids having other researchers rerun sometimes costly experiments.
- I would like to see an actual rubric tree somewhere. Maybe I missed it (or it is in the appendix somewhere), but it would help me fully understand what the leaf nodes are and how they work.


The rest of my "weaknesses" are really questions for the authors. I overall think this is a strong submission.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Can you share the few-shot LLM generation setup? What prompts were used (just example in appendix).
2. How did you go from 1713 high-quality instances to 654 instances after rubric?
3. “With explicit instructions to ignore position bias and remain objective”. How is this done? Why don’t we randomly swap position to avoid position bias? And “remain objective”, how does that work? What is the impact? You analyze this later, but what happens if you would do the evaluation with both positions (we can’t do this with humans since they will remember, but we can do this with LLMs).
4. “Imposing rigid structured metrics might constrain the models’ inherent reasoning processes” — Did you use constrained decoding here? Or is it still free-form output (but with the mentioning of the rubric)?
soundness: 3
strengths: 1. A good analysis on the new benchmark that is introduced, in a space where benchmarks (and especially meta benchmarks) are much needed.
2. Draws conclusions that seem sound, and are useful in both designing solutions as designing other benchmarks.
summary: Introduces WebDevJudge. A new meta benchmark to evaluate LLM-as-a-judge performance in web development. They show existing LLM judges underperform versus human experts in their setup. The benchmark is not solved. The benchmark supports both static text and real-time interaction setups.
weaknesses: 1. Details about the agentic setup are lacking. They have some details in the appendix, but no analysis of where the agentic setup fails. We know agentic setups with interaction with GUIs are still not great (and we know that they often are better with e.g. selecting using elements vs coordinates), but more details on how useful the agentic side is for the final conclusion would be good for the paper.
2. In the appendix I see examples, but none have images. If this is a known limitation, can you highlight it? E.g. some websites are designed around images and will look worse if no images are used.
3. Could propose (even in a weak-form) potential solutions to the current short-comings.

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: 1. Do you think the rubric tree used during human evals introduces any undesirable biases that could cause agreement to artificially go up?
soundness: 3
strengths: - The paper focuses on an important and understudied problem, that of establishing the reliability of LLM judges in web dev scenarios.
- The idea of using a query-grounded rubric tree for annotation is a novel and strong methodological contribution for collecting complex annotations. The high inter-annotator agreement (89.7%) validates this rubric-based approach and is especially motivating compared to w/o rubric.
- The evaluation of existing evaluators is comprehensive, testing LLMs, MLLMs, and agentic workflows across static and interactive paradigms.
- The error analysis is well done. It identifies precise failures like the inability to recognize "functional equivalence" and provides a clear -diagnostic analysis using the WebDevJudge-Unit dataset.
summary: The paper introduces WebDevJudge, a benchmark to evaluate LLM judges in the web development domain. The benchmark consists of paired web implementations with human preference labels. The paper uses this benchmark to evaluate LLMs, MLLMs, and agentic workflows. The findings show a significant performance gap between the best LLM judges and human experts. The paper also identifies failure modes in LLM judges, such as a positional bias and inability to recognize "functional equivalence" between implementations.
weaknesses: - The benchmark is fairly small (654 examples), which may limit the generalizability of the strong claims.
- Some of the claims about the benefits of agentic evaluators might need caveats since the main result is based on a single agent (UI-TARS-1.5). 
- The paper identifies a precision/recall trade-off between static LLMs and interactive agents and suggests an "ideal evaluator" would combine them, but doesn’t actually run this experiment.


Decision: Accept (Oral)