LiveWeb-IE: A Benchmark For Online Web Information Extraction
Authors: Seungbin Yang
Jihwan Kim
Jaemin Choi
Dongjin Kim
Soyoung Yang
ChaeHun Park
Jaegul Choo
Venue: ICLR 2026 Poster (2026)
Keywords: web information extraction
web scraping
Primary area: datasets and benchmarks
TL;DR: We introduce a benchmark for evaluating information extraction on live websites and a visual grounding scraper method.

Abstract: Web information extraction (WIE) is the task of automatically extracting data from web pages, offering high utility for various applications.
The evaluation of WIE systems has traditionally relied on benchmarks built from HTML snapshots captured at a single point in time.
However, this offline evaluation paradigm fails to account for the temporally evolving nature of the web; consequently, performance on these static benchmarks often fails to generalize to dynamic real-world scenarios.
To bridge this gap, we introduce LiveWeb-IE, a new benchmark designed for evaluating WIE systems directly against live websites.
Based on trusted and permission-granted websites, we curate natural language queries that require information extraction of various data categories, such as text, images, and hyperlinks.
We further design these queries to represent four levels of complexity, based on the number and cardinality of attributes to be extracted, enabling a granular assessment of WIE systems.
In addition, we propose Visual Grounding Scraper (VGS), a novel multi-stage agentic framework that mimics human cognitive processes by visually narrowing down web page content to extract desired information. 
Extensive experiments across diverse backbone models demonstrate the effectiveness and robustness of VGS.
We believe that this study lays the foundation for developing practical and robust WIE systems.

Published as a conference paper at ICLR 2026
LIVEWEB-IE: A BENCHMARK FOR ONLINE WEB IN- FORMATION EXTRACTION
Seungbin Yang o0-
Jihwan Kim 2- Jaemin Choi
Dongjin Kim
Soyoung Yang 9
ChaeHun Park 9 Jaegul Choo
KAIST AI
Letsur School of Computer Science and Engineering, Chung-Ang University
{sby99, jihvvan.king dj kim, yang, jchoo (kaoot@kkistt.. wakaasssac...... act.aac.k taeminldccau.ae kr, ckrcogns22228grm com
ABSTRACT
Web information extraction (WIE) is the task of automatically extracting data from web pages, offering high utility for various applications. The evaluation WTE systems has traditionally relied on benchmarks built from HTMI snapshots cap- tured at a single point in time. However, this offline evaluation paradigm fails to account for the temporally evolving nature of the web; consequently, perfor- mance on these static benchmarks often fails to generalize to dynamic real-world scenarios. To bridge this gap, we introduce LIVEWEB-IEE signed for evaluating WIE systems directly against live websites. Based on trusted and permission-grantes information extraction of various data categories, such as text, images, and hy- perlinks. We further design these queries to represent four levels of complex- ity, based on the number and cardinality of attributes to be extracted, enabling a granular assessment of WIE systems. In addition, we propose Visual Grounding Scraper (VGS), a novel multi-stage agentic framework that mimics human cog- nitive processes by visually narrowing down web page content to extract desired information. Extensive experiments across diverse backbone models demonstrate the effectiveness and robustness of VGS. We believe that this study lays the foun- dation for developing practical and robust WIE systems
new benchmark de-
websites, we curate natural language queries that require
INTRODUCTION
With the growth of data on the web, automatic information extraction from web pages has be- come crucial a wide range of applications, such as large- information analysis and decision- making (Crescenzi et al. 2001 "hhpelo et al. [021 Li et al.) 2023: Pichiyan et al.) 2023). The conventional approach web information extraction (WTE) has been dominated by wrapper-based methods, which define set of extraction rules based on the structural patterns of web pages (Ler-) man et al. 2003 Reis et al. 2004; Omari et al. 2017). With the advent of large language models (LLMs), language-agesn -gwet- based methods have been proposed, leveraging LLMs extract informa- tion directly from web pages (UncleCode) L Lorenzo Padoan 2024). However, both approaches face significant limitations: wrapper-based methods often brittle and require substantial human effort to create and maintain Dalvi et al. (2009), while using LLMs directly as extractors is imprac- tical for large-scale scraping tasks due the substantial costs incurred by processing each web page individually. To address this problem, hybrid approaches that leverage LLMs to generate reusable wrappers have emerged as a promising solution (Huang et al. 2024
Kaur 2025).
Despite these advancements, it is unclear whether the success of WIE systems on existing bench- marks reliable indicator of their performance in real-world scenarios. While existing bench- marks rely on static HTML snapshots. real-world web pages frequently change their layouts and structures over time (Hao et al. 2001T] Bronzi et al., 2013 Lockard et al. 2019). Given the strong dependence of WIE performance on the structural properties of web pages
Zheng al. 2007
Wang
"Equal contribution
Our dataset available httth:://tttiiiiio com/ 8by99/ vewebb-

---

Published as a conference paper at ICLR 2026
Existing Benchmarks
LiveWeb-IE [Ours]
Query Get the thumbnall on the ICLR 2026 homepage.
Existing Methods
Rio de Janeiro,
Brazil
N years
Full HTME
Wrong Answer
VGS (Duns)
Screenshot
Grounded
Offline (Frozen Snopshot)
(Evolving Web)
Online
Web Page
Region
Local HTML
Right Answer
(A)
(B)
Figure 1: Overview of the conventional WIE paradigm's 's limitations and our solutions involving new benchmark and an extraction method. (A) While existing offline benchmarks are constructed from static HTML snapshots, LIVEWEB-IE evaluates WIE systems on live websites to reflect the evolving nature the web. (B) On a complex live web page, methods that process full HTML often fail, whereas VGS leverages visual cues from the rendered page for accurate information extraction.
et al. 2022). performance measured on these offline benchmarks, which fail to capture such tempo- ral shifts, may not correlate with efficacy on the live websites (Mialon et al. 2023 Pan et al. 2024). Our empirical analysis in Appendix Fa confirms this discrepancy, showing that LLM- LM-edded wrapper generation methods suffer an average Fl degradation of over 15% when applied to the structurally evolved versions of the same websites compared to their static snapshots.
To fill this gap, we introduce LIVEWEB-IE, a benchmark for online web information extraction designed with four key features: (1) Live web evaluation. While conventional benchmarks eval- uate WIE systems on static HTML snapshots, LIVEWEB-IE mandates evaluation directly on live websites. By requiring WIE systems access the target URL., LIVEWEB-IE enables an assessment of their performance on the live web page at the time of evaluation, as shown in Figure (A). (2) Diverse and reliable websites. LIVEWEB-IE is constructed from 15 comiisiion-grantdd websites across 8 diverse domains. We select websites by considering high content stability, a crucial factor for ensuring the long-term validity of data annotation (Mialon et al., 2023). Specifically, we define this content stability as the temporal persistence of informational content, distinguishing it from layout stability. Our benchmark design ensures this content stability by curating queries for fact- based information that is unlikely to change. By targeting stable values within dynamic layouts, LIVEWEB- enables a robust evaluation of a system's 's ability to handle the web page structures at the moment of evaluation. (3) Query-driven WIE
diverse data categories. LIVEWEB IE follows an on-demand IE setup, allowing users to request information through free-form natural lan- guage queries (Sekine 2006; Tiao al., 2023 Qi et al. 2024). In real-world scenarios, these queries are not limited to text; users often want to extract non non-textual information, such as product images or event banners. However, existing benchmarks have focused on text-only extraction, failing to represent this full scope. LIVEWEB-IIY addresses this gap by comprising a diverse set of manually curated queries that require WIE systems to extract various data categories, including text, images, and hyperlinks, to closely align with these real-worke scenarios. (4)
task com- plexity. To support a comprehensive evaluation. LIVEWEB-IE organizes extraction tasks based on two properties: the number of attributes and the cardinality of their values. This combination yields four complexity levels, from extracting a single attribute with a single value (Type T) to multiple attributes with lists of values (Type IV).
Extensive experiments on LIVEWEB-IE show that even methods with strong performance on ex- isting benchmarks struggle significantly, especially complex extraction tasks. We attribute this performance degradation to approaches that only rely on parsing HTML, as its inherently verbose nature makes it challenging to extract the desired information (Bevendorff al. [023 He et al., 2024). As the structure of web pages grows more complex over time (Vogel & Springer 2022), the evolving nature of the web further amplifies the performance degradation of such methods.
2

---

Published as a conference paper at ICLR 2026
Based on these observations, we propose Visual Grounding Scraper (VGS), novel multi- stage framework that advances the promising paradigm of generating reusable wrappers. VGS emulates how humans find information gion where the information is, and then identifies the specific items within that region. This strategy bypasses the noise inherent in raw HTML by focusing the rendered web page, thereby generat- ing a reliable wrapper. Our method establishes state-of-the-art results not only on our challenging LIVEWEB-IT but also on existing WIE benchmarks. In summary, our contributions are as follows:
web page: VGS first visually scans the web page to find the
New Benchmark for Online WIE:
We introduce LIVEWEB-IE, a benchmark designed to over- come the limitations of offline evaluation by assessing WIE systems on online websites. It fea- tures diverse data categories, query-driven tasks, and a multi-dimensional complexity scheme.
Jisoon-Based Scraping Framework: We propose VGS, a framework that emulates how hu- mans find information on a web page. VGS visually identifies a relevant region on the web page and then pinpoints the specific target elements to generate accurate wrappers.
Extensive Evaluation: We prove the effectiveness and robustmess of VGS through extensive experiments across various backbone models, showing state-of-the-art performance on both our challenging LIVEWEB-IE and existing WIE benchmarks. Furthermore, we suggest directions for future work through human evaluations and detailed analysis performance.
RELATED WORK
WIE Benchmark. Extracting structured information from web pages is long-standing research
challenge due to the diverse and unstructured nature of web documents (Etzioni al. 2008 Manabe
& Tajimaa 2015 Wang et al.] 2022). This task has been evaluated using benchmarks ks that rely on static HTML snapshots (Hao et al.! 2011 Lockard et al. 2019 San al. 2023 Hotti al. 2024). These datasets, while valuable, are limited by an offline evaluation paradigm, which fails to account for the evolving nature of the web. Another line of research has introduced general-purpor
web benchmarks (Shi et al. [2017] et al. 2018 Humphreys et al. 2022 Deng et al.] [023 Zhou et al.) He et al. 2024] Wu al.] (025). The focus of these web agent benchmarks lies in evaluating an agent's 's ability to perform multi page task completion (e.g., booking a flight) through a series of web element interactions. Consequently, these benchmarks prioritize the evaluation of sequential action execution, which is distinct from the core WIE objective that requires the precise identification and extraction of target information from within DOM structures and visual layouts. Compared with these benchmarks, described in Table [ LIVEWEB- is specifically designed to assess the task of WIE, addressing a gap between existing WIE evaluation paradigms and real-world web scraping.
WIE Methodology. Early WIE systems mainly studied rule-based wrapper induction systems Kushmerick] 2000; Wu et al. 2009: Dalvi et al. [011 Furche et al., 2016). The advent of deep learning introduced suite of methods to better interpret HTML structures, progressing from quence models to more advanced architectures such as FreeDOM (Lin et al. 2020) with its CNN- BLSTM text encoder and WebFormer (Guo al. 2022), which integrates graph attention into Transformer framework. The development of pre-trained models specialized for web data, such as MarkupLM (Li et al.] 2022). and the incorporation of visual cues with models like WIERT (Li ct al. 2023), further enhanced extraction accuracy by understanding HTML semantics and layouts. Recently, the advent of LLMs has introduced language-agent-bun based systems that leverage reasoning capabilities for direct extraction (UncleCode) 2024, Lorenzo Padoan 2024). While flexible, these approaches are impractical for large-scale scraping tasks due to the significant latency incurred by performing inference on each page. To address this challenge, |Huang et al. (2024) and Kaur (2025) proposed hybrid approaches that utilize LLMs to generate reusable wrappers. The novelty of these works lies in their HTML-based pre-processing techniques, which prune the HTML content to filter out irrelevant noise for XPath generation. However, even the state-off thethe methods struggle on complex web pages, primarily due to over-reliance on HTML. To fill this gap. we propose VGS, a Dision-groundee framework that marks a methodological shift from prior approaches by emulating the human cognitive process and using visual information to sequentially filter out irrelevant noise.
3

---

Published as a conference paper at ICLR 2026
Comparison between LIVEWEB-IE and other web information extraction benchmarks. The query in ClosedIE corresponds to attributes, and in OpenIE to predicates. 1 denotes the page groups, clusters of structurally similar web pages within the website.
Table 1:
Benchmarks
Task
Modality
Online
Domain
Website
# Query
SWDE (Ha et al
2011
Closed IE
Text
X
8
80
32
[Bronzi dl
2013
Closed
Text
X
DSI Oman 7017
Closed IE
Text
X
Expanded SWDE (Lock
2019
Open IE
Text
X
3
21
748
PLAIE (San ct al 2023
Closed IE
Text
x
1
43
LIVEWEB-IE (Ours)
On- Ra-Demand IE
Text & Image
V
8
15 (461)
342
LIVEWEB-IE
In this section, we present LIVEWEB-IE, benchmark designed facilitate the evaluation of WIE systems that operate on live websites. Conventional benchmarks that rely on static HTML snapshots, often captured years in the past, fail to represent the current structural properties of web pages. WIE performance is highly dependent on these properties, it is crucial to evaluate systems against the most current version of a website. LIVEWEB-IE addresses this gap by mandating that systems access the target URL during evaluation. This protocol forces the system to process the web page's DOM structure as exists at the moment of testing. To maintain the validity of this evaluation over time, our benchmark is carefully designed to target ground-truth values that are factually stable, even as the website's s layout and structure evolve. We first define the task @ 3.D), then detail the dataset construction process ( 3.2), and conclude with statistical analysis of the benchmark ( 3.3
3.1 TASK DEFINITION
Formally, given target URL U and natural language query Q that requests the extraction of target attributes A = {ai,..., ). the objective is to extract the set of ground-truth values (ui,..., #*). The WIE system must infer the set of target attributes Á {â..... à+} from the query Q before extracting their corresponding values. Each value e; V that corresponds to an attribute a, E A can be single item or list of items according to the task type:
Type I: o = {()1, v1)}, where |A| = 1 and V1 is single item.
Type II: {(11, mi)....................... .... (ax, 4x where [A] > 1 and each is a single item.
Type III: 0 ((a), v1), where |A| = and 01 is list of items.
Type TV: 0 ((..,))..... (......,,,,,,))))) v1), where [A] > 1 and each is a list of items.
The performance is evaluated by first aligning the inferred attributes A with the ground-truth at- tributes A, and then comparing the extracted value é; with the ground-truth value v; This query- based information extraction setup reflects practical scenarios by aiming extract user-desired in-
formation that is often not covered by conventional ontologies (Jiao et al. 2023 Qi et al. 2024).
3.2 DATASET CONSTRUCTION
As shown in Figure 2 we construct LIVEWEB-IE in four stages: website selection, page grouping, data annotation, and human verification. Further details are described in Appendix
Website Selection. Our website selection process follows two core principles: ensuring bench- mark validity and adhering to the ethical standards for data acquisition. To ensure diversity, we select representative websites from a wide range domains, such as academic, e-commerce, and sports. A key criterion for selection is the stability and reliability of the websites, ensuring that their content remains consistent over time. For instance, a web page recording the results the 2022 World Cup final between Argentina and France contains factual information that is unlikely to be changed. By ensuring the content stability, we guarantee the benchmark validity and enable repro- ducible evaluation (Mialon et al. 2023 Guo et al. 2024: Pan et al., 2024). For ethical data sourcing, we implement a rigorous three-stage validation process. First, we review zobots t xxt files of each site check crawling policies. Second, we screen the Terms of Use confirm that data usage
4

---

Published as a conference paper at ICLR 2026
1. Website Selection
Data Annotation
4. Human Verification
WORMS
Attribute Identification
Whire
Dral
SAM 2:
Segment Anything in Images and Videos
SportsDB
ICLR
Nihil Revi Volentin Gobet Yuee-Tirg Hu Rondhwng Hu Cholianyn
Byeli: "engyu Ma Haltham Khedr mor Baile Chice Bullaid
Raw Dataset
Page Grouping
Laura Gustafsan Eric Mintun lunt ng Pan Kalaan Vasudev Alwala Nicolas Carlon Chao
Chao--uan Yuan Wu- Boss Ginshick Flttr Dolar Christoph Felchtenhiter
Castems 3t
ICLR
Sciti proseitic Suree GAld aecukador a
Refine Remove
(b) Query Curation
"Get the paper's title and abstract link""
(c) Value Collection
Title "SAM Segment Anything in Images
Group 1
Group
Abstract link: httpss//chhedd//ooott/hhbbttttt Jposter/ffbbstract details'
Verified Dataset
Figure 2: Dataset construction pipeline for LIVEWEB-IE. We first select diverse set of websites and group the web pages within each website by layout. We then annotate the attributes, queries, and values for each page group, followed by a human verification process to ensure data quality.
for research permissible. Finally, we directly contact the website administrators to obtain explicit consent. This systematic approach guarantees the ethical and legal integrity of our benchmark.
Page Grouping. Following the website selection process, perform page grouping, which in- volves clustering web pages within each website that exhibit similar layouts. This approach distin- guishes work from conventional WIE benchmarks, which are typically constructed using web pages with uniform structural patterns. By categorizing the structural variations within each web- site, enhance the structural diversity of the benchmark and facilitate an evaluation of the ability to handle the varied page layouts. A detailed description of each website is provided in Appendix H
Data Annotation. For each page group, manually curate natural language queries that request the extraction of specific attributes. Each of the five authors annotates queries for three distinct websites, following structured procedure: First, annotators identify all potential attributes that can be extracted from given page group, selecting only those corresponding values that are unlikely to change time. This criterion guarantees the long-term validity of the evaluation and eliminates the need for continuous human intervention to update ground-truth values. Furthermore, we select these attributes to encompass images (e.g., fanart) and hyperlinks (e.g., profile link) alongside text, reflecting the diverse nature of real-world user queries. Each identified attribute is then categorized by whether its value single item or list of items, which correspond to Type and Type III, respectively. To construct tasks for Type IT and Type TV, these attributes are combined while minimizing redundancy. Then, we curate a query in various formats to instruct the extraction of the selected attribute(s). Following the query annotation process, we collect the corresponding ground-truth values for each query. To efficiently handle the large-scale data extraction process, develop scraping scripts tailored for each page group. These scripts automatically parse the HTML content to extract the values corresponding to the attributes. For non-textual attributes, we functionally integrate by collecting their source values, such as @src for images and @href for hyperlinks, as the ground truth.
Human Verification. To ensure the quality of LIVEWEB-IE, we implement rigorous verification protocol. Each annotated sample is cross-validated by two annotators who are not involved in its creation. This review process focuses on three criteria: (1) whether the target attributes are clearly identifiable on the web page and correctly classified according to the task type, (2) whether the query temporally stable and accurately specifies the desired information, and (3) whether the annotated ground -round-truth values are correct. Samples that failed to meet these standards are either refined or removed by the verifiers. The details of dataset construction provided in Appendix C
5

---

Published as a conference paper at ICLR 2026
Input game
VGS
Aetrieve
Step Attribute Identification
Step 3: Element Pinpointing
Rendered Page
Step Grounding
Helicatta
VCM
A&A
XPath Synthesis
FasMt
Reglons
VLM
Paraei
Local HTML
Final
Maxe Teans
Output
The framework of VGS. sequentially narrows the observation space, from identifying target attributes, grounding the region, pinpointing the exact items, and generating the XPaths.
Figure 3:
3.3 DATASET ANALYSIS
Task Type
Data Category
Through the multi-stage data construc- tion process, construct LIVEWEB-IE, a benchmark designed for evaluating WIE systems live websites. The final dataset spans 8 domains, 15 websites, and 46 dis- tinct page groups, comprising 342 natural language queries and 97 unique attributes. The distribution of the queries across task type and data categories is illustrated in Figure Detailed statisties for each website are provided in Table to Table 9
19.01% 24.564
Typs 11
44.51%
33.33%
Text Irage Hype ryperLink
Typs 111 Type
20.07%
Figure 4: The task type and data category distribution.
4 VGS: VISUAL GROUNDING SCRAPER
In this section, we introduce Visual Grounding Scraper (VGS), a novel agentic framework designed generate robust wrappers. As shown in Figure 3 we employ multi-stage process that mimics the human cognitive process for seeking information on a web page. First, VGS decomposes a nat- ural language query into a of target attributes ( 4.1 For each attribute, it identifies relevant region on the web page ( 4.2). pinpoints the exact items of interest using bounding boxes ($ 4.3), and finally synthesizes generalizable XPaths using the highly focused visual and structural context & 4.4). This strategy accurately identifies the information required to generate wrappers by sequen- tially narrowing the observation space. The overall workflow of VGS illustrated in Figure 3
4.1 ATTRIBUTE IDENTIFICATION
The initial step is to decompose the user query Q into structured set of target attributes Á {ā H using an LLM guided by an attribute identification instruction La:
L.M Q)
This step translates the unstructured query into a concrete set extraction goals, thereby establish- ing clear objective that guides the entire subsequent process.
4.2 VISUAL GROUNDING
Following the attribute identification step, perform visual grounding to locate the relevant web page regions for each target attribute. The web page for a target URL is conceptualized as a sequence of vertical regions R {r where each region ry corresponds screenshot with

---

Published as a conference paper at ICLR 2026
fixed width and height. Since different attributes can be located in distinct regions of the web page, visual grounding is executed individually for each attribute à; Á. Given visual grounding instruction y.. a vision language model (VLM) evaluates the regions R to identify the pertinent region r for each attribute å, Through this evaluation, mapping R' is established, linking each predicted attribute à; with its corresponding visually grounded region r/:
R' = {(ā,, r where r VLM R,
This step prunes the observation space by isolating the most relevant regions of the web page. Con- sequently, subsequent stages can operate on condensed and contextually rich subset of the page.
4.3 ELEMENT PINPOINTING
Once the region containing the value for a target attribute is identified, we pinpoint the exact loca- tion of the value to identify the items required for XPath generation. For precise identification of attribute-relevant items, we adopt a two-stage approach. First, we generate a set of candidate bound- ing boxes B, (br,s }=11 around potential items of interest within r/- The strategy for generating these candidates is adapted based on the type of the attribute âq. For attributes whose values are directly rendered text, we visually scan the region ri and identify all relevant text items using the VLM. For non-textual attributes such as images or hyperlinks, we identify candidate items based on tags relevant to the attribute type (e.g., <img> tag for images). To mark these candidates in the region, we employ Set-of-Mark Prompting (Yang et al.) (023). Specifically, we use JavaScript tool to generate and overlay bounding boxes with unique numerical labels onto the candidate items within the region. The overlay function O applies the candidate bounding boxes B, to the region ri generating the visually marked region r = O(B,, Then, the VLM processes the marked region rf based on the pinpointing instruction Lp to select the subset of bounding boxes B S B, corresponding to the actual values of attribute à; This pinpointing process is formally defined as:
B HE where B VLM a?)
(3)
We finalize the localization process by pinpointing the values corresponding to the target attributes. The resulting set of bounding boxes supplies the evidence for the subsequent XPath synthesis.
4.4 XPATII SYNTHESIS
In the final stage, we synthesize reusable XPaths using the set of validated bounding boxes B For each bounding box by, E 13; we first identify the primary Document Object Model (DOM) element eb corresponding to the coordinates of the box. To provide sufficient context while minimizing noise, it then extracts local HTML segment hss consisting of and neighboring elements within a distance d. The VLM synthesizes a generalizable XPath x; for an attribute â, using an XPath synthesis instruction L,, the set of all relevant local HTML segments H4 {hat 1 the target attribute à, and the further pinpointed region A; = O rf). The set of XPaths X' that constitutes the final wrapper is formally defined as:
X== ( where x; VLM(L,, H4, a.)
By grounding the generation both visual evidence and localized information, the resulting XPath achieves high precision and generalizability. Each XPath 2; within the set of XPaths X' is used to generate the predicted value 0 for its corresponding attribute a,.
EXPERIMENTS
5.1 EXPERIMENTAL SETTING
Baselines. We compare VGS against methods that leverage LLMs to generate reusable wrappers. We adopt Chain-of- -fann-o (CoT) (Wei et al., 2022). Reflexion (Shinn et al.) 2023), and Auto- Scraper (Huang al. 2024) as baselines. CoT generates XPath in single pass, whereas Reflex- ion operates iteratively to refine its output based on execution failures. AutoScraper also follows an iterative process, but proactively simplifies the web page by pruning the DOM tree. We also report the performance a human evaluation conducted by six experts with relevant backgrounds. Details for human evaluation available in Appendix B.3

---

Published as a conference paper at ICLR 2026
Table Main experimental results on LIVEWEB-IE, measured by Precision (P), Recall (R), and FI score (F1). The best score for each metric is highlighted in bold.
Models
Method
Type
Type
Type
III
Type
P
F1
R
F1
P
F1
R
F1
P
R
F1
Proprictary Models
COT
132.04
49.01
34.36
32
54,80
37
9.66
13.99
8.19
10.16
10.90
7.40
20.36
31.33
20.53
Gemini-2.5.Flash
Reflexion
33.00
48.31
35.96
89.87
39.59
39.40
10.29
13.00
8.30
12.83
23.11
8.67
23.53
29.66
22.39
AutoScraper
36.52
42.35
37.25
31.47
88.76
34.39
10.94
18.61
10:23
14.36
23.75
9.66
22.41
22.02
VGS
47,83
55.85
49.02
42.36
53.76
44.82
43.61
43.49
42.92
38
38.32
38.13
42.81
47.46
43.44
COT
32.15
41.08
33.41
31.86
54,23
34,80
8.88
18.45
8.76
12.22
22.01
8.26
20.58
33,66
20.63
GPT-4o-mini
Reflexion
31.97
38.15
33.17
24.84
31.37
25.95
8.26
9.09
7.60
9.68
10.38
7.05
17.75
21.18
17.47
AutoSerapee
35.10
43.23
36.25
31.14
55.17
34.03
11.74
19.26
11.05
13.67
22.62
9.20
22.10
34.73
21.85
VGS
49.11
57.55
50.36
32.37
46.30
33.32
33.87
34.58
33.95
30.29
30.72
29.68
35.48
41.28
35.85
CUT
45.96
52.83
47.54
39.53
49.51
40.84
12.14
11.44
8.15
11.20
16.
7.24
26.03
31.27
24.60
Go
Reflexion
47.75
50.75
48.75
39.46
43.54
39.64
15.9
10.34
10.24
7.24
3.69
3.76
26.47
25.71
24.22
AutoScraper
53.23
59.52
55.22
41.93
$2.79
42.65
13.95
12.10
9.10
12.60
13.37
6.92
28. 94
32.86
26.76
VGS
64.51
69.78
65.87
55.49
46.35
45.25
45.73
45.38
41.33
41.86
41.50
47.78
52.09
48.58
Open-Source Models
COT
12.14
25.23
14.62
3.81
10.11
4.71
5.63
7.22
5.90
0.60
1.19
0.79
5.11
9.98
5,98
Gemma-3-4B
Reflexion
12.19
15.94
12.45
10.54
30.00
13.61
9.57
32.23
10.99
4.19
4.79
4.21
9.01
21.78
10.35
AnteSeraper
14.30
18.06
14.95
11.54
30.19
14.04
9.67
35.85
10:27
7.08
17.94
7.61
10.44
26.48
11.57
VGS
19.97
21.47
20.10
21.03
27.66
22.33
15.02
24.92
15.70
4,98
29.53
7.19
15,18
26.17
16.30
COT
31.53
47.08
32.63
17.06
46.96
20.07
9.56
17.39
10.69
6.15
17.24
7.29
15.01
31.29
16.65
Gemma-3-222 3-27B
Retlexion
36.71
25.73
12.03
17.47
AutuScraaee
33.50
45.85
34.63
19.51
$3.35
22.87
9.81
16.58
10.27
10.92
13.36
10.31
17.30
31.67
19.04
VGS
36.99
38.72
61.27
29.73
29.65
29.87
30.79
COT
34.41
43.21
35.08
9.44
16.25
9.82
4.37
15.85
4.98
2.28
8.16
3.45
10.99
19.27
11.67
Qwen-2.5-7B
Reflexion
36.56
46.98
38.12
19.75
13.30
5.64
23.71
7.76
16.55
13.22
14.53
AutoScraper
36.00
42.08
36.75
19.41
41.57
19.53
7.35
11.64
7.50
3.77
21.55
5.90
15.30
28.25
16.04
VGS
37.99
47.36
37.31
23.00
43.67
16.63
16.64
13.07
15.28
13.58
21.60
30.23
21.74
COT
28.83
42.59
30.78
32.49
32.08
7,20
16.11
6.23
4.04
10.67
4.61
17.63
25.76
17.74
Qwen-2.5.12n
Reflexion
37.30
44.91
37.57
30.90
48.48
33.05
8.47
15.02
6.72
7.53
13.38
7.98
20.01
29.68
20.28
AutoScraper
40.63
44.72
41.07
35.42
46.48
36.20
7.11
14.63
8.37
5.48
9.27
5.23
21,02
27.97
21.61
VGS
46.85
52.45
46.76
39.78
40.07
37.83
30.33
30.61
29.90
28.09
38.31
28.67
35.57
39.31
35.05
COT
35.77
39.98
69
9.48
7.53
18.75
21.01
22.06
Qwen-2.5-72B
AutoScraper
Reflexion
38.03
30.32
61.04
33.69
41.71
31.33
30.15
33.53
51.69
35.29
31.38
37.61
10.69
10:05
18:33
8.96
8.62
8.42
6.53
17.00
7.94
4.95
21.90
18.86
35.64
20.79
23.28
18.36
VGS
48.09
53.96
48.63
37.38
60.63
41.10
35.30
39.43
36.53
30.42
31.91
31.06
37.12
46.30
38.77
Human
83.31
92.40
87.13
85.56
94.05
86.06
89
30.30
89.50
89.04
82.74
87.51
84.14
85.24
90.86
86.60
Models. We employ diverse set of both proprictary and open-source models. For proprictary models, we use Gemini-2.55Fiaan (Comanici et al. 2025), GPT-4o-mini, and GPT-4o (Hurst al. 2024]. For open-source models, we utilize Gemma-3--4B/7B. Instruct (Team et al. 2025) and Qween-2.5-7B/22B772BB
(Yang et al., 2024a). Since the Qwen-2.5-Inntrrue models are text- only, we employ their corresponding vision-language counterparts of the same size, Qwen-2.5-VL. 7B/32B/72B-Instrucc (Bai et al. 2025), for the visual analysis steps within our method. As the other them consistently for all stages of our method.
models inherently multi-modas
Evaluation. We evaluate how effectively WIE system identifies the target attributes from nat- ural language query and extracts their corresponding values. First, we determine if the WIE system correctly identifies the target attributes. To account for the semantic flexibility of natural language, we employ an LLM-basee alignment strategy. using GPT-4o to match the inferred attributes with their ground-truth counterparts (Jiang et al.) 2024) Chen al. 2024). Inferred attributes that do not align are considered an extraction failure. Subsequently, for each aligned attribute, we compare the extracted values against the ground-truth values. line with existing evaluation schemes for WIE, measure performance using precision, recall, and Fl score (Hao et al.) 2011 Lockard et al. 2019 Huang et al. 2024). Further details on the evaluation setup are available in Appendix B
Implementation Details. All experiments are conducted a zero-shot setting due to the context length limitations of the models. As the baseline methods do not account for interaction with the live web page, we provide the HTML of a given web page as input. For each sample, a set of XPaths is generated from the first web page in group and then applied all other pages within that group
8

---

Published as a conference paper at ICLR 2026
CoT
Refiexion
AutoScraper
VGS (Ours)
SWDE
Expanded SWDE
Fl Score (%)
80-
Gemini-2.5 2.5- Flash
GPF do mini
GPT-40
Gemma- 3.40
Owen Gemma-3-2778
Gemma 3-278
Qwen-2.5-328
Qwen Qwen- 2.5.228
2wen-225-72B 728
Gemini G.mini-225-Flaat 2.5-flash GPT-40 mini
GPT-40
Gemma- 3.40a
Qwen-2.5.71 Gemma-3-270 3-270
Qwen-2 5.3328
Qwen 0.en-2.5-720 2.5-728
Gemma-
Figure 5: Fl score comparison on existing web information extraction benchmarks. VGS generally
outperforms baselines across different backbone models. Full results shown in Table 5
to extract the target values. We use Playwright for rendering and navigating web pages, as well as for executing the generated XPaths. Further details on experiments are available Appendix A
5.2 MAIN RESULTS
presents the main results for VGS and baseline methods across various backbone models, showing that VGS generally outperforms the baselines across all task types. The performance gap between VGS and the baselines is particularly pronounced on the Type III and Type which require the extraction of multiple attributes. Specifically, when using GPT-4o as the backbone model, VGS outperforms AutoScraper by a significant margin of 36.28% and 34.58% in Fl score on Type III and Type IV tasks, respectively. This indicates that our vision-grounde identifying the precise target elements facilitates the generation accurate XPaths on live websites. It is noteworthy that even when using the powerful backbone models, GPT-4o and Qwen-2.5-72B. the best perfformin VGS falls short of human performance by 38.02% and 47.83% in overall score, respectively. This finding demonstrates that our dataset is challenging benchmark and un- derscores the need for further advancements to handle live web environments.
Table 2
tasks,
approach to
5.3 ABLATION STUDY
VGS filters out irrelevant information by pro- gressively narrowing down the observation space. This process is driven by two sequential steps: visual grounding and element pinpoint- ing. To analyze the contribution of each step, conduct ablation study. As shown in Ta- 3le[]] the removal of any component results in performance drop. Notably, when using Qwen- 2.5-72B as the backbone, removing both com- ponents simultaneously causes a performance drop of 7.59%, a more pronounced degradation than removing either component individually. These findings validate the effectiveness of our sequential, vision-based filtering approach.
Table 3: Ablation study on visual grounding and element pinpointing. We report the Fl score.
Grounding Pinpointing
GPT-4e
Qwen-2.5-72B
/
48.58
38.77
/
x
43.89 (L 4.69%)
34.72 (L 4.05%)
x
/
45.62 (L 2.96%)
37.06 (1 1.71%)
x
x
43.17 5.41%) 31.18 (L 7.59%)
DISCUSSION
6.1 EXPERIMENTS OVER EXISTING BENCIIMARK
To demonstrate the robustness of proposed method, evaluate existing WIE benchmarks, SWDE (Hao et al.] 2011) and Expanded SWDE (Lockard et al.] 2019). Following the evaluation protocol Huang et al. (2024), we prompt each method to extract the target attributes using natural language instructions. These benchmarks only focus on preserving textual content and the basic DOM structure, often resulting in a rendered web page with visual artifacts, such overlapping text and expired images (Yang et al.) (204b). Since our methodology operates on screenshots of rendered web pages, we manually exclude samples exhibiting significant rendering issues. Further details are provided in Appendix B.2 Our experimental results demonstrate that VGS generally achieves
hhups///laywriihh.dew

---

Published as a conference paper at ICLR 2026
strong performance on both benchmarks, shown in Figure 5 Notably, VGS exhibits significant performance improvement over the baselines when utilizing small-sized backbone models, such as Gemma-3-4B and Qwen-2.5-7B. The results confirm the robustness and effectiveness of our vision- grounded approach, validating its effectiveness even on traditional benchmarks.
6.2 PERFORMANCE ANALYSIS DATA CATEGORY
LIVEWEB-IE incorporates non-textual data, such as images and hyperlinks, to reflect the diverse requirements of practi- cal WIE scenarios. We analyze how per- formance varies across different data cat- egories. Figure [6] illustrates the perfor- mance when using GPT-4o and Qwen-2.5- 72B as backbone models. A key obser- vation is that both baseline methods and VGS exhibit performance degradation when extracting non-textual data. While VGS obtains relatively high Fl scores of 44.05% for images and 43.13% for hyperlinks using the GPT-4o, these results indicate that there still room for improvement. The difficulty of non-textual information extraction suggests that a direction for future research is to improve the capability to extract information that is not explicitly represented.
CoT
Reflexion
AutoScraper
VG5 (Ours)
GPT-4o
Qwen-2.5-72B
Scare (%)
40
Text
Image
Hyperlink
Text
Image
Hyperlink
Figure 6: score comparison of VGS and baseline methods across three data categories.
6.3 ERROR ANALYSIS
To better understand the failure cases of VGS, we manually analyze 100 ran- dom samples for each of the Qwen-2.5- 7B/32B/72B, GPT-4o-mini, and GPT-4o models. The errors categorized into four main failure types, corresponding to the main steps of VGS: misinterpreting the target attribute, failing to ground the re- gion, incorrectly pinpointing the target el- ement, and generating wrong XPaths. Fig- ure presents the distribution of these er- ror types. Our analysis reveals that sig- nificant portion of errors occurs during the stages that leverage visual modality to identify information corresponding to the target attributes. While leveraging visual context filter out irrelevant information proves beneficial for generating robust XPaths, precisely identifying the target information remains a key challenge.
Error Grounding Failure
Anppiiniing XPeth Error
GPT-4o-mini
30%
29%
19%
80%
GPT-4o
68%
Owen-2.5-76.
33%
30%
23%
90%
Qwen-25-328
31%
28%
20%
82%
Qwen-2255776
20
40
60
80
100
Error Proportions (%)
Figure 7: Distribution of error types for VGS across different backbone models.
7 CONCLUSION
In this work, we introduce LIVEWEB-IE a benchmark designed to evaluate the capabilities of web information extraction systems under the online web environment. LIVEWER-IE features a collec- tion of live websites and natural language queries designed to cover four levels of complexity. Our benchmark reveals that even methods exhibiting strong performance on existing benchmarks strug- gle generalize to live websites. To address this, we propose VGS, multi-stage framework that leverages visual context to accurately identify the elements required to generate XPath by progres- sively filtering out irrelevant information. Experiments show that VGS outperforms strong baselines on both LIVEWEB- IE and existing benchmarks. Our work sets a new paradigm for evaluation and methodology, driving progress toward more practical web information extraction systems.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
This work was supported by the Starting growth Technological R&D Program(TIPS Program, (No. RS-2023-00272605)) funded by the Ministry of SMEs and Startups(MSSS, Korea) 2023; Insti- tute for Information & communications Technology Planning & Evaluation(IITP) grant funded by the Korea govermeen((MSII) (RS-2019-1119007) Artificial Intelligence Graduate School Pro- gram(KAIST)): and the National Research Foundation of Korea(NRF) grant funded by the Korea MSV)) (No. RS-2025-005556221)
ETHICS STATEMENT
Ethical considerations are essential this study, particularly concerning the construction of the LIVEWEB-IE benchmark. Our protocol for website selection is guided by the principle of respecting website operators and user privacy. For each of the 15 websites that constitute our benchmark, we undertake a multi-step approval process: (1) we first review their robots. txt files and Terms of Use ensure our research activities are not in violation of their stated policies. (2) Following this, we proactively contact the administrators of each website to explain the purpose of our research and formally request permission include their website in our benchmark. We proceed only after receiving explicit consent. The natural language queries in LIVEWEB-IE are intentionally curated to extract publicly available information. Our data collection and annotation processes are designed to minimize any risk of exposing private data. We will release our benchmark under a license that restricts its application to research purposes only, accompanied by clear guidelines.
REPRODUCIBILITY STATEMENT
To ensure reproducibility, we provide our LIVEWEB-IE and all experimental details. The dataset is accessible through an anonymized GitHub repository /ettps://githee com/sby99 LiveWeb-IEl Appendix A documents the experimental environment and hyperparameters. Ap- pendix B describes the evaluation protocols for our dataset and existing benchmarks. All prompts used in the experiments are provided in Appendix
REFERENCES
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502. 13923, 2025.
Janek Bevendorff, Sanket Gupta, Johannes Kiesel, and Benno Stein. An empirical comparison of web content extraction algorithms. In Proceedings of the 46th International ACM SIGIR Confer- on Research and Development in Information Retrieval, pp. 2594 2603, 2023.
Mirko Bronzi, Valter Crescenzi, Paolo Merialdo, and Paolo Papotti. Extraction and integration of partially overlapping web sources. Proceedings of the VLDB Endowment, 6 6(10):805-8) 816, 2013.
Ruirui Chen, Chengwei Qin, Weifeng Jiang, and Dongkyu Choi. large language model good annotator for event extraction? In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 17772-17780, 2024.
Gheorghe Comanici, Eric Bieber, Mike Schaekermann, Ice Pasupat, Noveen Sachdeva, Inderjit Dhillon, Marcel Blistein, Ori Ram, Dan Zhang, Evan Rosen, et al. Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capa- bilities. arXiv preprint arXiv:2507.0626: 2025.
Valter Crescenzi, Giansalvatore Mecca, and Paolo Merialdo. Roadrunner: Towards automatic data extraction from large web sites. In Proceedings of the 27th International Conference Very Large Data Bases, VLDB '01, pp. 109-118, San Francisco, CA, USA, 2001. Morgan Kaufmann Publishers Inc. ISBN 1558608044.

---

Published as a conference paper at ICLR 2026
Nilesh Dalvi, Philip Bohannon, and Fei Sha. Robust web extraction: an approach based a prob- abilistic tree-edit model. In Proceedings of the 2009 ACM SIGMOD International Conference
Management of Data, SIGMOD '09, pp. 335-348, New York, NY, USA, 2009. Associa- tion for Computing Machinery. ISBN 9781605585512. doi: 10. 10.1145/1559845.1 1559882. URL.
https:://oi..or http///i.orgg//00 1145/ 01984555994555 1559882
Nilesh Dalvi, Ravi Kumar, and Mohamed Soliman. Automatic wrappers for large scale web ex- traction. Proc. VLDB Endow, 4(4):219-230, January 2011. ISSN 2150-8097. doi: 10.14778/
1938545.1938547
1938544
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-28 14, 2023.
Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S. Weld. Open information extraction from the web. Commun. ACM, 51(12))68-74, December 2008 ISSN 0001-0782 doi: 10.1145/ 1409360. 1409377. URL.
000009336011199997 1409378
Tim Furche, Jinsong Guo, Sehastian Maneth, and Christian Schallhart. Robust and noise re- sistant wrapper induction. In Proceedings of the 2016 International Conference on Manage- ment of Data, SIGMOD '16, pp. 773-784, New York, NY, USA, 2016. Association for Com- puting Machinery. ISBN 978 4503353335. doi: 10.1 00.1145/88299332115224
URL https:
//doi. 10. 111////22903333 29152144
Yu Guo, Zhengyi Ma, Jiaxin Mao, Hongjin Qian, Xinyu Zhang, Hao Jiang, Zhao Cao, and Zhicheng Dou. Webformer: Pre-training with web pages for information retrieval. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1502-1512, 2022.
Zhicheng Guo, Sijie Cheng, Hao Wang. Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Maosong Sun, and Yang Liu. StableToolBench: Towards stable large-scale benchmarking on tool learning of large language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Findings of the Association for Computational Linguistics: ACL 2024, pp. 1 11143-11111 11156, Bangkok, Thailand, August 2024, Association for Computational Linguistics. doi: 10. 10.18653/v122244 2t0.5////1/2224...ninn--aaaaa 664. URL
anthology org/ 2024. findings acl. 6647
Qiang Hao, Rui Cai, Yanwei Pang, and Lei Zhang. From one tree to forest: a unified solution for structured web data extraction. In Proceedings of the 34th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '11, pp. 775-784, New York, NY. USA, 2011. Association for Computing Machinery. ISBN 9781450307574. 100991600010000. URL
doi: 10.1145/
htt..0.1112222..00 2t1.111..00000000 2010020
Hongliang He, Wenlin Yao, Kaixin Ma, Wenhao Yu, Yong Dai, Hongming Zhang, Zhenzhong Lan, and Dong Yu. WebVor::::: Building end-to-end web agent with large multimodal models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Anmual Meet- ing of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6864 6890, Bangkok, Thailand, August 2024. Association for Computational Linguistics. doi: 10. 18653v1vl 2024.acl-long.371 URL httt:ttt:: acl anthology.... www.oorg/22244 acl long..... 3717
Alexandra Hotti, Riccardo Sven Risuleo, Stefan Magureanu, Aref Moradi, and Jens Lagergren. The klarna product page dataset: Web element nomination with graph neural networks and large language models, 2024. URL ptattpss//aaxiv... org. 2111. 02168
Wenhao Huang, Zhouhong Gu, Chenghao Peng, Jiaqing Liang, Zhixu Li, Yanghua Xiao, Liqian Wen, and Zulong Chen. AutoScraper: A progressive understanding web agent for web scraper generation. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Pro- ceedings of the 2024 Conference on Empirical Methods Natural Language Processing, pp- 2371-2388 Miami, Florida, USA, November 2024. Association for Computational Linguis- tics. doi: 10.186533w1222244emmlp eemn p- ma n. 1417
141. URL ht htttpp://aaaaatthooooo.....//// wtwp:hhhollyy.o//222200.
12

---

Published as a conference paper at ICLR 2026
Peter C. Humphreys, David Raposo, Tobias Pohlen, Gregory Thornton, Rachita Chhaparia, Alistair Muldal, Josh Abramson, Petko Georgiev, Alex Goldin, Adam Santoro, and Timothy P. Lilli- crap. A data-driven approach for learning to control computers. In International Conference on Machine Learning, 2022. URI. hwt.hhttt://eett www.ntm www.annttcccollll ar. /rggCorppsTTD 246867455
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Pengcheng Jiang, Jiacheng Lin, Zifeng Wang, Jimeng Sun, and Jiawei Han. Genres: Rethinking evaluation for generative relation extraction in the era large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 2820-2837, 2024.
Yizhu Jiao, Ming Zhong, Sha Li, Ruining Zhao, Siru Ouyang, Heng Ji, and Jiawei Han. In- struct and extract: Instruction tuning for on-demand information extraction. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Meth- ods in Natural Language Processing, pp. 1005,, Singapore, December 2023. Associa- tion for Computational Linguistics. doi: 10.18653/v1/22 1/2033/////223.2 http//22...mmmmmmae/m mennl-mminn6622 URL https:
emtpp-3.mmmmpmmmaaaaaa 6207
Ashmeet Kaur. Automating xpath query generation using nip for streamlined web crawling and gui testing. In 2025 2nd International Conference on Trends in Engineering Systems and Technologies (ICTEST), volume pp. 1 1-6, 2025. doi: 00.1100/CTEES6667700000111
Nicholas Kushmerick. Wrapper induction: efficiency and expressiveness. Artif. Intell., 118(1-2): 15-68, April 2000. ISSN 0004-3702. doi: 10. //doi.org/// 10.gg110.00 2011//0000433300000 (99) 00100-9
URL https:
Kristina Lerman, Steven Minton, and Craig Knoblock. Wrapper maintenance: A machine learning approach. J. Artif. Intell. Res. (JAIR), 18: 18:149-181, 02 2003. doi: 10.161 1145.
Junlong Li, Yiheng Xu, Lei Cui, and Furu Wei. MarkupLM: Pre-training of text and markup language for visually rich document understanding. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6078-6087, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653/v 1/2022...2/.......oooooggg 122 URL https: atwa/aaaaaa http:::s//.llanthoooyy....o org/ 20w. long 4207|
Zimeng Li, Bo Shao, Linjun Shou, Ming Gong, Gen Li, and Daxin Jiang. Wiert: web informa- tion extraction via render tree.
AAAI Press, 2023. ISBN 978-1-
57735-880-0. doi: 10. 1l.1609/aaii.v37ii 26546.
v37il1 26546
Bill Yuchen Lin, Ying Sheng, Nguyen Vo, and Sandeep Tata. Freedom: A transferable neural architecture for structured information extraction on web documents. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 1092- 1102, 2020.
Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning web interfaces using workflow-guided exploration. In International Conference on Learning Representations, 2018.
Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. OpenCeres: When open information extrac- tion meets the semi-structured web. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp- 5047-3056 Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. doi: 00.18653/v1/119. URI. htR.:ttts:://// antho ogy. ogy.orrg/199-1300
13

---

Published as a conference paper at ICLR 2026
Marco Vinciguerra Lorenzo Padoan. Scrapegraph-ai, 2024. URL https: httpb/ggttub..t.// com/
VinciGito0/Scxaperaaaat A Python library for scraping leveraging large language models.
Tomohiro Manabe and Keishi Tajima. Extracting logical hierarchical structure of html documents based on headings. Proc. VLDB Endow.. 8(12):1606-1611 August 2015. ISSN 2150-8097. doi: 10. 10.14778/2222002.2244063
htt.:tttp://oo..ooo wtt..//////...11///111 www.111272220033 2824058
Grégoire Mialon, Clémentine Fourrier, Craig Swift, Thomas Wolf, Yann LeCun, and Thomas Scialom, Gaia: benchmark for general ai assistants, 2023. URL httphhttp//// htttp::/azxxv.orr
abs/ 22as/2311.. 12983
Adi Omari, Sharon Shoham, and Eran Yahav. Synthesis of forgiving data extractors. In Pro- ceedings of the Tenth ACM International Conference on Web Search and Data Mining, WSDM '17, pp. 385-394, New York, NY, USA, 2017. Association for Computing Machinery. ISBN URL cettps://dci.org//
9781450346757. doi: 10.1145/301 8661 13611.00
3018661 3018740
Yichen Pan, Dehan Kong, Sida Zhou, Cheng Cui, Yifei Leng, Bing Jiang, Hangyu Liu, Yanyi Shang, Shuyan Zhou, Tongshuang Wu, and Zhengyang Wu. Webcanvas: Benchmarking web agents in online environments, 2024. URL Ittpps://axxiii org/ abs / 12373
Vijayaragavan Pichiyan, S Muthulingam, Sathar G, Sunanda Nalajala, Akhil Ch, and Manmath Nath Das. Web scraping using natural language processing: Exploiting unstructured text for data ex- traction and analysis. Procedia Comput. Sci., 230(C) Jamuary 2023. ISSN 1877-0509. doi: 10. prtp://jpocc.222.... 12. 074
URL https: doi org http:/...0011.1111111111 wtwp://.p............ 2023. 1016/1
Yunjia Qi, Hao Peng, Xiaozhi Wang, Bin Xu, Lei Hou, and Juanzi Li. ADELIE: Aligning large language models on information extraction. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 7371-7355 Miami, Florida, USA, November 2024. Association for Computational Linguisties. doi: 18.1 102/5///1/222.... www.:4emmllllmmim......m 2024. emm main. 419/
htLhttss::/ /att:ss:/aclaatthooloo
D. C. Reis, P. B. Golgher, A. S. Silva, and A. F. Laender. Automatic web news extraction using tree edit distance. In Proceedings of the 13th International Conference on World Wide Web, WWW '04, pp. 502-511, New York, NY, USA, 2004, Association for Computing Machinery. ISBN doi: 10.1 021114/998772.98874488 988740
Aidan San, Yuan Zhuang, Jan Bakus, Colin Lockard, David Ciemiewicz, Sandeep Atluri, Kevin Small, Yangfeng Ji, and Heba Elfardy. PLAtE: A large-scale dataset for list page web extrac- tion. In Sunayana Sitaram, Beata Beigman Klebanov, and Jason D Williams (eds.), Proceedings of the ólst Annual Meeting of the Association for Computational Linguistics (Volume 5: Indus- try Track), pp. 284-294, Toronto, Canada, July 2023. Association for Computational Linguis- acl-inuustry.. 1nn.:///tt//2.....iinnunttt.t..
tics. doi: 10. 10.18653/v1/20233
URL https: http://acccachhcaaeaoacaaccao www.hholgy....gg/////
a sel-industry.27.
Satoshi Sekine. On-demand information extraction. Proceedings of the COLING/ACL 2006 Main Conference Poster Sesvions, pp. 731-738, Sydney, Australia, July 2006. Association for Computational Linguistics. URL https: httl:ttts//ccaaathoo.tt http///aaaaaaa.....ooo.o wtwp:/looy..r./P0000000000 2094/
Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits:
An open-domain platform for web-based agents. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Pro- ceedings of Machine Learning Research, pp. 3135-3144. PMLR, 06- Aug 2017. URL https: htttps::/proceediing wttp:/gg.......e/m.ee..
m mlr.mmmr......... www.ss/ss/ / shi 17a html
Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion:
Language agents with verbal reinforcement learning. Advances in Neural Information Processing Systems, 36:8634-4666 2023.
14

---

Published as a conference paper at ICLR 2026
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint
2025.
Tsaone Swaabow Thapelo, Molaletsa Namoshe, Oduetse Matsebe, Tshiamo Motshegwa, and Mary- Jane Morongwa Bopape. Sasscal websapi: A web scraping application programming interface to support access sasscal's weather data. Data Science Journal, Jul 2021.
UncleCode. Crawl4ai: Open-source Ilm friendly web crawler & scraper. https://g/thhb.
com/ wwtp://ccceeeeeeeeeee//e/// 2024.
Lucas Vogel and Thomas Springer. An in-depth analysis of web page structure and efficiency with focus on optimization potential for initial page load. In International Conference on Web Engi- neering, pp. 101-116. Springer, 2022.
Qifan Wang. Yi Fang, Anirudh Ravula, Fuli Feng, Xiaojun Quan, and Dongfang Liu. Webformer: The web-page transformer for structure information extraction. In Proceedings of the ACM Web Conference 2022, pp. 3124 3133, 2022.
Jason Wei, Xuezhi Wang. Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoe Le, Denny Zhou, et al. Chain-of-though prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Bo Wu, Xueqi Cheng, Yu Wang, Gang Zhang, and Guodong Ding. Facilitating wrapper generation with page analysis. In Proceedings of the 2009 IEEE International Conference Intelligence and Security Informatics, ISI pp. 191-193. IEEE Press, 2009, ISBN 9781424441716.
Jialong Wu, Wenbiao Yin, Yong Jiang, Zhenglin Wang, Zekun Xi, Runnan Fang, Linhai Zhang, Yulan He, Deyu Zhou, Pengjun Xie, and Fei Huang. WebWalker: Benchmarking L.MSs in web traversal. In Wanxiang Che, Joyce Nabende, Ekaterina Shutova, and Mohammad Taher Pilehvar (eds.), Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1; Long Papers). pp. 10290-10305, Vienna, Austria, July 2025. Association for Com- putational Linguistics. ISBN 979---99176-251-0, doi: 10. 18653/11/20222 acl-on.ooo URL https: https://aclaantooooo.oo////////2
5087]
An Yang. Baosong Yang, Beichen Zhang, Binyuan Hui. Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115. 2024a.
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mant prompting unleashes extraordinary visual grounding in gpt-4v. arXiv preprint arXiv:2310.1144)) 2023.
Yifei Yang, Tianqiao Liu, Bo Shao, Hai Zhao, Linjun Shou, Ming Gong, and Daxin Jiang. Hypertext entity extraction in webpage, 2024b. URL
698
Shuyi Zheng, Ruihua Song, and Ji-Rong Wen. visual consistency. In AAAI, volume 7, pp. 1507-1513, 2007.
news extraction based on
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for build- ing autonomous agents. In The Twelfth International Conference Learning Representations.
15

---

Published as a conference paper at ICLR 2026
A IMPLEMENTATION DETAILS
Environment Setup. We perform experiments on Linux server with two NVIDIA H200 GPUs. Our implementation relies on Python 3.12.11, CUDA 12.9, and the Playwright library (version 1.54.0) for web browser automation. We set a web page viewport size of 1280 x 1100 pixels and a neighbor distance of 2. For open-source model inference, we utilize the vLLM library (version 0.10.1.1).
Hyperparameters. ensure consistent and reproducible results across all experiments, we apply uniform generation hyperparameters to all models. We use a temperature of 0.0 and a max sequence length of 8192 for the generation hyperparameters.
LLM-based Wrapper Generation. Raw HTMI content contains numerous irrelevant elements that interfere with LLMss while the full content often too large to fit within the context length limitations of LLMs. Following Huang et al. (024), provide simplified HTML to LLM-based wrapper baselines. Specifically, we first remove elements that contain <script> and <style> from the DOM tree. Then, we preserve the @class attribute alongside essential functional at- tributes, including fhref for hyperlinks and @rrc and @alt for images.
B EVALUATION DETAILS
B.I LIVEWEB-IE
We employ the powerful LLM (GPT-4o) to judge whether predicted attributes from WIE systems correspond to ground-truth attributes. This approach is crucial for accommodating the semantic flexibility inherent in natural language. For instance, if the ground-truth attribute is "author profile link" and the system predicts "author info page link", our evaluation method correctly identifies these synonymous. To ensure the validity of this automated evaluation, manually review the 50 samples where the predicted attributes were not exact literal matches to the ground truth. The judgments from GPT-44o achieve 98% agreement rate with manual assessments, confirming its reliability.
B.2 EXISTING BENCIMMRRSS
Following Huang et al. 2024), we sample 100 web pages per website and exclude those with render- ing issues such as text overlap. For the SWDE benchmark, we remove two websites, CollegeTooolks and FanHouse, from the university and NBAPlayer domains, respectively. As each website has four associated cases, this reduces the initial 320 cases by 8. resulting in a final set of 312 cases. For the Expanded SWDE benchmark, we first subsample 294 attributes by aligning relations with an established attribute set and discarding outlier cases (Huang et al., 2024). We then filter out one website with rendering problems, which removes its 14 associated attributes. This yields final set of 280 attributes for evaluation. Figure] IT shows an example of an excluded web page.
B.3 HUMAN EVALCATION
To establish human performance for LIVEWEB-IE we recruit six evaluators with Bachelor's de- grees in Computer Science and experience with web scraping. We select a balanced subset 120 samples across websites and task types, distributing them equally among the evaluators. Evalua- tors write executable web scraping scripts according to these guidelines: (1) create executable web scraping scripts for each query its corresponding page group, (2) for non-textual content such as images and hyperlinks, extract the value of the corresponding source attribute (e.g., a tag for hyper- link), and (3) avoid receiving direct help from Al assistants for generating web scraping scripts. We then execute the web scraping scripts submitted by each evaluator. The results are compared against the ground truth using the same evaluation methodology.
16

---

Published as a conference paper at ICLR 2026
Type
Type
Type
Type
Fl Score (%)
Vil 1uas
Vid 10ast
Vos (aan)
Vid (Dans
Extraction Time (6)
Extraction Time (5)
Extraction Time (s)
Extraction Time (s)
Figure 8: FI score and extraction time comparison between VGS and a direct extraction approach across the four task types. The extraction time is measured in seconds (s) per page group.
C DETAILS FOR LIVEWEB-IE CONSTRUCTION
The dataset construction process involves five authors with Bachelor's degrees higher in Com- puter Science. We annotate the data through attribute identification, query formulation, ground-truth value extraction, and human validation. To identify the attributes from the web page, we the original text from the source web pages attributes (Lockard et al. 2019). If multiple values share identical text within a page group (c.g.. "link" referring both league link and team link), we add descriptive prefixes to create distinct attributes such as "league link" and "team link". For weh pages where attributes are not explicitly identified, we assign semantically clear attributes derived from the meaning of their associated values. When formulating queries, we create instructions that accurately specify the desired information. We design temporally stable queries that avoid time-dependent ref- erences. For example, "Extract logo information for all seasons" updates new seasons are added. For ground-truth truth value extraction, we extract values according to their data categories. We extract @rrc values for images and @href values for hyperlinks. These correspond to target destinations and source paths as specified in HTML standard 914
D IMPACT OF CONTEXT DISTANCE
In our XPath Synthesis stage $ 4.4). the dis-
50
tance parameter d defines the size of the lo-
F1 Score (%)
cal HTML segment used as context. This pa-
40
rameter manages the trade-off between pro- viding sufficient structural information and in- troducing noise. To investigate its effect WIE performance, we analyze by varying d E {0, 2, 4}. As shown in Figure 9 perfor- mance degrades at the extremes, specifically at and 4. We hypothesize that this is because an insufficient context (d = 0) leads to overly specific XPaths that lack generalizabil- ity, while excessive context (d 4) confuses the VLM with irrelevant information, leading to imprecise XPath generation.
30
Gemini
min-4o-mini
20
GPT4o
2
3
4
Distance
Figure 9: Performance variation across distances.
E COMPARISON WITH DIRECT EXTRACTION VIA LANGUAGE AGENT
Another approach for WIE involves utilizing language agents to reason directly over HTML content to extract desired information. To compare VGS with this approach, we establish baseline that prompts an LLM to extract the target attributes directly. Figure illustraaee the comparison result. While LLM extractor offers high flexibility, its page-by-page reasoning process incurs substantial inference time. Moreover, the direct extraction approach is vulnerable to model context length limitations because it generates all target values directly. This leads significant performance drop in scenarios that require extracting a large number values, such as Type IV. In contrast,
17

---

Published as a conference paper at ICLR 2026
Table Performance comparison between the SWDE-sub and SWDE-2025. The SWDE-sub refers to snapshot manually sampled from the original SWDE dataset, where the attributes and values remain identical the live web pages. SWDE-2025 denotes the corresponding web pages captured in November 2025, AFI represents the difference in Fl scores, obtained by subtracting SWDE-sub from SWDE-2025, which highlights the consistent performance decline across all approaches.
SWDE-sub
SWDE-2025
Model
Method
A F1
P
R
F1
CoT
95.05
85.0
82.62
97.94
70.32
68.95
-13.67
Reflexion
91.98
95.53
90.70
98.47
73.24
73.86
-16.93
GPT-4o
AutoScraper
99.47
90.79
92.66
99.06
77.48
77.83
-14.83
VGS (ours)
98.12
96.27
94.53
98.21
78.94
86.38
-8.15
CoT
99.47
71.58
73.32
95.08
57.38
56.33
-16999
Reflexion
94.62
79.74
76.67
94.12
60.89
58.79
-17.88
Qwen-2.5-72B
AutoScraper
95.57
82.11
80.0
92.40
66.16
62.58
-17.51
VGS (ours)
93.53
85.59
83.48
94.05
76.12
74.85
-8.63
VGS operates as hybrid method that employs a VLM to generate reusable XPaths, offering an efficient solution for large-scale data extraction.
F PERFORMANCE GAP BETWEEN STATIC SNAPSHOTS AND LIVE WEB PAGES
To investigate whether the WIE performance reported on offline benchmarks generalizes to current web environments, we conduct comparative evaluation of WIE systems on both static snapshots and their corresponding live web pages. We utilize the SWDE dataset, which consists of HTML snapshots from the early 2010s. From this dataset, manually curate
test set by comparing these snapshots to their corresponding current versions (November 2025). To ensure a valid compar- ison, select only web pages where the target value remains present and identical the original SWDE ground truth. This process yields a curated dataset covering 10 websites with 10 pages each, detailed in Table We then evaluate both the L.LM-based wrapper generation baselines and VGS on the original snapshots and their live counterparts.
As shown in Table 4 the performance on offline snapshots fails generalize to the current version of the same websites. With the GPT-4o model, the Fl scores for HTML-based baselines drop by an average of 15.14%, and VGS drops by 8.15%. Similarly, on the Owen-2.55722 model, the baselines show an average F1 degradation of 17.46%, whereas VGS degrades by 8.63%. This validates our core motivation and demonstrates the need for benchmark akin to LIVEWER-IIE which evaluates systems directly on live web pages. Furthermore, it suggests our vision-groundes approach is more robust to structural evolution.
G EFFICIENCY COMPARISON WITH LLM-BASED WRAPPER GENERATION BASELINES
To compare the efficiency of VGS against LLM- based wrapper generation baselines, we conduct an experiment across varying levels of web page complexity. We stratify the LIVEWEB-IE samples based on average HTML length, serving as proxy for complexity. Within each complexity group, we measure the average score and the mean inference time for VGS and the baselines.
As illustrated in Figure TO VGS exhibits higher initial latency on simpler web pages compared to the baselines. However, as structural complexity increases, the inference latency of iterative HTML- based methods (Reflexion and AutoScraper) increases substantially, eventually converging with that of VGS on highly complex pages. Crucially, VGS consistently maintains superior extraction perfor- mance across all complexity levels. Considering that robust extraction from complex, content-heavy websites represents the primary bottleneck roal-world WIE, we argue that VGS offers a practical and favorable solution for real-world deployment.
18

---

Published as a conference paper at ICLR 2026
GPT-4o
Qwen-2.5-72B
100
CoT
Reflexion
Inference Time
AutoScraper
50
F1 Score (%)
VGS (ours)
40
40
20
8
0-20
20-40
40-60
60-80
80+
0-20
20-40
40-60
60-80
Character Length of HTML (k)
Figure 10: Efficiency comparison between VGS and LLM-based wrapper generation baselines. The character length represents the average HTML character length of the web pages for each data sample.
H WEBSITE DESCRIPTIONS FOR LIVEWEB-IE
This section provides detailed descriptions of the websites included in LIVEWEB-IE.
H.1 ACADEMIC
Arxiv. ArXiv repository is a major preprint database for scientific literature, widely used in com- puter science. The platform hosts vast collection of scientific articles, each accompanied by rich metadata including titles, authors, and abstracts. For this website, we constructed queries related to extracting paper titles, author names, and direct PDF links.
Machine Learning Conferences. The ICLR ICMI and Neurlp' platforms represent top-tier machine learning conferences with standardized academic content structures. These websites host the proceedings for each conference, including lists of accepted papers, author information, and abstracts. Example tasks include finding the abstract page for given paper or extracting the URLs for all papers in specific session.
Hugging Face Daily Papers. Hugging Face Daily Paper is a community platform that aggre- gates and ranks recent Al research papers. The platform features daily feed of new papers, each with community-driven features like discussion threads, like counts, and links to associated code models. Our queries for this platform include extracting paper's publication date and author names.
10 serves as the authoritative global database for marine taxonomic information, providing comprehensive data on marine organisms worldwide. The platform features multimedia galleries, species profiles, and taxonomic hicrarchies, offering rich visual content alongside traditional text-based scientific information. Our queries include finding an image of given marine species and the name of the photographer.
World of Marine Species (WoRMS). WORMS
hhupss//rxxivorr///
"hups:/ficlr.c//
'httss//iemmlcc/
"ntts://neurips.cd
chup://mmrrnespccissorrr/
19

---

Published as a conference paper at ICLR 2026
H.2 AUTOMOTIVE
Fueleconomy. Fueleconomy serves as the official U.S. government resource for vehicle fuel economy information, operated jointly by the Environmental Protection Agency (EPA) and the De- partment of Energy (DOE). The platform provides standardized fuel efficiency data, including Miles Per Gallon (MPG) ratings, and environmental impact information across diverse vehicle categories, ranging from sedans to trucks. Our queries include finding the annual fuel cost for a specific car model and year.
H.3 SPORTS
TheSportsDB. TheSportsDB operates as an open, crowd-sourced sports database providing comprehensive artwork and metadata for sports events, teams, and leagues worldwide. The plat- form contains detailed information about football clubs, basketball teams, player statistics, match results, and toumnmmen histories across multiple sports disciplines. Our queries include extracting YouTube links for specific matches and retrieving tournament banners for given competitions.
12
H.4 LIBRARY
DPLA. DPLA serves the central portal for America's digital cultural heritage, aggregating millions of items from libraries, archives, and museums across the United States. The collection includes historical photographs, manuscripts, books, newspapers, audio recordings, and artifacts spanning American history from colonial times to the present. Queries include extracting images, links, and titles for specific articles historical documents within the vast digital collection.
H.5 FOOD
providdss comprehensive open database of meals and recipes from around the world, offering detailed cooking instructions and ingredient information. The database includes thousands of recipes spanning various cuisines, complete with ingredient measurements, cooking steps, and high-quality food photography. Our tasks involve extracting ingredient images and names for specific dishes from the database.
TheMea TheMealDE provides
TheCocktaiLDB. TheCocktailDD operates as a complementary database to TheMealDB, spe- cializing in cocktail and drink recipes with detailed preparation instructions. The collection fea- tures hundreds of cocktail recipes, including classic drinks, modern mixology creations, and non- alcoholic beverages with ingredient lists, preparation methods, and serving suggestions. Example queries include extracting cocktail images and names for specific drinks from the comprehensive database.
H.6 BOOK
Books to Scrape. Books to Scrape serves sandbox website designed specifically for web scraping practice, mimicking the structure and functionality of real online bookstores. The catalog contains 1,000 fictional books across various genres, including mystery. romance, science fiction, and non-fiction, complete with cover images, star ratings, prices, and availability status. Our tasks include extracting book titles, stock status, and pricing information from product catalog structures.
16
http::/wwww.fueeccooomt gov/
" whtps://www.tteppprssb..cmm
'huts://dp.la//
" hhttp://www.themealldee
https:/www..hccooktallb..cc
16 "htps:/hboks
20

---

Published as a conference paper at ICLR 2026
H.7 E-COMMERCE
Sandbox Oxylabs. Sandbox Oxylab provides controlled e-commerce environment specifi- cally designed for web scraping testing, featuring a realistic gaming product marketplace interface. The platform showcases various video games across different platforms, including PC, PlayStation, Xbox, and Nintendo, with detailed product descriptions, screenshots, system requirements, and user reviews. Our queries focus extracting game titles, genres, and pricing information.
H.8 OTHERS
Quotes to Scrape. Quotes Scrape operates sandbox website featuring collections of fa- mous quotes with author attribution, designed for web scraping education and practice. The site contains inspirational and philosophical quotes from notable figures throughout history, including writers, philosophers, scientists, and world leaders. organized with tagging systems and author bio- graphical information. Tasks include extracting quotes and their corresponding author names to test basic text parsing capabilities.
18
Scrape This Site. Scrape This Site
19 provides a comprehensive collection of web scraping chal- lenges with various HTML structures and content types across different pages. The platform offers multiple datasets, including country statistics with population and area data, hockey team informa- tion, movie databases with Oscar winners, and sandbox environments for testing different scraping scenarios. Our queries span extracting movie titles and awards, country names, and capitals.
CASE STUDY
We provide successful inference cases of VGS for each task type. The examples for Type I to Type IV presented in Figure Figure 29 For visualization purposes, the inferred bounding boxes are outlined with thick borders.
J PROMPT TEMPLATES
We provide the prompt templates used in experiments. The prompts for VGS organized by each step: attribute identification (Figure 12). visual grounding (Figure 13 element pinpoint-
ing (Figures 14 and 15). and XPath synthesis (Figure T6). For comparison,
also provide the
prompts for the baseline methods evaluated in our study. These include the prompts for CoT (Fig-
ures[I7 and Reflexion (Figures 19 20 and (1). AutoScraper (Figures 22 23 and 24h. and LLM
Extractor (Figure
K LIMITATIONS
Our work, while advancing the evaluation of WIE systems, has few limitations. First, the cur- rent version of LIVEWEB-IE is constructed exclusively from English-based websites and natural language queries. Consequently, the performance of WIE systems on our benchmark may not gen- eralize to web pages and user instructions in other languages and cultural contexts. Extending the benchmark to encompass multilingual and multicultural scenarios ward building universal WIE systems. Second, limitation pertains to the scope of the benchmark. LIVEWER-IE is constructed from 15 websites across 8 domains. While this curated selection en- sures stable and high-quality evaluation, the findings may not fully generalize the vast array of websites, such as highly dynamic social media feeds. Expanding the domain coverage and scale of the benchmark is crucial step in more rigorously validating generalization performance. Third, attributes and queries are restricted to information with temporally stable values. To ensure repro- ducible evaluation and maintain stable ground-trus -trtth annotations, we curated queries for information
considerable next step to-
17 "ttpss//annbbox.xxylabiiioo https://quooess.tssaap...o comm
21

---

Published as a conference paper at ICLR 2026
that is unlikely change over time. Consequently, while LIVEWEB-IE evaluates system's 's bustmess to structural drift by accessing live websites, its queries are limited to temporally stable information and do not include dynamic data like current stock prices. While this design aligns with the foundational WIE challenge, analyzing web page structure, and mitigates the overhead of continuous ground-trufe maintenance, acknowledge that expanding the benchmark include time-sensitive queries is a valuable future direction that would broaden the diversity of tasks.
L
THE USE OF LARGE LANGUAGE MODELS
Following the ICLR 2026 policies on LLMs usage, we disclose our use of LLMs in the writing process of this paper. The role of LLMs was confined that of writing assistant, helping to improve grammatical correctness and readability. It is important to note that the LLM was not used for generating core research ideas and drafting the primary structure of the paper. All model- generated suggestions were critically evaluated, and the final text was written by the authors, who bear full responsibility for the entirety of this work.
22

---

Published as a conference paper at ICLR 2026
Table 5: Results on existing benchmarks.
SWDE
Expanded SWDE
Models
Method
R
R
F1
Proprietary Models
COT
92.62
76.82
75.33
88.47
65.08
63.74
Reflexion
94.31
84.53
84.40
89.50
80.14
79.22
Gemini 2.5 Flash
AutoScraper
93.87
78.79
78.07
88.58
74.14
73.44
VGS
96.20
86.58
88.42
95.76
81.88
83.19
COT
88.01
73.21
69.85
87.97
71.59
68.92
Reflexion
92.11
68.79
68.15 15
94.14
68.51
68.05
GPT-4o-mini
AutoSeraper
91.67
77.94
77.00
91.46
79.88
78.62
VGS
95.69
81.72
83.49
92.08
72.80
73.48
COT
92.81
77.50
77.50
91.92
74.07
76.43
Reflexion
94.69
85.31
86.88
93.59
78.69
78.47
GPT-4o
AutoScraper
94.06
88.44
87.50
91.11
84.66
81.71
VGS
96.92
87.56
89.77
96.07
84.25
84.19
Open-Source Models
COT
92.35
14.40
13.99
88.16
8.99
7.60
Reflexion
90.71
15.92
14.57
92.61
4.74
4.24
Gemma-3-4B
AutoScraper
86.75
24.59
23.56
87.85
15.86
14.48
VGS
66.62
57.43
58.86
65.32
55.28
55.67
COT 87.12
49.89
47.24
82,99
43.84
39.01
Reflexion
88.22
54.27
52.02
88.94
49.15
46.29
Gemma-3-27B
AutoScraper
88.90
58.65
57.67
88.24
59.02
55.28
VGS
77.14
67.07
68.28
76.06
62.53
62.97
COT
91.91
33.91
33.36
91.21
28.20
26.68
Reflexion
93.33
39.46
39.27
91.28
35.30
34.49
Qwen-2.5-7B
AutoScraper
92.68
32.81
32.50
94.87
30.95
30.60
VGS
71.39
60.16
61.88
73.42
60.55
61.75
COT
89.65
69.55
67.91
89.10
67.87
65.08
Reflexion
95.48
71.82
71.35
94.29
67.44
67.42
AutoScraper
93.94
69.03
68.03
91.82
64.67
63.56
VGS
97.04
77.87
80.29
95.92
74.98
77.07
COT
92.83
71.91
70.54
89.23
61.00
58.76
Reflexion
94.95
73.38
73.85
91.70
65.89
65.33
Qwen-2.5-72
AutoScraper
92.78
83.51
83.02
89.85
78.63
77.83
VGS
94.38
81.65
83.66
92.95
79.75
80.74
23

---

Published as a conference paper at ICLR 2026
Table 6: Statistics of SWDE-sub, a manually curated subset where target values remain identical to recent web pages.
Domain
Website
Attributes
# web pages
espn nba
name
10
name
10
NBAPlayer
usatoday
height, name
10
yahoo
height, name, weight
10
wiki
height, name, weight
10
allmovie
director, title title director, title director, title director, title
10
boxofficemojo
10
Movie
imdb
10
metacritic
10
rottentomatoes
10
24

---

Published as a conference paper at ICLR 2026
College Links: College SearchCompare Student Loanst Scholarships
Granniaal College Evplration AdwIssoos Aid and Bboking for acilesibeekallep option?
Paanner PGdken
Colpare
Sellul through, Rundeeds wiins
Usernume
Logtt)
In right for youl We lixt degreee from
Lackos
Hormopagge
At Anedittional
eccrecited online programa
Callege
Search
Minda
Concentration:
Searoo Iy
Nojor
Degree Typet
Degree
Schools:
Search
Become Nember
Lonz
Tide
Compare
Online
Buxnmoo School Sertion
Local Contes
Malbara
Artanses
Figure example of a web page excluded due to text overlap.
25

---

Published as a conference paper at ICLR 2026
Table 7: Dataset statistic (Part 1): Academic and Auto domains.
Domain
Website
Attribute
# web page # Group
author
author profile link
ArXiv
pdf link
87
3
subject
title
abstract link
author
ICLR
292
2
author profile link
title
abstract link
author
NeurIPS
311
2
author profile link title
abstract link
author
ICML
225
2
author profile link
title
Academic
author
author count
citing model link
model citation count
Hugging Face
230
2
paper link
publish date
source link
title
author
author profile link
country
editor profile link
editor type
group
WoRMS
group image
73
4
group link
institute
photo title
species
species image
species link
annual cost
driver mpg
tank size
Auto
Fueleconomy
vehicle
69
3
vehicle image
vehicle link
vehicle type
26

---

Published as a conference paper at ICLR 2026
Table 8: Dataset statistic (Part 2): Sports, Library, and Food domains.
Domain
Website
Attribute
# web page # Group
banner
event link
fanart
league date
league link
lineup
logo
member image
member link
Sports
TheSportsDB
261
poster
profile link
search link
season badge
scason poster
team
team badge
team link
thumb image
youtube link
article image article link
collection
collection image
collection link
cxhibition link
Library DPLA
source image
184
5
source link
source title
subject link
title
topic
topic link
flag image
ingredient ingredient image
TheMealDB
ingredient link
298
2
meal
meal image meal link
Food
cocktail
cocktail image
cocktail link
TheCocktailDB
ingredient
ingredient image
ingredient link
27

---

Published as a conference paper at ICLR 2026
Table 9: Dataset statistic (Part 3): Book, E-commerce, and Others domains.
Domain
Website
Attribute
web page # Group
book image
hook link
category
count
Book
Books to Scrape
1100
price
stock status
title
upc
developer
game count
genre
image
E-commerce Sandbox Oxylabs
1143
link
platform
price
title
arca
awards
capital
country
goals for
nominations
population
Scrape This Site
team title turtle image
32
Others
turtle link
turtle name
wins
year
author
author profile link birth
Quotes to Scrape
70
3
quote
tag link
28

---

Published as a conference paper at ICLR 2026
VGS Prompt for Attribute Identification
You are an Abbti Patracior Chab converts usec estiaaction seguesta into stractared JaDN payloads for web scraping systens.
Transtorn usee's data what attributes extract.
request conpact apecification that
Instruntioons
ideatify apecific peints they exizacted rocue the mininal set of sttributer needed tultill user's request
the
Bules:
JSCN Structura: Return cSCN object with exactly one key: "actrizutes"
The "actrinutas" key mast contoin a list of actriaute nanes
Kininal Suspe: Inclade only abiributes explicit's ielitiily requeated ar cleatly implied by the cserfa Lask Output Ouly: Beturn the JSDN abject with no code Sences, explanat lons, or addil additional cex:
Please output in the lowing J50N fornac:
"ssteibutso":
"antribors_nams_..
Figure 12: VGS prompt template for attribute identification.
VGS Prompt for Visual Grounding
You are vieual grounding acsistan: that identities Kalch webpage coctaine specitic UI element tron set of viesport screenshote.
Task: diven a sat of viavpert scresashots and sesrested atcrabuze, decernine which specific regionn anyl poncaing the target atcribute clearly visible within boundaries.
Instrumt ions:
receive multiple viesport screenshste representing
zegions Nebpage
screenshos independently region ideatitier Aralyze
containa carest attribute
Rules:
Evaluate each region screenshot individually for the presence of the target attribate
If mull mpligle regions contain the Lazget altribuls, selvct the one dhece is most elearly visible
and prominent
Do not nake ansumptions about page scructure or iater elerents beyond what is directly observable
Do not assune clanenca exist basad on typical seboaqe patterns or complates
Pocus only on eiencazs that are clearly sithin each reqion's viewport boundarios
Return only the sing best metching region
Please culput
fullowing J80N Formel:
regionn: tratehing_reessont
Single regior
target
Figure 13: VGS prompt template for visual grounding.
29

---

Published as a conference paper at ICLR 2026
VGS prompt for Element Pinpointing (Element Scanning)
aze precise visual extractor analyzes screenshot excract apecific actribuce modality claseicication.
sttribute viewport screenshot
attribute deternine the nodelity from perfern appropriste extraction task.
Inatruct.ionz
classify the actribute into exactly one modality based naming patserns:
1, TEXT Modalsty - For textual I actribuzes
Escract all visible text strings exectly as they appear
Preserve natural reading order (tep to botton, left to right)
No text normalizatinn zstion or peraphresing
"tags" key MUST be emppy.
Ourpot Kodalisy
attributes
line only
Ignore
emply
IMAGE Nodslity
visual contert attributes Cutput "ino" tags only
Ionare any text ovezlays on inages
"texta" key MUST be empty
Rulea:
Select exactly ene modality - never nix text and tag outpots
For TEXT modality: desupiicate while preserving reading order
EYPERLINK
nodalitien: lovercase EIN caq nanas only, keep mininal genezic
output in lowing
"Lexaa": TEXP midalily: visible
"tags": For non-TExT modalities: Lovercase HTML seading octles, wnactly tlay appeas
Figure 14: VGS prompt template for element pinpointing (element scanning).
VGS prompt for Element Pinpointing (Element Selection)
preci viaual elenent seleczor that anslyzes annotated region screenshot identiffy specific DI alements matching a target atcribute.
Task: Siven an annocated region scrcenstor where petential condidato elements have colored bounding boxes with unique nuierr ical labela, select the exect. boanding box I0s thei contain valuee Jea for the specifled tazget atcribute.
Instruct lone:
Sach sisment outlined
top-richt corner colored background matching the box Jocus on elements that directly cortain represent the values
box containias snall label
shoxs intoger
white
tasget atcesbure
Rules:
Relevance Only: Return
bounding boxes that directly contain values the target
attribute
Leaf-Level Selecticn: when aested boxes exist, prefer the innezmos: hox ccntaining the Fax data Complets Caverage: Return all natching icaf elencats visitle on screen, net just the cirst
occusrence
No Asiiguuoua containees: Avoid selsecing wzapper diva, sections, or containers whea laser data
elementa exist
Whiteliet Only: only outpot IDs that exiet in the screenshots do not invent or guese numbers
Plezse output only integce
simpis 11st:
Figure 15: VGS prompt template for element pinpointing (element selection).
30

---

Published as a conference paper at ICLR 2026
VGS Prompt for Xpath Synthesis
You are an expert XPath generator that creates rabust, generolizable colecters for web scraping on segnenls, nacked zegiuny Lazgel altzibace.
screenshot, target xpata
provided segrents, regior structural
pages,
Instructione;
HIMI segments marked
identity structural
stable anchoring
elenenta
Use the mazked zeglon screenshot as visual context to understand the carget alemenz's lccation end appearance
Rules:
-Battorn Rezognition: Identity comnon structural potterns acrocs all HTML segments
Context Intopration: Leverage bota vicual evicence and locelized 87ML internation
suneralisablligg Ensure works seliably acrosa sinilas page atructuzes Içnoze irrelevant or nolay samples forming genaralized solector
Please autput in
lowing
"xpta";; " Single XPath solector targeting appropriste
Figure 16: VGS prompt template for Xpath synthesis.
CoT Prompt for Top-Down Operation
good reading mnderstanding
give
Please read telleving HTML code, return XPethe elemente in the satietying the instructione belox.
tiold tast directly
Instruction: [0]
Rulea:
Field wetemninal lunt
Use explicitly listed tield rames tron the instruction.
otherwi se, anter ninimal relevart fields present in the DCM without inventing uncupported tields
XFach Construction:
Aveid embeiding exact literal velues visible
HTHL content
Dowfer
altribules hiezarchieal relationships britlle idettifiess
Field Organazationn XPathe
Generate
maitiple nodes exist instruction reguirenents for each Field
tield
Maintain inst XPathe that
Data Structura Raquirenents:
Ensuzo spath and value dietionaries have identical key sets
ili the XPath and value lista so each position oue responds to the sane Larget rode Sxtract Lext from single nodes viuh concalenel ing sentenl from mulciple elesenta
Kissing Informat ion Bandling:
Return empty lieta tor explicitly enumerated tielde when NTM lacks suitable content Return empty ohjects for both spach and veluee when no inst rettuucion fie are apeci Fied and no content found
Please output in fullowing
"toought*; Beiet reasoning field selection and XPath decivation of extrected text cont ent
tualme"; Fie' e-to-string-11k + dynamic keys
Sxpatif::
Field to XPath list napping for Larget node selection () dynanie keys
Bocora ocde:
Figure 17: CoT prompt template for top-down operation.
31

---

Published as a conference paper at ICLR 2026
CoT Prompt for Synthesis Operation
are pezswos inieeeni shich is guod RTHL understanddng as well.
Pollowing ruction,,
sequence written several.
cornespending
potentiall
edepted
webpeges
Bere inseructions the tack: Inatructione: (0)
The action sequences and the corresponding extracted results with different sequences on different vebpages are es fel ows:
(1)
Please output in the tollowing JSON tornat:
"thoughe": ** thinking abouz
sequerce
candi
Figure 18: CoT prompt template for synthesis operation.
Reflexion Prompt for Top-Down Operation
web parsor is good reading undorstanding
can cive clear
following HTML oade,
uscuped field that directly
elaments in
astictying the instructione
Instruction: [0]
Rules:
Field Deterninat ion:
Use expl
icllly liy Listed field cames fron the instruction.
otherwise, inter ninimal reievant tields present in the DCM without inventing uncupported tields
XPath construction
Aveid enbedding exact literal
visible
HTML
Create structurally sebest selectors using attelbate and structurai patterns
attributes
relationships
idertifiers
Field Organizza Waths
maltiple aodes exist field instruction requivenents
Maintain distinct XPathe that
cield
Data recttuee requirenente:
Ensuue spath and valus dictionaries have idantical key sets
Align the XPath and value lists so each position corresponds the same target node rxt text from single nodes vishout concatenal ing pontent from nulsigle elanents Kiasing Informat lun Bandling:
Retuin ampty l:ata for explicisly emamrrated fields vhen HEs lacks suitable conteat Return eepty objects for both xpach: and value when no instruction Helds are specities and no suitable content found
Please output the tolloxing J30N format:
thoouhh" Briet ceasoning field select XPach derivacion ot extrected text content
tvalue"; tre.eete-stringg-
a dynemic
"ppttt;; Hiole-to-xtath-liit napping for target nedo selection () dynenie keys
Berefs RTN,, coder
(11
Figure 19: Reflexion prompt template for top-down operation.
32

---

expected ues, Published as a conference paper at ICLR 2026
Reflexion Prompt for Self-Reflection Operation
Plazse read the tel-cking HTML codd, and then retura all possibls XPaths that can recognize the alenwata
matching the instrsotion belox.
Inatruction: [0]
an
Evaltiats cincistency retwesa extraction resuits expected valuas incerrectly captured Clack for eaply sesults indicacing failed exiract ion
Identity isrelevant clements
Accept sav valuss with zedundant sepasatora as consistent, since post-procesaing will handle the
Value Assessment:
Te exani in expected values in the context of the HtM structure
Detemiine aptinal Xeath stratagios foz locating target content
Conside= strectural patterns and element reletionshipe for reliable targeting
RuCinement:
Genezate stining
curreat ores they demorscret
nest cequizements quality
decixions analysis findings
enbeiding
literal
aTM with ditterent neanings
overly
selectors that
muttilee
Use specific fic class sttributes and positional Lndicatora te differentiate tazget rodes Haintain precision through atractural anchors and attribute-based tarosting Hi asing Content llnn ing:
Return empty out puts when ITM. lacks information matching the instrucsion Anknowledge ex esctaclion lon Limi iniiation lona callie Chan foruing inappropriace nalihhe
Please output in the tolloxing J30N tornst:
"thought";
reasonicg
selaction XPath derivacion. extracced
yea/no direasly
"valuu": Pield string list napping
"kpath"; 1 Pield-ce-tRtta-list napping for tazget node selection
() 1 dynamic keys
And herefe the history about the thoughts, XPaths, and resuits extracted by the crexier: (11
Begera oode:
Figure 20: Reflexion prompt template for self-reflection operation.
Reflexion Prompt for Synthesis Operation
are porfect dLscrininator,
good HIML undarstanding well.
Pollowing the inst
BINL, sequeene writlen palentially
uociwapeiding adapted the excraction
other
vabsice.
Here the thne ruot lons for the tasks Instructione: (0)
The icn sequentes and the cocresponding extrasted resulta with differen sequences on diween velpagea are as follows:
(1)
Please autpot in the lowing 350N forne::
"thought"; en thinking about
ex
awguerse
candi
Figure 21: Reflexion prompt template for synthesis operation.
33

---

Published as a conference paper at ICLR 2026
AutoScraper Prompt for Top-Down Operation
aze wab parser good reading understanding browser.
can give
read eewing иMM,
grouped Field that direct
claments in the astictying the instructions
Instructions [0]
Rules:
Vield neterninat ion:
Use spllicily Listed field cames from the inst ion
otherwise, inter ninimai rolevant fields preseat in the DCH without inventing uncupported rislds
xeath Construction:
Aveid enbedding exact literal
RIML content
Create structurally sebest selectora attelbate atructuraz patterns idertifiens
relationships
Wield Organizazionn Watha
nultiple modes exist instructicn requivenets
field
Haintain distinct XPathe
siold
Data Structure rqquireeent::
Enauze spath and value dictionaries have idantical key sets
Align the XPath and value lists so each position corresponds the same tazqet node ret text from single nodes wishout concatenst ing content from nuletple elenerts
Kissing Informtt ion Handl ing:
Retuin emply Lass for expl ieicly emamrrated fields when HEa lacks suitable conteal seturn erptý objects for both xpact and value when no instruction tiels are specitled and no suitable content 15 tound
Pleaso output
colloxing J808 format:
" Brief zeasoning field select WPath decuvalion extracted text content
"value"; sta.e-te-string-iie
dynamic
"ppatt"; 2lole-to-KBath-list napping for target neds selection (1 dynemic keys
Here'' the HTNL code:
(1]
Figure 22: AutoScraper prompt template for top-down operation.
AutoScraper Prompt for Step-back Operation
al1 expected which And here!'s the value (noter there will be most 13 expected values! The iee bode is fol lowss
judig the following ressognizedd beforehand. instructioos
(2)
Please output in the following J3ON fornac:
"thought + briwf thinkinn mng aboue whether ITT, code consaine expected "usgemmett": whether the EINL code contains extracted valcos. Betuen yes/to directly.
Figure 23: AutoScraper prompt template for step-back operation.
34

---

Published as a conference paper at ICLR 2026
AutoScraper Prompt for Synthesis Operation
You are perfect discrininetor, which is good at ITM, understanding as well.
Felloving instruction, there some action sequeace writton severol sorseapending
polentially
adapied the
otherz aubgages
vebsi.e.
Here the instructions the tack: rustrrct lons: (0:
The action esquances and the cozrespending extracted recults with dirfercaz sequeness different velpaage are es follows:
(1)
Please output in the foll lowing J50N fornet:
"thought":
thinking about
"numees"
sequense chosen zrom candidatos,
puspat
Figure 24: AutoScraper prompt template for synthesis operation.
LLM Extractor Prompt
Exsractor perforns STRECT sing page
Rules:
Inlez abtribute nanes fron the cawz inaczuction in natural language Return impty lista for attributes kish values
octpet ONLY the cinal 3SCN object wish NO additicasl text or explanaticos
Bern are the instruct lons of the task: (03
EINS: (1)
Please output collowing JaON Fornac:
"cattributes"; ("valuei", "valca2"]
for
Figure 25: LLM Extractor prompt template.
35

---

Published as a conference paper at ICLR 2026
Step Attribute Identification
Step Visual Grounding
youtube
Attribute
Give me the youtube the game
youuuub link")
LLM
Step 3: Element Pinpointing
Step Xpath Synthesis
Attribute: youtube 1ink
Attribute:
youtube
15nk
XPathh
[eide featuree
j/div/div[p1//iv 2]/contor(1]/At]
VLM
da Ionelee heteppeeepp
"outlube
Figure 26: Complete VGS trajectory for Type task LIVEWEB-IE.
https://hookss http://hhokks.sscapp....o wwwpssttt.tcccccccatottocceeotee httpp//ccaaaaccauu/nntt.. day green smoothie cleanse lose-up-tee- to 1o-15--ounnd--n 10-days 581/Index.html
Step Attribute Identification
Step Visual Grounding
Altribute:
title
Astribute
Extract and UPC of the book
("eitle" "upc"")
Attribute:
VLM
Step 3: Element Pinpointing
Step 4: Xpath Synthesis
Attribute: title
Actribute: title
XPath
di content
j/articlo/d [v
Attiri bute: upc
VLM
Attribute:
upc
VLM
1/" [/"dd ['lddd''content inner |/article/E
able/tbodyttn(1]
Figure 27: Complete VGS trajectory for Type II task on LIVEWEB-IE.
36

---

Published as a conference paper at ICLR 2026
Step 1: Identify Attribute
Step Visual Grounding
Attribute:
Attribute
HITE
Please extract sub toplic titles
("eppi sab eistitt
LLM
Step 3: Element Pinpointing
Step 4: Xpath Synthesis
Attribute: topic sub title
Attribute: topic sub title
Wldd //
VLM
/ /a/div[2]/h2
Figure 28: Complete VGS trajectory for Type III task LIVEWEB-IE.
https http://www.....ooeooeooooeeeeeee oww.://wwwooooomo.....------eeed 2tt.://ee00e---oaaaaaoooee------
Step Identify Attribute
Step Visual Grounding
Attribute:
("car nane", "car image")
and image
LLM
Attribute:car Image
VLM
Step 3: Element Pinpointing
Step 4: Xpath Synthesis
(eelee row' 1//0
Attribute: car Image
VLM
Attribute: car Isage
VLM
17*De "main- content [1stt [11/tabde/tbedn tt////////////t dEal/ama(2)
Figure 29: Complete VGS trajectory for Type IV task LIVEWEB-IE.
37

---

Published as a conference paper at ICLR 2026
A
D
P
GRANII
Exhibitonns
collection
voliiis and iessums aocul iiald paskes outinsse int des steer
Exhibitions
hrgjet
Sten es of onai dignificacce drown from source materials in librates, anchives, and museurs across the Ji Rtdd Statos.
CASE
Years
Empire
Tragody
Ane ensuen Brigkr
Butlle Ballut aane
Trepuiig Non
Political Ussiddess Presidercia Elections
The Murterr Mary Phagan and the Lanching of Leo Frank
Ohe
ganc of froee
(A)
(B)
Figure 30: Comparison of layout changes on the DPLA website over time. (A) illustrates the web page from a past version (January 2018), which retrieved from the Web Archive (https: //web.a a rchh rehive..ii.. weeeev..o org). digital archive preserving over trillion historical web page snapshots, while (B) depicts the current version (November 2025) of the same URL. The attributes "title", "article image", and "article link" retain identical values across both versions, but the visual layout and underlying DOM structure of the page have diverged. By targeting attributes whose information remains consistent regardless of layout changes, we enable stable evaluation of WIE system's ss information extraction performance on web pages as they exist at the current moment.
TheMealDB
THEMEALDB
Thai Green Curry
Thai Green Curry
(A)
BB)
Figure 31: Comparison of layout changes on the TheMealDB website over time. (A) illustrates specific meal page from past version (October 2017). while (B) depicts the current version (November 2025) of the same URL. The attributes "flag", "ingredient image", "ingredient link", "meal image", and "meal" retain identical values across both versions, but the visual layout and underlying DOM structure of the page have diverged.
38


Reviews:
[{"review_id": "vaGboF8IPm", "reviewer": "Reviewer_v3S5", "summary": "The paper tackles the task of Web Information Extraction (WIE) directly on live websites, while prior work has mainly considered static websites.  For this, they introduce a new benchmark LiveWeb-IE which existing Web IE approaches seem to struggle on. The paper also introduces Visual Grounding Scraper (VGS) that leverages VLMs to narrow down the relevant web element to extract desired information. Experiments show that VGS outperforms current WebIE approaches on LiveWeb-IE while also demonstrating improvements on existing WebIE benchmarks.", "questions": "1. Given that the use of live websites within this benchmark creates the problem of information on these websites changing over time, how do the authors  plan to address this? Will the answers regularly be refreshed, similar to FreshQA [2]?\n\n[2] FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation; Vu et al 2023", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The methodology is described well and the paper is easy to read. \n2. The experimental results are pretty comprehensive, with a variety of backbone LLMs used. \n3. The LiveWeb-IE benchmark can be a very valuable resource to the research community.", "weaknesses": "1. The novelty of the VGS approach is limited. The method mainly incorporates VLMs for prompting to narrow down relevant web elements, with the XPath generation part already being done in prior work [1]. \n2. The related work section is pretty lacking, with no discussion of distinctions/comparisons of VGS with prior WebIE methodologies.\n3. While VGS is relatively performant, the authors should also show a cost comparison with prior baselines. The approach of iteratively pass all regions of the webpage to VLM can be cost intensive. The paper does not have any discussion of the additional cost of the proposed approach. \n\n[1] Automating xpath query generation using nlp for streamlined web crawling and gui testing; Kaur et al 2025", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DuuEMDcwBq", "reviewer": "Reviewer_RXSR", "summary": "The paper introduces a new benchmark, LiveWeb IE, which focuses on addressing the limitations of previous offline benchmarks that capture only a fixed snapshot in time. LiveWeb IE is claimed to evaluate models directly on live websites, thereby reflecting their performance on temporally evolving web content. The paper further proposes a Web IE framework, Visual Grounding Scraper, which leverages visual cues as guidance rather than directly locating the target element.", "questions": "What is the key difference between the WIE task and a subset of GUI tasks (where the agent do not need to perform action but simply perform the perception)?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-Empirical results show that the newly proposed benchmark is more challenging, and the performance on state-of-the-art LLM/LMMs are less saturated; showing a gap between WIE systems and humans on more up-to-date live websites.\n-The proposed VGS framework is effective on both closed-source and open-source models\n-The paper writing is clear and contains sufficient ablations on the VGS components", "weaknesses": "-While the LiveWeb-IE benchmark is claimed to be “evaluating directly against live websites”, it is not clear how the benchmark automatically evolves as the website updates over time. The dataset construction pipeline is still based on a snapshot of a certain time and requires human verification to curate the data. It is potentially an overclaim that the benchmark is “Live”.\n-It is also not clear how to handle layout changes through time while still keeping the evaluation/annotation valid; and how much human efforts are required to keep the benchmark up-to-date\n-The WIE task is very related to GUI and Computer Use tasks (especially in the settings where the html/a11y tree is available for the perception step); there is a lack of discussion and comparison with the widely studied GUI agent models and literature.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nSVO1gJSW9", "reviewer": "Reviewer_uvkz", "summary": "This paper introduces LIVEWEB-IE, an online Web Information Extraction (WIE) benchmark, and proposes a multi-stage agentic framework called VGS (Visual Grounding Scraper). By mimicking human cognitive processes, VGS accurately identifies the information that needs to be extracted. Experiments demonstrate that VGS outperforms existing methods on both LIVEWEB-IE and several established WIE benchmarks.", "questions": "No specific questions. Please refer to the weakness section for detailed concerns.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The proposed Visual Grounding Scraper (VGS) framework that mimics human information-seeking behavior on web pages is novel and practical. Multi-stage visual grounding (region → element → XPath) effectively reduces HTML noise, achieving great performance on both LIVEWEB-IE and other offline benchmarks.", "weaknesses": "1. Weak Motivation. While the paper argues that performance on offline benchmarks fails to generalize to live websites due to temporal changes in web structures, this claim lacks sufficient empirical evidence. For instance, there is no direct comparison showing how existing methods degrade over time on the same websites, nor quantitative data on the frequency or impact of such changes. This undermines the core motivation, as it's unclear whether the offline-to-online gap is as significant as asserted, potentially overstating the need for LIVEWEB-IE.\n2. Reproducibility Undermined. Live evaluation causes inconsistent results across runs/time windows, breaking fair comparison and replicability. Website states can vary at each evaluation (e.g., due to updates in layout or content), leading to inconsistent results across runs. Different systems or papers might be tested in non-overlapping time windows, making it impossible to ensure fair comparisons.\n3. Expenditure and efficiency in VGS. The multi-stage VGS framework relies heavily on VLMs for visual grounding and pinpointing, which could incur significant computational costs and latency, especially for large-scale scraping. The paper does not discuss efficiency metrics (e.g., inference time or resource usage) or optimizations, making it less practical for real-world deployment compared to simpler HTML-based methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "w3xMtnrint", "reviewer": "Reviewer_D9Lx", "summary": "The paper presents LIVEWEB‑IE, a benchmark for evaluating web information extraction systems on so‑called live websites, and introduces a baseline model, Visual Grounding Scraper (VGS). The benchmark is constructed through website selection, annotation, and verification, emphasizing content stability for reproducibility. However, since it deliberately focuses on static webpages and stable answers, the setting does not clearly demonstrate advantages over conventional snapshot‑based benchmarks in reflecting real temporal dynamics.", "questions": "Could the authors clarify what specific phenomena of “liveness” are actually captured in the current setup, beyond what a static snapshot benchmark would provide?\n\nGiven that the dataset construction deliberately filters for stable content, how does the current benchmark evaluate a system’s robustness to temporal or structural changes—if at all?\n\nMultimodality is highlighted as a major aspect of the work, but its functional role is unclear. Can the authors specify which tasks or results concretely rely on non‑textual modalities, and what insights would be lost if these were removed?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The benchmark clearly defines four task types covering single/multi‑attribute and single/multi‑value extraction, providing a systematic and comprehensive framing of the web information extraction problem.\n\nThe paper presents a coherent pipeline for website selection, annotation, and verification, showing solid engineering effort.\n\nThe introduction of the baseline model (Visual Grounding Scraper – VGS) illustrates the benchmark’s intended use and offers an initial reference for comparison.", "weaknesses": "Although the paper highlights the live nature of the benchmark, most of the evaluated webpages are essentially static and rarely updated. As a result, the work does not clearly demonstrate any advantage of using “live” data over conventional snapshot‑based settings that are already stable and reproducible. Even HotpotQA, which is built on fixed Wikipedia content and unchanging commonsense facts, offers a comparable level of variability.\n\nThe benchmark explicitly filters for stable pages and attributes “unlikely to change,” ensuring reproducibility but sidestepping what genuinely makes web extraction live—coping with temporal and structural drift. A truly live evaluation should treat the target as a function responsive to time and environment rather than a fixed, time‑agnostic value.\n\nIncorporating multiple modalities in web information extraction is intuitively reasonable, but the paper does not clearly explain how these modalities are functionally integrated or why they are critical to the benchmark’s objectives. The description and experiments leave it unclear whether multimodality influences task formulation, model behavior, or evaluation outcomes, making this aspect feel under‑motivated despite being prominently emphasized.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)