GEM: Gestalt Enhanced Markup Language Model for Web Understanding via Render Tree
Authors: Zirui Shao
Feiyu Gao
Zhongda QI
Hangdi Xing
Jiajun Bu
Zhi Yu
Qi Zheng
Xiaozhong Liu
Venue: EMNLP 2023 Main (2023)
Keywords: Gestalt
Markup Language
Web Understanding
Language Model
TL;DR: The Gestalt Enhanced Markup (GEM) Language Model is proposed, which flexibly incorporates heterogeneous visual information from rendered web pages into the LM without adding visual modality input.

Abstract: Inexhaustible web content carries abundant perceptible information beyond text. Unfortunately, most prior efforts in pre-trained Language Models (LMs) ignore such cyber-richness, while few of them only employ plain HTMLs, and crucial information in the rendered web, such as visual, layout, and style, are excluded. Intuitively, those perceptible web information can provide essential intelligence to facilitate content understanding tasks. This study presents an innovative Gestalt Enhanced Markup (GEM) Language Model inspired by Gestalt psychological theory for hosting heterogeneous visual information from the render tree into the language model without requiring additional visual input. Comprehensive experiments on multiple downstream tasks, i.e., web question answering and web information extraction, validate GEM superiority.

GEM: Gestalt Enhanced Markup Language Model for Web Understanding via Render Tree
Zirui Shao', Feiyu Gao", Zhongda Qi', Hangdi Xing',
Jiajun Bu', Zhi Yu't, Qi Zheng', Xiaozhong Liu
'Zhejiang Provincial Key Laboratory of Service Robot, Zhejiang University Allibbba Group Worcester Polytechnic Institute {shaozirui, qzd, xinghd, bjj, yuzhi renzheJuuze, edu. cn feiyu. ibaba-iaann inc. com, yongqi zq@taobao. com, xliul 4elp14wwpi..m edu
Abstract
Rendered Page
Xprna
Inexhaustible web content carries abundant perceptible information beyond text. Un- fortunately, most prior efforts in pre-trained Language Models (LMs) ignore such cyber- richness, while few of them only employ plain HTMLs, and crucial information the ren- dered web, such as visual, layout, and style, are excluded. Intuitively, those perceptible web information can provide essential intelligence facilitate content understanding tasks. This study presents an innovative Gestalt Enhanced Markup (GEM) Language Model inspired by Gestalt psychological theory for hosting het- erogeneous visual information from the render into the language model without requiring additional visual input. Comprehensive experi- ments on multiple downstream tasks, i.e., web question answering and web information traction, validate GEM superiority.
10%
385
52"
19u
442
2900nst
View
revia
Confuse Focus
IS
Anawe!
Steeiigg
Lo S-
HHirarchy
Question:
Whht is the key performance of Xperia?
Figure 1: From the HTML code view, the page consists of seven siblings <li>, which can be divided into three groups from the visual perception view (represented by differently colored boxes). By taking hierarchical relationships of the visual view into account, the system can answer the question "What is the key performance of the Xperia?" better than only considering the sibling relationship.
1 Introduction
Web pages serve crucial carriers for humans to acquire and perceive information. Due to the sig- nificant wealth of these documents, long-standing efforts have been undertaken address web under- standing tasks (Chen al., 2021; Hao al., 201 Dong al., 2014; Escudeiro and Escudeiro, 2009; Yn et al., 2012). However, understanding web pages can be challenging for automated systems compared to humans, as the design of layout and visual style caters specifically to human perceptual patterns, thereby facilitating comprehension.
while neglecting the advantageous visual informa- tion of rendered pages, such as style and layout, which essential for web understanding. Several recent works investigate web pages as pictures of screenshots (Xu et al., 2020b,a; Vishwanath et al., 2018; Garncarek et al., 2020; Huang et al., 2022), emphasizing the importance of the fixed layout of the input image. However, the dynamic nature of web page rendering can lead to significant ap- pearance variations across devices and browsers, resulting the limited applicability of such mod- els. Additionally, these methods require an OCR² system to extract text with coordinates, which leads to extra overhead and neglect the hierarchical infor- mation provided by the markup language.
Recently, the pre-trained Language Models (LMs) (Li et al., 2022; Deng et al., 2022) advance web understanding and demonstrate superior per- formance on various related tasks by jointly pre- training on text and markup information. Neverthe- less, these models oversimplify web pages as plain HTML (HyperText Markup Language) sequences
"Equal contribution.
Corresponding author.
"https://en. htti::://en..ikipedii.. www.wwwwwwii...... org/wiki wiki http:/pptcllcchaaaaaaae ter ter_recoonition
hhttps: //en.

---

Regardless of render conditions, humans are ca- pable of rapidly comprehending the content of ren- dered web pages, which indicates that human visual perception is an effective way to consume web se- mantics (Xiang et al., 2007; Xu and Miller, 2016). To leverage this advancement, we propose to en- code visual information of web pages into the LM flexibly via render-tree-enabble pre-training tasks. It is inspired by human perceptive patterns rather than taking visual features as input directly. Gestalt psychological theory (Wertheimer, 1938; Koffka, 1955), a prominent cognitive model, explains the human perceptive processes that elements with similar visual styles and proximate locations are commonly regarded to be of similar semantic func- tions. For instance, as Figure I depicts, while the seven nodes of given web page are siblings in the markup language, their different rendering ap- pearances and positions categorize them into three semantic blocks. Without visual awareness (i.e., only using HTML tags), is challenging to answer downstream questions like "What is the key perfor- mance of Xperia?". However, when considering the visual cues of the web page, particularly the answer is comprised of nodes in the same row and color, this question becomes rather achievable.
web pages, which combine the DOM and CSSOM (CSS Object Model) that contain comprehensive information about rendered web pages, including textual, structural, and visual information. In prac- tice, GEM can learn a strengthened representation of markup language with the enhancement of visual prior knowledge.
To validate GEM superiority, we conduct experi- ments on two downstream tasks, i.e., web ques- tion answering and web information extraction. GEM consistently surpasses several strong base- lines. Moreover, we verify the model architecture adaptability of Gestalt objectives. In addition, we compare the performance of GEM and large lan- guage models (LLMs) (Brown et al., 2020; Ouyang et al., 2022a; Chowdhery et al., 2022).
Our main contributions are as follows:
The proposed GEM model introduces a render tree as a powerful approach to enhance the pre- training of language models with considerable visual knowledge acquired from web pages.
Based on Gestalt psychological theory, two innovative Gestalt pre-training objectives have been proposed to enable visual perception of GEM, which has been proven beneficial for various downstream tasks.
In this paper. we propose Gestalt Enhanced Markup (GEM) language model designed to host visual knowledge into the language model with- out additional visual input requirement. The GEM utilizes visual information of rendered web pages solely during the pre-training, while during fine- tuning or inference, the rendering unnecessary.
The pre-trained model and code of GEM are publicly available at GitHubS.
Preliminaries
To incorporate visual perception into language models, two further pre-training objectives are pro- posed, each relating one Gestalt law. The one is Same Textual Style Prediction (STSP), which based on the Gestalt laws of similarity that humans tend to perceive objects with similar appearances as a group. This task enlightens the language model with an appearance perception to learn semantics. On the other hand, the Proximate Nodes Predic- tion (PNP) enables the model to understand the relationships of elements from visual positions, only from the D (Document Object Model) tree. The PNP task proposed on the Gestalt prox- imity law, according to which humans consider objects close to each other perceptively coherent. Our corpora are built from render trees of rendered
2.1 Review of Gestalt Theory
Gestalt, a German word, is referred to "unified whole". The Gestalt laws proposed by German psychologist Max Wertheimer (Wertheimer, 1938) describe how humans group elements in percep- tion. Graphic designers use these laws arrange elements on web pages and other interfaces (Gra- ham, 2008). Web pages that violate the Gestalt laws result in comprehension difficulties due to mis- matched semantic and perceived structures (Sani and Shokooh, 2016). Thus, such pages are likely to be phased out (Xiang et al., 2007). We can assume modern pages are mostly well-designed and follow Gestalt laws.
"https://ww.ww wwtp:/wwww.ro.oo.o...mmooo www..oog////csoo111///
"nttps://github.com https///ggtttaccooolllaaaaaeeerr
hhttps://ent htttp:://enwwikipedia. http:ippiia..ogiiii//oooooo wwt..//ik//bccuuett.ooo Model
tterelaccheee//rree///////oooeet /GEM

---

This paper applies two Gestalt laws, similarity and proximity, enhance the LM with visual per- ception, as detailed below.
Sinstartty
Pelvoe Avd33
OY 09%
363 TECEAA
#ebul
Low
Pronieity
19MI 200g
RAM logedi
2900mlh
The Gestalt law of similarity. According to this law, similar objects are perceptually grouped (Wertheimer, 1938). In web pages, the similarity is based on rendered appearance. For instance, in Figure 2, the font color of the upper two sellow-framed nodes white, forming a group related to the popularity of the phone.
Detuet asc
High Presiiity
Figure 2: Illustrations of the Gestalt law of similarity and proximity.
Methodology
3.1 Preparing Inputs
The Gestalt law of proximity. This law states that nearby objects are perceptually grouped (Wertheimer, 1938). In web pages, proxim- ity measured by rendered positions. For instance, in Figure 2, the lower two green- framed nodes are close, creating group that highlights the key performance of the phone.
Web pages are easily gathered at scale, yet they not provide straightforward supervision for vi- sual perception. Thus, several pre-processings are conducted during pre-training. We filter out pages that are non-renderable using headless chrome and render the remaining ones. Next, we randomly sample a rendered page from the remaining pool, which can avoid any bias towards certain websites or domains and ensure the diversity of the data. Additionally, selenium' is applied to store HTML containing only visible nodes and to record each node's specified CSS properties (required for pre- training). Since many web pages retain thousands of tokens, we truncate them into sub-pages em- ploying sliding windows proposed by Deng et al. (2022). As shown in Figure 3, GEM takes HTML as input, which is processed into text tokens and corresponding XPath (XML Path Language) ex- pressions. The CSS properties are utilized the ground truth for the Gestalt pre-training tasks that are covered in section 3.2.
Based on this knowledge, pre-training tasks can be developed according the Gestalt laws, en- abling the language model to simulate the percep- tive processes of humans and better understand the semantic relationships among web contents.
2.2
Render Tree
The render tree is of great significance in ren- dering web pages, which is processed through four steps. The first two steps parse HTML and CSS² (Cascading Style Sheets) documents to create DOM and CSSOM trees, which are independent objects describing content and style rules. In the third step, DOM and CSSOM are merged into the render tree by reserving the visible DOM nodes and mounting CSSOM style information to the cor- responding node. Eventually, the browser traverses the render tree, calculates each node's exact size and position, and transforms nodes to actual pixels the screen. In brief, the concept of render trees contains comprehensive information on rendered web pages, encompassing textual content, HTML structure, stylistic, positional, and other visual in- formation. In this study, render trees are employed to build the corpora for pre-training.
3.2 Pre-training Objectives
As Figure 3 depicts, propose two Gestalt objec- tives to inject visual awareness into the language model. To preserve contextual and hierarchical information, GEM also employs the markup objec- tives proposed by et al. (2022). The ultimate pre- training objective is the summation of the markup objectives and the Gestalt objectives.
The following two subsections elaborate on the proposed Gestalt objectives respectively.
"https://developer whwrooompppp......eeeeeeeoe wwtpb/tg.ccccctttccc.cc.ooo.occt hhaeee/eeeeeeeeccccccccoooo hrome/
Chttps://welt dew.crrtticll-reneee renh-rend er te--tree-oomm conrreeccosttucttin 'https: //en. http://wwwwwiiiiiieeeeee wwwipedia.aa... org/wiki/CSs
"ht //www. selenium dev/
"https://en http::/:///wwwwipeeeeeeeeeee wikipedia. org/wikiiXPatt.

---

Gestalt Objactivia
Supervision
Task 1: Sienn Texcuaal Style Prediction
Task 2: Proximate Nade Prediction
Textual Style Metaadata
Posttional Metadata
Rendered Weo Page
Saft Fee!
Teets 24pt/ Box
A, B:
Hrghht:
12/4
A Proximate
Not
Nade
Node
feet: 20pa 30pa
Same
1318 Mayos
Proximate
Presuura
por
Node
Transferme Layors
Input
SPowe
Aftw
Aihel
Kirw
Affrod
AFridd
HTML
Node A
Node
Node C
Figure 3: An overview of the proposed GEM. Given a rendered web page, a browser is utilized obtain its render tree and automatically extract HTML and CSS properties. The HTML is utilized as input, which is further processed into text tokens and XPath expressions, while CSS properties are employed as supervision for two Gestalt objectives. These objectives include Same Textual Style Prediction (STSP), based on the law of similarity, and Proximity Node Prediction (PNP), based on the law of proximity. STSP equips the LM with visual-appearance perception by predicting whether the textual styles of sampled node pairs are the same, while PNP facilitates the LM to perceive relationships by learning whether sampled node pairs are proximate. Note that the markup objectives of GEM are hidden in this figure.
3.2.1 Same Textual Style Prediction
3.2.2 Proximate Nodes Prediction
The first Gestalt objective is Same Textual Style Prediction (STSP), utilizing the Gestalt law of sim- ilarity (see Section 2.1). The similarity of web ele- ments is defined by their appearances, which con- sist of size, background, and foreground (Xu and Miller, 2016). For the sake of simplicity, hereafter, those attributes are collectively called textual style. Referring CSS Reference" we figure CSS prop- erties of font, color, and background-color as joint control of textual style.
Besides the textual style information, the visual- position relationships of nodes are also essential for visual awareness in web understanding. Hence, the Proximate Node Prediction (PNP) objective is proposed, leveraging the Gestalt law of proximity (see Section 2.1). In web pages, the rendering re- gions of nodes are extracted from the render tree to evaluate their proximity. Notably, these rendering regions incorporate the padding, which is the space between the rendering border and content.
In pre-training the proximity is interpreted by comparing the position of nodes' edges (i.e., left, top, right, and bottom) that are calculated based on their rendering regions. If any edges of two nodes share the same value, they are considered proximate. As an example in Figure 3, the bottom edge of Node B and the top edge of Node C' co- incide, indicating they are proximate. Conversely, Node A and Node B share no edges and thus are non non-proxinee prox-roximate.. We randomly sample node pairs and ask the model to predict if they are proximate using the features of the first token from each node.
In pre-training, the Gestalt law of similarity is translated into classifying pair of text nodes with "same" or "not same" textual style. A given pair of text nodes are considered the same style if all textual-style--onntollnng CSS properties are the same. For instance, in Figure 3, Node A and Node B are assigned the "not same" label due to their non-identical font size. We randomly sample node pairs from one page, and the model is required to classify the pairs with the features from the first token of each node.
The STSP task provides supervision for textual- style-similarity clues, which enables GEM to un- derstand semantic relationships by incorporating prior knowledge of textual-style design.
Take situation where nodes are far in the DOM tree but visually close, providing vital clue that these nodes may be semantically related. The PNP task enables GEM to consider the semantic relation- ships among nodes utilizing the prior knowledge of proximity design apart from the structure of the

---

DOM tree.
4.2 Fine-tuning
We experiment on two downstream tasks to eval- uate GEM: web question answering and web in- formation extraction. Note that both GEM and GEM-R do not require rendered web pages in fine- tuning.
4 Experiment
4.1
Pre-training Setups
4.1.1 Data
Our corpora are built from the Common Crawl dataset. We derive approximately 2 million train- ing samples from 100k renderable web pages by pre-processing. Details of the data pre-processing are available in Section 3.1. We recognize the pos- sibility of noisy samples in our corpus that don't 't adhere to the Gestalt principles. However, such instances are minimal since, as demonstrated by Xiang et al. (2007), web pages that contradict the Gestalt principles typically experience short lifes- pans on the internet. In order to ensure fairness, we remove all the web pages that appear in the downstream task datasets. The settings of markup objectives follow et et al. (2022). both the STSP
4.2.1 Web Question Answering
Web question answering a task that automati- cally answers questions about a given web page, which requires a system to comprehensively under- stand the spatial and logical structure of the web page. We employ the Web-based Structural Read- ing Comprchension (WebSRC) dataset (Chen et al., 2021) verify the ability of GEM. WebSRC con- tains 400K question-answe answer pairs from 6.5K web pages and provides corresponding HTML source codes, screenshots, and metadata. The answers are either text spans on pages or yes/no. We follow previous work (Li et al., 2022) to take WebSRC as a typical extractive reading comprehension task, in which the token representations are fed into an output layer predict the start and end indexes of the answer (Devlin et al., 2019). The evalua- tion metries are Exact match (EM), F1 score (F1), and Path overlap score (POS), as defined the original paper (Chen et al., 2021), where POS is a tag level metric that measures the accuracy of locating HTML tags. We follow Li et al. (2022) to conduct the experiment on the official train/dev sets and report the results the development set. We fine-tune the pre-trained LMs for 10 epochs with a batch size of 32, and the learning rate is le-5.
and PNP tasks, we initially traverse and label all node pairs in a given training sample as positive or negative, and store them in separate pools. Sub- sequently, we randomly sample 100 samples from each pool (totaling 200 samples) for pre-training. If either pool has fewer than 100 samples, we employ oversampling to guarantee data balance. Using this approach, each node pair has an equal probability being labeled as "same" in STSP and as "proxi- mate" in PNP.
4.1.2 Implementation
We set up GEM following the MarkupL and initialize it with the pre-trained weight provided by Li et al. (2022). Additionally, we also implement a RoBERTa -based based GEM named "GEM-R" on the same corpora. MarkupLM enhances RoBERTa by incorporating a new XPath embedding layer to model HTML structures. GEM-R is pre-trained us- ing the MLM objective and the Gestalt objectives. As in the GEM, we utilize the first token features as the node features in GEM-R. The input format of GEM-R is the same T-PLM (Chen et al., 2021), i... pure text extracted from HTML. Our imple- mentation uses Adam optimizer (Kingma and Ba, 2014) with a learning rate of le-5 and a batch size of 128 training samples with a maximum of 384 tokens. The pre-training is done on 8 Nvidia-V109 100 GPUs for 300K steps. We evaluate the pre-training performance in the Appendix A.5.
4.2.2 Web Information Extraction
We use the Structure Web Data Extraction (SWDE) dataset (Hao et al., 2011) to evaluate GEM, real- world web page collection for automatic informa- tion extraction the web. The SWDE consists of over 124k web pages from 80 websites of verti- cals (10 websites per vertical). The task requires the model to extract the values for several given attributes (3 to 5 different attributes according to the vertical) from a page. In the actual application scenario, due to labor costs, only limited labeled data can be obtained for given vertical. However, the system is required to work much larger website set. Thus, we evaluate GEM each ver- tical independently with few-shot setting, where 50% of websites are training data, and the rest are testing data. Note that websites in the testing set are unseen during training. Precision, Recall, and
Ehttps: pttttp://ccomoncaawl.coom

---

Category
Model
EM
F1
POS
Category Model
P
R
F1
T-PLM
52.12
61.57
79.74
SSM
74.10
Non-PL.M
Pure
RoBERTabes baase
51.89
62.48
80.11
FreeDOM-Full
92.56
-based
Text
RoBERTa haase
52.56
62.61 80.38
SimpDOM
93.75
GEM-R
54.88
64.81 81.60
95.31 93.55
94.05
RoBERTabase
H-PLM
61.51
67.04 82.97
(+1.26)
(#1.99)
(#1.80
PLMs with
HTML
MarkupLM base
66.67 73.74 87.47
95.43
93.46 94.09
RoBERTabaso
(H)
MarkupLMpase
66.83 73.24 86.63
(+1.62) (#2.34) (#2.15)
pure text
GEM
69.12 75.93 88.41
95.91
94.05 94.57
GEM-R
(+1.25)
(*1.77) (#1.60)
H+sual V-PLM
62.07 66.66 83.64
95.99
95. 16 95.57
MarkupL base
Table I: Web question answering results on the Web- SRC dataset, categorized by input modality. "*" means further pre-traiming on our corpora with the original objective only. Underlined figures represent the best re- sults the text input group, while bold figures indicate the best results of all models.
(+:1.49)
(#1.70) (*1.65)
PLMs with
96.04
95.14 95.59
Maar
(+1.54)
(=1.87) (#1.81)
HTML
96.84 95.66 96.04
GEM
(+1.42) (#1.59) (+1.53)
Table 2: Evaluation results on web information extrac- tion task (SWDE dataset), categorized by method type. Some P (Precision) and R (Recall) values are left blank due to unreported original papers. Each value is re- ported "mean + standard deviation" calculated from 80 experiments. "*" stands for the model further pre- trained on our corpora with original objective only. Underlined figures represent the best results of the text input group, while bold figures indicate the best results of all models.
Fl score on page level are evaluation metrics on this task, following Hao et al. (2011). To make follow Zhou et al. (2021) pre- and post-process data. The results of each vertical are the average of 10 training set permuta- tions. The final experiment results are obtained by taking the average of all 8 verticals. We tune the pre-trained LMs for 10 epochs with a batch size of 64, and the learning rate is 2e-5.
fair comparison,
Moreover, GEM-R achieves the best results among all models with text input, indicating the robust adaptability of the Gestalt tasks in model archi- tecture. Notably, GEM-R does not incorporate an XPath embedding layer and relies solely on non-structural pure text as input, suggesting that Gestalt tasks can capture visual information with- out the explicit incorporation of HTML structure. Additionally, unlike V-PLM, which relies input screenshots for visual perception, GEM solely uti- lizes HTML as input and does not require rendered web pages during fine-tuning and inference, which provides significant deployment convenience ad- vantages while preserving visual perception.
4.3 Results
4.3.1
Web Question Answering
The results of web question answering are shown in Table I. Several baselines are compared, which are categorized into three groups based on their in- put modality: (1) Pure Text: T-PLM, RoBERTa, and GEM-R utilize non-structural pure text by deleting all HTML tags follow Chen et al. (2021); (2) HTML: H-PLM, MarkupLM,, and GEM take HTML input, using different per-processing methods, where H-PLM follows Chen et al. (2021)
and MarkupLM, GEM follow Li et al. (2022). (3) HTML+ Visual: V-PLM (Chen et al., 2021) lever- ages both HTML and screenshots as inputs. Since GEM does not have visual input, other strong mod- els containing visual features or metadata attained by rendered web pages (e.g., coordinates of ele- ments) are not included.
4.3.2 Web Information Extraction
The results of web information extraction are in Table 2, where the compared models are classified into three groups: (1) Non-PLM-basse methods: SSM (Carlson and Schafer, 2008), FreeDOM-Full
As depicted in the results, GEM consistently sur- passes other baselines. The gap between GEM and MarkupLM* illustrates the effectiveness of incor- porating visual awareness through Gestalt tasks.
(Lin al., 2020), SimpDOM (Zhou et al., 2021);
(2) PLMs with pure text as input: RoBERTa, GEM-
R are detailed in Section 4.3.1; (3) PLMs with

---

WebSRC Challenge Set
The specific pre-trained model weight we utilize, Alama-2-7b-chat-4 is available on Huggingface
Model
EM
F1
POS
The objective of the Web Question Answering task (with the WebSRC dataset) is to explore the model's capacity to comprehend the spatial and logical structure of given web page. LLM, host- ing an extensive knowledge repository, can answer general questions (Petroni et al., 2019), regardless of the given web page. To the project scope, we employ a subset from WebSRC, tagged as the "challenge ser", by eliminating questions that GPT- 3.5 can respond to correctly based solely on the question content. The challenge set comprises over 5,000 questions, with further details provided in Appendix A.I.
GEM
67.04
72.69
87.00
GPT-3,5-turbo
66.99
71.92
68.43
Illam22
25.91
31.54
37.97
Llama2-FT
48.95
53.50
63.17
SWDE Random Subsets
Precision
Recall
F1
GEM
97.06
96.09
96.36
GPT-3.5-turbo
28.61
25.93
26.18
GPT-3.5-turbo*
27.54
26.61
26.35
Llama2
25.98
26.10
26.04
Llama2*
26.27
26.10
26.06
The prompt template for WebSRC employs in- context learning (Brown et al., 2020), incorporating task description, selected demonstrations, and a test instance. For each question- page pair, domly select n demonstrations from the same verti- cal ensure semantic relevance. Due the token limit, we set 3. Further details are available Appendix A.2. Additionally, to address discrepan- cies in training data, we fine-tune Llama2 using the WebSRC training set. Further details about Llama2 fine-tuning are in the Appendix A.3.
Table 3: Performance comparison of LLMs and GEM on downstream tasks, with retested results for GEM on WebSRC Challenge and SWDE Random Set. "Llama2-FT" denotes Llama2 fine-tuned the Web- SRC training set. The prompt of "*" is limited to the text of nodes.
HTML as input: MarkupLM and GEM, which are the same as Section 4.3.1.
GEM achieves similar results on web infor- mation extraction as web question answering. GEM outperforms all basclines, and GEM-R is the best model in the "PLMs with pure text" group. The improvement achieved by GEM is significant, considering that the baselines demonstrate remark- able performance, and each value Table the average of 80 experiments. Another notable obser- vation that Gestalt tasks decrease the standard deviation of the results, indicating that incorporat- ing visual perception can enhance the robustness of models.
As shown in Table 3, GEM and GPT-3.5 achieve comparable performance on Exact Match and Fll score, on POS, GEM significantly outperforms GPT-3.5. Regarding Llama2, whether or not it un- dergoes fine-tuning, GEM significantly surpasses it. The fine-tuning outcomes align with expectations that fine-tuning improves Llama2's performance on the WebSRC dataset. The experimental results highlight GEM's superiority in consuming HTML structural information and validate LLM's inca- pability to provide answers despite access to the highly pertinent web page, mainly attributed to the LLM's lack of comprehension and familiarity with the HTML structure.
4.4 Discussion on Large Language Model (LLM)
Recently, large language models (LLMs) have been gaining adoption in different domains. Hence, we assess LLM on both downstream tasks, as shown in Table 3. We compare two baselines: 6PT-3.5-turbo (Ouyang et al., 2022b; Brown et al., 2020) and Llama2 (Touvron et al., 2023). GPT-3.5-turbo represents one of the current state-of-the-art LLMs and is accessible via the OpenAl 13 On the other hand, Llama2 is a prevalent open-source large model in academia.
For the web information extraction experiment using the SWDE dataset, considering that the SWDE requires 80 experiments and its web pages are of considerable length, we, due to limited re-
sources, randomly select 800 samples for testing and do not conduct experiments on fine-tuning Llama2. The prompt template adheres to the in- context learning principle, with 3 randomly lected demonstrates. All web pages in the prompt
"nttps://huggingf http://uuggingfaac..com/tttt ww/setatonnaaeeeeee LamaaLLamaa2--
Phttps://platee www.:m.ccppaicommmmmmmmm
-chat

---

Dataset
Objectives
WebSRC
SWDE
# Markup
STSP
PNP
EM
F1
POS
P
R
F1
96.04
95.14
95.59
66.83
73.24
86.63
(+1.54)
(+1.87)
(+1.81)
96.58
95.26
95.64
2
V
67.78
74.27
87.55
(+1.10)
(#1.49)
(#1.39)
96.62
95.48
95.81
67.58
74.15
86.68
(+:1.00)
(#:1.3)
(#1.644
96.84
95.66
96.04
69.12
75.93
88.41
(+1.42)
(+1.59)
(#1.53)
Table 4: Ablation study on the WebSRC and SWDE dataset. "STSP" and "PNP" stand for Same Textual Style Prediction and Proximate Nodes Prediction. "Markup" denotes the original pre-training objectives of MarkupI.M. The results on the SWDE dataset are reported as "mean # standard deviation" over 80 experiments.
(1) Attention activation of MarkupLM
in Table 4. The models with different objectives pre-trained in the same settings in Section 4.1.2. Experiment results validate our hypothet- ical proposition: both style and positional infor- mation extracted from rendered web pages benefit web understanding. For Instance, in terms the EM metric on the WebSRC, the proposed STSP shows an improvement of 0.95%, while the im- provement achieved with the PNP is 0.75%. Mean- while, the resonance between positional (PNP) and style (STSP) tasks and multi-task co-training can be vital to enhance the model performance. The models trained by an individual Gestalt task, with- out investigating multi-view landscapes, can hardly achieve superiority compared to the #4 model with holistic perception.
high
BORN RUN
(2) Attention activation of GEM
high
BORN RUN
Ral the Hendered Web Page
(26) HTML
Figure 4: Visualization of node-level self-attention weights between green node and other nodes, shown in both the rendered page and HTML (model's input).
4.6 Attention Maps Visualization
To further investigate the effect of injected visual awareness, we choose a case in WebSRC as an example and visualize the attention activation be- tween the green node and other nodes in the last layer of the encoder. The checkpoints we used are #1 and #4 in Table 4, which have not been fine- tuned. Both MarkupLM and GEM utilize HTML as their input, and to facilitate understanding, we provide visualizations on the rendered pages, shown in Figure 4. The color blocks are manually painted using the activation of the first token of each node, which serves as the node representa- tion in pre-training. MarkupLM essentially relies on the DOM tree since the high activation nodes are sequential HTML code (shown in Figure 4 lb). In contrast, GEM incorporates additional vi-
undergo pre-processing according to Zhou et al. (2021), producing XPath-text node pairs. Note that the pre-processed web content consists segments rather than the entire web page, which is consistent with the input of the baselines described in Section 4.3.2. Moreover, to eliminate XPath's potential noise, conduct experiment with the restriction that only the text of nodes is provided. The prompt template details are in Appendix A.4. Experiment results confirm LLM's incapability for HTML structure comprehension.
set of
4.5
Ablation Study
To further investigate the effectiveness of GEM, we perform a series of ablation experiments, as shown

---

sual prior knowledge beyond the DOM tree. Even without visual input, GEM pays more attention to nodes that are close to the green node, as well as those with similar textual styles (as displayed in Figure 4 2a). Visualization results demonstrate that the Gestalt tasks modify the attention mechanisms intended. GEM's attention map aligns with the web page's visual appearance, indicating that GEM can establish connections between visual and struc- tural/semantic information of the DOM tree. The attention mechanism of GEM is beneficial for web understanding, as verified in Section 4.3.1.
2022) ushered in new age of AI applications. While these advances have demonstrated remark- able capacity, there remains an acute need for a comprehensive evaluation of their application to web understanding.
Conclusion
This paper presents GEM, Gestalt Enhanced Markup (GEM) Language Model that leverages Gestalt psychological theory. GEM innovatively enriches the language model with heterogencous visual information from render trees of web pages without requiring visual modality input. As part of this innovation, two distinctive Gestalt pre-training objectives are formulated in order codify visual qualities such style and position within the lan- guage model. Evaluations of different downstream tasks and backbones show that GEM can learn a stronger representation of markup language with visual knowledge enhancement.
5 Related Work
Web pages contain rich perceptible information be- yond text. Extracting this information is crucial for web understanding but also challenging. Pre- vious works on this task mainly use rule-based modules (Soderland, 1999; Cohen et al., 2002; Gul- hane al., 2011; I; Hao et al., 2019), which are not robust different websites.
Limitations
Representation learning for plain text documents has been well-studied. Pre-trained Language Mod- els (PLMs), which use text encoders with self- supervised objectives, achieve remarkable perfor- mance numerous NLP tasks (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2019). Some recent works extend these methods web pages by taking HTML documents as input and encoding seman- tic information with specific pre-training tasks (Li et al., 2022; Deng et al., 2022).
Despite the success of the model GEM, some draw- backs need to be addressed in the future. Primarily, the proposed model relies on the render trees of web pages, which may not be immediately avail- able or perfectly precise for certain complex or advanced web pages. Additionally, the potential of large language models (LLMs) is undeniable, and there is much incentive to exploit it. This paper refrains from integrating GEM with LLMs, it represents challenging prospect that requires ex- tensive research. We aim to delve more deeply into this problem and make pioneering progress in our future research.
However, these models ignore the visual features of web pages, which are essential for understanding them. Some other works address this issue by treat- ing web pages as images of screenshots (Xu et al., 2020b,a), but they lost the hierarchical structure of HTML. Zhao et al. (2022) and Xie et al. (2021) incorporate coordinates of web elements into the input along with the HTML. However, they all rely on the fixed layout of the input image, which could vary with different devices and browsers. There- fore, these models have limited applicability.
Acknowledgements
This work is supported by the National Natural Sci- ence Foundation of China (Grant No. 61972349 the National Key R&D Program of China (No. 2018YFC2002603), and Llibaba-Zhejialt Univer- sity Joint Institute of Frontier Technologies.
Our work diverges from prior attempts in two crucial areas: First, we model web pages by lever- aging both HTML documents and render trees, pro- viding the LM with visual perception. Addition- ally, we harness knowledge from web page visual aspects by encoding it into the text representation rather than using visual information input.
References
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger. Tom Henighan, Rewon Child. Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens
Recent developments in LLMs (Brown et al., 2020; Ouyang et al., 2022a; Chowdhery et al.,

---

Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack
4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Xin Dong, Evgeniy Gabrilovich, Geremy Heitz. Wilko Horn, N. Lao, Kevin P. Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: a web-scale approach to probabilistic knowl- edge fusion. Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining.
Andrew Carlson and Charles Schafer. 2008. Bootstrap- ping information extraction from semi-structured web pages. In ECML/PKDD.
Nuno Filipe Escudeiro and Paula Maria Escudeiro. 2009. Exploring html tags and metadata to improve the expressiveness of web search engine's 's queries. 2009 Second International Conference on Computer and Electrical Engineering, 1:569-573.
Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. 2021. WebSRC: A dataset for web- based structural reading comprehension. In Proceedings of the 2021 Confer- ence on Empirical Methods in Natural Language Pro- cessing, pages 4173-4185, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Lukasz Garnaarek, Rafal Powalski, Tomasz Stanistawek, Bartosz Topolski, Piotr Halama, Michal P. Turski, and Filip Grali' nski. 2020. Lambert: Layout-aware language modeling for information extraction. In IEEE International Conference on Document Analysis and Recognition.
Aakanksha Chowdhery, Sharan Narang. Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebas- tian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinod- kumar Prabhakaran, Emily Reif, Nan Du. Benton Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Gareía, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontack Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz. Erica Moreira, Re- won Child, Oleksandr Polozov, Katherine Lee, Zong- wei Zhou, Xuezhi Wang. Brennan Saeta, Mark Díaz. Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstere Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. ArXiv, abs/2204.0231
Lisa Graham. 2008. Gestalt theory in interactive media design. Journal of Humanitiex & Social Sciences, 2(1).
Pankaj M. Gulhane, Amit Madaan, Rupesh R. Mehta, Jeyashankher Ramamirtham, Rajeev Rastogi, Sandeepkumar Satpal, Srinivasan H. Sengamedu, Ashwin Tengli, and Charu Tiwari. 2011. Web- scale information extraction with vertex. 2011 IEEE 27th International Conference on Data Engineering. pages 1209-1220
Junheng Hao, Muhao Chen, Wenchao Yu, Yizhou Sun, and Wei Wang. 2019. Universal representation learn- ing of knowledge bases by jointly embedding in- stances and ontological concepts. Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining.
Qiang Hao, Rui Cai, Yanwei Pang, and Lei Zhang. 2011. From one tree to a forest: A unified solution for struc- tured web data extraction. SIGIR page 775-784, New York, NY, USA. Association for Computing Machinery.
William W. Cohen, Matthew Hurst, and Lee S. Jensen. 2002. A flexible learning system for wrapping tables and lists in html documents. In Proceedings of the 11th International Conference on World Wide Web, WWW '02, page 232-241 New York, NY, USA. Association for Computing Machinery.
Edward J Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adap- tation of large language models. arXiv preprint arXiv:2106.09685
Xiang Deng, Prashant Shiralkar, Colin Lockard, Binx- uan Huang, and Huan Sun. 2022. Dom-Im: Learning generalizable representations for html documents.
Yupan Huang. Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutlmv3: Pre-training for doe- ument with unified text and image masking. Pro- ceedings of the 30th ACM International Conference Multimedia.
Jacob Devlin, Ming-Wei Wei Chang, Kenton Lec, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume 1 (Long and Short Papers), pages
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR. abs/1412.69988
K Koffka. 1955. Principles of gestalt psychology.

---

Vladimir Levenshtein. 1966. Binary codes capable of correcting deletions, insertions, and reversals. Sovier physics doklady, 10(8):707-710.
SnehaY., G. Mahadevan, and Madhura Prakash. 2012. A personalized product based recommendation sys- tem using web usage mining and semantic web. In- ternational Journal of Computer Theory and Engi- neering, pages 202-205.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan
Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zeulemoyer. 2019. Bart: Denoising srquence-to-sequene
Stephen Soderland. 1999. Learning information extrac- tion rules for semi-structured and free text. Machine Learning, 34:233-272.
pre-training for nat- ural language generation, translation, and compre- hension. In Anmual Meeting of the Association for Computational Linguistics.
Hugo Touvron, Louis Martin, Kevin R. Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Daniel M. Bikel, Lukas Blecher, Cris- tian Cantón Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony S. Hartshorn, Saghar Hos- seini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel M. Kloumann, A. V. Korenev, Punit Singh Koura, Marie-Anee Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov. Pushkar Mishra, Igor Molybog. Yixin Nie, Andrew Poulton, Jeremy Reizenstein. Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, R. Subramanian, Xia Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zhengxu Yan. Iliyan Zarov, Yuchen Zhang, An- gela Fan, Melanie Kambadur, Sharan Narang, Aure- lien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023. Llama 2: Open foundation and fine-tuned chat models. ArXiv, abs/2307.092
Junlong Li. Yiheng Xu, Lei Cui, and Furu Wei. 2022. Markuplm: Pre-training of text and markup language for visually rich document understanding, In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 6078-6087.
Bill Yuchen Lin, Ying Sheng. Nguyen Vo, and Sandeep Tata. 2020. Freedom: A transferable neural archi- tecture for structured information extraction on web documents, KDD '20, page 1092-1 100,, New York, NY, USA. Association for Computing Machinery.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: robustly optimized bert pretraining ap- proach. ArXiv, abs/1907.11692
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Pe- ter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022a. Training language models to follow instructions with human feedback. ArXiv, abs/2203.021555
D Vishwanath, Rohit Rahul, Gunjan Sehgal, Swati, Arindam Chowdhury, Monika Sharma, Lovekesh Vig, Gautam M. Shroff, and Ashwin Srinivasan. 2018. Deep reader: Information extraction from document images via relation extraction and natural language. ArXiv, abs/ 88 12.04777
Max Wertheimer. 1938. Gestalt theory.
Long Ouyang, Jeffrey WU, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022b. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744.
Peifeng Xiang, Xin Yang, and Yuanchun Shi. 2007. Web page segmentation based on gestalt theory. 2007 IEEE International Conference on Multimedia and Expo, pages 2253-2256 IEEE.
Chenhao Xie, Wenhao Huang, Jiaqing Liang, Cheng- song Huang, and Yanghua Xiao. 2021. Webke: Knowledge extraction from cemi-structured web with pre-trained markup language model. Proceedings of the 30th ACM International Conference on Informa- tion & Knowledge Management.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin. Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2463- 2473, Hong Kong, China. Association for Computational I Linguistics.
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang. Yijuan Lu, Dinci A. F. Florêncio, Cha Zhang, Wanxiang Che, Min Zhang, and Lidong Zhou. 2020a. Layouthmv2: Multi-modal pre-training for visually-rich document understanding. ArXiv, abs/2012.14740.
Somaych Mehrizi Sani and Yeganch Keyvan Shokooh. 2016. Minimalism designing user interface of com- mercial websites based on gestalt visual perception laws (case study of three top brands in technology scope). 2016 Second International Conference on Web Research (ICWR), pages 115-124.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. 2020b. Layoutlm: Pre-training of text and layout for document image understanding.

---

Zhen Xu and James Miller. 2016. Identifying semantic blocks web pages using gestalt laws of grouping. World Wide Web, 1955)957-977.
Prompt
Here is a question from web page titled { (Title) Question: (Ques- tion] You need to answer the ques- tion. The format of your reply: an- swer|[NNSER]] And do not reply to other content.
Zihan Zhao, Lu Chen, Ruisheng Cao, Hongshen Xu, Xingyu Chen, and Kai Yu. 2022. Tie: Topological information enhanced structural reading comprehen- sion on web pages. In NAACI..
Title
The title of the given web page from Web- SRC.
Yichao Zhou, Ying Sheng, Nguyen Ha Vo, Nick Ed- monds, and Sandeep Tata, 2021. Simplified dom trees for transferable attribute extraction from the web, ArXiv, abs/2101.02415.
Slots
Question
The question about the given web page from WebSRC.
A Appendix
A.1
WebSRC Challenge Set Selection
Table 5: The prompt for WebSRC Challenge Set selec- tion.
Due to limited cost, we randomly select 18,131 questions from the WebSRC development set, which contains 52,826 samples. We utilize the prompt defined in Table 5 to query GPT-3.5 and then compared the responses the ground truth by calculating their similarity using the formula:
Task
R
F1
STSP 61.70
76.24
65.15
PNP
72.33
65.59
84.60
similarity = 4 + 1 d
Table 6: Pre-traine Pre-training performance evaluation on held- out training data.
1 +
where 4, and l are the lengths of the responses and ground truth, whereas d denotes the Levenshtein Distance (Levenshtein, 1966) between them.
comprehensive evaluation, we reserve subset of the training data the evaluation set. We use classification metrics, and the results presented in the Table 6.
Ultimately, we select 5,324 questions with a sim- ilarity below 20% for the challenge set.
A.2
The Prompt Template of WebSRC
To evaluate LLM on WebSRC, we define the prompt template as shown in Table 7.
A.3
Details of Fine-tuning Llama2 on WebSRC
We fine-tune Llama2 model utilizing Parameter- Efficient Fine- Finn-TTning (PEFT) and Low-Rank Adap- tation (LoRa) (Hu et al., 2021) techniques with the entire training set of WebSRC. The fine-tuning is done on 2 NVIDIA A100 GPUs for 1 epoch.
A.4 The Prompt Template of SWDE
To evaluate LLM on SWDE, we define the prompt template as shown in Table 8. In order to facilitate understanding, web page is sampled from the "auto" vertical, and the values of all slots in the sampled web page are presented in Table
A.5 Pre-training Performance Evaluation
We evaluate the pre-training performance of the two innovative pre-training tasks, STSP and PNP, both which are binary classification tasks. For

---

Prompt
Here is question and its corresponding page. You need to answer the question, and the ANSWER either text spans page or yes/no. You need to answer the TID, which the deepest tag in the DOM tree which contain the answer. For yes/no question, there is no tag associated with the answer, so the TID is You also need to answer the STARTING INDEX the answer, which is the char offset of the answer from the start of the content the tag specified by TID. Note that before counting this number, we first eliminate all the inner tags in the specified tag and replace all the consecutive whitespaces with one space, For yes/no questions, STARTING INDEX answer "no".
| is for answer "yes" and 0 for
The format of your reply: answer: [ANSWER] tid: [TID) starting index: (START- ING INDEX] And do not reply other content.
Here are some demonstration:
(Demonstrates)
Here is the question and its corresponding page:
Question:
Ceastionn(Quettion Question) Page: ( [Paee]
reply:
Slots
Question
Question about the given web page from WebSRC.
Page
HTML souce code of the given web page from WebSRC.
Demonstrates
The selected demonstration with its ground truth response.
Table 7: The prompt for evaluating LLM on WebSRC.
Prompt
You are a web scraper, you will extract the values of (Atrributes) from a given web page segment.
You are not allowed to use any tools, you can only use the information in the input JSON object.
The web page is provided in the form of JSON object, which contains several key-value pairs. Each key is INDEX, and each value is another JSON object that has two fields: "XPATH" and TEXT. The "XPATH" field is string that represents the location of an element in an HTML document. The "TEXT" field string that contains the text content of that element.
The format the answer, and do not reply other content: [Answer_Format, Here are some demonstrations:
[Demonstrates]
Here is the given web page segment.
JSON object: (Web Page]
reply:
Slots
Attributes
Given attributes in the current vertical. Response in the format based on the given attributes. The selected demonstration with its ground truth response. The web page is provided in the form contains several key-value pairs. Each key a INDEX, and each value another JSON object that has two fields: "XPATH" and "TEXT".
Answer_Fonme Format
Demonstrates
Web Page
JSON object, which
Table 8: The prompt for evaluating LLM on SWDE.

---

Attributes Web_ Page
model", "price", "engine", _eve___ccoomm" (I: ("XPATH": "/ html/ body/ div[2]/ div[3/ div/ table/ tr/ td[2]/ d/v div[1j/ div[ly/ div[11/ ht/ br[il", "TEXT": "2011 Kia Sportage"), ), 2: ("XPATH": "/ html/ body/ div[2]/ div[3) div/ table/ tr/ td[2]/ div[1]/ div[1) div[1]/ div[2]/ div(1]/ div[2]/ div[I]/ table/ tr[1/ td[2]/ b". "TEXT": "18, 295-24,,99",,, 3: ("XPATH": "/ html/ body/ div[2]/ div[3/ div/ table/ tr/ td]2]/ / div[1// div[1] div[2]/ div[1]/ div[2/ div[iy/ table/ tr[]/ td[2]", "TEXT": 91,,9330-232280",,,,
36: ("XPATH": html/ body/ div/ div[4) table/ tr(5]/ td[1]",
div[2] div[3]/ div/ table/ tr/ td[2]/ div[1]/ "TEXT": "2011 Kia Sportage"))
Former
model": [INDEXI, INDEX2, INDEX1] [] "price": [INEEXI, IN- DEX2, INDEXn] or []: "engine": [INDEXI, INDEX2, INDEXn] or []: INDEX2, INDEXn] Π].
"fuel
Demonstrates
JSON object: (1: ("XPATH": "/ html/ body/ div[2]/ div[3]/ div/ table/ tr/ td[2// div[lJ/ div[1/ div[1/ [ h1/ br[1]". "TEXT": "2010 Toyota Sequoia"), 20: ("XPATH": "/ html/ body/ div[2]/ div[3]/ div/ table/ tr/ td[2]/ div[]/ div[4] div[3]/ table/ tr[17/ td[2]", "TEXT": "19")]
reply: model": "price":[2, 9]: "engine" [113]; peceee"ccom"" 20]
Table 9: The values of all slots in web page sampled from "auto" vertical. The "Web_ Pape__Paeeee and "Demonstrates" values are partially hidden due to space limitations.


Reviews:
[{"review_id": "dfOB81usfb", "reviewer": "Reviewer_6rWs", "summary": "", "questions": "* A claim in the Introduction states \"Regardless of render conditions, humans are capable of rapidly comprehending the content of rendered web pages, which indicates that human visual perception is an effective way to consume web semantics.\" Could you please point to any relevant citations for this? I'd like to understand what aspects of the human visual perception system are radically different from a regular OCR equipped vision model (especially one as large as CLIP or PaLI where the network has enough capacity to learn high-level concepts).\n* There seem to be errors in the explanation of the STSP pretraining objective and the example provided in the paper to explain it. Based on the definition, nodes B and C in Figure 3 should be different, but they are marked as same. Could you please explain why that is, considering the background color is different?\n* I think it's possible to achieve better POS results on GPT 3.5 with a little bit of fine-tuning or few-shot prompting. It is really good at reading comprehension, and given a large enough context length it should be able to get the right subset. The fact that it is able to generate the right ground truth tells me some prompt changes or post-processing might have gotten equal results. Were you able to try any alternate prompts or few-shot strategies?\n* Apologies if I'm misunderstanding this, but based on the example in Table 8, it seems like the xpath for the elements does not contain any style information (whether specified inline in the HTML or via separate CSS files). Does that mean whatever mechanism you use to extract xpath for each element of the HTML does not return styling information? If the input to your transformer layers is completely devoid of any style information, I'd imagine the model would have a harder time learning those distributions. So I'm wondering how much of the improvements you see is the results of pretraining MarkupLM for longer vs your pretraining objectives. Do you have a comparison between MarkupLM and your model where the length of pretraining is the same?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for dedicating your time to reviewing our paper and offering insightful feedback. We are grateful for your recognition of the novelty and potential of our idea. About the concerns you raised, we have carefully considered them and would like to respond as follows.\n\n# Response to Concerns regarding GPT 3.5\n\n> I think it's possible to achieve better POS results on GPT 3.5 with a little bit of fine-tuning or few-shot prompting. It is really good at reading comprehension ... some prompt changes or post-processing might have gotten equal results. Were you able to try any alternate prompts or few-shot strategies?\n\nThank you for raising this question. We conduct experiments with various prompts for WebSRC and discover that the one presented in Appendix A.2 produces optimal outcomes. The task description of this prompt is derived from the official WebSRC repository [2], which ensures that GPT-3.5 comprehends the task requirements. We will provide an additional narrative in our revision.\n\nWe also agree that in-context learning (ICL) based few-shot strategies can be essential for this study, while similar approaches show remarkable performance in some tasks. Following your suggestions, to evaluate its effectiveness for WebSRC, we conduct a supplementary experiment using the prompt format suggested by [1]. This format consists of a task description, selected demonstrations, and a test instance. For each question-page pair, we randomly select $n$ demonstrations from the same vertical to ensure semantic relevance. Due to the token limit imposed by the OpenAI API, we set $n$ to 3. To minimize the influence of randomness, we repeat the experiment 5 times and report the average results. The following table presents the results:\n\n| Model       | EM    | F1    | POS    |\n|-------------|-------|-------|--------|\n| GPT-3.5     | 66.80 | 72.01 | 66.24  |\n| GPT-3.5-ICL | 66.99 | 71.92 | 68.43  |\n| GEM         | 67.04 | 72.79 | 87.00  |\n\n\n\"GPT-3.5-ICL\" denotes the result using the ICL prompt, and \"GPT-3.5\" and \"GEM\" results are from Table 3 of our paper. As illustrated by the table, GPT-3.5-ICL achieves a similar result to GPT-3.5 in terms of EM and F1, indicating that GPT-3.5's QA capability has been fully activated. Regarding POS, ICL shows modest improvement, but GEM still outperforms GPT-3.5 significantly, which implies GEM's superiority in consuming HTML structural information. We will include these experimental results in our revision for a complete comparison.\n\n> The paper compares performance of their pre-trained + fine-tuned encoder-decoder model against a larger decoder-only model (GPT 3.5) in a zero-shot untuned manner. This seems like an uneven comparison.\n\nWe appreciate your observation regarding the comparison between GEM and GPT-3.5. The purpose of Section 4.4 is to demonstrate that GEM can achieve comparable results on tasks that require consuming HTML structure, which is a key focus of GEM, rather than to claim that GEM outperforms GPT-3.5 in every aspect. \n\nWe acknowledge that the comparison can be further enhanced by leveraging model fine-tuning with task-specific training data. Therefore, to align these differences, we intend to fine-tune GPT-3.5 with the same dataset as GEM utilizing the OpenAI API, and we will host this result in the revision. However, the preliminary experiment indicates that fine-tuning GPT-3.5 cannot significantly improve its performance since it is already trained on massive web-like data, such as CommonCrawl [1]. We will validate this hypothesis through experiments and report the results in the revision.\n\n> I felt the paper is limited in scope to an improvement to a specific prior model in this domain. To establish the viability of the pre-training objective, it needed to show results on some more contemporary models, such as smaller-sized decoder-only models with similar fine-tuning as was done.\n\nWe appreciate your concern regarding the viability of the Gestalt tasks. As illustrated in Table 1 and 2 of our paper, the Gestalt tasks can enhance the performance of RoBERTa and MarkupLM, two different architecture encoder-only PLMs. Furthermore, we would like to highlight that GEM is the current state-of-the-art PLM in this domain. Therefore, our experiments provide strong evidence for the viability of the Gestalt tasks. Additionally, we recognize the distinction between designing pre-training tasks on web understanding for encoder-only and decoder-only models. We are in the process of adapting the Gestalt tasks to decoder-only architectures, and this is a promising direction for future research.\n\n# Response to Concerns Regarding Pre-training\n\n> Apologies if I'm misunderstanding this, but based on the example in Table 8, it seems like the xpath for the elements does not contain any style information ...  Does that mean whatever mechanism you use to extract xpath ... does not return styling information? ...  I'd imagine the model would have a harder time learning those distributions. ... Do you have a comparison between MarkupLM and your model where the length of pretraining is the same?\n\nThank you for your thoughtful questions. The XPath expressions GEM uses are standard and do not incorporate any stylistic information from the web pages. We would like to clarify that GEM does not require any stylistic information or additional visual input, as it only takes HTML and XPath as input. Unlike prior costly OCR-based efforts, GEM learns the visual perception from two Gestalt pre-training tasks, which use CSS properties as their ground truth labels. Moreover, GEM utilizes the visual information of rendered web pages only during the pre-training and does not need any rendering during fine-tuning or inference.\n\nTo compare GEM and MarkupLM fairly, we experiment with the same pre-training condition (including pre-training steps and dataset). In Table 1 and 2,  \"MarkupLM\\*\" denotes MarkupLM further pre-trained on our corpora with its original objective for the same pre-training steps. The gap between GEM and MarkupLM\\* demonstrates the effectiveness of Gestalt tasks. We will provide additional narrative in the paper to make the experiment results more noticeable.\n\n\n> The paper lacks adequate details on the pretraining performance, for e.g. comparison of the loss or perplexity compared to MarkupLM or RoBERTa on which this model is based. Considering the pre-training tasks are the main contributions of this paper, this is a significant omission.\n\nWe appreciate your insightful suggestion. We concur that comparing loss and perplexity would provide a more thorough evaluation of the model's pre-training performance. Due to space constraints, we are unable to include these results in the original version. We will manage to incorporate various figures of loss curves and a comprehensive analysis of perplexity.\n\n# Response to Rest Concerns\n\n> The paper does not provide adequate explanation or examples of Gestalt principles being flouted in web pages and it's effect on web Q&A and information extraction. A wikipedia page has section and para sub-headings - which are grouped closer together when in collapsed state. While looking similar and being co-located in this state, they are not semantically related ... Similarly for news headlines, sports scores, etc.\n\nWe appreciate your comment and acknowledge that some web pages in our corpora do not follow the Gestalt principles. However, we would like to clarify that the Gestalt task is not the only pre-training task for GEM. It is an essential complementary task that leverages non-text information (i.e., vision) to enhance the existing text-based tasks (e.g., MLM). The Gestalt task is an enhancement, not a replacement, of the text-based tasks. Moreover, as [3] has shown, web pages that violate the Gestalt principles tend to have low survival rates on the internet. Hence, the proportion of noisy data samples is relatively low. We will provide more evidence to support this claim in our revision.\n\nRegarding the section headings and paragraph sub-headings, we believe that they are not very semantically related, but they have analogous semantic functions as headings. Nevertheless, we concede that the Gestalt task may be less effective on the example you gave. Therefore, we are intrigued by the work of [4], who have developed methods to eliminate noisy blocks from web pages (such as advertisements). We will cite their papers in future work and further optimize the model performance. Finally, we would like to emphasize that our empirical results (Table 1, 2, 4) demonstrate the effectiveness of our method based on the current corpora. We will provide an additional narrative in the revision to address this problem.\n\n\n> A claim in the Introduction states \"Regardless of render conditions, ... , which indicates that human visual perception is an effective way to consume web semantics.\" Could you please point to any relevant citations for this? I'd like to understand what aspects of the human visual perception system are radically different from a regular OCR equipped vision model ...\n\nWe agree that we should provide additional references to support this argument. Two studies, [3] and [5], endorse our claim on human perception. We will cite these references clearly in our revision to enhance its rigor. \n\nRegarding your question about the radical difference between human perception and the OCR-equipped vision model, we would like to clarify that our objective is to enhance the language model rather than the vision model. Nevertheless, we can identify two differences. First, Gestalt psychology is a prominent cognitive theory that is derived from human experiences of perceiving semantic groups, which motivates us to enhance language models with visual perception without additional visual input. In contrast, most vision models, such as CLIP or PaLI, aim to learn statistical alignment between modalities from large-scale data but do not explicitly model semantic groups.  Second, the input of GEM is HTML. OCR-equipped vision models take images as input and emphasize the importance of the fixed layout of the input image. However, the dynamic nature of web page rendering can lead to significant appearance variations across devices and browsers. Therefore, GEM is more suitable for web understanding. We will provide more detailed explanations in our revision.\n\n> There seem to be errors in the explanation of the STSP pretraining objective and the example provided in the paper to explain it. Based on the definition, nodes B and C in Figure 3 should be different, but they are marked as same. Could you please explain why that is, considering the background color is different?\n\nFollowing your question, we will enhance Figure 3 and make it more comprehensible. We would like to clarify that the background colors of example nodes, e.g., Node B and C, are intended to highlight them, and actually, they both have a white background on the web page. As stated in Section 3.2.1, Node B and C have the same textual style because they share the same *font, color*, and *background-color* CSS values. However, we acknowledge that this may not be clear to the readers and may create confusion about the original background color of example nodes. We will enhance the clarity of the figure to ensure that it is unambiguous and not misleading.\n\nAgain, we would like to express our gratitude for evaluating our paper carefully. Thank you for your time and effort in providing a comprehensive review.\n\n\n# Reference\n[1]\tBrown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., ... & Amodei, D. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-1901.\n\n[2]\thttps://github.com/X-LANCE/WebSRC-Baseline\n\n[3]\tXiang, P., Yang, X., & Shi, Y. (2007, July). Web page segmentation based on gestalt theory. In 2007 IEEE International Conference on Multimedia and Expo (pp. 2253-2256). IEEE.\n\n[4]\tUma, R., & Latha, B. (2019). Noise elimination from web pages for efficacious information retrieval. Cluster Computing, 22, 14583-14602.\n\n[5]\tXu, Z., & Miller, J. (2016). Identifying semantic blocks in Web pages using Gestalt laws of grouping. World Wide Web, 19, 957-978.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The idea of using Gestalt Psychology theory as pre-training tasks has novelty and promise. This theory provides a computationally lightweight, reasonably accurate baseline assumption to make about the semantic connections between components on the page - and hence a reasonable proxy for using actual visual information about the page.", "reasons_to_reject": "While I like the novelty of using Gestalt theory for pretraining tasks, there are a few shortcomings of this paper:\n* The paper lacks adequate details on the pretraining performance, for e.g. comparison of the loss or perplexity compared to MarkupLM or RoBERTa on which this model is based. Considering the pre-training tasks are the main contributions of this paper, this is a significant omission.\n* The paper does not provide adequate explanation or examples of Gestalt principles being flouted in web pages and it's effect on web Q&A and information extraction. A wikipedia page has section and para sub-headings - which are grouped closer together when in collapsed state. While looking similar and being co-located in this state, they are not semantically related (one section could be about Etymology whereas another about References). Similarly for news headlines, sports scores, etc.\n* The paper compares performance of their pre-trained + fine-tuned encoder-decoder model against a larger decoder-only model (GPT 3.5) in a zero-shot untuned manner. This seems like an uneven comparison.\n* I felt the paper is limited in scope to an improvement to a specific prior model in this domain. To establish the viability of the pre-training objective, it needed to show results on some more contemporary models, such as smaller-sized decoder-only models with similar fine-tuning as was done.", "questions_to_authors": "* A claim in the Introduction states \"Regardless of render conditions, humans are capable of rapidly comprehending the content of rendered web pages, which indicates that human visual perception is an effective way to consume web semantics.\" Could you please point to any relevant citations for this? I'd like to understand what aspects of the human visual perception system are radically different from a regular OCR equipped vision model (especially one as large as CLIP or PaLI where the network has enough capacity to learn high-level concepts).\n* There seem to be errors in the explanation of the STSP pretraining objective and the example provided in the paper to explain it. Based on the definition, nodes B and C in Figure 3 should be different, but they are marked as same. Could you please explain why that is, considering the background color is different?\n* I think it's possible to achieve better POS results on GPT 3.5 with a little bit of fine-tuning or few-shot prompting. It is really good at reading comprehension, and given a large enough context length it should be able to get the right subset. The fact that it is able to generate the right ground truth tells me some prompt changes or post-processing might have gotten equal results. Were you able to try any alternate prompts or few-shot strategies?\n* Apologies if I'm misunderstanding this, but based on the example in Table 8, it seems like the xpath for the elements does not contain any style information (whether specified inline in the HTML or via separate CSS files). Does that mean whatever mechanism you use to extract xpath for each element of the HTML does not return styling information? If the input to your transformer layers is completely devoid of any style information, I'd imagine the model would have a harder time learning those distributions. So I'm wondering how much of the improvements you see is the results of pretraining MarkupLM for longer vs your pretraining objectives. Do you have a comparison between MarkupLM and your model where the length of pretraining is the same?", "paper_topic_and_main_contributions": "The paper provides a new method to pre-train language models to better handle markup language-based text, such as the content on web pages (which includes HTML and the styles associated with the elements). Two new pre-training tasks are suggested which aim to capture the rendered visual aspects of the markup - these tasks are inspired by Gestalt Psychology theory which states that components containing similar content are co-located and look similar. The paper hypothesizes this way of pre-training improves the model's ability to understand web page content, and shows this by the improved performance on web Q&A and information extraction task.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "@misc{gur2023understanding,\n      title={Understanding HTML with Large Language Models}, \n      author={Izzeddin Gur and Ofir Nachum and Yingjie Miao and Mustafa Safdari and Austin Huang and Aakanksha Chowdhery and Sharan Narang and Noah Fiedel and Aleksandra Faust},\n      year={2023},\n      eprint={2210.03945},\n      archivePrefix={arXiv},\n      primaryClass={cs.LG}\n}"}}, {"review_id": "p3VES9mfBl", "reviewer": "Reviewer_W8C9", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and your time in reviewing our paper. We appreciate your positive comments on our method's novelty, exposition, motivation, and intuition. We are committed to publishing the code and the model to facilitate further research in this area. We have provided the following explanation in response to your insightful suggestion about clarification.\n\n> Ablation - while visual information can be useful for an extractive Q&A task like in the case of WebSRC dataset, it can be particularly useful to see how it fares in the Information Extraction task because the latter involves discerning right elements from incorrect ones based on limited context and visual information. Was this experiment performed? It would be interesting to learn how well the approach shines here.\n\nThank you for your question regarding the ablation experiment on the SWDE dataset. We perform this experiment but cannot include it in the paper because of the space limit. The table below shows the experimental results, which achieve similar results as on web question answering. We will provide this experiment in the appendix of our revised version.\n\n| # | Markup | STSP | PNP | Precision  | Recall     | F1          |\n|---|--------|------|-----|------------|------------|-------------|\n| 1 | √      |      |     | 95.99±1.49 | 95.16±1.70 | 95.57±1.65  |\n| 2 | √      | √    |     | 96.58±1.10 | 95.25±1.49 | 95.64±1.39  |\n| 3 | √      |      | √   | 96.62±1.20 | 95.48±1.73 | 95.80±1.64  |\n| 4 | √      | √    | √   | 96.84±1.42 | 95.66±1.59 | 96.04±1.53  |\n\n> Webpage processing - the paper uses a sliding window to truncate long webpages into sub-pages, the details of which are missing. How is a page split? Sharing this is critical to understanding how a HTML \"section\" does not inadvertently land into separate sub-pages.\n\nThank you for your valuable suggestions. We employ the sliding window algorithm proposed by [1], which attempts to prune the nodes in the tree while preserving the relationship between the nodes. We will supplement these details in Section 3.1 of the revision and provide an additional narrative to help readers reproduce the experiment outcomes.\n\n> Data collection - How are the 100k pages in the pre-training corpora selected from Common Crawl? Are there any websites more preferred than others, or is it a random sample? How do you ensure diversity, availability of modern pages that render well?\n\nThank you for your insightful questions. We select the 100k pages from Common Crawl using a two-step process. First, we filter out pages that are non-renderable using headless chrome. Second, we randomly sample 100k pages from the remaining pool. This way, we avoid any bias towards certain websites or domains, and ensure a diverse and representative sample of modern web pages that render well. We will provide more details on this process in Section 3.1 of our revision to clarify it. Because of space limitations, more data collection details will be available in the appendix.\n\n> Sampling - The sampling of node pairs for both the pre-training objectives (STSP, PNP) is an imbalanced data problem. How do you sample negatives? How do you sample \"hard\" negatives (pairs that are actually negative, but may have similar perceptual appearance and/or proximity)?\n\nWe appreciate your feedback on sampling negative samples in our pre-training objectives. We follow a two-step procedure for both STSP and PNP to obtain balanced samples. First, we traverse and label all node pairs in a given web page as positive or negative, and store them in separate pools. Second, we randomly sample 100 samples from each pool (totaling 200 samples) for pre-training. If either pool has less than 100 samples, we use oversampling to ensure data balance. We agree that sampling hard negatives could improve our performance, and we intend to explore this direction in our future work. We apologize for omitting these details in the paper due to space constraints, but we will include them in the appendix of our revised version.\n\n> Error analysis - Finally, while the pre-training task clearly helps deliver better performance on both the tasks considered, the paper can do better in throwing light on pending error patterns, by including an analysis and/or discussion of few examples where the approach fell short.\n\nWe are grateful for your insightful suggestions. In our revised version, we will include more analysis to illustrate the limitations and challenges of our method in certain scenarios.\n\n\n\nOnce again, we would like to thank you for your constructive feedback. Your comments and suggestions were very insightful and helped us improve the quality of our work.\n\n# Reference\n\n[1] Deng, X., Shiralkar, P., Lockard, C., Huang, B., & Sun, H. (2022). Dom-lm: Learning generalizable representations for html documents. arXiv preprint arXiv:2201.10608.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "A. The paper introduces a novel pre-training approach based on a psychological theory that integrates visual information, HTML structure and text (markup and tokens) in a unified manner that ultimately delivers better performance on downstream tasks.\n\nB. The paper is very clear in its exposition, right from motivation, introduction, situating itself in the broader work on the topic, methods, results and reflection.\n\nC. The authors plan to publish the code and the model, which should promote further work using the idea, thereby stimulating more research in this relatively nascent area.\n\n", "reasons_to_reject": "Please note the following are some avenues where the paper clarity can be improved; they are not necessarily strong reasons to reject.\n\nA. Webpage processing - the paper uses a sliding window to truncate long webpages into sub-pages, the details of which are missing. How is a page split? Sharing this is critical to understanding how a HTML \"section\" does not inadvertently land into separate sub-pages.\n\nB. Data collection - How are the 100k pages in the pre-training corpora selected from Common Crawl? Are there any websites more preferred than others, or is it a random sample? How do you ensure diversity, availability of modern pages that render well?\n\nC. Sampling - The sampling of node pairs for both the pre-training objectives (STSP, PNP) is an imbalanced data problem. How do you sample negatives? How do you sample \"hard\" negatives (pairs that are actually negative, but may have similar perceptual appearance and/or proximity)?\n\nD. Ablation - while visual information can be useful for an extractive Q&A task like in the case of WebSRC dataset, it can be particularly useful to see how it fares in the Information Extraction task because the latter involves discerning right elements from incorrect ones based on limited context and visual information. Was this experiment performed? It would be interesting to learn how well the approach shines here.\n\nE. Error analysis - Finally, while the pre-training task clearly helps deliver better performance on both the tasks considered, the paper can do better in throwing light on pending error patterns, by including an analysis and/or discussion of few examples where the approach fell short.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents a novel approach to training a pre-trained language model for modern webpages that require understanding the complex interplay between HTML structure, text and visual appearance and layout. The idea is based on a psychological theory called Gestalt laws of perception, one of which is about how humans perceive similar objects to form a group, and another is about how we perceive closely situated objects to form a coherent group. The authors apply these Gestalt laws of similarity and proximity of objects to the pre-training task by designing appropriate contrastive pre-training objectives, in addition to employing the more commonly used masked language modeling objective based on the text. \n\nThe beauty of the approach lies in 1) the ability to leverage the intuitive laws in infusing visual appearance information into the pre-training task for better page understanding (in addition to structure and text considered previously), and in doing so, 2) also circumventing the challenge of requiring visual \"screenshots\" as input which has been a big hurdle for all prior attempts that aimed to integrate text, markup and visual information in a single model.\n\nThe paper is clear in its exposition, giving the readers a solid motivation and intuition behind the proposed approach, and also in clearly demonstrating its benefits via comparative evaluation, ablation and a qualitative example analysis. Particularly interesting is the fact that the model doesn't require HTML rendering for fine-tuning or inference, which can be immensely useful for a practitioner in Information Extraction, Question Answering and other relevant areas that depend on page understanding. The paper clearly demonstrates how this is enabled by the Gestalt laws that can capture visual information useful toward downstream tasks without explicit incorporation of HTML structure.\n\nHaving said that, the paper can be improved by clarifying a few aspects around webpage preprocessing, data collection, sampling approach for training, ablation on the IE task, and including a discussion on error patterns that remain.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "G7CLI4DviP", "reviewer": "Reviewer_bTQ3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your careful review of our paper and your constructive feedback. We are glad that you recognized our main contribution and ablation studies. Regarding your concern, we hope that our explanation below can address that.\n\n> Performance gain is relatively small compared to MarkupLM, especially for web-info extraction experiment.\n\nWe appreciate your concern about the relatively small performance gain in the SWDE dataset. In order to enhance experiment validity, the experiment results for each vertical are an average of 10 training set permutations, and the final results are averaged across all verticals, resulting in an average of 80 experimental results (at lines 360-363). Meanwhile, it is noteworthy that MarkupLM has already demonstrated exceptional performance in SWDE (with an F1 score of 95.57%), and further efforts can only lead to marginal improvements. Additionally, as shown in Table 2, GEM achieves a smaller standard deviation than MarkupLM, indicating that visual information enhances the stability of GEM. We hope this explanation addresses your concern, and we will provide additional narratives to address this problem in the revision.\n\nFinally, we would like to express our gratitude again for your review.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* Proposed a unique method that can enhance LM's ability to understand HTML text. This is important now as a lot of people is trying to enhance LLM with the ability to browse the web (e.g. WebGPT).\n* Ablation studies showed that the proposed method is crucial to improve the final performance.", "reasons_to_reject": "* Performance gain is relatively small compared to MarkUp LM, especially for web-info extraction experiment.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposed a novel way to train a model that can be used on structure data like HTML. Borrowing idea from psychology, they proposed two pre-training task specifically designed to improve downstream performance on HTML datasets. One pre-training task is Binary Textual Style Prediction, which predicts whether two HTML nodes have the same style or not, and another pre-training task is Proximate Note Prediction, which predicts whether two HTML nodes are nearby or not. Both tasks takes text embeddings of the corresponding HTML nodes as input. After pre-training, they fine-tune the model on two HTML-related downstream task: web QA and web information extraction. The results show that the Gestalt-inspired pre-training methods indeed helped the downstream performance.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a novel method for building pre-trained language models that are capable of handling markup language-based text and understanding the complex interplay between HTML structure, text and visual appearance, and layout. The idea behind the new approach is grounded in Gestalt laws of perception, which describe how humans perceive similar objects to form a group and how closely situated objects are also perceived to form a coherent group. The paper hypothesizes that this way of pre-training improves the model's ability to understand web page content, and the experiments confirm that this helps improve its performance on downstream tasks.

All reviewers agree that the paper has **a number of strengths and merits**. Specifically:
1. *Novelty and theoretical as well as practical appeal of the proposed idea*: This strength is highlighted by all reviewers.
2. *Overall clarity of writing*: Reviewer W8C9 points out that the paper is clear, well organized and well written.
3. *Reproducibility* is noted by all reviewers, and Reviewer W8C9 lists the fact that the authors plan to publish the code and the model among their reasons to accept. 
4. In addition, Reviewer bTQ3 points out that *ablation studies* suggest that the proposed method is crucial to the improvement of the final performance of the model.

At the same time, all reviewers also identified **some weaknesses** and further areas for improvement, including:
1. *Lack of meaningful comparison and further model's analysis*: Reviewer 6rWs points out that comparison of the loss or perplexity of the proposed model to MarkupLM or RoBERTa on which the model is based is missing. In addition, Reviewer 6rWs has further feedback on the comparison to GPT 3.5.
2. *Evaluation and analysis of the results*: Reviewer 6rWs points out that it would be desirable for the paper to show improvements on a broader scale and using some more contemporary models. Furthermore, Reviewer W8C9 proposes improvements in terms of ablation studies and error analysis. Finally, Reviewer bTQ3 considers the performance gain to be relatively small compared to MarkupLM.
3. *Further methodological explanations*: Reviewer 6rWs also suggests that the paper should provide more detailed explanation and / or examples of Gestalt principles.
4. *Implementation details*: Finally, while formulated not as clear weaknesses, Reviewer W8C9 makes suggestions for further improvement and clarification of certain implementation details – see their review.

The authors did a good job addressing the concerns expressed and the questions posed by the reviewers. Most suggestions made by the reviewers have been acknowledged by the authors and seem to be likely to be integrated in the revised version of the paper.
Decision: Accept-Main