Towards Zero-shot Relation Extraction in Web Mining: A Multimodal Approach with Relative XML Path
Authors: Zilong Wang
Jingbo Shang
Venue: EMNLP 2023 Findings (2023)
Keywords: web mining
xml path
document ai
zero shot
transfer learning

Abstract: The rapid growth of web pages and the increasing complexity of their structure poses a challenge for web mining models. Web mining models are required to understand semi-structured web pages, particularly when little is known about the subject or template of a new page. Current methods migrate language models to web mining by embedding the XML source code into the transformer or encoding the rendered layout with graph neural networks. However, these approaches do not take into account the relationships between text nodes within and across pages. In this paper, we propose a new approach, ReXMiner, for zero-shot relation extraction in web mining. ReXMiner encodes the shortest relative paths in the Document Object Model (DOM) tree of the web page which is a more accurate and efficient signal for key-value pair extraction within a web page. It also incorporates the popularity of each text node by counting the occurrence of the same text node across different web pages. We use contrastive learning to address the issue of sparsity in relation extraction. Extensive experiments on public benchmarks show that our method, ReXMiner, outperforms the state-of-the-art baselines in the task of zero-shot relation extraction in web mining.

Towards Zero-shot Relation Extraction in Web Mining: A Multimodal Approach with Relative XML Path
Zilong Wang
Jingbo Shang
University of California, San Diego zlwang@ucsd.
University of California, San Diego jshang@ucsd.
Abstract
DOM Tree
Mone Wertinl
MRA Firtical
heral
The rapid growth of web pages and the in- creasing complexity of their structure poses a challenge for web mining models. Web min- ing models are required to understand semi- structured web pages, particularly when little is known about the subject or template new page. Current methods migrate language models to web mining by embedding the XMI. source code into the transformer or encoding the rendered layout with graph neural networks. However, these approaches do not take into account the relationships between text nodes within and across pages. In this paper, pro- pose a new approach, ReXMiner, for zero-shot relation extraction in web mining. ReXMiner encodes the shortest relative paths in the Doc- ument Object Model (DOM) tree of the web page which is more accurate and efficient signal for key-value pair extraction within web page. It also incorporates the popularity of each text node by counting the occurrence of the same text node across different web pages. We use contrastive learning address the issue of sparsity in relation extraction. Extensive periments on public benchmarks show that our method, ReXMiner, outperforms the state-of- the-art baselines in the task of zero-shot relation extraction in web mining.
Ioad
beddy
2ead
body
Aode
Eaady
Relative
Path
Patlern
Abs.
Paths
apan
span
11
The structural information from semi- structured web pages. Based the DOM Tree from the HTML source code, the absolute and relative XML Paths are extracted. We believe the web page structure is well modeled by the XML Paths predict the attribute of text nodes and the relative XML Paths provide extra signals to predict the relation between text nodes.
Figure I:
The dynamic nature of the modern internet poses significant challenges for web mining models due to its rapid pace of updates. It is infeasible to anno- tate emerging web pages and train targeted models for them. Modern web mining models are expected to perform zero-shot information extraction tasks with little prior knowledge of emerging subjects templates (Lockard et al., 2020; Chen et al., 2021). In this context, the multimodal features extracted from the HTML/XML source code as well as the textual contents are crucial for dealing with zero- shot information extraction task on the countless emerging web pages.
1 Introduction
The internet is a vast repository of serii-sruettured web pages that are characterized by the use of HTML/XML markup language. Compared to plain text in traditional natural language understanding tasks, these web pages possess additional multi- modal features such as the semi-structured visual and layout elements from the HTML/XML source code. These features can effectively generalized across different websites and provide a richer un- derstanding of the web pages (Lockard et al., 2018, 2019, 2020). * Jingho Shang is the corresponding author.
Previous approaches to the problem of zero-shot web mining have primarily focused on creating rich representations through large-scale multimodal pre- training, utilizing XML Paths of text nodes' (Lin et al., 2020; Zhou et al., 2021; Li et al., 2022b). As shown in Figure 1, XML Paths are sequences of tags (e.g., div, span, 1i) indicating the location of
'https://en httts:://enwwikieett wikipedia. aeteiii..rgg/iiee wrg/wwki/XPattt

---

the text node in the DOM Tree² of the page. These pre-training approaches extend vanilla language models by embedding the absolute XML Paths but fail to take into account the relative local relation- ship expressed by the relative XMI. Paths. The related nodes tend to be close to each other in the DOM tree, which results in a long common prefix in their XML Paths, as shown in Figure 1. Such local relation is more common than the absolute XML Path patterns. Therefore, it is easy to trans- fer the relative XML Paths to new web pages, and the relative XML Paths serve as a more efficient and meaningful signal in predicting the relation between text nodes.
adopting contrastive learning during training which is widely used in related works (Su et al., 2021; Hogan et al., 2022; Li al., 2022a). We randomly generate negative cases and restrict their ratio to the positive ones, allowing the model to properly discriminate related node pairs from others.
By learning from the relationships between text nodes within and acmoss pages, ReXMiner is able to effectively transfer knowledge learned from ex- isting web pages to new ones. We validate our approach on web pages from three different ver- ticals from the SWDE dataset (Hao et al., 201 1), including Movie, University, and NBA. The rela- tion labels are annotated by Lockard et al. (2019). We summarize our contribution follows. We propose novel multimodal framework, ReXMiner, that effectively exploit the relative local relationship within each web page and in- corporate the popularity of text nodes across dif- ferent web pages in the relation extraction task. We represent the relative local relation and the popularity of text nodes in the language models through relative XML Paths in the DOM Tree and the occurrence number of text nodes across different web pages.
Additionally, existing web mining approaches tend to treat each web page separately and focus on memorizing their various templates, ignoring the fact that the relevance across different web pages of the same website is also meaningful to identify the related text nodes (Zhou et al., 2021; Li et al., 2022b; Lockard et al., 2020). Intuitively, text node is more likely be key word if it appears frequently in collection of web pages and its sur- rounding words are not fixed. For example, in web pages about NBA players, the statistics about the height, age are common text fields in the player introduction, so the text nodes, such as "Height:" and "Age:" should appear more frequently than other text nodes and the surrounding text contents should be different.
Extensive experiments on three different verti- cals from SWDE dataset demonstrate the effec- tiveness of ReXMiner in the zero-shot relation extraction task in web mining.
Reproducibility. The code will be released on G
In light of the aforementioned challenges in web mining, we propose a web mining model with Relative XML Path, ReXMiner, for tackling the zero-shot relation extraction task from semi- structured web pages. Our approach aims to learn the local relationship within each web page by ex- ploiting the potential of the DOM Tree. Specif- ically, we extract the shortest path between text nodes in the DOM Tree as the relative XML Path, which removes the common prefix in the XML Paths. Inspired by the relative position embed- ding in T5 (Raffel et al., 2020), we then embed the relative XML Paths as attention bias terms in the multi-layered Transformer. Additionally, we incor- porate the popularity of each text node by counting the number of times occurs across different web pages, and embed the occurrence logarithmically in the embedding layer. Furthermore, we address the data sparsity issues in the relation extraction task by
Related Work
Information Extraction in Web Mining How to efficiently and automatically gathering essen- tial information from the internet is always a hot topic in the academia of natural language process- ing and data mining due to the enormous scale and vast knowledge within the internet. The open information extraction task in web mining is origi- nally proposed by Etzioni et al. (2008) and further developed by following works, including Fader et al. (2011), Bronzi al. (2013); Mausam (2016) which rely on the syntactic constraints or heuris- tic approaches to identify relation patterns, and Cui et al. (2018); Lockard et al. (2018, 2019); Xie et al. (2021); Li et al. (2023) which introduce neural networks to solve the task under supervi- sion or distant supervision settings. Our proposed method follows the task formulation of the zero- "ggttuubccom//wwwggeee... httpc///wwwwaaneee/////eeeeeeeee
Phttps://ent http::://en.wikieedii.. www.wedia.aa.mm wttp///ii///oocee.tttt. Object_Model

---

Verticals
Websites
Web Pages
ones (Lockard al., 2020). The unseen web pages should be orthogonal to the existing ones with re- gard to vertical, topic, and template. The zero-shot setting requires the web mining models to extract relevant pairs based on both the textual content and the DOM Tree structure of web pages. Specifi- cally, each web page is denoted as a sequence of text nodes, P [41, 42, .... ........ (2.,
Movie
IMDB, Hollywood. Yahoo
Tranic
(3 to loual)
ESPN, Fan Houve Fox Sports, Today, ...8 toind)
NBA
120000 in latul)
Miot Aoy-dllon.bem well-Kentn knni
University
News, Embark, (8 in total)
where n is the number of nodes in the page. Each node involves textual contents and the XML Path extracted from the DOM Tree, Ty appth:.. The goal of the zero-shot relation extraction task is to train a model using related pairs, (z: + xz)). from a set of web pages, and subsequently extract related pairs from unseen ones. For example, shown in Figure 2, one of our tasks is to train models with web pages from Movie and NBA verticals and test the models with web pages from the University vertical.
Kn].
Figure 2: The web pages in the SWDE dataset. There are three verticals, Movie, NBA, University. Each ver- tical includes several websites. Each website includes hundreds web pages.
shot relation extraction in web mining proposed by ZeroShotCeres (Lockard al., 2020) where the models are required to transfer relation knowledge from the existing verticals to the unseen ones. Ze- roShotCeres adopts the graph neural network to understand the textual contents and model the lay- out structure. It finally produces rich multimodal representation for each text node and conduct bi- nary classification extract related pairs.
Methodology
We extend the text-only language models with mul- timodal features and propose a novel framework, ReXMiner, for zero-shot relation extraction task in web mining. Figure 3 shows the components in our framework. We adopt the absolute XML Path embedding in MarkupLM (Li et al., 2022b), and further extend it with popularity embedding and relative XML. Path attention. To cope with the sparsity issue in the relation extraction task, adopt the contrastive learning strategy where conduct negative sampling control the ratio between positive cases and negative cases.
Layout-aware Multimodal Transformers The pre-trained language models, such as BERT (Ken- ton and Toutanova, 2019), XLNet (Yang et al., 2019), GPT (Brown et al., 2020), T5 (Raffel et al.. 2020), are revolutionary in the academia of natural language processing. It achieves state-of-the-art performance in text-only tasks. To further deal with multimodal scenarios, various features are extracted and incorporated into the Transformer framework. Recent study has shown that it is bene- ficial to incorporate multimodal features, such as bounding box coordinates and image features,into pre-trained language models to enhance overall performance in understanding visually-rich docu- ments (Xu al., 2020, 2021: Huang et al., 2022; Gu al., 2021: Wang et al., 2022b). Similarly, web pages are rendered with HTML/XML markup language and also represent layout-rich structures. Multimodal features from the DOM Tree or ren- dered web page images are incorporated in the pre- trained language models to solve the tasks in the semi-structurrd web pages (Lin et al., 2020; Zhou al., 2021; Li al., 2022b; Wang et al., 2022a).
4.1 Absolute XML Path Embedding
We follow the idea in MarkupLM and embed the absolute XML Paths in the embedding layer. We introduce it in this section for self-contained pur- pose. The XML Path is a sequence of tags from HTML/XML markup language (e.g., div, span, 1i). Both of the tag names and the order of tags are important to the final representation. There- fore, in the embedding layer, we first embed each tag as embedding vector, and all these tag em- beddings are concatenated. To be more specific, we pad or truncate the XPath to a tag sequence of fixed length, [t1,..., tn], and embed the tags Emb(41) ), ..., Emb((n) where 4, is the i-th tag and Emb((,) E I is its embedding. We further con- catenate the vectors as Emb(1)) o Emb(t,) E Rhs to explicitly encode the ordering information, where o is the operation of vector concatenation.
3 Problem Formulation
The zero-shot relation extraction in web mining to learn knowledge of related pairs in the existing web pages and transfer the knowledge to the unseen

---

Training Set
ReXMiner
Testing
Relative XML.
Relative Attention
Vertical:
Veeticals
Web Page:
htn3
body
Web Page: Ray Allenttml
I+1
Sweet Hete
Common Prefix
Length: 3
DOM Tiee
ReIXPothi-3 (opan, divrill
Contractive Learning
<heml>>
RelXPatha (liydivrepan
/cCuany
icCanada
Positive
cheads
<body>
Multimodal Embedding Layer
hCannds
hConny
The Sticet
<aiivo>
Popularity Embedding
Hereofier
hine
Negaatie
Absolute XML Path Embedding
Capan?
<li>
Word
Embeddiing
hecaary
hTCe
XPath0
XPathD Sweer
XPathO
XPath1
XPath2
#Pos + #Neg
Country
Canada
The
Hereafter
Country
Canada
#Pos: #Neg
Figure 3: The framework of ReXMiner. We extract the DOM Tree of each web page from the HTML source code and further extract the absolute and relative XML Paths. We embed the popularity of text nodes and absolute XML Paths in the embedding layer and embed the relative XML Paths in the attention layers. We reduce the binary classification loss the relation pairs sampled by negative sampling. In this figure, we train ReXMiner using web pages from the Movie vertical and test it unseen web pages from the NBA vertical.
To fit in with the hyperspace of other embedding layers E R4, a linear layer is used to convert the concatenation into the right dimension.
quently in the group of web pages.
o P) - [1 1, if Expath', allw, e P otherwise
AbsXPathEmb(zzatti)
=Proj(Emb(tt) .... Emb(t,)) Rd
pop(w)
Co(w, P.)
where Proj(-) is linear layer with parameters W E Ruaxd and b Rd.
where pop(w) is the popularity of w. Then we normalize logarithmically and convert the value into indices ranging from r. Each index corre- sponds to an embedding vector.
4.2 Popularity Embedding
We propose Popularity Embedding to incorporate the occurrence of the text nodes into the pre-trained framework. Web pages from the same website use similar templates. The popularity of a certain text node across different web pages of the same website is meaningful in the relation extraction task in the web mining. Intuitively, a text node is more likely to be key word if it appears frequently and its neighboring words are not fixed.
PopEmb(ww) = Emb (|- pog(w)())
1) C IRd
log N
where Emb(-) is the embedding function; 7 is the total number of popularity embeddings; d is the dimension of embedding layers.
Formally, along with the absolute XML Path embedding, the embedding of the i-th text node, (w,, xpath:), is as follows.
In details, given a text node (w. rpolh) and N web pages Pl...., PN from the same website, we iterate through all the text nodes in each web page and compare their textual contents with w, regard- less of their XML Paths. We count the web pages that involves nodes with the same text and define the number of these web pages as the popularity of w. Thus, higher popularity of text node means that the same textual contents appears more fre-
6; :PPoopmm(u;)) ux:) AbsXPathEmb(zpatt)) WordEmb(w1) + PosEmb(1)
4.3 Self-Attention with Relative XML Paths
The local relation within each web page is essential to the zero-shot relation extraction since the related nodes are more likely to be close the DOM Tree. As shown in Figure 4, they present a long common

---

Dom Tree
Prefix
nodes in the DOM Tree:
root
Common Prefix of and Ay
100 Longth
ReIXPath(a; = xz) = [rev(ppath, ); xpath;] ReIXPath(z; + ri) [rev(xpath] t;; xpath,]]
Sub Tree ReIXPath(y + x)
sputh
where rev(:) is to reverse the tag sequence; is the lowest common ancestor of x; and zj (e.g. 83 Figure 4). the case of Figure 4, rev(opath5) equals [4o, (s]. the lowest common ancestor is l3- and xpath, equals [t4.]. so ReIXPath(r; = 24) equals [ts, ts, t3. f4].
ReIxPathx x) spath,
Lowest Common Ancestor
Figure 4: The relative XML Path illustration. In Prefix, we focus on the length of the common prefix of the pair nodes showing their depth in the DOM Tree, and embed it in the first attention layers. In Sub Tree, we focus on the shortest path between the pair of nodes, and embed it the following attention layers.
Adding Bias Terms In the first a layers of the self-attention, embed the common prefix length dej as bias terms. The attention weight between zj and uy is computed
1 (WPes) (wWk) ez) + b bdrrd))))
AG
prefix in their XML Paths, and the rest parts of their XML Paths compose the relative XML Paths between them. The relative XML Paths can be seen the shortest path between text nodes in the DOM Tree. Therefore, the relative XML Paths useful signals and could be well transferred into un- seen web pages. Enlightened by Yang et al. (2019); Raffel et al. (2020); Peng et al. (2021, 2022), we model the relative XML Paths as bias terms and in- corporate them into the multi-layer self-attention of Transformer. Specifically, we embed the common prefix length in the first a layers of self-attention and embed the relative XML Paths tags in the next 3 layers of self-attention. where (a + 3) equals to the total number of layers. the case of Figure we embed the common prefix length the relative XML Paths (ta, t3, ts, to].
where the common prefix length dij is a bounded integer and each integer is mapped to specific bias term by b
In the next B layers of the self-attention, we em- bed the relative XML Paths as bias terms. Fol- lowing the absolute XML Path embedding (in- troduced in Section 4.1), we project the embed- ding of tags in ReIXPath(z; = 2y) into bias terms. Specifically, we split the relative XML Path the lowest common ancestor tag and embed each part separately. When embedding ReIXPath(r; = 2y), the two sub-sequences of tags are [rev(xpath, ); 6 and xtathhh))).
In the equation, fm is the lowest common ances- tor (e.g. to in Figure 4); [t,,...ttm, tm] is the path from 2; to the lowest common ancestor (e.g. [ta, ty) in Figure 4); (m..... (n] the path from the lowest common ancestor to xy (e.g. [ta, ts, to] in Figure 4). The bias term is follows,
well as
Extracting Relative XML Paths Given pair text nodes, 3; and Zj. we first extract the com- mon prefix of their XML Paths which shows the path from the root to the lowest common ances- tor of these two nodes in the DOM Tree (e.g. [fo. f1, t2, t3] in Figure 4). We denote the prefix length as dy. The rest parts in the XML Paths shows the path from the lowest common ances- for to the text node. We denote them xpath; and xpath, which are the XMI. Paths without the common prefix (c.g. [ts, to] and [ta,] in Figure 4). They also compose the shortest path between these
xpath (1is (x4)
= b(Fmb((1) Emb(tm)) + b D'(Emb((1n) o... o Emb'(4,)) E R
where is the operation of vector concatenation; Emb is the embedding function; b is linear layer projecting embedding to R. We also use two sets of modules to differentiate the two sub-sequences of tags, (b, Emb) and (b', Emb'). Thus, the attention weight between x; and w, is computed as
1 1G
boaath (x;,
Vd

---

4.4 Contrastive Learning
Pairs per Web Page
Vertical
# Websites
Web Pages
We observe the sparsity issues in the relation extrac- tion task, where only a small proportion of nodes are annotated as related pairs so the negative cases are much more than the positive ones. To tackle this issue, we adopt the contrastive learning and conduct negative sampling to control the ratio be- tween the positive cases and negative ones.
Movie NBA
16000
34.80
3551
11.94
University
8090
28.44
Table 1: The statistics of the SWDE datset.
different verticals, Movie, NBA, and University, and each vertical includes websites of the corre- sponding topic. For example, http://in http:://mmmb.////// com and http://rottenttome in the Movie vertical, and http://esppt go com and htttp://nbll com are collected in the NBA ver- tical. Then the SWDE dataset collects web pages in each website and extracts their HTML source code for web mining tasks. Based on the original SWDE dataset, Lockard al. (2019, 2020) further annotates the related pairs the web pages, and propose the zero-shot relation extraction task in web mining. The statistics the SWDE dataset is shown in Table I, where we report the total number of websites in each vertical, the total number of web pages in each vertical, and the average number of annotated pairs in each web page.
Negative Sampling The number of positive cases and negative cases in the sampling should follow,
com are collected
#Pos #Neg = ?;; #Pos: #Neg = μ
where we denote the number of related pairs in the groundtruth as #Pos and the number of neg- ative samples #Neg; 7, and i two hyper- parameters.
Loss Function To distinguish the positive sam- ples from the negative ones, we train our model with cross-entropy loss. First, we define the proba- bility of related pair, (z; + æz) using the Biaffine attention (Nguyen and Verspoor, 2019) and the sig- moid function a.
Biaffine(u, u1, v) = u T Me + W( + P(r; + xz) (Biaffine(hy,, hz))
5.2 Experiment Setups
The zero-shot relation extraction task requires that the unseen web pages the testing set and the ex- isting web pages in the training set are of different verticals. Therefore, we follow the problem set- tings, and design three tasks based on the SWDE dataset, where we train our model on web pages from two of the three verticals and test our model on the third one. We denote the three tasks as, Train models with the Movie and NBA verticals, and test them on the University vertical;
where h; and hj are the hidden states from ReXMiner corresponding x, and æy; M, W, b are trainable parameters; o is the vector concatena- tion. During training, we reduce the cross entropy of training samples against the labels.
Σ CrossEntropy(P(z, + 2z). L(I,, (11,0))
Movie+NBA=
where [x (y)) is the label of (Iz, 2z)), either posi- tive negative, indicating whether these two nodes are related or not.
NBA+Univ= Train models with the NBA and University verticals, and test them on the Movie vertical;
Movie= NBA: Train models with the Uni- versity and Movie verticals, and test them the NBA vertical.
5 Experiments
We conduct experiments and ablation study of zero- shot relation extraction on the websites of differ- ent verticals from the SWDE dataset following the problem settings proposed Lockard al. (2020).
We report the precision, recall, and F-1 score.
5.3 Compared Methods
We evaluate ReXMiner against several baselines.
5.1 Datasets
Our experiments are conducted on the SWDE dataset (Hao et al., 2011). As shown in Figure 2, the SWDE dataset includes websites of three
Colon Baseline The Colon Baseline is heuris- tic method proposed in Lockard et al. (2020). It identifies all text nodes ending with a colon (":")

---

Unseen Vertical
Average
Model
Movie
NBA
University
Pre
Rec
Pre
Rec
F1
Pre
Rec
F1
F1
Colon
47
19
27
33
40
46
31
37
35
ZSCeres-FFNN'
42
38
40
44
46
45
50
45
48
44
ESCeres-GNNy
43
42
42
49
48
49
45
47
46
MarkupLM
48.93
40.56
44.35
44.45
71.35
54.78
58.50
62.37
60.37
53.17
Ours
45.36
49.36
47.28
65.86
64.94
65.40
68.43
60.97
64.48
59.05
Table 2: The experiment results of ReXMiner and baseline models. 1 The results of Colon Baseline and 7e-
roshotCeres (ZSCeres) are from Lockard al. (2020). t We introduce the contrastive learning module of ReXMiner to the M. framework to solve the relation extraction task.
the relation strings and extracts the closest text node the right below the object. The Colon Baseline needs no training data, it satisfies the requirement of the zero-shot relation extraction.
mance in the zero-shot relation extraction task in all three verticals of the SWDE dataset. Specifi- cally, ReXMiner surpasses the second-best model, MarkupLM, by 5.88 the average F-1 score. each task, we can observe a remarkable improve- ment of 2.93, 10.62 and 4.11 in F-1 score when the Movie, NBA, or University verticals are considered as the unseen vertical, respectively.
ZeroshotCeres ZeroshotCeres (Lockard al., 2020) is a graph neural network-based approach that learns the rich representation for text nodes and predicts the relationships between them. first extracts the visual features text nodes from the coordinates and font sizes, and the textual fea- tures by inputting the text into pre-trained BERT model (Kenton and Toutanova, 2019). Then the features fed into graph attention network (GAT) (Veličković al., 2018) 8), where the graph is built based on the location of text nodes in the ren- dered web page to capture the layout relationships. The relation between text nodes predicted as a binary classification on their feature concatenation.
ZeroshotCeres is the previous state-of-art model proposed to solve the zero-shot relation extraction which leverages the graph neural network to model the structural information. We copy its perfor- mance from Lockard et al. (2020). In the com- parison with MarkupLM and ReXMiner, we ob- serve that directly modeling the XML Path infor- mation using Transformer framework achieves bet- performance, where MarkupLM and ReXMiner surpass ZeroshotCeres by 7.17 and 13.05 in av- erage F-1 score. The multimodal attention mech- anism with absolute XML Path embedding from MarkupLM enhance the performance in each task, and ReXMiner achieves the state-of-the-art overall performance after incorporating the relative XML Paths and the popularity of text nodes.
MarkupLM MarkupLM (Li et al., 2022b) is a pre-trained transformer framework that jointly models text and HTML/XML markup language in web pages. It embeds absolute XML Paths in the embedding layer of the BERT framework and proposes new pre-training tasks to learn the corre- lation between text and markup language. These tasks include matching the title with the web page, predicting the location of text nodes in the DOM Tree, and predicting the masked word the input sequence. We use MarkupLM as a backbone model and append it with the contrastive learning module of ReXMiner to solve relation extraction task.
Though the performance of ReXMiner varies in different verticals, we can safely come to the conclusion that our proposed model, ReXMiner, is superior to the baselines solving zero-shot relation extraction task in web mining. Further analysis is conducted in Ablation Study and Case Study study the multimodal features.
5.5 Ablation Study
5.4 Experimental Results
In the ablation study, we aim at studying the role of multimodal features proposed in ReXMiner, in- cluding the Relative XML Path Attention and the Popularity Embedding. We introduce three abla- tion versions of ReXMiner by removing certain
We report the performance of ReXMiner in Ta- ble 2 and compare it with baseline models. From the result, we can see that our proposed model, ReXMiner, achieves the state-of-the-art perfor-

---

Unseen Vertical
Average
Model
Movie
NBA
University
Pre
Rec
F1
Pre
Rec
F1
Pre
Rec
F1
F1
ReXMiner
45.36
49.36
47.28
65.86
64.94
65.40
68.43
60.97
64.48
59.05
w/o RelXPath
45.60
45.68
45.64
47.13
73.54
57.44
54.82
74.63
63.21
55.43
w/o ReIXPuth PopEmb
48.93
40.56
44.35
44.45
71.35
54.78
58.50
62.37
60.37
53.17
Table 3: The results of ablation study, where we compare ReXMiner with two ablation variants, ReXMiner w/o ReIXPath and ReXMiner w/o RelXPath + PopEmb. PopEmb denotes the popularity embedding. and RelXPath denotes the relative XPath bias terms.
NBA+Univ Movie (Prediction result on Quiz Show.hmm
Extracted Puirs
Relative XML
Model
Positive
(Color type, Technicolar prints):
(Moods, Thoughtj; (Set 1958); (Genres, Drama); (Sound by. Dolby':
R
(w/ RelXPath PupEmb)
ReXMiner
(Produced by. Buena Vista);
(Director, Americanat; (Types, Drama); (MPA Rating, USAJ; (Keywords, Scandal)
(Rsom Romombasing--ene -immaloaing--mmoiiot
(Keywords, Advertislag);
(Types, Docudrama)
span
(Color type, Tochnioolor prints). (uudds Foud for Thoughty, (Genres, Drama); (Sound by, Dolby); (From bouk, Remembering Americui;
ReXMiner
(Director, Americanal; (Genree, Dolby);
(wie ReIXPath, w/ EmpEmb)
(Moods, Tochmiculor prints);
(Types, Drama); (Tomes, Technicolor prints)
ReXMiner
(Color type, Technicolor prints).
(vele RelXPath + PapE
(Muods, Food for Thought)
(Director, Drama);; (Flags, Americans)
Movie The green pairs Table The extraction results of the ablation models on Quiz Stml in NBA denote the new true positive predictions compared with the previous ablation model, and the red pairs denote the missing true positive predictions compared with the previous ablation model.
features in Table 3. From Table 3, compare the performance of all ablation models. We find that using the Popularity Embedding enhances F-1 score by 2.84 and 2.66 in Novie+NBA=Unih Univ task and Movie task, respectively. After incorporating the Relative XML Path Attention, the F-1 score are further improved in all three tasks. Thus, the ablation model with all multimodal fea- tures achieve the highest F-1 score. We conclude that the Relative XML Path contributes the high precision while the popularity embedding enhances recall leading to the best performance in F-1 score.
bedding, ReXMiner (w/o RelXPath, w/ PopEmb) depends on the frequency when predicting the lated pairs so intends extract more text node pairs and contributes to a higher recall. After adding the Relative XML Path Attention, the ex- tracted pairs are further filtered by the relative XML Path patterns in ReXMiner (w/ RelXPath + PopEmb) so it can extract similar number of true positive pairs and largely reduce the number of false positive cases, but it leads to the missing traction of (From book, Remembering America).
Conclusion and Future Work
5.6 Case Study
In this paper, we present ReXMiner, a web min- ing model to solve the zero-shot relation extraction task from semi-structured web pages. It benefits from the proposed features, the relative XML Paths extracted from the DOM Tree and the popularity of text nodes among web pages from the same web- site. Specifically, based on MarkupLM, and we further incorporate the relative XML Paths into the attention layers of Transformer framework as bias terms and embed the popularity of text nodes in the embedding layer. To solve the relation ex-
Table 4, we show the extraction results of the ablation models on Quiz show.html in NBA+Univ=Movie. We select one relative XML Path pattern, sspan;div;us al and list the corresponding extracted pairs into two groups, true positive extractions and false positive extractions. From the results, we can see that ReXMiner with all proposed features shows the best performance, which is also demonstrated the ablation study. Specifically, by incorporating the Popularity Em

---

traction task, we append the backbone model with the contrastive learning module and use the neg- ative sampling to solve the sparsity issue of the annotation. In this way. ReXMiner can transfer the knowledge learned from the existing web pages to the unseen ones and extract the related pairs from the unseen web pages. Experiments demonstrate that our method can achieve the state-of-the-an performance compared with the strong baselines. For future work, we plan explore the new problem settings with limited supervision, such as few-shot learning and distant supervision, and fur- ther study the topologiaal structure information in the DOM Tree to explore more meaningful signals in understanding the semi-structured web pages in web mining tasks.
authors and should not be interpreted necessarily representing the views, either expressed or implied, of the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for government purposes not withstanding any copy- right annotation hereon.
References
Mirko Bronzi, Valter Crescenzi, Paolo Merialdo, and Paolo Papotti. 2013. Extraction and integration of partially overlapping web sources. Proceedings of the VLDB Endowment, 6 6(10):805-816
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in newral information processing systems, 33:1877-1901.
7 Limitations
We build ReXMiner based on MarkupLM and in- corporate new features, including the relative XML Paths, the popularity of text nodes, and the con- trastive learning. After initializing our model with the pre-trained weights of MarkupLM, the addi- tional modules are finetuned on the datasets of downstream tasks without large-scale pre-training. due to the limited computing resource. We believe more promising results can be achieved if it pos- sible to pre-train our proposed framework enabling all parameters to be well converged.
Jiaoyan Chen, Yuxia Geng, Zhuo Chen, Ian Horrocks, Jeff Z Pan, and Huajun Chen. 2021. Knowledge- aware zero-shot learning: Survey and perspective. arXiv preprint arXiv:2103.0000
Lei Cui, Furu Wei, and Ming Zhou. 2018. Neural open information extraction. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 407- 413.
Oren Etzioni, Michele Banko, Stephen Soderland, and Daniel S Weld. 2008. Open information extrac- tion from the web. Communications of the ACM, 51(12):68-74,
8 Ethics Statement
Our work focus on the relation extraction in web mining under zero-shot settings. We build our framework using Transformers repository by Hug- gingface (Wolf et al., 2019), and conduct experi- ments on the SWDE datasets (Hao et al., 2011). All resources involved in our paper are from open source and widely used in academia. We also plan to release code publicly. Thus, we do not antic-
Anthony Fader, Stephen Soderland, and Oren Etzioni. 2011. Identifying relations for open information ex- traction. In Proceedings of the 2011 I conference empirical methods in natural language processing. pages 1535-1545.
Xavier Glorot and Yoshua Bengio. 2010. Understanding the difficulty of training deep feedforward neural net- works. In Proceedings of the thirteenth international conference on artificial intelligence and statistics. pages 249-256. JMLR Workshop and Conference Proceedings.
ipate any ethical concerns.
Acknowledgments
Jiuxiang Gu, Jason Kuen, Vlad I Morariu, Handong Zhao, Rajiv Jain, Nikolaos Barmpalios, Ani Nenkova, and Tong Sun. 2021. Unidoe: Unified pretraining framework for document understanding. Advances in Neural Information Processing Systems, 34:39-50.
We want to thank the anonymous reviewers for their insightful comments. Our work is spon- sored in part by NSF CAREER Award 2239440,
NSF Proto-OKN Award 2333790, NIH Bridge2AI Center Program under award IU54HG0125 10-001, CisCo-UCSD Sponsored Research Project, well as generous gifts from Google, Adobe, and Tera- data. Any opinions, findings, and conclusions or recommendations expressed herein are those of the
Qiang Hao, Rui Cai, Yanwei Pang, and Lei Zhang. 2011. From one tree a forest: unified solution for struc- tured web data extraction. In Proceedings of the 34th international ACM SIGIR conference on Research and development Information Retrieval, pages 775- 784.

---

William Hogan, Jiacheng Li, and Jingbo Shang. 2022. Fine-gu Sine-grained contrastive learning for relation extrac- tion. In Proceedings of the 2022 Conference Em- pirical Methods in Natural Language Processing, pages 1083-1095.
Mausam Mausam. 2016. Open information extraction systems and downstream applications. In Proceed- ings of the twenty-fifth international joint conference on artifiical intelligence. pages 4074-4077.
Dat Quoe Nguyen and Karin Verspoor. 2019. End- to-end neural relation extraction using deep biaffine attention. In European conference on information retrieval, pages 729-738. Springer.
Yupan Huang. Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. 2022. Layoutimv3; Pre-training for doc-
ument ai with unified text and image masking. In Proceedings of the 30th ACM International Confer- ence on Multimedia, pages 4083-4091
Han Peng, Ge Li, Wenhan Wang. Yunfei Zhao, and Zhi Jin. 2021. Integrating tree path in transformer for code representation. Advances in Neural Information Processing Systems, 34:9343-9354.
Jacob Devlin Ming-Wes Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding, In Proceedings of NAACL-HLI, pages 4171-44 1116.
Han Peng, Ge Li, Yunfei Zhao, and Zhi Jin. 2022. Re- thinking positional encoding in tree transformer for code representation. In Proceedings of the 2022 Con- ference on Empirical Methods in Natural Language Processing, pages 3204-3214.
Dongyang Li, Taolin Zhang, Nan Hu, Chengyu Wang. and Xiaofeng He. 2022a. Hielre: A hierarchical con- trastive learning framework for distantly supervised relation extraction. In Findings of the Association for Computational Linguistics: ACL 2022, pages 2567- 2578.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., (1(10)):
Junlong Li. Yiheng Xu, Lei Cui, and Furu Wei. 2022b. Markuplm: Pre-training of text and markup language for visually rich document understanding. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume 1: Long Papers), pages 6078-6087.
Peng Su, Yifan Peng, and K Vijay-Shanker. 2021. Im- proving bert model using contrastive learning for biomedical relation extraction. In Proceedings of the 20th Workshop on Biomedical Language Processing, pages 1-10.
Zimeng Li, Bo Shao, Linjun Shou, Ming Gong, Gen Li, and Daxin Jiang. 2023. Wiert: web informa- tion extraction via render tree. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 37, pages 13166-13173.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lio, and Yoshua Bengio. 2018. Graph attention networks. In International Conference on Learning Representations.
Bill Yuchen Lin, Ying Sheng, Nguyen Vo, and Sandeep Tata. 2020. Freedom: A transferable neural architec- ture for structured information extraction on web doc- uments. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1002-1102.
Qifan Wang. Yi Fang, Anirudh Ravula, Fuli Feng, Xi- aojun Quan, and Dongfang Liu. 2022a. Webformer: The web-page transformer for structure information extraction. In Proceedings of the ACM Web Confer- ence 2022, pages 3124-3133.
Colin Lockard, Xin Luna Dong, Arash Einolghozati, and Prashant Shiralkar. 2018. Ceres: Distantly su- pervised relation extraction from the semi-structured web. Proceedings of the VLDB Endowment, 11())).
Zilong Wang. Jiuxiang Gu, Chris Tensmeyer, Nikolaos Barmpalios, Ani Nenkova, Tong Sun, Jingho Shang, and Vlad Morariu. 2022b. Mgdoc: Pre-training with multi-granular hierarchy for document image under- standing. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 3984 3993.
Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. 2019. Openceres: When open information extraction meets the stuutstre web. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, Volume I (Long and Short Papers), pages
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019. Huggingface's 's transformers: State-of- the-art natural language processing. arXiv preprint
Colin Lockard, Prashant Shiralkar, Xin Luna Dong, and Hannaneh Hajishirzi. 2020. Zeroshotceres: Zero- shot relation extraction from semi-structured web- pages. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8105-8117.
Chenhao Xie, Wenhao Huang, Jiaqing Liang, Cheng- song Huang, and Yanghua Xiao. 2021. Webke: Knowledge extraction from semi structured web with pre-trained markup language model. In Proceed- ings of the 30th ACM International Conference Information & Knowledge Management, pages 2211- 2220.
Ilya Loshchilov and Frank Hutter. 2018. Decoupled weight decay regularization, In International Confer- ence on Learning Representations.

---

Yang Xu, Yiheng Xu, Tengehao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Florencio, Cha Zhang. Wanxiang Che, et al. 2021. Layoutmmv2: Multi-modal pre-traiming for visually-rich document understanding. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 1thh International Joint Conference on Natural Language Processing (Volume 1: Long Pupers), pages 2579 2591.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang. Furu Wei, and Ming Zhou. 2020. Layouthm: Pre-training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD Interna- tional Conference Knowledge Discovery Data Mining, pages 1192-1200.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in neural informa- tion processing systems, 32.
Yichao Zhou, Ying Sheng. Nguyen Vo, Nick Edmonds, and Sandeep Tata. 2021. Simplified dom trees for transferable attribute extraction from the web. arXiv preprint 201.:22000.224

---

Appendix
B Zero-shot Relation Extraction on Unseen Websites
A Implementation Details
In this paper, we propose ReXMiner to solve the zero-shot relation extraction task where web pages in the training set and the testing set from dif- ferent verticals. Here, we report the results of the additional experiments for the zero-shot relation extraction on unseen websites. To be more specific, in these additional experiments, the web pages the training set and the testing set are from the same vertical but different websites. For each vertical in the SWDE dataset, we select a subset of websites as the testing set and train the model with the rest websites. We select "rottentomatoes" and "yahoo" from Movie Vertical, "yahoo" from NBA Vertical, and "ecampustours" and "usnews" from University Vertical. We report the results in Table
We use the open-source Transformers framework from Huggingface (Wolf et al., 2019) and build ReXMiner on the base of MarkupL.M (Li et al., 2022b). We initialize ReXMiner with the pre- trained weights of MarkupLM, initialize the tra modules with Xavier Initialization (Glorot and Bengio, 2010), and further finetune ReXMiner on the relation extraction tasks. We not incorpo- rate further pre- pre-training on extra corpus. We use one NVIDIA A6000 to train the model with batch size of 16. We optimize the model with Adam W optimizer (Loshchilov and Hutter, 2018), and the learning rate is 2 10-5 We set the number of popularity embeddings (r) as 20, the number of at- tention layers with the common prefix length (a) as 12, the number of attention layers with the relative XML Path (ß) as 3, the total number of samples (n) as 100, and the ratio between the positive and negative samples (/) as 5.
Movie
Model
Pre
Rec
F1
HSNNt
37
50
45
7SCeres-GNN'
50
MarkupLM
55.98
71.30
62.72
Ours
55.15
79.50
65.12
NBA
Model
Pre
Rec
F1
ESCeres-FENNt
ZSCeres-ONN
47
39
42
MarkupLMM
46.49
73.66
57.00
Ours
68.06
62.56
65.19
University
Model
Pre
Rec
F1
HSNNI
47
59
52
ZSCCeresGNNN'
50
49
50
MarkupLM
67.09
70,56
68.78
Ours
73.08
68.73
70.84
Table 5: The experiment results of ReXMiner and base- line models. 'The results of ZeroshotCeres (ZSCeres) are from Lockard et (2020). I We introduce contrastive learning module of ReXMiner to the Marku- pLM framework to solve the relation extraction task.


Reviews:
[{"review_id": "ZTu82cIymT", "reviewer": "Reviewer_uVRd", "summary": "", "questions": "1. What are the zero-shot results of recent LLMs, e.g., gpt4, on the task?   \n2. This point may worth discussion: in industrial practice, few shot may always be acceptable, so why do authors insist on zero-shot as the paper position?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Reproduction**\n\nWe believe it is essential to open-source our code to reproduce the result and benefit future research. As claimed in the last paragraph of the Introduction Section, we will open-source the code on GitHub. We cannot open-source it now simply because of the anonymity requirements from EMNLP.\n\n**The Motivation of Zero-shot WebIE**\n\nZero-shot web mining is an important task in both academia and industry. It has been pursued for years and has significant industry applications [1,2,3,4]. It benefits future research and applications since online services need to be timely updated and adapted to the new changes on the internet but it is not easy to get either few-shot examples or updated models in a short time.\n\nDue to the fast pace of internet development, it is not always possible to get timely updates on the current model. Although it might be true that few-shot examples can be easily obtained in industrial practice, it also takes money and time to update the ongoing model with the few-shot data, especially when it comes to small businesses or even individual users. Therefore, it is worthwhile to explore web mining algorithms or frameworks that can be well generalized to unknown data in a zero-shot setting. \n\n**Experiment Result with OpenAI LLMs**\n\nWe also observe the extraordinary performance of current LLMs in zero-shot and few-shot learning and we believe our focus of this paper is orthogonal to the development of LLMs. \n\nLLMs are good at understanding natural language and solving general NLU tasks thanks to the extensive pre-training data. However, in our task, we focus on a specific task: the relation extraction in web pages. We seek signals from both the XML paths and the textual contents on the web page. We design special modules for transformer architecture to let LMs better leverage the XML paths. We believe that our work shows the importance of relative relation between HTML nodes in web mining. \n\nWe believe the HTML data should be already included in the pre-training of LLMs from OpenAI and assume the HTML source code is presented in the textual format or rendered as images. However, it is still an open question of how to leverage the structural signals from HTML in a fine-grained way as what we can realize in the era of BERT-like models.\n\nFurthermore, if LLMs are used to solve the web mining problem, we believe that the challenge is to design proper prompt templates and select suitable demo samples to let LLMs understand the HTML structure in the pure text format. Due to the time constraint,  We ran the experiment with GPT-4 on 200 random samples in Movie vertical and reported the performance here. We input the same information to GPT-4 as our model.\n|                     | Precision | Recall | F1-Score |\n|---------------------|-----------|--------|----------|\n| GPT-4 (200 samples) | 33.60%     | 19.74%  | 24.87%    |\n| ReXMiner (Fulll set)          | 45.36%     | 49.36%  | 47.28%    |\n\nWe can observe that although GPT-4 is powerful in zero-shot and few-shot learning, it is still a non-trivial task to learn from XML paths to extract related pairs.\n\n**Experiment Result with WebKE**\n\nWe thank the reviewer for pointing out the missing reference. I will cite the paper in our final version. Here we compare the difference between our framework and WebKE and report the performance with the WebKE model on our task. We will include this in our future version.\n\n1. *Different task settings:* WebKE model is designed for site-specific OpenIE in web pages. As defined in [1] and [2], the site-specific OpenIE depends on the knowledge learned from the web pages of the same website. To be more specific, the training set and testing set of WebKE are from the same website. However, following the zero-shot setting defined in [2], our task is zero-shot OpenIE which is harder than site-specific OpenIE. In our problem setting, the training set and testing set are of different domains. The model is asked to transfer knowledge from one domain to another. So we have three settings: Movie+NBA⇒Univ, NBA+Univ⇒Movie, and Univ+Movie⇒NBA. The three settings defined by [1] and [2] are as follows (from Table 1 of [2]):\n\n| Setting Level | Site-specific | Test set's relation to Training set     | Example                                                                      |\n|---------------|---------------|-----------------------------------------|------------------------------------------------------------------------------|\n| III (WebKE)       | Yes           | Different web pages of the same website | Train with amazon.com/001.html and test with amazon.com/002.html             |\n| II            | No            | Different websites of the same vertical | Train with amazon.com/001.html and test with <another_shopping>.com/001.html |\n| I (Ours)      | No            | Different verticals                     | Train with amazon.com/001.html and test with NBA.com/001.html                |\n\n2. *Different web page structure encoding:* As discussed in the paper, one motivation of this work is to let language models better understand the structural information of web pages. In WebKE, they render the web pages into an image and use the coordinates of the text nodes in the web page to encode the structural information. In comparison, we skip the rendering and directly encode the XML paths. We believe that our method is more efficient since rendering may take a long time to process, and our method is more robust since the coordinates of the text nodes may be influenced by the different rendering platforms (web pages may look different from cell phone and laptop) but the XML paths are always the same.\n\n3. *Different solution for negative cases:* Both our work and WebKE have noticed a large number of negative cases in the relation extraction. In our work, we solve this issue through negative sampling and contrastive loss. In WebKE, they extract relations and objects by predicting the start and end of each phrase and also use heuristic filtering to further process the prediction results.\n\nTo quantify the difference between WebKE and our method, we run the exact same code of WebKE and only change the training set and testing set to the zero-shot relation extraction setting. Due to the time constraint, we only run the experiment setting of Movie+NBA⇒Univ. The results are:\n|                     | Precision | Recall | F1-Score |\n|---------------------|-----------|--------|----------|\n| WebKE | 0.41%     | 13.59%  | 0.80%    |\n\nWe believe the difference is due to the different task settings and how to encode the web page structure. In the WebKE setting, the training set and testing set are different web pages but from the same website. The layout of the web pages can be similar so the coordinates can be easily used to learn the relation and further predict the object. However, in our zero-shot relation extraction setting, the training set and testing set are of different domains, making it hard to extract these pairs through coordinates. The relation revealed by the XML paths in our approach stays the same and contributes to our better performance.\n\n**Each Component in Our Framework**\n\nHere we clarify our insight into designing each module in our framework.\n\n1. *Relative XML Path:* While it is common to acquire meaningful signals from the relative position of tokens in a sequence, it hasn’t been a feasible way to leverage relative XML paths in web mining. In our work, we observe the long common prefix in the relative node pairs and implement the relative XML path modules. We separate the relative XML information into two parts: the depth and the exact relative path, which serves as a clear encoding of the relative XML paths.\n\n2. *Popularity Embedding:* The frequency is commonly used to reveal the importance of a phrase in the text. As discussed in the paper, the relation text nodes are more frequent than the other nodes and their neighboring words are different. Therefore, we combine the frequency information into the context embeddings to see the performance.\n\n3. *Contrastive Learning:* In the task setting of zero-shot relation extraction, the relationship is explicitly extracted. For example, “Age:”, and “24” are two text nodes in the web page, and we need to extract both of them as a related pair. So the task reduces to a binary classification between any two text nodes, which introduces a large proportion of negative cases. To balance the positive and negative cases, we use negative sampling to control the ratio.\n\n- [1] OpenCeres: When Open Information Extraction Meets the Semi-Structured Web\n- [2] ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages\n- [3] CERES: Distantly Supervised Relation Extraction from the Semi-Structured Web\n- [4] Zero-shot Entity Extraction from Web Pages", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. easy to follow\n2. The technical details are solid and reasonable.", "reasons_to_reject": "1. lack comparison to existing PLM-based methods, such as [1,2] and openai-gpt based methods.\n2. The components in the paper are popular approaches, I suggest that the authors give more insight and example on how each component (e.g. contrasive loss) works on each scenario.\n3. poor reproducibility, without any code available for evaluation.\n\n\n\n\n\n\nReferences (see missing references)", "questions_to_authors": "1. What are the zero-shot results of recent LLMs, e.g., gpt4, on the task?   \n2. This point may worth discussion: in industrial practice, few shot may always be acceptable, so why do authors insist on zero-shot as the paper position?", "paper_topic_and_main_contributions": "This paper addresses the challenge of understanding semi-structured web pages in web mining, particularly when little is known about the subject or template of a new page. The authors note that current methods, which involve embedding XML source code into the transformer or encoding the rendered layout with graph neural networks, do not adequately account for the relationships between text nodes within and across pages.\n\nThe main contributions of this paper are the development and implementation of ReXMiner, which offers a more accurate and efficient method for key-value pair extraction within a web page. ReXMiner achieves this by encoding the shortest relative paths in the Document Object Model (DOM) tree of the web page. It also incorporates the popularity of each text node by counting the occurrence of the same text node across different web pages.\nThe authors use contrastive learning to address the issue of sparsity in relation extraction.", "excitement": 3, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "Too many arxiv references, some of which may already have a conference version"}, "extra_text": {"Missing_References": "[1] Li, Zimeng, et al. \"WIERT: Web Information Extraction via Render Tree.\" Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 37. No. 11. 2023.\n\n[2] Xie, Chenhao, et al. \"Webke: Knowledge extraction from semi-structured web with pre-trained markup language model.\" Proceedings of the 30th ACM International Conference on Information & Knowledge Management. 2021."}}, {"review_id": "qW1hcix7Au", "reviewer": "Reviewer_KQqa", "summary": "", "questions": "A. The relative path sequences seem to be simply put together so information about direction and lowest common ancestor is not fed into the model. Does this information have any effect on the model?\n\nB. In this model, path prefixes are added to the first several layers of attention weight as bias and relative paths are added to the next layers of attention weight as bias. How do they interact with each other and ultimately affect the model?\n\nC. The number of these layers is controlled by hyperparameters, how are these hyperparameters determined and do they have a significant effect on the model?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and suggestions. We will clarify the confusing points and add them to our later version.\n\n**Experiments on other datasets and analysis**\n\nWe understand that it is important to validate a method with more data and experiments. However, through our literature review, the SWDE with extended relation labels [1] is the only existing dataset designed for HTML-aware relation extraction in web pages. Most of the existing datasets for relation extraction tasks consist of pure text and skip the rich HTML data even if their data source may be web pages [2][3].\n\nWe believe that the current SWDE dataset is extensive enough which includes 3 verticals, 21 websites, and 27641 web pages all in total, and this is also the dataset used by baselines to demonstrate their performance in the relation extraction tasks.\n\nTo further evaluate our method, we add more experiment settings. In our current version, we explore the performance of ReXMiner when transferring the knowledge from existing verticals to unknown verticals (e.g. Movie + University → Nbaplayer). Now, we add one more setting according to the task definitions in [4]. We train our model on a set of known websites and apply it to unknown websites. Here the training set and testing set are of the same vertical but different websites. Due to time constraints, Here are the new results of Movie vertical. We will report the other two verticals in our final version.\n\n|                | Precision | Recall | F1-Score |\n|----------------|-----------|--------|----------|\n| ZeroshotCeres  | 49.0      | 51.0   | 50.0     |\n| MarkupLM       | 55.98     | 71.30  | 62.72    |\n| ReXMiner(Ours) | 55.15     | 79.50  | 65.12    |\n\nMore details about the problem settings defined in [1] and [4] (from Table 1 of [4]).\n\n| Setting Level | Site-specific | Test set's relation to Training set     | Example                                                                      |\n|---------------|---------------|-----------------------------------------|------------------------------------------------------------------------------|\n| III           | Yes           | Different web pages of the same website | Train with amazon.com/001.html and test with amazon.com/002.html             |\n| II (Added new setting)           | No            | Different websites of the same vertical | Train with amazon.com/001.html and test with <another_shopping>.com/001.html |\n| I (Our current setting)      | No            | Different verticals                     | Train with amazon.com/001.html and test with NBA.com/001.html                |\n\n**Details about the direction of relative XML path and lowest common ancestor**\n\nIn short, we indeed encode the direction of XML paths and the lowest common ancestor in our Self-Attention with Relative XML Paths. We understand that the direction of relative XML paths and lowest common ancestor are of great importance to the relationship between text nodes in web mining. We encode them as follows.\n\n1. *Lowest common ancestor:* As shown in Line 352 and 353, the t_m denotes the lowest common ancestor and its embedding is part of the XML Path bias.\n2. *The order of XML tags in relative XML paths:* We encode the order of XML tags through the concatenation of embedding vectors. As shown in Line 352 and 353, we first embed each XML tag into a short vector and concatenate these vectors sequentially. Then we use a linear layer to convert the concatenated vector into a bias term. Here the concatenation ensures that the bias term involves the direction information. \n3. *The direction of relative XML paths:* Similarly, in Line 352 and 353, the relative XML path from node $x_i$ to $x_j$ is divided into two parts, and we embed each part with different linear layers, b, and b$^\\prime$, which ensures that the added bias terms of $x_i\\to x_j$ and $x_j\\to x_i$ are different.\n\n**Details about the Self-Attention with Relative XML Paths**\n\nIn the design of Self-Attention with Relative XML Paths, we would like to encode both the depth information and the exact relative XML paths. The depth is a simple scalar and easy to learn as a bias term, while the exact relative XML paths are more informative and computing-intensive to learn as a bias term. We do a trade-off between these two and embed them separately in the multi-layer attention as bias terms. In the first alpha layers, the added terms are from the depth value which is easy to learn and well combined with the pretrained weight. In the next beta layers, the added terms are from the exact relative XML paths and the hidden states already contain rich semantic information, so the combined signals can be well kept during the final prediction.\n\n**Hyperparameters of the Self-Attention with Relative XML Paths**\n\nAs discussed, we would like to encode both the depth information and the exact relative XML paths into the attention layers, so we design two different bias terms. To balance the computing, we use more layers to encode the depth information and the last 3 layers to encode the exact relative XML paths. We believe the choice of hyperparameters is not a big issue in our framework and other combinations might lead to a better result.\n\n- [1] OpenCeres: When Open Information Extraction Meets the Semi-Structured Web\n- [2] REDTab: A Relation Extraction Dataset for Knowledge Extraction from Web Tables\n- [3] DocRED: A Large-Scale Document-Level Relation Extraction Dataset\n- [4] ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This paper proposes a kind of novel and effective feature fomulation for web pages by encoding the shortest relative paths in the DOM tree and counting the occurrence of the text node.\n\nThe transformer-based model ReXMiner using the above features reaches the state-of-the-art in the task of zero-shot relation extraction for web mining, which is hopeful to facilitate the exploitation of the unlabelled web corpus.", "reasons_to_reject": "The contribution of this paper seems somewhat insufficient as a long paper. As a considerable part of ReXMiner comes from the existing work of the model MarkupLM, the adding part including relative path bias and popularity embeddings are more like an incremental work. I believe this paper should be enriched in breadth or depth to fully demonstrate the advantages and importance of the new approach. For example, more experiments on other datasets and tasks about web pages, like what MarkupLM does, or more analytical experiments illustrating the principles of the new method.", "questions_to_authors": "A. The relative path sequences seem to be simply put together so information about direction and lowest common ancestor is not fed into the model. Does this information have any effect on the model?\n\nB. In this model, path prefixes are added to the first several layers of attention weight as bias and relative paths are added to the next layers of attention weight as bias. How do they interact with each other and ultimately affect the model?\n\nC. The number of these layers is controlled by hyperparameters, how are these hyperparameters determined and do they have a significant effect on the model?", "paper_topic_and_main_contributions": "In this paper, a new method called ReXMiner is proposed for zero-shot relation extraction in web mining. It encodes the shortest relative paths in the DOM tree of the web page and counts the occurrence of the same text node across different web pages. Experiments on public benchmarks show that this method outperforms the state-of-the-art baselines in the task of zero-shot relation extraction in web mining.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "lNA6p0cMoJ", "reviewer": "Reviewer_3bEb", "summary": "", "questions": "I am wondering if the author has considered IDF as the popularity of text node has been considered. \n", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's recognition of the effectiveness of our framework. Here we provide more experiments and results to further demonstrate our proposed method.\n\n**More experiments and results**\n\nOur motivation is to leverage the relative information of the XML path to help language models better understand the structure of web pages especially when handling unknown web pages. \n\nIn our current problem setting, we follow the task definition in [1] and explore the hardest setting where the models need to transfer the knowledge from the existing vertical to the unknown vertical.\n\nWe add one more problem setting where we ask the model to extract related pairs on the unknown websites. To be more specific, the training set and testing set are of the same vertical but from different websites. We believe this new setting is slightly easier than our current setting since websites of the same vertical may share common domain knowledge. Here are the results on the Movie vertical (Due to time constraints, we only finished this one setting. We will report the other two verticals in our final version).\n|                | Precision | Recall | F1-Score |\n|----------------|-----------|--------|----------|\n| ZeroshotCeres  | 49.0      | 51.0   | 50.0     |\n| MarkupLM       | 55.98     | 71.30  | 62.72    |\n| ReXMiner(Ours) | 55.15     | 79.50  | 65.12    |\n\nMore details about the problem settings defined in [1] (from Table 1 of [1]).\n\n| Setting Level | Site-specific | Test set's relation to Training set     | Example                                                                      |\n|---------------|---------------|-----------------------------------------|------------------------------------------------------------------------------|\n| III           | Yes           | Different web pages of the same website | Train with amazon.com/001.html and test with amazon.com/002.html             |\n| II (Added new setting)           | No            | Different websites of the same vertical | Train with amazon.com/001.html and test with <another_shopping>.com/001.html |\n| I (Our current setting)      | No            | Different verticals                     | Train with amazon.com/001.html and test with NBA.com/001.html                |\n\n**Use of IDF as the popularity of text node**\n\nWe thank the reviewer's insightful suggestions. Indeed, our definition of the node's popularity is quite related to the well-known word IDF. In our definition of popularity, we count the number of web pages where the given text node exists and embed it through the equation in Line 276: $\\lfloor \\tau \\log(pop(w)) / \\log(N) \\rfloor$. This definition is similar to the word IDF but uses a different way to normalize the value. Since the word IDF can range from 0 to +inf, which makes it hard to embed the values into vectors. We believe that our method produces bounded values and contains the same information as IDF.\n\n- [1] ZeroShotCeres: Zero-Shot Relation Extraction from Semi-Structured Webpages", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "Paper is well structured and clear to follow. \nPaper tackles one important problem in web text mining field. \nPaper shows the proposed solution can outperform the state of art models. \nPaper show cases the gained benefits by combining more structural or popularity information. \n", "reasons_to_reject": "Authors can consider to test more on other datasets. \nIt will be even better to compare the inference/training time. \n\n", "questions_to_authors": "I am wondering if the author has considered IDF as the popularity of text node has been considered. \n", "paper_topic_and_main_contributions": "This paper studies the approach to understand semi-structured web pages utilizing the relation between text nodes within and across pages. \nIt proposes a framework ReXMiner to encode the shortest relative paths in document object model and the popularity info of text node.\nIt also uses the contrastive learning approach to facilitate the training. \nExperiments conducted on SWDE dataset shows the model achieves or outperforms the SOTA performance. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UbfUI9GJBf", "reviewer": "Reviewer_tb1s", "summary": "", "questions": "A. There seems to be a confusion in the use of the notions of text nodes, words and tokens. A text node is an element from the DOM tree and can be a token or word, a phrase, a sentence or even a paragraph. Could you please clarify what is the nature of the elements in the input sequence for the model, i.e. what w_i means in the (w_i, xpath_i) notation? Is it a text node or a token? If they are tokens, are the text nodes tokenized with a BPE tokenizer? \n\nB. If the input of the model is a sequence of tokens (with their xpaths), then h_i and h_j in the Biafine equation (line 383) are representations of tokens, and x_i and x_j (line 384) are tokens, not text nodes. So how does the system group tokens to produce pairs of text nodes, such as \"(Color type, Technicolor prints)\" ?\n\nC. It is claimed that \"contrastive learning\" is used to train the model (section 4.4) but I am not sure if that's correct: the loss as defined in lines 377-393 seems to be a normal supervised cross-entropy loss (prediction vs true label on a single sample) and not a contrastive loss.  So it seems that the negative sampling is used simply to alleviate the problem of class imbalance and not for a contrastive loss. Could you please clarify?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer for the detailed comments and insightful suggestions. Here we would clarify the confusing points in our paper and will also revise the related parts in our future version. All the typos, missing references, and presentation issues will also be well handled.\n\n**Clarification of the model inputs**\n\nTo encode the textual contents of a web page into a language model, we follow the previous works [1][2] where the contents in all the nodes are concatenated as a long string to be fed into the model. We take note of the beginning and ending indices to reconstruct each node. As for the XML paths, all the words in a given node share the same XML path encoding. So, given a node in web page, $([w_1,...,w_n], xpath)$, we tokenize it into such input sequence: $(w_1, xpath), (w_2, xpath),..., (w_n, xpath)$.\n\n**Clarification of the extraction step**\n\nAs we discussed, when converting the nodes into the input sequence, we take note of the beginning and ending indices. Our task is to extract the related pairs of nodes, so we extract the hidden states corresponding to the end index as the representation for the given node. These hidden states are fed into the Biaffine layers to further predict the relationship between nodes.\n\n**Ambiguous use of contrastive loss**\n\nThanks for pointing out this issue. We will remove the ambiguous use of the term and make the narration clearer. \n\n- [1] MarkupLM: Pre-training of Text and Markup Language for Visually-rich Document Understanding\n- [2] DOM-LM: Learning Generalizable Representations for HTML Documents", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed extension of the MarkupLM model, with the encoding of text node \"popularity\" and relative XML paths between text nodes, is relatively simple and improves performance in zero-shot relation extraction setting, as shown by the experiments and the ablation study.", "reasons_to_reject": "- Two important technical aspects of the proposed method are missing or are not clear: \n\t- (1) Regarding the representation of model input: from section 4 and in particular lines 280-284, the model input is defined as the sequence of \"text nodes\" (w_i, xpath_i) from the XML/HTML document, which is confusing, as text nodes can be phrases, sentences or even paragraphs... In Figure 3 though, it seems that the model input is a sequence of tokens, not text nodes, which would make more sense, but the paper does not mention tokenization at all.\n\t- (2) Regarding the extractions: from the provided examples (Table 4), the model extracts pairs of text nodes, such as \"(Color type, Technicolor prints)\". If, as it is probably the case, model input is a sequence of tokens (from tokenized text nodes), then h_i and h_j in the Biaffine component (equation at line 383) are representations of tokens, not text nodes and hence, P(x_i, x_j) is about tokens x_i and x_j. Therefore, the paper lacks a description of how tokens are grouped in order to produce pairs of text nodes from the predicted pairs of tokens.", "questions_to_authors": "A. There seems to be a confusion in the use of the notions of text nodes, words and tokens. A text node is an element from the DOM tree and can be a token or word, a phrase, a sentence or even a paragraph. Could you please clarify what is the nature of the elements in the input sequence for the model, i.e. what w_i means in the (w_i, xpath_i) notation? Is it a text node or a token? If they are tokens, are the text nodes tokenized with a BPE tokenizer? \n\nB. If the input of the model is a sequence of tokens (with their xpaths), then h_i and h_j in the Biafine equation (line 383) are representations of tokens, and x_i and x_j (line 384) are tokens, not text nodes. So how does the system group tokens to produce pairs of text nodes, such as \"(Color type, Technicolor prints)\" ?\n\nC. It is claimed that \"contrastive learning\" is used to train the model (section 4.4) but I am not sure if that's correct: the loss as defined in lines 377-393 seems to be a normal supervised cross-entropy loss (prediction vs true label on a single sample) and not a contrastive loss.  So it seems that the negative sampling is used simply to alleviate the problem of class imbalance and not for a contrastive loss. Could you please clarify?", "paper_topic_and_main_contributions": "The paper addresses the problem of (untyped) relation extraction (i.e. extracted pairs are not labeled) from web pages and in a zero-shot setting (i.e. testing is done on web pages from a vertical/domain not seen during training). The difficulty is obviously in the generalization to documents that can be different not only in their textual and semantic content, but also in their HTML/XML structure and layout.   \n\nThe proposed solution follows existing multimodal methods that embed both text and document structure information. More specifically, and as stated in the paper, it is strongly inspired by the MarkupLM model (Li et al., 2021), which embeds absolute XML Paths along with textual elements within a transformer encoder. \n\nThe main contribution of the paper is the extension of the MarkupLM model (Li et al., 2021) with 2 features: popularity of text nodes (i.e. number of web pages where a text element occurs) and relative XML Path between text elements. The first feature is encoded in the input sequence embeddings (of text nodes), and the second is encoded as a bias term in the self-attention components. The experiments and ablation study show that the two features improve performance.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "IMO, the following paper should be cited and discussed. Although their experiments with the SWDE dataset are not comparable due to different settings, the main contribution of the paper is multimodal (text + XML structure) representation learning, where various structural features are encoded along with text elements:\n\nDeng, Xiang, Prashant Shiralkar, Colin Lockard, Binxuan Huang, and Huan Sun. ‘DOM-LM: Learning Generalizable Representations for HTML Documents’. arXiv, 25 January 2022. [https://doi.org/10.48550/arXiv.2201.10608](https://doi.org/10.48550/arXiv.2201.10608).", "Typos_Grammar_Style_and_Presentation_Improvements": "Typos:\n\nline 143: gathering --> gather\n\nfigure 2 caption: hundreds web pages --> hundreds of web pages\n\nThroughout the paper: the definite article \"the\" is sometimes used in plural noun phrases where it should not be, e.g. \"which rely on the syntactic constraints\" --> \"which rely on syntactic constraints\". Proofreading is probably needed to fix all the cases."}}]

Author rebuttal:

Meta-review: The paper presents ReXMiner, a new approach to zero-shot relation extraction from webpages that encodes relative shortest paths in the Document Object Model (DOM).  The sense of zero-shot here is following prior work (e.g. Lockard et. al.) in that the model is trained on one domain (e.g. NBA), and tested on webpages from another domain (e.g. Movies).

Reviewers appreciated the simplicity of the approach as a nice extension to MarkupLM, and the fact that the paper is relatively well written. 
 However, some reviewers comments raised concerns about whether the contribution is incremental with respect to prior work to justify acceptance as a long paper.  Reviewers also indicated they would like to see a comparison to a baseline that uses LLMs.

I have some concerns about the experiments, for example, what are the dev/test splits used to select hyperparameters - are these the same that were were used in prior work?  The ZeroShotCeres paper indicates they do not use a development set of pages from the target websites, whereas in Appendix A the paper mentions: "As for the hyper-parameters, we select a subset of web pages from each web site in the SWDE dataset as validation set to find the best hyper-parameters.", which seems to indicate that pages from the target website was included in the development set, so this seems like it may not be a fair comparison.
Decision: Accept-Findings