MUSTIE: Multimodal Structural Transformer for Web Information Extraction
Year: 2023

Abstract:
The task of web information extraction is to extract target fields of an object from web pages, such as extracting the name, genre and actor from a movie page. Recent sequential modeling approaches have achieved state-of-the-art results on web information extraction. However, most of these methods only focus on extracting information from textual sources while ignoring the rich information from other modalities such as image and web layout. In this work, we propose a novel MUltimodal Structural Transformer (MUST) that incorporates multiple modalities for web information extraction. Concretely, we develop a structural encoder that jointly encodes the multimodal information based on the HTML structure of the web layout, where high-level DOM nodes, low-level text, and image tokens are introduced to represent the entire page. Structural attention patterns are designed to learn effective cross-modal embeddings for all DOM nodes and low-level tokens. An extensive set of experiments has been conducted on WebSRC and Common Crawl benchmarks. Experimental results demonstrate the superior performance of MUST over several state-of-the-art baselines.

heading: 1 Introduction
text: The world wide web has grown explosively in the past decades, with millions of new web pages being created everyday. Web pages and documents have been widely used and become a powerful resource for humans to obtain information. For example, Figure 1 shows a movie page from the IMDB website, which contains structured movie information including movie name, description, genre, etc. This information is essential to facilitate new experiences in applications like web search and retrieval (Crescenzi and Mecca, 2004; Yan et al., 2009). There has been an enduring demand for automatic information extraction from unstructured
∗Corresponding authors.
or semi-structured web pages to create structured knowledge bases (Chang et al., 2006; Hao et al., 2011). Therefore, it is an important research problem to extract structured information from web pages (Carlson and Schafer, 2008).
Web information extraction (Manabe and Tajima, 2015; Wu et al., 2018) poses a lot of challenges to researchers in both academia and industry, due to the unstructured nature and the diverse layout patterns of the web documents (Xiong et al., 2019; Lockard et al., 2019). Moreover, web data often contains multiple modalities such as texts, tables, and images. A substantial amount of research (Katti et al., 2018; Zhang et al., 2021) has been proposed for automatic web information extraction, including early works of template-based extraction (Dalvi et al., 2011). However, these methods clearly do not scale up to billions of websites. Deep learning models (Gogar et al., 2016; Zhou et al., 2021) attempt to use supervisions from markup pages (Tempelmeier et al., 2018) to build different extractors for different fields.
With the recent development of natural language processing (Vaswani et al., 2017), language models have been successfully applied to web informa-
2405
tion extraction. These methods first convert the web document to a text sequence by concatenating all the text nodes (Gupta et al., 2020) or to a connected graph by using the rendered page (Qian et al., 2019), and then adopt sequential modeling such as LSTM (Lin et al., 2020) or attention networks (Hwang et al., 2021) to extract the target fields from the web. More recently, several multimodal language models (Dong et al., 2020; Xu et al., 2020) have been proposed to extract web information from both textual and visual signals. Despite achieving promising results on web information extraction, there are several major limitations for existing natural language models. First, they encode each modality of the web document independently with an individual encoder, which fails to capture the connections among different modalities, resulting in a less effective web representation. Second, they do not fully encode the semi-structure HTML layout, which carries important knowledge about the correlations between different fields. For example, in Figure 1, the DOM nodes corresponding to the movie ‘name’ usually appear directly after the image node in the HTML, while the ‘release date’ and ‘duration’ nodes are often siblings. Therefore, encoding the structural HTML would benefit the information extraction. Third, the texts and images from individual modalities are simply concatenated, making existing Transformer models incapable of handling large web documents.
To address these challenges, in this work, we propose a novel MUltimodal Structural Transformer (namely MUST), which incorporates multiple modalities for web information extraction. In particular, we design a multimodal encoder with a structural attention mechanism to jointly encode all the DOM nodes from multiple modalities, and learn the cross-modal embeddings for them. Intuitively, MUST leverages the web layout structure that naturally connects DOM nodes from all modalities for more effective attention weight computation. The information of the target fields is then extracted from the learned node embeddings. We conduct evaluations of our model on WebSRC and Common Crawl benchmarks, and show the superior performance of MUST over several state-of-the-art methods. The experimental results also demonstrate the effectiveness of the structural attention in modeling web documents with multimodal data. The main contributions are summarized as follows:
• We propose a unified Multimodal Structural
Transformer for web information extraction, which effectively models the multimodal data with the HTML layout and jointly extracts the information for the target fields.
• We design a structural attention mechanism to capture the correlation among different modalities of the web document for learning effective cross-modal embeddings.
• We conduct an extensive set of experiments on two benchmarks and demonstrate the effectiveness of the proposed approach.

heading: 2 Related Work
text: Web Information Extraction Early works in web information extraction are wrapper induction methods (Kim and Shim, 2011; Lockard et al., 2018), which construct templates by learning the desired patterns from the web documents. Several deep learning methods (Sleiman and Corchuelo, 2013; Wang et al., 2019) are proposed to extract or classify a text node to a set of fields using its textual and visual features, e.g., classify whether a text node is the ‘name’ field.
With the recent advancement in natural language processing (NLP) (Devlin et al., 2019), an increasing number of language models (Appalaraju et al., 2021; Wang et al., 2020a; Yang et al., 2022; Zhao et al., 2022) have been developed for web information extraction. These methods can be further divided into three main groups. The first group contains the sequential modeling approaches (Herzig et al., 2020; Majumder et al., 2020), which construct a text sequence by concatenating all the text nodes from the web and performing the extraction. Form2Seq (Aggarwal et al., 2020) designs a seq-toseq model with an RNN. WebFormer (Wang et al., 2022a) merges all the text nodes from the HTML and trains a model with hierarchical attention. The second group includes the graph learning models (Qian et al., 2019; Lockard et al., 2020), which treat the web document as a graph connecting multiple rendered components and directly learn the web representation on the graph. FormNet (Lee et al., 2022) generates a structure-aware graph from the rendered web document and uses the graph convolutional network (GCN) for obtaining the node embeddings. The third group consists of the multimodal methods (Gong et al., 2017; Liu et al., 2019; Wang et al., 2020b; Li et al., 2021), which learn to extract field information from both textual and
visual clues on the web. LayoutLMv2 (Xu et al., 2021) adopts a two-stream multimodal Transformer encoder to model the interaction among text and image.
Structure and Efficient Transformers Our work is also related to those Transformer models (Tay et al., 2022; Rae et al., 2020; Wang et al., 2022b) that focus on efficiently encoding structure and large sequences. ETC (Ainslie et al., 2020) and Longformer (Beltagy et al., 2020) describe a method to use a global memory with a relative attention pattern (Shaw et al., 2018, 2019) to represent the structure text input. Transformer XL (Dai et al., 2019) develops an approach to encode long text sequences beyond a fixed size. HIBERT (Zhang et al., 2019) uses hierarchical attention on the equally divided input blocks. Random sparse attention is utilized in BigBird (Zaheer et al., 2020) to reduce the quadratic computations to linear time. These methods achieve promising results in dealing with structure and large input. However, they cannot be directly applied to encode HTML layout with multiple modalities.

heading: 3 Multimodal Structural Transformer

heading: 3.1 Problem Setting
text: In this section, we formally define the problem of web information extraction. A web document can be essentially represented as a HTML DOM tree
H . It usually contains information from multiple modalities, such as texts and images, which are naturally the leaf nodes in the DOM tree (see Figure 2). In order to encode the target field, we create a special DOM node ‘Field’ under the root of the DOM tree, with a leaf node representing the text field attached to it. Similarly, for ‘<img>’ DOM nodes, we apply Optical Character Recognition (OCR) to obtain the texts from the image and add these OCR nodes under the image node. We denote the leaf nodes as C = (C1, C2, . . . , Cn), where Ci represents the i-th leaf node in the DOM tree. For each leaf node, it is either a text sequence or an image, i.e., Ci = (wi1, . . . , w i ni), where w i j is the j-th word or image token in Ci. The goal of web information extraction is that given a target field T , extract its corresponding information from the web document. For example, for the text field ‘Director’, we aim to obtain ‘Steven Spielberg’. And for the target field ‘Name’, ‘Jurassic Park’ would be the correct extraction.

heading: 3.2 Overview
text: The overall model architecture of MUST is shown in Figure 2, which consists of three key components, the embedding layer, the MUST encoder and the extraction layer. The embedding layer initializes the embeddings of both the text and image tokens (referred to as TI tokens in the rest of the paper), as well as the DOM nodes. The MUST encoder jointly encodes the multimodal information
from the DOM tree with structural attention patterns to capture the correlations among DOM nodes and text/image tokens. The extraction layer extracts the answer from the embedding of the ‘Field’ with a Transformer decoder.
There are several advantages to our modeling. (1) The multimodal information on the web is jointly encoded through a unified structural encoder, where the information from different modalities effectively communicates with each other. (2) We directly encode the HTML DOM tree instead of sequentializing the document (Chen et al., 2021; Wang et al., 2022a) which does not fully capture the structure information, or generating a graph from the web (Qian et al., 2019; Lee et al., 2022) which requires careful design of the nodes and edges. (3) Our model does not concatenate all the inputs, allowing it to scale to large documents.

heading: 3.3 Embedding Layer
text: Existing multimodal approaches (Xiong et al., 2019; Li et al., 2021) encode textual and visual features separately with individual encoders. Different from previous works, we jointly encode texts and images together with the DOM tree from the web document in a multimodal structural Transformer.
In the embedding layer, we initialize the embeddings for all DOM nodes and TI tokens with a ddimensional vector. The embedding of each DOM node can be viewed as a summarization of the subtree under it. For example, in Figure 2, the DOM node ‘<head>’ represents the whole web document and can be used for document-level classification. The ‘<img>’ DOM node essentially contains all the information about that image. For a DOM node, its embedding is constructed by adding a node embedding, a type embedding and a tag embedding. For a TI token, it is constructed by a word/patch embedding and a type embedding. The word embedding (Zou et al., 2013) is widely used in language models. The patch embedding is obtained by a linear projection of the visual feature from ResNet101 (He et al., 2016). The type embedding is used to indicate the type of the token, i.e., DOM node, text or image. The tag embedding represents the HTML tag of the DOM node such as ‘<div>’ and ‘<img>’. All these embeddings are trainable.

heading: 3.4 MUST Encoder
text: The MUST encoder contains a stack of L identical layers, which connects the DOM nodes, texts and images from multiple modalities with a structural
attention mechanism, and learns cross-modal contextual representations of the web document and field. In each encoder layer, there are four different attention patterns. First, structural attention among DOM nodes, which transfers the knowledge across the DOM tree. Second, bottom up attention from text/image token to DOM node. Third, top down attention that passes the information from DOM nodes to the text/image token. Fourth, local attention that learns contextual embeddings from other TI tokens in the same leaf node.
DOM-to-DOM Attention The DOM-to-DOM attention is designed to propagate the information from one DOM node to another, which essentially calculates the attention weights among the DOM nodes. We utilize the connections in the DOM tree H to compute the DOM-to-DOM attention, i.e., we allow each DOM node to attend to a set of DOM nodes in the DOM tree, including itself, its parent, children and siblings. For instance, the DOM node ‘<img>’ will attend to (besides itself) the parent node ‘<div>’, the children ‘<alt>’ and two ‘<OCR>’ nodes, and the sibling node ‘<div>’. Formally, given the DOM nodes embedding XD, the DOM-to-DOM attention is defined as:
eNNij = x D i W NN Q (x D j W NN K + t NN ij )
T / √ d
αNNij = exp(eNNij )∑
ℓ∈S(xD i
) exp(eNNiℓ )
, for xj ∈ S(xDi )
where S(xDi ) denotes the set of DOM nodes that xDi can attend to. W NN Q and W NN K are learnable weight matrices, and tNNij are learnable vectors representing the connection type between the two nodes, i.e. self, parent, child or sibling. d is the embedding dimension.
Bottom-Up Attention There are several choices for designing the Bottom-Up attention. For example, allowing full attention from TI tokens to a DOM node. However, the computation grows linearly with the total number of the TI tokens, which is costly for large web documents. Therefore, in the Bottom-Up attention, we only enable attention from TI tokens to the DOM node they belong to. Note that for Bottom-Up attention, only leaf nodes are involved. For instance, in Figure 2, the ‘<h1>’ DOM node only directly receives information from the text tokens within it, i.e., ‘Jurassic’ and ‘Park’. The information contained in other TI tokens will be propagated to the ‘<h1>’ DOM node
through DOM-to-DOM attention. Denote the TI token embeddings as XTI , the restricted Bottom-Up attention for a leaf node Ci is defined as:
eBUij = x D i W BU Q (x TI j W BU K )
T / √ d
αBUij = exp(eBUij )∑
ℓ∈Ci exp(e BU iℓ )
, for j ∈ Ci
where WBUQ and W BU K are weight matrices in Bottom-Up attention.
Top-Down Attention In Top-Down attention, each TI token directly connects with every DOM node, absorbing the high-level representation from these DOM nodes. For example in Figure 2, the text token ‘Jurassic’ from leaf node ‘<h1>’ attends to all DOM nodes in the DOM tree. The definition of the Top-Down attention is similar to the above Bottom-Up attention except that each TI token attends to all DOM nodes. Full details are in Appendix A.
Local Attention The local attention is the traditional attention mechanism used in various existing Transformer models (Devlin et al., 2019; Dosovitskiy et al., 2021), which learns contextual token embeddings from the input sequence. Again, in our design, we only restrict local attention between two TI tokens from the same leaf DOM node to further reduce the computational cost.
The final representation of the DOM nodes and TI tokens can be achieved by merging the above structural attention patterns. The output embeddings for DOM nodes and TI tokens ZD, ZTI are calculated as follows:
zDi = ∑
j∈S(xDi ) α DD ij x D j W D V + ∑ ℓ∈Ci α BU iℓ x TI ℓ W TI V
zTIi = ∑ ℓ∈Ci α LA iℓ x TI ℓ W TI V + ∑ j α TD ij x D j W D V
where all the attention weights αij are described above. WDV and W TI V are the learnable matrices to compute the values for DOM nodes and TI tokens respectively. Intuitively, these structure attention patterns effectively connect the DOM nodes and TI tokens on the web from different modalities, enabling efficient interactions across the DOM tree.

heading: 3.5 Extraction Layer
text: The extraction layer of MUST outputs the final answer for the target field from the web document. We use a Transformer decoder (Vaswani et al.,
2017) on the output embeddings of the DOM node ‘Field’ to generate the extraction word by word:
w̄t = argmax wt
(softmax(WdeX t de))
where Xtde is the decoder output at word position t. Wde is the output matrix which projects the final embedding to the logits of vocabulary size. A copy mechanism (Zhao et al., 2018) is employed into the decoder to allow both copying words from the text nodes, and generating words from a predefined vocabulary during decoding. To further improve the embedding learning, we supplement two auxiliary tasks as shown in Figure 2. (1) extracting the text spans from the text nodes via sequential tagging (Xu et al., 2019; Chen et al., 2021). (2) classifying the web document using the embedding from the ‘<head>’ node. The total loss is defined as:
L = LD + αLSeq + βLCls
where α and β are hyper-parameters to balance among different losses.

heading: 4 Experiments

heading: 4.1 Datasets
text: We evaluate our method on two multimodal benchmarks, WebSRC (Chen et al., 2021) and Common Crawl (Wang et al., 2022a; Li et al., 2022). WebSRC1 is designed for structural reading comprehension and information extraction on the web. It contains 6.5K web pages with their HTML sources and images from 10 domains, e.g. “Jobs”, “Books”, “Autos”, etc. We use the KV-type pages in our experiment, resulting in a subset of 3214 pages with 71 unique fields. These pages are all single object pages containing multiple key-value pairs, e.g. (“genre”, “Science Fiction”). The keys are used as the fields, while the values are the answers to be extracted from the web page. Common Crawl2 is commonly used in various web information extraction tasks. It contains more than 3 billion web pages from various domains, and we choose three domains Movies, Events and Products in the experiments. We further select web pages with schema.org annotations3, which contain the full markup information about the object and are used as the ground-truth labels. The
1https://x-lance.github.io/WebSRC/ 2https://commoncrawl.org/the-data/ 3https://schema.org/
fields are {“Name”, “Description”, “Genre”, “Duration”, “Director”, “Actor”, “Published Date”} for Movies, {“Name”, “Description”, “Date”, “Location”} for Events and {“Name”, “Description”, “Brand”, “Price”, “Color”} for Product pages. We downsample the web pages by allowing at most 2k pages per website to balance the data. More details are provided in Appendix B.

heading: 4.2 Baselines
text: Our model is compared with six state-of-the-art web information extraction methods.
GraphIE (Qian et al., 2019) propagates information between connected nodes through graph convolutions.
FreeDOM (Lin et al., 2020) proposes a twostage neural network to extract the information from text nodes.
SimpDOM (Zhou et al., 2021) treats the problem as a DOM node tagging task and uses a LSTM to jointly encode XPath with the text features.
V-PLM (Chen et al., 2021) models the HTML, text and visual signal together by concatenating their embeddings with individual encoders.
WebFormer (Wang et al., 2022a) concatenates the HTML and the text sequence and builds a sequential tagging model.
MarkupLM (Li et al., 2022) designs a multimodal pre-training model with text, layout, and image, and fine-tunes it for information extraction.

heading: 4.3 Settings
text: We implement MUST using Tensorflow and trained on a 32 core TPU v3 configuration. During training, we use the gradient descent algorithm with Adam optimizer. During inference, we conduct beam search with beam width 6. The details of all hyper-parameters are reported in Appendix C. Following previous works (Li et al., 2022), we use
Exact Match (EM) and F1 as the evaluation metrics. We repeat each experiment 10 times and report the metrics based on the average over these runs.

heading: 5 Results

heading: 5.1 Main Results
text: MUST outperforms the state-of-the-art web information extraction methods on all datasets. We report the performance comparison result on all datasets in Table 1. It is not surprising to see that the node-level extraction methods FreeDOM and GraphIE do not perform well, as they only extract the text from each text node independently or with local information based on the text features. SimpDOM uses a LSTM to jointly encode the XPath information with the text feature, and thus boosts the performance. V-PLM, WebFormer and MarkupLM achieve even stronger results compared to these methods due to the explicit modeling of the HTML. Nevertheless, it can be seen that MUST achieves the best performance over all the compared methods on all datasets. For example, the EM score of MUST increases over 2.57% and 4.61% compared with WebFormer and MarkupLM on Products. The reason is that these sequential modeling and multimodal methods separately encode HTML, text and image with individual encoders, and concatenate them into a single sequence for learning their embedding. In contrast, MUST jointly encodes the multimodal information from the web in a structural manner, which effectively transfers the knowledge among different modalities, leading to better cross-modal embeddings. We also report a field level results of MUST on the Products data in Table 2. We can see that MUST achieves higher performance on ‘Name’ and ‘Brand’ compared to the fields ‘Price’ and ‘Description’. More detailed analysis is provided in Appendix ??.

heading: 5.2 Results on Low-resource Scenario
text: MUST performs reasonably well in lowresource scenarios. We further evaluate the performance of MUST and all other baselines in a low-resource setting. Specifically, we randomly sample 20% and 10% training data from WebSRC and Common Crawl respectively and retrain the models. The F1 scores are reported in Table 3. There are several observation from these results. First, it is clear that all methods suffer from large performance drop. However, the performance gap between the low-resource and full-resource scenarios is relatively small for those methods that encode the HTML information, e.g., V-PLM, WebFormer, MarkupLM and MUST. Our hypothesis is that in the low-resource training, the HTML layout provides additional knowledge beyond the text for information extraction, which is particularly importance under low-resource settings. Second, MUST still outperforms the baselines in most cases. We also observe that MarkupLM achieves even stronger result than MUST on Products. We believe this is due to their large pretraining on web documents, which learns certain common knowledge in the HTML.

heading: 6 Analysis and Discussion

heading: 6.1 Importance of Different Modalities
text: HTML layout plays an important role for web information extraction, while OCR texts and visual information from the web images are also valuable sources that boost the extraction performance. To understand the impact of different modalities from the web document, i.e., HTML layout, OCR texts and visual signals, we conduct an ablation study by removing each modality from
F1
70
75
80
85
90
95
Name Description Brand Price Color
w/o OCR text w/o visual w/o HTML All
Figure 4: Field level importance of different modalities.
our model. Concretely, removing HTML layout means we do not leverage the DOM tree in MUST, but just concatenate the text and image tokens from all leaf nodes. Removing OCR texts or visual signals means delete the corresponding DOM nodes in the DOM tree during encoding. The results of F1 scores on all datasets are illustrated in Figure 3. It is clear that HTML layout plays a crucial role for the information extraction task on all datasets, which is consistent with our expectation. Moreover, both the OCR text and visual information help improve the extraction performances.

heading: 6.2 Field Level Importance of Different Modalities
text: Each modality has different impacts on different fields. While the visual signal is very useful for ‘Color’ extraction, OCR text benefits the extraction of both ‘Price’ and ‘Brand’. To further analyze the impact of different modalities on different fields, we conduct another field level ablation study on the Products data. The experimental settings are the same as in the above experiment, and we remove each modality at a time. The results of field level F1 scores are shown in Figure 4. We observe that HTML layout still plays an essential role across all fields. It can be seen from the results that the visual signal does not help too much on ‘Name’ and ‘Description’ extraction, but clearly improves the performance on ‘Color’ extraction. The reason is that many product images carry the information about the product color, and therefore can be useful when extracting the product ‘Color’. We also
observe that the OCR text boosts the extraction of ‘Brand’, as it is often the case that product ‘Brand’ is mentioned in the product image. We provide more case studies in Appendix ??.

heading: 6.3 Impact of Different Attention Patterns
text: Every attention pattern has a positive impact on the model performance, while MUST with all structural attention patterns achieves the best performance. In this ablation study, we evaluate the impact of different attention patterns on the model performance by eliminating each attention at a time. Concretely, we train three additional models without the three attentions respectively, i.e., DOM-to-DOM, Bottom-UP and Top-Down attention. Note that we always keep the Local attention as it is the fundamental component of Transformer models. The F1 scores of these three models together with the original MUST on all datasets are shown in Figure 5. First, we observe clear model performance drop without the Bottom-Up attention on all datasets. This is because the Bottom-Up attention is used to transfer knowledge from leaf nodes (containing text and image information) to DOM nodes, which is important for learning effective contextual embeddings for DOM nodes. We also observe some performance drop, around 1 to 2 percent in terms of F1 score, when eliminating one of the other two attention patterns. This observation validates that the structural attention mechanism is crucial for modeling the multimodal web documents and extracting the information from them. Nevertheless, it is clear that MUST with all attention patterns achieves the best performance.

heading: 6.4 Performance-Scale Trade-off
text: MUST with a 12-layer encoder and a 4-layer decoder achieves good performance-scale tradeoff. We conduct a performance-scale study on different MUST configurations. In particular, the MUST-base model uses a 12-layer encoder with
a 4-layer decoder. We evaluate the model performance with a different number of encoder layers in {2L, 6L, 12L, 24L}, and decoder layers in {2L, 4L, 12L}. The F1 scores of different models are reported in Table 4. It is not surprising to see that Encoder-24L and Decoder-12L obtain the best performances, which is expected. On the other hand, larger models usually require both longer training and inference time. Our MUST model with a 12- layer encoder and a 4-layer decoder performs reasonably well on all datasets, which achieves good performance-scale trade-off.

heading: 6.5 Impact of Multi-task Learning
text: Both text span extraction and web document classification help improve the model performance. To understand the impact of the auxiliary tasks, we evaluate the model performance by varying the hyper-parameters α and β from {0, 0.1, 0.5, 0.8, 2, 10}. Note that we modify one hyperparameter by fixing the other one to the optimal value (see Appendix C). The model performances with different hyper-parameter values are shown in Figure 6. It is clear that both tasks lift the model performance (0 value of α or β means removing that task). However, the text span extraction task plays a more important role compared to the web classification task.

heading: 7 Conclusions
text: This paper presents a novel Multimodal Structural Transformer (MUST) for web information extraction. A structural encoder is developed and used to
jointly encode the multimodal information associated with the HTML layout, where high-level DOM nodes, and low-level text and image tokens are introduced to represent the entire web. Structural attention patterns are designed to learn effective cross-modal embeddings for all DOM nodes and text/image tokens. Experimental results on WebSRC and Common Crawl benchmarks demonstrate the effectiveness of the proposed approach.
Limitations
There are two limitations of the current MUST model. First, although pre-trained language models can potentially boost the performance in web information extraction, pre-train a MUST on web documents has its unique challenges. There are several possibilities for our future exploration. For example, we plan to pretrain a MUST model by incorporating HTML-specific tasks, such as masking DOM nodes and predicting the relations between DOM nodes. Second, our model focuses on web pages with single-object, where each target field only has exactly one answer. For a multi-object page, e.g. a movie listing page, there are different movie names corresponding to different movies on the page. However, methods like repeated patterns (Adelfio and Samet, 2013) can be applied.

heading: A More Technical Details
text: We provide more technical details on our MUST in this section.
MUST Encoder As mentioned in the main paper, the MUST encoder is a stack of L identical layers:
X l = MUST(X l−1), 1 ≤ l ≤ L
where X0 is the input embedding for the first layer, which is obtained from the embedding layer. Each encoder layer contains a structural attention layer followed by a standard feed forward network:
Zk = StrAtt(Xk−1), Xk = FFN(Zk)
The StrAtt layer uses the structural attention mechanism described in the main paper. We supplement the full details of the Top-Down attention and the Local attention.
Top-Down Attention The Top-Down attention is defined as:
eTDij = xTIi W TD Q (x D j W TD K ) T
√ d
αTDij = exp(eTDij )∑ ℓ exp(e TD iℓ )
Local Attention The Local attention is defined as:
eLAij = xTIi W LA Q (x TI j W LA K ) T
√ d
αLAij = exp(eLAij )∑
ℓ∈Ci exp(e LA iℓ )
, for j ∈ Ci

heading: B Dataset
text: B.1 Data Processing The WebSRC dataset contains three types of web pages, i.e. KV (key-value), Comparison and Table. As stated in the main paper, we only use the KV type pages in our experiments. The reason is that both Comparison and Table web pages are more suitable for multi-object extraction, where those objects’ information are described in a table or list and can be obtained directly with repeated pattern or table extraction techniques (Wang et al., 2019). For the KV pages, the key-value pairs only contain value text without any span information in the text sequence of the web page. Therefore, we need to label the span of the value in the text sequence,
since the sequential tagging task in MUST requires token level spans during training.
The Common Crawl dataset contains a huge amount of web pages with schema.org annotations, which are used as the supervision in various information extraction tasks. An example of schema.org Event annotations is shown in Figure 7. It contains the annotation type “https://schema.org/Event”, as well as the annotations for all the event fields including name, description, date and location. In our experiments, we work on three big domains - Movies, Events and Products. We further filter these pages by restricting to English and single object pages (have one single schema.org type annotation). We also label the span corresponding to the field in the text sequence.
The process of labeling spans is straightforward as follows:
• Use white-space to tokenize the text on the web into unigrams. For example, ‘This is a very long paragraph about HelloKitty’ is tokenized to [‘This’, ‘is’, ‘a’, ‘very’, ‘long’, ‘paragraph’, ‘about’, ‘HelloKitty’]. In this step, all punctuations are removed.
• Use white-space to tokenize the answer into unigrams. For example, ‘very long’ is tokenized to [‘very’, ‘long’].
• Search and match the answer unigrams in the text unigrams.
• Map the unigram span of the answer to character bytes span.
There are 3.87% examples in the Common Crawl dataset, whose answer text can not be matched by this procedure. We simply exclude these examples in our experiments. Moreover, we also found there are roughly 21.54% examples where the answer has multiple occurrences in the text.
B.2 Statistics The statistics of the datasets with training time are shown in Table 5.
B.3 Baseline Discussion We want to provide some clarification on the results of the two baselines, WebFormer and MarkupLM, in Table 1. First, for both methods, we directly run their codes to obtain the results. The code/model of MarkupLM is publicly available. For WebFormer, we obtain the original code and model from its authors. Second, our results are consistent with MarkupLM on WebSRC (last row in their Table 1). Here we use stronger baseline MarkupLM-large for comparison. Third, for CommonCrawl, we reprocess the data by removing non-matched groundtruth (as discussed above), resulting in slightly less data (in our Table 5) compared to the data used in WebFormer (in their Table 1). This is the main reason why the reported numbers of WebFormer in this work are even higher than the original results.
C Implementation Details
For data pre-processing, we use open-source LXML library4 to process each page for obtaining the DOM tree structures. For all these baselines, we use the same English uncased WordPiece vocabulary as in BERT. The word embedding is initialized with the pretrained BERT-base. The encoder parameters used in MUST are 12 layers, 768 hidden size, 3072 hidden units (for FFN). The maximum text sequence length is set to 2048. The decoder parameters used in MUST are 4 layers, 768 hidden size, 3072 hidden units, max output sequence length is 128. During training, we use the gradient
4https://lxml.de/
descent algorithm with Adam optimizer. The initial learning rate is set to 2e−5. The batch size for each update is set as 64 and the model is trained for up to 15 epochs. The dropout probability for the attention layer is set to 0.1. The model parameters are provided in Table 6.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:

heading: 3 A1. Did you describe the limitations of your work?
text: Left blank.
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.

heading: 3 A3. Do the abstract and introduction summarize the paper’s main claims?
text: Left blank.
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 7 Did you use or create scientific artifacts? Left blank.
7 B1. Did you cite the creators of artifacts you used? Left blank.
B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Not applicable. Left blank.
B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Not applicable. Left blank.
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Not applicable. Left blank.
C 3 Did you run computational experiments? Left blank.

heading: 3 C1. Did you report the number of parameters in the models used, the total computational budget
text: (e.g., GPU hours), and computing infrastructure used? Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.

heading: 3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found
text: hyperparameter values? Left blank.

heading: 3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary
text: statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Left blank.

heading: 3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did
text: you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Left blank.
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: There are several possibilities for our future exploration. For example, we plan to pretrain a MUST model by incorporating HTML-specific tasks, such as masking DOM nodes and predicting the relations between DOM nodes. Second, our model focuses on web pages with single-object, where each target field only has exactly one answer. For a multi-object page, e. g. a movie listing page, there are different movie names corresponding to different movies on the page. However, methods like repeated patterns (Adelfio and Samet, 2013) can be applied