SlowGuess/NAACL_2022_OCR huggingface dataset:
Zihan Zhao, Lu Chen*, Ruisheng Cao, Hongshen Xu, Xingyu Chen and Kai Yu*  
X-LANCE Lab, Department of Computer Science and Engineering
MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, China  
Shanghai Jiao Tong University, Shanghai, China
State Key Laboratory of Media Convergence Production Technology and Systems
zhao_mengxin@sjtu.edu.cn, chenlusz@sjtu.edu.cn
{211314, xuhongshen, galaxychen, kai.yu}@sjtu.edu.cn
Recently, the structural reading comprehension (SRC) task on web pages has attracted increasing research interests. Although previous SRC work has leveraged extra information such as HTML tags or XPaths, the informative topology of web pages is not effectively exploited. In this work, we propose a Topological Information Enhanced model (TIE), which transforms the token-level task into a tag-level task by introducing a two-stage process (i.e. node locating and answer refining). Based on that, TIE integrates Graph Attention Network (GAT) and Pre-trained Language Model (PLM) to leverage the topological information of both logical structures and spatial structures. Experimental results demonstrate that our model outperforms strong baselines and achieves state-of-the-art performances on the web-based SRC benchmark WebSRC at the time of writing. The code of TIE will be publicly available at https://github.com/X-LANCE/TIE.
With the rapid development of the Internet, web pages have become the most common and rich source of information (Dong et al., 2014). Therefore, the ability to understand the contents of structured web pages will guarantee a rich and diverse knowledge source for deep learning systems. Each web page is mainly rendered from the corresponding HyperText Markup Language (HTML) codes. In other words, the understanding of a structured web page can be achieved by the comprehension of its HTML codes.
One of the commonly used tasks to verify the model's ability of comprehension is Question Answering (QA). However, previous QA models only focus on the comprehension of plain texts (Rajpurkar et al., 2016; Yang et al., 2018; Reddy et al., 2019; Zeng et al., 2020), tables (Pasupat and Liang,
(a) An example of the structured web pages' HTML codes and screenshots
HTML codes and screenshots
(b) The illustration of DOM Tree (logical structure)
(c) The illustration of NPR Graph (spatial structure)
Figure 1: An example of web pages in WebSRC and its corresponding Document Object Model (DOM) tree and Node Positional Relation (NPR) graph in WebSRC. The colored HTML tag in (a) is corresponding to the bounding box with the same color in (a) and the node with the same color in (b) and (c).
2015; Chen et al., 2020c, 2021b), or knowledge bases (KBs) (Berant et al., 2013; Talmor and Berant, 2018). These sources have either no topological structure or fixed-form structures. On the contrary, the topological structures of web pages are complex and flexible, which are less investigated in previous QA works.
Specifically, HTML codes can be viewed as multiple semantic unit separated by tag tokens (e.g.  $<\mathrm{div}>$ ,  $</\mathrm{div}>$ ). An HTML tag refers to a pair of matched start and end tags and all the content in between, which also corresponds to a part of the web page (illustrated in Fig. 1 (a)). Therefore, there are two kinds of topological structures in web pages: logical structures which contain the hierarchical relations and clustering of tags (see Fig. 1 (b)); and spatial structures which contain the relative positions between different tags in the web pages (see Fig. 1 (c)). These topological structures are as important as the semantics of HTML codes and screenshots.
Although previous works (Chen et al., 2021c; Li
*The corresponding authors are Lu Chen and Kai Yu.
Human Language Technologies, pages 1808 - 1821
et al., 2021) have tried to leverage the topological structures by adopting HTML tags or XPaths as tokens or position embeddings, only logical structures are encoded implicitly. However, it is obvious for humans to identify key-value pairs if two spans are located in the same row or column, while this relation may take various forms in the logical structures of different web pages. Moreover, tables have extremely simple spatial structures but will be super complex in terms of logical structures. Therefore, spatial structures are essential and complementary to logical structures.
The major obstacle that prevents previous models to leverage spatial relations is that both the two kinds of topological structures are organized at the tag level instead of the token level (Fig. 1 (b) and (c)). As token-level models, whose computation and prediction units are the tokens of web pages, it is extremely hard and anti-natural for them to encode the topological structures. Moreover, using token-level models also means that previous works have to implicitly imply the logical structures to the models, which may be less effective than explicitly telling with the help of prior knowledge.
To tackle these problems, we propose Topological Information Enhanced model (TIE), a tag-level QA model that operates on the representations of HTML tags to predict which tag the answer span belongs to. By switching from token level to tag level, various structures of web pages can be explicitly encoded into the model easily. Specifically, TIE encodes both the logical and spatial structures using Graph Attention Network (GAT) (Velickovic et al., 2018) with the help of two kinds of graphs. The first kind of graphs is Document Object Model (DOM) trees which is widely used to represent the logical structures of HTML codes. Secondly, to encode the spatial structures, we define the Node Positional Relation (NPR) graph based on the bounding box of HTML tags obtained by the browser. Detail definition can be found in Section 3.2.2.
Moreover, to accomplish the token-level prediction tasks by a tag-level QA model, we further introduce a two-stage process including node locating stage and answer refining stage. Specifically, in the answer refining stage, a traditional token-level QA model is utilized to extract answer span with the constraint of the answer node prediction by TIE in the node locating stage.
Our TIE model is tested on the WebSRC
Figure 2: Illustration of the relations between DOM trees and HTML codes. The italic tokens "<li>" Front Wheel Drive </li>" are the direct content of node  $n_i$
dataset  ${}^{1}$  and achieve state-of-the-art (SOTA) performances.
- We propose a tag-level QA model called TIE with a two-stage inference process: node locating stage and answer refining stage.
- We utilize GAT to leverage the topological information of both the logical and spatial structures with the help of DOM trees and our newly defined NPR graphs.
- Experimental results on the WebSRC dataset demonstrate the effectiveness of our model and its key component.
The Web-based SRC task (Chen et al., 2021c) is defined as a typical extractive question answering task based on web pages. Given the user query  $\mathbf{q} = (q_{1}, q_{2}, \dots, q_{|\mathbf{q}|})$  and the flattened HTML code sequence  $\mathbf{c} = (c_{1}, c_{2}, \dots, c_{|c|})$  of relevant web page as inputs, the goal is to predict the starting and ending position of answer span  $(s, e)$  in the HTML codes  $\mathbf{c}$  where  $|\mathbf{q}|, |\mathbf{c}|$  denote the length of the question and the HTML code sequence, respectively, and  $1 \leq s \leq e \leq |\mathbf{c}|$ . Notice that each token  $c_{i}$  in the flattened HTML codes  $\mathbf{c}$  can be a raw text word or tag symbol such as <div> while the user query  $q$  is a word sequence of plain text.
2.2 DOM Trees of HTML codes
The DOM tree is a special tree structure that is parsed from raw HTML codes by Document Object Model  ${}^{2}$  . Each node in the tree denotes a tag closure in the original HTML code. Specifically, each node contains a start tag token (e.g. <div>), an end
1https://x-lance.github.io/WebSRC/.
Document_Object模型
Figure 3: The two-stage architecture using TIE and traditional QA model (e.g. MarkupLM)
tag token (e.g.  $\langle$  /div>), and all the contents in between. One DOM node  $n_j$  is the descendant of another node  $n_i$ , iff the contents of node  $n_j$  is entirely included in the contents of node  $n_i$ .
Furthermore, we define the direct contents of each DOM node (and its corresponding HTML tag) as all the tokens in its tag closure that are not contained in any of its children (see Figure 2).
3 TIE
In this section, we will first introduce the architecture of the whole SRC system in Sec.3.1, and then the two kind of graph we used in Sec. 3.2. Finally, the structure of Topological Information Enhance model (TIE) is demonstrated in Sec.3.3.
3.1 Architecture of the Whole SRC System
With the help of DOM trees and NPR graphs, TIE can efficiently predict in which node the answer is located. Therefore, we modify the original architecture of the SRC system into a two-stage architecture: node locating and answer refining. The two-stage architecture is illustrated in Figure 3.
In the node locating stage, we first define the answer node as the deepest node in the DOM tree which contains the complete answer span. Then, TIE is utilized to predict the answer node  $n_a$  for the question  $q$  given the flattened HTML codes  $c$  and the corresponding DOM tree  $\mathcal{D}_c$  and NPR graphs  $\mathcal{G}_c$  (see Sec. 3.2). Formally,
$$
\operatorname {T I E} (q, c, \left(\mathcal {D} _ {c}, \mathcal {G} _ {c}\right)) = \boldsymbol {p} ^ {n},
$$
$$
n_{a} = \operatorname *{argmax}_{n_{i}\in V_{D_{c}}}(p_{i}^{n}),
$$
where  $p_i^n$  denotes the probability of node  $n_i$  being the answer node, and  $V_{D_c}$  is the node set of  $D_c$ .
Then, in the answer refining stage, we use the predicted answer node as a constraint during the prediction of the answer span. In more detail, we first use a QA model (e.g. MarkupLM) to obtain the start and end probabilities  $\pmb{p}^s$ ,  $\pmb{p}^e$  among all the tokens of HTML code sequence  $c$ . Then, the predicted answer span is chosen from the spans which are contained by the predicted answer node  $n_a$ . To conclude, provided that the starting and ending position of predicted answer node  $n_a$  in the HTML code  $c$  is  $s_a$ , and  $e_a$ , the second stage can be formulated as follows:
$$
\operatorname {Q A} (q, c) = \boldsymbol {p} ^ {s}, \boldsymbol {p} ^ {e}
$$
$$
(s_{pred},e_{pred}) = \operatorname *{argmax}_{(i,j):s_{a}\leq i <   j\leq e_{a}}(p_{i}^{s} + p_{j}^{e})
$$
3.2 Construction of GAT Graphs
Recently, Graph Neural Network (GNN) (Scarselli et al., 2008) has been widely used in multiple Neural Language Processing tasks, such as text classification and generation (Yao et al., 2019; Zhao et al., 2020), information extraction (Lockard et al., 2020), dialogue policy optimization (Chen et al., 2018a,b, 2019, 2020d), dialogue state tracking (Chen et al., 2020a; Zhu et al., 2020), Chinese processing (Gui et al., 2019; Chen et al., 2020b; Lyu et al., 2021), etc. Graph Attention Network (GAT) is a special type of GNN that encodes graphs with attention mechanism. In this work, to leverage both the logical and spatial structures, we introduce two kinds of graphs: DOM Trees and NPR graphs.
3.2.1 DOM Trees
The logical relations of HTML codes can be described with the assistance of its DOM Tree (see Sec. 2.2). However, the original tree is extremely sparse, which often leads to poor communication efficiency among nodes. To this end, we modify the structure to enlarge the receptive fields for each node. Mathematically, the resulting graph  $\mathcal{D}_c = (V_{D_c},E_{D_c})$  can be constructed from the original sparse form  $\mathcal{D} = (V_D,E_D)$
$$
\left\{ \begin{array}{c} V _ {D} = \text {a l l n o d e s i n t h e o r i g i n a l D O M t r e e}, \\ E _ {D} = \{(n _ {i}, n _ {j}) | n _ {i} \text {i s t h e p a r e n t o f} n _ {j} \} \cup \\ \{(n _ {i}, n _ {j}) | n _ {i} \text {i s a c h i l d o f} n _ {j} \}, \end{array} \right.
$$
into a denser one  $\mathcal{D}_c = (V_{D_c},E_{D_c})$
$$
\left\{ \begin{array}{r l} V _ {D _ {c}} & = V _ {D} \\ E _ {D _ {c}} & = \{(n _ {i}, n _ {i}) | n _ {i} \in V _ {D _ {c}} \} \cup \\ & \{(n _ {i}, n _ {j}) | n _ {i} \text {i s a n a n c e s t o r o f} n _ {j} \} \cup \\ & \{(n _ {i}, n _ {j}) | n _ {i} \text {i s a d e s c e n d a n t o f} n _ {j} \} \end{array} \right.
$$
Figure 4: The overall architecture of TIE
In this way, each node can directly communicate with all of its ancestors and descendants, so that the information can be transferred much faster.
3.2.2 NPR Graphs
To explicitly establish the positional relations between different texts, we define and construct Node Positional Relation (NPR) graph  $\mathcal{G}_c = (V_G,E_G)$  based on the rendered structured web pages.
Similar to DOM Tree, each NPR node  $n_i$  corresponds to a tag  $t_i$  in the HTML code of the web page. The content of NPR nodes is defined as the direct content of their corresponding HTML tags. It is worth noticing that under our definition, the node sets of the NPR graph and the DOM tree of the same web page are identical ( $V_G = V_D$ ).
Moreover, considering that the nodes with informative relations (such as "key-value" relations and "header-cell" relations) are usually located on the same row or column, we introduce four kinds of directed edges into NPR graphs: UP, DOWN, LEFT, and RIGHT. Specifically,  $(n_i, n_j) \in E_G^{\mathrm{UP}}$  when
$$
\left\{ \begin{array}{c} \min  \left(x _ {n _ {i}} + w _ {n _ {i}}, x _ {n _ {j}} + w _ {n _ {j}}\right) - \max  \left(x _ {n _ {i}}, x _ {n _ {j}}\right) \\ \geq \gamma \times \min  \left(w _ {n _ {i}}, w _ {n _ {j}}\right) \\ y _ {n _ {i}} \geq y _ {n _ {j}} \text {o r} y _ {n _ {i}} + h _ {h _ {i}} \geq y _ {n _ {j}} + h _ {n _ {j}} \end{array} \right. \tag {1}
$$
both hold, where  $(x_{n_i},y_{n_i}),(x_{n_j},y_{n_j})$  are the coordinates of the upper-left corner of the bounding boxes corresponding to the nodes  $n_i$  and  $n_j$ ;  $w_{n_i},w_{n_j}$  are the width of the two bounding boxes
while  $h_{n_i}, h_{n_j}$  are the height of the two bounding boxes; and  $\gamma$  is a hyper-parameter. Similar functions are used for  $E_G^{\text{DOWN}}$ ,  $E_G^{\text{LEFT}}$ , and  $E_G^{\text{RIGHT}}$ . Finally,  $E_G = E_G^{\text{UP}} \cup E_G^{\text{DOWN}} \cup E_G^{\text{LEFT}} \cup E_G^{\text{RIGHT}}$ . Figure 1 (a) and (c) show an example of the NPR graph and its corresponding HTML code.
To simplify the NPR graphs, we only consider the nodes whose direct contents contain text tokens. That means in NPR graphs, the nodes whose direct contents only contain tag tokens will be isolated nodes with no relation.
3.3 Design of TIE
The model we proposed, TIE, mainly consists of four parts: the Context Encoder Module, the HTML-Based Mean Pooling, the Structure Encoder Module, and the Classification Layer. The overall architecture of TIE is shown in Figure 4.
Context Encoder Module. We first utilize Pre-trained Language Model as our context encoder. It encodes the contextual information of the HTML codes and gets the contextual word embeddings used for node representation initialization. Specifically, we use two PLM in our experiments: H-PLM (Chen et al., 2021c) + RoBERTa (Liu et al., 2019) and MarkupLM (Li et al., 2021).
HTML-Based Mean Pooling. In this module, TIE initializes the node representations based on the contextual word embedding calculated by Context Encoder. Specifically, for each node, we initialize its representation as the average embedding
Type	Training set	Dev set
#QA	%	#QA	%
KV	129990	42.3	21798	41.3
Comparison	52893	12.2	9078	17.2
Table	124432	40.5	21950	41.6
Table 1: The statistics of QA pairs from different types of websites in WebSRC.
of its corresponding tag's direct contents. Formally, the representation of node  $n_i$  is calculated as:
$$
\boldsymbol {n} _ {i} = \underset {x _ {j} \in \mathrm {D C} \left(n _ {i}\right)} {\text {m e a n}} \left(\boldsymbol {x} _ {j}\right) \tag {2}
$$
where  $\mathrm{DC}(n_i)$  means the tokens set of the direct contents of node  $n_i$ ;  $\pmb{x}_j$  is the contextual embedding of token  $x_j$ .
Structure Encoder Module. TIE utilizes GAT to encode the topological information contained in DOM trees and NPR graphs. Specifically, for the i-th attention head of GAT:
$$
\boldsymbol {Q} _ {i} = \boldsymbol {W} _ {q, i} \boldsymbol {N}; \boldsymbol {K} _ {i} = \boldsymbol {W} _ {k, i} \boldsymbol {N}; \boldsymbol {V} _ {i} = \boldsymbol {W} _ {v, i} \boldsymbol {N}
$$
$$
\operatorname {G A T} _ {i} (N) = \operatorname {s o f t m a x} \left(\frac {\boldsymbol {Q} _ {i} ^ {T} \boldsymbol {K} _ {i}}{\sqrt {d}} + M _ {i}\right) \boldsymbol {V} _ {i}
$$
$$
m _ {j k} ^ {(i)} = \left\{ \begin{array}{l l} 0 & (n _ {j}, n _ {k}) \in \operatorname {E d g e} (G _ {i}) \\ - \infty & o t h e r w i s e \end{array} \right.
$$
$$
G _ {i} \in \left\{\mathcal {D} _ {c}, \mathcal {G} _ {c} ^ {\text {U P}}, \mathcal {G} _ {c} ^ {\text {D O W N}}, \mathcal {G} _ {c} ^ {\text {L E F T}}, \mathcal {G} _ {c} ^ {\text {R I G H T}} \right\}
$$
where  $\mathbf{N} = [\mathbf{n}_i]_{d\times |\mathcal{N}|}$ ;  $d$  is the dimension of the node representations  $\mathbf{n}_i$ ;  $\mathbf{W}_i$ s are the learnable parameters;  $M_i = [m_{jk}^{(i)}]_{|\mathcal{N}|\times |\mathcal{N}|}$  is the mask matrix for the i-th attention head. Finally, the outputs of all the attention heads are concatenated to form the node representations for the next GAT layer.
Classification Layers. Finally, we get the embeddings of all the nodes from the Structure Encoder Module and utilize a single linear layer followed by a Softmax function to calculate each node's probability of being the answer node.
We evaluate our proposed methods on WebSRC (Chen et al., 2021c). In more detail, the WebSRC dataset consists of 0.4M question-answer pairs and 6.4K web page segments with complex structures. For each web page segment, apart from its corresponding HTML codes, the dataset also provides
the bounding box information of each HTML tag obtained from the rendered web page. Therefore, we can easily use this information to construct the NPR graph for each web page segment.
Moreover, WebSRC groups the websites into three classes: KV, Comparison, and Table. Specifically, KV indicates that the information in the websites is mainly presented in the form of "key:value", where key is an attribute name and value is the corresponding value. Comparison indicates that each web page segment of the websites contains several entities with the same set of attributes. Table indicates that the websites mainly use a table to present information. The statistics of different types of websites in WebSRC are shown in Table 1.
We submit our models to the official of WebSRC for testing.
To keep consistent with previous studies, we adopt the following three metrics: (1) Exact Match (EM), which measures whether the predicted answer span is exactly the same as the golden answer span. (2) Token level F1 score (F1), which measures the token level overlap of the predicted answer span and the golden answer span. (3) Path Overlap Score (POS), which measures the overlap of the path from the root tag (<HTML>) to the deepest tag that contains the complete predicted answer span and that contains the complete golden answer span. Formally, the POS is calculated as follows:
$$
P O S = \frac {\left| P _ {p r e d} \bigcap P _ {g t} \right|}{\left| P _ {p r e d} \bigcup P _ {g t} \right|} \times 100 \% \tag{3}
$$
where  $P_{pred}$  and  $P_{gt}$  are the set of tags that on the path from the root (<HTML>) tag to the deepest tag that contains the complete predicted answer span or the ground truth answer span, respectively.
4.3 Baselines & Setup
We leverage the three models introduced in Chen et al. (2021c) and MarkupLM (Li et al., 2021) as our baselines. Specifically, T-PLM converts the HTML codes into plain text by simply removing all the HTML tags, while H-PLM treats HTML tags as special tokens and uses the origin HTML code sequences as input. Then, both of them utilize PLMs to generate the predicted answer span. To leverage visual information, V-PLM concatenates token embeddings resulting from H-PLM with visual embeddings and then feeds the results into multiple
Method	Dev	Test
EM↑	F1↑	POS↑	EM↑	F1↑	POS↑
BASE	T-PLM(BERT) (Chen et al., 2021c)	52.12	61.57	79.74	39.28	49.49	67.68
H-PLM(BERT) (Chen et al., 2021c)	61.51	67.04	82.97	52.61	59.88	76.13
V-PLM(BERT) (Chen et al., 2021c)	62.07	66.66	83.64	52.84	60.80	76.39
MarkupLM (Li et al., 2021)	68.39	74.47	87.93	-	-	-
MarkupLM*	68.99	74.55	88.40	60.43	67.05	80.55
TIE MarkupLM	76.83	82.77	90.90	71.86	75.91	85.74
LARGE	T-PLM(Electra) (Chen et al., 2021c)	61.67	69.85	84.15	56.32	72.35	79.18
H-PLM(Electra) (Chen et al., 2021c)	70.12	74.14	86.33	66.29	72.71	83.17
V-PLM(Electra) (Chen et al., 2021c)	73.22	76.16	87.06	68.07	75.25	84.96
MarkupLM (Li et al., 2021)	74.43	80.54	90.15	-	-	-
H-PLM(RoBERTa)*	70.90	75.15	87.16	67.76	74.61	86.29
TIEH-PLM(RoBERTa)	75.57	79.38	88.29	69.65	74.78	85.72
MarkupLM*†	73.38	79.83	89.93	69.09	76.45	87.24
TIE MarkupLM	81.66	86.24	92.29	75.87	80.19	89.73
self-attention blocks before generating predictions. Faster R-CNN is utilized to extract visual embeddings from screenshots of the corresponding web pages. On the other hand, MarkupLM leverages XPaths to encode the logical position of each token and use it as an additional position embedding.
In our experiments, we use 3 GAT blocks as the Structure Encoder Module of TIE. H-PLM(RoBERTa) and MarkupLM are leveraged as context encoders. The implementation of TIE is based on the official code provided by WebSRC  $^{3}$  and MarkupLM  $^{4}$ . We set the hyperparameter  $\gamma$  in Eq.1 to be 0.5. Finally, the models used in the answer refining stage are of the same architecture as the context encoder models of TIE while individually trained on WebSRC. For more setup details, please refer to Appendix. A
The experimental results on the development set and the test set are shown in Table 2. Specifically, the performances of TIE in the following sections refer to the performances of the proposed two-stage system, and the subscript of TIE refers to both the context-encoder for TIE and the QA model used in answer refining stage.
Table 2: The results of our proposed method on WebSRC. EM denotes the exact match scores; F1 denotes the token level F1 scores; POS denotes the path overlap scores. We submit the models to the official of WebSRC for testing. * denotes reproduction results. † denotes average results of 3 random seeds.
|S0|	|S1|	|S0|: |S1|
MarkupLM	873	692	1.26:1
TIEmarkupLM	944	314	3.1:1
Table 3: The statistics of samples on Compare websites in the development set with wrong predictions.  $S_0$  is the set of examples with 0 F1 scores.  $S_1$  is the set of examples with F1 scores between 0 and 1. The numbers are average results of 3 random seeds.
From the results, we can find out that our TIE consistently achieves better results compared with the corresponding baselines. Specifically,  $\mathrm{TIEMarkupLM}$  significantly outperforms the previous SOTA results, MarkupLM, by  $6.78\%$  EM,  $3.74\%$  F1, and  $2.49\%$  POS on the test set. Moreover, it is worth noticing that the performance of  $\mathrm{TIEMarkupLM - BASE}$  is even higher than the performance of the MarkupLM-LARGE model (76.83% v.s.  $73.38\%$  EM on the development set and  $71.86\%$  v.s.  $69.09\%$  EM on the test set). These results strongly demonstrate that TIE can effectively model the topological information of the semi-structured web pages with the help of its structure encoder.
Furthermore, we compare the performances of  $\mathrm{TIEMarkupLM}$  and MarkupLM on different types of websites. The results are shown in Figure 5. From the figure, we find that our method achieves signifi
$^{3}$ https://github.com/X-LANCE/WebSRC-Baseline $^{4}$ https://github.com.microsoft/unilm/tree/master/markuplm
Figure 5: The performance comparison on different types of websites of the development set.