SlowGuess/ACL_2020_OCR huggingface dataset:
Colin Lockard
lockards@cs.washington.edu
Prashant Shiralkar
shiralp@amazon.com
Xin Luna Dong
lunadong@amazon.com
Hannaneh Hajishirzi
University of Washington, Allen Institute for AI
hannaneh@cs.washington.edu
In many documents, such as semi-structured webpages, textual semantics are augmented with additional information conveyed using visual elements including layout, font size, and color. Prior work on information extraction from semi-structured websites has required learning an extraction model specific to a given template via either manually labeled or distantly supervised data from that template. In this work, we propose a solution for "zero-shot" open-domain relation extraction from webpages with a previously unseen template, including from websites with little overlap with existing sources of knowledge for distant supervision and websites in entirely new subject verticals. Our model uses a graph neural network-based approach to build a rich representation of text fields on a webpage and the relationships between them, enabling generalization to new templates. Experiments show this approach provides a  $31\%$  F1 gain over a baseline for zero-shot extraction in a new subject vertical.
Semi-structured websites offer rich sources of high-quality data across many areas of knowledge (Dong et al., 2014). These websites present information via text that is accompanied by rich visual and layout features that can be generalized beyond a single website. However, most prior work on information extraction (IE) from websites has largely ignored most of these features, instead relying only on HTML features specific to an individual website (Ferrara et al., 2014). This requires training data for every website targeted for extraction, an approach that cannot scale up if training data must be manually created.
To circumvent manual data annotation, previous work used a distant supervision process requiring a knowledge base aligned to the website targeted
Figure 1: Our zero-shot open-domain information extraction process learns generalizable graph-based representations of how relations are visually presented on semi-structured websites, allowing for training on one vertical (such University sites) and extraction from another (such as Movie sites).
for extraction (Gentile et al., 2015; Lockard et al., 2018), including for OpenIE extraction (Banko et al., 2007; Bronzi et al., 2013; Lockard et al., 2019). These methods, however, can only learn a website-specific model based on seed knowledge for the site, but cannot be generalized to the majority of websites with knowledge from new verticals, by long-tail specialists, and in different languages.
In this paper, we introduce the task of zero-shot relation extraction from semi-structured websites, in which a learned model is applied to extract from a website that was not represented in its training data (Figure 1). Moreover, we introduce ZEROSHOTCERES, a graph neural network model that encodes semantic textual and visual patterns common across different training websites and can generalize to extract information from documents with never-before-seen templates and topics.
8105
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8105-8117
Unlike unstructured text, which can be modeled as a sequence, or images, which can be modeled as a two-dimensional grid of pixels, it is not obvious how to operate over the many shapes and sizes of text fields on a semi-structured webpage. We illustrate our intuition using the webpage snippets in Figure 1: Despite their differences, each site uses alignment of relation and object strings, either vertically or horizontally, to help indicate relationships; in addition, relation strings are often more prominent than their objects, either in size or boldness. Such features are semantically meaningful to readers and often consistent from site to site; thus, encoding them into the representation of webpages will allow us to generalize to unseen sites.
Our model, ZEROSHOTCERES, encodes these diverse feature types in a graph representation in which each text field becomes a node in a graph, connected by edges indicating layout relationships on the page. This abstracts away the details of the page while maintaining the core visual structure presented to the reader. A graph neural network is then applied to produce a new representation of each text field, informed by the surrounding page context. This representation is then used to extract entities and relationships from the document. This allows us to extract not only in the closed-domain setting, but also allows us to conduct OpenIE on websites about entirely new subject verticals not seen during training.
Our contributions are threefold: (a) We introduce a graph neural network model for webpage representation that integrates multi-modal information including visual, layout, and textual features, enabling generalization for IE from never-beforeseen websites. (b) We propose the first approach to enable Open Information Extraction from semistructured websites without prior knowledge or training data in the subject vertical. (c) Our method works in both OpenIE and ClosedIE settings. We conduct evaluations showing the effectiveness of the technique and exploring the challenges of zero-shot semi-structured IE, achieving a  $31\%$  improvement in F1 compared to an OpenIE baseline. The graph model gives a  $26\%$  F1 boost when extracting according to a defined schema (ClosedIE).
DOM-based ClosedIE: The conventional approach to extraction from semi-structured websites is wrapper induction (Kushmerick et al., 1997), in
which training data for documents from a given template is used to learn a rule-based extractor based on DOM (i.e., HTML) features to apply to other documents of the same template, extracting relations according to a pre-defined ontology ("ClosedIE"). Since this approach requires training data for each template targeted for extraction, recent work has focused on reducing the manual work needed per site. Fonduer (Wu et al., 2018) provides an interface for easily creating training data, Vertex (Gulhane et al., 2011) uses semi-supervision to minimize the number of labels needed, LODIE (Gentile et al., 2015) and Ceres (Lockard et al., 2018) automatically generate training data based on distant supervision, and DIADEM (Furche et al., 2014) identifies matching rules for specific entity types.
DOM-based OpenIE: WEIR (Bronzi et al., 2013) and OpenCeres (Lockard et al., 2019) offer OpenIE approaches to DOM extraction. The latter method uses visual features in a semi-supervised learning setting to identify candidate pairs that are visually similar to known (relation, object) pairs; however, the ultimate extraction model learned is still site-specific and based on DOM features rather than the more generalizable visual or textual features. Pasupat and Liang (2014) present a zero-shot method for extraction from semi-structured webpages, but limit their work to extraction of entities rather than relationships and do not consider visual elements of the page.
Multi-modal extraction: The incorporation of visual information into IE was proposed by Aumann et al. (2006), who attempted to learn a fitness function to calculate the visual similarity of a document to one in its training set to extract elements like headlines and authors. Other recent approaches that attempt to address the layout structure of documents are CharGrid (Katti et al., 2018), which represents a document as a two-dimensional grid of characters, RiSER, an extraction technique targeted at templated emails (Kocayusufoglu et al., 2019), and that by Liu et al. (2018), which presents an RNN method for learning DOM-tree rules. However, none of these address the OpenIE setting, which requires understanding the relationship between different text fields on the page.
The approaches most similar to ours are GraphIE (Qian et al., 2019) and the approach by Liu et al. (2019). Both approaches involve constructing a graph of text fields with edges representing
8106
Figure 2: A depiction of the web page representation module (left) and relation classifiers (right).
horizontal and vertical adjacency, followed by an application of a GCN. However, neither approach makes use of visual features beyond text field adjacency nor DOM features, and both only consider extraction from a single text field rather than OpenIE. In addition, they show only very limited results on the ability of their model to generalize beyond the templates present in the training set.
3 Problem and Approach Overview
3.1 Zero-shot relation extraction from semi-structured websites
We address the problem of extracting entities and the relationships between them as expressed by never-before-seen semi-structured websites. A semi-structured website typically belongs to a subject vertical  $V$ , where  $V$  is a general field of knowledge such as movies, finance, or sports. A semistructured website consists of a set of detail pages sharing a similar template, each of which contains a set of facts about a page topic entity  $e_{topic}$ . The HTML document  $w$  defines a set of text fields  $T$  which the web browser renders as a webpage according to the instructions defined in the HTML and any referenced auxiliary files such as CSS or Javascript. The text fields have both textual and visual features, described in Section 4.2.1.
3.1.1 Relation Extraction
Our goal is to extract (subject, relation, object) knowledge triples, where the subject is  $e_{\text{topic}}$ , the object is a text field  $t \in T$  containing the name of an entity (or atomic attribute value), and the relation indicates the relationship between the two entities.
For this work, we assume the page topic entity has already been identified, (such as by the method
proposed by Lockard et al. (2018) or by using the HTML title tag) and thus limit ourselves to identifying the objects and corresponding relations. We consider the following two settings:
Relation Extraction (ClosedIE): Let  $R$  define a closed set of relation types, including a special type indicating "No Relation". Relation Extraction is the assignment of each text field  $t$  to one  $r_i \in R$ , which indicates the relationship between the entity  $e_{\text{object}}$  mentioned in  $t$  and  $e_{\text{topic}}$ .
Open Relation Extraction (OpenIE): Given a pair of text fields  $(i,j)$ , Open Relation Extraction is a binary prediction of whether  $i$  is a relation string indicating a relationship between the entity  $e_{object}$  mentioned in  $j$  and  $e_{topic}$ .
3.1.2 Zero-shot Extraction
Unlike prior work that requires the learning of a model specific to the semi-structured website targeted for extraction, we look at zero-shot extraction. Given a semi-structured website  $W$  targeted for extraction, zero-shot extraction is the learning of a model without any use of pages from  $W$  during training. We consider two zero-shot settings:
Unseen-Website Zero-shot Extraction is the learning of a model without any use of pages from  $W$ , but with pages from some other website(s) from vertical  $V$  during training.
Unseen-Vertical Zero-shot Extraction is the learning of a model without any use of pages from  $W$  or of pages from any website with vertical  $V$  during training.
3.2 Approach Overview
Figure 2 depicts our approach for zero-shot relation extraction (detailed in Section 5) leveraging a web page representation that will capture the
8107
similarities in visual and textual semantics across websites (Section 4). Our web page representation module first converts each page into a layout graph (Section 4.1) that abstracts away the details of the page structure while maintaining the adjacency relationships between text fields. We represent each text field with an initial feature vector of visual and textual attributes. This input is passed into a graph neural network that allows for information to flow between nodes, producing a new text field representation that captures contextual information (Section 4.2).
To obtain a web page encoding, we leverage a pre-training step with auxiliary loss function  $\mathcal{L}_{pre}$  that encourages the model to produce an intermediary representation useful for IE. This is performed via a three-way classification that determines if a text field contains a relation name, the object of some relation, or irrelevant text (Section 4.3). After pre-training, the weights of this GNN are frozen and it can be applied to new pages, with its output used as input into a relation extraction module, optimized with task-specific loss function  $\mathcal{L}_{task}$ , where the task is either OpenIE or ClosedIE, described in Section 5. The resulting approach minimizes our overall loss  $\mathcal{L}_{ZSCERES}$ , with:
$$
\mathcal {L} _ {\text {Z S C E R E S}} = \mathcal {L} _ {p r e} + \mathcal {L} _ {t a s k} \tag {1}
$$
4 Web Page Encoder
The key idea behind our solution is to train webpage representations to capture the fundamental similarities in visual and textual semantics across websites to express relations, objects, and their relationships. The fundamental characteristics we capture, generalizable across templates and verticals, thus allow us to carry over our knowledge across websites and enable zero-shot extraction.
There are two key parts in our solution. First, we build a graph to capture the layout relationships in a more abstract form that allows us to more easily learn the common features across different sites such as the fact that relation strings are often to the left or above their objects (Section 4.1). Second, we apply a Graph Neural Network (GNN) to learn representations for each node capturing contextual information about its neighborhood on the webpage (Section 4.2), allowing information to flow through the nodes, providing context (e.g., flowing through "Cast" to a far-away node "Uma Thurman" via the closer node "Ethan Hawke" in Figure 3). This
Figure 3: A cropped portion of the detail page from allmovie.com for the film Tape. Arrows overlaid showing the constructed page graph consisting of edges for each horizontal (purple), vertical (yellow) and DOM (green) relationship between text fields.
representation will be useful for relation extraction as described in Section 5.
4.1 Page graph construction
We encode the layout relationships between text fields in the form of a graph,  $G$ , consisting of a set of nodes  $N$ , each corresponding to a text field, and a set of edges  $E$  corresponding to relationships between the text fields. The edges capture three forms of adjacency, as shown in the example in Figure 3:
Horizontal: Edges are added when two text fields are horizontal neighbors on the page; that is, they have a shared vertical location and there are no other text fields between them.
Vertical: Edges are added when two text fields are vertical neighbors on the page; that is, they have an overlapping horizontal location and there are no other text fields between them.
DOM: Edges are added when two text fields are siblings or cousins in the DOM tree; that is, the absolute XPaths identifying their locations differ only at a single index value.
4.2 Graph Neural Network (GNN)
To build a representation of each text field that incorporates the surrounding page context, we use Graph Attention Networks (GAT) (Veličković et al., 2018). The feature vector for each text field (described below) and the page graph form the input to a GAT, which then produces a new representation for each text field based on the surrounding context
8108
in the graph. Specifically, for each text field  $i$ , GAT layer  $l$  computes a representation  $h_i^l$  as follows:
$$
h _ {i} ^ {l} = \sigma \left(\sum_ {j \in N _ {i}} \alpha_ {i j} W _ {G} ^ {l} h _ {j} ^ {l - 1}\right), \tag {2}
$$
where  $N_{i}$  is the set of neighbors of node  $i$  in the graph, and  $h_j^{l - 1}$  is the representation of node  $j$  from the preceding layer;  $h_j^0$  indicates the input features for the node. (For each node, we add a self loop to the graph; that is, including  $i$  in  $N_{i}$ .)  $W_{G}^{l}$  is a learned weight matrix applied to the node features for layer  $l - 1$  and  $\sigma$  is a non-linear function, in our case a ReLU. The attention weight  $\alpha_{ij}$  determines how influenced a node's representation is by each of its neighbors, calculated as follows:
$$
\alpha_ {i j} = \frac {\exp \left(\sigma \left(a ^ {\top} \left[ W _ {G} ^ {l} h _ {i} ^ {l - 1} ; W _ {G} ^ {l} h _ {j} ^ {l - 1} \right]\right)\right)}{\sum_ {k \in N _ {i}} \exp \left(\sigma \left(a ^ {\top} \left[ W _ {G} ^ {l} h _ {i} ^ {l - 1}; W _ {g} ^ {l} h _ {k} ^ {l - 1} \right]\right)\right)}, \tag {3}
$$
where  $a$  is a weight vector applied against the concatenation (represented by “;”) of the two node's features as transformed by  $W_{G}^{l}$  and  $\sigma$  is a ReLU. This produces a new contextualized set of features for each node that are informed by the surrounding page context. We describe the original input features for each text field in the next section.
4.2.1 Initial text field features
For each text field on the page, we produce an initial feature vector containing both visual feature vector  $V$  and textual feature vector  $T$ . We define the input feature vector  $h_i^0$  for text field  $i$  as:
$$
h _ {i} ^ {0} = [ T (i); V (i) ] \tag {4}
$$
where  $"\;"$  represents concatenation.
Visual Features: A numeric feature vector is constructed representing the bounding box coordinates of the text field, the height and width of the bounding box, and the font size, along with one-hot features representing the typeface, font weight, font style, color, and text alignment.
Textual Features: In ClosedIE, to capture its semantics, the textual content of the text field is processed with a pre-trained BERT (Devlin et al., 2018) model. To produce a representation of the entire text field, we simply average the BERT-Base output for each token in the text field. For OpenIE,
since the goal is to generalize to entirely new subject verticals that may contain text not seen during training, only a single textual feature is used<sup>1</sup>: the percent of pages on the site on which the string in the text field appears. This frequency measure helps differentiate relation strings, which are likely to be common, from object strings, which are more likely to be rare.
4.3 Pre-Training Web Page Encoder
To encourage the GNN weights to capture the features necessary to represent relationships on the page, we use a pre-training step to learn the GNN representation before incorporating it into the extraction model. The pre-training task is a simplified form of the OpenIE task. To speed up training by avoiding the pairwise decisions necessary for OpenIE, we instead perform a multi-class classification of each text field into a class  $c$  in the set {Relation, Object, Other}:
$$
p \left(c \mid h _ {i} ^ {l}; \theta\right) = \operatorname {s o f t m a x} \left(W _ {\text {p r e}} h _ {i} ^ {l}\right) \tag {5}
$$
where  $h_i^l$  is the output of the GNN for the text field,  $W_{pre}$  is a weight matrix, and  $\theta$  comprises  $W_G$  and  $W_{pre}$ . Given a training set with  $T$  text fields, each with a ground truth class  $y_i^{pre}$ , we minimize the cross-entropy loss  $\mathcal{L}_{pre}$ :
$$
\mathcal {L} _ {p r e} = - \sum_ {i = 1} ^ {T} \log p \left(y _ {i} ^ {p r e} \mid h _ {i} ^ {l}, \theta\right) \tag {6}
$$
To discourage overfitting to spurious details in the small number of websites in our training set, we freeze the GNN weights after pre-training and do not update them during the full OpenIE training. After pre-training we discard the linear layer  $W_{pre}$  since it is not needed for subsequent steps; instead, we directly use the GNN output  $h^l$ .
5 Relation Extraction Model
Once we have the new representation  $h_t^l$  of each text field  $t$  produced by the above GNN process, we can perform our final classification.
5.1 OpenIE
For OpenIE, the classification decision must be made over a pair of text fields,  $i$  and  $j$ , the first containing the candidate relation string and the second containing the candidate object string. To
This feature is also used during ClosedIE
8109
avoid examining all possible pairs of fields, we first apply the candidate pair identification algorithm from Lockard et al. (2019), which filters down to a set of potential pairs based on physical and layout distance between text fields.
For each candidate pair, we concatenate the GNN-produced contextual features  $h^l$  for both text fields with the original features  $h^0$  for both text fields (since some information can be diluted in the GNN), as well as a pairwise feature vector that simply contains the horizontal and vertical distance between the two text fields, and pass them into a binary classifier:
$$
r _ {i} ^ {O I E} = \operatorname {F N N} \left(\left[ h _ {i} ^ {0}; h _ {j} ^ {0}; h _ {i} ^ {l}; h _ {j} ^ {l}; \text {p a i r w i s e} _ {i, j} \right], \theta^ {O I E}\right) \tag {7}
$$
where FNN is a feed-forward neural network with parameters  $\theta^{OIE}$ , “;” indicates concatenation, and  $r_i^{OIE}$  is the predicted probability that the two text fields constitute a (relation, object) pair. We then optimize for cross-entropy loss across training examples  $T$  with  $y_i^{OIE} = 1$  if the pair is positive:
$$
\begin{array}{l} \mathcal {L} _ {O I E} = \sum_ {i = 1} ^ {T} y _ {i} ^ {O I E} \log r _ {i} ^ {O I E} \\ + \left(1 - y _ {i} ^ {O I E}\right) \log \left(1 - r _ {i} ^ {O I E}\right), \tag {8} \\ \end{array}
$$
5.2 ClosedIE
For ClosedIE, we perform a multi-class classification using the contextual representation produced by the GNN  $(h_i^l)$  along with the original features  $(h_i^0)$  for text field  $i$ :
$$
r _ {i} ^ {C I E} = \operatorname {F N N} \left(\left[ h _ {i} ^ {0}; h _ {i} ^ {l} \right], \theta^ {C I E}\right) \tag {9}
$$
where FNN is a feed-forward neural network parameterized by  $\theta^{CIE}$ , "," indicates concatenation, and  $r_i^{CIE}$  is the predicted probability of relation  $r$  in set  $R$ . We optimize for cross entropy loss  $\mathcal{L}_{CIE}$ :
$$
\mathcal {L} _ {C I E} = - \sum_ {i = 1} ^ {T} \log p \left(y _ {i} ^ {C I E} \mid h _ {i} ^ {0}, h _ {i} ^ {l}, \theta^ {C I E}\right) \tag {10}
$$
where  $y_{i}^{CIE}$  is the true class for example  $i$ . For both ClosedIE and OpenIE we use one hidden layer in the feed-forward network.
6.1 Dataset
For both OpenIE and ClosedIE, our primary dataset is the extended version (Lockard et al., 2019) of the
SWDE dataset (Hao et al., 2011), which contains gold labels for OpenIE extractions for 21 English-language websites (each with one template) in three subject verticals (Movie, NBA, and University), with between 400 and 2,000 pages per site. We generated ClosedIE labels by converting the OpenIE labels to ClosedIE labels via manual alignment of OpenIE relations between websites, giving a set of 18 relations for the Movie vertical, 14 for NBA, and 13 for University. More information on training data creation and a complete listing of ClosedIE relations is available in the Appendix.
We used three SWDE Movie sites (AMCTV, AllMovie, and IMDb) as a development set and did not evaluate on them for the reported results.
6.2 Experimental Settings
For each model tested (both our own and the baselines), we classify the training setting into the following categories indicating the level of vertical or site-specific knowledge used, in decreasing level of difficulty.