SlowGuess/ACL_2020_OCR huggingface dataset:
- Level I-Unseen-Vertical Zero-shot (OpenIE only): A model is trained on sites from two of the three verticals (e.g. NBA and University) and applied to sites from the other vertical (Movie). This is the hardest case and is important when we wish to extract knowledge from new verticals where we do not have any prior knowledge or annotations.
- Level II-Zero-shot with Vertical Knowledge: A model is trained on all sites but one (spanning Movie, NBA, and University) and then applied to the held-out site. As in cross-validation, experiments are repeated with each site having a turn being held out. It is easier than Level I but is still important for a new website that may not have data overlapping with other websites in the same vertical. For the ClosedIE setting, we train only on in-vertical sites.
- Level III-Site-specific Knowledge: This is the traditional setting used by two of our baselines where we have seed knowledge overlapping with the website data to allow training a specific model for the website. Whereas Level I-II are both zero-shot settings, Level III is not, as it allows site-specific training data via weak supervision. (We do not present results using full supervision from manual annotations since it is known from prior work (e.g., Gulhane et al. (2011)) that full
supervision from the target website yields highly accurate semi-structured extractors; we note that ZSCERES also achieves comparable results ( $\sim$  0.95 F1) in this setting.
We repeated our experiments 10 times and we report the results averaged across the runs. For OpenIE, we follow the "lenient" scoring method for SWDE introduced by Lockard et al. (2019), scoring an extraction as correct if the relation string matches any of acceptable surface forms listed by the ground truth for that object.
Models are constructed in PyTorch (Paszke et al., 2017), with graph functions implemented in DGL (Wang et al., 2019) and optimization performed using Adam (Kingma and Ba, 2014) and a batch size of 20. For OpenIE, we use a hidden layer size of 25 for the GAT and 100 for the feed-forward layer. For ClosedIE, we use a hidden layer size of 200 for all layers. We use a 2-layer GAT and dropout of 0.25. We obtain visual features by rendering the page using the headless Chrome browser and querying the values using Selenium<sup>2</sup>.
Extraction Threshold: Since our zero-shot setting means we cannot use a development set of pages from the target site to tune the decision threshold, we instead set the threshold for each experiment to the value that attains the optimal F1 on the experiments where other sites were held-out.
OpenIE Postprocessing Rules: To ensure consistency among the extracted values, we keep only the highest confidence extraction in the case that the same text field is extracted as both a relation and object, or if multiple relations are extracted for the same object. In addition, some pages in the dataset contain relational tables, from which we sometimes extract the column headers as relations with the column contents as objects. While we believe a post-processing step could potentially recover these relational contents from our extractions, the SWDE data does not contain ground truth for such facts. Instead, we apply the heuristics described by (Cafarella et al., 2008) to identify these tables and remove them from our extractions.
6.3 Baselines and Models
We compare against several baselines:
Colon Baseline (OpenIE) This is a heuristic technique that identifies all text fields ending in a colon
(":") and assumes they are relation strings, then extracts the text field to the right or below, whichever is closer, as the object. We consider it as Level I knowledge since it requires no training.
WEIR (OpenIE) This approach by Bronzi et al. (2013) discovers relations by aligning multiple pages about the same entity. Because it requires sites to be grouped by vertical and uses a gazetteer list of entity names for the alignment, it has Level III knowledge.
OpenCeres (OpenIE) This applies the model by Lockard et al. (2019), which requires a knowledge base matching some facts presented on the target website, using Level III knowledge.
ZSCERES-FFNN (Feed-forward neural network): This model takes the same features and training data as the full ZSCERES model but removes the GNN component, with versions tested with both Level I (ZSCERES-FFNN Unseen-Vertical) and Level II (ZSCERES-FFNN Unseen-Website) knowledge.
ZSCERES-GNN: This applies the full model described in Section 4.2, with versions tested with both Level I (ZSCERES-GNN Unseen-Vertical) and Level II (ZSCERES-GNN Unseen-Website) knowledge.
7.1 OpenIE
Level-I Knowledge: Table 1 shows that ZSCERES is able to extract facts in entirely new subject verticals  $31\%$  more accurately than the colon baseline. Across all SWDE sites (micro-averaging across all extractions), ZSCERES-GNN achieves an F1 of 0.45, in comparison with 0.43 for ZSCERES-FFNN, showing that the additional information provided by the page encoder allows for a better representation of the relationships between text fields.
By successfully learning general patterns of relational presentation on webpages, ZSCERES-GNN is able to train solely on a set of 16 websites about Movies and NBA players, and then extract from University websites more accurately than the WEIR and OpenCeres systems, which take advantage of Level III knowledge to learn models specific to those University sites. While OpenCeres's rich vertical knowledge allows it to attain better results in Movie and NBA, ZSCERES-GNN still posts much stronger results than the other baselines in these two verticals.
$^{2}$ https://www.seleniumhq.org
8111
System	Site-specific Model	Level	Movie	NBA	University	Average
P	R	F1	P	R	F1	P	R	F1	F1
OpenCeres	Yes	III	0.71	0.84	0.77	0.74	0.48	0.58	0.65	0.29	0.40	0.58
WEIR	Yes	III	0.14	0.10	0.12	0.08	0.17	0.11	0.13	0.18	0.15	0.13
ZSCERES-FFNN Unseen-Website	No	II	0.37	0.5	0.45	0.35	0.49	0.41	0.47	0.59	0.52	0.46
ZSCERES-GNN Unseen-Website	No	II	0.49	0.51	0.50	0.47	0.39	0.42	0.50	0.49	0.50	0.47
Colon Baseline	No	I	0.47	0.19	0.27	0.51	0.33	0.40	0.46	0.31	0.37	0.35
ZSCERES-FFNN Unseen-Vertical	No	I	0.42	0.38	0.40	0.44	0.46	0.45	0.50	0.45	0.48	0.44
ZSCERES-GNN Unseen-Vertical	No	I	0.43	0.42	0.42	0.48	0.49	0.48	0.49	0.45	0.47	0.46
Figure 4: For OpenIE, using the full SWDE set (except the test site), including in-vertical training data (i.e. Level II knowledge), allows for 5-10 point gains in precision at equivalent recall compared to using only out-of-vertical training data (Level I).
Table 1: With no vertical knowledge, ZSCERES-GNN achieves  $65\%$  higher recall and comparable precision in all verticals compared to the colon baseline. Even in comparison to approaches that use vertical knowledge to learn site-specific OpenIE models, ZSCERES achieves an F1 seven points higher in the University vertical.
System	Knowledge Level	P	R	F1
ZSCERES-FFNN	II	0.45	0.49	0.46
ZSCERES-GNN	II	0.62	0.55	0.58
Level-II Knowledge: Figure 4 shows that adding the in-vertical sites to the training set (but still withholding the test site) allows the model to achieve performance better than the Level I training set that uses only out-of-vertical data.
7.2 ClosedIE
Table 2 shows the results for ClosedIE extraction. ZSCERES-GNN attains an overall F1 of 0.58 averaged across the three verticals. This significantly outperforms the feed-forward model that did not use the GNN, which attained an F1 of 0.46. While our performance on this dataset is far below
Figure 5: Performance on the ClosedIE Movie vertical increases significantly as more sites are added to the training data.
Table 2: For ClosedIE, using the pre-trained GNN adds 12 F1 points in comparison to the baseline lacking contextual information.
OpenIE F1	ClosedIE F1
Full Model	0.71	0.73
No GNN	0.68 (0.03 ↓)	0.63 (0.10 ↓)
No pre-training	0.66 (0.05 ↓)	0.73
No DOM edges	0.65 (0.06 ↓)	0.58 (0.15 ↓)
No spatial edges	0.65 (0.06 ↓)	0.62 (0.11 ↓)
No visual features	0.55 (0.16 ↓)	0.73
No BERT features	-	0.10 (0.63 ↓)
Add BERT features	0.68 (0.03 ↓)	-
Table 3: Ablations on the Movie development set.
the state-of-the-art for semi-structured ClosedIE (above 0.9 for all verticals), prior systems all learn site-specific models based on manual labeling or prior knowledge aligned to the website, while we have only Level II Knowledge available.
Figure 5 shows how adding additional training data improves performance in the Movie vertical. It appears that adding additional training sites would further improve the performance.
8112
7.3 Ablation Study
Table 3 shows the contributions of different elements of the model in the OpenIE and ClosedIE settings as calculated on the development set of three sites in the Movie vertical. These ablations show that the GNN helps in both settings, with a larger effect in ClosedIE, which is likely due to sharing the rich information about the text of nearby text fields.
Pre-training is important in OpenIE but does not have a significant effect for ClosedIE. This is not surprising given that the pre-training task is closely related to the OpenIE task. Both DOM and spatial adjacency edges contribute to the success of the page layout graph for the GNN. In the ClosedIE setting, the text and layout relationships alone will generally contain sufficient information to make an extraction, while in OpenIE the visual elements (such as whether text is bold or underlined) are a strong source of consistency across websites.
7.4 Error Analysis
OpenIE: To understand what cases our ZSCERES-GNN model is missing, we sampled 100 error cases in each vertical from the Unseen-Vertical experiment and manually examined them. Some examples of both erroneous and correct extractions are shown in Table 4 in the Appendix. False positives were largely due to the presence of two different types of n-ary relationships on the page.
The first class of errors involving n-ary relationships, making up  $43\%$  of all false positives, were where several facts have a multi-way relationship with the page topic, but individually the fields are not meaningful. For example, the NBA site USAToday includes a "Latest notes" section with links to several articles relevant to the page topic entity, mentioning the date, headline, and summary. We extract all of these objects with the "Latest notes" relation, but to obtain meaningful knowledge it would be necessary to additionally associate the correct date, headline, and summary with each other. While we can envision methods for doing this via post-processing, the SWDE benchmark considers these to be errors.
In the second class, ZSCERES correctly extracted (relation, object) pairs, but from page sections that contain facts about entities other than the page topic. For example, on the MatchCollege site, a section of "Similar Local Colleges" contains some of the same relations presented for the page
topic, in similar formatting. These types of errors made up another  $6\%$  of false positives.
Of the remaining errors,  $33\%$  were due to the extraction of pairs where the extracted relation did not represent a relationship, while another  $14\%$  were due to the extraction of pairs with a correct relation string and incorrect object. Most false negatives occurred in long vertical lists, where some values were extracted, but not all.
ClosedIE: False negatives were most likely to occur on long lists of values (such as cast lists), where values toward the bottom of the list were sometimes missed. Recall also suffered on relations where the relation name varied significantly from site to site, or where ambiguity existed. For example, the string "Produced by" is used by some sites to indicate the producer of the film, while on other sites it indicates the production company.
We have introduced a zero-shot method for learning a model for relation extraction from semistructured documents that generalizes beyond a single document template. Moreover, this approach enables OpenIE extraction from entirely new subject verticals where no prior knowledge is available. By representing a webpage as a graph defined by layout relationship between text fields, with text fields associated with both visual and textual features, we attain a  $31\%$  improvement over the baseline for new-vertical OpenIE extraction. Future extensions of this work involve a more general pre-training objective allowing for the learned representations to be useful in many tasks as well as distantly or semi-supervised approaches to benefit from more data.
We would like to acknowledge grants from ONR N00014- 18-1-2826, DARPA N66001-19-2-403, NSF (IIS1616112, IIS1252835), Allen Distinguished Investigator Award, and Sloan Fellowship.
Yonatan Aumann, Ronen Feldman, Yair Liberzon, Binyamin Rosenfeld, and Jonathan Schler. 2006. Visual information extraction. Knowledge and Information Systems, 10:1-15.
Michele Banko, Michael J. Cafarella, Stephen Soderland, Matthew G Broadhead, and Oren Etzioni.
8113
2007. Open information extraction from the web. In *IJCAI*.
Mirko Bronzi, Valter Crescenzi, Paolo Merialdo, and Paolo Papotti. 2013. Extraction and integration of partially overlapping web sources. PVLDB, 6:805-816.
Michael J. Cafarella, Alon Y. Halevy, Yang Zhang, Daisy Zhe Wang, and Eugene Wu. 2008. Uncovering the relational web. In WebDB.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: Pre-training of deep bidirectional transformers for language understanding. In NAACL-HLT.
Xin Luna Dong, Evgeniy Gabrilovich, Geremy Heitz, Wilko Horn, Ni Lao, Kevin Murphy, Thomas Strohmann, Shaohua Sun, and Wei Zhang. 2014. Knowledge vault: a web-scale approach to probabilistic knowledge fusion. In KDD.
Emilio Ferrara, Pasquale De Meo, Giacomo Fiumara, and Robert Baumgartner. 2014. Web data extraction, applications and techniques: A survey. Knowledge-Based Systems, 70:301-323.
Tim Furche, Georg Gottlob, Giovanni Grasso, Xiaonan Guo, Giorgio Orsi, Christian Schallhart, and Cheng Wang. 2014. Diadem: Thousands of websites to a single database. PVLDB, 7:1845-1856.
Anna Lisa Gentile, Ziqi Zhang, and Fabio Ciravegna. 2015. Early steps towards web scale information extraction with lodie. AI Magazine, 36:55-64.
Pankaj Gulhane, Amit Madaan, Rupesh R. Mehta, Jeyashankher Ramamirtham, Rajeev Rastogi, Sandeepkumar Satpal, Srinivasan H. Sengamedu, Ashwin Tengli, and Charu Tiwari. 2011. Web-scale information extraction with vertex. ICDE, pages 1209-1220.
Qiang Hao, Rui Cai, Yanwei Pang, and Lei Zhang. 2011. From one tree to a forest: a unified solution for structured web data extraction. In SIGIR.
Anoop R. Katti, Christian Reisswig, Cordula Guder, Sebastian Brarda, Steffen Bickel, Johannes Höhne, and Jean Baptiste Faddoul. 2018. Chargrid: Towards understanding 2d documents. In EMNLP.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. *ICLR*.
Furkan Kocayusufoglu, Ying Sheng, Nguyen Vo, James Bradley Wendt, Qi Zhao, Sandeep Tata, and Marc Najork. 2019. *Riser: Learning better representations for richly structured emails*. In WWW.
Nicholas Kushmerick, Daniel S. Weld, and Robert B. Doorenbos. 1997. Wrapper induction for information extraction. In *IJCAI*.
Shengpeng Liu, Ying Li, and Binbin Fan. 2018. Hierarchical RNN for few-shot information extraction learning. In ICPCSEE.
Xiaojing Liu, Feiyu Gao, Qiong Zhang, and Huasha Zhao. 2019. Graph convolution for multimodal information extraction from visually rich documents. In NAACL-HLT.
Colin Lockard, Xin Luna Dong, Prashant Shiralkar, and Arash Einolghozati. 2018. Ceres: Distantly supervised relation extraction from the semi-structured web. PVLDB.
Colin Lockard, Prashant Shiralkar, and Xin Luna Dong. 2019. OpenCeres: When open information extraction meets the semi-structured web. In *NAACL-HLT*.
Panupong Pasupat and Percy Liang. 2014. Zero-shot entity extraction from web pages. In ACL.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in PyTorch. In NeurIPS Autodiff Workshop.
Yujie Qian, Enrico Santus, Zhijing Jin, Jiang Guo, and Regina Barzilay. 2019. GraphIE: A graph-based framework for information extraction. In *NAACL-HLT*.
Petar Velicković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph Attention Networks. ICLR.
Minjie Wang, Lingfan Yu, Da Zheng, Quan Gan, Yu Gai, Zihao Ye, Mufei Li, Jinjing Zhou, Qi Huang, Chao Ma, Ziyue Huang, Qipeng Guo, Hao Zhang, Haibin Lin, Junbo Zhao, Jinyang Li, Alexander J Smola, and Zheng Zhang. 2019. Deep graph library: Towards efficient and scalable deep learning on graphs. ICLR Workshop on Representation Learning on Graphs and Manifolds.
Sen Wu, Luke Hsiao, Xiao Cheng, Braden Hancock, Theodoros Rekatsinas, Philip Levis, and Christopher Ré. 2018. Fonduer: Knowledge base construction from richly formatted data. SIGMOD, 2018:1301-1316.
8114
A.1 ClosedIE Label Mappings
SWDE provides OpenIE labels for all binary relations between the objects mentioned on the page and the page topic entity. These labels include the relation string used to indicate the relationship, sometimes including multiple acceptable surface forms if there is more than one applicable string for the relation (usually due to more or less specific versions of the relation). The original SWDE data only includes ClosedIE labels for a small subset of relation types. To create ClosedIE ground truth for all relations on the sites, we examined all OpenIE relations across the SWDE sites and grouped them into a set of relations that each represented the same fundamental idea. In some cases, we chose to map relations into a somewhat more general category, such as mapping "Associate Producer" and "Executive Producer" into the same "Producer" concept. After obtaining this set, we eliminated all relations that appeared on fewer than 3 websites in the dataset. The set of relations used for the ClosedIE experiments is given in Table 5. The full mapping of OpenIE to ClosedIE relations can be found at https://github.com/cdlockard/expanded_swde.
A.2 Training Data Creation
The Extended SWDE dataset provides ground truth extractions of OpenIE predicate and object strings for the webpages it contains. However, it does not specify which text fields on the page were the source of the extractions. To create training data, we need to label a specific text field. It is usually the case that each ground truth string matches only one text field, so there is no ambiguity, but in cases where multiple text fields have the same value, we must disambiguate which one to use. We did this by identifying all matching text fields for the ground truth predicate and object and chose the pair in which the predicate and object strings have the closest Euclidean distance on the rendered page.
While this is generally a safe assumption, there are still occasional errors in the training data. In particular, we observed that the NBA vertical had considerably more ambiguous cases since most relations are numerical and the pages often contained large tables of numbers. We hypothesize that this may explain why performance on the NBA vertical is lower when using Unseen-Website training data compared to the Unseen-Vertical setting (Table 1).
During testing, we applied the same standard
used by prior work on the dataset and accepted an answer as correct if it matched the ground truth string, regardless of which text field produced the extraction.
Vertical	Site	Extraction	Correct	Notes
Page Topic	Relation	Object
Movie	Hollywood	Spanish Fly	Costume Designer	Jose Maria de Cos-sio	Yes
Movie	Metacritic	Saving Face	Reviewed by	Maitland McDon-agh	Yes
NBAPlayer	ESPN	Jameer Nelson	Birth Place	Chester, PA	Yes
NBAPlayer	MSNCA	Matt Bonner	College	Florida	Yes
University	CollegeProwler	Spring Arbor University	Admission Difficulty	Average	Yes
University	MatchCollege	Menlo College	College Credits Accepted	AP Credit	Yes
Movie	RottenTomatoes	Slow Burn	Tomatometer Percentage	97%	No	Subject of relation is not page topic but is an un-related recently released film
Movie	RottenTomatoes	Ginger Snaps 2	WHAT'S HOT ON RT	Trailer: Santa has a bloody Xmas	No	Extracted relation string is not a relation
Movie	Metacritic	The Constant Gardener	User Panel Options	The Constant Gar-dener	No	Extracted relation string is not a relation
University	CollegeProwler	Minnesota School of Business	CP Top 10 Lists	Best Performance Venues	No	Link to article not related to page topic, but is a “Similar School”
University	MatchCollege	Maric College	Highest Degree	Associate's	No	Subject of relation is not page topic
NBAPlayer	FoxSports	Tony Parker	Latest News	Mon. Dec 6, 2010	No	n-ary object
NBAPlayer	MSNCA	Gilbert Arenas	Birthplace	215	No	Erroneous extraction of weight for birthplace (both text fields are nearby)
Table 4: Selected OpenIE Extractions from ZSCERES-GNN with Level I training (no knowledge of the subject vertical).
8116
Vertical	Relation
movie	movie.aka
movie	movie.box_office
movie	movie.budget
movie	movie.cOUNTRY
movie	movie.directed_by
movie	movie.distributor
movie	movie.genre
movie	movie.language
movie	movie.produced_by
movie	movie.production_company
movie	movie.rating
movie	movie.release_date
movie	movie.runtime
movie	movie.starring
movie	movie.synopsis
movie	movie.written_by
movie	movie.year
nbaplayer	nbaplayer.age
nbaplayer	nbaplayer.assists
nbaplayer	nbaplayer.birthdate
nbaplayer	nbaplayer.birthplace
nbaplayer	nbaplayer.college
nbaplayer	nbaplayer.draft
nbaplayer	nbaplayer.experience
nbaplayer	nbaplayer.field_goal_percentage
nbaplayer	nbaplayer.height
nbaplayer	nbaplayer.points
nbaplayer	nbaplayer.position
nbaplayer	nbaplayerrebounds
nbaplayer	nbaplayer.weight
university	university.application_fee
university	university.calendar_system
university	university.control
university	university.enrollment
university	university.in_state_tuition
university	university.out_state_tuition
university	university.phone
university	university.religious_affiliation
university	university.settingting
university	university.tuition
university	university.undergraduate_enrollment
university	university.website
Table 5: A listing of ClosedIE relation types mapped from OpenIE labels in SWDE
8117