Event Coreference Data (Almost) for Free: Mining Hyperlinks from Online News

1 . Introduction . Cross-document event coreference resolution ( CDCR ) is the task of identifying and clustering mentions of real-world events in a given collection of documents . For example , CDCR systems need to decide whether the sentences “ On Monday , Lindsay Lohan checked into rehab in Malibu ” and “ Ms . Lohan entered a rehab facility ” from two different documents refer to the same event , by taking temporal , spatial and other cues from the document contexts into account . CDCR is a vital preprocessing step for downstream multi-document tasks such as question answering or fact checking . Annotation of CDCR is laborious and expensive , requiring expert annotation which can span weeks for several hundred documents [ Cybulska and Vossen , 2014b , Vossen et al. , 2018 ] . Crowdsourcing annotation has been proposed , however this requires extensive training of annotators [ Bornstein et al. , 2020 ] or post-processing by experts [ Bugert et al. , 2020 , 2021 ] , precluding large-scale studies . Annotating CDCR data in a different language requires great effort since language-specific guidelines [ Minard et al. , 2016 ] and enough annotators with proficiency in the target language are required . As a consequence , CDCR corpora had to compromise on size , domain coverage , and the density of annotated mentions and coreference links , as well as language coverage . 1 . Data and model available at https : //github.com/UKPLab/emnlp2021-hypercoref-cdcr This data bottleneck is problematic for three reasons . Firstly , recent state-of-the-art CDCR systems are based on pretrained language models [ Devlin et al. , 2019 , Liu et al. , 2019 ] fine-tuned with supervised learning on human-annotated corpora [ Zeng et al. , 2020 , Yu et al. , 2020 , Caciularu et al. , 2021 , Cattan et al. , 2021 ] . Achieving top results with such models still requires high-quality labeled data for fine-tuning [ Gururangan et al. , 2020 ] , yet the size of current corpora is insufficient for training large systems . Secondly , because corpora need to make compromises on domain coverage , the domain coverage of all existing CDCR corpora is limited even when combined . This holds back research on open-domain CDCR systems which could increase the ( currently limited ) applicability of CDCR in downstream tasks [ Bugert et al. , 2021 ] . Thirdly , because test splits consist only of a few hundred documents ( far less than what downstream applications may require ) , scalability to large corpora is a problem which could not be tackled so far [ Bugert et al. , 2021 ] . To overcome this gap , we leverage hyperlinks in online news articles : When referencing a significant real-world event in the body of an article , writers often add a hyperlink to a different article covering this event . We conjecture that by collecting hyperlinks which point to the same article ( s ) and interpreting anchor texts as mention spans , high-quality crossdocument event coreference links can be retrieved quickly and in large quantity ( see Figure 1 ) . To this end , we devise a data extraction pipeline which mines such datasets automatically from Common Crawl2 and apply it to create the HyperCoref corpus , consisting of 40 news outlets with over 2M mentions in total , far exceeding the size of existing CDCR corpora . HyperCoref achieves broader coverage in event types compared to manually annotated corpora . In experiments with a state-of-the-art CDCR model [ Cattan et al. , 2021 ] , we evaluate the relation between the amount of gold training data and test performance across three CDCR corpora : ECB+ , FCC-T , and GVC . We make the remarkable observation that models trained entirely on silver-standard data from HyperCoref perform on a similar level as models trained on gold-standard data ( between 4 pp . CoNLL F1 worse and 4 pp . 2. https : //commoncrawl.org/ better , depending on the corpus at hand ) . Overall , our findings lift the dependency on gold data for training CDCR systems and pave the way for large , robust and potentially multilingual systems , as our data extraction approach can be easily adapted to any language found on the web . Our contributions are : C1 . A novel approach for acquiring silver-standard cross-document event coreference links from hyperlinks , C2 . HyperCoref , a large corpus created with this approach , and its analysis compared to gold-standard CDCR corpora , C3 . out-of-domain transfer experiments with a state-of-the-art CDCR system , certifying the quality of this data . 2 . Fundamentals . We define cross-document event coreference resolution ( CDCR ) and its relation to hyperlinks in news . Task Definition CDCR consists of two steps : ( 1 ) identifying mentions of real-world or hypothetical events in a collection of documents ( event mention detection ) , and ( 2 ) recognizing which of these mentions refer to the same events , thereby producing a crossdocument clustering of mentions ( coreference resolution ) . Event mentions are commonly defined by four components : their action ( checked into ) , participants ( Lindsay Lohan ) , time ( on Monday ) and location ( rehab in Malibu ) [ Cybulska and Vossen , 2014a ] . Actions are the centerpiece of event mentions , and their token span is the main representative of an event mention in text . We also refer to this span as a mention span . Hyperlinks in News To establish the context of a recent news development , news journalists make reference to other events which have caused , influenced or are otherwise related to the recent newsworthy event . In online news , such references are often marked with a hyperlink to another article which covers the referenced event in greater detail . These hyperlinks can ( with some margin of error ) be interpreted as cross-document coreference links : The hyperlink ’ s anchor text ( its clickable text region ) corresponds to an event mention ’ s action , and the target URL identifies the referenced event . A pair of hyperlinks which point to the same URL but are located in different articles then correspond to two event mentions connected by a CDCR link . This is exemplified in Figure 1 . We propose to collect CDCR data by mining hyperlinks from online news . In the next section , we explain our data pipeline creating such data and the key issues one needs to overcome in the process . 3 . Data Extraction . Following an explanation of our data extraction pipeline , we describe its application for creating the HyperCoref corpus which we then compare to expert-annotated CDCR corpora . 3.1 Pipeline . We apply our pipeline on one news outlet at a time for greater computational efficiency . The following steps are visualized in Figure 2 . Given a news outlet : 1 . We source documents from Common Crawl ( CC ) , a public repository of crawled web pages . We retrieve a list of all pages available for this news outlet from the CC index . 2 . We download these web pages , remove excess markup and detect publication dates with the newspaper3k framework3 . 3 . We drop pages without in- or outgoing hyperlinks . A large portion of pages tend to be duplicates . We deduplicate pages based on their textual content using locality sensitive hashing [ Leskovec et al. , 2020 , ch . 3 ] and clustering of pages by their TF-IDF cosine similarity in a two-stage approach . 4 . We extract hyperlinks , remove their URL query strings , and harmonize the boundaries of their anchor text ( trimming of whitespace , exclusion of punctuation ) . 5 . The removal of hyperlinks which refer to non-eventive content such as overview pages , product reviews , or affiliate products plays a key role . We apply a series of filtering steps : ( a ) Links whose target domain ( amazon.com , facebook.com , etc . ) mismatches the article ’ s domain are removed . ( b ) We make the assumption that the majority of links do refer to eventive content , and that their URLs share syntactic similarities . We build a prefix tree from all URLs and retain only those links whose URLs are part of the 90 % most frequent prefixes . ( c ) Links to pages with high indegree or from pages with high outdegree are removed . ( d ) A set of handwritten rules targeting URLs and anchor texts is applied , which for example remove URLs containing /tag/ , /category/ , or anchor texts such as “ click here ” . 6 . Groups of hyperlinks sharing the same anchor text and target URL are removed entirely . This eliminates any remaining links to hub pages or “ read more ” -type links appearing out of context on multiple pages . All pipeline steps except for the handwritten rules in step 5d ) are language-independent , hence the pipeline can be easily applied to news in any language found on the web . We limit ourselves to English news in this work since the majority of gold-standard CDCR corpora are in English . In several test runs with Dutch , German , and French news we observed results of similar quality to those created from English news . 3. https : //pypi.org/project/newspaper3k/ 3.2 The HyperCoref Corpus . We apply our pipeline to 40 news outlets from English-speaking countries to produce HyperCoref , a corpus consisting of 2M documents , 0.8M hyperlink clusters ( =̂ events ) and 2.7M hyperlinks ( =̂ event mentions ) . The most recent documents included stem from the October 2020 crawl of CC . We compare HyperCoref to three CDCR corpora : EventCorefBank+ ( ECB+ ) [ Cybulska and Vossen , 2014b ] , a corpus containing news articles from a broad selection of 45 topics , the Football Coreference Corpus ( FCCT ) [ Bugert et al. , 2020 , 2021 ] which annotated football match events in sports news and the Gun Violence Corpus ( GVC ) [ Vossen et al. , 2018 ] which annotates fine-grained gun violence events in news . A size comparison is shown in Table 1a , demonstrating that HyperCoref is several orders of magnitude larger than expert-annotated corpora . Additional analysis ( see Appendix A.1 ) reveals that the distribution of cluster sizes is comparable between HyperCoref and expert-annotated corpora , however anchor texts tend to be longer phrases or even entire sentences , opposed to the minimum span annotations of action triggers pursued for the creation ECB+ . To keep further analysis and experiments manageable , the remainder of this work focuses on ABC News ( abcnews.go.com ) and BBC News ( bbc.com ) , two large and reputable news outlets . Event Types We heuristically determine the event types contained in each corpus by performing WSD on the syntactic head of mentions against WordNet [ Miller , 1995 ] , choosing the most frequent sense and counting word sense occurrences . Table 1b shows the top word senses of three ( sub- ) corpora . Compared to ECB+ , ABC contains a greater proportion of reporting events ( cf . Pustejovsky et al . [ 2003 ] ) and mentions using light verbs , which are challenging for coreference resolution [ Hovy et al. , 2013 , Choubey and Huang , 2017 ] . BBC consists of events from the sports domain . We count the number of unique verbal word senses of mentions to estimate the event type coverage per corpus ( see Table 1a ) . HyperCoref exhibits considerably broader coverage than previous CDCR corpora .

Summary: The paper collects a new ‘silver’ dataset HyperCoref for cross document coreference resolution task, constructing by using hyperlinks in online news articles. The paper shows that the dataset is considerably larger () than existing human-annotated datasets and has broader coverage in terms of event types. Experiments on three existing human-annotated datasets (ECB+, FCC-T, GVC) show that models trained on the silver-dataset perform comparable to those trained on gold train data.