Event Coreference Data (Almost) for Free: Mining Hyperlinks from Online News

1 . Introduction . Cross-document event coreference resolution ( CDCR ) is the task of identifying and clustering mentions of real-world events in a given collection of documents . For example , CDCR systems need to decide whether the sentences “ On Monday , Lindsay Lohan checked into rehab in Malibu ” and “ Ms . Lohan entered a rehab facility ” from two different documents refer to the same event , by taking temporal , spatial and other cues from the document contexts into account . CDCR is a vital preprocessing step for downstream multi-document tasks such as question answering or fact checking . Annotation of CDCR is laborious and expensive , requiring expert annotation which can span weeks for several hundred documents [ Cybulska and Vossen , 2014b , Vossen et al. , 2018 ] . Crowdsourcing annotation has been proposed , however this requires extensive training of annotators [ Bornstein et al. , 2020 ] or post-processing by experts [ Bugert et al. , 2020 , 2021 ] , precluding large-scale studies . Annotating CDCR data in a different language requires great effort since language-specific guidelines [ Minard et al. , 2016 ] and enough annotators with proficiency in the target language are required . As a consequence , CDCR corpora had to compromise on size , domain coverage , and the density of annotated mentions and coreference links , as well as language coverage . 1 . Data and model available at https : //github.com/UKPLab/emnlp2021-hypercoref-cdcr This data bottleneck is problematic for three reasons . Firstly , recent state-of-the-art CDCR systems are based on pretrained language models [ Devlin et al. , 2019 , Liu et al. , 2019 ] fine-tuned with supervised learning on human-annotated corpora [ Zeng et al. , 2020 , Yu et al. , 2020 , Caciularu et al. , 2021 , Cattan et al. , 2021 ] . Achieving top results with such models still requires high-quality labeled data for fine-tuning [ Gururangan et al. , 2020 ] , yet the size of current corpora is insufficient for training large systems . Secondly , because corpora need to make compromises on domain coverage , the domain coverage of all existing CDCR corpora is limited even when combined . This holds back research on open-domain CDCR systems which could increase the ( currently limited ) applicability of CDCR in downstream tasks [ Bugert et al. , 2021 ] . Thirdly , because test splits consist only of a few hundred documents ( far less than what downstream applications may require ) , scalability to large corpora is a problem which could not be tackled so far [ Bugert et al. , 2021 ] . To overcome this gap , we leverage hyperlinks in online news articles : When referencing a significant real-world event in the body of an article , writers often add a hyperlink to a different article covering this event . We conjecture that by collecting hyperlinks which point to the same article ( s ) and interpreting anchor texts as mention spans , high-quality crossdocument event coreference links can be retrieved quickly and in large quantity ( see Figure 1 ) . To this end , we devise a data extraction pipeline which mines such datasets automatically from Common Crawl2 and apply it to create the HyperCoref corpus , consisting of 40 news outlets with over 2M mentions in total , far exceeding the size of existing CDCR corpora . HyperCoref achieves broader coverage in event types compared to manually annotated corpora . In experiments with a state-of-the-art CDCR model [ Cattan et al. , 2021 ] , we evaluate the relation between the amount of gold training data and test performance across three CDCR corpora : ECB+ , FCC-T , and GVC . We make the remarkable observation that models trained entirely on silver-standard data from HyperCoref perform on a similar level as models trained on gold-standard data ( between 4 pp . CoNLL F1 worse and 4 pp . 2. https : //commoncrawl.org/ better , depending on the corpus at hand ) . Overall , our findings lift the dependency on gold data for training CDCR systems and pave the way for large , robust and potentially multilingual systems , as our data extraction approach can be easily adapted to any language found on the web . Our contributions are : C1 . A novel approach for acquiring silver-standard cross-document event coreference links from hyperlinks , C2 . HyperCoref , a large corpus created with this approach , and its analysis compared to gold-standard CDCR corpora , C3 . out-of-domain transfer experiments with a state-of-the-art CDCR system , certifying the quality of this data . 2 . Fundamentals . We define cross-document event coreference resolution ( CDCR ) and its relation to hyperlinks in news . Task Definition CDCR consists of two steps : ( 1 ) identifying mentions of real-world or hypothetical events in a collection of documents ( event mention detection ) , and ( 2 ) recognizing which of these mentions refer to the same events , thereby producing a crossdocument clustering of mentions ( coreference resolution ) . Event mentions are commonly defined by four components : their action ( checked into ) , participants ( Lindsay Lohan ) , time ( on Monday ) and location ( rehab in Malibu ) [ Cybulska and Vossen , 2014a ] . Actions are the centerpiece of event mentions , and their token span is the main representative of an event mention in text . We also refer to this span as a mention span . Hyperlinks in News To establish the context of a recent news development , news journalists make reference to other events which have caused , influenced or are otherwise related to the recent newsworthy event . In online news , such references are often marked with a hyperlink to another article which covers the referenced event in greater detail . These hyperlinks can ( with some margin of error ) be interpreted as cross-document coreference links : The hyperlink ’ s anchor text ( its clickable text region ) corresponds to an event mention ’ s action , and the target URL identifies the referenced event . A pair of hyperlinks which point to the same URL but are located in different articles then correspond to two event mentions connected by a CDCR link . This is exemplified in Figure 1 . We propose to collect CDCR data by mining hyperlinks from online news . In the next section , we explain our data pipeline creating such data and the key issues one needs to overcome in the process . 3 . Data Extraction . Following an explanation of our data extraction pipeline , we describe its application for creating the HyperCoref corpus which we then compare to expert-annotated CDCR corpora . 3.1 Pipeline . We apply our pipeline on one news outlet at a time for greater computational efficiency . The following steps are visualized in Figure 2 . Given a news outlet : 1 . We source documents from Common Crawl ( CC ) , a public repository of crawled web pages . We retrieve a list of all pages available for this news outlet from the CC index . 2 . We download these web pages , remove excess markup and detect publication dates with the newspaper3k framework3 . 3 . We drop pages without in- or outgoing hyperlinks . A large portion of pages tend to be duplicates . We deduplicate pages based on their textual content using locality sensitive hashing [ Leskovec et al. , 2020 , ch . 3 ] and clustering of pages by their TF-IDF cosine similarity in a two-stage approach . 4 . We extract hyperlinks , remove their URL query strings , and harmonize the boundaries of their anchor text ( trimming of whitespace , exclusion of punctuation ) . 5 . The removal of hyperlinks which refer to non-eventive content such as overview pages , product reviews , or affiliate products plays a key role . We apply a series of filtering steps : ( a ) Links whose target domain ( amazon.com , facebook.com , etc . ) mismatches the article ’ s domain are removed . ( b ) We make the assumption that the majority of links do refer to eventive content , and that their URLs share syntactic similarities . We build a prefix tree from all URLs and retain only those links whose URLs are part of the 90 % most frequent prefixes . ( c ) Links to pages with high indegree or from pages with high outdegree are removed . ( d ) A set of handwritten rules targeting URLs and anchor texts is applied , which for example remove URLs containing /tag/ , /category/ , or anchor texts such as “ click here ” . 6 . Groups of hyperlinks sharing the same anchor text and target URL are removed entirely . This eliminates any remaining links to hub pages or “ read more ” -type links appearing out of context on multiple pages . All pipeline steps except for the handwritten rules in step 5d ) are language-independent , hence the pipeline can be easily applied to news in any language found on the web . We limit ourselves to English news in this work since the majority of gold-standard CDCR corpora are in English . In several test runs with Dutch , German , and French news we observed results of similar quality to those created from English news . 3. https : //pypi.org/project/newspaper3k/ 3.2 The HyperCoref Corpus . We apply our pipeline to 40 news outlets from English-speaking countries to produce HyperCoref , a corpus consisting of 2M documents , 0.8M hyperlink clusters ( =̂ events ) and 2.7M hyperlinks ( =̂ event mentions ) . The most recent documents included stem from the October 2020 crawl of CC . We compare HyperCoref to three CDCR corpora : EventCorefBank+ ( ECB+ ) [ Cybulska and Vossen , 2014b ] , a corpus containing news articles from a broad selection of 45 topics , the Football Coreference Corpus ( FCCT ) [ Bugert et al. , 2020 , 2021 ] which annotated football match events in sports news and the Gun Violence Corpus ( GVC ) [ Vossen et al. , 2018 ] which annotates fine-grained gun violence events in news . A size comparison is shown in Table 1a , demonstrating that HyperCoref is several orders of magnitude larger than expert-annotated corpora . Additional analysis ( see Appendix A.1 ) reveals that the distribution of cluster sizes is comparable between HyperCoref and expert-annotated corpora , however anchor texts tend to be longer phrases or even entire sentences , opposed to the minimum span annotations of action triggers pursued for the creation ECB+ . To keep further analysis and experiments manageable , the remainder of this work focuses on ABC News ( abcnews.go.com ) and BBC News ( bbc.com ) , two large and reputable news outlets . Event Types We heuristically determine the event types contained in each corpus by performing WSD on the syntactic head of mentions against WordNet [ Miller , 1995 ] , choosing the most frequent sense and counting word sense occurrences . Table 1b shows the top word senses of three ( sub- ) corpora . Compared to ECB+ , ABC contains a greater proportion of reporting events ( cf . Pustejovsky et al . [ 2003 ] ) and mentions using light verbs , which are challenging for coreference resolution [ Hovy et al. , 2013 , Choubey and Huang , 2017 ] . BBC consists of events from the sports domain . We count the number of unique verbal word senses of mentions to estimate the event type coverage per corpus ( see Table 1a ) . HyperCoref exhibits considerably broader coverage than previous CDCR corpora .

Summary: This paper collects and releases a substantial new resource for the challenging task of cross-document event coreference, of "silver" coreference clusters automatically constructed from hyperlinks in web news articles.  It is substantially larger than previous human-annoated cross-doc event coref datasets, which is a task that is extremely challenging to annotate.  The authors also demonstrate that training a model on this silver data yields a high-performing model as evaluated on a previous human-annotated dataset.