SciCo: Hierarchical Cross-Document Coreference for Scientific Concepts

task in natural language understanding . Previous work on cross-document coreference resolution ( CDCR ) typically considers mentions of events in the news , which seldom involve abstract technical concepts that are prevalent in science and technology . These complex concepts take diverse or ambiguous forms and have many hierarchical levels of granularity ( e.g. , tasks and subtasks ) , posing challenges for CDCR . We present a new task of Hierarchical CDCR ( H-CDCR ) with the goal of jointly inferring coreference clusters and hierarchy between them . We create SCICO , an expert-annotated dataset for H-CDCR in scientific papers , 3X larger than the prominent ECB+ resource . We study strong baseline models that we customize for H-CDCR , and highlight challenges for future work . 1 . Introduction . Cross-document coreference resolution ( CDCR ) identifies and links textual mentions that refer to the same entity or event across multiple documents . This fundamental task has seen much work recently [ Choubey and Huang , 2017 , Kenyon-Dean et al. , 2018 , Barhom et al. , 2019 , Cattan et al. , 2021a , b , Caciularu et al. , 2021 ] and can benefit various downstream applications such as multi-hop question answering [ Dhingra et al. , 2018 , Wang et al. , 2019 ] , multi-document summarization [ Falke et al. , 2017 ] , and discovery of cross-document relations [ Hope et al. , 2017 , 2020 , 2021 ] . Existing datasets for CDCR , such as ECB+ [ Cybulska and Vossen , 2014 ] , focus on mentions of news events involving concrete entities such as people or places . Abstract technical concepts are largely unexplored despite their prevalence in domains such as science and technology , and can pose significant challenges for CDCR : they often take diverse forms ( e.g. , class-conditional image synthesis and categorical image generation ) or are ambiguous ( e.g. , network architecture in AI vs. systems research ) . These complex concepts also have many hierarchical levels of granularity , such as tasks that can be divided into finer-grained subtasks , where reference to a specific concept entails also a reference to the higher-level concept ( e.g. , CRF entails the sequence tagging task ) , unlike events and entities in ECB+ that are treated as non-hierarchical . In this paper , we formulate a novel task of hierarchical CDCR ( H-CDCR ) . The task is to infer ( 1 ) crossdocument entity coreference clusters of concept mentions in scientific papers , and ( 2 ) referential hierarchy between clusters , where referring to a child cluster entails reference to the parent . Figure 1 shows the structure we aim to induce given mentions in context . Our task is the first to consider unified CDCR and hierarchy between clusters of mentions in context , unlike most work focusing on lexicon-level taxonomies over uncontextualized words [ Shwartz et al. , 2016 , Zhang et al. , 2018 ] . H-CDCR in science can support many applications . One example illustrating our setting is faceted query by example [ Mysore et al. , 2021 ] , where given a paper , a user may highlight a specific span ( facet ) ( e.g. , network architecture ) , and explore other papers that refer to the same concept while resolving ambiguity and clustering diverse surface forms . Selecting a facet may also show concept hierarchies enabling users to browse and navigate collections [ Hope et al. , 2020 ] ( e.g. , by viewing a list of many types of network architectures , automatically identified ) . See §6 for more discussion of related areas and problems that could benefit from H-CDCR in the scientific domain . To advance research in this area and enable supervised model training and evaluation , we create a new large-scale dataset named SCICO ( Scientific Concept Induction Corpus ) , which is annotated by domain experts . SCICO consists of clusters of mentions in context and a hierarchy over them , as shown in Figure 1 . The corpus is drawn from computer science papers , and the concept mentions are methods and tasks from across CS . To build SCICO , we develop a new candidate generation approach built on three resources : a low-coverage KB , a noisy hypernym extractor , and curated candidates . We evaluate strong baseline models , finding that a cross-encoder model addressing coreference and hierarchy jointly outperforms others.1 Our main contributions include : • We formulate the novel task of hierarchical cross-document coreference ( H-CDCR ) , and explore it within scientific papers . • We release SCICO , an expert-annotated dataset 3X larger than the prominent ECB+ CDCR dataset . • We build a model for H-CDCR that outperforms multiple baselines while leaving much room for future improvement . 2 . The H-CDCR Task . 2.1 Problem Formulation . Our goal is to induce clusters of contextualized mentions that corefer to the same concept , and to infer a hierarchy over these concept clusters . Formally , we are given a set of documents D from a 1 . SCICO , code and models are available at https : //scico.apps.allenai.org/ diverse corpus . We assume each d ∈ D comes annotated with mentions ( spans of text , see Table 1 ) denoting concepts . Denote byMd = { m1 , m2 , ... , mn } the set of mentions in document d and by M the set of mentions across all d ∈ D. The first goal , similar to cross-document coreference resolution , is to cluster the mentions inM into disjoint clusters C = { C1 , ... , Ct } , with each cluster consisting of mentions { m|m ∈ Ci } that all refer to the same underlying concept ( see Figure 1 ) . To account for the difficulty in precisely delineating the “ borders ” of extremely fine-grained concepts in scientific literature , clusters are allowed to include subtle variations around a core concept ( e.g. , CRF model , CRF tagger ) . The second goal is to infer a hierarchy over clusters . Define a graph GC = ( C , E ) , with vertices representing C ( mention clusters ) , and directed edges ij ∈ E , each edge representing a hierarchical relation between clusters Ci and Cj which reflects referential hierarchy . A relation C1 −→ C2 exists when the concept underlying C2 entails a reference to C1 . For example , a text that mentions the concept “ BERT model ” also ( implicitly ) invokes several other concepts ( “ neural language model ” , “ neural nets ” , “ NLP ” ) but not others ( “ robotics ” , “ RoBERTa model ” ) . In section 3.3 , we ground this entailment definition with a faceted search application to assist the annotation . Put together , the goal in our Hierarchical Cross Document Coreference ( H-CDCR ) task is : Given documents D and mentionsM , construct clusters C and a hierarchy graph GC = ( C , E ) by learning from a set of N examples { ( Dk , Mk , GkC ) } Nk=1 . In our experiments , we focus on mentions referring to tasks and methods in computer science papers ( see examples in Table 1 and §3 ) . 2.2 Evaluation Metrics . Comparing an extracted set of mention clusters and hierarchies to a gold standard for evaluation is non-trivial . Evaluation metrics for coreference resolution ( e.g. , MUC , B3 , CEAFe , and LEA ) do not involve relations between clusters , and models for inferring hierarchical relations such as hypernymy do so over pairs of individual terms . Therefore , in addition to reporting established metrics for coreference resolution , we devise two specific metrics for our novel unified task . Cluster-level Hierarchy Score In H-CDCR , hierarchical relations are defined over clusters of mentions . This complicates the evaluation of the hierarchy , since a system may output a different set of clusters from the gold due to coreference errors . Our cluster-level score is intended to evaluate hierarchical links without penalizing coreference errors a second time . Let CSP −→ CSC be a hierarchical link output by the system between a parent cluster CSP and a child cluster CSC . We define this link to be a true positive iff there exists some pair of mentions in these two clusters that are also in a parent-child relationship in the gold hierarchy . That is , it is positive iff there exist mentions p ∈ CSP and c ∈ CSC which are also in two gold clusters , i.e . p ∈ CGP and c ∈ CGC , such that CGP −→ CGC in the gold.2 If not , the output link counts as a precision error . We define recall errors analogously , swapping system and gold in the above formalism . As an example , see Figure 2 , where p = IE and c = Pattern-based definition extraction . The metric treats this link as a true positive , since there is at least one pair of mentions in the gold over which the same hierarchy relation holds . Requiring at least one pair , rather than greater overlap , is intended to maximally decouple the coreference and hierarchy penalization . We also report results when requiring a more conservative 50 % overlap , to examine robustness to this choice . Path-Distance Score The cluster-level hierarchy score does not take into account the degree of separation between concepts in the graph . We devise a score that does , following previous work on graph-based semantic distances [ Lee et al. , 1993 ] . Here , we consider for each pair of mentions ( m1 , m2 ) the number of cluster-level directed edges needed to traverse from m1 to m2 , plus one ( mentions in the same cluster have a distance of 1 ) . For each mention pair , we compute this distance for both the ground-truth and the model ( see Figure 2 ) , and the ratio between the smaller of the two distances and the larger . For pairs that are disconnected in the gold or the system but not both , we treat the ratio as zero , and pairs disconnected in both are ignored . The ratios are averaged to obtain the final score . 2 . We also apply transitive closure , adding edges between pairs of clusters with indirect hierarchical relations ( e.g. , across the first and last nodes in the chain computer vision −→ image synthesis −→ categorial image synthesis ) , following common practice in hierarchy prediction [ Li et al. , 2018 ] . More formally , let W be the union of mention pairs that have a path in the gold or system cluster-graph . The path-distance score is defined as follows : 1 |W | ∑ i , j∈W min ( psys ( i , j ) , pgold ( i , j ) ) + 1 max ( psys ( i , j ) , pgold ( i , j ) ) + 1 ( 1 ) where pgold ( i , j ) and psys ( i , j ) are the distances between the mentions i and j in the gold and the system , respectively . This metric gives partial credit for similar but not exactly matching graphs , e.g. , a predicted parent-child relation is not considered a complete error if the gold specifies a coreference relation ( the ratio is 1/2 ) . This can help in ambiguous cases ; e.g. , whether CRF model is a parent of CRF tagger or belongs in the same cluster .

Summary: - The paper proposes the new task of hierarchical cross document coreference resolution (HCDCR), where mention clusters are organized in a hierarchy.  This can enable applications such as faceted query where the user can explore papers that refer to similar (but not necessarily the same) concepts (via a hierarchy).  To support this task, the authors contribute both a dataset (SciCO consisting of ~20K documents and ~26K mentions) and metrics for evaluation.  The dataset is constructed in a semi-automatic fashion, annotators are asked to annotate mention clusters and concept hierarchy, starting from candidate mention sets that are selected for annotators to label (by using a mix existing hierarchy of ML terms, automatically extracted hypernyms, and small curated list of lexically diverse concepts), while . Baselines (a coreference and a hierarchical model) and a unified model (predict both coreference and hierarchy) is proposed and compared on the dataset to provide an initial set of benchmark results.