Venue: AKBC 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary:
- The paper proposes the new task of hierarchical cross document coreference resolution (HCDCR), where mention clusters are organized in a hierarchy.  This can enable applications such as faceted query where the user can explore papers that refer to similar (but not necessarily the same) concepts (via a hierarchy).  To support this task, the authors contribute both a dataset (SciCO consisting of ~20K documents and ~26K mentions) and metrics for evaluation.  The dataset is constructed in a semi-automatic fashion, annotators are asked to annotate mention clusters and concept hierarchy, starting from candidate mention sets that are selected for annotators to label (by using a mix existing hierarchy of ML terms, automatically extracted hypernyms, and small curated list of lexically diverse concepts), while . Baselines (a coreference and a hierarchical model) and a unified model (predict both coreference and hierarchy) is proposed and compared on the dataset to provide an initial set of benchmark results.  

Quality: The submission appears to be technically sound with well-thought out metrics and a reasonable set of experiments and natural initial models for the proposed task.

Clarity: The paper is very well written with clear description of the proposed task, metrics, data annotation process and experiments.

Originality: The paper contributes several original ideas (the HCDCR task, evaluation metrics, and proposed models - the models are mostly based on prior work and applied to the new task)

Significance: The work could stimulate research in a more fine-grained connections between documents.

Pros:
- The paper presents and interesting new task with dataset and metrics
- A initial model and set of experiments is provided 

Cons:
- From a quick read over the paper, without careful examination of the data, it is difficult to judge the quality of the annotations and the dataset, and potential issues with the metrics. 
- The automated construction of the dataset may introduce biases that favor algorithms that use the same resources/methods.
- It's unclear if the task would be of interest to the broader community or just a very specialized subset
review_title: Interesting new task with dataset and metrics

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: This paper proposes a reformulation of CDCR to include an hierarchical component (H-CDCR), whereby an hierarchy of co-reference clusters is jointly 
inferred with the clusters. 

Along with the new task, two evaluation metrics are proposed, the cluster-level hierarchy score that evaluates hierarchical links while not penalizing co-reference errors repeatedly (TP when at least one Parent -> Child relation between mentions in common between the system output and the gold cluster hierarchy), the path-based score average path-distance ratios over all pairs of mentions. 

The authors propose SciCo, a resource and benchmark for (H-)CDCR thrice the size of the previous reference dataset (ECB+).
The dataset is constructed following a robust protocol that follows established conventions for the annotation of coreference data and that is based on two main components: a large corpus of scientific papers annotated with mentions and a set of seed taxonomies to select mentions that are likely to have hierarchical relations. 

The corpus of mentions is build on a corpus of 10M computer science abstracts  + SciREX using a state-of-the-art information extraction model DyGIE++. The model is trained on SciERC, a reference corpus of manually verified mention annotations. 

The selection of likely cluster candidates is performed using the papers with code taxonomy, hypernym relations extracted by DyGIE++ (aggregated by concept across all papers). The candidates are augmented with a manually curated list to add diversity in the dataset. 
Candidates are grouped into one-parent-multiple-children hierarchies and candidate mentions are retrieved for each group to form the CDCR "topics" using a vector-based retrieval of mentions from the large corpus annotated with mentions. 

4 annotators (PhD or graduate) were hired (upwork) to perform the annotations. The corpus shows good agreement (well above system performance, leaving room for improvement). 

The authors then perform an evaluation of several baseline systems adapted for the new formulation of the task (SOTA Coreference model and a textual entailment model (for the hierarchy)) and implement their own joint model that significantly outperforms the baseline models. 
The joint model casts the association of a pair of mentions as a text classification problem, where two mentions can belong to the following classes: coreference, m1 > m2, m2 > m1, unrelated. 

The evaluation is complemented with an ablation study that examines performance with coreference and hierarchy only variants. The impact of lexical diversity is also studied. 


Strengths: 
 - Well written paper
 - Very strong contributions, both data-wise and methodology-wise

Shortcomings: 
 - The paper feels somewhat compressed to fit into the format, some of the elements in the appendices should really be in the paper as the sole descriptions in the paper wouldn't allow to fully reproduce the work. I really needed to read the appendices to fully understand some aspects. 




Other comments and questions: 

"SciREX has the advantage of introducing mentions from full paper texts, vetted by a human annotator for quality" -> do you mean SciERC?

Formatting issue with the reference for "Hierarchical grouping to optimize an objective function." 

About the annotators, it's common to report whether the annotators were compensated fairly for the work and the associated workload. 

The textual description for the path-based measure is somewhat difficult to follow, a formula or small algorithm that summarizes the computation would be a welcome addition. 

review_title: An interesting reformulation of CDCR accompanied by a robust dataset and models 

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: # General comments
This paper presents a new dataset and baselines for the task of cross-document coreference resolution for concepts in CS papers. In addition to the regular coreference clusters, the authors introduce a hierarchical element, adding parent-child relations between the concepts. Overall, this is a rigorous annotation workflow and the baseline experiments are well designed show the challenging nature of the dataset. I only have a few issues.


# Specific comments/questions
While it is certainly novel in its current formulation (and in the context of scientific concepts), I would argue that the task of hierarchical cross-document coreference has been studied in the past, e.g. the work of Signh et al. (2011) "Large-Scale Cross-Document Coreference Using Distributed Inference and Hierarchical Models".

In Section 3.2 there is a mention of a "parent concept". How is that determined for the automatically generated groups and where is it specified for the curated list presented in Section 3.1?

The example (and justification) given for the referential hierarchy (Section 3.3 and Appendix A.3) isn't entirely convincing. From an information retrieval point of view, I wouldn't mind if a query for Penn TreeBank POS tagging returned documents related to POS tagging in general (in fact it might be more useful to place a specific tagging scheme in the context of its research area).

Why is the Longformer needed in Section 4.2 if the pairwise scorer f(.,.) accepts only the mentions as input? Are the contexts of each also used in the scorer?

# Minor comments
- Flip the order of the presentation of the metrics in Figure 2 to match the order of their introduction in the text.
- I assume the annotators hired (Section 3.3) were from the CS field?
- The CoNLL F1 metric is used in Section 3.4 before its definition in Section 5.
- Are "deep neural networks" and "neural networks" equivalent (as stated in Section 4.2)? I would argue the two terms have exactly the same relationship as "PTB POS tagging" and "POS tagging" (one being a subset of the other).
- For clarity, add the human IAA scores in Table 3.
- The names of the Ablation experiments (Table 4) don't match the names/descriptions used in the text.
review_title: An interesting expansion on cross-document coreference, rigorously annotated dataset and good baselines

Meta-review: This work introduces a new task of hierarchical cross-document coreference resolution, together with a new dataset and baselines for this task. Reviewers all agreed that this work makes strong contributions (the new task, evaluation metrics, and methods), and is well-written with clear description. Reviewers also raised minor concerns towards the quality of the dataset, annotation details, and the connection to prior work, for which the authors have provided detailed clarifications during the discussion stage. This is a clear accept and we encourage the authors to incorporate reviewers’ suggestions and their explanations in these authors' responses into the final version of the paper.
Decision: Accept