SlowGuess/NAACL_2022_OCR huggingface dataset:
vector, which they feed into a pairwise classifier to generate pairwise scores. We on the other hand use the sentences containing the span in question and additional context sentences from the document, keeping a total of 128 word piece tokens. This additional context from the document, despite keeping fewer tokens, accounts for much of the performance gain. This is discussed in further detail in §4.5.
4.3 Predicted Topic Clusters
We compare our model against the majority of earlier works that used predicted topic clusters and gold mentions (see Table 6 and Appendix A Table 8 for more complete results). We used the reference implementation by Pradhan et al. (2014) to score our models with singletons. Our model is competitive with earlier approaches (Yu et al., 2020; Zeng et al., 2020; Caciularu et al., 2021), despite using significantly fewer resources at inference time  $n$  transformer computations at inference time as oppose to  $n^2$  transformer computations. We also note that in contrast to our approach, Caciularu et al. (2021) used a total of 600 tokens from each document (most documents are within 512 tokens) whereas we only use 128 tokens. Models by Yu et al. (2020); Zeng et al. (2020) employ a BERT based semantic role labelling (SRL) model. On average, our model lags their models by approximately 1.1 CoNLL F1 points, however, we note that Yu et al. (2020) find that the SRL tagging accounted for roughly 0.4 CoNLL F1 points.
When comparing to other models that are linear in transformer computations, our model does well. Compared to the work by Allaway et al. (2021), our model surpasses their results by 3.7 CoNLL
Scaling	Adapt.	Fine-tuned	SRL	Encoder	System	MUC F1	B3F1	CEAF-e F1	CoNLL F1
Events	n2	✓	✓	✓	BERT-large	Baseline	76.7	77.5	73.2	75.7
✓	✓	RoBERTa-large	Zeng et al. (2020)	87.5	83.2	82.3	84.3
✓	✓	Longformer	Yu et al. (2020)	86.6	85.4	81.3	84.4
RoBERTa-large	Caciularu et al. (2021)	88.1	86.4	82.2	85.6
n	✓	✓	✓	BERT-base	Cattan et al. (2021a)	83.5	82.4	77.0	81.0
Allaway et al. (2021)	82.2	81.1	79.1	80.8
Ours
- RoBERTa-large	85.6	84.8	79.6	83.3
- RoBERTa-base	84.0	82.4	79.0	81.8
- BERT-large	82.8	82.3	77.9	81.0
- BERT-base	79.8	79.4	74.4	77.9
Entities	n2	✓	✓	Longformer	Baseline	70.7	61.7	56.9	63.1
RoBERTa-large	Caciularu et al. (2021)	89.9	82.1	76.8	82.9
✓	✓	✓	BERT-base	Cattan et al. (2021a)	83.6	72.7	63.1	73.1
Ours	Allaway et al. (2021)	84.3	72.4	69.2	75.3
- RoBERTa-large	87.1	80.3	73.1	80.2
- RoBERTa-base	83.6	74.1	68.5	75.4
- BERT-large	80.8	71.4	66.2	72.8
- BERT-base	78.2	68.9	62.7	69.9
F1 points on average. We note however, that their model used a BERT-base model and that they also introduced a novel sequential clustering approach. Our methodology used the larger RoBERTa-large model, and we utilized an agglomerative clustering algorithm as in previous works.
Finally, in contrast to earlier works, we note that our model performs equally well when using predicted clusters and  $\mathrm{ECB + }$  gold topics. In fact, our model does better (by 0.9 CoNLL F1 points) on entities when going to gold topics, and achieves the same performance on events using gold topics. This is related to how we selected our contrastive pairs - negative and positive pairs were selected from within each topic and so our model focused on the lexical ambiguity in the  $\mathrm{ECB + }$  corpus.
4.4 Training and Inference Time
Our model is larger than earlier models by Cattan et al. (2021a,b); Allaway et al. (2021). On a single V100 Tensor Core GPU with 32 GB of RAM, training took approximately two days. This is comparable to reported times for the cross-encoder model (using Longformer) by (Caciularu et al., 2021). We note that contrastive learning methods have been found to converge slowly (Sohn, 2016). At infer
Table 6: A comparison of methods utilizing contextual embedding models and their performance on the ECB+ test corpus using predicted topic clusters of Barhom et al. (2019). We have indicated the scaling at inference time (in terms of transformer computations) above. We have also indicated whether systems utilized adaptive pre-training (Adapt.), fine-tuned encoders (Fine-tuned), or utilized a semantic role labelling model (SRL). To better compare to earlier works, we have included results from using different encoders in our model and indicated which encoders were used in earlier works. Finally, Allaway et al. (2021) used sequential clustering algorithm whereas ours and Cattan et al. (2020) utilized an agglomerative clustering algorithm. Bold indicates best overall. Underlined results indicate our best overall.
Entities	Events
F1	Δ	F1	Δ
Pred. Topics	Our Model	80.2	83.3
-CLS representation	77.8	-2.4	82.3	-1.0
-mention representation	77.8	-2.4	82.0	-1.3
-no document context	74.2	-6.0	80.8	-2.5
Gold Topics	Our Model	81.1	83.3
-CLS representation	79.0	-2.1	81.3	-2.0
-mention representation	78.9	-2.2	79.6	-3.7
-no document context	75.1	-6.0	77.1	-6.2
Corpus	Our Model	78.7	75.9
-CLS representation	77.3	-1.4	74.9	-1.0
-mention representation	75.7	-3.0	72.0	-3.9
-no document context	72.8	-6.0	70.5	-5.4
Table 7: Ablation results (CoNLL F1) on the ECB+ test set with singletons.
ence time, however, our model takes approximately 15 seconds to evaluate on the  $\mathrm{ECB + }$  test set of events (using gold mentions and with singletons included). As a point of comparison, we ran the model of Cattan et al. (2021a,b) which likewise uses RoBERTa-large and is linear in transformer computations. We found that their model takes approximately 60 seconds under similar settings. In §4.5 we discuss experiments with smaller models.
We ablate several parts of our model using the headlines heuristic and examine the importance of the underlying language model, the token representations, and the document context.
Language Model We examined the effect different representations have on overall performance by ablating the language model used. We found that the larger and richer representations of the RoBERTa-large model performed better generically. We gained on average 5 CoNLL F1 points in using RoBERTa-large versus BERT-large. We gained on average 7.2 CoNLL F1 points versus the smaller BERT-base model. Details can be found in Table 6.
Token Representation To assess the effect of including the CLS token embedding in the final representations, we trained our model without using its representation, but keeping the mention representation. We find that the CLS representation accounts for roughly 1.3 CoNLL F1 points on average while the mention representation accounts for roughly 2.8 CoNLL F1 points on average (see Table 7 for details). We also examined our model without explicitly using the mention representation, but still tagging the span with  $[E], [/E]$  tokens. For our model, we find that the mention representation was a more important factor when considering events. We speculate that tagging the mention location with  $[E], [/E]$  tokens allows the transformer to attend to the mention. For events, which have a more complicated structure (e.g. arguments) this likely has a more important effect.
Document Context Finally, an important component of our model was including the first two sentences of each document in the spirit of Caci-ularu et al. (2021). For the  $\mathrm{ECB + }$  corpus, which is comprised of news articles, much contextual information is contained in the first two sentences of the document. We see that the document context contributes on average 5.4 CoNLL F1 points (see Table 7 for details). This is in line with our expectations for new articles and earlier observations by Caci-ularu et al. (2021). We suspect the importance of this feature is due to a property of the  $\mathrm{ECB + }$  corpus that has been highlighted by others - namely, the documents form fairly distinct clusters in themselves and so simple document embeddings are able to recover subtopics easily (Cattan et al., 2021a,b; Cremisini and Finlayson, 2020). Note
for instance that our model without using document context is competitive (compare with Table 6) when using predicted topics for pre-clustering. Recently, Eirew et al. (2021) sought to address this issue by creating a the Wikipedia Events Coreference (WEC) dataset. Applying our model to the WEC dataset, we found that our results surpass their benchmark by large margins (CoNLL F1 of 89.3 versus 62.3 (Eirew et al., 2021)). We plan to discuss these results in further detail in future work.
4.6 TextRank
A limitation of the current work is its specificity to formal text (i.e. news articles, Wikipedia articles). Given the importance of the headlines to our model, we also conducted experiments using the TextRank algorithm (Mihalcea and Tarau, 2004) to extract sentences that best summarize the content of the article instead of using the first two. We expect this method to be more applicable to less formal settings. We embedded each sentence in the document using SBERT and select the top two. On average we found that the headlines heuristic provided a 4.6 and 3.7 CoNLL F1 gain on on event and entity coreference resolution respectively (with singletons) over the TextRank extracted contexts (for detailed metrics see Table 8 in Appendix A). This is expected in the ECB+ context as the TextRank algorithm selects noisier sentences as compared to article headlines.
In this paper, we proposed a new model for within- and cross-document coreference resolution. We demonstrated that contrastive learning approaches are effective at learning representations for coreference resolution. We evaluated our model on gold topics and at the corpus level of the  $\mathrm{ECB + }$  corpus with and without singleton mentions—and found that our approach surpasses current state-of-the-art methods by large margins. We also evaluated our models with an initial document clustering method and found that our model was competitive with earlier works. We presented extensive ablations of our model and discussed limitations of our work including model size, training time, application to formal text domains (i.e. news articles and Wikipedia), and use of agglomerative clustering to generate final coreference clusters. Interesting directions for future work would be testing the TextRank algorithm
in less formal contexts (i.e. beyond news articles and Wikipedia articles), investigating higher-order tuples (e.g. triplets) to speed up model convergence, and extending our work to predicted mentions as opposed to gold mentions as has been done by others (Cattan et al., 2021a,b).
The authors thank the anonymous reviewers for their advice and comments.
In this work, we used the ECB+ corpus (Cybulska and Vossen, 2014) which consists of news articles from the open domain. Our use was consistent with the intended use of the dataset. Our model does not contain any intentional biases. As discussed in §3.4, §4.4, we ran our experiments on a single p3dn.24xlarge with 8 V100 32GB GPUs. Model training and inference was relatively short and does not present ethical issues.
Emily Allaway, Shuai Wang, and Miguel Ballesteros. 2021. Sequential cross-document coreference resolution. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 4659-4671, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Saliha Azzam, Kevin Humphreys, and Robert Gaizauskas. 1999. Using coreference chains for text summarization. In Coreference and Its Applications.
A. Bagga and B. Baldwin. 1998. Algorithms for scoring coreference chains. In The first international conference on language resources and evaluation workshop on linguistics coreference, volume 1, pages 563-566.
Shany Barhom, Vered Shwartz, Alon Eirew, Michael Bugert, Nils Reimers, and Ido Dagan. 2019. Revisiting joint modeling of cross-document entity and event coreference resolution. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4179-4189, Florence, Italy. Association for Computational Linguistics.
Cosmin Bejan and Sanda Harabagiu. 2010. Unsupervised event coreference resolution with rich linguistic features. In Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics, pages 1412-1422, Uppsala, Sweden. Association for Computational Linguistics.
Jane Bromley, Isabelle Guyon, Yann LeCun, Eduard Säckinger, and Roopak Shah. 1993. Signature verification using a "siamese" time delay neural network. In Proceedings of the 6th International Conference on Neural Information Processing Systems, NIPS'93, page 737-744, San Francisco, CA, USA. Morgan Kaufmann Publishers Inc.
Avi Caciularu, Arman Cohan, Iz Beltagy, Matthew Pe- ters, Arie Cattan, and Ido Dagan. 2021. CDLM: Cross-document language modeling. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2648-2662, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2020. Streamlining cross-document coreference resolution: Evaluation and modeling.
Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2021a. Cross-document coreference resolution over predicted mentions. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 5100-5107, Online. Association for Computational Linguistics.
Arie Cattan, Alon Eirew, Gabriel Stanovsky, Mandar Joshi, and Ido Dagan. 2021b. Realistic evaluation principles for cross-document coreference resolution. In Proceedings of *SEM 2021: The Tenth Joint Conference on Lexical and Computational Semantics*, pages 143–151, Online. Association for Computational Linguistics.
S. Chopra, R. Hadsell, and Y. LeCun. 2005. Learning a similarity metric discriminatively, with application to face verification. In 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05), volume 1, pages 539-546 vol. 1.
Andres Cremisini and Mark Finlayson. 2020. New insights into cross-document event coreference: Systematic comparison and a simplified approach. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pages 1-10, Online. Association for Computational Linguistics.
Agata Cybulska and P. Vossen. 2014. Using a sledgehammer to crack a nut? lexical diversity and event coreference resolution. In LREC.
Agata Cybulska and Piek Vossen. 2015. Translating granularity of event slots into features for event coreference resolution. In Proceedings of the The 3rd Workshop on EVENTS: Definition, Detection, Coreference, and Representation, pages 1-10, Denver, Colorado. Association for Computational Linguistics.
Bhuwan Dhingra, Qiao Jin, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2018. Neural models for reasoning over multiple mentions using coreference. In Proceedings of the 2018 Conference of
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 42-48, New Orleans, Louisiana. Association for Computational Linguistics.
Alon Eirew, Arie Cattan, and Ido Dagan. 2021. WEC: Deriving a large-scale cross-document event coreference dataset from Wikipedia. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2498-2510, Online. Association for Computational Linguistics.
Tobias Falke, Christian M. Meyer, and Iryna Gurevych. 2017. Concept-map-based multi-document summarization using concept coreference resolution and global importance optimization. In Proceedings of the Eighth International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 801-811, Taipei, Taiwan. Asian Federation of Natural Language Processing.
R. Hadsell, S. Chopra, and Y. LeCun. 2006. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'06), volume 2, pages 1735-1742.
Yin Jou Huang, Jing Lu, Sadao Kurohashi, and Vincent Ng. 2019. Improving event coreference resolution by learning argument compatibility from unlabeled data. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 785-795, Minneapolis, Minnesota. Association for Computational Linguistics.
Kian Kenyon-Dean, Jackie Chi Kit Cheung, and Doina Precup. 2018. Resolving event coreference with supervised representation learning and clustering-oriented regularization. In Proceedings of the Seventh Joint Conference on Lexical and Computational Semantics, pages 1-10, New Orleans, Louisiana. Association for Computational Linguistics.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E Gonzalez, and Ion Stoica. 2018. Tune: A research platform for distributed model selection and training. arXiv preprint arXiv:1807.05118.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.
Xiaoqiang Luo. 2005. On coreference resolution performance metrics. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 25-32, Vancouver, British Columbia, Canada. Association for Computational Linguistics.
Rada Mihalcea and Paul Tarau. 2004. TextRank: Bringing order into text. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing, pages 404-411, Barcelona, Spain. Association for Computational Linguistics.
Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632-642, Berlin, Germany. Association for Computational Linguistics.
Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Edward Hovy, Vincent Ng, and Michael Strube. 2014. Scoring coreference partitions of predicted mentions: A reference implementation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30-35, Baltimore, Maryland. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERTnetworks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, Hong Kong, China. Association for Computational Linguistics.
Peter J. Rousseeuw. 1987. Silhouettes: A graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53-65.
Kihyuk Sohn. 2016. Improved deep metric learning with multi-class n-pair loss objective. In Advances in Neural Information Processing Systems, volume 29. Curran Associates, Inc.
Shyam Upadhyay, Nitish Gupta, Christos Christodoulopoulos, and Dan Roth. 2016. Revisiting the evaluation for cross document event coreference. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 1949-1958, Osaka, Japan. The COLING 2016 Organizing Committee.
Jose L. Vicedo and Antonio Ferrandez. 2000. Importance of pronominal anaphora resolution in question answering systems. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics, pages 555-562, Hong Kong. Association for Computational Linguistics.
Marc Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model-theoretic coreference scoring scheme. In Proceedings of the 6th Conference on Message Understanding, MUC6 '95, page 45-52, USA. Association for Computational Linguistics.
Bishan Yang, Claire Cardie, and Peter Frazier. 2015. A hierarchical distance-dependent bayesian model for event coreference resolution. Transactions of the Association for Computational Linguistics, 3(0):517-528.
Xiaodong Yu, Wenpeng Yin, and Dan Roth. 2020. Paired representation learning for event and entity coreference.
Dmitry Zelenko, Chinatsu Aone, and Jason Tibbetts. 2004. Coreference resolution for information extraction. In Proceedings of the Conference on Reference Resolution and Its Applications, pages 24-31, Barcelona, Spain. Association for Computational Linguistics.
Yutao Zeng, Xiaolong Jin, Saiping Guan, Jiafeng Guo, and Xueqi Cheng. 2020. Event coreference resolution with their paraphrases and argument-aware embeddings. In Proceedings of the 28th International Conference on Computational Linguistics, pages 3084-3094, Barcelona, Spain (Online). International Committee on Computational Linguistics.
A Detailed Metrics
Below we give detailed metrics with singletons.
MUC	\(B^3\)	CEAF-e	LEA	CoNLL
R	P	F1	R	P	F1	R	P	F1	R	P	F1	F1
Events	Baseline	72.5	81.1	76.6	69.6	87.4	77.5	77.9	69	73.2	55.63	72.9	63.1	75.7
Zeng et al. (2020)	85.6	89.3	87.5	77.6	89.7	83.2	84.5	80.1	82.3	-	-	-	84.3
Yu et al. (2020)	88.1	85.1	86.6	86.1	84.7	85.4	79.6	83.1	81.3	-	-	-	84.4
Caciularu et al. (2021)	87.1	89.2	88.1	84.9	87.9	86.4	83.3	81.2	82.2	76.7	77.2	76.9	85.6
Cattan et al. (2021a)	85.1	81.9	83.5	82.1	82.7	82.4	75.2	78.9	77	68.8	72	70.4	81
Allaway et al. (2021)	81.7	82.8	82.2	80.8	81.5	81.1	79.8	78.4	79.1	-	-	-	80.8
Ours
-RoBERTa-large	87.9	83.4	85.6	86.2	83.4	84.8	76.9	82.4	79.6	74.1	74.2	74.1	83.3
-RoBERTa-base	83.6	84.5	84.0	78.9	86.1	82.4	79.5	78.5	79.0	67.1	75.8	71.2	81.8
-BERT-large	82.9	82.7	82.8	81.3	83.4	82.3	77.8	78.0	77.9	68.9	72.5	70.6	81.0
-BERT-base	80.3	79.3	79.8	78.0	80.9	79.4	73.8	75.0	74.4	63.4	68.8	66.0	77.9
-RoBERTa-large + TextRank	80.0	83.6	81.8	76.9	86.4	81.4	78.6	74.7	76.6	64.1	74.3	68.8	79.9
Entities	Baseline	58.7	88.6	70.7	46.2	93.1	61.7	79.7	44.2	56.9	35.6	68.2	46.8	63.1
Caciularu et al. (2021)	88.1	91.8	89.9	82.5	81.7	82.1	81.2	72.9	76.8	76.4	73	74.7	82.9
Cattan et al. (2021a)	85.7	81.7	83.6	70.7	74.8	72.7	59.3	67.4	63.1	56.8	65.8	61	73.1
Allaway et al. (2021)	83.9	84.7	84.3	74.5	70.5	72.4	70	68.1	69.2	-	-	-	75.3
Ours
-RoBERTa-large	83.1	91.6	87.1	72.2	90.4	80.3	81.1	66.5	73.1	63.7	79.3	70.6	80.2
-RoBERTa-base	77.2	91.1	83.6	61.6	92.8	74.1	81.0	59.4	68.5	52.2	79.2	62.9	75.4
-BERT-large	72.8	90.7	80.8	58.1	92.7	71.4	81.8	55.6	66.2	49.0	76.6	59.7	72.8
-BERT-base	69.9	'88.7	78.2	55.5	90.9	68.9	78.5	52.2	62.7	45.0	72.3	55.5	69.9
-RoBERTa-large + TextRank	75.6	91.2	82.7	59.1	93.2	72.3	80.8	57.4	67.1	49.6	78.9	60.9	74.1
Table 8: Detailed results comparing methods utilizing contextual embedding models and their performance on the  $\mathrm{ECB + }$  test corpus using predicted topic clusters. Note that the systems of Zeng et al. (2020); Yu et al. (2020); Caciularu et al. (2021) require significantly more resources than the others ( $n^2$  versus  $n$  transformer computations). Finally, Allaway et al. (2021) uses a BERT-base model and a sequential clustering algorithm whereas ours and Cattan et al. (2020) utilize RoBERTa-large models and an agglomerative clustering algorithm.
MUC	\( B^3 \)	CEAF-e	LEA	CoNLL
R	P	F1	R	P	F1	R	P	F1	R	P	F1	F1
Events	Gold Topics	Baseline	72.9	72.4	72.7	69.7	73.5	71.5	71.1	71.7	71.4	53.5	59.2	56.1	71.9
Cattan et al. (2021b)	80.1	76.3	78.1	77.4	71.7	74.5	73.1	77.8	75.4	62.9	59.1	61	76
Ours	87.8	82.9	85.3	86.5	83.1	84.8	76.9	82.8	79.7	74.4	74.0	74.2	83.3
Corpus	Baseline	72.9	60.5	66.1	69.7	56.4	62.4	51.5	68.6	58.8	45.3	42.6	43.9	62.4
Kenyon-Dean et al. \( (2018)^† \)	67	71	69	71	67	69	71	67	69	-	-	-	69
Ours	86.4	74.9	80.2	85.3	67.9	75.6	65.3	80.1	71.9	68.3	57.5	62.4	75.9
Entities	Gold Topics	Baseline	61.6	85.9	71.8	48.6	89	62.9	76.7	45.9	57.4	37.3	65.5	47.5	64
Cattan et al. (2021a)	-	-	-	-	-	-	-	-	-	-	-	-	70.9
Ours	84.5	90.1	87.2	79.3	86.6	82.8	78.7	68.6	73.3	70.3	75.7	72.9	81.1
Corpus	Baseline	61.9	77.5	68.8	48.7	79.6	60.4	68.2	46.1	55	35.2	57.8	43.7	61.4
Ours	83.9	86.6	85.2	78.5	82.7	80.5	73.0	67.9	70.4	67.8	71.7	69.7	78.7