Are Language Models Robust Coreference Resolvers?

Authors: nghia t le
alan  ritter
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 04:04:59

Abstract: Recent work on extending coreference resolution across domains and languages relies on annotated data in both the target domain and language. At the same time, pre-trained large language models (LMs) have been reported to exhibit strong zero- and few-shot learning abilities across a wide range of NLP tasks. However, prior work mostly studied this ability using artificial sentence-level datasets such as the Winograd Schema Challenge. In this paper, we assess the feasibility of prompt-based coreference resolution by evaluating instruction-tuned language models on difficult, linguistically-complex coreference benchmarks (e.g., CoNLL-2012). We show that prompting for coreference can outperform current unsupervised coreference systems, although this approach appears to be reliant on high-quality mention detectors. Further investigations reveal that instruction-tuned LMs generalize surprisingly well across domains, languages, and time periods; yet continued fine-tuning of neural models should still be preferred if small amounts of annotated examples are available.

Main paper: 
 introduction: 
 Entity coreference resolution aims to find all spans within an input text that refer to the same entity. As an important information extraction sub-task, coreference resolution has received considerable attention from the NLP community over the years, with recent progress driven mostly by neural coreference models (Lee et al., 2017; Wu et al., 2020; Joshi et al., 2020) . There has also been an increasing interest in the generalization of coreference systems to domains and languages beyond the popular CoNLL-2012 benchmark (Xia & Van Durme, 2021; Bohnet et al., 2022) . Most work on extending coreference resolution to new domains and languages relies on target language annotated data in the targeted domain, however the amount of labeled data needed to cover every possible domain in all languages is prohibitively expensive. Meanwhile, unsupervised (Haghighi & Klein, 2010) and few-shot (Le et al., 2022) coreference resolution has received less attention, despite the fact that learning with less labels is desirable when adapting to new languages or domains.Concurrently, there has been a great deal of progress on zero-and few-shot learning using pre-trained language models (LMs) (Ouyang et al., 2022; Touvron et al., 2023) . Attempts have been made at evaluating pre-trained LMs' coreference abilities under zero-and few-shot settings: Brown et al. (2020) demonstrated that prompting GPT-3 can resolve coreference on the Winograd Schema Challenges (WSC), Yang et al. (2022) showed that coreference resolution was a challenging task for GPT-2 when prompted with multiple-choice templates, and Agrawal et al. (2022) successfully reframed clinical pronoun resolution as span generation. While these studies reveal some evidence of the coreference abilities in large LMs, they either use methods that fail to beat reasonable baselines, or evaluate on sentencelevel, non-standard coreference datasets that are designed to benchmark the capabilities of LLMs, rather than providing an accurate evaluation of models' coreference capabilities in a realistic setting. In contrast, the traditional dataset for coreference resolution, CoNLL-2012/OntoNotes, contains real-world document-level examples with complex linguistic Figure 1 : An example of coreference resolution with LMs prompting. Here we show two prompt templates experimented in this work: Question-Answer and Document templates. In the QA template, the language model generates the answer when given a passage and an open-ended wh-question (Ouyang et al., 2022) . In contrast, the document template marks the candidate mentions and asks the LM to annotate the cluster IDs for each mention directly within the text (represented by different colors). Both templates require a mention detector to generate candidate mentions.( §A.1), we found that prompting GPT-4 with a QA template struggled to compete with Stanford's deterministic coreference system (Lee et al., 2013) , achieving 67 F 1 when comparing to 72 F 1 from Lee et al. (2013) . We also experimented with an alternative document-level template that can elicit more coreference links than the traditional QA template, achieving 86 F 1 (Table A.1) . In this template, the mentions of the input text are first marked with special tokens indicating a span to annotate (e.g., Mr. Clinton → [Mr. Clinton](#)). The LM is then given instructions to annotate this marked span with the cluster ID, (e.g., [Mr. Clinton](#) → [Mr. Clinton](#cluster 1)). Given strong results over the QA template, we used this document template for all subsequent experiments. 
conll-2012 experiments: 
 We investigate the coreference abilities of large LMs on the CoNLL-2012 benchmark (Pradhan et al., 2012) and compare these models against existing supervised and unsupervised baselines. We found that GPT model family (OpenAI, 2023) yield competitive results with previous unsupervised and rule-based models, while significantly outperforming them when gold mentions are provided.The English OntoNotes 5.0 dataset (Weischedel et al., 2011; Pradhan et al., 2012) is traditionally used to evaluate coreference systems. This dataset spans seven distinct genres such as news, telephone conversations, and religious text. We follow the standard train-dev-test splits from previous work and report CoNLL F 1 , an average of three coreference-based metrics MUC, B 3 , and CEAF ϕ 4 .We mainly consider Stanford's deterministic resolver, which is referred to as dcoref (Lee et al., 2013) . This coreference resolver consists of multiple sieves, with each sieve being a set of handcrafted rules that filters out mentions and ordered from highest to lowest precision to minimize cascading errors.2 For supervised systems, we compare to coref-mt5 (Bohnet et al., 2022) and coref-T0 (Zhang et al., 2023) , two text-to-text approaches based on seq2seq models. For unsupervised baselines, we include results from weak-SpanBERT (Stolfo et al., 2022) , a system that trained a SpanBERT-based architecture on dcoref coreference predictions.We use Llama 2 model family (Touvron et al., 2023) as the primary open-sourced language models, namely two instruction-tuned versions of base Llama-2, Llama-2-Chat and CodeLlama (Rozière et al., 2023) . To avoid hallucinations, we constrain the generations as follows: for each given mention, we ask the model to generate the cluster ID. We then update the input sequence by appending the generated ID with the text segment between the current and the next mention. The process is repeated until all the mentions in the document are annotated.We also report performance on the most recent OpenAI language models: the instruction-tuned 175B InstructGPT (text-davinci-003) (Ouyang et al., 2022) , ChatGPT (gpt-35-turbo), and GPT-4 (OpenAI, 2023) . Due to the cost of running these models, we generate outputs using unconstrained greedy decoding with a single generation per input document. For all our official experiments using InstructGPT, ChatGPT, GPT-4, we generated approximately 18 million tokens, 15 million tokens, 1 million tokens, respectively. All GPT experiments were conducted before December 2023.We report results under two settings: predicted mentions, where only raw text is provided as input, and gold mentions, where the gold mention boundaries are provided as input. To obtain predicted mentions, we use the mentions output by dcoref as input into language model prompts. (Koehn, 2004) .Table 1 shows the results between different coreference systems. We note that prompting InstructGPT and GPT-4 outperforms dcoref for predicted mentions, with performance gaps increasing for gold mentions. However, this approach still considerably underperforms fully supervised systems. While all Llama-2 model variants underperform dcoref baseline, we note that CodeLlama significantly outperforms Llama-2-Chat. CodeLlama 7B even matches the performance of Llama-2-Chat 70B.To further understand the strengths and weaknesses of instruction-tuned LMs for coreference, we break down the results according to different resolution classes (Lu & Ng, 2020) . Specifically, for each coarse-grained mention class (named entity, pronoun, nominal), we compute the resolution accuracy, which is the percentage of anaphors correctly linked to an antecedent (Figure 2 ). We observe that InstructGPT does particularly well in pronoun resolution, corroborating previous work (Agrawal et al., 2022) . It struggles more for named entities and the difficult nominal resolution. However, InstructGPT still remains competitive with dcoref for these classes, with the gaps increasing when gold mentions are provided. In particular, InstructGPT (and CodeLlama in gold mention setting) outperforms dcoref on challenging nominal phrases (Figure 2 ). Table 2 compares two finetuned Llama-2 models with two aforementioned supervised systems, coref-T0 11B parameters and SpanBERT+e2e. We note that finetuned Llama-2 achieves competitive results in this setting, surpassing SpanBERT+e2e and approaching coref-T0 despite having simpler text formats and generation procedures (e.g., no constrained beam search, no task-specific decoding actions). This indicates the feasibility of fine-tuning coreference with Llama-2 using our simple prompt template.While prompting LMs can be competitive with previous coreference systems, the quality of candidate mentions has a considerable effect on the final performance. We quantify the importance of high-quality Mention Detection (MD) by measuring the models' performance when inputting candidate mention sets generated by different mention detectors. Furthermore, we analyze the performance of prompting LMs for mentions with a simple template that outputs a list of named entities, pronouns, and nominal phrases, given an input text. We discuss these results below. InstructGPT consistently outperforms dcoref as MD performance increases (Figure 3 , left). In general, coreference performances of all models improve as mention detection score increases. This is not surprising, as it has been similarly reported in previous work studying mention detection of neural coreference resolution systems (Lu & Ng, 2020) . We further observe that CodeLlama underperforms while ChatGPT performs comparable to dcoref baseline. Nonetheless, we note that InstructGPT again consistently outperforms dcoref, regardless of MD performance. InstructGPT also exhibits some evidence of long-range ability when correctly resolving it to allegations of infidelity.shows that InstructGPT and GPT-4 perform much worse than dcoref. Further analysis by mention types shows they particularly struggle to recall nominal mentions. A qualitative example in Table 3 demonstrates that while InstructGPT was able to recover a considerable portion of named entities and pronouns, it also made numerous errors, including span errors, extra entities, and missing mentions (Kummerfeld & Klein, 2013) .Given that what constitutes a mention can depend heavily on the annotation guidelines of specific datasets and domains, it may be challenging to ask a MD system to predict mentions without any labeled examples. Since Mention Detection plays a crucial role in coreference resolution (Wu & Gardner, 2021) as well as its generalizability to different domains, high-quality mention detection appears to be a pre-requisite for prompt-based coreference resolution. Fortunately, however, mention annotation has been shown to be much less costly than annotating full coreference chains (Gandhi et al., 2022) . 
generalization beyond ontonotes: 
 Although supervised neural models achieve superior results for coreference, they are also known to struggle when generalizing across domains, sometimes even underperforming rule-based systems (Moosavi & Strube, 2017) . As such, recent research in coreference largely focus on the generalization ability of neural models beyond the OntoNotes dataset (Xia & Van Durme, 2021; Gandhi et al., 2022) . Given that large LMs are pre-trained on lots of general-purpose data and are not optimized for a single coreference dataset, it seems plausible that instruction-tuned language models might also be effective across diverse texts. To explore this question, we examine how well instruction-tuned LMs generalize to different domains ( §4.1), languages ( §4.2), and time periods ( §4.3). We mainly report results for InstructGPT and ChatGPT, given its competitive performance on OntoNotes while being less expensive than GPT-4 ( §3). The diverse coreference datasets considered in this analysis are given in Table 12 . Since mention detection has been shown to be fairly challenging ( §3.3), we evaluate the experiments in this section using gold mentions. dcoref was not trained on any corpus. Overall, InstructGPT exhibits strong generalization results when using out-of-the-box.To study the robustness of our approach across domains, we use the datasets benchmarked in Toshniwal et al. (2021) due to the diversity in genres (news, Wikipedia, conversations), document lengths (long vs. short), and annotation guidelines (singletons vs. non-singletons).For evaluation, we follow the annotation schema of the corresponding dataset (i.e., if the dataset contains singletons, then we also output singletons). Similar to previous work in coreference domain adaptation (Xia & Van Durme, 2021; Toshniwal et al., 2021) , we explore different systems where different types of source and target training data are available. Specifically, in addition to dcoref as in §3, we include the trained models TRANSFER-ON (Xia & Van Durme, 2021) and longdoc-PC (Toshniwal et al., 2021) , which were respectively trained on the train set of OntoNotes en (2,802 annotated documents of newswire and religious texts) and PreCo (36,120 documents of reading comprehension examinations, collected in Chen et al. ( 2018)). TRANSFER-ON was then further finetuned on 10 labeled documents from the target domains. Additionally, we include the pretrained encoder SpanBERT (Xia & Van Durme, 2021) as a fine-tuning baseline (on a small amount of annotated data), where a pretrained SpanBERT encoder was not trained on a large source corpus and instead directly finetuned on 10 target documents.3 InstructGPT is robust for coreference domain adapation. Table 4 shows the coreference domain generalization for various systems. While InstructGPT is competitive with longdoc-PC, it still trails behind TRANSFER-ON considerably. This indicates that transfer learning is still a preferred method for coreference domain adaptation, particularly when a large corpus of training data and a few annotated documents in the target domain are available.On the other hand, when compared to models that were not trained on source coreference datasets such as dcoref and SpanBERT, InstructGPT outperforms them by a significant margin. This demonstrates the robustness of InstructGPT for coreference domain adaptation when using as a black-box model.To test the generalization of InstructGPT on resolving coreference across multiple languages, we experimented with Chinese and Arabic portions of OntoNotes and the multilingual coreference SemEval-2010 dataset (Recasens et al., 2010) . A notable difference between OntoNotes and SemEval-2010 is the annotations of singletons, which has led to different evaluation methods for SemEval-2010. We follow the evaluation setting of previous work for each of the evaluated languages: excluding singletons from both predicted and evaluation clusters for Chinese and Arabic, while excluding singletons from predicted set but keeping them in evaluation sets for other languages. We refer to Section 5 of Bohnet et al. (2022) for more discussion on this.Similar to §4.1, we compare InstructGPT with neural transfer-learning models from Xia & Van Durme (2021), TRANSFER-EN and XLM-R. Both use a pretrained XLM-RoBERTa-large encoder fine-tuned with 10 documents from the target language. We note that TRANSFER-EN was previously trained on English OntoNotes before continuing training on the target language, which makes it a stronger model than XLM-R. TRANSFER-EN and XLM-R correspond to TRANSFER-ON and SpanBERT from §4.1, respectively, with the only difference being the pretrained encoder (XLM-R vs. SpanBERT). InstructGPT can also effectively resolve coreference across languages.From Table 5 , we observe similar conclusions to §4.1: continued learning using a large source corpus with a handful of annotated examples from target languages still performs the best. Nonetheless, InstructGPT was able to outperform XLM-R across all languages, and is even on par with TRANSFER-EN for Chinese and Dutch. This result indicates the importance of a source English coreference corpus for continued learning.An interesting dimension to analyze the robustness of coreference generalization is temporal changes (Agarwal & Nenkova, 2022; Liu & Ritter, 2023) , since having coreference systems that can generalize beyond datasets that were created over a decade ago (e.g., OntoNotes) can be beneficial. (Joshi et al., 2020) , which was fine-tuned on the in-domain OntoNotes train set, to obtain silver annotations for all three datasets. We then evaluate the models on these silver annotations, with mentions given as before. Further details on how we sampled and annotated these datasets are presented in §A.4. Prompting instruction-tuned LMs is robust to temporal changes. Table 6 shows the results. We first observe a decrease when moving from gold to silver annotations for all models. More importantly, we see more degradation and variance in performance of dcoref for the different temporal datasets, whereas the variance is less pronounced for InstructGPT and ChatGPT. While CodeLlama-34B underperforms dcoref baseline, it also observes less variance when evaluated on different temporal datasets. 2022) demonstrates that adapting mention annotations to new domains instead of the entire coreference chains is more cost-efficient while also improving domain adaptation performance. In contrast to the above work, we propose to prompt general-purpose language models for coreference resolution and show promising generalization capabilities across domains. Our findings also align with contemporaneous work Nori et al. (2023) , which shows that prompting can unlock specialized capabilities in general-purpose LLMs.Research in coreference resolution has been dominated by neural span-based models that score coreference links between spans (Lee et al., 2017; Joshi et al., 2020) . Recently, a new paradigm for coreference starts to emerge: formulating coreference resolution as conditional text generation (Liu et al., 2022; Bohnet et al., 2022; Zhang et al., 2023) . Both Liu et al. (2022) and Bohnet et al. (2022) fine-tuned T5-based models on sequences of structured-building actions, with the former achieving competitive results for structured prediction tasks and the latter achieving SOTA results for coreference resolution. Zhang et al. (2023) finetuned T0 models on a simpler text sequences that directly encode coreference annotations, yet achieved comparable results to Bohnet et al. (2022) . While our work falls into this category, we are interested the intrinsic ability of the language model to resolve coreference, using an autoregressive language model on an instruction-based prompt format. Another coreference annotation framework that is similar to ours is TANL (Paolini et al., 2021) , which also formulates structured prediction as a text generation task. The main difference between our work and TANL is that TANL is evaluated in fully supervised setting using multi-task learning, whereas we evaluated instruction-tuned LLMs without explicit fine-tuning on the CoNLL train set.With the success of zero-shot and few-shot prompting of large language models on various NLP benchmarks, we ask to what extent this success translates to more traditional NLP tasks like coreference resolution. Manning et al. (2020) shows evidence of linguistic abilities in masked LMs, and Blevins et al. (2022) presents a structured prompting approach that achieves strong few-shot results for sequence tagging tasks. For coreference resolution, prior work has mostly focused on few-shot learning for sentence-level, syntactically simple coreference datasets such as Winograd Schema Challenge (Levesque et al., 2012) and for pronoun resolution on clinical data (Agrawal et al., 2022) . 
conclusion: 
 In this paper, we study how well instruction-tuned language models resolve coreference via prompting. We demonstrate the feasibility of this approach on the CoNLL-2012 benchmark, surpassing previous unsupervised systems but still underperforming state-of-the-art supervised models. Interestingly, prompting instruction-tuned LMs appears to generalize well across a wide range of domains, languages, and time periods, particularly if no training examples are given. Nonetheless, it still trails behind continued learning with a large training corpus in the source domain and a handful of annotated examples in the target domain.Even with the surprising effectiveness of language model technologies in various applications, coreference resolution remains a challenging task. In addition, it continues to be an important subtasks of the Information Extraction (IE) pipeline (particularly for documentlevel IE), with usages in applications such as extracting scientific documents, answering queries that require aggregation over a collection of unstructured documents, etc. Given the emerging effectiveness of LMs on a variety of NLP tasks and the importance of adapting coreference models to different domains, we hope this study can help shed some light on future research in both directions. 
 
 Appendix: Because OpenAI GPT models are proprietary models, we do not know whether or not OntoNotes was included in its training data. However, at the time of writing, there is some evidence against OntoNotes data contamination. First, a previous probe that aimes to measure data contamination and memorization of OntoNotes on ChatGPT showed negative results. 4 Second, our experiment in §4.3 includes data sampled after the models' training cutoff date (September 2021), yet still shows a robust F 1 . Finally, the conclusions in this paper still stand regardless of whether or not these models trained on OntoNotes: (1) prompting instruction-tuned LMs is a feasible strategy for coreference resolution, and (2) although this approach has unique strengths and weaknesses, it is robust across many domains, languages, and time periods.Llama-2-70B (Gan et al., 2024) 36.1 GPT-4 (Gan et al., 2024) 51.4Llama-2-70B (Ours) 46.3 GPT-4 (Ours) 84.7Table 8 : CODI-CRAC 2022 results.detector from dcoref as well as the prompting of InstructGPT for MD using template in Table 11 . In addition, to see the effects of having high-quality mentions on dcoref and InstructGPT, we also consider outputs from SpanBERT-large trained on OntoNotes train set (Joshi et al., 2020) and a NER tagger with xlm-roberta-large (Conneau et al., 2020) trained on BIO labels adapted from OntoNotes annotations. We note that these systems are not directly comparable to each other, since they were trained on different annotatations: SpanBERT-large on full coreference data and xlm-roberta-large on non-nested MD data. The number of mentions per document is measured using the silver annotations from SpanBERT (Joshi et al., 2020) .To sample the appropriate data for this experiment, we start with the Wall Street Journal sections of the RealNews (Zellers et al., 2019) and OntoNotes dev set. We used SpanBERT (Joshi et al., 2020) to label all 56 WSJ articles from OntoNotes to obtain WSJ-1989 (CoNLL F 1 using SpanBERT on WSJ-1989 is shown on Table 10 ). To create WSJ-2019, we first labeled all 191 WSJ articles from RealNews using SpanBERT as above. We then sampled 56 articles using stratified sampling based on two features: document length and number of mentions per document. Specifically, we partitioned the WSJ RealNews articles into bins based on document lengths (bin size = 500 tokens), and for each document-length bin we further partitioned based on the number of mentions (mention size = 50). We then sampled the appropriate number of documents (i.e., the number of WSJ-1989 documents in each partition) for each bin to obtain WSJ-2019. For WSJ-2023, we randomly collected 56 articles from the WSJ website dated between May and June 2023 based on document lengths and topics. The distributions of three datasets are shown in Figure 4 . In the following text, list all named entities, pronouns, and nominal noun phrases according to the OntoNotes conventions. Input: In the summer of 2005, a picture that people have long been looking forward to started emerging with frequency in various major Hong Kong media. With their unique charm, these well-known cartoon images once again caused Hong Kong to be a focus of worldwide attention. The world's fifth Disney park will soon open to the public here. The most important thing about Disney is that it is a global brand.Named Entities: Hong Kong Pronouns: their, it, many, its, that, its, this Nominal Noun Phrases: these well-known cartoon images, the world's fifth Disney park Our code is available at https://github.com/nle18/coref-llmshttps://nlp.stanford.edu/software/dcoref.htmlFigure1ofXia & Van Durme (2021). Models summary detailed in Table13.https://hitz-zentroa.github.io/lm-contamination/blog/ 


Decision: The paper studies LLMs' ability to perform classic coreference resolution as defined by OntoNotes. The empirical process is solid, and the evidence supports the claims. Having said that, the research question seems less interesting in 2024; as reviewers P4nJ and 3vsC note, it is probably not going to significantly impact future research in language modeling.

[comments from the PCs] We encourage the authors to consider the framing/discussion/conclusion of this paper to show its pertinance to contemporary research and future challenges. This can enhance the paper's reception.

Reviews: review: paper_summary: paper_summary: This paper evaluates LLMs in their ability to perform coreference resolution using a traditional NLP framework of evaluation and dataset. For coreference resolution, the authors argues that prior work has mostly focused on few-shot learning for sentence-level, syntactically simple coreference datasets such as Winograd Schema Challenge and for pronoun resolution on clinical data. This paper therefore bridges a gap between the LLM literature and coreference resolution literature.  

Quality: this paper is overall high quality. The research question is well motivated and bridges a gap between LLMs and traditional NLP based coreference systems. The experiments are well designed with appropriate data benchmarks. The selection of additional research questions is also comprehensive and relevant. 

Clarity: this paper is very clear.

Originality: this paper provides a grounded and principled framework to compare prompt-based LLMs and traditional task based (deep learning based and rule based) linguistic models on a well established NLP task. It bridges a gap that shows it's originality. 

Significance: medium-high please see above
strength_weakness: strength_weakness: n/a
questions: questions: n/a
paper_summary: paper_summary: This paper reports on experiments evaluating prompt-based approaches to coreference. The approach is very systematic, comparing a number of open-source and proprietary LLMs with unsupervised and supervised models. The main evaluation in Section 3 is carried out on the standard CONLL2012 dataset, but in Section 4 other English datasets are considered, as well as datasets in other languages including  Arabic and Chinese Ontonotes and the SEMEVAL 2010 datasets. The main results for English are that the performance of LLMs on predicted mentions is much lower than that of state-of-the-art coref systems, but on gold mentions is comparable or better. Fine-tuning the LLMs improves the results. On other datasets, InstructGPT performs better out-of-the-box than supervised models trained on other domains, but not than supervised models fine-tuned on a specific domain. The same happens with other languages.

The main contributions / results are:
1. Two prompting strategies are proposed and tested, Question-Answering based and 'Document Template' based (aka document annotation)
2. The very systematic evaluation will be useful to others.
strength_weakness: strength_weakness: 1. The main weakness of this paper is that it is heavily experimental, not really introducing new ideas but focusing primarily on the evaluation side of things. (Which doesn't make it less useful.)

2. My main concern is the extent to which the results in Section 3 especially are affected by data contamination, given that Ontonotes-2012, Semeval-2010, etc have been around for over than years. But the results e.g., in Section 4.3 make it sound like it can't all just be data contamination.
questions: questions: Questions:

1. Why didn't you use some of the datasets that appeared more recently? The only recent-ish dataset is LitBank which came out in 2020 - e.g., what about the CODI-CRAC 2022 dataset, which would have also covered more dialogue? Or GUM v. 10?

Suggestions:

1. If accepted, the authors should include a comparison of their results with those of Gan et al 2024, which in my view is the most related work
paper_summary: paper_summary: The paper presents coref experiments using prompted LLMs. Two prompts are considered, one QA-based and the other framed as clustering all given mentions in the document. The latter is found to be clearly more performant. LLMs are given either gold or predicted mentions (using an off-the-shelf manual coref system dcoref); they're not competitive at mention finding themselves, which is likely due to the specific nature of mentions in OntoNotes and other coref datasets. In-context examples are not used (i.e., it's zero-shot) because they're not helpful for the document-level prompt.

The main results are: (1) when dcoref-predicted mentions are used (document-level prompt, zero-shot), strong LLMs like InstructGPT and GPT-4 beat an unsupervised approach weak-SpanBERT which is a SpanBERT trained on the output of dcoref, but fall behind supervised systems, (2) when gold mentions are used, they perform better than supervised and other baselines that also use gold mentions. The paper also performs various transfer and multilingual coref experiments, and find that LLMs generalize well.
strength_weakness: strength_weakness: - The prompting results are, as acknowledged in the paper and not surprisingly, not amazing compared to SOTA results. This furthermore has various downsides, such as a dependence on a third-party mention finder. Other than for checking off a box on the list of things to verify for LLMs, it's a little trick to extract a lot of lasting technical value from this work.
- The finding that gold mentions allow for amazing coref peformance is known. A supervised system can get ~95 F1 if given gold mentions, this result can be found in [1] but I think the general fact that the coref performance bottleneck is mention finding is a bit of an open secret. 
- It seems weak-SpanBERT is the only unsupervised (or distantly supervised) baseline. I don't have strong suggestions for specific additional baselines, but having dcoref and dcoref-based SpanBERT as the only non-LLM non-supervised baselines seems a bit insufficient to fully contextualize the performance of LLM prompting for coref.

[1] Seq2seq is All You Need for Coreference Resolution (Zhang et al., 2023)
paper_summary: paper_summary: The paper explores entity coreference resolution, focusing on leveraging LLMs through prompting. Results indicate promising potential, although the method still falls behind supervised models and requires a robust mention detector. Nonetheless, the study demonstrates the generalization capabilities of LLMs across different contexts.
strength_weakness: strength_weakness: 1. While OntoNotes has been a valuable resource for coreference research, its age may limit the generalizability of the findings. Consideration of experiments on newer datasets with more complex mentions and coreference examples would enhance the robustness of the study's conclusions.
2. The study's alignment with current practices in LLMs usage, without significant advancements or novel contributions, may limit its impact on future research.

Comments: comment: If extra experiments are carried out I'd be happy for the paper to be accepted
Good rebuttal, if those modifications are carried out I'd be happy for the paper to be accepted