Efficient and Accurate Coreference Resolution Defying Recent Trends (2024)

Abstract: Large autoregressive generative models have emerged as the cornerstone for achieving the highest performance across several Natural Language Processing tasks. However, the urge to attain superior results has, at times, led to the premature replacement of carefully designed task-specific approaches without exhaustive experimentation. The Coreference Resolution task is no exception; all recent stateof-the-art solutions adopt large generative autoregressive models that outperform encoderbased discriminative systems. In this work, we challenge this recent trend by introducing Maverick, a carefully designed – yet simple – pipeline, which enables running a state-ofthe-art Coreference Resolution system within the constraints of an academic budget, outperforming models with up to 13 billion parameters with as few as 500 million parameters. Maverick achieves state-of-the-art performance on the CoNLL-2012 benchmark, training with up to 0.006x the memory resources and obtaining a 170x faster inference compared to previous state-of-the-art systems. We extensively validate the robustness of the Maverick framework with an array of diverse experiments, reporting improvements over prior systems in data-scarce, long-document, and out-of-domain settings. We release our code and models for research purposes at https: //github.com/SapienzaNLP/maverick-coref.

heading: 1 Introduction
text: As one of the core tasks in Natural Language Processing, Coreference Resolution aims to identify and group expressions (called mentions) that refer to the same entity (Karttunen, 1969). Given its crucial role in various downstream tasks, such as Knowledge Graph Construction (Li et al., 2020), Entity Linking (Kundu et al., 2018; Agarwal et al., 2022), Question Answering (Dhingra et al., 2018; Dasigi et al., 2019; Bhattacharjee et al., 2020; Chen and Durrett, 2021), Machine Translation
(Stojanovski and Fraser, 2018; Voita et al., 2018; Ohtani et al., 2019; Yehudai et al., 2023) and Text Summarization (Falke et al., 2017; Pasunuru et al., 2021; Liu et al., 2021), inter alia, there is a pressing need for both high performance and efficiency. However, recent works in Coreference Resolution either explore methods to obtain reasonable performance optimizing time and memory efficiency (Kirstain et al., 2021; Dobrovolskii, 2021; Otmazgin et al., 2022), or strive to improve benchmark scores regardless of the increased computational demand (Bohnet et al., 2023; Zhang et al., 2023).
Efficient solutions usually rely on discriminative formulations, frequently employing the mentionantecedent classification method proposed by Lee et al. (2017). These approaches leverage relatively small encoder-only transformer architectures (Joshi et al., 2020; Beltagy et al., 2020) to encode documents and build on top of them task-specific networks that ensure high speed and efficiency. On the other hand, performance-centered solutions are nowadays dominated by general-purpose large Sequence-to-Sequence models (Liu et al., 2022; Zhang et al., 2023). A notable example of this formulation, and currently the state of the art in Coreference Resolution, is Bohnet et al. (2023), which proposes a transition-based system that incrementally builds clusters of mentions by generating coreference links sentence by sentence in an autoregressive fashion. Although Sequence-to-Sequence solutions achieve remarkable performance, their autoregressive nature and the size of the underlying language models (up to 13B parameters) make them dramatically slower and memory-demanding compared to traditional encoder-only approaches. This not only makes their usage for downstream applications impractical, but also poses a significant barrier to their accessibility for a large number of users operating within an academic budget.
In this work we argue that discriminative encoder-only approaches for Coreference Reso-
13380
lution have not yet expressed their full potential and have been discarded too early in the urge to achieve state-of-the-art performance. In proposing Maverick, we strike an optimal balance between high performance and efficiency, a combination that was missing in previous systems. Our framework enables an encoder-only model to achieve toptier performance while keeping the overall model size less than one-twentieth of the current stateof-the-art system, and training it with academic resources. Moreover, when further reducing the size of the underlying transformer encoder, Maverick performs in the same ballpark as encoder-only efficiency-driven solutions while improving speed and memory consumption. Finally, we propose a novel incremental Coreference Resolution method that, integrated into the Maverick framework, results in a robust architecture for out-of-domain, data-scarce, and long-document settings.

heading: 2 Related Work
text: We now introduce well-established approaches to neural Coreference Resolution. Specifically, we first delve into the details of traditional discriminative solutions, including their incremental variations, and then present the recent paradigm shift for approaches based on large generative architectures.

heading: 2.1 Discriminative models
text: Discriminative approaches tackle the Coreference Resolution task as a classification problem, usually employing encoder-only architectures. The pioneering works of Lee et al. (2017, 2018) introduced the first end-to-end discriminative system for Coreference Resolution, the Coarse-to-Fine model. First, it involves a mention extraction step, in which the spans most likely to be coreference mentions are identified. This is followed by a mentionantecedent classification step where, for each extracted mention, the model searches for its most probable antecedent (i.e., the extracted span that appears before in the text). This pipeline, composed of mention extraction and mention-antecedent classification steps, has been adopted with minor modifications in many subsequent works, that we refer to as Coarse-to-Fine models.
Coarse-to-Fine Models Among the works that build upon the Coarse-to-Fine formulation, Lee et al. (2018), Joshi et al. (2019) and Joshi et al. (2020) experimented with changing the underlying document encoder, utilizing ELMo (Peters et al.,
2018), BERT (Devlin et al., 2019) and SpanBERT (Joshi et al., 2020), respectively, achieving remarkable score improvements on the English OntoNotes (Pradhan et al., 2012). Similarly, Kirstain et al. (2021) introduced s2e-coref that reduces the high memory footprint of SpanBERT by leveraging the LongFormer (Beltagy et al., 2020) sparse-attention mechanism. Based on the same architecture, Otmazgin et al. (2023) analyzed the impact of having multiple experts score different linguistically motivated categories (e.g., pronouns-nouns, nounsnouns, etc.). While the foregoing works have been able to modernize the original Coarse-toFine formulation, training their architectures on the OntoNotes dataset still requires a considerable amount of memory.1 This occurs because they rely on the traditional Coarse-to-Fine pipeline that, as we cover in Section 3.1, has a large memory overhead and is based on manually-set thresholds to regulate memory usage.
Incremental Models Discriminative systems also include incremental techniques. Incremental Coreference Resolution has a strong cognitive grounding: research on the “garden-path” effect shows that humans resolve referring expressions incrementally (Altmann and Steedman, 1988).
A seminal work that proposed an automatic incremental system was that of Webster and Curran (2014), which introduced a clustering approach based on the shift-reduce paradigm. In this formulation, for each mention, a classifier decides whether to SHIFT it into a singleton (i.e., single mention cluster) or to REDUCE it within an existing cluster. The same approach has recently been reintroduced in ICoref (Xia et al., 2020) and longdoc (Toshniwal et al., 2021), which adopted SpanBERT and LongFormer, respectively. In these works the mention extraction step is identical to that of Coarse-to-Fine models. On the other hand, the mention clustering step is performed by using a linear classifier that scores each mention against a vector representation of previously built clusters, in an incremental fashion. This method ensures constant memory usage since cluster representations are updated with a learnable function. In Section 3.2 we present a novel performance-driven incremental method that obtains superior performance and generalization capabilities, in which we adopt a lightweight transformer architecture that retains the mention representations.
1Training those models requires at least 32G of VRAM.

heading: 2.2 Sequence-to-Sequence models
text: Recent state-of-the-art Coreference Resolution systems all employ autoregressive generative approaches. However, an early example of Sequenceto-Sequence model, TANL (Paolini et al., 2021), failed to achieve competitive performance on OntoNotes. The first system to show that the autoregressive formulation was competitive was ASP (Liu et al., 2022), which outperformed encoderonly discriminative approaches. ASP is an autoregressive pointer-based model that generates actions for mention extraction (bracket pairing) and then conditions the next step to generate coreference links. Notably, the breakthrough achieved by ASP is not only due to its formulation but also to its usage of large generative models. Indeed, the success of their approach is strictly correlated with the underlying model size, since, when using models with a comparable number of parameters, the performance is significantly lower than encoder-only approaches. The same occurs in Zhang et al. (2023), a fully-seq2seq approach where a model learns to generate a formatted sequence encoding coreference notation, in which they report a strong positive correlation between performance and model sizes.
Finally, the current state-of-the-art system on the OntoNotes benchmark is held by Link-Append (Bohnet et al., 2023), a transition-based system that incrementally builds clusters exploiting a multipass Sequence-to-Sequence architecture. This approach incrementally maps the mentions in previously coreference-annotated sentences to system actions for the current sentence, using the same shift-reduce incremental paradigm presented in Section 2.1. This method obtains state-of-the-art performance at the cost of using a 13B-parameter model and processing one sentence at a time, drastically increasing the need for computational power. While the foregoing models ensure superior performance compared to previous discriminative approaches, using them for inference is out of reach for many users, not to mention the exorbitant cost of training them from scratch.

heading: 3 Methodology
text: In this section, we present the Maverick framework: we propose replacing the preprocessing and training strategy of Coarse-to-Fine models with a novel pipeline that improves the training and inference efficiency of Coreference Resolution systems. Furthermore, with the Maverick Pipeline, we elim-
inate the dependency on long-standing manuallyset hyperparameters that regulate memory usage. Finally, building on top of our pipeline, we propose three models that adopt a mention-antecedent classification technique, namely Mavericks2e and Maverickmes, and a system that is based upon a novel incremental formulation, Maverickincr.

heading: 3.1 Maverick Pipeline
text: The Maverick Pipeline combines i) a novel mention extraction method, ii) an efficient mention regularization technique, and iii) a new mention pruning strategy.
Mention Extraction When it comes to extracting mentions from a document D, there are different strategies to model the probability that a span contains a mention. Several previous works follow the Coarse-to-Fine formulation presented in Section 2.1, which consists of scoring all the possible spans in D. This entails a quadratic computational cost in relation to the input length, which they mitigate by introducing several pruning techniques.
In this work, we employ a different strategy. We extract coreference mentions by first identifying all the possible starts of a mention, and then, for each start, extracting its possible end. To extract start indices, we first compute the hidden representation (x1, . . . , xn) of the tokens (t1, . . . , tn) ∈ D using a transformer encoder, and then use a fullyconnected layer F to compute the probability for each ti being the start of a mention as:
Fstart(x) = W ′ start(GeLU(Wstartx))
pstart(ti) = σ(Fstart(xi))
with W ′start,Wstart being the learnable parameters, and σ the sigmoid function. For each start of a mention ts, i.e., those tokens having pstart(ts) > 0.5, we then compute the probability of its subsequent tokens tj , with s ≤ j, to be the end of a mention that starts with ts. We follow the same process as that of the mention start classification, but we condition the prediction on the starting token by concatenating the start, xs, and end, xj , hidden representations before the linear classifier:
Fend(x, x ′) = W ′end(GeLU(Wend[x, x ′]))
pend(tj |ts) = σ(Fend(xs, xj)) with W ′end, Wend being learnable parameters. This formulation handles overlapping mentions since,
for each start ts, we can find multiple ends te (i.e., those that have pend(tj |ts) > 0.5).
Previous works already adopted a linear layer to compute start and end mention scores for each possible mention, i.e., s2e-coref (Kirstain et al., 2021), and LingMess (Otmazgin et al., 2023). However, our mention extraction technique differs from previous approaches since i) we produce two probabilities (0 < p < 1) instead of two unbounded scores and ii) we use the computed start probability to filter out possible mentions, which reduces by a factor of 9 the number of mentions considered compared to existing Coarse-to-Fine systems (Table 1, first row).
Mention Regularization To further reduce the computation demand of this process, in the Maverick Pipeline we use the end-of-sentence (EOS) mention regularization strategy: after extracting the span start, we consider only the tokens up to the nearest EOS as possible mention end candidates.2 Since annotated mentions never span across sentences, EOS mention regularization prunes the number of mentions considered without any loss of information. While this heuristic was initially introduced in the implementation of Lee et al. (2018), all the recent Coarse-to-Fine have abandoned it in favor of the maximum span-length regularization, which is a manually-set hyperparameter that regulates a threshold to filter out spans that exceed a certain length. This implies a large overhead of unnecessary computations and introduces a structural bias that does not consider long mentions that exceed a fixed length.3 In our work, we not only reintroduce the EOS mention regularization, but we also study its contribution in terms of efficiency, as reported in Table 1, second row.
Mention Pruning After the mention extraction step, as a result of the Maverick Pipeline, we consider an 18x lower number of candidate mentions for the successive mention clustering phase (Table 1). This step consists of computing, for each mention, the probability of all its antecedents being in the same cluster, incurring a quadratic computational cost. Within the Coarse-to-Fine formulation, this high computational cost is mitigated by considering only the top k mentions according to their probability score, where k is a manually set hyper-
2We note that all the well-established Coreference Resolution datasets are sentence-split.
3The max-length regularization filters out 196 correctly annotated spans when training on OntoNotes.
parameter. Since after our mention extraction step we obtain probabilities for a very concise number of mentions, we consider only mentions classified as probable candidates (i.e., those with pend > 0.5 and pstart > 0.5), reducing the number of mention pairs considered by a factor of 10. In Table 1, we compare the previous Coarse-to-Fine formulation with the new Maverick Pipeline.

heading: 3.2 Mention Clustering
text: As a result of the Maverick Pipeline, we obtain a set of candidate mentions M = (m1,m2, . . . ,ml), for which we propose three different clustering techniques: Mavericks2e and Maverickmes, which use two well-established Coarse-to-Fine mentionantecedent techniques, and Maverickincr, which adopts a novel incremental technique that leverages a light transformer architecture.
Mention-Antecedent models The first proposed model, Mavericks2e, adopts an equivalent mention clustering strategy to Kirstain et al. (2021): given a mention mi = (xs, xe) and its antecedent mj = (xs′ , xe′), with their start and end token hidden states, we use two fully-connected layers to model their corresponding representations:
Fs(x) = W ′ s(GeLU(Wsx))
Fe(x) = W ′ e(GeLU(Wex))
we then calculate their probability to be in the same cluster as:
pc(mi,mj) = σ(Fs(xs) ·Wss · Fs(xs′)+ Fe(xe) ·Wee · Fe(xe′)+ Fs(xs) ·Wse · Fe(xe′)+ Fe(xe) ·Wes · Fs(xs′))
with Wss,Wee,Wse,Wes being four learnable matrices and Ws,W ′s,We,W ′ e the learnable parameters of the two fully-connected layers. A similar formulation is adopted in Maverickmes, where, instead of using only one generic mentionpair scorer, we use 6 different scorers that handle linguistically motivated categories, as introduced by Otmazgin et al. (2023). We detect which category k a pair of mentions mi and mj belongs to (e.g., if mi is a pronoun and mj is a proper noun, the category will be PRONOUN-ENTITY) and use a category-specific scorer to compute pc. A complete description of the process along with the list of categories can be found in Appendix A.
Incremental model Finally, we introduce a novel incremental approach to tackle the mention clustering step, namely Maverickincr, which follows the standard shift-reduce paradigm introduced in Section 2.1. Differently from the previous neural incremental techniques (i.e., ICoref (Xia et al., 2020) and longdoc (Toshniwal et al., 2021)) which use a linear classifier to obtain the clustering probability between each mention and a fixed length vector representation of previously built clusters, Maverickincr leverages a lightweight transformer model to attend to previous clusters, for which we retain the mentions’ hidden representations. Specifically, we compute the hidden representations (h1, . . . , hl) for all the candidate mentions in M using a fully-connected layer on top of the concatenation of their start and end token representations. We first assign the first mention m1 to the first cluster c1 = (m1). Then, for each mention mi ∈ M at step i we obtain the probability of mi being in a certain cluster cj by encoding hi with all the representations of the mentions contained in the cluster cj using a transformer architecture. We use the first special token ([CLS]) of a single-layer transformer architecture T to obtain the score S(mi, cj) of mi being in the cluster cj = (mf , . . . ,mg) with f ≤ g < i as:
S(mi, cj) = Wc ·(ReLU(TCLS(hi, hf , . . . , hg)))
Finally, we compute the probability of mi belonging to cj as:
pc(mi ∈ cj |cj = (mf , . . . ,mg)) = σ(S(mi, cj))
We calculate this probability for each cluster cj up to step i. We assign the mention mi to the most probable cluster cj having pc(mi ∈ cj) > 0.5 if
one exists, or we create a new singleton cluster containing mi.
As we show in Sections 5.3 and 5.5, this formulation obtains better results than previous incremental methods, and is beneficial when dealing with long-document and out-of-domain settings.

heading: 3.3 Training
text: To train a Maverick model, we optimize the sum of three binary cross-entropy losses:
Lcoref = Lstart + Lend + Lclust
Our loss formulation differs from previous transformer-based Coarse-to-Fine approaches, which adopt the marginal log-likelihood to optimize the mention to antecedent score (Lee et al., 2018; Kirstain et al., 2021). Since their formulation “makes learning slow and ineffective, especially for mention detection” (Zhang et al., 2018), we directly optimize both mention extraction and mention clustering with a multitask approach. Lstart and Lend are the start loss and end loss, respectively, of the mention extraction step, and are defined as:
Lstart = N∑
i=1
−(yi log(pstart(ti))+
(1− yi) log(1− pstart(ti)))
Lend =
S∑
s=1
Es∑
j=1
−(yi log(pend(tj |ts))+
(1− yi) log(1− pend(tj |ts))) where N is the sequence length, S is the number of starts, Es is the number of possible ends for a start s and pstart(ti) and pend(tj |ts) are those defined in Section 3.1.
Finally, Lclust is the loss for the mention clustering step. Since we experiment with two different mention clustering formulations, we use a different loss for each clustering technique, namely Lantclust for the mention-antecedent models, i.e., Mavericks2e and Maverickmes, and Lincrclust for the incremental model, i.e., Maverickincr :
Lantclust =
|M |∑
i=1
|M |∑
j=1
−(yi log(pc(mi|mj))+
(1− yi) log(1− pc(mi|mj)))
Lincrclust =
|M |∑
i=1
(
Ci∑
j=1
−(yi log(pc(mi ∈ cj))+
(1− yi) log(1− pc(mi ∈ cj))))
where |M | is the number of extracted mentions, Ci is the set of clusters created up to step i, and pc(mi|mj) and pc(mi ∈ cj) are defined in Section 3.2.
All the models we introduce are trained using teacher forcing. In particular, in the mention token end classification step, we use gold start indices to condition the end tokens prediction, and, for the mention clustering step, we consider only gold mention indices. For Maverickincr, at each iteration, we compare each mention only to previous gold clusters.

heading: 4 Experiments Setup

heading: 4.1 Datasets
text: We train and evaluate all the comparison systems on three Coreference Resolution datasets:
OntoNotes (Pradhan et al., 2012), proposed in the CoNLL-2012 shared task, is the de facto standard dataset used to benchmark Coreference Resolution systems. It consists of documents that span seven distinct genres, including full-length documents (broadcast news, newswire, magazines, weblogs, and Testaments) and multiple speaker transcripts (broadcast and telephone conversations).
LitBank (Bamman et al., 2020) contains 100 literary documents typically used to evaluate longdocument Coreference Resolution.
PreCo (Chen et al., 2018) is a large-scale dataset that includes reading comprehension tests for middle school and high school students.
Notably, both LitBank and PreCo have different annotation guidelines compared to OntoNotes, and provide annotation for singletons (i.e., singlemention clusters). Furthermore, we evaluate models trained on OntoNotes on three out-of-domain datasets:
• GAP (Webster et al., 2018) contains sentences in which, given a pronoun, the model has to choose between two candidate mentions.
• LitBankns and PreCons, the datasets’ test-set where we filter out singleton annotations.
• WikiCoref (Ghaddar and Langlais, 2016), which contains Wikipedia texts, including documents with up to 9,869 tokens.
The statistics of the datasets used are shown in Table 2.

heading: 4.2 Comparison Systems
text: Discriminative Among the discriminative systems, we consider c2f-coref (Joshi et al., 2020) and s2e-coref (Kirstain et al., 2021), which build upon the Coarse-to-Fine formulation and adopt different document encoders. We also report the results of LingMess (Otmazgin et al., 2023), which is the previous best encoder-only solution, and f-coref (Otmazgin et al., 2022), which is a distilled version of LingMess. Furthermore, we include CorefQA (Wu et al., 2020), which casts Coreference as extractive Question Answering, and wl-coref (Dobrovolskii, 2021), which first predicts coreference links between words, then extracts mentions spans. Finally, we report the results of incremental systems, such as ICoref (Xia et al., 2020) and longdoc (Toshniwal et al., 2021).
Sequence-to-Sequence We compare our models with TANL (Paolini et al., 2021) and ASP (Liu et al., 2022), which frame Coreference Resolution as an autoregressive structured prediction. We also include Link-Append (Bohnet et al., 2023), a transition-based system that builds clusters with a multi-pass Sequence-to-Sequence architecture. Finally, we report the results of seq2seq (Zhang et al., 2023), a model that learns to generate a sequence with Coreference Resolution labels.

heading: 4.3 Maverick Setup
text: All Maverick models use DeBERTa-v3 (He et al., 2023) as the document encoder. We use DeBERTa because it can model very long input texts effectively (He et al., 2021).4 Moreover, compared to the LongFormer, which was previously adopted by several token-level systems, DeBERTa ensures a larger input max sequence length (e.g., DeBERTalarge can handle sequences up to 24,528 tokens while LongFormer only 4096) and has shown better performances empirically in our experiments on the OntoNotes dataset. On the other hand, using
4This is because its attention mechanism enables its input length to grow linearly with the number of its layers.
DeBERTa to encode long documents is computationally expensive because its attention mechanism incurs a quadratic computational complexity. Whereas this further increases the computational cost of traditional Coarse-to-Fine systems, the Maverick Pipeline enables us to train models that leverage DeBERTalarge on the OntoNotes dataset, without any performance-lowering pruning heuristic. To train our models we use Adafactor (Shazeer and Stern, 2018) as our optimizer, with a learning rate of 3e-4 for the linear layers, and 2e-5 for the pretrained encoder. We perform all our experiments within an academic budget, i.e., a single RTX 4090, which has 24GB of VRAM. We report more training details in Appendix B.

heading: 5 Results

heading: 5.1 English OntoNotes
text: We report in Table 3 the average CoNLL-F1 score of the comparison systems trained on the English OntoNotes, along with their underlying pre-trained language models and total parameters. Compared to previous discriminative systems, we report gains of +2.2 CoNLL-F1 points over LingMess, the best encoder-only model. Interestingly, we even outperform CorefQA, which uses additional Question Answering training data.
Concerning Sequence-to-Sequence approaches, we report extensive improvements over systems with a similar amount of parameters compared to our large models (500M): we obtain +3.4 points compared to ASP (770M), and the gap is even wider when taking into consideration Link-Append (3B) and seq2seq (770M), with +6.4 and +5.6, respectively. Most importantly, Maverick models surpass the performance of all Sequence-toSequence transformers even when they have several billions of parameters. Among our proposed methods, Maverickmes shows the best performance, setting a new state of the art with a score of 83.6 CoNLL-F1 points on the OntoNotes benchmark. More detailed results, including a table with MUC, B3, and CEAFϕ4 scores and a qualitative error analysis, can be found in Appendix C.

heading: 5.2 PreCo and LitBank
text: We further validate the robustness of the Maverick framework by training and evaluating systems on the PreCo and LitBank datasets. As reported in Table 4, our models show superior performance when dealing with long documents in a
data-scarce setting such as the one LitBank poses. On this dataset, Maverickincr achieves a new stateof-the-art score of 78.3, and gains +1.0 CoNLLF1 points compared with seq2seq. On PreCo, Maverickincr outperforms longdoc, but seq2seq still shows slightly better performance. This is mainly due to the high presence of singletons in PreCo (52% of all the clusters). Our systems, using a mention extraction technique that favors precision rather than recall, are penalized compared to high recall systems such as seq2seq.5 Among our systems, Maverickincr, leveraging its hybrid architecture, performs better on both PreCo and LitBank.

heading: 5.3 Out-of-Domain Evaluation
text: In Table 5, we report the performance of Maverick systems along with LingMess, the best encoderonly model, when dealing with out-of-domain texts, that is, when they are trained on OntoNotes and tested on other datasets. First of all, we report considerable improvements on the GAP test set, obtaining a +1.2 F1 score compared to the previous state of the art. We also test models on WikiCoref, PreCons and LitBankns (Section 4.1). However, since the span annotation guidelines of these corpora differ from the ones used in OntoNotes, in Table 5 we also report the performance using gold mentions, i.e., skipping the mention extraction step (gold column).6 On the WikiCoref benchmark, we achieve a new state-of-the-art score of 67.2 CoNLLF1, with an improvement of +4.2 points over the previous best score obtained by LingMess. On the same dataset, when using pre-identified mentions the gap increases to +5.8 CoNLL-F1 points (76.6 vs 82.4). In the same setting, our models obtain up to +7.3 and +10.1 CoNLL-F1 points on Precons and LitBankns, respectively, compared to LingMess. These results suggest that the Maverick training strategy makes this model more suitable when dealing with pre-identified mentions and out-of-domain texts. This further increases the potential benefits that Maverick systems can bring to many downstream applications that exploit coreference as an intermediate layer, such as Entity Linking (Rosales-Méndez et al., 2020) and Relation Extraction (Xiong et al., 2023; Zeng
5Precision and Recall scores are reported in Appendix C. 6We do not include autoregressive models because none of the original articles report scores on out-of-domain datasets. We also could not test those models because they do not provide the code to perform mention clustering alone, and performing it with such approaches is not as straightforward as in encoder-only models.
et al., 2023), where the mentions are already identified. Among our models, on LitBankns and WikiCoref, Maverickincr outperforms Maverickmes and Mavericks2e, confirming the superior capabilities of the incremental formulation in the long-document setting. Finally, we highlight that the performance gap between using gold mentions and performing full Coreference Resolution is wider when tested on out-of-domain datasets (on average +17%) compared to testing it directly on OntoNotes (83.6 vs 93.6, +10%).7 This result, obtained on three different out-of-domain datasets, suggests that the difference in annotation guidelines considerably contribute to lower the OOD performances (-7%).
7An evaluation of the proposed Maverick models in terms of mention extraction and mention clustering using gold mentions scores can be found in Appendix C.

heading: 5.4 Speed and Memory Usage
text: In Table 3, we include details regarding the training time and the hardware used by each comparison system, along with the measurement of the inference time and peak memory usage on OntoNotes the validation set. Compared to Coarseto-Fine models, which require 32GB of VRAM, we can train Maverick systems under 18GB. At inference time both Maverickmes and Mavericks2e, exploiting DeBERTalarge, achieve competitive speed and memory consumption compared to wl-coref and s2e-coref. Furthermore, when adopting DeBERTabase, Maverickmes proves to be the most efficient approach8 among those directly trained
8In terms of inference peak memory usage and speed.
on OntoNotes, while, at the same time, attaining performances that are equal to the previous best encoder-only system, LingMess. The only system that shows better inference speed is f-coref, but at the cost of lower performance (-3.0).
Compared to the previous Sequence-toSequence state-of-the-art approach, Link-Append, we train our models with 175x less memory requirements. Comparing inference time is more complicated, since we could not run models on our memory-constrained budget. For this reason, we report the inference times from the original articles, and hence times achieved with their high-resource settings. Interestingly, we report as much as 170x faster inference compared to seq2seq, which exploits parallel inference on multiple GPUs, and 85x faster when compared to the more efficient ASP. Among Maverick models, Maverickincr is notably slower both in inference and training time, as it incrementally builds clusters using multiple steps.

heading: 5.5 Maverick Ablation
text: In Table 6, we compare Mavericks2e and Maverickmes models with s2e-coref and LingMess, respectively, using different pre-trained encoders. Interestingly, when using DeBERTa, Maverick systems not only achieve better speed and memory efficiency, but also obtain higher performance compared to the previous systems. When using the LongFormer, instead, their scores are in the same ballpark, showing empirically that the Maverick training procedure better exploits the ca-
pabilities of DeBERTa. To test the benefits of our novel incremental formulation, Maverickincr, we also implement a Maverick model with the previously adopted incremental method used in longdoc and ICoref (Section 2.1), which we call Maverickprev-incr. Compared to the previous formulation we report an increase in score of +3.9 CoNLL-F1 points. The improvement demonstrates that exploiting a transformer architecture to attend to all the previously clustered mentions is beneficial, and enables the future usage of hybrid architectures when needed.
As a further analysis of whether the efficiency improvements of our systems stem from using DeBERTa or are attributable to the Maverick Pipeline, we compared the speed and memory occupation of a Maverick system using as underlying encoder either DeBERTalarge or LongFormerlarge. Our experiments show that using DeBERTa leads to an increase of +77% of memory space and +23% of time to complete an epoch when training on OntoNotes. An equivalent measurement, attributable to the quadratic memory attention mechanism of DeBERTa, was observed for the inference time and memory occupation on the OntoNotes test set. These results highlight the efficiency contribution of the Maverick Pipeline, which is agnostic to the document encoder and can be applied to future coreference systems to ensure higher efficiency.

heading: 6 Conclusion
text: In this work, we challenged the recent trends of adopting large autoregressive generative models to solve the Coreference Resolution task. To do so, we proposed Maverick, a new framework that enables fast and memory-efficient Coreference Resolution while obtaining state-of-the-art results. This demonstrates that the large computational overhead required by Sequence-to-Sequence approaches is unnecessary. Indeed, in our experiments Maverick systems demonstrated that they can outperform large generative models and improve the speed and memory usage of previous best-performing encoder-only approaches. Furthermore, we introduced Maverickincr, a robust multi-step incremental technique that obtains higher performance in the out-of-domain and long-document setting. By releasing our systems, we make state-of-the-art models usable by a larger portion of users in different scenarios and potentially improve downstream applications.

heading: 7 Limitations
text: Our experiments were limited by our resource setting i.e., a single RTX 4090. For this reason, we could not run Maverick using larger encoders, and could not properly test Sequence-to-Sequence models as we did with encoder-only models. Nevertheless, we believe this limitation is a common scenario in many real-world applications that would benefit substantially from our system. We also did not test our formulation on multiple languages, but note that both the methodology behind Maverick and our novel incremental formulation are language agnostic, and thus could be applied to any language.

heading: Acknowledgements
text: We gratefully acknowledge the support of the PNRR MUR project PE0000013-FAIR.
Roberto Navigli also gratefully acknowledges the support of the CREATIVE project (CRoss-modal understanding and gEnerATIon of Visual and tExtual content), which is funded by the MUR Progetti di Rilevante Interesse Nazionale programme (PRIN 2020). This work has been carried out while Giuliano Martinelli was enrolled in the Italian National Doctorate on Artificial Intelligence run by Sapienza University of Rome.

heading: A Multi-Expert Scorers
text: In Maverickmes, the final coreference score between two spans is calculated using 6 linguistically motivated multi-expert scorers. This approach was introduced by Otmazgin et al. (2023), which demonstrated that linguistic knowledge and symbolic computation can still be used to improve results on the OntoNotes benchmark. In Maverickmes we adopt this approach on top of the Maverick Pipeline. We use the same set of categories, namely:
1. PRON-PRON-C. Compatible pronouns based on their attributes, such as gender or number (e.g. (I, I), (I, my) (she, her)).
2. PRON-PRON-NC, Incompatible pronouns (e.g. (I, he), (She, my), (his, her)).
3. ENT-PRON. Pronoun and non-pronoun (e.g. (George, he), (CNN, it), (Tom Cruise, his)).
4. MATCH. Non-pronoun spans with the same content words (e.g. Italy, Italy).
5. CONTAINS. One contains the other (e.g. (Barack Obama, Obama)).
6. OTHER. The Other pairs.
To detect pronouns we use string match with a full list of English pronouns.
To perform mention clustering, we dedicate a mention-pair scorer for each of those categories.
Specifically, for the mention mi = (xs, xe) and its antecedent mj = (xs′ , xe′), with their start and end token hidden states, we first detect their category kg using pattern matching on their spans of texts. Then we compute their start and end representations, using the specific fully-connected layers for the category kg:
F kg s (x) = W ′ kg,s(GeLU(Wkg,sx))
F kg e (x) = W ′ kg,e(GeLU(Wkg,ex))
The probability pkgc of mi and mj is then calculated as:
p kg c (mi,mj) = σ(F kg s (xs) ·Wss · F kgs (xs′)+
F kg e (xe) ·Wee · F kge (xe′)+
F kg s (xs) ·Wse · F kge (xe′)+
F kg e (xe) ·Wes · F kgs (xs′))
With Wss,Wee,Wse,Wes being four learnable matrices and W ′kg,e ,W ′ kg,s ,Wkg,e ,W ′ kg,s
the learnable parameters of the two fully-connected layers. In this way, each mention-pair scorer learns to model the probability for its specific linguistic categories.

heading: B Training details
text: B.1 Datasets We report technical details of the adopted datasets.
• OntoNotes contains several items of metadata information for each document, such as genre, speakers, and constituent graphs. Following previous works, we incorporate the speaker’s name into the text whenever there is a change in speakers for datasets that include this metadata.
• LitBank contains 100 literary documents and is available in 10 different cross-validation folds. Our train, dev, and test splits refer to the first cross-validation fold, LB0. We report comparison systems results on the same splits. Since training DeBERTalarge is particularly computationally expensive, as introduced in Section 4.3, we train on LitBank by splitting in half each LitBank training document.
• The authors of PreCo have not released their official test set. To evaluate our models consistently with previous approaches, we use the official ’dev’ split as our test set and retain the last 500 training examples for model validation.
B.2 Setup
All our experiments are developed using the Pytorch-Lightning framework.9 For each Maverick model, we load the pre-trained weights for the base10 and large11 version of DeBERTa−v3 from the Huggingface Transformers library (Wolf et al., 2020). We accumulate gradients every 4 steps and use a gradient clipping value of 1.0. We adopt a linear learning rate scheduler a warm-up of 10% of the total steps check validation scores every 50% of the total number of steps per epoch. We select our model upon validation of Avg. CoNLL-f1 score and use patience of 20.

heading: C Additional Results
text: In Table 7 we report the performance of models according to the standard Coreference Resolution metrics: MUC (Vilain et al., 1995), B3(Bagga and Baldwin, 1998), CEAFϕ4 (Luo, 2005) and AVG CoNLL-F1. Scores for Maverick models are computed using the official CoNLL coreference scorer.12
9 https://lightning.ai
10 https://huggingface.co/microsoft/deberta-v3-base 11 https://huggingface.co/microsoft/deberta-v3-large 12 https://conll.github.io/reference-coreference-scorers
C.1 Error Analysis To better understand the quality of Maverick predictions, we conduct an error analysis on our best system trained on OntoNotes, Maverickmes. In Table 8, we report the score of performing only mention extraction (F1) or mention clustering with gold mention (CoNLL-F1) with our systems. Our results highlight that our models have strong capabilities of clustering pre-identified mentions, but limited performance in the identification of correct spans. We investigated this phenomenon by conducting a qualitative evaluation of the outputs of our best system, Maverickmes, and found out that OntoNotes contains several annotation errors. We report examples of errors in Table 9. The main inconsistency we found in the gold test set is that many documents have incomplete annotations, which directly correlates with the mention extraction error.

heading: Future Work
text: The improvement demonstrates that exploiting a transformer architecture to attend to all the previously clustered mentions is beneficial, and enables the future usage of hybrid architectures when needed. As a further analysis of whether the efficiency improvements of our systems stem from using DeBERTa or are attributable to the Maverick Pipeline, we compared the speed and memory occupation of a Maverick system using as underlying encoder either DeBERTalarge or LongFormerlarge. Our experiments show that using DeBERTa leads to an increase of +77% of memory space and +23% of time to complete an epoch when training on OntoNotes. An equivalent measurement, attributable to the quadratic memory attention mechanism of DeBERTa, was observed for the inference time and memory occupation on the OntoNotes test set. These results highlight the efficiency contribution of the Maverick Pipeline, which is agnostic to the document encoder and can be applied to future coreference systems to ensure higher efficiency