Seq2seq is All You Need for Coreference Resolution
Authors: Wenzheng Zhang
Sam Wiseman
Karl Stratos
Venue: EMNLP 2023 Main (2023)
Keywords: coreference resolution; sequence-to-sequence models;

Abstract: Existing works on coreference resolution suggest that task-specific models are necessary to achieve state-of-the-art performance. In this work, we present compelling evidence that such models are not necessary. We finetune a pretrained seq2seq transformer to map an input document to a tagged sequence encoding the coreference annotation. Despite the extreme simplicity, our model outperforms or closely matches the best coreference systems in the literature on an array of datasets. We consider an even simpler version of seq2seq that generates only the tagged spans and find it highly performant. Our analysis shows that the model size, the amount of supervision, and the choice of sequence representations are key factors in performance.

Seq2seq is All You Need for Coreference Resolution
Wenzheng Zhang' Sam Wiseman Karl Stratos'
Rutgers University 2 Duke University
{wenzheng zhang, karl. stratos jerutgesees edu swiseman@cs duke. edu
Abstract
A natural question is: such task-specific mod- els necessary for coreference resolution, or can we approach as standard seq2seq problem? There have been previous efforts to reduce corefer- ence resolution as a seq2seq problem (Urbizu et al.. 2020; Paolini et al., 2021), ), but they underperform task-specific models, suggesting that task-specific models are perhaps necessary.
Existing works on coreference resolution sug- gest that task-specific models are necessary to achieve state-of- -ot--of--heeeee the-art performance. In this work, we present compelling evidence that such models are not necessary. We finetune a pre- trained seq2seq transformer to map input document to a tagged sequence encoding the coreference annotation. Despite the extreme simplicity, our model outperforms closely matches the best coreference systems in the lit- erature array of datasets. We also propose an especially simple seq2seq approach that gen- erates only tagged spans rather than the spans interleaved with the original text. Our analy- sis shows that the model size, the amount of supervision, and the choice of sequence repre- sentations are key factors in performance.
In this work, we present the first full seq2seq reduction of coreference resolution that matches or outperforms the best coreference systems in the lit- erature, demonstrating that task-specific models are not necessary to obtain state-of-the-ar performance and questioning the need develop specifci solutions. Our approach extremely simple. We treat the raw document as a source sequence and the coreference annotation as a target sequence, then finctune pretrained encoder-decoder model like T5 (Raffel et al., 2020) TO (Sanh et al., 2022) without any modification to the architecture.
1 Introduction
The seminal work by Lee al. (2017) popularized end-to-end models for coreference resolution based on searching over all possible spans and their clus- tering. However, even with substantial refinement and simplification in followup works (Lee et al., 2018; Xu and Choi, 2020; Wu et al., 2020; Kirstain et al., 2021), the models are highly task-specific, involving many specialized hyperparameters such how many candidates to retain in the mention proposal phase.
There is great deal of flexibility in the choice of target sequence representation. Our main model represents the coreference annotation sequence of actions that either copy token from the source sequence, start/end mention span, or tag pre- dicted mention span with an integer. At test time, the model always produces a valid coreference clus- tering by constrained beam search. We consider an even simpler version in which the model gen- erates only the tagged spans, and find that it also yields surprisingly high performance. This simpler version is advantageous because it results in faster inference.
There is recent line of works that take an alter- native approach, leveraging advances in pretrained sequence-to-ssqquene (seq2seq) models. Liu et al. (2022) propose ASP, an autoregressive pointer- based model with a multitasking head for bracket pairing and span labeling. Bohnet et al. (2023) propose caasition-based system with carefully designed states and actions, simulated by seq2sey model that processes one state at a time. However, these seq2seq-ban based models are still task-specific. requiring modification the seq2seq architecture or a derivation a transition-bast system with state manipulation.
Our seq2seq model obtains strong results on an array of coreference datasets. On English OntoNotes (Pradhan et al., 2012), with a 3B- parameter TO the model obtains 82.9 test aver- age F1, outperforming the corresponding ASP (Liu al., 2022) initialized from the same base model (82.3). With a 11B-parameter TO, our model achieves 83.2 FI, outperforming CorefQA

---

(Wu et al., 2020) (83.1) and getting close to the best known result using 1aB-parameter model (83.3). On PreCo (Chen et al., 2018), the model achieves 88.5, outperforming the task-specific model of Toshniwal et al. (2021) (87.8). On Lit- Bank (Bamman et al., 2020), which has substan- tially smaller training data, the model obtains 78.3 cross-validation F1 lagging behind Toshniwal et al. (2021) who report 79.2. But when trained the union of LitBank, OntoNotes, and PreCo, obtains 81.2 split-0 FI, significantly outperforming their 78.2. Our analysis shows that the model size, the amount of supervision, and the choice of sequence representations are key factors in performance. We make our code publicly available at: https: wtt.e/bb...o//eeeeennnaaaa Seaaaannggggggseerr.......
initions (Section 3). ASP (Liu et al., 2022) is an autoregressive pointer-based model for structured prediction. It is modified transformer that at step conditions on the input document z and sequence represen- tation of the annotation so far 2şt and predicts a tuple of model-specific actions (ou, (1) used for structure building. For instance, E {0... 1} is a bracket pairing action parameterized with a feedforward layer that consumes the previous hid- den states (h,, h,, (i.e., a pointer network). ASP obtains state-of-the-ar results on OntoNotes (up to 82.5 average F1). We outperform ASP with a standard transformer.
Bohnet al. (2023) develop a transition- based coreference system that can be implemented by a seq2seq model. The system autoregressively maps a state to a prediction, where a state is previous coreference-annorated sentences along with the next sentence and a prediction is system actions (e.g., link and append). While the system can be reduced to seq2seq predictions, it processes sentence
2
Related Work
this section, we give a more focused treatment of previous works on seq2seq-style approaches to coreference resolution to make our contributions more clear. Urbizu et al. (2020) propose framing coreference resolution as seq2seq task, but their work is proof of concept. They naively predict boundaries, cluster numbers, and blanks (c.g.. "(O 0) - (2) I 1)"" with suboptimal modeling choices (c.g., their decoder only receives these sym- bols and no document content). They only report results on the ARRAU corpus and significantly fall behind existing works (e.g., 66.5 vs 78.8 under B3). In contrast, we solve much more challeng- ing problem of developing state-of-the-ann seq2seq coreference systems.
time by applying the predicted ac- tions the current state. We show that such an explicit state-by-state transition is not necessary, and that a standard transformer can directly reason with all coreference clusters simultaneously.
Seq2Seq Methods
Let denote the vocabulary. For any sequence E D' and position 1 E {1... .....Tr T}, we use the prefix notation act (a1 (Q1...:.1)) E yt-1 and agi (a1... a.) E Vt. We assume a gener- alized seq2seq setting in which E yT" is the source sequence, y E D is the target sequence (i.e., a sequence of labels), and VT is an addi- tional sequence fed the seq2seq decoder where at = F(x, Z<tt Metr) some fixed deterministic mapping. The "generalized" model uses parame- ters 0 to define the conditional distribution
There are recent works upproaching structured prediction with a general seq2seq-style solution. One example is TANI. (Paolini et al., 2021), which frames entity/relation extraction, semantic role labeling, and coreference resolution as seq2seq tasks. Again, TANL fails demonstrate compet- itive performance on standard coreference resolu- tion datasets, obtaining only 72.8 average F'l on OntoNotes (compared to the current state-of-the- art performance level which is > 80). Their target sequence representation corresponds to our full lin- earization with token action except that they tag the cluster information by preceding mention strings (e.g.. "[ his Barack Obama |""), which yields long target sequences and introduces clustering ambigu- ity. While we improve the performance of TANL. to 79.6 in our own implementation, achieving the best performance (83.2) requires our sequence def-
po(y II po zg().
(1)
We emphasize that the generalization above does not modify the standard seq2seq framework. Dur- ing training, we feed (xr to the model as the usual source-target sequence pair and minimize the cross-entropy loss using y as per-token labels. At test time, we apply the mapping F at each step by postprocessing predictions.

---

All our methods use (1) and only change the variable definitions. The encoder input always the document to be processed. The decoder input 2 is a sequence representation or "linearization" of the coreference annotation of x. The decoder output is any action sequence from which 2 can be extracted deterministically at each step.
the corresponding spans in the source sequence? A simple solution popular in general seq2seq ap- proaches to tagging to exhaustively predict all the tokens in x (Daza and Frank, 2018; De Cao et al., 2021). The example (2) is then linearized as
a <m> <m> </m> e 2 </m> d <m> 2 </m>
We call this representation full linearization. Full lincarization completely climinates the problem of alignment ambiguity at the cost of longer target sequences. We also consider an alternative shorter representation that we call partial linearization in which only the tagged mentions are encoded. this case, (2) is linearized as
3.1 Linearization of the Coreference Annotation
For document x V"", the coreference anno- tation S is a set of spans clustered into C. groups formalized as
S {(.,,,1): : ISiSiST, 1 13
<m>
<m> </m> 2 </m> <m> e </m>
Note that the spans can be nested. We assume that the spans are ordered in non-decreasing lengths. We will use the following as running example:
Partial linearization has the potential to drastically shorten the target length if the mentions are sparse, but it requires an explicit aligmment step to transfer the predictions to the input document. We defer the discussion of the alignment problem to Section 3.4.
x b,, d, = (2,, 1), (5, 5, 2), (2,3
(2)
3.2 Action Sequences
(i.e., the clustered spans are {(b) ((b.c),e{)) The goal is to express S as sequence D for some length T. A minimal formulation is to literally predict the integer triples, for instance 2 sSr(S) where str is the string conversion in Python. However, while short, such non- linguistic representation was found to perform poorly likely because it is not compatible with lan- guage model pretraining.
Given a choice of lincarization 2 E V' (i.e., input to the decoder), we can choose any action sequence E Vd (i.e., the actual predictions by the decoder) such that at each step f,, we can extract 2 from the document & and the past information z<t and yett- We assume that = 
 and 27+1 
 are the standard start and end symbols for the target quence. A straightforward action sequence is given by 1/: Σ++1 for / 1 ... . which we call token action. Token action corresponds to a common language modeling setting where the per-step pre- diction is simply the next token. For instance, the annotation x (a, b) and S = {(2, 2, 1)} under full lincarization and token action is assigned
A better approach is to predict the mentions. We represent the tagged span (i,j, in document x as
<m> Fa 23 </m>
where <m> </m>> E V are special symbols indicat- ing the start and end of a mention representation, and E V indicates the end of a mention string. Nested spans are handled naturally by linearizing the spans in order (i.e.. from the shortest to the longest). For instance, the subsequence (b, c) E 22 in the running example (2) will become the length- 10 sequence
y = (a, <m> b, 1//m,,,//////// 
)
: = (
, a, <m> b, |, 1, </m>, 
)
(3)
Under full lincarization, action space A <<>,<m>> </m>, 
) U u where <c> is the special "copy" symbol which means copying single token from the document and advancing the index by one, and 4 C V is the subset of the vocabulary used for encoding integers. We call this choice copy action, formally defined
can
smaller
<m> <m> b </m> cl </m>
In contrast a transition-based approach (Bohnet al., 2023), we decode all coreference clusters in the document in a single pass. Thus there an issue of alignment: if the model predicts multiple men- tions with the same string form, how do we know
as
<C>
if 2111 & (<m>, </m>, 
) Uu otherwise
Ut
2441

---

The example (3) under copy action becomes
Consider the document (a, b, d, e, b). The
lincarization consisting of two length-1 mentions
y = (<c>, <m> <c>, 1, </m>, 
) 2 - (
, a, <m>, b, 1, </m><<//>>>/// 
)
<m> </m> <m> b </m>
(5)
Copy action in conjunction with constrained beam search is a natural way to prevent deviations be- tween the source and target sequences (Daza and Frank, 2018). Partial linearization is not compati- ble with copy action since the model skips the gap between mentions.
correspond to either { 2, 1), (6, 1)} or S { 6, 1), (7, 1)). We align mentions by aligning tokens with gaps (i.e., a token may be aligned to nothing). Prior to aligning tokens, we remove all special symbols while saving the span information for each mention (omitting the clus- ter information for simplicity). For instance, (5) becomes (b, b) with spans (1, and (2. 2).
3.3 Integer-Free Representation
So far we have relied on the separation symbol and an integer to label every mention with its clus- ter identity. Because the number of mentions in document is quite large, we consider ways avoid these two symbols. One way is hard-code the cluster information directly in the mention bound- aries by introducing a special symbol </m,> E for each cluster number = .,....C C. Under this scheme, we may assign to the annotation æ = (a,b) b)
|
We then find a highest-scoring alignment where the score measures token matching (1 if matched, -1 1 if mismatched) and length-n affine gap penalty g()) = - p(n 1), where is a hyperparame- ter'. The affine gap penalty encourages long unseg- mented gaps, capturing the intuition that mentions tend appear in high density. The above example has the optimal aligmment with 2 matches (6 ++ and + 2) and a length-ö gap in the source se- quence ()...)). Plugging in the token matches in the span information, we identify the locations (6, 6) and (7, 7) corresponding to the second an- notation. This approach naturally handles nested mentions.
and S = {11, (2, 1)):
y (<m>, <<>; </m|> <m>, <C>; </m;> 
) 2 (
, <m>, a, </m?> <m>, </m}> 
)
The performance of this representation was found to be surprisingly poor, likely because the model has hard time learning to predict so many new symbols. We tackle this issue by introducing "new" action <new> that delegates the burden of specifying an unseen cluster number to postprocess- ing. The example (4) now assigned the action sequence
For a document of length T' and partial linearization (with special symbols removed) of length K, there are (T** possible alignments with gaps. We exactly find an optimal alignment in O K) time by Gotoh's s algorithm (Gotoh, 1982). An oracle experiment shows that our approach is highly effective, reaching 98.0 average F1 on OntoNotes if we use the gold partial linearization. We further improve the alignment scheme by in- serting sentence markers in the input document and linearization, which allows us to constrain the alignment to sentence pairs at test time.
y = (<m>, <c>, <new>, <M>> <C>, </m/s/////////// 
) = (
, <m>,<m </m1>, <m> b. </m]> 
)
this way, whenever the model predicts <new> we can feed </m/+1> as the decoder input where is the previous number of clusters. The model is only responsible for predicting </m,> for expanding a known cluster L. We call this integer-free repre- sentation and show that it is nearly performant corresponding baseline that relies on and I while being shorter and notationally cleaner.
Discussion
Having presented our technical approach, we dis- cuss how it relates other approaches to corefer- resolution that also use sequence-based mod- els.
Our approach is pure seq2seq because we use both the standard architecture and the system de- sign that is applicable to any seq2seq task, such as machine translation and text summarization. In contrast, the approach of Bohnet et al. (2023) is not
3.4 Mention Alignment
The issue of alignment arises only under partial lin- earization and token action. this case, is possi- ble to have linearized mentions whose correspond- ing locations in the input document are ambiguous.
II our experiment, we set = 0 for simplicity, as we ob- served negligible performance differences when p S 0.0001.

---

considered pure seq2seq because their transition- based system specifically designed for the coref- erence task (e.g., their "Link" and "Append" ac- tions are meant to cluster mentions together), even though the system itself is implemented using the standard seq2seq architecture. To understand the practical difference, note that their system requires M encoder forward passes for single document during inference where M is the number of sen- tences, whereas ours requires one. The success of the transition system of Bohnet al. (2023) does not imply the success the general seq2seq system in our work.
# Docs
Dataset
Train
Dev
Test Words Mentions Cluster Size
OntoNotes
2802
343
348
467
56
4.4
LitBank
80
10
10
2105
291
3.7
PreCo
36120
500
500
337
105
1.6
Table Data statistics OntoNotes, LitBank, and PreCo datasets. The number of documents in cach split, average word count per document, average mention count per document, and average mention count per cluster are listed.
et al. (2021) and downsample the OntoNotes and PreCo datasets to 2K samples per epoch.
We adopt the generalized seq2seq framework that may require postprocessing during generation (c.g., for copy action) for improved modeling flexi- bility and performance, but the postprocessing step does not change the standard seq2seq architecture and system design. Furthermore, our model with- our postprocessing is nearly as effective. Specifi- cally, our "Full linear + token action + T03n" model in Table 2 is a standard seq2seq model with no post- processing during inference but achieves an 82.4 test F1 score, which is competitive with our best same-size "Full linear + copy action + TO0nn" model that does require token-level postprocessing (82.9).
5.2 Implementation Details
We initialize our model using the T5 model fam- ily (Raffel et al., 2020), which includes models of various sizes. Specifically, we use T5 (Raffel et al., 2020), TO (Sanh et al., 2022), and FLAN-T5 (Chung et al., 2022) with model sizes base, large, XL/3B, and XXL/pp. We use the pretrained models available in the Hugging Face Transformers library (Wolf et al., 2019).
To train large models with limited resources, we use Deepspeed (Rasley et al., 2020) with ZeRO optimizers (Rajbhandari et al., 2020) and enable gradient checkpoiningg We divide the document into overlapped segments, each with maximum length of 2048 tokens and overlapping length of 1024 tokens. During inference, the maximum input length is 4096 tokens for all our experiments. We use constrained beam search with beam size 4.
We use constrained decoding during generation to ensure valid coreference annotation, which is a standard practice (Daza and Frank, 2018; De Cao et al., 2021). The details can be found in Appendix A.
5 Experiments
For optimization, we use the Adam optimizer (Kingma and Ba, 2015) with the learning rate of 5e-4 for base/large models, 5e-5 for XL/3B, and 3e-5 for XXL/pp models. We use linear learning rate decay scheduler with a warmup proportion of 0.1. We train models using batch size of per GPU, using 8 A100 40G GPUs for models of size up to 3B and 8 A100 80G GPUs for models of size 11B.
5.1 Datasets
We train and evaluate on three widely used datasets for coreference resolution: OntoNotes (Pradhan et al., 2012), PreCo (Chen et al., 2018) and Lit- Bank (Bamman et al., 2020). The data statistics are summarized in Table 1. is important note that these datasets exhibit sigmificant variations in terms of size, document length, number men- tions/clusters, and domain. Following Kirstain et al. (2021), we incorporate the speaker's name into the text whenever there is a change in speakers for datasets that include speaker metadata. In addition to training and evaluating these three datasets individually, we also perform an additional exper- iment involving joint training on the combined dataset comprising all three datasets. To address the issue of data magnitude imbalance in joint training, we adopt the methodology suggested by Toshniwal
5.3 Baselines
We categorize the baselines into the following three groups.
Non-seq2seq Models in this category are specifi- cally designed for coreference resolution and em- ploy sophisticated coreference-specific architec- tures. Most models in this category follow the approach introduced by Lee et al. (2017) to de- tect mention spans in the input text and establish

---

MUC
B
CEAF
Avg.
Model
P
F1
R
F1
P
F1
FI
et al., 2017
78.4
73.4
75.8
68.6
61.8
62.7
60.8
67.2
Lee et al. (2018)
81.4
79.5
80.4
72.2
69.5
70.8
68.2
67.1
67.6
73.0
Joshi et al. (2019)
84.7
82.4
83.5
76.5
74.0
75.3
74.1
69.8
71.9
76,9
Yu al. (2020)
82.7
83.3
83.0
75.6
72.2
71.6
76.4
Joshi et al. (2020)
85.8
84.8
85.3
78.3
77.9
78.1
76.4
74.2
75.3
79.6
Xia al. (2020)
85.7
84.8
85.3
78.1
77.5
77.8
76.3
74.1
75.2
79.4
Toshniwal
(2020)
85.5
85.3
77.3
74.2
75.3
Non-Seq2seg
Wu et al. (2020)*
88.6
87.4
88.0
82.4
82.0
82.2
79.9
78.3
79.1
83.1
Xu and Choi (2020)
85.9
85.5
85.7
79.0
78.9
79.0
76.7
75.2
75.9
80.2
Kirstain al. (2021)
86.5
85.8
77.9
76.8
76.1
Dobrovolskii (2021)
84.9
87.9
86.3
77.4
82.6
79.9
76.1
77.1
76.6
81.0
Toshniwal al. (2021)
79.6
Liu (2022)+ T0w
85.8
86.9
83.3
78.3
78.4
82.3
Liu et al. (2022) FLAN-T5XXL
86.1
88.4
87.2
80.2
83.2
81.7
78.9
78.3
78.6
82.5
Transition Seq2seq
Bohnet et al. (2023) + mmT5xx.
87.4
88.3
87.8
81.8
83.4
82.6
79.1
79.9
79.5
83.3
Paolini et al. (2021)4T5m.
81.0
69.0
68.4
72.8
Paolini (2021)+T0),
85.0
86.0
85.2
78.5
76.5
76.0
Partial linear + T0g
83.9
87.6
85.7
76.6
82.1
79.3
77.7
76.5
77.1
80.7
Seq2seq
Integer free T0s
84.9
88.8
86.8
78.9
84.0
81.4
78.1
79.3
78.7
82.3
Full inear token action Tosn
85.9
88.6
87.2
79.6
83.5
81.5
78.9
78.0
78.5
82.4
Full linear + copy action T0
85.8
89.0
87.4
80.0
84.3
82.1
79.1
79.4
79.3
82.9
Full linear copy action Tope
86.1
89.2
87.6
80.6
84.3
82.4
78.9
80.1
79.5
83,2
Table 2: Results on the OntoNotes (CoNLL-12 12 English) test set. The average CoNLL F1 score of MUC, B and CEAF the main evaluation criterion. Models marked with are our implementation. marks models using additional training data.
antecedent relationships by computing span repre- sentation similarity at either span level (Lee et al., 2017, 2018; Joshi et al., 2019, 2020; Xu and Choi, 2020; Liu et al., 2022) or word level (Kirstain et al., 2021; Dobrovolskii, 2021). In contrast, Wu al. (2020) use QA model predict coreferent men- tion boundaries, while Xia et al. (2020), Toshniwal et al. (2020), and Toshniwal et al. (2021) use mem- ory augmented models.
ensure fairer comparison, we implement their method and include the results by training and eval- uating with larger model (T03g) using the same input length
(2048 tokens). All of our mod- els fall into this category. We include both full linearization models and partial linearization mod- els as baselines. For full linearization models, we consider variants that use token action sequence and copy action sequence.
Transition-based Seq2seq Models in this cat- egory are based on a designed transition system. Currently Bohnet et al. (2023) the only model in this category.
5.4 Results
5.4.1 English OntoNotes
Table shows our main results on the test portion of English OntoNotes. We first point that it is generally challenging ensure full comparabil- ity due to numerous ways the approaches differ. For instance, CorefQA (Wu et al., 2020) achieves strong performance with a relatively small model BSpanBERT--aarge,, 340M parameters), but it uses additional training data to optimize the mention proposal network (without which the performance drops 75.9) and is slow at test time because it runs extractive reader on each mention. On the other hand, the best results obtained with ASP (Liu al., 2022) and the transition- based system (Bohnet et al., 2023) rely on much larger models (FLAN-T5xII. 1IB parameters; mT5xxI. 13B pa-
Seq2seq Models in this category leverage pequence-to-sequene architecture to predict lin- earized coreference annotations without relying on coreference-speciff model architectures or special- ized system designs. Existing models in this cate- gory include Urbizu et al. (2020) and Paolini et al. (2021). However, Urbizu et al. (2020) do not evalu- ate standard coreference resolution datasets like Notes, and their performance is not competi- tive enough, we do not compare with them. On the other hand, Paolini et (2021) do not report performance using larger T5 models, and their in- put length is shorter than ours (1024 words). To

---

Model
PreCo LitBank LitBanko
close 83.3 obtained by 13B--paameterr transition- based system. While we do not have the computa- tional budget to train a 13B-parameter model, as a point of comparison, Bohnet et al. (2023) report the dev average F1 of 78.0 using a 3.7B- a. B-parameter m our model using the same base model with full linearization and copy action obtains 79.6.
Xia and Van Durme (2021)
88.0
76.7"
Thirukovalluru al. (2021)
78.4
and Gardner (2021)
85.0
Toshniwal et al. (2020)
76.5
Toshniwal al. (2021)
87.8
79.3
77.2
Toshniwal (2021 J+Jiint
87.6
78.2
Our copy action T0se
88.5
78.3
77.3
Our copy action T0se Joint
88.3
81.2
5.4.2 PreCo and LitBank
Table 3: Results on Preco, Litbank test set. The aver- age CoNLL Fl score MUC, B² and CBAFs, is the evaluation metric. We report both the 10-fold cross- validation results (official setting) and the results of split 0 (Littannk)) following Toshniwal al. (2021). Joint denotes training on the union of OntoNotes, PreCo and LitBanks. * marks transfer learning results which uses additional pretraining.
We further verify the effectiveness of our seq2seq model by taking the same 3B -parameter TO setting (full linearization, copy action) for OntoNotes to additional datasets in Table 3. We consider PreCo (Chen et al., 201 8) and LitBank (Bamman et al., 2020) following the same experimental setup Toshniwal et al. (2021). On PreCo, which pro- vides the largest training dataset (36K documents, compared to 2.8K in OntoNotes), our model out- performs all previous works at an average FI score of 88.5.
rameters), where running such large models is not always feasible depending on the authors' avail- able computational resources (e.g., we do not have the resources to scale to 13B parameters). We do our best interpret the results as objectively as possible?.
LitBank, on the other hand, is a very small dataset with only 100 annotated documents split into 80-10-10 for training, validation, and test, Its official evaluation metric is an average over 10-fold cross-vass validation results; we also report results on split O to compare with prior work. Despite the small training data, our model achieves competi- tive results of 77.3 on split 0 and 78.3 cross- validation, though lagging behind the task-specific model of Toshniwal et al. (2021). When the model is trained on the union of OntoNotes, Preco and LitBank split 0 training portion, the model achieves a significantly better performance 81.2, beating 78.2 in the previous work. This shows that (1) when there is sufficient training data, our seq2seq model can easily obtain tate-of-the-an results, and (2) even when that is not the case, the performance is still relatively strong and can be easily improved by including other available datasets.
The first observation is that models based on a sizeable T5 outperform those that are not (with the exception of CorefQA which uses additional train- ing data). Focusing on models with 3B parameters and input length 2048, our T0-based seq2seq model with full linearization and copy action achieves an average Fl score of 82.9, outperforming ASP (82.3) and TANL (79.6) using the same base model and input length. fact, our 3B-model outperforms ASP using a 11B model (82.5). While full lineariza- tion with copy action is the most performant, full linearization with token action (which requires no postprocessing during generation) is almost as com- petitive (82.4). Partial lincarization with token ac- tion using the alignment method in Section 3.4 also obtains a nontrivial Fl of 80.7. outperforming most non-seq2seq baselines. In terms of performance and speed, partial lincarization demonstrates lower accuracy but faster inference compared to full lin- earization. On the English OntoNotes development data, the inference time for partial lincarization is about 22 minutes, whereas full linearization takes approximately 40 minutes for both token action and copy action sequences.
5.5 Ablation Studies
In this section, conduct ablation studics to in- vestigate different aspects related to sequence rep- resentations, decoder input choices, and pretrained models of varying sizes. Unless specified, all the ablation experiments are conducted using the TO3B model.
With the BB-parameter T0pp initialization, our model reaches 83.2 F1, better than CorefQA and
5.5.1 Action Sequence
To analyze the impact of sequence representation, we present the results of an ablation study in Ta- ble 4.
'While the model sizes are similar, the comparison with Bohnet et al. (2023) is not fully equivalent due to their focus on multilingual coreference using the m model.

---

Sequence Representation
Avg. F1
Decoder Input
Avg. F1
Baseline
82.6
Baseline
82.6
Copy action sequence
56.4
Copy action + integer free
82.6
Token sequence + copy action sequence
82.5
Copy action + integer before
82.3
full linear
Token action integer
82.5
Table 5: Ablation study for decoder input on OntoNotes
Token action + antecedent string
79.5
development set. Average CoNLL F1 is reported.
Baseline
81.1
partial lincar
w/o sentence marker
79.9
Oracle align
99.2
Pretrained Model
params
Avg. F1
Oracle align w/o sentence marker
98.0
T5ese
220M
76.2
T5
770M
77.2
Table 4: Ablation study for sequence representations OntoNotes development set. Average CoNLL F1 is reported.
T338
3B
81.6
FLAN-T5KI.
3B
82.5
T0sa
82.6
FLAN-T5xoL
11B
82.9
T0pp
11B
83.0
Full Linearization The baseline for full lin- earization utilizes copy action sequence and repre- sents cluster identity using integers. In Section 3.3, we introduce an integer- freee representation which achieves performance comparable to the baseline (both achieving 82.6). Although the integer-free representation is more complex in design, it offers a cleaner and shorter sequence representation. No- tably, placing the integer before the mention string leads to a noticeable drop in performance (82.3 vs 82.6), emphasizing the importance of predicting cluster identity after detecting mentions due to the autoregressive nature of the seq2seq model. Addi- tionally, replacing the copy action in the baseline with a token action results in slightly worse per- formance (82.5 vs 82.6), indicating that a smaller action space is beneficial. Moreover, using the antecedent mention string to link coreferent men- tions (similar TANL (Paolini et al., 2021)) sig- nificantly decreases performance (79.5 vs 82.6). This demonstrates the superiority of using integers to represent cluster identity over using antecedent mention strings for linking coreferent mentions.
Table 6:
Ablation study for pretrained model on
OntoNotes development set. Average CoNLL F1 is reported.
These results validate the effectiveness sentence markers in alignment.
5.5.2 Decoder Input
We present an ablation study for decoder input in Table 5. The baseline uses linearized token quence as the decoder input. Replacing the token sequence with copy action sequence (similar Urbizu et al. (2020)) yields significantly worse per- formance compared to the baseline (56.4 vs 82.6). Averaging token and action embeddings as the in- put embedding is also less effective than the base- line (82.5 82.6). These results emphasize the importance of providing the decoder with lin- earized token sequence.
5.5.3 Pretrained Model
Table 6 shows an ablation study for pretrained model. We observe an improvement in perfor- mance the model size increases. For models of the same size, both FLAN-T5 and TO surpass the performance of the original T5 model. TO achieves better performance than FLAN-TS when compared at the same size.
Partial Linearization Partial linearization is in- compatible with the copy action since the model skips the gap between mentions. For partial lin- earization, the baseline employs a token action se- quence with explicit sentence markers. Sentence markers prove to be useful for the model, allowing it to focus on each sentence individually during generation and aiding in alignment. Removing sen- tence markers leads to significant deterioration in performance (79.9 81.1). To further understand the benefits of sentence markers for alignment, we conduct oracle experiments using gold lineariza- tion with and without sentence markers, obtaining average Fl scores of 99.2 and 98.0, respectively.
5.6 Error Analysis
To better understand the model behavior, we con- duct error analysis on the dev set in Table 7. The ex- periments are based the TO3B copy action model. The unlabeled mention detection F1 is 89.2. On the other hand, the clustering performance reaches 95.8 average Fl when restricted correctly recov- cred mentions, and 94.8 when we assume perfect mention detection. This shows that mention detec-

---

choice of representation evidenced in past works, However, once an effective representation is found, as the case in this work, we expect the perfor- mance on the task will only improve with future ad- vances in pretrained transformers. Another limita- tion, shared across all approaches built on seq2seq models, is the fact that training with language mod- eling loss is computationally intensive. Even with our substantial effort in scaling the model, with our budget we could not consider properly training models with more than 1 IB parameters. But we expect this limitation to benefit from the general effort in improving the efficiency of training and deploying transformers.
Mention detection
89.2
Detected mention clustering
95.8
Oracle mention clustering
94.8
Table 7: Error analysis on OntoNotes development set. We report mention detection Fl and mention clustering average CoNLL FI.
tion is the performance bottleneck; once correct mentions are obtained, the model can accurately infer coreference clusters. Upon qualitative anal- ysis of randomly sampled gold clusters and their best matches, we find that major source of error is annotation mistakes. For instance, one gold an- notation dictates "you do not want to [facel17 the dilemma. But [it]17 can not be avoided" while our model correctly predicts "you do not want to face [tee dilemmal20- But [it]20 can not be avoided"; another gold annotation dictates "[ [ William 19 and she Jio saw each other, was such wonderful reunion for [ them 110 to just hug, and he would hug her 12 and look at her 12 " while our model predicts "[ William lı and [ she 12 J12 saw each other, it was such wonderful reunion for [ them J12 to just hug, and he Iı would hug her 12 and look at her I2"
Acknowledgements
We thank the anonymous reviewers for their helpful comments.
References
David Bamman, Olivia Lewke, and Anya Mansoor. 2020. An annotated dataset of coreference in English literature. Proceedings of the Twelfih Language Resources and Evaluation Conference, pages 44554 Marseille, France. European Language Resources Association.
6 Conclusions
Bernd Bohnet, Chris Alberti, and Michael Collins. 2023. Coreference Resolution through a seq2seq Transition- Based System. Transactions the Association for Computational Linguistics, 11:212-226.
We have presented a highly performant seq2seq reduction of coreference resolution. Unlike in pre- vious works that rely on task-specific approaches to obtain strong performance, we use standard encoder-decoder model that receives the document as an input sequence and predicts a sequence repre- sentation of its coreference annotation as the target sequence. Contrary to previously reported weak results using seq2seq reductions, we show for the first time that with suitable definitions of the se- quence representation of the task, it is possible to achieve state-of-the-art performance, reaching 83.2 test average F1 English OntoNotes with an 1BB 11--arameter T0pp initialization. Our model's strong results fundamentally challenge the default spec-sp approach to coreference resolution.
Hong Chen, Zhenbua Fan, Hao Lu, Alan Yuille, and Shu Rong. 2018. PreCo: A large-scale dataset in preschool vocabulary for coreference resolution, Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 172- 181, Brussels, Belgium. Association for Computa- tional Linguistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling fantetueo language models. arXiv preprint arXiv:2210.1110 14416..
Angel Daza and Anette Frank. 2018. A sequence-to- sequence model for semantic role labeling. In Pro- The
ceedings of The Third Workshop on Representation Learning for NLP, pages 207-216.
Limitations
Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. 2021. Autoregressive entity retrieval. In International Conference on Learning Representa- tions.
While our approach is standard seq2seq modeling and does not require architectural modifications, it requires an effort in identifying an effective se- quence representation of coreference resolution; the model may perform poorly with a suboptimal
Vladimir Dobrovolskii. 2021. Word-level coreference resolution, In Proceedings of the 2021 Conference on

---

Empirical Methods in Natural Language Processing, pages 7670-7675.
coreference in ontonotes. In Joint conference on EMNLP and CoNLL-shares task, pages 1-40.
Osamu Gotoh. 1982. An improved algorithm for match- ing biological sequences. Journal of molecular biol- ogy, 162(3):75--708
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-texs trans- former. The Journal of Machine Learning Research, 21(1):5485-5551
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zeullemyer, and Omer Levy. 2020. Spanhert: Improving pre-training by representing and predict- ing spans. Transactions of the Association for Com- putational Linguistics, 8:64-77.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. 2020. 7ero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Com- puting. Networking, Storage and Analysis, pages 1- 16. IEEE.
Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel S Weld. 2019. Bert for coreference reso- lution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natu- val Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNI.P) pages 5803-5808.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowl- edge Discovery & Data Mining, pages 5505-3506.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: method for stochastic optimization. In ICLR (Poster).
Yuval Kirstain, Ori Ram, and Omer Levy. 2021. Coref- erence resolution without span representations. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 14-19. Online. Association for Computational Linguistics.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim. Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang. Mike Tian-Jian Jiang. Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Fevry. Jason Alan Frics. Ryan Techan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In International Conference on Learning Representations.
Kenton Lee. Luheng He, Mike Lewis, and Luke Zettle- moyer. 2017. End-to-end neural coreference reso- lution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to- fine inference. In P Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Papers), pages 687-692, New Orleans, Louisiana. Association for Computational Linguistics.
Raghuveer Thirukovalluru, Nicholas Monath, Kumar Shridhar, Manzil Zaheer, Mrinmaya Sachan, and An- drew McCallum. 2021. Scaling within document coreference to long texts. In Findings of the Associ- ation for Computational Linguistics: ACL-IJCNLP 2021, pages 3921-3931.
Tianyu Liu, Yuchen Eleanor Jiang, Nicholas Monath, Ryan Cotterell, and Mrinmaya Sachan. 2022. Autore- gressive structured prediction with language models. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 993-1005, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Shubham Toshniwal, Sam Wiseman, Allyson Ettinger. Karen Livescu, and Kevin Gimpel. 2020. Learning to ignore: Long document coreference with bounded memory neural networks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pages 8519-8526.
Shubham Toshniwal, Patrick Xia. Sam Wiseman, Karen Liveseu, and Kevin Gimpel. 2021. On generaliza- tion in coreference resolution. In Proceedings of the Fourth Workshop on Computational Models of Ref- erence, Anaphora and Coreference, pages 111-120, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Ci- cero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation be- tween augmented natural languages. In International Conference on Learning Representations.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll- 2012 shared task: Modeling multilingual unrestricted
Gorka Urbizu, Ander Soraluze, and Olatz Arregi. 2020. Sequence to sequence coreference resolution. In Pro- ceedings of the Third Workshop on Computational

---

Models of Reference, Anaphora and Coreference. pages 39-46, Barcelona, Spain (online). Association for Computational Linguistics.
2. Inside Mention: Open mentions exist (i.e., the count of separation tokens is less than that of mention start tokens <m>), but not in Inside Cluster Identity state.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. 2019, Huggingface' transformers: State-of- the-art natural language processing, arXiv preprint 1rXiv:190.03371.
Outside: Not in Inside Mention or Inside Clus- ter Identity states.
Token Action. Depending on the current state, different tokens are permitted:
Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. 2020. Corefqa: Coreference resolution as query- based span prediction. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 9953-6963.
Outside: The next token from the input source and the mention start token <m> are allowed.
The next token from the input source, the mention start token <m>, and the separation token are allowed.
Inside Mention:
Zhaofeng Wu and Matt Gardner. 2021. Understanding mention detector- leteetor-linker interaction in neural coref- erence resolution. In Proceedings of the Fourth Workshop on Computational Models of Reference, Anaphora and Coreference, pages 150-157.
Inside Cluster Identity: All integer tokens and the mention end token </m> are allowed.
Patrick Xia, João Sedoc, and Benjamin Van Durme.
2020. Incremental neural coreference resolution in constant memory. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP)), pages 8617-8624.
Copy Action. For the Copy Action model, trained to generate a copy action token <c> rather than the actual next source token, we manipulate the logits scores to enforce the generation of the actual source token. Specifically, the logits score of the actual next source token is set to that of the copy token <c>. The copy token <c> is then masked out. The rules for permissible tokens remain consistent with those for Token Action.
Patrick Xia and Benjamin Van Durme. 2021. Moving on from ontonotes: Coreference resolution model transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5241-5256.
Liyan and Jinho D. Choi. 2020. Revealing the myth of higher- order inference in coreference resolution. In Proceedings of the 2020 Conference on Empirical Methods Natural Language Processing (EMNLP), pages 8527-8533, Online. Association for Computa- tional Linguistics.
A.2 Partial Linearization
Due alignment issues with the input source partial linearization, it's infeasible to impose con- straints on source input token generation as in full linearization. Nonetheless, sentence-level con- straints can be applied by utilizing sentence bound- ary markers <sentence> and </sentence> in both the input source and lincarization. We identify the current state of the generated tokens in partial lin- earization based on sentence markers and integer cluster identity as:
Juntao Yu, Alexandra Uma, and Massimo Poesio. 2020. A cluster ranking model for full anaphora resolution. Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 1-20.
A Constrained Decoding
We implement constrained decoding through vo- cabulary masking, which depends on the current state of the generated sequence. Below, outline the masking rules for various models. For more details, please check our code.
Complete Sentence: Equal counts of sen- tence start <sentence> and end markers </sentence>.
A.1 Full Linearization
We categorize the state of the generated tokens thus far in full lincarization with integer cluster identity as follows:
2. Inside Sentence i: Within the i-th sentence (i.e., <sentence> count is and not Com- plete Sentence state).
1. Inside Cluster Identity: Cluster identity gen- eration stage (i.e., the count of mention end
Inside Cluster Identity: Cluster identity gen- eration stage (i.e., the count of mention end tokens </m> is less than that of separation to- ken ))
tokens </m> is less than that separation to- ken

---

4. Inside Mention: Open mentions exist (i.e., the count of separation tokens is less than that of mention start tokens <m>), but not Inside Cluster Identity state.
Inside Mention Seen 1: The next token from the input source, the mention start token <m?, and cluster identity hard-coded mention end tokens </mo?, </m:///m............ </m/?, </m/+1> are allowed.
5. Outside: Not in Inside Mention or Inside Clus- ter Identity states.
Outside Mention: The next token from the input source and the mention start token <m> are allowed.
Depending the current state, different tokens permitted:
Complete Sentence: only sentence start marker token <sentence> allowed.
Outside and Inside sentence i: All the tokens from the i-th sentence in the input source, the mention start token <m> and the sentence end marker token </sentence> are allowed.
Inside Mention and Inside Sentence i: All the tokens from i-th sentence the input source, the mention start token <m>, the sep- aration token and the sentence end marker token </sentence> are allowed.
Inside Cluster Identity and Inside Sentence i: All the tokens from the i-th sentence in the input source, all integer tokens, the mention end token </m> and the sentence end marker token </sentence> are allowed.
A.3 Integer-Free Representation.
the integer-free model, which is trained pre- dict <new> for unseen clusters instead of </m/+1>, where </mo>, </m]?,... </m/ have already ap- peared, we manipulate the logits scores to enforce the generation of the </m;11> token instead of <new> token for unseen cluster. Specifically, the logits score of the </m/+>> token is set to that of the <new> token. The <new> token is then masked We categorize the state of the generated to- kens thus far in full linearization with integer-free cluster identity as follows:
1. Inside Mention Seen I: Open mentions ex- ist (i.e., the count of cluster identity hard- coded mention end tokens </my> is less than that of mention start tokens <m>) and </mo>, </m/>//..............//>/ ...,, </m;> has been seen far.
2. Outside Mention: No open mentions.
Depending on the current state, different tokens are permitted:


Reviews:
[{"review_id": "ARmY9queAC", "reviewer": "Reviewer_Jhxb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the review and suggestions. \n-  Our method is fundamentally different from Bohnet et al., 2023. The core of the transition-based seq2seq model is the transition-based system, not the seq2seq model itself. In their paper, the seq2seq model serves to verbalize their coref-specific transition-based system. In contrast, our approach uses a standard seq2seq model to address coreference resolution by predicting a coreference annotation of the text directly.  The success of the transition system of Bohnet et al. does not imply the success of the general seq2seq system in our work, because the transition system is explicitly designed to solve the mention clustering problem, whereas the general seq2seq system is not.\n- We will release all our code.\n- We will incorportate the writing suggestions in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- A working \"linear\" seq2seq approach for ECR that beats SOTA and yet is simple\n- experiments on a broad range of datasets\n- a solid ablation study", "reasons_to_reject": "- The intro is very confusing without diagrams or examples. In general, the paper would read a lot better with fewer numbers\n- The seq2seq approach is fundamentally similar to Bohnet et al., 2023. The authors only provide contributions by simplistic variations of the decoding strategies and model attribute variations.\n- While the results on a spectrum of models of different sizes were interesting, an error analysis focusing on the gaps between the smallest and largest models would have been really useful. The error analysis in this paper seems limited.\n- The authors do not provide supplemental code to back the claims of the paper, given that the main contribution of this paper is an engineering experiment (could consider using anonymous GitHub)", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper presents a seq2seq transition-based reduction of entity coreference resolution (ECR) that the authors claim performs better than SOTA or at least matches it on numerous datasets. Furthermore, the authors conduct empirical analyses to theorize the impact of different model attributes and sequence generation strategies on coreference performance, employing a series of ablation experiments.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- lines 062-074: extremely difficult to follow without an example. I would suggest adding a full system architecture diagram with an example.\n- lines 075-096: too many numbers makes it really difficult to read. Maybe you can re-write it without the result numbers and summarize the key contributions (or the takeaways in your case)\n- lines 288, 289, 298, 299 might read better without the commas like in 217\n- Table 2 has too many numbers. Since you only talk about the CoNLL F1 in Results, it might be better if you move this table to appendix and use a condensed table with CoNLL F1 only. The freed up space could be then utilized for a better presentation of the system.\n- It would also be worth adding the LEA score if you want to talk about the precision and recall too specifically in your ablation study and error analysis. \n- Table 6 might look better and more informative as a plot (# params on x, performance on y)"}}, {"review_id": "zhjRNzUut9", "reviewer": "Reviewer_SffD", "summary": "", "questions": "A) L219: What specifically do you mean by a single pass?\n\nB) L327: What is the reason for choosing 0.0001?\n\nC) Table 1: are the values for # clusters correct? These are described as the average number of clusters in a document but e.g. PreCo has significantly more than 1.6 entity clusters per document. Maybe this is the average mentions per cluster?\n\nD) Table 1: from the numbers it appears that you are not using the original PreCo test set. How exactly did you split the dev and test splits for PreCo?\n\nE) What is the reason for using a batch size of 1 per GPU? Is this to accommodate the largest possible maximum input length? \n\nF) How long does it take to train the model in terms of iterations and wall-clock time? For the inference times given in minutes is this using the same number of GPUs as training?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the review and comments. \n\n**Responses to concerns**\n\n- In response to the reviewer's questions regarding the task-specific claims and our comparison with Bohnet et al, 2023, we clarify why we categorize Bohnet et al, 2023 as a task-specific model and outline the core distinctions between our methodology and that of Bohnet et al, 2023, as follows:\n    - The “model” in our claim refers to both the model architecture and the system design.  The method of Bohnet et al, 2023 is considered task-specific because of its task-specific system design, while all other baseline models are considered task-specific because of their unique task-specific model architecture. Our method doesn’t require any coref-specific architecture or system. All we need is a seq2seq model, which is general and task-agnostic, having been widely employed in a diverse array of tasks like machine translation and text summarization, which can be very different from each other by nature. Bohnet et al, while not needing a task-specific architecture, does necessitate a task-specific transition-based system, which is specifically designed and adapted for the coreference task described in the paper.  Due to the nature of coreference resolution that clusters mentions, the “Link” and “Append” actions in the transition-based seq2sq system are suitable and effective to cluster mentions together, that’s why it’s effective and achieves strong performance on coreference resolution. The success of the transition system of Bohnet et al. does not imply the success of the general seq2seq system in our work, because the transition system is explicitly designed to solve the mention clustering problem, whereas the general seq2seq system is not.\n   - The core of the transition-based seq2seq model is the transition-based system, not the seq2seq model itself. In their paper, the seq2seq model serves to verbalize their coref-specific transition-based system. In contrast, our approach uses a standard seq2seq model to address coreference resolution by predicting a coreference annotation of the text directly. \n\n- We will strengthen the error analysis section by looking into more error patterns in the final version. \n-  It’s well known that Seq2seq models, such as T5, necessitate an adequate model size for satisfactory performance. This observation is consistent across recent coreference papers, including Liu et al., 2022 and Bohnet et al, 2023.  However, once the model size reaches sufficient capacity, further enlarging it yields diminishing returns as performance plateaus. Despite scaling up task-specific models with ample computational resources, it's unlikely that they would surpass the performance of our approach or recent models based on sizeable (m)T5. \n- Our paper incorporates data on our model's inference speed. Ideally, we would also obtain the inference speed of baseline models by running them under identical settings to ours. However, this is not feasible as those baseline models either lack fully reproducible code, as is the case with Bohnet et al., 2023 and Wu et al., 2020, or their implementation is inefficient, as seen in Liu et al., 2022, which employs a rudimentary model parallel technique instead of a more efficient framework like Deepspeed we use.\n\n**Responses to questions**\n- A)  Our model runs in a single pass for each document, generating a target sequence all at once based on the entire input document. In contrast, Bohnet et al's approach processes one sentence at a time from a raw document, manually augmenting it for next sentences. For each raw sentence, the model predicts \"Link\" and \"Append\" actions based on that sentence, then manually augments it with the prediction. The augmented sentence is then used as history context for the following conditional generation.\n- B) The 0.0001 is a hyperparameter. We didn’t explore this much since we found the performance is stable and good when it’s small (<=0.0001). We’ll indicate it as a hyperparameter in the writing for the final version.\n- C) Thanks for correcting this mistake! It’s actually the average mentions per cluster. We will fix this in the final version.\n- D) We follow the setting of Toshniwal et al (https://aclanthology.org/2021.crac-1.12.pdf) and use the last 500 docs of the original training data as dev and the original dev data as test.\n- E) Yes, this is to accommodate the largest possible maximum input length.\n- F) The training time of 3 billion models is around 20 hours on 8 A100 40G GPUs and the training time of the 11 billion models is around 36 hours on 8 A100 80G GPUs.  The inference uses the same number of GPUs as training.\n\n**Presentation improvements**\n- Thanks for pointing out the OntoNotes annotation guidelines! We will fix this error in the future version by using this replacement example: The gold annotation omitting “he” and “she” is “as [[ William ]9 and  she ]10  saw each other, it was such a wonderful reunion for [ them ]10 to just hug, and he would hug [ her ]2 and look at [ her ]2 ” while our model correctly predicts  “as [ [ William ]11 and [ she ]2 ]12 saw each other, it was such a wonderful reunion for [ them ]12 to just hug, and [ he ]11 would hug [ her ]2 and look at [ her ]2”. \n- We will change the “large T5” wording to “sizeable T5” in the final version.\n- We will make Table 2 more clear in the final version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The proposed method achieves state-of-the-art results in all four training configurations including when trained on OntoNotes, the most well studied entity coreference evaluation.\n* The paper includes extensive baselines including training the seq2seq model of Paolini et al. (2021) with a larger T0-3B language model.\n* The writing is very clear and the methodology is convincing.\n* The paper shows the impact of various design decisions at a large scale (finetuning T0-3B) through both ablations and the comparison of proposed modeling decisions (partial linear vs full linear, token action vs copy action, integer vs integer free).", "reasons_to_reject": "* The paper is framed as providing evidence that task-specific models are not necessary for achieving state-of-the-art performance on coreference resolution; however, this finding is not surprising given the prior work of Bohnet et al. (2023) which achieved state-of-the-art performance on entity-based coreference resolution datasets, including OntoNotes, by formulating coreference resolution as a sequence-to-sequence task and finetuning T5 and MT5 on said formulation. Bohnet et al. (2023) was published in TACL on 14 March 2023 which was more than 3 months prior to the EMNLP submission deadline of 23 June 2023. My understanding is that this would therefore not be considered contemporaneous work as per the EMNLP 2023 CfP. The paper does compare against Bohnet et al. (2023), but this comparison should be more extensive to support the claim that the proposed method is not task-specific whereas Bohnet et al. (2023) is considered in the paper to be task-specific.\n* The provided error analysis is very brief with limited insight into model performance beyond the observation that mention detection is a bottleneck.\n* The evaluation is largely focused on F1 score as opposed to other dimensions of performance such as inference speed and model size. Based on Table 6, it appears that at comparable model sizes, task-specific models significantly outperform the proposed method.", "questions_to_authors": "A) L219: What specifically do you mean by a single pass?\n\nB) L327: What is the reason for choosing 0.0001?\n\nC) Table 1: are the values for # clusters correct? These are described as the average number of clusters in a document but e.g. PreCo has significantly more than 1.6 entity clusters per document. Maybe this is the average mentions per cluster?\n\nD) Table 1: from the numbers it appears that you are not using the original PreCo test set. How exactly did you split the dev and test splits for PreCo?\n\nE) What is the reason for using a batch size of 1 per GPU? Is this to accommodate the largest possible maximum input length? \n\nF) How long does it take to train the model in terms of iterations and wall-clock time? For the inference times given in minutes is this using the same number of GPUs as training?", "paper_topic_and_main_contributions": "This paper presents a supervised method for directly applying pre-trained sequence-to-sequence models to the problem of entity coreference resolution without the need for task-specific architectural components. The method functions by taking the input sequence to be the raw string of tokens composing a document, and the output sequence to be the document with special tokens marking entity boundaries and cluster identities. For example, the input sequence might be the tokenization of the string “Emily thinks she is tall” and the corresponding output sequence would then be the tokenization of “<m> Emily | 1 </m> thinks <m> she | 1 </m> is tall” where the integers after each mention indicate the cluster identity. The output sequence is decoded one token at a time by concatenating the previous output to the input sequence.\n\nThe paper explores possible decisions for how to represent mention’s cluster identity and how to represent the output sequence. For the output sequence, the paper shows that replacing non-special tokens with a “copy” token performs better than other methods. The paper experiments training the proposed model on OntoNotes, PreCo, LitBank, and a joint dataset of all three. For OntoNotes, the paper shows that finetuning T0-3B (3 billion parameters) and T0-pp (11 billion parameters) using the proposed method outperform existing methods. The best configuration achieves a CoNLL F1 score of 83.2 on OntoNotes.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* In the Error Analysis, I do not believe the “annotation mistakes” are necessarily mistakes but rather unintuitive results of the complexity of coreference annotation. For example, bare plurals in OntoNotes such as “students” are not annotated as coreferring except in certain cases such as when the antecedent of a pronoun which then starts a new coreference chain. While unintuitive, this example does appear to follow the OntoNotes annotation guidelines.\n* L464: The wording of “large T5” is confusing since there is a large-size version of the T5 model.\n* Table 2: It took me a while to figure out of the 5 models which action or linearization were being used when not specified."}}, {"review_id": "aRIlBPDKxk", "reviewer": "Reviewer_msDa", "summary": "", "questions": "First, this was a cool result! I had a couple questions.\n\n* A. The claim throughout the paper is that model size has a big impact. This is neither surprising nor novel – for years, we’ve seen the base LM scale up for coref and it leads to (free) improvements on the downstream task. Table 6 claims that this is yet another example of this phenomenon. How do we know it is actually model size and not pre-training data? Both FLAN-T5 and T0 are trained on substantially more data. I don’t think there is any test set contamination, but do we actually think it is model size? If so, why are the 11B models only slightly better than the 3B ones? \n\n* B. I offer a different hypothesis as to why seq2seq “suddenly” works now, and it isn’t entirely due to the linearization scheme (although that likely plays a big role). In Table 6, we see that T5_base/large do *worse* than SpanBERT-base/large or RoBERTa based models. This implies that had you done this study a few years ago, it would have been deemed unsuccessful. So why do you think seq2seq now works on the newer generation of models? \n    * i. There were already some hypotheses put forth in the paper (like how it needs to be more natural-language), it would be nice to have them organized, as a discussion/retrospective section, as to why seq2seq should be the path forward. \n* C. What do the sentence boundary markers look like? This isn’t mentioned in section 3.3/3.4 and it seems kind of important if it actually resolves many of the alignment errors.\n  * i. If you had oracle alignment, (how) would the final score improve? It might be possible to “cheat” and coerce the intended alignment out of the model by looking at the attention weights in the final layer.\n* D. The paper frequently emphasizes about not being “task-specific” – this doesn’t seem quite true and I suggest rephrasing this wording significantly? True to the definition in the paper, some training decisions still need to be made, like the hyperparameters for finetuning the T5/T0 model and max sequence length. These may need to be different depending on the size of the training set. Outside of this paper, a model that is not task-specific would be a general (coref) model that (at inference time) can predict properly on all tasks or datasets for the task, like a general \"NLI\" model or general QA model. If I understand the paper correctly, this paper is not claiming to be that omnibus model?\n* E. The treatment for CorefQA in this paper might actually be a little unfair? How is their method more expensive than these 3B/11B models that required lots of extra pretraining for T5/T0? I know that for a long time, papers wrote them off as being unreproducible, unfair training conditions, “need TPUs”, etc. But given the sizes of the models in this work, is CorefQA finally comparable on paper?\n* F. I'm still not clear on how <new> and <m_l> work. Does this imply that if a document has 100 clusters, there will be 100 special <m_l> tokens that are selectable by the decoder? I know this is rare, but there's no recency bias imposed to select between these 100, right?", "limitations": "", "rating": 5, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for the positive review and constructive comments. \n\n**Responses to the questions**\n- A.  T5-3B is substantially better than T5-large (77.2), and they are trained on the same C4 dataset. Model size becomes particularly significant when it is not adequately large. However, once the model size reaches or exceeds 3 billion parameters, further enlarging it yields diminishing returns as performance plateaus. \n- B. \n  - The main hypothesis for why seq2seq now works is that the model size is now sufficiently large. T5-3B achieves substantial improvements over T5-large, which is pretrained on the same C4 dataset. \n  - Thanks for your valuable suggestion! We will include a separate discussion section for the hypotheses in the final version.\n- C.  \n  - Two special tokens, \"<sentence>\" and \"</sentence>\", serve as sentence boundary markers, indicating the start and end of each sentence respectively in both the source and target sequences. During generation at inference time, the model determines the sentence index where the next token will belong by keeping track of the number of “<sentence>” and “</sentence>” tokens generated so far. The constrained decoding restricts the model to generating tokens that correspond to the same sentence index in the source sequence, as well as coreference-specific special tokens like \"<m>\", \"</m>\", \"|\", and integers. We will include the details of constrained decoding rules for all the linearization and action sequence variants in the Appendix in the final version. \n  - If oracle alignment were available, the final score would likely improve. However, it is doubtful that this would surpass the performance of the full linearization variants, given the reduced contextual information in the target sequence. The examination of attention weights in the final layer is really an interesting approach for alignment. Thanks for proposing that!  We will include this in the final version. \n- D.  The task-specific in our paper means that our approach is not task-specific since seq2seq is a general approach that can be used for many tasks. The specifically trained model is definitely task-specific. We are not aiming to train a general model like ChatGPT that can solve many tasks. Within the coref task, we trained dataset-specific coref models as well as a joint multi-dataset coreference model with our method. Table 3 presents the performance. The OntoNotes performance  (82.9) for the joint model will be included in the final version. When assessed on each respective training dataset, the multi-dataset joint model either surpasses or matches the performance of dataset-specific models.This shows the possibility of training a general coref model on sufficient training datasets with our approach. \n- E. CorefQA uses extra training data specifically for mention proposal training, a crucial element of their approach. This stage is more appropriately considered as additional fine-tuning rather than pretraining, which is commonly associated with general language model training. This distinction is a primary reason why recent studies employing large-scale models like 3B/3.7B/11B/13B (m)T5/T0, such as ASP, Bohnet et al 2023, and our work, often regard CorefQA as not directly comparable. Furthermore, CorefQA remains unreproducible. We plan to make all our code available to ensure reproducibility.\n- F. Due to the autoregressive generation process, if k clusters has been identified prior to a given step, then k+1 special mention end boundary tokens (\"</m1>\", \"</m2>\", ..., \"</mk>\", \"<new>\") become selectable options. If a document contains 100 clusters and all have been identified prior to a generation step, 100 special \"</m_l>\" mention end boundary tokens will be available for selection. No recency bias is applied when choosing among these 100 tokens.\n\n**Presentation improvements**\n\n- We will make a footnote that comparison is still not entirely equivalent for Bohnet et al., 2023. \n- We will release all the code for reproducibility. We will also include the constrained decoding details in the Appendix in the final version.\n-  Ideally, we would also obtain the inference speed of baseline models by running them under identical settings to ours. However, this is not feasible as those baseline models either lack fully reproducible code, as is the case with Bohnet et al., 2023 and Wu et al., 2020, or their implementation is inefficient, as seen in Liu et al., 2022, which employs a rudimentary model parallel technique instead of a more efficient approach like Deepspeed we use.\n- Thanks for pointing out the OntoNotes annotation guidelines! We will fix this mistake by using this replacement example: The gold annotation omitting “he” and “she” is “as [[ William ]9 and  she ]10  saw each other, it was such a wonderful reunion for [ them ]10 to just hug, and he would hug [ her ]2 and look at [ her ]2 ” while our model correctly predicts  “as [ [ William ]11 and [ she ]2 ]12 saw each other, it was such a wonderful reunion for [ them ]12 to just hug, and [ he ]11 would hug [ her ]2 and look at [ her ]2”. \n- We will tone down the claims in the future version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* The paper is scientifically comprehensive and easy to follow; where possible, the authors tried to make fair comparisons.\n* The authors give a timely demonstration that the seq2seq approach to coref, while not novel, can be highly competitive with much more room to explore.\n* This simple approach also shifts the paradigm of “task-specific” coref models to more general models, where coref can simply be another dataset, fine-tunable with standard T5/T0 hyperparameters and methods.\n\nNotably, even if there are major flaws/concerns with experimental design of the paper or reasons for the success of its method, it still presents a strong empirical result under the seq2seq paradigm, whereas many in the past have failed.", "reasons_to_reject": "There are some minor terminology issues described below which make the comparisons appear fairer than they really are. Further, some of the conclusions drawn about why seq2seq now suddenly works need more justification, and a couple sentences don’t appear correct to me.", "questions_to_authors": "First, this was a cool result! I had a couple questions.\n\n* A. The claim throughout the paper is that model size has a big impact. This is neither surprising nor novel – for years, we’ve seen the base LM scale up for coref and it leads to (free) improvements on the downstream task. Table 6 claims that this is yet another example of this phenomenon. How do we know it is actually model size and not pre-training data? Both FLAN-T5 and T0 are trained on substantially more data. I don’t think there is any test set contamination, but do we actually think it is model size? If so, why are the 11B models only slightly better than the 3B ones? \n\n* B. I offer a different hypothesis as to why seq2seq “suddenly” works now, and it isn’t entirely due to the linearization scheme (although that likely plays a big role). In Table 6, we see that T5_base/large do *worse* than SpanBERT-base/large or RoBERTa based models. This implies that had you done this study a few years ago, it would have been deemed unsuccessful. So why do you think seq2seq now works on the newer generation of models? \n    * i. There were already some hypotheses put forth in the paper (like how it needs to be more natural-language), it would be nice to have them organized, as a discussion/retrospective section, as to why seq2seq should be the path forward. \n* C. What do the sentence boundary markers look like? This isn’t mentioned in section 3.3/3.4 and it seems kind of important if it actually resolves many of the alignment errors.\n  * i. If you had oracle alignment, (how) would the final score improve? It might be possible to “cheat” and coerce the intended alignment out of the model by looking at the attention weights in the final layer.\n* D. The paper frequently emphasizes about not being “task-specific” – this doesn’t seem quite true and I suggest rephrasing this wording significantly? True to the definition in the paper, some training decisions still need to be made, like the hyperparameters for finetuning the T5/T0 model and max sequence length. These may need to be different depending on the size of the training set. Outside of this paper, a model that is not task-specific would be a general (coref) model that (at inference time) can predict properly on all tasks or datasets for the task, like a general \"NLI\" model or general QA model. If I understand the paper correctly, this paper is not claiming to be that omnibus model?\n* E. The treatment for CorefQA in this paper might actually be a little unfair? How is their method more expensive than these 3B/11B models that required lots of extra pretraining for T5/T0? I know that for a long time, papers wrote them off as being unreproducible, unfair training conditions, “need TPUs”, etc. But given the sizes of the models in this work, is CorefQA finally comparable on paper?\n* F. I'm still not clear on how <new> and <m_l> work. Does this imply that if a document has 100 clusters, there will be 100 special <m_l> tokens that are selectable by the decoder? I know this is rare, but there's no recency bias imposed to select between these 100, right?", "paper_topic_and_main_contributions": "This paper shows that linearizing coreference resolution problems and then using seq2seq models is highly performant. They argue that past models are all *task-specific* – i.e. having task-specific hyperparameters. Meanwhile, they present a method which linearizes the target sequence and guarantees well-formed output. They show competitive performance on several widely-used datasets and have ablations/analysis to explain the tradeoffs of certain annotation decisions and sources of errors.", "excitement": 5, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "* One of the limitations of this work is that it is only looking at English coreference. That’s fine, but Bohnet et al., 2023 is multilingual with mT5, and so that’s worth footnoting somewhere that even though model sizes are similar, comparison is still not entirely equivalent.\n* It could be useful for reproducibility to include the constrained decoding rules/grammar in the appendix.\n* This approach has a nice property that the runtime looks quite efficient (unless the alignment is really slow). With space, it would be nice to compare the asymptotic runtime of the full algorithm vs. some of the more recent models like Bohnet et al., 2023, Liu et al., 2022, and Wu et al., 2020, and Paolini et al., 2021.\n* L617-624: I believe in OntoNotes, generic mentions are not intentionally treated as *not coreferring*. It’s worth double-checking in their guidelines, but then this example about “students” should not be used as an annotation error. \n* There's something about this writing style/tone that reads a little arrogant/overconfident, especially given some of the questions/comments in my review -- specifically the final line \"Our model’s strong results fundamentally challenge the default task-specific approach to coreference resolution.\" even though prior work has aimed at non-task-specific models (e.g. Toshniwal et al., 2022's work that trained on multiple datasets, or Bohnet et al., 2023's work on a single multilingual model). Other examples of overconfident statements in intro/abstract, e.g. by claiming size is the main factor or that the model actually is not \"task-specific.\"\n"}}]

Author rebuttal:

Meta-review: The paper presents a seq2seq approach to coreference resolution. Unlike Bohnet et al (2023) which use a transition-based approach to processing a sentence at a time. The proposed approach directly predicts the documents with intext coreference annotations. The evaluation of a number of corpora showed results matching or exceeding the previous state-of-the-art results. The paper also reports results on the smaller LMs which is useful for researchers who do not have access to the resources needed for the large models. The main issue of the paper, however, is they made some strong claims to oversell the paper, e.g. claims on Bohnet et al (2023) are task-specific system etc. Apart from this, I would also expect the author to make the code freely available, and address other issues raised by the reviewers. A discussion on post-processing is also needed, given the LMs might not always generate texts following the input text. How you align the input and output can be a piece of useful information, e.g. Bohnet et al used the 3-word immediately after the mention to find the position of the mention.
Decision: Accept-Main