SlowGuess/NAACL_2021_OCR huggingface dataset:
2This third issue is the reason why (Vala et al., 2016; Yu et al., 2020a) used lenient metrics for scoring split-antecedent resolution, although ones that did not score single antecedent resolution as well.
3We can compute precision by switching the role of system and key entities in LEA computations.
Train/Dev	Test
Documents	353	60
Sentences	7524	1211
Tokens	195676	33225
Mentions	61671	10341
Singletons	26368	4158
Non-referring expressions	8159	1391
Single-antecedent anaphors	20127	3568
Split-antecedent anaphors	356	60
Split-antecedents	878	137
Importance As discussed, the number of entities that contain split-antecedents in our annotated data is negligible compared to entities with singular mentions. Therefore, we will not see a big difference in the overall score when the system resolves both singular and plural mentions. In order to put more emphasize on harder coreference links, i.e., resolving split-antecedents, we adapt the importance measure to assign a higher weight to entities containing split-antecedent as follows:
$$
\operatorname {i m p o r t a n c e} (e) = \frac {\operatorname {i m p o r t a n c e - f a c t o r} (e) * | e |}{\sum_ {e _ {i}} \operatorname {i m p o r t a n c e - f a c t o r} (e _ {i}) * | e _ {i} |}
$$
The importance-factor assigns  $\mathrm{Imp}_{split}$  times higher importance on plural entities compared to entities of singular mentions:
$$
\text {i m p o r t a n c e - f a c t o r} (e) = \left\{ \begin{array}{l l} \operatorname {I m p} _ {s p l i t} & \text {I f e i s a p l u r a l e n t i t y} \\ 1 & \text {I f e i s s i n g u l a r} \end{array} \right.
$$
We evaluated our system on the RST portion of the ARRAU corpus (Uryupina et al., 2020). ARRAU provides a wide range of anaphoric information (referring expressions including singletons and non-referring expressions; split-antecedent plurals; generic references; discourse deixis; and bridging references) and was used in the CRAC shared task (Poesio et al., 2018); RST was the main evaluation subset in that task; the RST portion of the ARRAU corpus consists of 1/3 of the Penn Treebank (news texts). Table 1 summarizes the key statistics about the corpus.
5.2 Separate and Joint Evaluation Methods
In separate evaluation, we follow standard practice to report CONLL average F1 score (macro average of MUC,  $\mathbf{B}^3$  and  $\mathrm{CEAF}_{\phi 4}$ ) for single-antecedent anaphors, and F1 scores for
Table 1: Statistics about the corpus used for evaluation.
Parameter	Value
BiLSTM layers/size/dropout	3/200/0.4
FFNN layers/size/dropout	2/150/0.2
CNN filter widths/size	[3,4,5]/50
Char/Glove/Feature embedding size	8/300/20
BERT embedding layer/size	Last 4/1024
Embedding dropout	0.5
Max span width (l)	30
Max num of clusters	250
Mention/token ratio	0.4
Optimiser	Adam (1e-3)
Non-referring method	Hybrid
Prefiltering threshold	0.5
Adjustment parameter (α)	0.01
Loss weight (β)	0.1
Table 2: Hyperparameters for our models.
non-referring expressions. For split-antecedent anaphors, we report three F1 scores: the strict F1 score that only gives credit when both anaphors and all their split-antecedents are resolved correctly; the lenient F1 score that gives credit to anaphors that resolved partially correct (Vala et al., 2016); and the anaphora recognition F1 score.
For joint evaluation of single/split-antecedent anaphors, we report the LEA score using the upgraded script described in Section 4.
We use the default parameter settings of Yu et al. (2020b) and use their hybrid approach for handling the non-referring expressions. The split-antecedent part of the system uses an FFNN with two hidden layers and a hidden size of 150. The negative example loss adjustment parameter  $\alpha$  and the loss weight parameter  $\beta$  (used for JOINT learning) are set to 0.01 and 0.1 respectively after tuning on the development set. Table 2 provides details on our parameter settings.
6.1 Separate Evaluation on
Single/Split-antecedent Anaphors
We first evaluate our two proposed systems in the separate evaluation setting, in which we report separate scores for single-antecedent anaphors, nonreferring expressions and split-antecedent anaphors. Showing individual scores for different aspects provide a clear picture of the different models.
Training settings In the JOINT setting, the system is trained end-to-end with a weighted loss func
Here we report F1 instead of accuracy used in Yu et al. (2020a) as our evaluation is based on predicted mentions.
CoNLL F1	Non-referring	Anaphora Rec_splt	Lenient_splt	Strict_splt
R	P	F1	R	P	F1	R	P	F1	R	P	F1
Recent-2	-	-	-	-	31.7	42.2	36.2	10.3	15.6	12.4	5.0	6.7	5.7
Recent-3	-	-	-	-	31.7	42.2	36.2	16.9	17.0	17.0	0.0	0.0	0.0
Recent-4	-	-	-	-	30.0	40.9	34.6	18.4	14.2	16.0	0.0	0.0	0.0
Recent-5	-	-	-	-	28.3	39.5	33.0	16.9	10.7	13.1	0.0	0.0	0.0
Random	-	-	-	-	31.7	42.2	36.2	5.9	3.7	4.5	0.0	0.0	0.0
JOINT	77.1	72.6	77.2	74.8	50.0	51.7	50.9	39.0	35.3	37.1	15.0	15.5	15.3
PRE-TRAINED	77.9	72.4	78.0	75.1	45.0	71.1	55.1	30.2	46.1	36.4	16.7	26.3	20.4
Table 3: Separate evaluation of our systems on the test set.  $\mathrm{X}_{split}$  are the scores for the split-antecedent anaphors.
\( \mathbf{Imp}_{split} = 1 \)	\( \mathbf{Imp}_{split} = 10 \)
R	P	F1	R	P	F1
Recent-2	70.5	66.9	68.7	61.5	61.3	61.4
Recent-3	70.5	66.9	68.7	61.6	61.1	61.4
Recent-4	70.6	66.9	68.7	61.8	61.1	61.5
Recent-5	70.5	66.9	68.7	61.5	61.2	61.3
Random	70.4	66.7	68.5	60.9	60.0	60.4
Our model	70.8	67.2	69.0	63.8	64.4	64.1
Table 4: LEA evaluation on both single- and split-antecedent anaphors.  $\mathrm{Imp}_{split}$  indicates the split-antecedent importance.
tion. In the PRE-TRAINED setting, we use the pretrained model provided by Yu et al. (2020b), and train only the split-antecedent part of the system.
Baselines Like (Vala et al., 2016; Yu et al., 2020a), we include baselines based on heuristic rules or random selection. For all baselines, we use the same model as used by our PRE-TRAINED approach to supply the candidate split-antecedent anaphors/singular clusters. The anaphora recognition baseline classifies as split-antecedent anaphors the discourse-new mentions belonging to a small list of plural pronoun (e.g., they, their, them, we). The recent-x baseline chooses the  $x$  closest singular clusters as antecedents for these candidates. The random baseline assigns two to five antecedents randomly to each chose split-antecedent anaphors.
Results Table 3 shows the comparison between our two systems and the baselines. Since plural pronouns are the most frequent split-antecedent anaphors, the simple heuristic gives a reasonably good F1 score of up to  $36.2\%$  for anaphora recog
nition. In term of the scores on full resolution, the baselines only achieved a maximum F1 of  $17\%$  and  $5.7\%$  when evaluated in the lenient and strict settings respectively. The low F1 scores indicate that split-antecedent anaphors are hard to resolve.
When compared with the baselines, both of our approaches achieved much better scores for all three evaluations. Our models achieved substantial improvements over the baselines of up to  $19\%$ ,  $19.9\%$  and  $14.7\%$  for anaphora recognition, full resolution (lenient and strict) respectively. The model trained in a JOINT setting achieves a better recall for both lenient evaluation and anaphora recognition, while the PRE-TRAINED setting has much better precision. We expect this is because the joint system could have an impact on candidate mentions/clusters, hence potentially recover more antecedent-anaphora pairs. By contrast, the candidate mentions/clusters are fixed in the PRE-TRAINED setting. Overall, the JOINT model achieves a slightly better lenient F1 score but a lower strict F1 score, whereas the PRE-TRAINED setting has a better overall performance when compared with the JOINT model. The JOINT system also has a lower CONLL average F1 score and non-referring F1 score when compared with the system trained in a PRE-TRAINED fashion. This indicates that jointly training is not helpful for the single-antecedent anaphors and non-referring expressions. Hence we use the PRE-TRAINED approach for further experiments.
6.2 Evaluating single and split antecedent anaphors jointly
We then evaluate our models with the newly extended LEA scores to show how split-antecedent anaphors could impact the results when evaluated together with single-antecedent anaphors. Table 4 shows the LEA score comparison between our best
6We also tried a random selection based approach, but such an approach only gets less than  $5\%$  split-antecedent anaphors correctly.
model (PRE-TRAINED) and the baselines. As only half of the test documents contain split-antecedent anaphors, we report the results on those test documents to give a clear picture on the evaluation.
We carried out two evaluations. The first setting is the traditional evaluation setting for coreference, in which split-antecedent anaphors are weighed equally as single antecedent anaphors (i.e., they are treated in LEA as a single mention,  $\mathrm{Imp}_{split} = 1$ ). We do not believe, however, that treating all anaphors equally is the most informative approach to evaluating coreference, for it is well-known that some anaphors are much easier to resolve than others (Barbu and Mitkov, 2001; Webster et al., 2018). LEA makes it possible to give more weight to anaphors that are harder to resolve. So in our second evaluation we give more importance to split-antecedent anaphors ( $\mathrm{Imp}_{split} = 10$ ) since they are much harder to resolve and also infrequent when compare to the single-antecedent anaphors. To have slightly higher importance for split-antecedents will give us a better view of their impact. The results in Table 4 show that our best model achieved moderate improvements of  $0.3\%$  -  $0.5\%$  on the first LEA score setting when compared with the baselines. This is mainly because the split-antecedent anaphors are less than  $1\%$  of the mentions. But ss expected, the improvements become more clear in the second evaluation setting, in which our model is  $2.6\%$  -  $3.7\%$  better than the baselines.
6.3 State-of-the-art Comparison
To compare with the state-of-the-art system on ARRAU, (Yu et al., 2020a), we train our best setting (PRE-TRAINED) as Yu et al. (2020a) did, i.e., assuming both gold mention and gold split-antecedent anaphors are provided. We first train the base model using gold mentions, then train the split-antecedent part of the system using gold mentions and gold split-antecedent anaphors. Since Yu et al. (2020a)'s system is evaluated on the full ARRAU corpus and with a customised train/test split priorities the split-antecedent anaphors, we retrain their system using the same standard RST split as used in our evaluation. We train their system with both baseline and the best settings using a single auxiliary corpus (SINGLE-COREF).<sup>7</sup> As shown in Table 5, our best model achieved both better lenient and
Lenient	Strict
R	P	F1	Accuracy
Yu et al. Baseline	61.0	52.5	56.5	21.7
Yu et al. Best model	69.1	63.9	66.4	35.0
Our model	71.3	65.1	68.1	45.0
Table 5: State-of-the-art comparison on the test set.
better strict accuracy than the Yu et al. (2020a) system, even though theirs is a dedicated system concerned only with split-antecedent anaphora. The results suggest the pre-trained mention/cluster representations are suitable for low-resource tasks that reply heavily on such representations.
6.4 Analysis
In this section, we carry a qualitative analysis on the system outputs to find out the main courses of the performance gaps between the gold and predicted settings. We also report a more detailed comparison between our system and the Yu et al. (2020a) system to see if there is a systematic difference between the two systems on the gold settings.
The Challenge of Using Predicted Setting The split-antecedent anaphora resolution task is more complex than its single-antecedent counterpart. The semantic relation between each individual antecedent and the anaphora is not identity, but element-of; and the number of antecedents can also vary. The results on evaluations with gold mentions and gold split-antecedent anaphors provided are promising. However, when evaluated using predicted mentions we have two main challenges: anaphora recognition and noisy candidate mentions/clusters. For anaphora recognition, our best model (PRE-TRAINED) only recalls  $45\%$  of the anaphors. The performance of our anaphora recognition is affected by the predicted mentions, and further capped by the fact that we only attempt to classify as split-antecedent the mentions classed as discourse-new by the base model. To assess the impact of these two factors, we computed the recall of split-antecedent anaphors by predicted mentions and discourse-new mentions. Virtually all split-antecedent anaphors are recalled among the predicted mentions  $-98.33\%$  -but only  $65\%$  are recalled among the discourse-new mentions. This has a big impact on our results for split anaphora recognition, since  $35\%$  of the anaphors are not accessible to our system. To understand the impact of this gap on the result, we supply to our system the  $98.33\%$  of split-antecedent anaphors recognized as
<sup>7</sup>The best setting, that uses multi-auxiliary corpora, is more complex to train and only moderately improves the results.
predicted. We keep everything else- predicted mentions, and clusters-unchanged. When run this way, our system achieves a lenient F1 score of  $47.7\%$  which is 11.3 p.p. better than the score  $(36.4\%)$  achieved using predicted anaphors, although still  $20.4\%$  lower than the model trained and evaluated with gold mentions and gold split-antecedent anaphors  $(68.1\%)$ . We suggest this additional difference is mainly a result of noise in the predicted mentions and clusters. Overall, then, the noise in the predicted mentions and clusters contributed  $2/3$  of the score difference, while problems with anaphora recognition are responsible for the rest.
In Depth Comparison with Yu et al. (2020a). Next, we compared our model's outputs in the gold setting with those of the best model of Yu et al. (2020a) in more detail. We split the test set in two different ways and compute the system performances on different categories. First, we follow Yu et al. (2020a) and split the split antecedent anaphors in the test set into two classes according to the number of gold split-antecedents: one class includes the anaphors with two split-antecedents, whereas the second class includes the anaphors with three or more split-antecedents (about  $23\%$  of the total). Table 6 compares these two classes. As we can see from the Table, with lenient evaluation the two systems work equally well for the anaphors with two split-antecedents, but our model is  $8.5\%$  better for mentions with three or more split-antecedents. In terms of strict evaluation, our model outperforms the (Yu et al., 2020a) model by  $8.7\%$  and  $14.3\%$  for two classes respectively. Overall, the model presented here achieved substantial performance gains on anaphors with three or more split-antecedents.
We then split the data into two classes according to a different criterion: the part-of-speech of the anaphor. The first class consists of pronoun anaphors, such as "they" or "their". The second class consists of all other split antecedent anaphors, such as "those companies" or "both". As shown in Table 7, the (Yu et al., 2020a) model achieves better scores for pronoun anaphors (mainly "they" and "their"). However, our new model outperforms the old system with non-pronominal anaphors by  $5.4\%$  according to lenient F1, and doubled their strict accuracy.
In this paper, we introduced a neural system performing both single and split-antecedent anaphora
Yu et al.	Our model
Count	Lenient	Strict	Lenient	Strict
2	46	71.9	45.7	70.9	54.4
3+	14	52.5	0.0	61.0	14.3
Table 6: Scores for anaphors with different number of antecedents.
Yu et al.	Our model
Count	Lenient	Strict	Lenient	Strict
PRP	24	82.4	58.3	76.4	54.2
Other	36	57.5	19.4	62.9	38.9
Table 7: Scores for pronoun and other anaphors.
resolution, and evaluated the system in a more realistic setting than previous work. We extended the state-of-the-art coreference system on ARRAU to also resolve split-antecedent anaphors. The proposed system achieves much better results on split-antecedent anaphors when compared with the baselines using heuristic and random selection when using the predicted mentions/clusters. Our system also achieves better results than the previous state-of-the-art system on ARRAU (Yu et al., 2020a), which only attempted single-antecedent anaphora resolution from gold mentions, when evaluated on the same task.
In addition, we also proposed an extension of the LEA coreference evaluation metric to evaluate both single and split-antecedent anaphors in a single metric.
Catalina Barbu and Ruslan Mitkov. 2001. Evaluation tool for rule-based anaphora resolution methods. In Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics, pages 34-41, Toulouse, France. Association for Computational Linguistics.
Kevin Clark and Christopher D. Manning. 2016a. Deep reinforcement learning for mention-ranking coreference models. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2256-2262, Austin, Texas. Association for Computational Linguistics.
Kevin Clark and Christopher D. Manning. 2016b. Improving coreference resolution by learning entity
level distributed representations. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 643-653, Berlin, Germany. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yufang Hou. 2020. Bridging anaphora resolution as question answering. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 1428-1438, Online. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference resolution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803-5808, Hong Kong, China. Association for Computational Linguistics.
Ben Kantor and Amir Globerson. 2019. Coreference resolution with entity equalization. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 673-677, Florence, Italy. Association for Computational Linguistics.
Varada Kolhatkar and Graeme Hirst. 2014. Resolving shell nouns. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 499-510, Doha, Qatar. Association for Computational Linguistics.
Varada Kolhatkar, Adam Roussel, Stefanie Dipper, and Heike Zinsmeister. 2018. Anaphora with non-nominal antecedents in computational linguistics: a Survey. Computational Linguistics, 44(3):547-612.
Heeyoung. Lee, Angel. Chang, Yves. Peirsman, Nathaneal. Chambers, Mihai. Surdeanu, and Dan. Jurafsky. 2013. Deterministic coreference resolution based on entity-centric, precision-ranked rules. Computational Linguistics, 39(4):885-916.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. 2017. End-to-end neural coreference resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing,
pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to-fine inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pages 687-692, New Orleans, Louisiana. Association for Computational Linguistics.
Quan Liu, Hui Jiang, Andrew Evdokimov, Zhen-Hua Ling, Xiaodan Zhu, Si Wei, and Yu Hu. 2017. Cause-effect knowledge acquisition and neural association model for solving a set of winograd schema problems. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelligence, IJCAI-17, pages 2344-2350.
Ana Marasovic, Leo Born, Juri Opitz, and Anette Frank. 2017. A mention-ranking model for abstract anaphora resolution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 221-232, Copenhagen, Denmark. Association for Computational Linguistics.
Nafise Sadat Moosavi and Michael Strube. 2016. Which coreference evaluation metric do you trust? a proposal for a link-based entity aware metric. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 632-642, Berlin, Germany. Association for Computational Linguistics.
Haoruo Peng, Daniel Khashabi, and Dan Roth. 2015. Solving hard coreference problems. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 809-819, Denver, Colorado. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Massimo Poesio and Ron Artstein. 2008. Anaphoric annotation in the ARRAU corpus. In Proceedings of