Investigating Multilingual Coreference Resolution by Universal Annotations
Authors: Haixia Chai
Michael Strube
Venue: EMNLP 2023 Findings (2023)
Keywords: Multilingual coreference resolution
coreference analysis

Abstract: Multilingual coreference resolution (MCR) has been a long-standing and challenging task. With the newly proposed multilingual coreference dataset, CorefUD (Nedoluzhko et al., 2022), we conduct an investigation into the task by using its harmonized universal morphosyntactic and coreference annotations. First, we study coreference by examining the ground truth data at different linguistic levels, namely mention, entity and document levels, and across different genres, to gain insights into the characteristics of coreference across multiple languages. Second, we perform an error analysis of the most challenging cases that the SotA system fails to resolve in the CRAC 2022 shared task using the universal annotations. Last, based on this analysis, we extract features from universal morphosyntactic annotations and integrate these features into a baseline system to assess their potential benefits for the MCR task. Our results show that our best configuration of features improves the baseline by 0.9% F1 score.

Investigating Multilingual Coreference Resolution by Universal Annotations
Haixia Chai and Michael Strube Heidelberg Institute for Theoretical Studies gGmbH
Schloss-Wool Woliossssolfsrrunnnneeee 35 69118 Heidelberg, Germany (haixia. michael.
Abstract
Recently, Nedoluzhko al. (2022) propose a new set of multilingual coreference datasets, Coref- UD, built upon the framework Universal De- pendencies (de Marneffe et al., 2021), allowing coreference researchers to conduct cross-linguistic studies across 17 datasets for 12 languages. The datasets serve as resource for the CRAC 2022 shared task multilingual coreference resolution (Zabokrtský and Ogrodniczuk, 2022). Given the harmonized universal morphosyntactic and coref- erence annotations, we raise the question whether there are any universal features that are common to all languages and to what extent they can contribute to the development of an MCR system.
Multilingual coreference resolution (MCR) has been long-standing and challenging task. With the newly proposed multilingual coref- erence dataset, CorefUD (Nedoluzhko et al., 2022), we conduct an investigation into the task by using its harmonized universal mor- phosyntactic and coreference annotations. First, we study coreference by examining the ground truth data at different linguistic levels, namely mention, entity and document levels, and across different genres, to gain insights into the char- acteristics of coreference across multiple lan- guages. Second, we perform an error amalysis of the most challenging cases that the SotA sys- tem fails to resolve in the CRAC 2022 shared task using the universal annotations. Last, based on this analysis, we extract features from universal morphosyntactic annotations and in- legrate these features into baseline system to assess their potential benefits for the MCR task. Our results show that our best configuration features improves baseline by 0.9% score.
In this work, we conduct an in-depth investiga- tion into the MCR task by using universal annota- tions in CorefUD. First, we analyze ground truth data from different linguistic levels, including men- tion, entity and document levels, and across differ- ent genres, to gain an understanding of coreference across various languages. Second, we conduct an error analysis of the most challenging cases that MCR systems fail resolve. Last, based on this analysis, we integrate several features extracted from universal morphosyntactic annotations into a baseline system to examine their potential benefits for the MCR task. To the best of our knowledge, our method represents the first attempt to leverage universal annotations for MCR.
1 Introduction
Coreference resolution is the task identify ex- pressions in a given text that refer to the same en- tity. While considerable progress has been made in coreference resolution for English (Lee et al., 2017, 2018; Joshi et al., 2019, 2020; Kirstain et al., 2021; Grenander et al., 2022), extending this task to multiple languages presents significant challenges due the linguistic diversity and complexity of different languages. The multilingual coreference resolution (MCR) task (Recasens et al., 2010; Prad- han et al., 2012) focuses on developing a general and robust system that can effectively handle mul- tiple languages and wide range of coreference phenomena (e.g., pronoun-drop).
Our findings reveal: (i) There are indeed com- monalities across languages. For example, we observe a common pattern where the closest an- tecedent of an overt pronoun mainly corresponds to the subject or object position. These common- alities are valuable for potential future rescarch, such as linguistic investigations aimed further comprehending the linguistic phenomenon of coref- erence. However, it important to note that explor-
"On of the benefits of Universal Dependencies is that it provides cross-linguistic guidelines for morphosyntactic annotation in a consistent and language-independeet manner.
OOu code and model are publicly available at https:// gi thub. cww//aom/////eeeeeoo xi www.aiiaaaamiiiiii core..

---

ing universal features is a challenging task due to the inherent variability among languages, e.g., the expression of definiteness. (ii) common issue encountered in all languages by MCR systems is the difficulty of correctly detecting nominal nouns within some two-mention entities. (iii) Our experi- mental results show that our best configuration of features improves the baseline by 0.9% Fl score.
specific linguistic phenomena, our research delves into gold annotations to explore multilingual coref- erence including phenomena like zero pronouns from a linguistic perspective, uncovering valuable insights to foster further research.
MCR Systems. In the past decade, numerous MCR approaches have been proposed, including rule-based approaches, various training methodolo- gies such as cross-lingual and joint training, and methods that leverage linguistic information: (i) Rule-based. It requires a complete redefinition of a set of rules to transform a monolingual coreference resolution system into multilingual one, for ex- ample when using Stanford's multi-pass sieve CR system (Lee et al., 2011). The adaptation process is time-consuming and requires a language expert to develop the rules. (ii) Translation- projec- tion. This is technique that involves the automatic transfer of coreference annotations from a resource- rich language to low-resource language using parallel corpora (Rahman and Ng, 2012; Chen and Ng. 2014; Martins, 2015: Novák al., 2017; Lapshinova- Kapstinova-Koltunski et al., 2019). The primary challenge of this approach the occurrence of a large number of projected errors, such as a nomi- nal phrase in English is translated as a pronoun in German. (iii) Latent structure learning. Fernandes et al. (2012) and Björkelund and Kuhn (2014) use latent structure perceptron algorithm to predict document trees that are not provided in the training data. These document trees represent clusters using directed trees over mentions. This approach has achieved the best results the CoNLL-2012 shared task for English, Chinese and Arabic at that time. (iv) Joint training. This is a technique that finetunes multilingual word embeddings on the concatena- tion of the training data in multiple languages. It allows the model to learn shared representations and help in cases where the target language has limited training data. (Straka and Straková, 2022; Pražák et al., 2021: Pražák and Konopik, 2022) (v) Methods with linguistic information. Several studies have incorporated syntactic and semantic information into their models (Zhou et al., 2011; Jiang and Cohn, 2021; Tan et al., 2021). These works either focus coreference resolution within single language or employ machine learning ap- proaches address the MCR task. Different from the above, our work incorporates universal mor- phosyntactic information into an end-to-end joint training method across multiple languages.
2 Related Work
Analysis in Multiple Languages. Coreference is a complex linguistic phenomenon that requires linguistic expertise, even more so when study- ing it in multilingual context. Oftentimes, re- searchers primarily focus investigating coref- erence within single target language in which they possess expertise, enabling them to gain valu- able insights specific to that language (Ogrodniezuk and Nitor, 2017; Urbizu et al., 2019; Sundar Ram and Lalitha Devi, 2020). However, a few studies have been conducted on coreference across mul- tiple languages by using multilingual coreference datasets (Recasens et al., 2010; Pradhan et al., 2012; Nedoluzhko et al., 2022). These studies include sta- tistical unalysis of the datasets (Nedoluzhko et al., 2022), as well as efforts to improve the perfor- mance and generalizability of MCR systems from a technical standpoint (Kobdani and Schütze, 2010; Björkelund and Kuhn, 2014; Straka and Straková, 2022). It is apparent that analyzing coreference across multiple languages is a challenging task due to the expertise required of each language. How- ever, CorefUD helps such analyses by providing universal annotations. Our work is the first at- tempt to analyze cross-linguistic patterns and gain a broader understanding of coreference across dif- ferent languages and language families in a com- prehensive and comparative manner.
In the field of MCR, there has been notable atten- tion directed towards the research of two types of languages. One prominent area of investigation is around pro-drop languages, such as Chinese (Kong and Ng, 2013; Song et al., 2020; Chen et al., 2021;
Zhang et al., 2022), Italian (Tida and Poesio, 2011 I) and Arabic (Aloraini and Poesio, 2020, 2021). An- other research dircction involves the study of mor- phologically rich languages, such German and Arabic (Roesiger and Kuhn, 2016; Aloraini and Poesio, 2021). In contrast to the aforementioned work, which primarily focuses on enhancing the model's capabilities through technical analysis of

---

3 Linguistic Analyses CorefUD 1.1
tr itcc
is the latest version of CorefUD (Nedoluzhko et al., 2022) for the CRAC 2023 shared task on multilingual coreference resolution, including 17 datasets for 12 languages." 4 In the fol- lowing subsections, we conduct linguistic study on it by using the ground truth from the training datasets, examining coreference phenomena from different linguistic levels, namely mention, entity and document perspectives, and across different genres, in multiple languages.
CorefUD 1.13
no bokmaalnarc
nynorsknarc
karker
nucrccor
en parcorfull
de potsdamce
parcorfull
fr democrat
lt.ccc
aancora
plupce
hi mil.stegedloooef
gimm
cs.pdt
poodt
ancora
40
8i
100
3.1 Mention
Proper Noun
Zero
A mention is the smallest unit within a coreference relation, comprising one or more words (maybe even less than a word in some cases).
Figure Percentage of mention types in datasets.
Position of Head. The head of mention typi- cally represents the entity being referred to. The remaining words in the mention either provide ad- ditional information that precedes the head word (pre-mot pore-modification, c.g., a highly radioactive ele- ment) or further specify the meaning of the head after it (post-modification e.g., car with leather seats.). Note that the modifying words can be dom- inant in the mention in some cases, e.g., the first floor, making resolution of those mentions harder sometimes.
main categories of mentions in most datasets. EN PARCORFULL (Lapshinova-Kolune Koltunski et al., 2018), DR_ 1E__ARCORFULLL (Lapshinova-Koltunk) et al., 2018) and FR_ [__DEMOCRAT (Landragin. 2021) are the datasets having most overt pronouns, around 46% of mentions. In contrast, resolving zero pronouns is more crucial in the Czech datasets (Nedoluzhko al., 2016; Hajië al., 2020) where the number of zero pronouns higher than that of overt pronouns.
Universal Dependency Categories. By using universal dependency (UD) relations between words in a sentence, we can understand the hier- archical structure of the sentence and identify the potential antecedents of referring expressions. We classify UD relations of heads of gold mentions into 12 categories according the UD taxonomy as illustrated Table 2.
ca
Cs
en
hu
pl
cs
13%
22%
27%
51%
14%
14%
It
fr
de
ru
no
tr
52%
28%
32%
24%
18%
52%
Table 1: Percentage of pre-modified mentions in the respective languages.
Table shows that Hungarian, Lithuanian and Turkish all have high percentage of pre-modified mentions. They are from the Uralic, Baltic and Turkic language families that are considerably dif- ferent from the other languages.
A naphor-Anteeedent Relation. Given mention types and UD categories presented above, we have a particular interest in analyzing the UD category of the closest antecedent to an anaphor based on its mention types (e.g., core arguments _urrttctssssuttteett overt pronoun). We consider all mentions in an entity observed anaphors, but exclude the first mention.
Mention Types. To gain insight into how men- tions represent and refer entities, we categorize five types of mentions by the universal part-of- speech (UPOS) tags of the head words in gold mentions, namely nominal noun, proper noun, overt pronoun, zero pronoun and others.
The results in Figure 2 present UD rela- tions that are most frequently associated with nominal noun and overt pronoun. We found that non-core dependents (e.g., oblique nomi- nal), nominal dependents (e.g.. numeric mod- index.
Unsurprisingly, in Figure 1, we observe that nominal noun and proper noun are the two
'See Appendix A the statistics CorefUD 1.1. 'https: //ffall mff, cuni, ct/ cz/corefud/crac23

---

tr
Juicc Do nynorshaar bokmaslnrr:
Do nynorkmare
no bokmaslnurr:
korker
hiLkoskar
nuncer
copparoduull
potslumer de parcodful
potslanet de parcorfull Iolooox
I Jemmara
lJcz es
cr acco
place
PLPCe
seggelkoree
M
M
%
soegedkord
8
M
e0 gm
en gum
rupd
pooct
poodt
(a) Nominal Noun
(h) Overt Pronoun
Figure Ranking of relations between UD categories antecedents and mention types of anaphors each dataset. For instance, figure (a) shows, in each row, UD categories (initials) ordered according to their frequency of association with nominal noun on a dataset. See Table 2 for the full name of UD categories.
UD CATEGORIES
UD RELATIONS
person concept. Each gold entity in all datasets of CorefUD 1.1 has 3 to mentions on average without considering singletons.
arguments_
subject (S)
nsubj
core arguments_ object (0)
The first mention within a men- tion chain serves to introduce the entity into context. Thus, this mention could be the most informative expression in the entity, In CA_ANCORA (Recasens and Martí, 2010), for ex- ample, 97% of first mentions belong to mention types of nominal noun or proper noun, which convey a richer semantic meaning than pronouns. Furthermore, we observe consistent trend across all languages that the ratio of entities with the first mention being the longest mention in the entity ranges from 70% to 90%.7 7 The longer mention is, the more information it represents, e.g., a person vs. a person that works at Penn. Overall, the first mention captures semantic meaning an entity.
First Mention.
obj. iobj
non-core dependents_ nominals (D)
obl, vocative, expl. dislocated
nominal dependents nominals (N)
mmod, appos, nummod
clauses (C)
esubj, ccomp, xcomp, advel, acl advmod, discourse, amod aux, cop mark det, clf, case conj. ce
modifier words (M)
function words (F)
conrdination (R)
MWE (W)
fixed, flat, compound parataxis
loose (L.)
special (P)
orphan, goeswith, reparandum punct, root, dep
other (T)
Table 2: Universal dependency categories.
ifier, nominal modifier and appositional mod- ifier), core arguments_objeee _objcctt__ttctttc are the primary UD relations of antecedents for nominal noun, e.g., Sam, my brother, John 's cousin, arrived. In contrast, the closest antecedents of overt pronoun mainly cor- respond to subjects objects within core argu- ments. It is important that these findings ap- plicable across all languages, emphasizing their universal relevance in the context of the multilin- gual coreference resolution task.
and core
Semantic Similarity. In addition to the first men- tion, entity accumulate information with each subsequent mention. The mentions can be identical, slightly different, or completely differentt when compared to other mentions within the same entity. To examine the semantic similarity coref- erent mentions, we compute the Euclidean distance between the embeddings of each gold mention pair encoded using mBERT (Devlin et al., 2019).
In Figure 3, greater distance indicates that the mentions have a bigger semantic distance, while still referring the same entity. Conversely, a smaller distance suggests that the mentions are semantically more similar, if not identical. We speculate that the genres of the datasets have an
3.2 Entity
In text, entity have multiple mentions all referring to the same identifiable object, such as
"See Appendix A.1 for the details of proper noun and zero pronoun.
"See Appendix A.2 for the details.

---

CB aanorra
liee
peede (400
no nymorsinnre
bokmaatnure
tu paurueor
loclor
pitt
co.pparcocull
(30
de parextull
Furkmoeaat
(2457)
(95)
bl pcu
spegeilooee
pl poc
MLL aecoriulll 1256)
eupooii
(2438)
amorra (74%)
de Exteddance
Faclidean Distamore
299%)
aeticofflll (103)
(3156) ft
Figure 3: Mean and variance of Euclidean distances be- tween pairs of mentions within the same entities across all datasets.
(40%)
Figure 4: Average number of competing antecedents for pronominal anaphors. The percentages next to the datasets represent the percentage of valid examinations of overt pronouns.
impact on the analysis above. For example, in narrative texts such as EU Bookshop publications in EN_ _NN_PARCORFULL (Lapshinova-Kolun Koltuuuk et al., 2018) and Hungarian Wikipedia in HU_KOR KOR (Vadász, 2020; Vadász, 2022), an entity can be real- ized with different expressions. Thus, the semantic similarity of mentions tends to be greater. Recall that nominal noun and proper noun categories of mention types. So, it is challenging resolve mentions that have bigger semantic dis- tance.
ambiguity caused by the presence of multiple potential antecedents from distinct entities or singletons. We focus on those ambiguous cases that have potential antecedents with gender and number agreement. Both the pronouns and their antecedents are located in the same or the immediately preceding sentence.
main
Figure 4 shows that in CA_ANCORA and ES AS_LAN (Recasens and Martí, 2010), over 70% of overt pronouns satisfy the analysis con- ditions mentioned in the previous paragraph. This percentage is notably higher compared to the other datasets. Additionally, the average number of com- peting candidates in these two datasets is around six. This highlights the considerable difficulty in distinguishing the true antecedent(s) the pronoun among a pool of antccedents. To address such com- plex scenarios, one heuristic and explainable ap- proach is to leverage centering theory (Grosz et al.. 1995). It suggests that a pronoun tends to refer to the center or the most prominent entity in the pre- ceding context. Specifically, by tracking the center transitions,
3.3 Document
a document, there can be multiple entities, with some entities spanning the entire document while others appearing only in very few adjacent sen- tences. Occasionally, these entities may overlap within certain sections of the document, particu- larly in areas where complex relationships between entities are discussed. Table 3 shows an example text.
Competing Antecedents of Pronominal Anaphors. In a local context, the resolution of pronouns can become difficult due to their
identify potential antecedents based on salience and continuity of the entity. Cen- tering theory applicable across all languages, as is not dependent on any specific language.
The study of how Ipeoplels, as [fans]s, access and man- age information within a transmedia system provides valu- able insight that contributes not only to [practitioners]; and [scholars of the media industryls. but to the wider context of cultural studies, by offering findings on this new model of [the fan]s as [consumer], and [information- user]s. For [us]:. as [digital humanists), defining [the "transmedia fan"); )2 is of particular relevance as [we]] seek to understand contemporary social and cultural trans- formations engendered by digital technologies.
Besides analyzing overt pronouns, we also exam- ine the competing antecedents for zero pronouns. In the Czech datasets (Nedoluzhko al., 2016; Ha- jič et al., 2020), the average number of competing antecedents less than four, which lower than that of CA_ CA_ORA and ES AN_ORAA 8 This im- plies that identifying the true antecedents of zero
Table 3: Text with indexed entities in the two adjacent sentences. All mentions in bold are valid competing antecedents of the pronominal anaphor in blue.
"See Appendix A.3 for the details.

---

anaphors is not very difficult in the Czech datasets. In pro-drop languages, a more coherent discourse tends to facilitate or encourage the use of zero pro- noun especially in dialogue or social media con- texts. We found that the nearest antecedents of some zero pronouns can either be overt pronouns or zero pronouns that are less informative. Hence, resolving anaphoric zero pronouns is difficult subtask that requires contextual information.
speech, indicating a strong continuity in reference. particularly related to the story's characters. This finding is in line with the results of Dorgeloh and Wanner (2022). As for written non-fiction, particu- larly in academic, news and voyage (describing a journey or trip), there is a lower use of pronouns, with academic texts showing the lowest frequency.
4 Error Analysis of MCR Systems
Apart from studying coreference on gold annota- tions solely, we also investigate the ground truth that the MCR systems failed to address. Our partic- ular focus is on two-mention entities, which com- prise over 80% of the gold entities where the recall is zero. 9 Here, we analyze the predictions of two MCR systems: BASELINE (Pražák al., 2021), an end-to-end based system, and ÚFAL (Straka and Straková, 2022), the winning system in the CRAC 2022 shared task on MCR. 10 Figure 6 presents the error analysis in a tree structure conducted on ÚFAL.
3.4 Genre
A document can be different in types of discourse with respect to referring expressions. For example, authors may use diverse expressions (c.g., dog own- ers, owners, puppy owners and they) when referring to the same entity for the physical continuity of the text. In contrast, spoken discourse, especially in conversations, tends to have higher density of re- ferring expressions, including many pronouns and ellipsis, which contribute to the grammatical coher- ence within the discourse (c.g., Sue? Is not here.), and relies mostly on shared situational knowledge between speaker and listener (known as the 'com- mon ground'). (Dorgeloh and Wanner, 2022)
4.1 Undetected Mentions
The primary factor leading to unresolved two- mention entities is the inability to detect one or both of the mentions. ÚFAL identifies 22% of the mentions, while BASELINE detects 19%. ÚFAL employs pipeline approach, treating mention de- tection as a separate token-level classification task. The proposed tags for tokens can handle embedded and also overlapping mention spans. We specu- late that the mention detection module contributes slightly more to the identification of mentions.
1000
Number of Persinal Pronouns
800-
600
400-
200-
0-
academic
comversation
bio
fiction
intarview
speech
tentbook
vlog
voyage
whow
We further analyze the mention types and length of the undetected mentions. 9 (i) More than 50% of the undetected mentions on average are nomi- nal nouns, so we try to analyze the types of these noun phrases based on definiteness, such as demon- strative articles (c.g., that house) and proper noun- modified noun phrases (e.g., Barack Ohama presi- dency). However, due to the highly variable nature of definiteness across languages and the lack of consistent annotations at this level of granularity, encounter challenge in implementing this anal- ysis. For example, in Lithuanian, definiteness is encoded within adjectives or nouns, and possessive adjectives in Hungarian can only be inferred from word suffixes. Moreover, some languages, such
Figure 5: Number of personal pronouns per 8000 words in various genres within EN_GUM (Zeldes, 2017 77).
In Figure 5, we present the frequency of personal pronouns usage per eight thousands words in each genres from the English corpus, EN_ __GUM (Zeldes, 2017). The results show that vlog, as a type of web discourse, has the highest frequency of pronoun usage. Different from conversation, content cre- ators record themselves on video for their audience without engaging in real-time interaction during the recording process. When they share their thoughts experiences, they tend to use first-person pro- nouns (e.g., / and we) more frequently compared to other genres. We also observe that the frequency of pronouns in fiction is high, surpassing even that of
'See Appendix for the details of the error analysis. 19The two system outputs from the development sets of CorefUD 1.0 are publicly accessible at https: //fall mff. cuni, cw/corefu//cracc/

---

Entities (recall=0) 39%
Two-mention Entities (81%)
Undetected Mentions (78%)
Missing Links
Length (in Words)
Position of Head
Anaphor-Antecedent
Distance (in Sentences)
1-2 (55%)
Mention Types
Mention Types
UD Categories Mention Types
premodification
(37%)
(39%)
nominal noun (62%)
overt
(53%) postmodification
pronoun
(23%)
nominal noun- nominal noun (40%)
nominal dependents- nominal noun
subject in core
proper
arguments- pronoun
pronoun
Figure 6: Error analysis of entities where ÚFAL fails to resolve, meaning that the recall of these entities zero. For example, 81 % of unresolved entities consist of two mentions. One of the reasons for the failure to resolve two-mention entities is that 78% of the mentions within those entities are not detected. The figures are computed on average across all datasets.
Slavic ones, do not have grammaticalized defi- niteness at all. (ii) Analyzing mention length, observe that the majority of mentions in Hungarian (70%) and Lithuanian (80%) consist of only one or two words. One of the reasons is that Hungar- ian, for example, is an agglutinative language"!
is important to improve the capability of resolving noun phrases. Lastly, examine the gold anaphor- antecedent relations between the mentions of the unresolved entities. We found that the most fre- quent UD relation associated with the antecedents of nominal nouns are nominal dependents (e.g.. nominal modifier and appositional modifier). For antecedents of overt pronouns, the subject in core arguments is the most common UD relation.
When dealing with such languages, is plausible to include a preprocessing stage to handle word splitting.
4.2 Missing Links
Modeling with Universal Annotations
We also explore the relationship between the two mentions the unresolved entities." 9 First, we no- tice that in BASELINE, more than 45% of the en- tities have both mentions located the same sen- tence. To resolve those entities, syntax information that captures the grammatical relationships and de- pendencies between words within the sentences is beneficial. One approach is employing binding theory (Chomsky, 1993). On the other hand, in ÚFAL, 39% of the entities have their two mentions spanning across multiple sentences. To address this issue, an approach is to use knowledge extracted from the discourse structure of the text. Second, for both systems, resolving cases where both men- tions are nominal nouns presents difficulties across all languages. Additionally, our analysis in Sec- tion demonstrates that there are mention pairs referring the same entities, but showing lower semantic similarity, These findings suggest that
Based on the findings above, we can gain additional insights and clues regarding MCR. For example, we found that the closest antecedents of overt pro- nouns are always located in subject position. This pattern is common in nearly all languages shown linguistic infor-
in Figure (b). Therefore, mation extracted from universal annotations for the purpose of modeling and examine its effectiveness, in the following section.
5.1 Model
Baseline. We adopt the model proposed by Pražák et al. (2021) as our base model, which is an end-to-end neural model inspired by the method introduced by Lee et al. (2017). It serves as the baseline for the CRAC 2022 shared task multi- lingual coreference resolution.
Incorporating Linguistic Information. Given an input document consisting of n tokens, we first generate a contextual embedding for each token using mBERT denoted as X (X1, Xz). The tokenization is based on either word forms (wf) or
" Words are constructed by combining stem forms with multiple affixes to convey diverse grammaticall features such as tense and number, for example, beleselkedtem (I look into) and Odafigyelhettel volna (You could have paid attention to it).

---

CS
DE
MODELS
AVG
CA
HU
Pl.
L.T
FR
RU
PCED
PDT
GUM
PARC
POTS
PARC
BASELINE
53.7
55.2
68.4
64.3
48.8
46,4
50.2
57.6
64.2
57.0
33.7
43.3
43.0
66.9
lem
51.7
52.8
65.0
62.7
48.1
44.0
44.3
54.6
60.2
56.7
30.6
40.8
48.4
64.5
ours wf
54.6
55.7
68.5
64.9
50.1
47.1
50.4
57.7
62.1
58.6
35.1
44.9
48.5
66.5
ua
-0.51
-0.03
-0.23
0.00
-0.75
-0.21
-0.72
40.34
+1.57
-1.78
+0.81
-2.90
-4.04
+1.36
lang
-0.87
-0.45
-0.11
-0.65
-1.29
-0.71
-0.19
-0.14
+2.09
-1.62
-1.44
-1.61
-5.58
+0.40
Table F1 scores on the test set in our setting are reported on average across three runs. & rules out the additional
features extracted from universal annotations (ua) and languages (lang) from wf for the ablation study.
lemmas (lem). Then we define the embedding of each candidate span e as:
Training and Inference. Since ÚFAL (Straka and Straková, 2022) demonstrated that a multilin- gual model based on multilingual language model outperforms monolingual models on the MCR task, adopt similar approach. Our model is jointly trained mixture of datasets of 10 languages from CorefUD 1.0 (Nedoluzhko et al., 2022) us- ing mBERT (Devlin al., 2019) as the pretrained language model. Then we use this trained model to predict mention clusters on the target language- specific datasets.
e Xeatart Xeend & (se))
where Xestart and Xeend denote the embeddings of the boundary tokens. & is the addition of atten- tionally weighted token representations in the can- didate. (s,,) a concatenated feature vector that includes the width, UPOS tags, UD relations, men- tion types and UD categories the span. We select the token with the maximum attention weight as the head of the candidate to compute the mention types and UD categories discussed in Section 3. We measure how likely candidate is mention by using a mention score fm(():
5.2 Experiments
Settings. We verify the effectiveness of our mod- els on CorefUD 1.0 (Nedoluzhko et al., 2022). Be- cause the test datasets are not publicly available, partitioned approximately 10% of the training datasets to create are reported using the CoNLL Fl score - the av- erage of MUC (Vilain et al., 1995), B3 (Bagga and Baldwin, 1998), CEAFe (Luo, 2005). The final ranking score is calculated by macro-averaging the CoNLL Fl scores over all datasets. To ensure a fair comparison, we keep all parameters the same as the baseline (Pražák et al., 2021). All our exper- iments are performed V100 32G GPU. We examine two models, namely ours_ wf and ours Jem, discussed in Section 5.1, in comparison with the baseline model trained on our specific setting.
(c) = FFNN (lec, due)))
test datasets. The results
where (lu)) encodes the UPOS tag, UD relation, mention type and UD category of the candidate determined by its 'head' word as mentioned above.
After extracting the top An mentions based on the mention score, compute the likelihood of candidate mention being antecedent of query mention y by a scoring function (c.g)).
single NVIDIA Tesla
f :) = FFNN, Cg. ee Gg ((c,
((c, q) denotes the embeddings of some general features the document: language and word order of the language For each query mention, our model predicts a distribution P(g) over its candi- dates, E (c):
Table presents our results. Our model ours_w wf shows a modest improvement over the baseline with a margin of 0.9% Fl score on average across all languages. The model performs best on Germanic datasets, whereas the I.T I.C_ (Zitkus and Butkiene, 2018) and RU RUCOR (Toldova et al., 2014) datasets present the greatest difficul- ties, indicating that these two Baltic and Slavic languages are particularly difficult to handle. In the ablation study, we observe that including general features like language and word order also yields
Results.
exp(f(c,g) q))
P(g) EkeY cxp(((ck))
Note that if the query mention is singleton, set the scoring function zero. 13
12 "htttss//wall" info/
13 For more details, please refer to the original papers, Pružák et al. (2021) and Lec et al. (2017).

---

positive effects on performance, in addition to in- corporating universal annotations.
6 Discussion and Conclusion
It has become apparent that leveraging universal morphosyntactic annotations can be advantageous in various ways, like exploring underlying patterns of coreference, performing in-depth analysis and making a contribution to the development of an MCR system. However, there are still language- specific characteristics that hinder the comprehen- sive study of multiple languages together, particu- larly when it involves analyzing intricate aspects of the morphological layer, like definiteness and compound nouns in German. In addition, while multilingual datasets are harmonized to some ex- tent, there are still cases where certain information, such as entity types, is only provided for a limited number of languages. This limitation prevents us from conducting further analyses, such as examin- ing semantic class agreement across languages. We study MCR primarily focusing identity corefer- ence since it is the most important relation across all datasets. However, there exist various other anaphoric relations, such as bridging and discourse deixis (Yu et al., 2022). that remain unexplored. In this work, we analyze coreference across multiple languages by leverag- ing the harmonized universal morphosyntactic and coreference annotations in CorefUD. This analysis provides valuable insights into common features and challenges in MCR. We demonstrate the bene- fits of incorporating linguistic features for enhanc- ing the MCR system performance.
In contrast, the performance of ours Jem shows decline compared with BASELINE. The method is specifically designed to address data spar- sity and handling out-of-vocabulary words in morphological
languages. However, lemmati- zation can result in different words being mapped to the same lemma and loss of valuable morpho- logical information present in word forms. In order to handle multiple languages together, it is crucial to employ trade-off strategy to implement preprocessing approach.
Error Analyses. We employ the same analysis methodology as presented in Section 4 for the error analysis of our model ours_ wf and BASELINE in our setting. We found that ours_ wf predicts more clusters correctly than BASELINE, either in full or partially (i.e., the rate of gold entities with recall of zero is lower average, 39.19% vs. 39.77%.). entities are the most difficult cases for the two examined systems. In these unresolved two- mention entities, ours_wf has fewer undetected mentions on average especially in [R_DEMOCRAT and DE_PARCORFULL, as illustrated in Figure Among those undetected, there are more mentions consisting of more than two words compared with BASELINE. For the missing links, the two systems produce similar results. Both mentions in two- mention entities are primarily nominal nouns. And the most frequent UD relation associated with the antecedent of nominal still nominal dependents. Overall, our model ours_ ours_wf can resolve slightly more entities and shows a superior performance in mention detection compared with BASELINE. Nevertheless, there is still room for improvement.
important to note that
Limitations
In this work, our analyses are mainly corpus- based studies. The reliance on selected specific corpora may result in a focus on particular genres, domains, or time periods that may not be representative of other contexts. However, with the high number of datasets from diverse genres and domains, we believe the findings still can provide some valuable insights into MCR. The languages examined in our study belong to the European language group. would be interesting to involve languages from other regions, like Arabic and Chinese.
Percent of undetected mentiens
Bapsiie
80
40
aLancora
Suecest
hu, Regertiore
en gum
Al pic
ancora
tic
democrat
narcorrut
de nott mece
en percertat
Pu rucor
Acknowledgements
We thank the anonymous reviewers for their helpful feedback that greatly improved the final version of the paper. We also thank Margareta Kulcsar for her early experiments contributing to this work.
Figure 7: Percentage of undetected mentions in unre- solved cro-mention entities.

---

This work has been funded by the Klaus Tschira Foundation, Heidelberg, Germany. The first author has been supported by a HITS Ph.D. scholarship.
Heidrun Dorgeloh and Anja Wanner. 2022. Discourse Syntax. Cambridge University Press.
Eraldo Fernandes, Cícero dos Santos, and Ruy Milidiú. 2012. Latent structure perceptron with feature in- duction for unrestricted coreference resolution. Joint Conference on EMNLP and CoNLL - Shared Task, pages 41-48, Jeju Island, Korea. Association for Computational Linguistics.
References
Abdulrahman Aloraini and Massimo Poesio. 2020. Anaphoric zero pronoun identification; A multilin- gual approach. In Proceedings of the Third Workshop on Computational Models of Reference, Anaphora and Coreference, pages 22-32, Barcelona, Spain (on- line). Association for Computational Linguistics.
Matt Grenander, Shay B. Cohen, and Mark Steedman. 2022. Sentence-incrrmeentl neural coreference res- olution. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. pages 427-443, Abu Dhabi, United Arab Emirates. Association for Computational Linguisties.
Ahdulrahman Aloraini and Massimo Poesio. 2021. Data augmentation methods for anaphoric zero pronouns. In Proceedings of the Fourth Workshop Computa- tional Models of Reference, Anaphora and Corefer- ence. pages 82-93, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Barbara Grosz, Aravind K. Joshi, and Scott Weinstein. 1995. Centering: A framework for modeling the local coherence of discourse. Computational Linguistics. 21(2):203-223
Amit Bagga and Breck Baldwin. 1998. Algorithms for scoring coreference chains. In The First Inter- national Conference on Language Resources and Evaluation, pages 563-566, Granada, Spain.
Jan Hajië, Eduard Bejěek, Jaroslava Hlavacova, Marie Mikulová, Milan Straka, Jan Štěpánek, and Barbora Štěpánková. 2020. Prague dependency treebank consolidated 1.0. In Proceedings of the Twelfth Lan- guage Resources and Evaluation Conference, pages 2208-5215 Marseille, France. European Language Resources Association.
Anders Björkelund and Jonas Kuhn. 2014. Learn- ing structured perceptrons for coreference resolution with latent antecedents and non-local features. In Proceedings of the 52nd Annual Meeting of the As- sociation for Computational Linguistics (Volume 1: Long Papers), pages 47-57, Baltimore, Maryland. Association for Computational Linguistics.
Ryu lida and Massimo Poesio. 2011. A cross-lingual ILP solution to zero anaphora resolution. In Proceed- ings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Tech- nologies, pages 504-813, Portland, Oregon, USA. Association for Computational Linguisties.
Chen Chen and Vincent Ng. 2014. Chinese overt pro- noun resolution: A bilingual approach. In Proceed- ings of the AAAI Conference on Artificial Intelligence, volume 28, pages 1615-1621.
Fan Jiang and Trevor Cohn. 2021. Incorporating syntax and semantics in coreference resolution with hetero- geneous graph attention network. In Proceedings of the 2027 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 1584-1591, Online. Association for Computational Linguistics.
Shisong Chen, Binbin Gu, Jianfeng Qu, Zhixu Li, An Liu, Lei Zhao, and Zhigang Chen. 2021, Tackling zero pronoun resolution and non-zero coreference resolution jointly. In Proceedings of the 25th Confer- ence on Computational Natural Language Learning. pages 5 518-527, Online. Association for Computa- tional Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span- BERT: Improving pre- pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Noam Chomsky. 1993. Lectures on Government and Binding. De Gruyter Mouton, Berlin, New York.
Cathe--Ctteeinee de Marneffe, Christopher D. Man- ning, Joakim Nivre, and Daniel Zeman. 2021. Uni- versal Dependencies. Computational Linguisties, 47(2):255-308
Mandar Joshi, Omer Levy, Luke Zettlemoyer, and Daniel Weld. 2019. BERT for coreference reso- lution: Baselines and analysis. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5803-5808, Hong Kong, China. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-48 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Yuval Kirstain, Ori Ram, and Omer Levy. 2021. Coref- erence resolution without span representations. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language

---

Processing (Volume 2:
Short Papers), pages 14-19, Online. Association for Computational Linguistics.
André F. T. Martins. 2015. Transferring coreference re- solvers with posterior regularization. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1427-1437, Beijing, China. Association for Computational Linguistics.
Hamidreza Kobdani and Hinrich Schütze. 2010. SU- CRE: A modular system for coreference resolution. In Proceedings of the Sth International Workshop on Semantic Evaluation, pages 92-95, Uppsala, Sweden. Association for Computational Linguistics.
Anna Nedoluzhko, Michal Novák, Silvie Cinková, Marie Mikulová, and Jiří Mírovský, 2016. Corefer- ence in Prague Czech Banch-hnglihhhh Dependency Treebank. In Proceedings of the Tenth International Conference Language Resources and Evaluation (LREC' 16), pages 169-176, PortoroZ, Slovenia. European Lan- guage Resources Association (ELRA).
Fang Kong and Hwee Tou Ng. 2013. Exploiting zero pronouns to improve Chinese coreference resolution. In Proceedings of the 2013 Conference on Empiri- cal Methods in Natural Language Processing, pages 278-288, Seattle, Washington, USA. Association for Computational Linguistics.
Frédéric Landragin. 2021. Le corpus Democrat et son exploitation. présentation. Langages, (224):11 1-24.
Anna Nedoluzhko, Michal Novák, Martin Popel. Zdeněk Zabokrtsky. Amir Zeldes, Daniel Zeman, Peter Bourgonje, Silvie Cinková, Jan Hajič, Chris- tian Hardmeier, Pauline Krielke, Frédéric Landra- gin, Ekaterina Sapshinova-Koltunaa Martí, Marie Mikulová, Maciej Ogrodniczuk, Marta Recasens, Manfred Stede, Milan Straka, Svetlana Toldova, Veronika Vineze, and Voldemaras Zitkus. 2022. Coreference in universal dependencies 1.0 (CorelUD 1.0). LINDAT/CLARIAT brary at the Institute of Formal and Applied Linguis- tics (ÚFAL), Faculty of Mathematics and Physics, Charles University.
Ekaterina Lapshinova- KapshinovaKKottuskk,, Christian Hardmeier, and Pauline Krielke. 2018. ParCorFull: parallel corpus annotated with full coreference. In Proceed- ings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018), Miyazaki, Japan. European Language Resources As- sociation (ELRA).
M. Antônia
digital li-
Ekaterina Lapshinova-Ke Koothinova-Koltunkk,,, Sharid Lodiciga, Christian Hardmeier, and Pauline Krielke. 2019. Cross-lingual incongruences in the annotation of coreference. In Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference, pages 26-34, Minncapolis, USA. Association for Computational Linguistics.
Michal Novák, Anna Nedoluzhko, and Zdeněk Zabokrt- sky. 2017. Projection-based coreference resolution using deep syntax. In Proceedings of the 2nd Work- shop on Coreference Resolution Beyond OntoNotes (CORBON 2017), pages 56-64, Valencia, Spain. As- sociation for Computational Linguisties.
Heeyoung Lee, Yves Peirsman, Angel Chang, Nathanael Chambers, Mihai Surdeanu, and Dan Ju- rafsky. 2011. Stanford's s multi-pass sieve coreference resolution system at the CoNLL-2011 shared task. In Proceedings of the Fifteenth Conference on Compu- tational Natural Language Learning: Shared Task, pages 28-34, Portland, Oregon, USA. Association for Computational I Linguistics.
Maciej Ogrodniczuk and Bartiomiej Nitoó. 2017. Im- proving Polish mention detection with valency dictio- nary. In Proceedings of the 2nd Workshop on Corefer- ence Resolution Beyond OntoNotes (CORBON 2017), pages 17-23, Valeccia, Spain. Association for Com- putational Linguistics.
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettle- moyer. 2017. End-to-end neural coreference reso- lution. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 188-197, Copenhagen, Denmark. Association for Computational Linguistics.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. CoNLL- 2012 shared task: Modeling multilingual unrestricted coreference in OntoNotes. In Joint Conference on EMNLP and CoNLL Shared Task, pages 1-40, Jeju Island, Korea, Association for Computational Lin- guistics.
Kenton Lee, Luheng He, and Luke Zettlemoyer. 2018. Higher-order coreference resolution with coarse-to- fine inference. In Proceedings of the 2018 Confer- ence of the North American Chapter of the Associ- ation for Computational Linguistics: Human Lan- guage Technologies, Volume 2 (Short Pupers), pages 687-692, New Orleans. Louisiana. Association for Computational Linguistics.
Ondřej Pražák and Miloslav Konopik. 2022. End-to- end multilingual coreference resolution with mention head prediction. In Procerdings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution, pages 27- Gyeongju, Republic of Korea. Associa- tion for Computational Linguistics.
Xiaoqiang Luo. 2005. On coreference resolution perfor- mance metrics. In Proceedings of Human Language Technology Conference and Conference on Empiri- cal Methods in Natural Language Processing, pages 25-32.
Ondfej Pražák, Miloslav Konopík, and Jakub Sido. 2021. Multilingual coreference resolution with harmonized annotations. In Proceedings of the International Con- ference on Recent Advances in Natural Language Processing (RANLP 2021), pages 1119-1123.

---

Altaf Rahman and Vincent Ng. 2012. Translation-oon based projection for multiligual coreference resolution. In Proceedings of the 2012 Conference of the North American Chapter of the Association for Computa- lional Linguistics: Human Language Technologies, pages 720-730, Montréal, Canada. Association for Computational Linguistics.
Gorka Urbizu, Ander Soraluze, and Olatz Arregi. 2019. Deep cross- -ons-ll coreference resolution for less- resourced languages: The case of Basque. In Pro- ceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference, pages 35-41, Minneapolis, USA. Association for Computational Linguistics.
Marta Recasens, Lluís Mårquez, Emili Sapena, M. Antô- nia Marti, Mariona Taulé, Véronique Hoste, Massimo Poesio, and Yannick Versley. 2010. SemEval-2010 task I: Coreference resolution in multiple languages. In Proceedings of the 5th International Workshop on Semantic Evaluation, pages 1-8, Uppsala, Sweden. Association for Computational Linguistics.
Noémi Vadász. 2022. Building a manually annotated Hungarian coreference corpus: Workflow and tools. In Proceedings of the Fifth Workshop Computa- rional Models of Reference, Anaphora and Corefer- ence, pages 38-47, Gyeongju, Republic of Korea. Association for Computational Linguisties.
Marta Recasens and M Antônia Martí. 2010. Ancora- Co: Coreferentially annotated corpora for Spanish and Catalan. Language Resources and Evaluation, 44:315-345.
Noémi Vadász. 2020. KorKorpusz: kézzel annotált, többrétegū pilotkorpusz építése. In XVI. Mag- yar Számitógépes Nyelvészeti Konferencia (MSZNY 2020), pages 141-154, Szeged. Szegedi Tudománye- gyetem, TTIK, Informatikai Intézet.
Ina Roesiger and Jonas Kuhn. 2016. IMS HotCoref DE:
A data-driven co-reference resolver for German. In Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC'16), pages 155-160, Portorož, Slovenia. European Lan- guage Resources Association (ELRA).
Mare Vilain, John Burger, John Aberdeen, Dennis Connolly, and Lynette Hirschman. 1995. A model- theoretic coreference scoring scheme. In Sixth Mes- sage Understanding Conference (MUC-6): Proceed- ings of a Conference Held in Columbia, Maryland, November 6-8, 1995.
Linfeng Song, Kun Xu, Yue Zhang, Jianshu Chen, and Dong Yu. 2020. ZPR2: Joint zero pronoun recovery and resolution using multi-task learning and BERT. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 5429- 5434, Online. Association for Computational Lin- guistics.
Juntao Yu, Sopan Khosla, Ramesh Manuvinakurike,
Lori Levin, Vincent Ng, Massimo Poesio, Michael Strube, and Carolyn Rosé. 2022. The CODI-CRAC 2022 shared task on anaphora, bridging, and dis- course deixis in dialogue. In Proceedings of the CODI-CRAC 2022 Shared Task on Anaphora, Bridg- ing, and Discourse Deixis in Dialogue, pages 1-14. Gyeongju, Republic of Korea. Association for Com- putational Linguistics.
Milan Straka and Jana Straková. 2022. ÚFAL CorPipe CRAC 2022: Effectivity of multilingual models for coreference resolution. In Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Res- olution, pages 28-37, Gyeongju, Republic of Korea. Association for Computational Linguistics.
Zdeněk Žabokrtský and Maciej Ogrodniczuk, editors. 2022. Proceedings of the CRAC 2022 Shared Task on Multilingual Coreference Resolution. Association for Computational Linguistics, Gycongju, Republic of Korea.
Vijay Sundar Ram and Sobha Lalitha Devi. 2020. Han- dling noun-noun coreference in Tamil. In Proceed- ings of the WILDRE5- 5th Workshop on Indian Lan- guage Data: Resources and Evaluation. pages 20-24, Marseille, France. European Language Resources As- sociation (ELRA).
Amir Zeldes. 2017. The GUM Corpus: Creating Mul- tilayer Resources the Classroom. Language Re- sources and Evaluation, 51(3):881-612.
Fiona Anting Tan, Sujatha Das Gollapalli, and See- Kiong Ng. 2021. NUS-IDS at CASE 2021 task I: Improving multilingual event sentence coreference identification with linguistic information. In Proceed- ings of the 4th Workshop Challenges and Appli- cations of Automated Extraction of cocio-political Events from Text (CASE 2021), pages 105-112, On- line. Association for Computational Linguistics.
Shulin Zhang, Jixing Li, and John Hale. 2022. Quan- tifying discourse support for omitted pronouns. In Proceedings of the Fifih Workshop on Computational Models of Reference, Anaphora and Coreference, pages 1-12. Gyeongju, Republic of Korea. Associa- tion for Computational Linguistics.
Huiwei Zhou, Yao Li, Degen Huang, Yan Zhang, Chun- long Wu, and Yuansheng Yang. 201 1. Combining syntactic and semantic features by SVM for unre- stricted coreference resolution. In Proceedings of the Fifteenth Conference on Computational Natural Language Learning: Shared Task, pages 66-70, Port- land, Oregon, USA. Association for Computational Linguistics.
Svetlana Toldova, A. Roytberg, Alina Ladygina, Maria Vasilyeva, Ilya Azerkovich, Matvei Kurzukov, G. Sim. D.V. Gorshkov, A. Ivanova, Anna Nedoluzhko, and Y. Grishina. 2014. Ru-eval-2014 4: Evaluating anaphora and coreference resolution for Russian. Komp' 'juternjjaj Lingvistiku Intellek- tual'nye Tehnologit, pages 68 1-69.

---

Voldemaras Zitkus and Rita Butkiene. 2018. Corefer- ence annotation scheme and corpus for lithuanian language. In 2018 Fifth International Conference on Social Newworks Analysis, Management and Security (SNAMS), pages 243-250.
68. RECOWA
(71%
A Linguistic Analyses on CorefUD 1.1
cxppolt
(40%
We present the statistics CorefUD 1.1 Table to provide a basic understanding of all the datasets.
A.1 Anaphor-Antecedent Relation
pa (38%)
Figure 8 demonstrates the analysis of anaphor- antecedent relations where anaphors are proper noun and zero pronoun.
Figure 9; The figure displays the average number competing antecedents for zero pronouns. The percent- ages indicated next to the datasets represent the percent- age of valid examinations zero pronouns.
w_hcc
lockor
nLrucor
B Error Analysis
de_ oo_sotstance
de parcorfall
Figure presents the percentages of mention types of undetected mentions based on the predictions of ÚFAL.
55 ancor
pl. pet
bu stagelooree
poed
G 200022
ru.rucor
parcorfull
(a) Proper Noun
de patsdaadmm de parcorfull fr democrat
is mynnnnsemmee
bokmaslnant
lochor
lt.
tn parcororfall
ancora
de poballent
60 parcorfall
pl.pcc
democral
szegedkoref
33,00002
en gum
pce
cs.pdt
bn saggerrro
gum
pcedt
caLancora
200002
20
64
80
100
Nominal Noun
Overt
Other
(b) Zero Pronoun
Proper
Figure 8: The figure presents the ranking of relations be- tween UD categories antecedents and mention types of anaphors in each dataset. The mention types include proper noun (a) and zero pronoun (b). See Table 2 for the full name UD categories.
Figure 10: Coverage of mention types in the undetected mentions divided percentage for ÜFAL
show the distances between mentions in the unresolved wwo-mention entities for BASELINE and ÚFAL
In Figure 11,
A.2 First Mention
Table 7 shows various analyses conducted to ex- plore the underlying reasons of unresolved entities in both BASELINE and ÚFAL systems.
Table shows the statistics of first mentions that are the longest mentions in entities.
A.3
Competing Antecedents of Pronominal Anaphors
Figure/ shows the analysis of competing an- tecedents zero pronouns on three languages, Catalan, Czech and Spanish.

---

Does
Tokens/Sent Sents/Doc
Entities
Mentions
Mentions/Enttty
CA ANCOOA
1,011
10.52
30.39
13,589
48,323
3.56
CS SCPEC
1,875
21.24
23,42
39,945
136,932
3.43
CD_PDT
2,533
15.29
16.85
36,378
120,024
3.30
EN EN_GUM
151
56.61
17.05
5,816
25,615
4.40
HU_ SZEGEDKOREF
320
22.31
14.08
3,862
12,278
3.18
PC_PCC
1,463
19.63
15.03
17,748
65,915
3.71
ES "SNCOAAA
1,080
10.50
31.63
15,532
56,668
3.65
LC_LCC
80
16.63
22.62
879
3,609
4.11
FR_ [__DMMCCRAT
50
207.64
21.58
5,718
38,490
6.73
DE_ PARCORFULLL
15
30.47
18.93
192
737
3.84
DE POTSDAMCC
142
12.80
14.68
715
2,027
2.83
EN PARCORFULL
15
30.47
19.18
158
710
4.49
RU RUCOR
145
48.75
17.48
2,803
12,509
4.46
HU KORKOR
76
14.29
17.91
874
3,178
3.64
NO BOKMAALNARC
284
46.02
15.55
4,647
21,828
4.70
NO_NYNON NYNORSKNARC
336
30.71
16.74
4,307
18,354
4.26
TR IR_IIC
19
185.89
12.46
523
2,602
4.98
Table 5: Statistics CorefUD 1.1 based on number of documents (Docs), average number of sentences per document (Sents/Doc), average number of tokens per sentence (Tokens/Sent), number of entities, number of mentions, and average number mentions per entity (Mentions/Eatiity) in each of the datasets.
nunnoor
mr.pororfull
demorrst
atoora
Entities (%)
place
oo
ANCORA
76.92
PCEDT
87.10
rowh
75.22
GUM
69.55
201
60
69
30
100
HU SU_SSZEGEDKORF
80.11
Intermmeenta [-
interentental
PC_
83.13
ES ANCORA
77.25
neor
LI LCC
$1.80
purcrfall
FR DEMOCRAT
82.46
potalamss
DE PARCORFULL
88.02
de.poncorful
DE POTSDAMCC
77.48
fr.
dederccrat
EN PARCORFULL.
89.24
Jocor
RU RUCOR
83.41
HU KORKOR
82.27
saggtioon
NO B0_BOMMAALAARC
81.62
NO WO_NYNORSKNARC
78.
pood:
TR TTCC
70.36
ancora
201
Table 6: Percentage of first mentions that are the longest mentions within entities corresponding datasets.
Figure 11: Coverage of distance between mentions in the unresolved two-mention entities divided in percent- age. The top figure for BASELINE, and the bottom one corresponds to ÚFAL.

---

C
UFAL
ÚFAL
ÚFAL
UFAL
ÚFAL
ÚFAL
CALANCORA
36.20
CS. PCEDT
33,80
20.71
88.15
M8.49
77.74
68.83
41.
36.90
53.70
59.5
7.74
6.35
CS. PDT
35,50
23.56
84.70
88.17
82.78
79.22
46.60
54.90
$5.2
5.87
EN GUM
38.85
25.04
75.19
83.33
86.45
79.66
48.
50.60
61.30
65.4
4.87
4.83
HO SZEGEDKOREF
39.23
34.01
80,92
K3.33
78.21
82.80
69.70
70.20
56.60
53.1
2.38
2.32
PL PCC
36.97
23.05
85.78
90.25
83.07
18
61.20
31,00
31.8
4,05
ES. ANCORA
35,50
16.33
85.29
90.37
81.32
78.52
32,00
37.20
68. 20
66.3
9.29
7.78
FR DEMOCRAT
43.22
25.20
76.18
87.63
85.19
76.07
52.80
56.30
10
37.3
3.62
3.61
DE. PARCORFULL
45.45
36.36
90.00
87.50
72.22
64.29
$3.50
66.70
46.20
55.6
3.54
DE POTSDDACCC
50.59
27.06
69.77
65
98.33
84.09
47.50
$1,40
74.60
75.7
3.58
3.87
EN PARCORFULL
43.75
37,50
85.71
100.00
66.67
75.00
37.50
55.60
62.50
44.4
6.25
5.44
RUCOR
22.17
Table 7: The table shows the results of the error analysis on BASELINE (BL) and ÚFAL across all datasets
CorefUD 1.0. All results presented percentages excluding F. The initials for the sub analyses stand for: A: Unresolved entities (recall=0). B: Two-mention entities in the unresolved entities. C: Udetected mentions in the unresolved two-mention entities. Mentions that only have one or two words in the undetected mentions. E: Undetected mentions that pre-mooe me-moiiifiattins. F: Average length of mentions that are not detected.


Reviews:
[{"review_id": "1BZ5uf6vkG", "reviewer": "Reviewer_qVbW", "summary": "", "questions": "(A) Why didn't you include your own system in the error analysis, or the system of Straka and Strakova in the Sec. 6 comparison?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper and your valuable feedback. \n> no provision is made to tease apart confounders such as genre/domain of the texts, differences in annotation styles (even within a unified scheme - this was found in UD as a significant source of variation) etc.\n\nWe will conduct an additional genre/domain analysis on a few datasets, such as English-Gum (Zeldes, 2017), as only a limited number of datasets have genre/domain annotations in CorefUD. As for the differences in annotation styles, our work mainly focuses on analyzing the linguistic phenomenon of coreference, based on the multilingual coreference dataset proposed by Nedoluzhko et al. (2022). In Nedoluzhko et al. (2022), the authors have already discussed a lot about the diverse annotation schemes employed by all teams.\n\n> Leaving out the common theme of using the CorefUD corpus, the three parts of this article are pretty much independent of each other rather than building on a common technical contribution - in particular the system building experiment in section 5 does not in any way profit from the corpus analysis or error analysis sections.\n\nActually, the findings of the analyses in the first two parts are the foundation of our proposed model. For example, we found that the closest antecedents of overt pronouns are always located in subject positions. This pattern is common in nearly all languages showing in Figure 2 (b). Our model benefits from those patterns by incorporating features of UD categories and mention types. We will make a more explicit connection between our analyses and the modeling process in the final version. \n\n> the quantitative analysis stays at the level of what could be done in a few days by taking at face value the corpus annotations rather than looking more deeply into possible confounders.\n\nTo be honest, we spent 2 months on the first part of the analysis. We conducted numerous preliminary analyses to understand the datasets first and refined our analysis method in iterative rounds. From these, we present only the most valuable analyses in the paper. Beyond the workload, we would emphasize the analysis methods we employed, such as anaphor-antecedent relation and competing antecedents of pronominal anaphors, which require linguistic expertise and a lot of experience in the field of coreference studies. Please refer to the first answer for the concern of confounders.\n\n> Similarly the error analysis in section 4 states relatively obvious facts that are going to affect any SotA coreference system\n\nActually, we uncovered new findings through our error analysis. For example, when examining anaphor-antecedent relationships, we found the most frequent UD relation associated with the antecedents of nominal nouns are nominal dependents (e.g., nominal modifier and appositional modifier).  To the best of our knowledge, this paper is the first work to conduct an error analysis from a linguistic perspective across 10 languages. Our approach is supported by substantial experiments and detailed results. \n\n> it presents the rule-based approach of Lee et al 2011 as \"Template-based\" which I don't find very helpful as a term\n\nWe agree with this point and will replace ‘Template-based’ with ‘Rule-based’ in our final version. \n\n> in the subsequent text fails to cleanly separate multilingual (training data in multiple languages) from cross-lingual (training data in one language and testing in another language) tasks or even mixes training methodology (latent structure, joint training) and features (methods with linguistic information).\n\nIn the final version, we will reorganize the section of MCR Systems in Related Work by providing categories of training methodologies and approaches that incorporate linguistic information. Under the training methodology category, we will have two additional subcategories: cross-lingual approaches and multilingual approaches.\n\n> Why didn't you include your own system in the error analysis\n\nWe will include this error analysis following the method outlined in section 4 in the final version where we will have one more page available.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper produces several angles of analysis which together help us learn more about the CorefUD corpus than was available before", "reasons_to_reject": "Leaving out the common theme of using the CorefUD corpus, the three parts of this article are pretty much independent of each other rather than building on a common technical contribution - in particular the system building experiment in section 5 does not in any way profit from the corpus analysis or error analysis sections. If one wanted to argue against this paper, it is that neither of the sections has the contribution height for being accepted as a main conference paper - the quantitative analysis looks like it stays at the level of what could be done in short time by taking at face value the corpus annotations rather than looking more deeply into possible confounders (and I believe the authors if they say this part has gotten substantial attention and rounds of refinement - in UD it took a village of experts several releases to unearth cross-language inconsistencies); The error analysis in section 4 states relatively obvious facts that are going to affect any SotA coreference system, while this is a truism for error analyses this is expected to benefit from the cross-system comparison with their own system that the authors mention in the rebuttal.\nIn the related works presentation, the classification of MCR systems seems a bit confused - it presents the rule-based approach of Lee et al 2011 as \"Template-based\" which I don't find very helpful as a term, and in the subsequent text fails to cleanly separate multilingual (training data in multiple languages) from cross-lingual (training data in one language and testing in another language) tasks or even mixes training methodology (latent structure, joint training) and features (methods with linguistic information).", "questions_to_authors": "(A) Why didn't you include your own system in the error analysis, or the system of Straka and Strakova in the Sec. 6 comparison?", "paper_topic_and_main_contributions": "The article presents three contributions: One is a quantitative analysis of the data in the CorefUD 1.1 coreference corpus, the second one is an error analysis of the coreference resolution system of Straka and Strakova (2022), and the third is the authors' own coreference system based on the baseline system of Prayak et al (2021) where the authors add a small number of additional features.\nFor the first part, the authors look at mention internal structure, the coreference chains (\"entities\") themselves, and finally document-level information such as the number of competing antecedent candidates. Looking at both papers that mainly concern themesleves with (prior or current versions of) the CorefUD corpus - Nedoluzhko et al 2022 and Zabortsky et al 2022, the quantitative analysis seems to be novel with respect to these works which do not touch the content of the coreference corpora at all.\nThe quantitative analysis looks at properties of each of the language-specific subcorpora of CorefUD; in doing so, no provision is made to tease apart confounders such as genre/domain of the texts, differences in annotation styles (even within a unified scheme - this was found in UD as a significant source of variation) etc., but would still be somewhat useful to researchers implementing their own coreference systems or preparing deeper analyses.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "l. 088 - The term two-mention entities isn't clear in context\nl. 130 - morphologically rich"}, "extra_text": {}}, {"review_id": "NBv2nd6bAS", "reviewer": "Reviewer_5r1p", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper and your valuable feedback.\n\n> The paper do not explain the notions presented in the paper such as Root Cause Analysis or the method to find undetected mentions, the antecedents.\n \nIn lines 60-67, we provide an explanation of root cause analysis. We will substitute the term with 'error analysis' for easier comprehension. We employ the evaluation metric LEA (Moosavi and Strube, 2016) to identify unresolved clusters (recall=0) and find the undetected mentions by comparing the gold mentions with the predicted mentions generated by MCR systems.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper focuses on linguistic properties of mentions, entities and documents and analyzes the data to avoid frequent coreference anotation errors. It is very important to have data sets checked by human annotators.", "reasons_to_reject": "The paper do not explain the notions presented in the paper such as Root Cause Analysis or the method to find undetected mentions, the antecedents.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper presents an application of the data sets from CorefUD. The goal is to understand multilingual coreference resolution and to provide a solution for this.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "zJMbuLdI8I", "reviewer": "Reviewer_eyYv", "summary": "", "questions": "In Table 4, what is the different of between the baseline model and the proposed one without UA and LANG? Specifically, what is the characteristic of the model that accounts for 0.03? ", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for reviewing our paper and your valuable feedback.\n\n> The statistical significance of the results is not addressed. It is also unclear whether the results are from a single run or the average of repeated runs.\n\nWe follow the standard practice in the coreference resolution literature, and repeat our experiments three times to ensure the reliability of our results. We report the mean and standard deviation of F1 scores in our setting, as shown in the table below. We note that the models we investigated exhibited relatively large standard deviations across datasets. We attribute this variance gap to the differences in dataset sizes, as the models appear to yield higher variances when evaluated on a small corpus such as DE-PARC and EN-PARC.\n\n| Models| CA|CS-PCED| CS-PDT|EN-GUM|HU|PL|ES|LT|FR|DE-PARC|DE-POTS|EN-PARC|RU|\n|-|-|-|-|-|-|-|-|-|-|-|-|-|-|\n|Baseline|55.2±0.6|68.4±0.5|64.3±0.4|48.8±1.2|46.4±1.0|50.2±0.5|57.6±0.5|64.2±1.5|57.0±0.7|33.7±1.7|43.3±1.4|43.0±1.5| 66.9±1.1|\n|ours_lem|52.8±0.5|65.0±0.4|62.7±0.3|48.1±1.0|44.0±1.1|44.3±0.6|54.6±0.4|60.2±1.7|56.7±0.8|30.6±1.5|40.8±1.5|48.4±1.8|64.5±0.9|\n|ours_wf|55.7±0.5|68.5±0.4|64.9±0.5|50.1±1.1|47.1±0.9|50.4±0.5|57.7±0.6|62.1±1.5|58.6±0.8|35.1±1.8|44.9±1.4|48.5±1.8|66.5±1.1|\n\n> No error analysis is presented.\n\nWe will include this error analysis following the method outlined in section 4 in the final version where we will have one more page available.\n\n> The claim of 0.9 F1 improvement is somehow misleading. The F1 ranges from 0 to 1. Thus, without clear context, it looks impressive, but it is because the authors use a range between 0 to 100. (0.9 F1 -> 0.9% F1)\n\nWe will use 0.9% F1 in the final version.\n\n> In Table 4, what is the different of between the baseline model and the proposed one without UA and LANG? Specifically, what is the characteristic of the model that accounts for 0.03?\n\nWe are sorry but we do not understand what you are referring to with 0.03. Please clarify during the discussion period.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The multilingual in-depth description of CorefUD 1.1\n\nThe hybrid coreference model, which combines mBERT with morphosyntactic information.", "reasons_to_reject": "The statistical significance of the results is not addressed. \n\nNo error analysis is presented. (During the rebuttal phase, the authors indicate that they may put it into the final version, but it is unclear what analysis will be done).", "questions_to_authors": "In Table 4, what is the different of between the baseline model and the proposed one without UA and LANG? Specifically, what is the characteristic of the model that accounts for 0.03? ", "paper_topic_and_main_contributions": "This paper presents an in-depth multilingual description of CorefUD 1.1, highlighting various coreference properties differences in the 12 languages represented in this corpus. Then, a proposed model, which concatenates the UPOS tag, UD relation, mention type and UD category of the candidate, is proposed. This model is compared with the baseline model used in the shared task of multilingual coreference resolution (CRAC 2022) using CorefUD 1.0 (10 languages). Finally, the proposed model outperforms the baseline by 0.9% of F1, and an ablation test is presented.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "0.9 F1 -> 0.9% F1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper provides an in-depth analysis of the CorefUD multi-lingual coreference corpora used in the CRAC2022 shared task, in different linguistic levels (mention, entity and document). Then they did the error analysis on the best-performing system of the shared task to find out the challenges faced by the system. Last they proposed using UD-related features (e.g. UPOS, UD relation) to mitigate the errors. The evaluation on 10% of training data suggests using such features in the baseline system can achieve a marginal improvement of 0.9%. The analysis on the CorefUD corpus and error analysis provides useful insight into multi-lingual coreference resolution. The evaluation, however, is less convincing given that only one system is evaluated on a non-standard test set, which makes it hard to compare with other systems in the future. It is also unclear the reason behind the decision to use CoNLL 2012 F1 instead of the same evaluation metric (CorefUD scorer) used in the CRAC 2022 shared task, This again makes the evaluation non-comparable with the shared task systems even use exactly the same portion of the training set for the test.  The switch between CorefUD 1.1 and CorefUD 1.0 is also a bit confusing, it would be more helpful if the paper stuck to one dataset or at least explained their decision more clearly. There are a few other points that need to be clearly mentioned as well to improve the reproducibility and avoid confusion, e.g. according to the author's response in the discussion the error analysis is done on the dev set. On the UD resources side, (UPOS, UD relations etc.), the author claims during the discussion that those resources that come with the CorefUD are all predicted, but according to the shared task overview paper ( Žabokrtský et al. 2022) in Section 2.1 it mentioned many corpora has UD columns manually annotated (e.g. Prague Dependency Treebank (Czech)) and at the end of Section 2 the organiser mentioned "With some exceptions, if the original resources contained manual annotation of morpho-syntax, it has been kept also in CorefUD. " so it does contain manually annotated UD columns which I would request the author to investigate which corpora they used are with manual UD annotations and make it clear that the results they achieved are not benefit from the gold annotation, e.g. there is no distinguish improvements gained from the datasets with gold annotations.
Decision: Accept-Findings