Coreference resolution for spoken French

Loïc Grobol

► To cite this version:

Loïc Grobol. Coreference resolution for spoken French. Computation and Language [cs.CL]. Université Sorbonne Nouvelle - Paris 3, 2020. English. NNT: tel-02928209v2

HAL Id: tel-02928209

https://hal.science/tel-02928209v2

Submitted on 9 Mar 2021

**HAL** is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
École Doctorale 622 — Sciences du langage
Lattice, Inria

THÈSE DE DOCTORAT EN SCIENCES DU LANGAGE
DE L'UNIVERSITÉ SORBONNE NOUVELLE

Coreference resolution for spoken French

présentée et soutenue publiquement par

Loïc GROBOL

le 15 juillet 2020

Sous la direction d'Isabelle TELLIER† et de Frédéric LANDRAGIN

Et co-encadrée par Éric VILLEMONTE DE LA CLERGERIE et Marco DINARELLI

**Jury :**

Massimo Poesio, Professor (Queen Mary University), Rapporteur,
Sophie Rosset, Directrice de recherche (LIMSI), Rapportrice,
Béatrice Daille, Professeur (Université de Nantes), Examinatrice,
Pascal Amsili, Professeur (Université Sorbonne Nouvelle), Examinateur,
Frédéric Landragin, Directeur de recherche (Lattice), Directeur,
Éric Villemonte de la Clergerie, Chargé de recherche (Inria), Co-encadrant,
Marco Dinarelli, Chargé de recherche (LIG), Co-encadrant.
# Reconnaissance automatique de chaînes de coréférences en français parlé

## Résumé

Une chaîne de coréférences est l'ensemble des expressions linguistiques — ou mentions — qui font référence à une même entité ou un même objet du discours. La tâche de reconnaissance des chaînes de coréférences consiste à détecter l'ensemble des mentions d'un document et à le partitionner en chaînes de coréférences. Ces chaînes jouent un rôle central dans la cohérence des documents et des interactions et leur identification est un enjeu important pour de nombreuses autres tâches en traitement automatique du langage, comme l'extraction d'informations ou la traduction automatique. Des systèmes automatiques de reconnaissance de chaînes de coréférence existent pour plusieurs langues, mais aucun pour le français ni pour une langue parlée.

Nous nous proposons dans cette thèse de combler ce manque par un système de reconnaissance automatique de chaînes de coréférences pour le français parlé. À cette fin, nous proposons un système utilisant des réseaux de neurones artificiels et ne nécessitant pas de ressources externes. Ce système est viable malgré le manque d'outils de prétraitements adaptés au français parlé et obtient des performances comparable à l'état de l'art. Nous proposons également des voies d'amélioration de ce système, en y introduisant des connaissances issues de ressources et d'outils conçus pour le français écrit. Enfin, nous proposons un nouveau format de représentation pour l'annotation des chaînes de coréférences dans des corpus de langues écrites et parlées et en nous en donnons un exemple en proposant une nouvelle version d'ANCOR — le premier corpus de français annoté en coréférence.

**Mots-clés** : anaphore, coréférence, réseaux de neurones artificiels, apprentissage artificiel, ressources annotées, corpus, formats d'annotation, traitement automatique du langage naturel
Coreference resolution for spoken French

Abstract

A *coreference chain* is the set of linguistic expressions — or *mentions* — that refer to the same entity or discourse object in a given document. *Coreference resolution* consists in detecting all the mentions in a document and partitioning their set into coreference chains. Coreference chains play a central role in the consistency of documents and interactions, and their identification has applications to many other fields in natural language processing that rely on an understanding of language, such as information extraction, question answering or machine translation. Natural language processing systems that perform this task exist for many languages, but none for French — which suffered until recently from a lack of suitable annotated resources — and none for spoken language.

In this thesis, we aim to fill this gap by designing a coreference resolution system for spoken French. To this end, we propose a knowledge-poor system based on an end-to-end neural network architecture, which obviates the need for the preprocessing pipelines common in existing systems, while maintaining performances comparable to the state-of-the art. We then propose extensions on that baseline, by augmenting our system with external knowledge obtained from resources and preprocessing tools designed for written French. Finally, we propose a new standard representation for coreference annotation in corpora of written and spoken languages, and demonstrate its use in a new version of ANCOR, the first coreference corpus of spoken French.

**Keywords:** anaphora, coreference, artificial neural networks, machine learning, annotated resources, corpus, annotations representation, natural language processing
Remerciements

Mes tous premiers remerciements vont à Isabelle Tellier, ma directrice de thèse, à qui je dois
doublement mes débuts dans le monde de la recherche et de l'enseignement supérieur. Tu
resteras pour moi un modèle et je ne pourrais jamais assez te remercier pour ta gentillesse, ton
aide et ta confiance.

Je remercie de tout cœur Frédéric Landragin, mon directeur de thèse, qui a admirablement repris ce flambeau et dont l'infaillible optimisme et le soutien constant ont été les moteurs de toute la deuxième partie de cette thèse. Tu es incontestablement non seulement un excellent encadrant, mais aussi un collègue remarquable et un véritable ami. Je remercie aussi Marco Dinarelli, mon co-encadrant de thèse, à qui je dois mon introduction aux réseaux de neurones, plusieurs fructueuses collaborations et d'excellents moments, au Lattice comme en coreference. Je remercie encore Éric de la Clergerie, mon co-encadrant de thèse pour son soutien dans des moments compliqués, pour sa clairvoyances et pour nos très nombreuses discussions, j'ai grâce à toi de quoi m'occuper pendant plusieurs carrières et je ne sous-estimerai jamais les méthodes symboliques !

Je remercie les membres du jury d'avoir accepté d'évaluer cette thèse et sa soutenance dans des conditions loin d'être idéales, pour leurs nombreuses remarques, conseils et questions qui ont grandement contribué à l'amélioration et à la finalisation de ce travail. Des remerciements tout particuliers à Pascal Amsili, pour son aide dans la préparation de la soutenance à distance. Je remercie également les membres de mon comité de suivi de thèse, pour leur soutien et leurs conseils déterminants pour la fin de la thèse et pour la structure du présent manuscrit.

Je remercie le Lattice, ma *happy place* tout au long de cette thèse. Je pense tout particulièrement aux résidents de Thésardland, à la solidarité sans égal, mais aussi à l'ensemble des membres du laboratoire, passés et présentés, permanents et temporaires, sans qui ces années n'auraient pas été si douces.

Je remercie également l'ensemble des membres d'ALMAnaCH, pour leur accueil, leur soutien et l'ambiance stimulante de l'équipe qui m'a poussé tout au long de cette thèse à dépasser mes limites. Je remercie mes collègues enseignant·e·s du master pluriTAL, et en particulier Serge Fleury, Kim Gerdes, Sylvain Kahane, Jean-Michel Daube et Damien Nouvel, aux côtés desquels j'ai été heureux de pouvoir me consacrer à mon activité favorite, ainsi que l'ensemble de mes étudiant·e·s, et en particulier Natalia Kalshnikova et Veronika Solopova que j'ai eu le plaisir d'accompagner dans leurs travaux de recherche.

Je remercie les membres du consortium DEMOCRAT, pour leur implication dans un projet qui, s'il n'est finalement pas le cœur de cette thèse, a été déterminant pour mon travail durant ces années. Des remerciements particuliers à Serge Heiden pour ses encouragements, ses conseils et nos collaborations.

Je remercie les membres du groupe RITUEL, et en particulier Anne-Lyse Minard, Agatha Savary, Lotfi Abouda et la cabale de la coréférence : Emmanuel Schang, Adam Lion-Bouton, Sylvie Billot, Anaïs Lefeuvre-Halftermeyer et Jean-Yves Antoine, pour nos collaborations fructueuses – passées, présentes et futures! – et pour leur amitié
Des remerciements tous particuliers à Iris Eshkol-Taravella, à qui je dois la découverte du TAL et mes premières expériences de recherche et avec qui j'ai toujours plaisir à travailler. Je remercie très chaleureusement Sophie Prévost et Benoît Crabbé, dont le soutien a été crucial pour toute la fin de cette thèse et avec qui je poursuis avec jubilation mes aventures de recherche en TAL.

Je remercie l'ensemble des *Cool Young Academics*, habituel·le·s et de passage, toutes et tous des *lovely humans*. En particulier, merci à Olga, Maximin, Sacha, Alice, Gael, Alexis, Ilaine, Zak, Marine C., Arthur B., KyungTae, Lara, Sanjay, Hicham, Élise, Pierre, Lucie, Ariane, Jade, Hyun Jung et Marie.

Je ne pourrais assez remercier le club des Chouettes Individus aux Nombreuses Qualités, Marie-Amélie, Marine, Auphélie, Mathilde, Miléna, Tian et Yoann ; vous êtes le houmous du houmous, du fond du cœur, merci. Mille mercis également à Mathieu, Mathilde, Paul, Émilie, Chloé, Florent, Adrian, Pedro, Clémentine, Alix, Jayjay, Sara, Arthur, Margot, Cassandra, Murielle, FX, Maxime, Nico, Manon, Lucas, Anaïs et toutes celles et tous ceux qui avant et pendant ces années m'ont soutenu sans faillir.

Enfin, je remercie ma famille et ma belle-famille, sur qui je sais pouvoir compter en toutes circonstances, et Hélène, spectatrice et actrice de cette aventure, partenaire de crime idéale et mon humaine préférée.
# Contents

<table><tr><td>Front matter</td><td>ii</td></tr><tr><td>Résumé</td><td>ii</td></tr><tr><td>Abstract</td><td>iii</td></tr><tr><td>Abstract</td><td>iii</td></tr><tr><td>Remerciements</td><td>iv</td></tr><tr><td>Contents</td><td>vi</td></tr><tr><td>1 Introduction</td><td>1</td></tr><tr><td>1.1 Coreference resolution</td><td>1</td></tr><tr><td>1.2 Contributions</td><td>4</td></tr><tr><td>1.2.1 A coreference resolution system for spoken French</td><td>4</td></tr><tr><td>1.2.2 Leveraging imperfect knowledge</td><td>5</td></tr><tr><td>1.2.3 A standard for multigenre coreference resources</td><td>5</td></tr><tr><td>1.3 Outline</td><td>5</td></tr><tr><td>1.4 Terminology and notations</td><td>6</td></tr><tr><td>2 Coreference, anaphora and related phenomena</td><td>8</td></tr><tr><td>2.1 Referring mechanisms in language</td><td>8</td></tr><tr><td>2.1.1 Discourse models and coreference</td><td>8</td></tr><tr><td>2.1.2 Referring mechanisms and interpretation</td><td>10</td></tr><tr><td>2.2 Coreference in natural language processing</td><td>13</td></tr><tr><td>2.3 Related tasks</td><td>15</td></tr><tr><td>2.3.1 Named entity recognition and Entity linking</td><td>15</td></tr><tr><td>2.3.2 Narrow coreference resolution</td><td>17</td></tr><tr><td>2.3.3 Anaphora resolution</td><td>19</td></tr><tr><td>2.4 Evaluating coreference resolution</td><td>20</td></tr><tr><td>2.4.1 Mention detection</td><td>20</td></tr><tr><td>2.4.2 Coreference resolution metrics</td><td>21</td></tr><tr><td>2.4.3 Corpus-level evaluation</td><td>26</td></tr><tr><td>3 Annotated resources for coreference</td><td>29</td></tr><tr><td>3.1 Annotated resources for coreference</td><td>29</td></tr><tr><td>3.1.1 Objectives</td><td>30</td></tr></table>
Coreference resolution for spoken French

<table>
  <tbody>
    <tr>
      <td>3.1.2</td>
      <td>Scopes</td>
      <td>31</td>
    </tr>
    <tr>
      <td>3.1.3</td>
      <td>Parallel and supporting annotations</td>
      <td>32</td>
    </tr>
    <tr>
      <td>3.1.4</td>
      <td>Ontonotes</td>
      <td>33</td>
    </tr>
    <tr>
      <td>3.2</td>
      <td>Resources for French</td>
      <td>35</td>
    </tr>
    <tr>
      <td>3.2.1</td>
      <td>Historical resources</td>
      <td>35</td>
    </tr>
    <tr>
      <td>3.2.2</td>
      <td>ANCOR</td>
      <td>36</td>
    </tr>
    <tr>
      <td>3.2.3</td>
      <td>DEMOCRAT</td>
      <td>40</td>
    </tr>
    <tr>
      <td>3.3</td>
      <td>Representing coreference annotation</td>
      <td>40</td>
    </tr>
    <tr>
      <td>3.3.1</td>
      <td>The URS metamodel for anaphora and coreference</td>
      <td>41</td>
    </tr>
    <tr>
      <td>3.3.2</td>
      <td>A TEI encoding and serialization for URS</td>
      <td>42</td>
    </tr>
    <tr>
      <td>3.3.3</td>
      <td>Application to ANCOR and DEMOCRAT</td>
      <td>44</td>
    </tr>
    <tr>
      <td>4</td>
      <td>Coreference resolution systems</td>
      <td>49</td>
    </tr>
    <tr>
      <td>4.1</td>
      <td>Representations</td>
      <td>49</td>
    </tr>
    <tr>
      <td>4.1.1</td>
      <td>Mentions</td>
      <td>50</td>
    </tr>
    <tr>
      <td>4.1.2</td>
      <td>Mention pairs</td>
      <td>55</td>
    </tr>
    <tr>
      <td>4.1.3</td>
      <td>Mention sets</td>
      <td>56</td>
    </tr>
    <tr>
      <td>4.2</td>
      <td>Coreference resolution algorithms</td>
      <td>57</td>
    </tr>
    <tr>
      <td>4.2.1</td>
      <td>Ordering</td>
      <td>57</td>
    </tr>
    <tr>
      <td>4.2.2</td>
      <td>Link-centric algorithms</td>
      <td>59</td>
    </tr>
    <tr>
      <td>4.2.3</td>
      <td>Entity-mention models</td>
      <td>62</td>
    </tr>
    <tr>
      <td>4.3</td>
      <td>Mention detection</td>
      <td>64</td>
    </tr>
    <tr>
      <td>4.4</td>
      <td>Systems for French</td>
      <td>65</td>
    </tr>
    <tr>
      <td>5</td>
      <td>Automatic coreference resolution</td>
      <td>67</td>
    </tr>
    <tr>
      <td>5.1</td>
      <td>System scope</td>
      <td>68</td>
    </tr>
    <tr>
      <td>5.2</td>
      <td>The DeCOFre system</td>
      <td>69</td>
    </tr>
    <tr>
      <td>5.2.1</td>
      <td>General architecture</td>
      <td>69</td>
    </tr>
    <tr>
      <td>5.2.2</td>
      <td>Representing text spans</td>
      <td>69</td>
    </tr>
    <tr>
      <td>5.2.3</td>
      <td>Mentions detection</td>
      <td>76</td>
    </tr>
    <tr>
      <td>5.2.4</td>
      <td>Coreference resolution</td>
      <td>76</td>
    </tr>
    <tr>
      <td>5.3</td>
      <td>Training</td>
      <td>79</td>
    </tr>
    <tr>
      <td>5.3.1</td>
      <td>Optimization algorithm</td>
      <td>80</td>
    </tr>
    <tr>
      <td>5.3.2</td>
      <td>Learning rate</td>
      <td>80</td>
    </tr>
    <tr>
      <td>5.3.3</td>
      <td>Regularization</td>
      <td>81</td>
    </tr>
    <tr>
      <td>5.3.4</td>
      <td>Mention detection</td>
      <td>83</td>
    </tr>
    <tr>
      <td>5.3.5</td>
      <td>Antecedent scoring</td>
      <td>84</td>
    </tr>
    <tr>
      <td>5.3.6</td>
      <td>Spans representations</td>
      <td>85</td>
    </tr>
    <tr>
      <td>5.4</td>
      <td>Experiments</td>
      <td>86</td>
    </tr>
    <tr>
      <td>5.4.1</td>
      <td>Hyperparameters</td>
      <td>87</td>
    </tr>
    <tr>
      <td>5.4.2</td>
      <td>Baseline results</td>
      <td>89</td>
    </tr>
    <tr>
      <td>5.4.3</td>
      <td>Transfer learning</td>
      <td>90</td>
    </tr>
    <tr>
      <td>5.4.4</td>
      <td>Hyperparameters influence</td>
      <td>91</td>
    </tr>
    <tr>
      <td>5.5</td>
      <td>Conclusion</td>
      <td>92</td>
    </tr>
  </tbody>
</table>
<table><tr><td>6</td><td>Explicit knowledge augmentations for coreference resolution systems</td><td>95</td></tr><tr><td>6.1</td><td>Raw features</td><td>96</td></tr><tr><td>6.1.1</td><td>Span features</td><td>96</td></tr><tr><td>6.1.2</td><td>Pair features</td><td>97</td></tr><tr><td>6.2</td><td>Semi-supervised knowledge</td><td>99</td></tr><tr><td>6.2.1</td><td>Static word embeddings</td><td>99</td></tr><tr><td>6.2.2</td><td>Contextual word embeddings</td><td>100</td></tr><tr><td>6.3</td><td>Shallow linguistic knowledge</td><td>103</td></tr><tr><td>6.4</td><td>Structural linguistic knowledge</td><td>104</td></tr><tr><td>6.4.1</td><td>Non-recursive structures</td><td>105</td></tr><tr><td>6.4.2</td><td>Syntactic parsing</td><td>106</td></tr><tr><td>6.4.3</td><td>Results</td><td>107</td></tr><tr><td>6.5</td><td>Conclusion</td><td>109</td></tr><tr><td>7</td><td>Conclusion and perspectives</td><td>111</td></tr><tr><td colspan="2">Bibliography</td><td>113</td></tr><tr><td colspan="2">List of Figures</td><td>150</td></tr><tr><td colspan="2">List of Tables</td><td>151</td></tr><tr><td colspan="2">Licence</td><td>153</td></tr><tr><td colspan="2">Back</td><td>154</td></tr></table>
À Dino, mon grand-père, linguiste sans le savoir, à qui je dois ma rencontre avec l'études des langues. Et à Isabelle, grâce à qui je peux en faire mon métier.
<figure><img src="image_2.png" /></figure>
# Chapter 1

## Introduction

### 1.1 Coreference resolution

This thesis is about coreference resolution for spoken French. A coreference chain is the set of linguistic expressions — or mentions — that refer to the same entity in a given document. Coreference resolution consists in detecting all the mentions in a document and partitioning their set into coreference chains. Coreference chains play a central role in the consistency of documents and interactions, and their identification has applications to many other fields in natural language processing that rely on an understanding of language, such as information extraction, question answering or machine translation.

(1) [Susan Calvin]₁ had been born in [the year 1982], [they]° said, [which] made [her] seventy-five now. [Everyone]ᵇ knew [that]*. Appropriately enough, [U.S. Robot and Mechanical Men, Inc.]₂, was seventy-five also, since [it] had been in [the year of [[Dr. Calvin]₁'s birth]] that [Lawrence Robertson]₃ had first taken out [incorporation papers] for [what]₂ eventually became [the strangest industrial giant in [[man]'s history]]₂. Well, [everyone]ª knew [that], too.

At [the age of [twenty]], [Susan Calvin]₁ had been part of [the particular Psycho-Math seminar at which [Dr. Alfred Lanning of [U.S. Robots]₂]₅ had demonstrated [the first mobile robot to be equipped with [a voice]]₄]₆. [It]₄ was [a large, clumsy unbeautifull robot]₄, smelling of [machine-oil] and destined for [the projected mines on [Mercury]].—But [it]₄ could speak and make sense.

[Susan]₁ said nothing at [that seminar]₆; took [no part] in [the hectic discussion period that followed]. [She]₁ was [a frosty girl], plain and colorless, [who]₁ protected [herself]₁ against a world [she]₁ disliked by [[a mask-like expression] and [a hypertrophy of intellect]]. But as [she]₁ watched and listened, [she]₁ felt [the stirrings of [a cold enthusiasm]].

*(Asimov 1950)*

Example 1 gives a possible output for a hypothetical coreference resolution system. Entity mentions are denoted by brackets and coreferent mentions are subscripted with the same index.
Even in such a short<fnref n="1" /> example, we can already make some observations on the structure of the task. For instance, it seems that most mentions do not corefer with any other mention, and that, conversely, one entity — Dr. Susan Calvin — makes up most of the mentions, especially in the last paragraph. This suggests that the sizes of coreference chains are not uniformly distributed. We can also note that most mentions are noun phrases and pronouns, and that most pronouns corefer with a noun phrase that is not too far away from them.

This example also shows instances of many theoretical issues:

* Is [they]ª really a mention of an entity, and if yes, what is the nature of this entity?

* What does [that]* refer to? The date of birth of Susan Calvin? Her age? Should we then also consider that the whole first sentence is also a mention coreferent with [that]*?

* Are the boundaries in [incorporation papers] correct for this mention or should they encompass the whole phrase 'incorporation papers for what eventually became the strangest industrial giant in man's history'?

* Are [everyone]ᵇ and [everyone]ᶜ coreferent? What about the mentions of Dr. Calvin at different periods of her life?

* Is [a large, clumsy unbeautifull robot]₄ a mention of entity 4 or one of its attributes? Are coordinations like ‘a mask-like expression and a hypertrophy of intellect’? Is [no part]?

From a technical point of view, this also shows that unlike many tasks in natural language processing, coreference resolution is a document understanding task that is hard to reduce to sentence-level processing or structural analysis. Indeed, understanding that [the strangest industrial giant in [[man]'s history]]₂ corefers with [U.S. Robots]₂ cannot be done reliably without building a model of the world described in this paragraph. It is also unlikely that an automatic system could rely on existing ontologies or knowledge bases to obtain a pre-existing model of this world, since it is entirely fictional.

Conversely, many references are relatively easy to resolve with simple heuristics: for instance, since [Susan] is the only noun phrase in the last paragraph whose grammatical gender is compatible with all the subsequent ‘she’, it would be relatively safe for an automatic system to bet on their coreference. Similarly, it is very unlikely that [U.S. Robot and Mechanical Men, Inc.]₂ and [U.S. Robots]₂ are not coreferent.

Coreference resolution — at least in limited forms — has a long history in natural language processing, both as a tool for other tasks, e.g. in Winograd (1972)'s language understanding system SHRDLU and as an issue in itself (Hobbs 1986). These early works were mostly focused on the specific case of pronominal coreference, traditionally formulated as the task of finding noun phrase antecedents for pronouns. The term of *coreference* itself rose to fame in natural language processing on the occasion of the MUC-6 (MUC Consortium 1995c) and MUC-7 (MUC Consortium 1998), where it was a part of a set of specific document understanding and information extraction tasks.

<footnote n="1">¹Coreference resolution is usually formulated at the level of whole documents, in which case the example would be a whole book chapter.</footnote>
During this time, many attempts at designing an accurate coreference resolution system were made, usually following the trends in other fields of natural language processing: moving from rule-based systems to data-driven machine learning systems and in the later years to the specific area of neural network-powered machine learning, led by the increasing availability of annotated datasets and computational power. These efforts mostly focused on coreference resolution for English and new paradigms often emerged from research on this language, encouraged by a greater availability of resources. However, the techniques developed for English have also been applied, usually successfully to other languages as soon as suitable corpora became available.

During these years, there has been a great diversity in approaches to coreference resolution, with none gaining a clear and durable upper-hand. This might be explained the difficulty of properly modelling a task that concerns many related but heterogeneous cognitive phenomena. This diversity manifests itself not only in the design of automatic systems, but also in the quantitative metrics used to assess their performances and in the frameworks and representations used in existing annotated data.

In this landscape, French appears as an odd language out: despite its usual status as a well-resourced language, no corpus with coreference annotation for French was available until 2014, far later than other well-resourced languages (such as Czech, Polish, German, Spanish or Catalan). This lack of resources was solved by the release in 2014 of ANCOR (Muzerelle et al. 2013a, 2014). However, while this corpus, comparable in size with the main coreference corpora for other languages, did provide the missing resource, it did so in another unusual way. Indeed, most coreference corpora are based on either standard written texts, or transcripts of planned speech and more rarely on spontaneous speech or non-standard writings, but ANCOR is made entirely of spontaneous speech with various degrees of interactivity. Therefore, research using ANCOR as a resource is confronted not only with the problem of coreference resolution in French but also of coreference resolution in spontaneous speech, both relatively unexplored tasks.

This is not to say that ANCOR is not a valuable resource. On the contrary: despite the lesser attention given to spontaneous speech in natural language processing, this genre has a lot of relevance for final applications — since spontaneous speech is the most common form of language produced and consumed by humans. It is also of a great interest for fundamental research: with a low availability of resources but close to a well-resourced language (in that case written French) it can serve as an experimental ground for work on truly lesser-resourced languages.

So far, however, no attempt at using ANCOR to design and evaluate a complete end-to-end coreference resolution system for French has been made public. Our objective in this thesis is to address this lack, by proposing such a system and exploring ways to overcome the issues caused by the specificities of ANCOR. These issues mostly derive from the differences between coreference in spontaneous speech and other genres: as suggested by works such as Antoine (2004) and Prince (1981), coreference chains do not follow the same patterns in spoken and written language, but these patterns have not been studied extensively enough to provide a simple way to transform a system design for written language into a system suitable for
spoken language. Furthermore, due to the lesser availability of resources for spoken languages,
designing a coreference resolution system for spoken French using the same architecture as
one designed for e.g. written English is not necessarily possible.

We propose to deal with these issues in the following manner:

* Since we lack a proper formalization of the specificities of coreference in spoken French, we do not attempt to design a system based on handcrafted rules, but instead defer the construction of these rules to data-driven machine learning techniques. Our concern is mostly to design suitable parametric language representations and processing architectures that can be optimized using the annotated data of ANCOR. Fortunately, the recent advances in machine learning gave us with suitable tools for this task and the current state-of-the-art in coreference resolution for other languages suggest that this is a viable approach with the amount of data available to us.
* Since resources for spoken French are scarce, we start by building a system that depends on as few resources as possible and ensure its viability before attempting to augment it with the imperfect knowledge that can be gathered from tools designed for written French. There again, recent advances in natural language processing brought by end-to-end neural architecture let us hope that this approach can indeed work in our situation.

## 1.2 Contributions

Our contributions in this thesis address several issues in coreference resolution for spoken French: designing a knowledge-poor coreference resolution system suitable for this language and genre, asserting the relevance of noisy knowledge sources for this task and improving the state of coreference resources for French by proposing a unified representation for coreference copora in both written and spoken language.

### 1.2.1 A coreference resolution system for spoken French

We propose DeCOFre, an end-to-end coreference resolution system, capable of identifying entity mentions in raw documents and specifically designed for transcriptions of spontaneous spoken French. This system does not rely on external knowledge<fnref n="2" /> and its architecture should therefore be generalizable to other genres and languages as long as an annotated dataset of a size similar to ANCOR is available.

The architecture of this system is based on both components from neural coreference resolution
systems for English (K. Lee et al. 2017, 2018) and general-purpose representation techniques
coming from neural systems for other natural language processing tasks (Cross and Huang
2016; Devlin et al. 2018; Ling et al. 2015). Quantitatively speaking, the results obtained are
better than those obtained by the only previous work on unrestricted coreference resolution
for French (Désoyer et al. 2015a), which did not propose a way to detect entity mentions and
used extensive gold-standard knowledge. This makes DeCOFre the first proper end-to-end

<footnote n="2">²Beyond the use of pretrained word embeddings, which we show in section 6.2 to be unnecessary although they provide some tangential benefits.</footnote>
coreference resolution for French and the first coreference resolution system designed for and evaluated on spoken language transcriptions.

### 1.2.2 Leveraging imperfect knowledge

Having produced a knowledge-poor coreference resolution system, we propose several extensions of this system using knowledge sources of different types: from heuristics on the raw material of ANCOR that do not need external resources to knowledge extracted in semi-supervised ways from raw linguistic material, shallow linguistic processing and structured linguistic processing such as syntactic parsing. None of these knowledge sources were designed for spoken French, but either for heterogeneous web texts or standard written French.

We find that the relevance of these resources is highly variable, and in particular that general-purpose representations learned in semi-supervised ways are much easier to integrate in our system and improve it significantly, while knowledge relying on explicit linguistic processing does not seem to provide many improvements.

### 1.2.3 A standard for multigenre coreference resources

Finally, in order to support our usage of ANCOR in designing our system, we provide a new version of this corpus, including various improvements to its structure and content, correcting some mistakes and provide a reference word-level segmentation and standard split that we hope will help others to reproduce our experiments and compare our results to those of future works.

This resource comes in a new format, built upon the recommendation of the Text Encoding Initiative (TEI consortium 2020) in order to move from the ad-hoc representations of previous versions of ANCOR. This format is designed to allow coreference annotations for all documents encodable in TEI formats, including that of the DEMOCRAT corpus for coreference in written French (Landragin 2016) released during the elaboration of this thesis and to which we participated. It also allows for other types of annotations beyond coreference and can serve as a base in the conception of rich multi-annotated corpora using a large variety of materials.

## 1.3 Outline

This remaining of this document is organized as follows: in chapter 2, we give a discourse-oriented model of coreference along with the terminology that we adopt in this work, describe the task of coreference resolution as it is understood in natural language processing, outline some popular related tasks. We then provide definitions for the quantitative measurements used to assess the performances of coreference resolutions systems.

In chapter 3, we give an overview of the existing resources for coreference with elements of typology that we apply to the currently most popular resources for coreference resolution — the Ontonotes corpus (Hovy et al. 2006; Pradhan et al. 2007a) — and the existing resources for French. We also describe our contributions in the form of the XML-TEI-URS format for coreference annotations and its use ANCOR and DEMOCRAT.
In chapter 4, we describe the building blocks for existing coreference resolution systems in general and for French in particular, along with their respective requirements and the resources and algorithms that they rely on.

In chapter 5 we describe the knowledge-poor version of DeCOFre, our coreference resolution system in details, from its architecture to the training procedure that we used to optimize its performances of ANCOR and motivate some of our technical choices by empirical experiments.

Finally, in chapter 6, we describe our experiments of knowledge augmentations for DeCOFre, using publicly available resources – mainly designed from written French –, the results of these experiments and elements of interpretation.

## 1.4 Terminology and notations

Throughout this document, we use the following notations

**Vector concatenation** is noted by brackets. If $a = (a_1, ..., a_n)$ and $b = (b_1, ..., b_m)$ are vectors, $[a, b] = (a_1, ..., a_n, b_1, ..., b_m)$ is the concatenation of $a$ and $b$.

**Sequences** are noted with indexed parentheses or in **bold** where parentheses would make a formula hard to read and the meaning is unambiguous. For instance, depending on the ambiguity of the context, the sequence $(h_1, ..., h_n)$ could also noted $(h_i)_{1 \le i \le n}$, $(h_i)_i$ or **h**. If $S = (i_1, ..., i_n)$ is a sequence, we also write $(h_i)_{i \in S}$ for $(h_{i_1}, ..., h_{i_n})$.

**Cardinality** is noted with $|\cdot|$, for instance the cardinality of a set or an indexed family $S$ would be noted $|S|$.

**Neural network layers** are noted as follows

**Feedforward layers** also called ‘multilayer perceptrons’ are noted FFNN. Formally a feedforward layer of depth $n \in \mathbb{N}$ and dimensions $(k_0, ..., k_n) \in \mathbb{N}^{n+1}$ is defined as

$$ \text{FFNN: } \mathbb{R}^{k_0} \to \mathbb{R}^k \\ x \mapsto \text{FFNN}(x) = (f_n \circ \dots \circ f_1)(x) \qquad [1.1] $$

with for all $i \in [1, n]$,

$$ f_i: \mathbb{R}^{k_{i-1}} \to \mathbb{R}^{k_i} \\ x \mapsto f_i(x) = a_i(W_i \times x + b_i) \qquad [1.2] $$

where $a_i$ is an elementwise non-linear function, $W_i$ is a matrix of dimension $k_i, k_{i-1}$ and $b_i \in \mathbb{R}^{k_i}$.

**Long Short-Term Memory layers** (Hochreiter and Schmidhuber 1997) are noted LSTM and are seen as operating on finite ordered sequences of vectors. More specifically, since we mostly use bidirectional LSTMs (Graves and Schmidhuber 2005; Schuster and Paliwal 1997), we note LSTM and LSTM their forward and backward directions and LSTM the stacked bidirectional version.
**Gated Recurrent Units** (Cho et al. 2014) are noted GRU. As with LSTMs, they are seen as operating on finite ordered sequences of vectors and we note $\overline{\text{GRU}}$, $\overline{\text{GRU}}$ and GRU their forward, backward and bidirectional versions respectively.

Definitions for technical terms can usually be found in either Jurafsky and J. H. Martin (2019) or (using their translation in French) in Cornuéjols and Miclet (2010). We defer to Diestel (2017) for graph theory terminology.
Chapter 2

Coreference, anaphora and related
phenomena

2.1 Referring mechanisms in language

2.1.1 Discourse models and coreference

Consider a device designed to read a text in some natural language, interpret it, and
store the content in some manner, say, for the purpose of being able to answer questions
about it. To accomplish this task, the machine will have to fulfill at least the following
basic requirement. It has to be able to build a file that consists of the records of all the
individuals, that is, events, objects, etc., mentioned in the text and, for each individual,
record whatever is said about it.

(Karttunen 1976)

1. One objective of discourse is to enable a speaker to communicate to a listener a model s/he has of some situation. Thus, the ensuing discourse is, on one level, an attempt by the speaker to direct the listener in synthesizing a similar model.

2. Such a discourse model can be viewed as a structured collection of entities, organized by the roles they fill with respect to one another, the relations they participate in, etc. (Webber 1978)

Turning back to discourse, let us say that a **text** is a set of instructions from a speaker to a hearer on how to construct a particular **discourse-model**. The model will contain **discourse entities**, **attributes**, and **links** between entities. A discourse entity is a discourse-model object, akin to [Karttunen (1976)] **discourse referent** ; it may represent an individual (existent in the real world or not), a class of individuals, an exemplar, a substance, a concept, etc. Following Webber (1978), entities may be thought of as hooks on which to hang attributes.

(Prince 1981)

Language, then, is not merely interpreted with respect to worlds, models, contexts,
situations, and so forth. Rather, it is involved in constructions of its own. It builds
up mental spaces, relations between them, and relations between elements within
them. To the extent that two of us build up similar space configurations from the same linguistic and pragmatic data, we may ‘communicate’; communication is a possible corollary of the construction process.

(Fauconnier et al. 1994)

Following Fauconnier et al. (1994), Karttunen (1976), Prince (1981) and Webber (1978), for the purpose of this work, we focus on discourse (either uni- or multi-directional) as the construction of a model shared by all participants. This model is not simply a model of the real world, since the entities that it contains may be fictional, as the unicorn in example 2, or hypothetical, as the car in example 3.

(2) Bill saw a **unicorn**. The **unicorn** had a golden mane. (Karttunen 1976)

(3) If Mary had a **car**, she would take me to work in it. (Karttunen 1976)

It is not only a model of a physical world — real or fictional — either, since it can also involve facts (as ‘le’ in example 4) abstract notions or concepts (‘the topological entropy of the tent function’) and even nonsensical or paradoxical entities (‘a four-sided triangle’, ‘a barber that shaves all those, and those only, who do not shave themselves’ (Russell 1919) or ‘the smallest uninteresting integer’).

(4) [...] faut que je le note d' ailleurs vous me le faites rappeler

([...] I must write it down by the way you remind me of that)

(ANCOR, Muzerelle et al. 2014)

Finally, a discourse model might contain entities that depends on external factors, either because they are part of the general knowledge of a participant, as ‘Noam Chomsky’ in example 5, or because they refer to the situation at hand, as ‘cette belle affiche’ in example 6 (respectively *unused* and *situationally evoked* entities in the terminology of Prince (1981)).

(5) **Noam Chomsky** went to Penn. (Prince 1981)

(6) Madame nous sommes autorisés par la municipalité pour mettre cette belle affiche

(Madam we are allowed by the municipality to put **this beautiful poster**)

(ANCOR, Muzerelle et al. 2014)

The entities and their attributes can change as the discourse unfolds, as in example 7, where it is clear that the ‘zucchini’ entity has significantly different attributes between the first and last sentences.

(7) Pick a **ripe, plump zucchini**.<fnref n="1" /> Prepare **it** for the oven, cut **it** into four pieces and roast **it** with thyme and smoked paprika for 1 hour. Serve **it** with white rice and enjoy **its** sweet and lightly spicy taste.

In this framework, we hold it for given that entities are represented in language by parts of sentences (or utterances) — typically constituents or catena — the prototypical cases being noun phrases and pronouns. When this is the case, we will say that these parts are *mentions* and that they all *refer* to entities.

<footnote n="1">¹With all due apologies to G. Brown and Yule (1983) for making their famous example more animal-friendly.</footnote>
Given a document D and M, the set of all the mentions it contains, we will say that two
mentions are *coreferent* or that they *corefer* if they refer to the same entity — which is clearly
an equivalence relation on M.

Let ⋔ᵢ Cᵢ = M be the associated partition of M in equivalence classes, we will say that the Cᵢ are
the *coreference chains* of D (see section 2.2 for a motivation of the term *chain*). In other words, a
coreference chain is the set of all the mentions that refer to a single entity.

We will also say that the first mention of an entity in a document is *discourse-new* and conversely
that its other mentions are *discourse-old*. For convenience, in a slightly abusive borrowing from
the terminology of anaphora, given a discourse-old mention *m*, we will say that the mentions
preceding *m* in a discourse and coreferent with *m* are its *antecedents*.

Finally, we will say that a mention is *coreferring* if it is not the only element in its coreference
chain. Conversely we will call<fnref n="2" /> non-coreferring mentions *singleton mentions* or *singletons* when
there is no ambiguity.

Note that this only define an *identity coreference* relation and does not address the case of *near-identity coreference* (Recasens et al. 2011a), which supersedes the notion of coreference, allows a much richer representations of the relations between mentions and entities and elegantly resolves a number of conundrums in coreference annotation. For the time being, the annotated resources are still too scarce to make near-identity identification systems practical — both in terms of conception and in terms of evaluation.

2.1.2 Referring mechanisms and interpretation

When referring to an entity (and possibly simultaneously introducing it in a discourse model),
there are several possibilities for the emitter as to the choice of a referring expression. The
primary constraint in this choice is that the chosen expression can reliably be interpreted by
the receiver as referring to the correct entity — even if the choice can also be influenced by the
need to convey attributes of this entity (such as in example 8) among other considerations.

(8) *Do you remember Eyjafjallajökull? The eruption of this Icelandic volcano in 2010 stopped the air traffic for 48 hours.*

It is also to be expected that participants in an interaction try to conform to some sort of
convention, e.g. along the lines of Grice's cooperation principles (Grice 1989), which usually
implies that the chosen referring expression will be as concise as possible, all other things being
equal.<fnref n="3" />

When referring to an entity, a speaker has a choice between several modes of reference. This
choice is conditioned by the current *linguistic, situational and discursive contexts*. To be
precise:<fnref n="4" /> by *discursive context*, we mean the state of the discourse model ; by *situational context*,

<footnote n="2">²This identification between singleton mention and singleton entities has sometimes been the cause of heated
debates — as have been other fine points regarding singletons.</footnote>

<footnote n="3">³Let us stress, however, that this is a very rough approximation that should not be taken as an absolute formal
requirement or — in the words of Ellen Prince — ‘TO BE TAKEN WITH LARGE GRAIN OF SALT’.</footnote>

<footnote n="4">⁴But by no mean exhaustive. For example this does not account for the fact that the extralinguistic context might</footnote>
we mean the extralinguistic (including metalinguistic) elements of the interaction and their
perceptions by the participants ; and by *linguistic context*, we mean the linguistic content of the
interaction up to the production of a referring expression - most notably the order, recency and
forms of preceding entity mentions.

This gives us a rough typology for referring expressions according to their respective degree of
dependency to these contexts.

There are referring expressions whose interpretations depend essentially on encyclopedic know-
ledge that is assumed to be shared between discourse participants. Proper nouns (example 9) are
the prototypical case for this mode of reference, although some definite descriptions (example 10)
also fall in this category.

(9) *Lovelace* was the only legitimate child of poet Lord Byron and his wife Lady Byron

(10) *The first exoplanets to be discovered are orbiting the pulsar PSR B1257+12, also known as 'Lich'.*

These expressions can be thought of as referring to a discourse entity *via* a link to a entity in
that shared knowledge space.<fnref n="5" /> As such, their interpretations does not rely on the pre-existence
of the corresponding entities in the discourse model or the forms of their previous mentions.
However, their forms are not completely independent of the context, since the assumptions
made by the participants on their common knowledge have an influence on the degree of
specificity that is used in these designations.

For instance, if a participant is assumed to ignorant of an entity, its first mention will also serve
as an introduction. This can be seen in example 11: while the natures of ‘Orléans’ and ‘Paris’
can be assumed to be part of the general knowledge of fluent French speakers and are thus
simply designated by their proper name. ‘Lycée Pothier’ is not as well know, yet it is not further
specified. This is presumably because the speaker assumes that the interviewer has a some
knowledge of the Orléans area — where the interview takes place and ‘Lycée Pothier’ is part
of the general knowledge. Conversely, ‘Argenteuil’, a relatively well-known French city, is
qualified as ‘in the suburbs’. This might be due to the assumption from the speaker that their
interlocutor — not being French themselves — might be unfamiliar with Argenteuil and that
in this case this will give them enough information to infer the attributes of the ‘Argenteuil’
entity that are relevant at this point in the interaction: its nature and location.

(11) [...] je suis né à Orléans [...] et j'ai fait mes études au lycée Pothier [...] j'ai travaillé à Paris également à Argenteuil dans la banlieue [...]

([...] I was born in Orléans [...] and I did my education at the lycée Pothier [...] I have worked in Paris also in Argenteuil in the suburbs [...]) (ANCOR, Muzerelle et al. 2014)

not be the same for all participants or intentional obfuscation from the emitter. Still, it should be sufficient for our
needs in this work.

<footnote n="5">⁵For the purpose of this discussion, we will assume that encyclopedic knowledge functions as a background
mental space specific to each individual and that participants in an interaction can make assumptions and deductions
about one another’s encyclopedic knowledge.</footnote>
Note on the other hand that the actual referential nature of proper nouns is not always so clear-cut. For instance in example 12, the ‘Nobel’ part of the proper name ‘Nobel Prize’ does not necessarily create a discourse entity corresponding to Alfred Nobel, althoug in the terminology of Prince (1981), it does make it inferable, which allows the use of the pronoun ‘he’ in the next sentence.

(12) Marie Skłodowska-Curie is the only person to have received the **Nobel Prize** in two different scientific fields. Had **he** known her, he would probably have agreed that this distinction was well-deserved.

Similarly, the referential mechanism of *deictic expressions* mostly depends on a shared perception of the *situational context*. For instance, in example 13, the reference from ‘ici’ to the ‘Orléans’ entity is only inferable through knowledge of the location where the interaction is taking place.

(13) [...] j’ai des camarades qui ne sont pas du tout d’Orléans et [...] qui ne trouvent pas la ville sympathique mais moi j’ai toujours vécu ici [...]

([...]) I have friends who are not from Orléans at all and [...] who do not think the city is nice at all but I, I have always lived here [...])

(ANCOR, Muzerelle et al. 2014)

The situational context might include multimodal communication elements, as in example 14 where an accompanying gesture is the main factor of interpretation of ‘celui-là’

(14) j’aurais voulu la même chose en français mais pas celui-là [...]

(I would like the same thing in French but not **that one** [...])

(ANCOR, Muzerelle et al. 2014)

Following Muzerelle et al. (2014), we also include discourse and metalinguistic deixis in this category, as ‘la question’ and ‘« on »’ respectively in example 15.

(15) nous avons dit « **on** » je vous quand je vous ai posé **la question** j’ai demandé est-ce qu’on fait assez pour les habitants d’Orléans ? et « on » ça représente qui pour vous alors ?

(we said ‘on’ I you when I ask you **the question** I asked if ‘on’ did enough for the people of Orléans? and who is ‘on’ for you then?)

(ANCOR, Muzerelle et al. 2014)

Expressions that refer intensionally to entities via their attributes, their relations to other entities or both depend mostly on the discursive context. The precision of these intensional descriptions might range from complex predicates (example 16) to very general qualifications (example 17) and might be based on indirect information, such as morphological marks (example 18).

(16) [...] alors comme vacances avec **mon amie de Tours** j’espère euh aller au à la Costa Brava [...]

([...]) so as vacations with **my friend from Tours** I hope uhm to go to the Costa Brava

(ANCOR, Muzerelle et al. 2014)

(17) I got on **a bus** yesterday and **the driver** was drunk. (Prince 1981)

<footnote n="6">⁶Not translated to preserve the metalinguistic meaning in this example, but might be translated by **they** although **on** might also include first and second person referents, see Delaborde and Landragin (2019) for more details.</footnote>
(18) *I saw a man with a dog yesterday and it was beautiful.*

Note that this mode of reference does not imply that the corresponding entity is already present in the discourse model at the time of utterance. Indeed, in example 17, ‘a bus’ and ‘the driver’ are actually the first mentions of their respective entities. In extreme cases such as example 19,<fnref n="7" /> the intensional description can even be completely implicit.

(19) *She doesn't bike, though she owns one.*

Finally, the interpretation of some referring expressions mostly depends on the *linguistic* context. In these cases, the *discourse* reference to an entity might be *via* a linguistic reference to another expression. For instance, due to the syntax of dislocations in French, ‘il’ in example 20 is acting as a placeholder for the expression ‘votre journal habituel la République du Centre’, so they must refer to the same entity.

(20) [...] *votre journal habituel la République du Centre est-ce qu'il contient une rubrique sur le langage?*

([...] *your usual newspaper La République du Centre does it include a column on language?*)

(ANCOR, Muzerelle et al. 2014)

Note that a given expression does not necessarily have a single mode of reference: for instance disambiguating between two entities with the same proper name might necessitate further qualifications when both are plausible in the current context.

## 2.2 Coreference in natural language processing

The term *coreference* in the context of natural language processing was introduced as part of the MUC-6 shared task (MUC Consortium 1995c) in the group of the ‘SemEval’ group of subtasks. Its definition was mostly informal and not linked to a particular theoretical framework of *reference*

*The basic criterion for linking two markables is whether they are coreferential. Whether they refer to the same object, set, activity, etc. It is not a requirement that one of the markables is ‘semantically dependent’ on the other, or is an anaphoric phrase.*

(MUC Consortium 1995b)

With ‘markables’ defined by

*The coreference relation will be marked between elements of the following categories: nouns, noun phrases, and pronouns.* Elements of these categories are *markables*.

[...] *The relation is marked only between pairs of elements both of which are markables.*

(MUC Consortium 1995b)

and annotated as

<COREF ID="100">Lawson Mardon Group Ltd .</COREF>
said <COREF ID="101" TYPE="IDENT" REF="100">it</COREF>

<footnote n="7">⁷Marcel Bollman is to thank for this one.</footnote>
Therefore, in the context of this task, *coreference resolution* is defined as the process of detecting every mention *m* in a document that

* Is a constituent within a restricted set of types

* Is coreferent with at least one mention *m'* found earlier in the document, its 'antecedent' and annotating both *m*, *m'* and the *directed link* *m*' ← *m*.

This formulation makes use of the fact that coreference is an *equivalence relation* to simplify the task. Consider the graph Γ over the set M of all mentions induced by the coreference relation: the coreference chains are its connected components, and since coreference is transitive, they are completely connected graphs — or *cliques*. Therefore, identifying coreference chains can be reduced to the identification one of its *spanning trees* and, since coreference is symmetric, it suffices to identify one of its *spanning directed trees*, i.e. to associate a unique antecedent to all but one of its mentions.

The search space is often restricted even further by constraining the direction of the links to be consistent with the order of appearance of the mentions in the discourse or in the extreme case, to allow only chains<fnref n="8" /> (hence the expression *coreference chain*) or stars.<fnref n="9" /> In the latter case, this formulation makes coreference resolution a proper generalization of the pronominal anaphora resolution task (see section 2.3.2) by identifying a canonical antecedent for every discourse-old mention. In this approach, the units of interest are the directed links between coreferent mentions and the entities are *latent*. This formulation is called the *mentions pair* or *link-centric* model of coreference (Recasens 2010).

It is also possible to use the dual, more discourse-oriented, approach of modelling *reference* directly and keeping *coreference* implicit. In that formulation it is not the coreference links between the mentions that are modelled but the links between the mentions and their corresponding entities in what is called the *entity-mention* model of coreference. It comes in two flavours: one (seen for instance in OntoNotes (Hovy et al. 2006)) models reference as a property of individual mentions, and deals with directed mention→entity links and makes the *mentions* the central units ; the other (seen for instance in ACE (Doddington et al. 2004)) models reference as a property of entities, deals with directed entity→mention links and makes *entities* the central objects.

Choosing among these three models of coreference has an influence in all the parts of the
natural language processing treatment of coreference, which can be seen clearly in the diversity
of the system evaluation metrics (section 2.4.2), annotation conventions (chapter 3) and system
designs (chapter 4). Note however that these models are all equivalent and that the choice of
one in a part of a task does not mandate the choice for the other parts, for example it is possible
to evaluate an entity-centric entity→mention system (for instance CISTELL (Recasens 2010))
using a link-based metric (for instance MUC (Vilain et al. 1995)).

It is also possible to combine several approaches: e.g. a resource might include annotations

<footnote n="8">⁸Trees of maximal degree 2.</footnote>

<footnote n="9">⁹Trees of diameter 2.</footnote>
for both coreference chains and coreference links (as in the URS representation of ANCOR,
see section 3.3). In practice, most implementations have in common a relative lack of details
regaring the attributes of the entities and the type of (co)reference relations (but see section 2.3.3)
and identify entities and their corresponding coreference chains.

As a result, in the context of natural language processing, given a document, *coreference resolution*
is usually understood as the twofold task of

**Mention detection** Identifying the mentions in this document

**Coreference resolution proper** Identifying the coreference chains, with three alternatives

1. Associate every discourse-old mention with one of its antecedents (*mention-pairs* or *link-centric* model)

2. Affect every mention to the corresponding entity (*mention-centric entity-mention model*)

3. Partition the set of all mentions (*entity-centric entity-mention model*)

Note that these subtasks are not necessarily treated sequentially (see chapter 4).

A recurring question in practice is the treatment of singletons. Some implementations (such
as MUC and OntoNotes) consider that since singleton mentions are not actually **coreferring,**
they should not be counted as mentions for a coreference task. As a result, for these shared
tasks, the associated resources and the systems designed for them, ‘mention’ does not include
singletons. The opposite point of view is that singleton mentions are still *referring* and that as
such they still define coreference chains, if degenerated ones.

While both alternatives have upsides and downsides, this choice clearly has a substantial impact
on resource constitution, system design and evaluation and should be made explicit when
describing a task, a resource or a system.

## 2.3 Related tasks

### 2.3.1 Named entity recognition and Entity linking

*Named entities recognition* is closely linked to coreference resolution since their common inception as subtasks in MUC-6. Ubiquitous in information extraction, the concept of *named entity* is notoriously hard to define formally, partly due to its origin in applicative rather than theoretical contexts. The most comprehensive attempt at a formal definition is perhaps due to Ehrmann (2008):

> *Étant donnés un modèle applicatif et un corpus, on appelle entité nommée toute expression linguistique qui réfère à une entité unique du modèle de manière autonome dans le corpus.*

*(Given an applicative model and a corpus, we call **named entity** any linguistic expression that autonomously refers to a single entity of this model) (Ehrmann 2008)*
Thus, named entities *recognition* can be seen as a restriction of the mention detection sub-
task of coreference resolution, where the mentions that are considered are those that refer
‘autonomously’ — i.e. depending only on encyclopedic knowledge as defined in section 2.1.2 —
to a specific set of entities that are relevant for a specific applicative task. In practical imple-
mentations, the set of mentions is often further restricted to specific types of expressions. For
instance, while arguably autonomously referring (Ehrmann 2008), definite descriptions and
— more heterodoxically — speaker-referring pronouns, are seldom included in named entity
recognition tasks.

This *detection* task is almost always associated with a task of *typological classification*, where
detected mentions are sorted according to the nature of the corresponding entity in a given
taxonomy. The extent and the granularity of these taxonomies is also task-dependent, from
the relatively rough categories of MUC-6 (organisation, person, location, date, time, monetary
amount, percentage) to the highly specific hierarchies of the Quaero corpus (Rosset et al. 2011)
or Sekine and Nobata (2004) for instance.

*Entity Linking* is a further extension of named entity recognition, where the detection of
named entities is associated with a *linking* task that associates the detected mentions with
the corresponding entities, usually modelled as entries in a knowledge base. This can be
seen as an extreme case of the entity type detection — where the types are of the maximum
possible granularity — or as the analogue for named entities of the second subtask of coreference
resolution. Historically, entity linking was initially proposed as an *extension* of coreference
resolution to cross-document settings (Bagga and Baldwin 1998b) and by cross-pollination
with word sense disambiguation challenges gradually evolved to its current form as large scale
knowledge bases became available (Hachey et al. 2013).

While these two tasks are obviously closely related to coreference resolution, there has been
relatively little synergy in system designs. This is in part due to the fact that their restricted
set of types for mentions and entities allows for very convenient heuristics. In particular, most
implementations only consider mentions that are either non-nestable (as in MUC-6 (MUC
Consortium 1995a)) or whose structures are restricted to a limited degree of nesting (Rosset
et al. 2011), contrasting with the potentially unrestricted nested structures of mentions in core-
ference resolution. With these restrictions, named entity recognition can be seen *segmentation*
task, which in turn can be conveniently cast as a *sequence labelling* problem or — in the case
of nested mentions — to successive sequence labellings (Dupont 2017; Straková et al. 2019).
This formulation has the advantage of enabling the use of well-studied and efficient labelling
algorithms. In particular, machine-learning algorithms such as linear-chain conditional random
fields with Viterbi decoding (Lafferty et al. 2001) have yielded remarkable advances in named
entity recognition in the last two decades.

As for entity linking, due to both historical reasons and the relative easiness of narrowing the
referent of a mention down to a few homonymous entities in the reference knowledge base
(Hachey et al. 2013), the techniques employed are closer to those used in word disambiguation
and usually rely on ontological properties of the candidates entities. These approaches are not
readily usable for coreference resolution, where entities are not necessarily well-defined and
their properties are not usually easily inferable (see section 2.1.1).

### 2.3.2 Narrow coreference resolution

In some contexts, it might be beneficial to attempt restricted versions of coreference resolution rather than the full version we describe in section 2.2. Indeed, since reference encompasses a great variety of mechanisms, it seems only natural that designing a system focusing on a narrower problem could be both easier and more useful in some situations. In particular, several tasks that focus only on certain types of mentions or certain types of relations have received significant attention, both historically before the advent of unrestricted coreference resolution tasks, and currently in the form of domain-specific tasks. We call these tasks *narrow* coreference resolution tasks and describe in this section some of the most important ones.

Note that coreference resolution as it is implemented in existing campaigns and datasets is itself actually narrow, as it usually focuses only on nominal and pronominal reference. The distinction we make is that in these cases, the narrowness is not usually a choice motivated by a theoretical or applicative objective, but rather a circumstantial necessity caused by the difficulty of actually annotating unrestricted coreference.

#### Pronominal reference resolution

*Pronominal reference resolution* is the restriction of coreference resolution to pronominal mentions, usually formulated as an antecedent-finding task. Formally: 'For every pronoun *p* in a document, find a noun phrase *n* such that *p* and *n* are coreferent'.

Historically, this task was the first type of coreference resolution implemented in natural language processing systems. Poesio et al. (2016c) identifies two reasons for this

> There are two reasons for this focus on pronouns: a theoretical one – pronominal ana-
> phora is much more governed by grammatical competence that full nominal anaphora
> – and a practical one – interpreting pronouns depends less on lexical, commonsense
> and encyclopedic knowledge than other types of anaphoric interpretation; hence, shal-
> low approaches are more likely to achieve good results for this type of anaphora
> (Poesio et al. 2016c, p. 78)

to which we add another practical reason: in downstream applications of coreference resolution, such as natural language understanding or machine translation, a reliable interpretation of pronominal reference is crucial, as a system can not rely solely on the content of a pronoun to derive a useful meaning.

This should not taken as meaning that pronominal reference resolution is a generally easier restriction of coreference resolution. If very simple baselines – such as choosing the closest noun phrase with compatible morphology (Recasens and Hovy 2010) – and naïve applications of government and binding heuristics can be sufficient for many pronouns, a non-negligible number of pronominal reference do not follow these criteria. To give but a single example Antoine (2004) reports up to 12% of non-agreement between pronouns and their antecedents.
Moreover, ambiguous pronominal reference is both far from uncommon in attested corpora and challenging even for the current best systems (Webster et al. 2018). In fact, Levesque et al. (2012) propose to use ‘Winograd schemas’ — specific examples of pronominal reference — as an alternative to the Turing test. Specifically, considering the following pair of sentences:

(21) (a) *The trophy would not fit in the brown suitcase because it was too big.*

(b) *The trophy would not fit in the brown suitcase because it was too small.*
(Levesque et al. 2012)

reliably identifying the referent of ‘it’ in both sentence would involve a deep understanding of both the linguistic content of the sentences and the situation that they entail. It could actually be considered an AI-complete problem given certain guarantees to ensure that no side-channel is available. Studies on human performances on this task (Amsili and Seminck 2017) have shown that in the overwhelming majority of cases, human subjects do not experience any issue in solving Winograd schemas.

This is still an active area of research, with the recent work of Sakaguchi et al. (2019) providing more robust ‘adversarial’ Winograd schemas as an alternative to those introduced by Levesque et al. (2012) and those used as part of the GLUE (A. Wang et al. 2018) and SuperGLUE (A. Wang et al. 2019) benchmarks for natural language understanding.

## Entity detection and tracking

*Entity detection and tracking* is an extension of the named entities recognition task to unrestricted mentions. Like named entity recognition, it focuses only on a few types of entities, but instead on focusing only on autonomously referring mentions, it consists in detecting *all* the mentions of these entities and their types (the ‘detection’ part) and linking coreferent mentions (the ‘tracking’ part). As such, unlike named entities recognition, it is a true narrow coreference resolution task.

In the context of the Automatic Content Extraction (ACE) shared task (Doddington et al. 2004) — the most famous implementation this task —, the types considered were ‘Person’, ‘Organization’, ‘Location’, ‘F’acility’, ‘Weapon’, ‘Vehicle’ and ‘Geo-Political Entity’, thus for instance ‘a hoax’ in ‘President Bush says it’s all a hoax’ is not considered as a mention.

## Event coreference resolution

*Event coreference resolution* is the restriction of coreference resolution to a specific set of entities that represent events of a limited number of types. It is usually treated separately from strictly (pro)nominal coreference resolution, mainly because the mentions that are considered are much more heterogeneous — ranging from pronouns to full propositions — and because their meaning is usually less readily accessible (Lu and Ng 2018). In addition to detection of event mentions and coreference resolution, implementations of this task also usually include the identification of some characteristics of events such as their types, participants, times or reality (Song et al. 2015).
While this narrow coreference resolution task is theoretically rather specific, neglecting it can
have considerable consequences even on nominal coreference resolution. Consider for instance
the case of example 22:

(22) [...] *j'ai pas d'argent* euh je suis en dettes à l'EDF déjà de trois ou quatre millions enfin ça
va ça ne durera pas de toutes façons ça

([...] *I have no budget* uhm I have debts to EDF of already three or four millions, but it's
alright, it won't last anyway this) (ANCOR, Muzerelle et al. 2014)

In that case, not taking ‘j’ai pas d’argent’ into account would make the corresponding chain
consist only of pronouns, which significantly lessens the usefulness of detecting this chain.
Experimental results (H. Lee et al. 2012) have also shown that including event coreference
as part of a coreference resolution system can significantly improve the quantitative results
for both tasks. Accordingly, in some implementations — such as Ontonote’s (Pradhan et al.
2007b)— event mentions are annotated if they corefer with at least one (pro)nominal mention,
thus reaching a middle ground between purely (pro)nominal coreference annotation and the
considerably more complex task of annotating all reference phenomena (see chapter 3).

2.3.3 Anaphora resolution

Following the definition of Poesio (2016), we say an expression is *anaphoric* if it '[depends] on the linguistic context, i.e., on objects explicitly mentioned or objects whose existence can be inferred from what has been said' (Poesio 2016, p. 24). As we have seen in section 2.1.2, the interpretation of referring expressions can be (and is in fact often) context-dependent, which makes these expressions anaphoric. But anaphoricity is not limited to the *identity reference* phenomenon on which we focus on in this work and also encompass the cases of near-identity coreference (Recasens et al. 2011a), discourse deixis (Webber 1991) – a superset of event coreference –, but also pro-forms, ellipsis and numerous others (see Hirst (1981) for a comprehensive review).

*Anaphora resolution* is the task of detecting anaphoric expressions and the contextual elements on which its interpretation depend, usually in the form of other linguistic expressions called their *antecedent*.<fnref n="10" /> In practical implementations, it also usually includes resolving non-anaphoric identity references such as named entity coreference.

As we mentioned, the first implementations of a coreference resolution task in natural language
processing was pronominal coreference resolution, which is usually anaphoric. As a result,
and because of other uses of the terms *anaphora* and *coreference* in other areas of linguistics,
a durable terminological conundrum has taken place, described in details by Poesio (2016,
p. 38–39) where *anaphora* and *coreference* are used to denote various closely linked phenomena.
This interpretation has also largely influenced research on coreference resolution, both in its
terminology – for instance the use of the word *antecedent* for a preceding coreferent mention –
and in its techniques – for instance the formulation of coreference resolution as an antecedent-
finding task (see chapter 4).

<footnote n="10">¹⁰Note that in that case, the antecedent does not necessarily precede the mention, although it is by far the most
common situation. The case where the mention precedes its antecedent is traditionally called ‘cataphora’.</footnote>
## 2.4 Evaluating coreference resolution

As we have seen in section 2.2, coreference resolution is a twofold task, with two very different components.

The first subtask — mention detection — has a strong syntactic component, and has historically been highly dependent on syntactic analysis. However, it is not a purely syntactic task, since not all noun phrases and pronouns are referential and reliably identifying these is far from trivial, even with expert linguistic analysis. For instance, discriminating non-referring, partially or fuzzily referring and fully referring pronouns is still an active area of research (Delaborde and Landragin 2019).

The second subtask — identifying coreference chains — is almost purely a discourse-level task and is usually more complex. Consequently, most of the focus in the literature has been on this subtask, both in terms of system conception and of evaluation design, with the result that this it is usually called ‘corefrence resolution’ itself and that the full task including mention detection is sometimes called ‘end-to-end’ coreference resolution. Another influential factor is the choice in the MUC-6 and MUC-7 shared tasks — the initial task codifiers — and in Ontonotes — the most common benchmark for coreference resolution in English — not to annotate (and therefore not to *evaluate* on) singleton entities. This conflation of mention detection and singleton detection makes it hard to disentangle the contribution of mention detection to the performance of end-to-end systems and provides little incentive to study and evaluate the performance on mention detection only.

Yet, there is a large body of evidence (Moosavi et al. 2019; Moosavi and Strube 2016; Popescu-Belis et al. 2004; Uryupina et al. 2016; Uryupina and Moschitti 2013) for the claim that the performance of mention detection has a crucial impact on the performances of end-to-end coreference resolution systems. Following Popescu-Belis et al. (2004), Recasens (2010) and Recasens and Hovy (2011), we thus treat mention detection as a task of its own, with dedicated evaluation.

### 2.4.1 Mention detection

Mention detection is in particular a *detection* task. As such, it is only natural to evaluate the output of a mention detection system — whether it comes from a dedicated module or is part of a holistic end-to-end processing — using the precision, recall metrics introduced by Kent et al. (1955) and the unweighted F-score defined by van Rijsbergen (1979) and popularized by Chinchor (1992) for information retrieval and commonly used for other detection tasks.

Formally, given a set of *t* true samples, i.e. elements to find in a collection, and a system output of *p* positive samples, i.e. elements — mentions in our case — detected by a system in the same collection, of which *tp* elements are in fact true positive samples, the precision *P*, the recall *R* and
the F-score F₁ are defined by

$$ 
\begin{align*}
P &= \frac{tp}{p} \\
R &= \frac{tp}{t} \\
F_1 &= \frac{2}{\frac{1}{P} + \frac{1}{R}} = \frac{2 \cdot P \cdot R}{P + R} = \frac{2 \cdot tp}{t + p}
\end{align*} 
\qquad [2.1] $$

Note that F₁ is in particular the unweighted harmonic mean of P and R, viz. the Pythagorean mean that is the most punitive to imbalance.

Yet this does not wholly solve the problem of mention detection evaluation. Indeed, if these metrics measure the global quality of a system output, one still has to define what it means to have successfully detected a given mention. According to the definitions given in section 2.1.1, a mention is a linguistic expression that refer to an entity. If for some mentions such as pronouns such as ‘they’ or proper nouns such as ‘William Labov’ the exact boundaries are clear, this leaves a wide range of options regarding what to include in a given mention.

Consider for example a text span such as ‘Haden MacLellan PLC of Surrey, England’ (Hirschman and Chinchor 1998). From a purely syntactic point of view, it behaves as a single element that could be replaced by a pronoun and that a detection system should therefore detect all of it. However, from a task-oriented perspective, detecting only ‘Haden MacLellan’ is not completely wrong, and it would seem reasonable to give partial credit to a system that would do so. This was dealt with in MUC by providing maximal and minimal spans for every mention, and report two separate sets of measures: one for strict detection of the maximal spans and one for any detecting any span that would be both a proper subspan of the maximal span and a proper superspan of the minimal span.

Yet, a double annotation for every mention is a time-consuming task and the time it takes is usually more useful to invest in other task for an annotation project. To address this problem, Recasens (2010) proposes an alternative ‘lenient matching’ where the syntactic head of a mention serves as its minimal span, and Moosavi et al. (2019) a more complex algorithm to infer minimal span from parse trees. However, both of these proposals still require access to a reliable syntactic analysis, which is not always the case, leaving open the question of how to evaluate partial mention detection in general. Options such as those sometimes used for named entity recognition of attributing partial credit for correctly detecting one of the two boundaries of a mention or using overlap ratios has not yet seen much interest for mention detection evaluation.

### 2.4.2 Coreference resolution metrics

Quantitative evaluation is a very important component in the definition of most natural language processing tasks and is usually conditioned on the definition of an evaluation *metric*, which evaluates the proximity of a system’s output from a reference annotation. In that regard, coreference resolution stands out by its unusual number of competing metrics. Since the first proposition of Vilain et al. (1995) for a dedicated metric to evaluate and compare the
performances of coreference resolution systems (for the MUC-6 shared task), there has been at least five main counterproposals, with each one designed to overcome the shortcomings of the previous ones. In the current state of the art, there is no clear consensus (Poesio et al. 2016b) as to which of these metrics — which are not necessarily consistent with each other — should be used to rank coreference resolution systems.

There are multiple reasons for this situation: *historical* — the development and use of metrics being dependent on the availability of suitable resources — and *theoretical*, with the difficulty of formally defining a single degree of correctness for a given system output. Indeed, the addition of a single spurious mention in a chain can be a glaring error — for example a mistaken attribution in a Winograd Schema such as example 21 —, but grouping together nearly-coreferent mentions might be tolerable for most purposes — for example including ‘*le commun des soldats*’ in *the emphasized chain* in example 23.

(23) *Les soldats qu'il avait commandés en Sicile se donnaient un grand festin [...] ils se trouvaient nombreux, ils mangeaient et ils buvaient en pleine liberté [...] le commun des soldats était répandu sous les arbres [...]*

*(The soldiers that he commanded in Sicily were having a great feast [...] they were many, they were eating and they were drinking liberally [...] The common soldiers were spread out under the trees)*
(Salammbò, Flaubert 1910)

Moreover, the human-judged severity of these errors is often inversely correlated with the propensity of existing systems to fall into them. This is the case for example 23: it is not terribly complicated for an automated system to learn that a partitive such as ‘Le commun des’ entails a restriction and prevents identity coreference, but as we saw in section 2.3.2, reliable automatic solving of Winograd schema is by design highly non-trivial and potentially AI-complete. This issue is also magnified by the relative scarcity of the phenomena causing these errors in actual corpora.

In the following, we consider a single document and the output of a coreference resolution system for this document and we note

* $M_K$, the set of all gold (or ‘key’ mentions), $M_R$ the set of all system (or ‘response’) mentions and $M = M_K \cup M_R$ the set of all mentions

* K, the set of all gold coreference chains, which is a partition of $M_K$ and R, the set of all system coreference chains, which is a partition of $M_R$

Except for MELA, these metrics all define a precision P and a recall R and a $F_1$-score defined as usual as the unweighted harmonic mean of P and R:

$$F_1 = \frac{2 \cdot P \cdot R}{P + R} \qquad [2.2]$$
MUC

The MUC score (Vilain et al. 1995), was the first metric used for a full coreference resolution task. Vilain et al. (1995) defines the MUC precision and recall as

$$
\begin{align}
P &= \frac{\sum_{r \in R'} (|r| - |p(r, K)|)}{\sum_{r \in R'} (|r| - 1)} \nonumber \\
R &= \frac{\sum_{k \in K'} (|k| - |p(k, R)|)}{\sum_{k \in K'} (|k| - 1)} \tag{2.3}
\end{align}
$$

where $K' = \{k \in K | |k| > 1\}$ and $R' = \{r \in R | |r| > 1\}$ are the sets of non-singleton chains in key and response respectively and $p(x, E) = x/E = \{x \cap A | A \in E\}$ is the partition of a chain $x$ induced by a partition $E$ of $M$.

These correspond to the precision and recall over the set of links between discourse-old mentions and their closest antecedent. Indeed, for a given non-singleton key entity $k$, there are $|k| - 1$ such links and $|p(k, R)| - 1$ of them are missing in the response. Thus, summing over $K$, $R$ is the proportion of the key links that are present in the response. A similar analysis applies to $P$ by switching the roles of the key and response.

Introduced as part of the MUC-6 conference, MUC is designed accordingly to its link-based coreference model and inherits its biases. The most notable of these biases is the choice not to annotate singleton mentions, reflected in the assumption in MUC that chains contain at least one link. This is not to say that singleton chains, in particular *response* singleton chains are completely ignored, since they still contribute to the $|p(r, \cdot)|$ term.

A more severe issue with the MUC score is that since it only consider links to the closest antecedent, mistakenly including a mention in an chain counts as an error in both precision and recall, while mistakenly merging two entities only counts as a single recall error (Bagga and Baldwin 1998a; Recasens and Hovy 2011). In fact, Finkel and Manning (2008) note that the edge heuristic of including all mentions in a single response chain would obtain a MUC F₁-score<fnref n="11" /> of 88.2%, which was higher than all the then-published coreference resolution system.

B³

The B³ score<fnref n="12" /> (Bagga and Baldwin 1998a) was an answer to the aforementioned shortcomings of MUC. It is a mention-centric metric, originally defined<fnref n="13" /> as

$$
\begin{align}
P &= \frac{\sum_{m \in M_R} \frac{|r(m) \cap k(m)|}{|r(m)|}}{|M_R|} \nonumber \\
R &= \frac{\sum_{m \in M_K} \frac{|r(m) \cap k(m)|}{|k(m)|}}{|M_K|} \tag{2.4}
\end{align}
$$

where $r(m)$ (resp. $k(r)$) is the response (resp. system) chain containing mention $m$.

<footnote n="11">¹¹On the MUC-7 evaluation dataset.</footnote>

<footnote n="12">¹²Or ‘B-CUBED’ in the original version, given without explanation for the name.</footnote>

<footnote n="13">¹³In the unweighted version that is the most common in subsequent works. They also define a more general version with per-mention weights that amounts to computing a weighted average instead of an unweighted one.</footnote>
This definition assumes that R and K are indeed partitions, which historically has not always been the case due to the presence of duplicated mentions in actual system responses. Due to the choice in the CoNLL campaigns not to enforce this condition, an alternative definition of B³ was proposed by Pradhan et al. (2014):

$$ 
\begin{aligned}
P &= \frac{1}{|M_R|} \sum_{r \in R} \sum_{k \in K} \frac{|r \cap k|^2}{|r|} \\
R &= \frac{1}{|M_K|} \sum_{k \in K} \sum_{r \in R} \frac{|k \cap r|^2}{|k|}
\end{aligned} 
\quad [2.5] $$

However, this still does not address all the issues that can arise with duplicated mentions. In fact, when dealing with duplicated system mentions, the recall is not necessarily smaller than 1 and can even be driven to an arbitrary high values.

B³ takes into account the correct identification of singleton entities, which was one of the motivations to create it and use it over MUC. However, the reward for correctly identifying singletons might be too high given the large ratio of singleton in actual documents: Recasens and Hovy (2011) found a B³ score of 59% on ACE-2004 for an all-singleton baseline while noting that the singleton rate in ACE is actually lower than in other corpora.

## CEAF

The Constrained Entity Aligned F-Measure (CEAF) if a family of metrics introduced by Luo (2005) and designed to address the issues arising from using B³ to evaluate the output of end-to-end systems, which usually contain spurious mentions and can potentially contain duplicates of gold mentions. Luo (2005) define it as

$$ 
\begin{aligned}
P &= \frac{\sum_{r \in R} \phi(A^{-1}(r), r)}{\sum_{r \in R} \phi(r, r)} \\
R &= \frac{\sum_{k \in K} \phi(k, A(k))}{\sum_{k \in K} \phi(k, k)}
\end{aligned} 
\quad [2.6] $$

where $\phi: \mathcal{P}(M)^2 \to \mathbb{R}^+$ is a scoring function and A is a one-to-one mapping<sup>14,15</sup> from key chains to response chains that maximizes $\sum_{k \in K} \phi(k, A(k))$. Of course, the value of these metrics is highly dependent on the choice of $\phi$, which should intuitively be a measurement of how similar two entities are, with the constraints that for all (E, F) $\in \mathcal{P}(M)^2$, $\phi(E, F) = \phi(F, E) \le \phi(E, E) \ne 0$ and that if $E \cap F = \emptyset$, then $\phi(E, F) = 0$.

The two commonly used variants are the ‘mention-based’ CEAF<sub>m</sub>, which uses $\phi_3$, the cardinal of the intersection, for $\phi$

$$ \phi_3 : (k, r) \mapsto |k \cap r| \quad [2.7] $$

and the ‘entity-based’, CEAF<sub>e</sub>, which uses the Sørensen-Dice coefficient $\phi_4$ (Dice 1945; Sørensen 1948)

$$ \phi_4 : (k, r) \mapsto \frac{2|k \cap r|}{|k| + |r|} \quad [2.8] $$

<footnote n="14"><sup>14</sup>If K and R are not of the same size, they might be augmented with copies of the empty set.</footnote>

<footnote n="15"><sup>15</sup>One such mapping can be obtained efficiently using the Kuhn-Munkres algorithm (H. Kuhn 1955; Munkres 1957).</footnote>
The qualifications of ‘mention-based’ (corresponding to the mention-centric entity-mention model of coreference) for CEAF<sub>m</sub> comes from the fact that φ<sub>3</sub>(e, e) = |e|, which makes it the average over all mentions *m* of an attribution score of 1 if the key and response chains corresponding to *m* are aligned and 0 else. Similarly, CEAF<sub>e</sub> can be seen as the average overlap between the aligned key and response chains, since for all *e*, φ<sub>4</sub>(e, *e*) = 1 and thus correspond to the entity-centric entity-mention model of coreference.

CEAF solves most of the issues identified for B³ relative to the evaluation of system outputs with
spurious and potentially duplicated mentions by enforcing a single alignment between key and
response chains. However, it suffers from the same issues regarding singleton mentions: when
singletons are in majority in K, they make up most of the score, leaving relatively little room
for meaningful comparisons (Recasens and Hovy 2011). It also has a bias<fnref n="16" /> against oversplitting
chains: since the alignment is one-to-one, splitting a large key chain in two response chain of
similar size results in a heavy loss for a mistake that is not necessarily critical.

MELA (CoNLL)

The Mention, Entity, and Link Average score (MELA), introduced by Denis and Baldridge (2009) combines the previous metrics in an attempt to compensate for their different biases. It is simply computed as the weighted average of the F₁ score of MUC (a Link-based metric), B³ (a Mention-based metric) and a CEAF metric (an Entity-based¹⁷ metric). Despite its mostly empirical motivation, it gained a lot of exposure following the use of the unweighted variant for the CoNLL-2011 and 2012 shared tasks on coreference resolution, where it provided a single score to rank the participating systems, leading to the name ‘CoNLL score’ that is now common in the literature. While it could be argued that by combining three differently biased metrics, this one might have even worse biases, in actual evaluation campaigns using the OntoNotes dataset, the rankings given by the CoNLL score were usually consistent with those given by each individual metric. An assessment of this finding is still to be done for other implementations of the task.

BLANC

The BiLateral Assessment of Noun-phrase Coreference (BLANC), introduced by Recasens and Hovy (2011) and extended to the case M<sub>K</sub> ≠ M<sub>R</sub> by Luo et al. (2014) uses a variation of the Rand index (Rand 1971) to measure the cluster similarity between K and R by focusing on the *links* between mentions.

A coreference (resp. non-coreference) link is a couple *m*,*m'* of mentions such that *m* ≠ *m* and
*m* and *m'* are coreferent (resp. non-coreferent). Let C<sub>K</sub> (resp. N<sub>K</sub>) be the set of coreference
inte (resp. non-coreference) links according to K and C<sub>R</sub> (resp. N<sub>R</sub>) the one according to R. As
seen in table 2.1, the BLANC index is the arithmetic mean of the F<sub>1</sub>-scores for the detection of
coreference and non-coreference links.<sup>18</sup>

<footnote n="16"><sup>16</sup>Considered by its authors to be a feature of these metrics rather than a bug.
<sup>17</sup>Interestingly, Denis and Baldridge (2009) used CEAF<sub>m</sub>, which is arguably more mention-centric, but the subsequent uses — starting with the CoNLL-2011 shared task — used CEAF<sub>e</sub> instead.
<sup>18</sup>See Luo et al. (2014) and Recasens and Hovy (2011) for the edge cases where these definitions would imply</footnote>
<table><caption>Table 2.1: Definition of the BLANC metric</caption><thead><tr><th></th><th>Coreference</th><th>Non-coreference</th><th>Average</th></tr></thead><tbody><tr><th>Precision</th><td>P<sub>C</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding-bottom: 1px;">|C<sub>R</sub> ∩ C<sub>K</sub>|</div><div style="padding-top: 1px;">|C<sub>R</sub>|</div></div></td><td>P<sub>N</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding-bottom: 1px;">|N<sub>R</sub> ∩ N<sub>K</sub>|</div><div style="padding-top: 1px;">|N<sub>R</sub>|</div></div></td><td>BLANC<sub>P</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding-bottom: 1px;">P<sub>C</sub> + P<sub>N</sub></div><div style="padding-top: 1px;">2</div></div></td></tr><tr><th>Recall</th><td>R<sub>C</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding-bottom: 1px;">|C<sub>K</sub> ∩ C<sub>R</sub>|</div><div style="padding-top: 1px;">|C<sub>K</sub>|</div></div></td><td>R<sub>N</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding-bottom: 1px;">|N<sub>K</sub> ∩ N<sub>R</sub>|</div><div style="padding-top: 1px;">|N<sub>K</sub>|</div></div></td><td>BLANC<sub>R</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding-bottom: 1px;">R<sub>C</sub> + R<sub>N</sub></div><div style="padding-top: 1px;">2</div></div></td></tr><tr><th>F<sub>1</sub></th><td>F<sub>C</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding-bottom: 1px;">2 · P<sub>C</sub> · R<sub>C</sub></div><div style="padding-top: 1px;">P<sub>C</sub> + R<sub>C</sub></div></div></td><td>F<sub>N</sub> = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding-bottom: 1px;">2 · P<sub>N</sub> · R<sub>N</sub></div><div style="padding-top: 1px;">P<sub>N</sub> + R<sub>N</sub></div></div></td><td>BLANC = <div style="display: inline-block; vertical-align: middle; text-align: center;"><div style="border-bottom: 1px solid black; padding-bottom: 1px;">F<sub>C</sub> + F<sub>N</sub></div><div style="padding-top: 1px;">2</div></div></td></tr></tbody></table>

BLANC was defined in reaction to the issues of CEAF and B³ with singleton entities, which was becoming more glaring as more corpora with singleton mentions (in particular AnCora (Taulé et al. 2008) and SemEval 2010 (Recasens et al. 2010)) were released. However, it is not often reported in works on coreference resolution for English, due to the lack of singleton mentions in the most common benchmark for English—Ontonotes—and the well-documented hegemony of English in natural language processing literature (Bender 2019). This situation is also self-reproducing, since new works on coreference resolution have to compare to previous works (which was one of the reason to report MUC in the CoNLL evaluation campaigns despite its well-known flaws) there is little incentive to compute and report more recent metrics.

Finally, there is a certain degree of confusion as to the usage of BLANC for system mentions introduced by Luo et al. (2014): in that case, contrarily to the claims of Moosavi and Strube (2016), links where one end is a spurious system mention are *not* counted as valid non-coreference links. Therefore, as with CEAF, the addition of spurious system mentions actively degrades the final precision score.

### 2.4.3 Corpus-level evaluation

A lot has been written about the aforementioned metrics, comparing their advantages and disadvantages. Yet, there is a distinct lack in this body of work: all the definitions that we have given concern the evaluation of the output of a coreference resolution system on a single document. However, in actual evaluation campaigns, the evaluation usually concern multi-document corpora. To our knowledge, a standard way to obtain corpus-level evaluations from document-level evaluations has never been specified for any of these metrics.

Thus, the *de facto* standard is the one set by the reference CoNLL scorer (Pradhan et al. 2014). As of version 8.0.1,<fnref n="19" /> the scheme used to derive corpus-level evaluation is defined similarly to the usual method of computing the micro-averaged precision and recall in multi-class classification tasks.

<footnote>dividing by zero.</footnote>

<footnote n="19">¹⁹https://github.com/conll/reference-coreference-scorers/tree/v8.01</footnote>
Remember that in classification tasks, the *precision* for a given class *c* is defined as

$$P_c = \frac{|tp_c|}{|p_c|} \qquad [2.9]$$

where $tp_c$ is the set of true positives samples for *c* and $p_c$ is the set of positives samples.

This can be seen as the average correctness of class attributions (1 if the class was correctly attributed, else 0) over all positive samples for class *c*. The micro-averaged precision over the set of all classes $\mathcal{C}$ is the average correctness computed over all samples regardless of their class.

$$P = \frac{|\bigcup_{c \in \mathcal{C}} tp_c|}{|\bigcup_{c \in \mathcal{C}} p_c|} = \frac{\sum_{c \in \mathcal{C}} |tp_c|}{\sum_{c \in \mathcal{C}} |p_c|} \qquad [2.10]$$

or equivalently as the average of the per-class precisions weighted by the numbers of positives samples per class

$$P = \frac{\sum_{c \in \mathcal{C}} |p_c| \cdot P_c}{\sum_{c \in \mathcal{C}} |p_c|} \qquad [2.11]$$

Note that $c \in \mathcal{C}$ is in fact the total number of samples, so this is also equal to the micro-averaged recall or accuracy. This is readily interpretable as an empirical estimation of the probability that a given sample obtained by sampling uniformly in the dataset would be correctly classified.

Now, as we have seen in section 2.4.2, the existing metrics for coreference all$^{20}$ define a precision and a recall by averaging a measure of correctness over some units. These units are either links, mentions or entities, depending on the chosen model of coreference.

If we note $E_d$ the set of the units of interest in a system response for a given document *d* and *s* a measure of correctness, the precision for this response would be:

$$P_d = \frac{\sum_{e \in E_d} s(e)}{|E_d|} \qquad [2.12]$$

Proceeding by analogy with eq. (2.10), it then seems natural to compute the average precision for a corpus $\mathcal{D}$ as

$$P = \frac{\sum_{d \in \mathcal{D}} \sum_{e \in E_d} s(e)}{\sum_{d \in \mathcal{D}} |E_d|} \qquad [2.13]$$

which is what the reference scorer implementation does, and is again equivalent to computing the average of the per-document precisions weighted by the size of their respective set of units of interest

$$P = \frac{\sum_{d \in \mathcal{D}} |E_d| P_d}{\sum_{d \in \mathcal{D}} |E_d|} \qquad [2.14]$$

In practice, the units of interest for the previously defined metrics are

**MUc** links between discourse-old mentions and their closest antecedent

**B**$^3$ mentions

<footnote>$^{20}$For this analysis, following the reference implementation, we consider the coreference and non-coreference metrics in BLANC as separate metrics and compute the final $F_1$ score using their corpus-level values.</footnote>
CEAF<sub>m</sub> mentions

CEAF<sub>e</sub> entities

BLANC links (coreference and non-coreference)

There is an issue with this definition: even for classification tasks, using *micro-average* measures supposes that there is a reasonable balance across classes. Depending on the task, this can mean either that the classes' numbers of elements are close, or that the minority classes are not particularly relevant. If this assumption is not valid, a micro-averaged measure will be biased toward the majority class, thus blurring the value of this measure to estimate the actual effectiveness of a system. Similarly, for coreference, micro-averaged measures are only relevant if the *documents* are of similar sizes — where *size* is the number of units of interest. Otherwise, a micro-average is not a faithful account of system performances on smaller documents. Since coreference is a *document-level* phenomenon, artificially truncating or merging documents in order to achieve size similarity is also not feasible.

For the existing metrics, this issue has the most effect on BLANC: for a document with N mentions, the number of *coreference* links is roughly linearly dependent on N, but the number of *non-coreference* links scales *quadratically* with N. Therefore, in a heterogeneous corpus, the contribution of the smaller documents to BLANC is marginal when using micro-averages. Moreover, note that number of units of interest per document depends not only on the document size, but also of its genre, its topic...

However, taking the opposite option of using *macro-averages* i.e. unweighted averages of the per-document scores does not seem reasonable either. That choice would lead to an equal contribution of every document to the final score, which could lead the final evaluation to be overly optimistic, since coreference resolution is usually easier for shorter documents.

We propose to take a middle ground: averaging per-document scores, weighted by the number of mentions. While not necessarily ideal, this has the advantage of yielding the same results as micro-averages for B³ and CEAF<sub>m</sub> and comparable results for CEAF<sub>e</sub>.

We implement this approach in *scorch*<sup>21</sup> our scorer for *coreference chains*, along with a complete reimplementation of MUC, B<sup>3</sup>, CEAF and BLANC with a focus on straightforwardness, legibility and faithfulness to the most recent definitions of these metrics. Our implementation is continuously checked against the complete test suite distributed with the reference scorer, thus ensuring that the document-level<sup>22</sup> scores are exactly the same.

<footnote n="21"><sup>21</sup>https://github.com/LoicGrobol/scorch</footnote>

<footnote n="22"><sup>22</sup>The reference scorer does not provide a corpus-level evaluation test suite.</footnote>
# Chapter 3

## Annotated resources for coreference

In chapter 2, we have given a brief overview of identity reference phenomena in language and described coreference resolution as it is understood in natural language processing. Modern natural language processing is heavily reliant on annotated resources, which is not surprising: designing a system that simulates a human capacity is easier when examples of humans demonstrating said capacity are available. Coreference resolution is no different in that regard, and if early efforts such as those of Hobbs (1986) had to make do without such resources, all the significant system design efforts of the last two decades relied on annotated corpora.

In this chapter, we study how coreference is annotated in existing resources (section 3.1), with a specific focus on resources for French (section 3.2) and make a proposal for a new annotation model and an accompanying standard representation that can be used for coreference, designed to be able to take into account a large variety of source materials consistently (section 3.3).

### 3.1 Annotated resources for coreference

Leech (2005) defines corpus annotation as

> [T]he practice of adding interpretative linguistic information to a corpus (Leech 2005)

and later

> [A]nnotation is a means to make a corpus much more useful — an enrichment of the original raw corpus. From this perspective, probably a majority view, adding annotation to a corpus is giving ‘added value’, which can be used for research by the individual or team that carried out the annotation, but which can also be passed on to others who may find it useful for their own purposes.

(Leech 2005)

Generally speaking, an annotated resource is a mean to enable further research on the annotated phenomena, both from corpus linguistics and natural language processing perspectives. More precisely, from a natural language processing perspective, an annotated resource is a tool to develop natural language processing systems through rule design or machine learning, a tool to evaluate the performances of such a system and to study its behaviour in itself, and a tool to
compare the performances and behaviours of different systems.

As such, it would be hard nowadays to imagine developing a coreference resolution system
without having access to a corpus of a sufficient size and quality with coreference annota-
tions. Consequently, the design of such a corpus, its contents and the choices made during its
production strongly condition the systems developed with (or for) it.

In this section, we provide a rough overview of the characteristics of existing resources and
their disparities and a brief description of the Ontonotes/CoNLL-2011/CoNLL-2012 (Hovy et al.
2006) corpus, a resource whose influence on the development of coreference resolution systems
has been critical since its publication and its use in the eponymous shared tasks (Pradhan et al.
2012, 2011) and which explains some characteristics of the systems that we describe in chapter 4.

**3.1.1 Objectives**

Following Fort et al. (2011) and Habert (2000), we consider that

*Quelle que soit sa richesse, une annotation est cependant toujours orientée par une
tâche, même si cela est implicite.*

*(Whatever its richness, an annotation is always shaped by a task, even when this is
implicit.)*

*(Habert 2000)*

Accordingly, a corpus with coreference annotations is built toward one or several goals that shape all the steps of its constitution, from the choice of raw linguistic material and the theoretical framework underlying its annotations to the actual annotation process and its distribution and preservation — along with the material constraints within which these are done. As noted by Leech (2005), this does not preclude further reuse and applications beyond this original goal, but it does have an influence on them, with existing annotations becoming less useful the further away their uses stray from their original purpose.

The original motivations for existing annotated resources for coreference can be roughly sorted
in two categories: those where the study of referential phenomena is seen as an end in itself,
and those were it is a tool in broader studies in natural language processing, corpus linguistics<fnref n="1" />
or digital humanities in general. Borrowing from the terminology of Fort et al. (2011), we will
call these respectively *final* and *intermediary* applicative goals.

For example, among the best known coreference corpora for English, MUC (MUC Consortium 1995c, 1998) and ACE (Doddington et al. 2004) are typical of corpora with *intermediary* applicative goals. In both cases, their coreference annotations were part of larger efforts of semantic annotations meant as tools to design and evaluate natural language understanding systems This is clearly visible in their design choices: coreference was seen as complementary to named entity recognition, resulting in the lack of annotations of singleton entities in MUC (since those entities that were neither a named entity nor part of a larger coreference chain were considered of lesser importance to the message understanding task) and the restriction of the entities in

<footnote n="1">¹'La linguistique de corpus peut ainsi être objective, mais non objectiviste, puisque tout corpus dépend étroitement du point de vue qui a présidé à sa constitution' (Rastier 2002)</footnote>
ACE to named entities. The influence of the end goal is also visible in their choices of base
materials: written news and military reports for MUC, and news from diverse sources (written,
spoken, filmed) for ACE whose target was explicitly ‘extraction of information from audio and
image sources in addition to pure text’ (Doddington et al. 2004).

On the other hand, the ARRAU corpus (Poesio and Artstein 2008; Uryupina et al. 2019), explicitly designed to study anaphora, both from a linguistic perspective and to help designing anaphora resolution systems is built from raw materials in various genres, includes both written and spoken language and annotations for a large subset of reference and anaphoric phenomena.

Of course, this should not be taken to mean that these categories do not intersect, since improving
system design and evaluation should also be beneficial for downstream applications. For instance
the motivations given for GAP (Webster et al. 2018), a corpus of ambiguous pronominal reference
with enforced grammatical gender balance, were to provide both a benchmark for existing
systems on this particular kind of reference *and* a resource to design coreference resolution
systems that would be more useful for downstream applications e.g. machine translation and
information retrieval.

3.1.2 Scopes

As we saw in section 2.3, in of natural language processing, *coreference resolution* is more a
spectrum of closely related tasks rather than a strictly defined task. It is only natural, then, that
the same diversity exists among the existing annotated resources. These different conceptions
reflect in part the applicative goals for these resources (see section 3.1.1) but also the theoretical
choices of their authors and the material (including human and financial) circumstances of their
constructions. The breadth and depth of the annotations in a resource necessarily constrain
its possible uses. Corpus linguistic studies and coreference resolution systems can only use
the features that are provided and machine learning systems can only learn the phenomena
that are annotated. However, all annotations come with a cost, and one that is particularly
high for coreference, since it involves processing of complex phenomena at the level of whole
documents. More precisely, in the grid of analysis proposed by Fort et al. (2012), even the
subtask of resolving pronominal references to pre-detected mentions is already at maximal
complexity for the ‘context weight’, ‘ambiguity’ and ‘tagset dimension’ criteria,<fnref n="2" /> which puts it
among the most complex of the common annotations tasks.

The most notable of the disparities observable in the existing corpora for coreference include:

* The breadth (see section 2.3.2) of the annotated referential phenomena (possibly beyond identity). For instance ANCOR (Muzerelle et al. 2014) includes annotations for identity and bridging anaphora, but only for strictly nominal mentions, while Ontonotes (Hovy et al. 2006) includes verb mentions for event coreference but does not include annotations for bridging anaphora.
* The inclusion or exclusion of singleton mentions. While for Baldwin et al. (1998) ‘the

<footnote n="2">²This assumes that this subtask is formulated as a labelling task, which might not reflect the actual annotation process, but is a relatively straightforward way to obtain a lower bound for the complexity of annotating coreference.</footnote>
decision to annotate singletons is a bit of a philosophical issue' and MUC (MUC Con-
sortium **1995c**) and Ontonotes (see section 3.1.4) among others famously do not include
singleton annotations, Kummerfeld et al. (2011), Uryupina et al. (2019) and Uryupina
and Moschitti (2013) remark that this makes the development and evaluation of mention
detection systems more complex.

* The additional features annotated for mentions (which can include named entity type, definiteness, syntactic and morphosyntactic features...) These can be supporting annotations (see section 3.1.3) or only added for mentions, as is the case with ANCOR (see section 3.2.2)

* The model or models of coreference used in the annotations, which has an influence on the expressivity of the annotations for relations and entities and should be coherent with the planned downstream applications for intermediary applicative goals. For instance, for corpora designed with applications to EDT (see section 2.3.2) such as ACE (Doddington et al. 2004), using a mention-centric model is much more sensible than a link-centric model, since the definition of the task implies the existence of explicit (rather than latent) entities.

* A less studied choice is the categorization of certain anaphoric phenomena. For example, in ANCOR, references to the participants of an interaction are annotated as exophoric singletons rather than coreferent mentions. Another contentious case is that of appositives, annotated as identity reference in MUC and ACE — a choice famously criticized by van Deemter and Kibble (2000) on the grounds of formal interpretation—, annotated separately in Ontonotes and ignored in many other corpora such as ANCOR.

**3.1.3 Parallel and supporting annotations**

Beyond annotations directly related to coreference resolution, many resources annotated with coreference also include other kinds of annotations. Indeed, for annotators — both human and machines — these annotations can reduce the complexity of the task, sometimes significantly.

The annotations most commonly found alongside coreference are syntactic annotations, which make the mention detection subtask far easier if the mention candidates are restricted to syntactic constituents (or subtrees in dependency syntax analysis) since they reduce the task of findings mentions boundaries to an easier classification task. They can also help for annotations of minimal spans (see section 2.4.1), which can for instance be reduced to detecting a core constituent or a head word. Finally, until recently (see chapter 4), most automatic coreference resolution systems relied on morphosyntactic and syntactic features, both for mention detection and for coreference resolution, which makes gold annotation of these features desirable in the perspective of shared evaluation tasks, since it removes a bias in the comparisons.

For these reasons, many of the available corpora with coreference annotations also include
some form of syntactic parsing and in fact, most of these were actually annotated in syntax
before the coreference annotation process: Ontonotes (Hovy et al. 2006), AnCora (Taulé et al.
2008), TüBa-D/Z (Hinrichs et al. 2004) and the Prague Dependency Treebank (Nedoluzhko et al.
2016) are examples of coreference corpora that are full treebanks. For other corpora such as COREA (Hendrickx et al. 2013), the Polish Coreference Corpus (Ogrodniczuk et al. 2016) or EPEC-KORREF (Soraluze et al. 2012), the annotations are limited to morphosyntax or chunks.

In these cases, it is common to consider these annotations as lower layers, and to use them to support coreference annotations, rather than as parallel annotations. For instance, in AnCora, mentions are directly annotated by labelling constituents in the syntax trees. This ensures the consistency between the different annotations, but it also requires that the support annotations are either gold-standard before the coreference annotation procedure or an additional precorrection.

### 3.1.4 Ontonotes

The corpus known in part of the recent literature on coreference resolution systems as ‘CoNLL-2012’ or sometimes ‘CoNLL’ began its existence as Ontonotes (Hovy et al. 2006), a large (2.9 Mwords for the 5.0 release (Weischedel et al. 2013)), multilingual (English, Chinese and Arabic), multi-genres (it includes journalistic data, web texts and transcriptions of TV and radio broadcasts and telephonic conversations) and multi-annotation (Weischedel et al. (2013) mentions syntax, propositions, word senses, named entities and coreference).

The initial goals of Ontonotse was to be a resource with comprehensive semantic annotations for two types of applications:

* Machine learning systems for the natural language processing tasks corresponding to these annotations, by providing them with a large annotated dataset.

* Downstream natural language processing systems (the authors identified machine translation, question answering and summarization), to encourage a shift towards the use of rich inputs instead of ‘impoverished text models like bags of words or n-grams’ (Weischedel et al. 2013).

Ontonotes indeed saw some uses as a resource for other related tasks such as named entity recognition (Finkel and Manning (2009) is famous example) and narrow coreference resolution (e.g. for pronominal coreference as a tool for summarization by Gillick et al. (2009)), but its first systematic use for coreference resolution was for a subset of it (120 kwords from its English part) in the first shared task of the 2010 SemEval workshop (Recasens et al. 2010).

From then, it rose to omnipresence with its use as a benchmark in the coreference shared tasks of the CoNLL 2011³ (Pradhan et al. 2011) and 2012 (Pradhan et al. 2012). These campaigns were (and are arguably still are) the largest-scale shared tasks for coreference resolution with over 23 participants for the 2011 edition and 16 for the 2012 edition. The combination of the size and richness of the dataset and the availability of results for many systems made it hard for subsequent efforts in automatic coreference resolution (at least for English) to avoid working on the corresponding datasets and comparing their performances to those of the participants to the shared task.

<footnote n="3">³Where only the English part was used.</footnote>
As a result, the dataset thereafter often referred to as ‘CoNLL’ in coreference literature became a *de facto* standard benchmark for coreference and its idiosyncrasies became those of most of the coreference resolution systems developed for its three languages. This, combined with the well-documented tendency of part of the natural language processing community to focus exclusively on English (Bender 2019), also had the effect that most of the recent advances in coreference resolution algorithms were first developed with and for Ontonotes or Ontonotes-like data. Recent community efforts, such as the creation of workshop on Coreference Resolution Beyond Ontonotes (Ogrodniczuk and Ng 2016, 2017) attempt to remedy this state of affairs.

Generally speaking, the coreference annotations in Ontonotes can be seen as extensions of
those in MUC and ACE. They follow a mention-centric entity-mention model where coreferent
mentions are given a common @ID attribute. Their most notable specificities are the exclusion of
some phenomena (see Poesio et al. (2016a) for a more comprehensive and detailed examination):

* Pleonastic and generic pronouns are not considered to be mentions

* Coreference between generic mentions is not annotated

* Copular, predicative and appositive constructions are not annotated as identity reference (and are ignored in the CoNLL shared tasks, which is not unusual in coreference corpora in general)

In contrast, it does include event coreference by marking coreferent verb mentions.

Finally, as we mentioned in section 2.2 and section 3.1.2, the most controversial (and possibly influential) choice in Ontonotes is the non-annotation of singleton mentions. As a matter of fact, this choice is never actually made explicit in the release publication (Pradhan et al. 2007b) or the dataset documentation (Weischedel et al. 2013), which merely notes that ‘mentions of the same entity, concept or event are co-referenced as IDENT’, which is to be understood as implying that *non-coreferring* ones — i.e. singletons — are not annotated at all. A plausible motivation for this convention would be that since the choice was made to annotate verb mentions, annotating singletons would have involved either annotating most verbs or having different conventions for noun and verb mentions.

Beyond the effects on system design mentioned in chapter 4, this choice also had an effect on the very vocabulary used in the literature on coreference resolution. Indeed, the documentation of Ontonotes and of the CoNLL shared tasks use the term *mention* to refer to *coreferring* mentions (for instance in the title of one of the secondary tasks ‘Predicted plus gold mentions’). Following this terminology, some works on mention detection — for instance Kummerfeld et al. (2011) — include singleton detection (and removal), which implies document-level processing, since being a singleton is not an intrinsic property of a mention.

The portion of Ontonotes used in the SemEval corpus includes automatic singleton annotations,
described by the authors as

[A]ll NPs and possessive determiners were annotated as singletons excluding those functioning as appositives or as premodifiers but for NPs in the possessive case. In coordinated NPs, single constituents as well as the entire NPs were considered to
be mentions. There is no reliable heuristic to automatically detect English explet-
ive pronouns, thus they were (although inaccurately) also annotated as singletons.
(Recasens et al. 2011b)

which goes to show that there is no easy way to reconstitute this information short of manual
reannotation.

## 3.2 Resources for French

French is usually counted as a well-resourced language, both in terms of theoretical linguistics
works and in terms of language resources.<fnref n="4" /> However, for a long time, there was a distinct
lack of annotated resources for coreference in French, compared to those available in other
well-resourced languages (see for instance Poesio et al. (2016a) for a recent cross-language
summary of these). There is no easy explanation for this situation, though Salmon-Alt et al.
(2004) attribute it to the combination of the lack of interest until 2002 for generic resources on
anaphora and the lack of freely available resources with pre-annotations (particularly syntactic
annotations) to support coreference annotations after that — which is still true: the recent
large-scale annotation projects for coreference in French did start from unannotated data.

The complementary releases of ANCOR (Muzerelle et al. 2013a, 2014) for spoken French and DEMOCRAT (Landragin 2016) for written French put an end to this situation: there are now two large-scale corpora with coreference annotations for French, made publicly available. This unlocked the development of machine-learning systems for coreference resolution for French, of which our own work, presented in chapter 5 and chapter 6, on coreference resolution for *spoken* French is an example.

In this section, we give a brief overview of the resources for coreference in French made available
before 2013, then go on to describe ANCOR — our reference corpus — and DEMOCRAT.

### 3.2.1 Historical resources

The state of coreference resources prior to 2002 is described by Salmon-Alt (2002) as lacking in
both quality and quantity:

les ressources françaises annotées en relations anaphoriques sont insuffisantes, tant au niveau quantitatif qu'au niveau qualitatif (Salmon-Alt 2002)

We reproduce in table 3.1 their inventory of the existing resources at the time, excluding those
that only concern non-identity anaphora.

Of these resources, only Popescu-Belis (1999) was both<fnref n="5" /> available and general enough, and it was
far too small for data-driven natural language processing. Salmon-Alt (2002) aimed to address
this lack by providing a publicly available large-scale corpus with coreference annotations, but

<footnote n="4">⁴At the time of writing, it stood at the third place in ELRA's map of language resources (Calzolari et al. 2010).</footnote>

<footnote n="5">⁵ARCADE (Tutin et al. 2000) was later publicly released, but remained too specific.</footnote>
<table><caption>Table 3.1: French corpora with anaphora annotations prior 2002 (from Salmon-Alt 2002)</caption><thead><tr><td>Reference</td><td>Words (#)</td><td>Mentions</td><td>Open access</td></tr></thead><tbody><tr><td>Bruneseaux and Romary (1997)</td><td>30 k</td><td>Characters, places and items</td><td>yes</td></tr><tr><td>Popescu-Belis (1999)</td><td>10 k</td><td>NPs</td><td>yes</td></tr><tr><td>Clouzot et al. (2000)</td><td>95 k</td><td>3rd person pronouns</td><td>unclear</td></tr><tr><td>Tutin et al. (2000)</td><td>1 M</td><td>closed class anaphoric expressions</td><td>no</td></tr><tr><td>Trouilleux (2001)</td><td>45 k</td><td>3rd person pronouns</td><td>no</td></tr></tbody></table>

the project fell short of that objective.<fnref n="6" />

A later initiative was DéDé (Gardent and Manuélian 2005), a corpus of definite description with a fine-grained typology, but it is both too small (40 kwords) and too specific (since it does not include annotations of pronouns) for general coreference resolution.

### 3.2.2 ANCOR

ANCOR (Muzerelle et al. 2013a, 2014) is the first public large-scale corpus with unrestricted nominal coreference annotation for French. It started with a proof-of-concept study by Muzerelle et al. (2013b) that resulted in a smaller corpus named CO2.

The initial goals of CO2 were — in the short term — to provide a systematic study of anaphora in spoken French and to study the influence of homogeneity constraints such as identity of number or grammatical gender in spoken language. In the long term, the authors hoped that this resource could help designing and evaluating of named entity recognition and entity detection and tracking systems, improving the analysis of anaphora for human-computer communication and evaluate the performances for spoken language of anaphora resolution systems designed for written language.

The motivation for ANCOR was to go further in terms of corpus size: since the current state-of-the-art coreference resolution systems were (and still are) machine-learning systems, the development of such a system for French required more data than was available in CO2 or in the other corpora for French that were either non-public or too narrow (see section 3.2.1). It was also an opportunity to integrate more diverse language and to refine the annotation procedure, building on the experience of CO2. Generally speaking, ANCOR is a corpus with mostly *final* applicative goals, with some long-term intermediary applicative goals, which have had relatively little influence on its design.

The base material for ANCOR consists of samples from three corpora: ESLO (Enquête SocioLin-guistique à Orléans, Eshkol-Taravella et al. 2011), and OTG (Office du Tourisme de Grenoble) and Accueil UBS (Université Bretagne-Sud), both part of the Parole Publique corpus (Nicolas

<footnote n="6">⁶It then enjoyed a relatively short second life as part of the FReeBank project Salmon-Alt et al. (2004), now only reachable through the Internet Archive Wayback Machine.</footnote>
et al. 2002). ELSO is only partially included in ANCOR and the included portion consists of two subparts: ANCOR-CO2, used in the pilot study and ANCOR-ESLO. Table 3.2 gives the relative contributions of these parts.

<table><caption>Table 3.2: ANCOR subcorpora dimensions (from Muzerelle et al. 2014)</caption><thead><tr><th rowspan="2">Corpus</th><th colspan="2">Duration</th><th colspan="2">Words</th></tr><tr><th>h</th><th>%</th><th>#</th><th>%</th></tr></thead><tbody><tr><td>ESLO-ANCOR</td><td>25</td><td>81.97</td><td>417 k</td><td>85.45</td></tr><tr><td>ESLO-CO2</td><td>2.5</td><td>8.20</td><td>35 k</td><td>7.17</td></tr><tr><td>OTG</td><td>2</td><td>6.56</td><td>26 k</td><td>5.33</td></tr><tr><td>UBS</td><td>1</td><td>3.28</td><td>10 k</td><td>2.05</td></tr><tr><td>Total</td><td>30.5</td><td></td><td>488 k</td><td></td></tr></tbody></table>

All of these corpora are of transcribed *spontaneous* spoken French, with slightly different genres and transcription conventions between the ESLO and Parole Publique parts.

The ESLO parts are both from the ESLO1 part of ESLO (Lonergan et al. 1974), transcriptions of recordings from 1968 to 1971 in Orléans, France. These recordings were originally with the two goals of constituting an educational resource for French as a Foreign Language students and teachers and a sociolinguistic corpus, the ‘soundscape of a city’ (Baude and Dugua 2011). The parts kept for ANCOR are interviews of citizens of Orléans. Their form resembles that of sociological interviews (Weber and Beaud 2003), with prompts in the form of open questions from the interviewer and answers of variable length from the interviewee. The topics range from the interviewee’s uses of language to their day-to-day lives, with the intent of leading them to talk about their perceptions of their city and their language. Since the objective in this kind of interview is to elicit personal observations, sentiments and thoughts from the interviewee, the questions and possibly phatic feedbacks from the interviewers are usually kept as short as possible and the answers can be long monologues, even though some of them are simple yes/no answers. It also contains some extra-interview interactions, such as interview set-up and small talk between participants.

The Parole Publique parts are transcriptions of much more utilitarian speech. They record interactions in physical presence at Grenoble’s tourist office in 2001 for OTG and telephonic conversations from the reception of the Université Bretagne-Sud in 2013 for Accueil UBS. Both consist of short interactions of no more than a few minutes, where one of the participants (caller or visitor) seeks an answer to a question or asks for a document. Most of the interactions consist of clarifications and negotiations, sometimes with a few extra comments or some small talk. Utterances are much shorter than in ESLO, and the interactions are much more interactive and noisy.

ANCOR is a corpus for *anaphora*, including bridging anaphora, but strictly restricted to nominal and pronominal expressions. Generally speaking, the authors strove for *consistency* over completeness. For instance the adverb *demain* ('tomorrow'), though considered to be a mention is never annotated. This was motivated by the need for a strict syntactic characterization of
mentions, annotation experiments in the pilot campaign having shown that broader and fuzzier
definitions were causing the annotators troubles. Similarly, as we mentioned in section 3.1.2,
mentions of the participants of the interactions were annotated as singletons rather than
coreferent mentions, since the authors considered that their coreference was independent of
the context, i.e. non-anaphoric and therefore not to be annotated. This last choice is somewhat
surprising, given that while the same argument could be made for named entities, these are still
annotated as coreferent.

In addition to mentions, expletive pronouns are also annotated, to help system designers ‘car ceux-ci peuvent tromper les systèmes de résolution des anaphores’. The pronouns in the fixed idioms *s’il vous plaît* and *il y a*, however, are not.

The following annotations are also provided for mentions:

* Gender, number and part of speech

* Definiteness (indefinite, definite, demonstrative or expletive form)

* Inclusion or not in a prepositional phrase

* Named entity type (for named entities)

* “NEW” for the first mention of a coreference chain

The representation of reference follows a link-centric model as is common<sup>7</sup> for anaphora corpora,
since it gives a natural common representation for all anaphoric phenomena. Specifically, the
chosen antecedent for a mention is the first *nominal* mention of its chain (thus excluding
pronominal antecedents, even in cases of pronominal cataphora — where a pronoun is the first
mention —).

The anaphora relations are also typed, with the following hierarchy:

**Identity** anaphora are split between

**Nominal** anaphora, when the mention is a noun phrase. It falls between two subtypes

**Direct** annotated *DIRECTE* when the mention and its antecedent have the same (nominal) head

**Indirect** annotated *INDIRECTE* otherwise

**Pronominal** anaphora, annotated *ANAPHORE* when the mention is a pronoun.

**Bridging** anaphora are split between

**Nominal** bridging anaphora annotated *ASSOC*

**Pronominal** bridging anaphora, annotated *ASSOC_PR*

Apart from transcription information (which are not annotations *per se* but rather parts of the rich source material), these corpora had not been the object of any kind of annotation. Most notably, they included no canonical word segmentation and no syntactic analysis. In contrast

<footnote n="7"><sup>7</sup>But not the only solution: a mention-centric representation would also be suitable for instance.</footnote>
with comparable resources such as Tüba-D/Z (Telljohann et al. 2006), the PCC (Ogrodniczuk et al. 2016) or Ontonotes (Pradhan et al. 2007b), this meant that the identification of mentions with their exact character-level boundaries had to be done from scratch by the annotators. To mitigate this difficulty, the annotation procedure was done in two distinct phases: one for mention identification and one for mention linking, with adjudication rounds by an expert annotator between and after them.

The nature of the source material (speech transcriptions) and the initial goals (studying anaphora in speech), a format that allowed for accurate representations was a necessity for ANCOR. Utterance boundaries, overlaps, speakers, disfluencies, non-verbal events... are all crucial parts of spoken language and destroying this information would have seriously impaired the potential subsequent studies. It was also crucial to preserve the synchronization with the original audio signal: the original corpora are nor word- or phoneme-aligned but there are sufficiently many timestamps to do it automatically in the current state of technology, which would allow studies on the relations between anaphora and prosody.

For convenience reasons, the choice of the original work was then to keep the source material in its original Transcriber XML-TRS format (Barras et al. 1998). However, this format is not meant for extensive annotations besides those necessary for transcription and was not natively supported by existing annotation software. As a result, the annotators simply worked on XML-TRS serializations in Glozz as if they were raw texts.

While this removed the need for file conversions or the adaptation of an annotation software to Transcriber, this approach has two critical issues. One is practical: since the annotation in Glozz are stand-off with character offsets, and since XML serialization offers no guarantee of character-level stability (i.e. a single XML document can have different characters stream representations), in order to annotate XML documents with Glozz, one has to link the annotations not only to a specific version of these documents, but to a specific serialization, which can be problematic. In addition to this, even if character-level stability could be guaranteed, it would still forbid any subsequent modification of the source material (corrections or metadata change) or at least making annotation transposition to these new version very complex.

The other issue is more fundamental: overlaying a new structure on an already structured document without linking the two structure makes it hard to enforce structural constraints, such as the one imposed by the ANCOR annotation guidelines that a mention cannot cross utterances boundaries.

A step in this direction was taken by Désoyer et al. (2015b), who converted the original stand-off annotations of Glozz to custom inline annotations. The result was a format that is not usable in either Glozz or Transcriber anymore but it at least ensured the consistency between the two structure by forcing them into a common XML tree and removed the brittle dependency on character positions. However, this format was completely ad-hoc, mostly undocumented and hard to use in practice, which prompted our own standardization effort (see section 3.3).
### 3.2.3 DEMOCRAT

Started in 2016, the Description et modélisation des chaînes de références (DEMOCRAT, Landragin et al. 2018) project aims at building a coreference corpus for written French. The first version of this corpus<fnref n="8" /> has been released in June 2019 and as of 2020 the project is reaching its end.

The publication of ANCOR in 2013 endowed French with its first corpus with annotations for unrestricted nominal coreference, but the specific nature of its source material (transcriptions of spontaneous spoken language), as interesting as it is, made it an outlier compared to existing corpora for other languages. In particular, it made comparisons of the performances of coreference resolution systems quite complex, since most of these systems and particularly state-of-the art systems were designed for and using the English part of the CoNLL-2012 corpus (of which only a small part consists of spontaneous speech). Beyond the differences in annotation conventions, that could have been lessened, the differences between reference mechanisms in spoken and written languages (noted by Prince (1981) for instance) make the simultaneous change of these two parameters (language and genre) problematic. In that context, one of the motivation of DEMOCRAT is to fill this gap, by providing a corpus that would be complementary to ANCOR.

DEMOCRAT’s source material is a compilation of libre texts in written French, from fiction and news texts in modern French and a smaller collection of historical texts.

Unlike ANCOR, the annotations in DEMOCRAT are strictly limited to coreference and do not include other anaphoric phenomena. It also lacks the richer features provided in ANCOR for mentions such as mentions types, named entities or definiteness. However, it does include an automatic tokenization, and part-of-speech and lemma for its tokens, provided by Treetagger (Schmid 1994).

The first release uses the XML-TEI TXM format (Heiden et al. 2010) with stand-off XML-TEI-URS (see section 3.3) annotations. The published material uses annotations for both entity-mention models: every mention is annotated with an identifier for the corresponding entity, and chains are represented by URS schemes pointing to all of their mentions.

## 3.3 Representing coreference annotation

The earliest attempt at devising a standard for coreference annotation comes from Poesio et al. (1999) in the context of the MATE project, as a proposal for a scheme to be used in the MATE workbench (Isard et al. 2000) and later refined in Poesio (2004) for the GNOME corpus. It durably influenced subsequent coreference corpora, most notably ARRAU (Poesio and Artstein 2008; Uryupina et al. 2019), LiveMemories (Rodríguez et al. 2010–0021) and EPEC-KORREF (Soraluze et al. 2012). It also influenced part of the annotations for the AnCora (Taulé et al. 2008), Tüba-D/Z (Telljohann et al. 2006) or the Prague Dependency Treebank (Nedoluzhko et al. 2016).

However, it only provides facilities to annotate coreference links — which is only natural since it was designed for general anaphora annotation rather than coreference. This is an issue

<footnote n="8">⁸https://www.ortolang.fr/market/corpora/democrat</footnote>
when using an entity-mention coreference model and led to some fragmentation from projects that chose another model, and Ontonotes (Hovy et al. 2006) or the Polish Coreference Corpus (Ogrodniczuk et al. 2016) for instance do not use it.

Regarding resources for French, as we have seen in section 3.2, the two main corpora — ANCOR and DEMOCRAT — use different models of coreference, with ANCOR using a link-centric model (entity-level annotations are planned but have not been yet been produced) and DEMOCRAT using an entity-centric model. In order to build an homogenous resource for French combining these two corpora and enable a consistent access to their annotations without constraining one of them to the other’s model (and thus losing information), we had to devise a new representation for coreference annotations. Moreover, the specific constraints of the rich source material of ANCOR (that encode speech transcriptions) and our plans to enrich these corpora with further annotation encouraged us to devise a format that would allow for as much interoperability as possible with encodings beyond coreference.

In the remaining of this section, we describe our proposal for a model and a representation format for coreference annotations. This proposal is implemented in the XML-TEI URS used for the first release of DEMOCRAT and a new release of ANCOR, first described in Grobol et al. (2017a), Grobol et al. (2018b) and Grobol et al. (2018a).

### 3.3.1 The URS metamodel for anaphora and coreference

URS (Units, Relations, Schemes) is an annotation *metamodel* introduced by Widlöcher (2008) in the context of discourse analysis, but designed to be general enough for a large class of linguistic annotations. It is based on the three eponymous classes of annotations

**Units** are parts of the source material. In the first implementation of URS in Glozz (Widlöcher and Mathet 2012), they were restricted to *contiguous* segments, but in principle, they can be any relevant part.

**Relations** are binary *links* between two annotations, modelling binary relations, either directed or undirected.

**Schemes** are *n*-ary relations between annotations, either homogenous or structured.

Note that ‘annotation’ in the definitions above refer to all three kinds of annotations and that e.g. a relation can link two other relations, a unit and a scheme or any other combination. All annotations can also bear more information in the form of attached feature structures, which are themselves non-relational, i.e. a feature can not refer to another annotation.

The first use of URS for coreference annotations is almost fortuitous: as we mentioned in section 3.2.2, the members of the CO2 and ANCOR (Muzerelle et al. 2014, 2013b) annotation projects chose to use Glozz mostly for its ergonomics and its integrated inter-annotator agreement computation module. In ANCOR, the annotations are anaphoric relations, and they are naturally represented as directed URS relations. Later on, for the MC4 project (Mélanie-Becquet and Landragin 2014) that served as pilot study for the DEMOCRAT annotation campaign, the designers of the Analec annotation platform (Landragin et al. 2012) chose to use URS as their
base metamodel to ensure the compatibility with Glozz. As a result, their annotations of coreference are also formulated in URS terms: in that case, the annotations are coreference chains as chains of mentions, represented as URS schemes.

Consequently, when we started to work on a representation format for DEMOCRAT, with the intention of being as compatible as possible with ANCOR and the MC4 pilot corpus, it appeared natural to merge these representations. We propose the following representation for anaphora and coreference:

**Mentions** are represented URS *units*: they are the portions of the source material that are the observable parts of anaphoric and referential phenomena. In most corpora they are contiguous spans for convenience but that is not a necessary constraint: for instance in the case of overlapping speech or with disfluencies, it would be perfectly reasonable to consider non-contiguous mentions.

**Anaphoric relations** including, but not limited to *identity* reference are URS *relations*. In most settings, it is natural to use *directed relations*, either between a mention and an antecedent — for link-centric models — or between a mention and an entity.

**Entities** represented as URS *schemes*, by identifying them with their coreference chains.

This representation has the major advantage of being able to reify — and consequently to * annotate* — the three base objects for coreference (mentions, links and entities). This comes at the price of either redundancy or irregularity: since there are several ways to encode identity reference (as relations or as schemes), corpus designers have to either encode it several times (with the need to enforce consistency) or to chose one of the possible representations (harming cross-corpora compatibility). Our recommendation is to encode it both ways: annotate links between mentions and their canonical antecedents and coreference chains. This allows a uniform treatment of all anaphoric phenomena and annotation of entity-level properties (such as properties of an entity) and link-level properties (e.g. the mechanism of reference, see section 2.1.2). The consistency between the annotations should be automatically enforced by annotation softwares, which are already necessary for coreference annotation, and thus should not be a burden for the annotators themselves.

Note that this model does not impose many hard constraints on actual realizations: different projects and different resources can still use annotations conventions that suit their specific needs. For instance, it does not impose any rule for the choice of a canonical antecedent, or a specific typology of anaphoric links. This also allows for more complex annotations, for instance near-identity relations as described by (Recasens et al. 2011a) are easily described as URS relations, either between a mention and an entity or between two entities.

### 3.3.2 A TEI encoding and serialization for URS

In section 3.3.1, we described a model for coreference annotations as an implementation of the URS metamodel. However, while this solves the issues of the annotation *semantic*, it does not provide a standard annotation *representation* or actual * serialization*. Indeed, URS itself is not necessarily bound to a representation format and its first implementations in Glozz (Widlöcher
and Mathet 2012) and Analec (Landragin et al. 2012) use very different representations.

Theoretically, it might not be a problem for interoperability: as long as conversion tools are able to transform third-party serializations into abstract representations, third-party softwares could each use their preferred native representation and still be able to use this common model. This is still a problem in practice: no matter the goodwill of the community, it is not reasonable to expect extensive support for every arbitrary native representation from third-party software designers. This issue is by no mean new — the need for interoperability of data is older than computer science itself — and the solution is well known: standardization.

For documents, in the context of digital humanities, the Text Encoding Initiative (TEI, TEI consortium 2020<sup>9</sup>) is one of the longest-standing and most successful efforts of standardization. Its guidelines for document encoding encompass the needs of most topics in digital humanities (which for the purpose of this discussion includes computational linguistics) in terms of expressivity and versatility. Of course, this sometimes comes at the expense of brevity and straightforwardness, but that can hardly be completely avoided — and in any case, it is not our main concern here.

Accordingly, in our effort to develop an *interoperable* and *versatile* format for coreference annotation, using a TEI representation seemed to be the best choice, with two main motivations. Firstly, a TEI-backed format benefits from the work done during the 30 years of development of the standard, including battle-tested representations and support from a large panel of software solutions. Even without explicit support for the precise semantic of the URS metamodel, features such as the validation of properties or basic representation facilities thus come at no cost. Secondly, the TEI already support a variety of rich input formats: for instance transcription of speech via the spoken module, which is of crucial interest for us given the nature of ANCOR.

We are not the first to make that choice either: as we mentioned earlier, TEI-compatible encodings of reference were already a proposal of (Bruneseaux and Romary 1997) and current reference corpora such as the Polish Coreference Corpus (Ogrodniczuk et al. 2016) use TEI encodings. However, these uses are all strictly designed for referential and anaphoric annotations, while we propose to represent the whole URS metamodel, which has the potential to represent many other types of annotations (see section 3.3.3 for a concrete example).

We propose to represent

**Units** as `<tei:span>`<sup>10</sup> elements, whose purpose is to 'associate' an interpretative annotation directly with a span of text' (TEI consortium 2020), marking its nature by setting its `tei:type` attribute to "unit".

**Relations** as `<tei:link>` elements, which 'define' an association or hypertextual link among elements' (TEI consortium 2020), with a `tei:type` attribute set to "relation". The linked annotations are given by the `tei:target` attribute as a space-separated couple of

<footnote n="9"><sup>9</sup>Due to the grassroot and evolutive nature of the TEI, chosing a single authority to cite is not a trivial task. Readers not familiar with the TEI might want to start with Burnard (2014) rather than diving head first in the technicalities of the guidelines.</footnote>

<footnote n="10"><sup>10</sup>In all of this section, we refer to the TEI concepts as of the 4.0 version of the P5 guidelines (TEI consortium 2020)</footnote>
`teidata pointer`. If a directed relation is used, the convention should be that the first target is the source of the link and the second one is the destination.

**Schemes** as `<tei:link>` elements with a `tei:type` attribute set to "schema".

Associating features with these elements can be done straightforwardly, through the integration of the SemAF standard (ISO 2017), implemented in the TEI in the form of feature `<tei:f>` and feature structures `<tei:fs>`. This representation, with its serialization in XML forms the XML-TEI URS format.

Additionally, we also recommend a stand-off integration of these elements to TEI documents, preferably through the use of the `<tei:standOff>` element proposed by Romary (2017), and officially made part of the TEI guidelines in version 4.0.0. This is motivated by the common need in real-world corpora for parallel annotations (see section 3.1.3), which are much easier to manage in separate files (or at least separate locations) rather than as inline elements interleaved in the source material. It is also far more convenient for dealing with concurrent annotations of the same phenomena — either by different annotators or as different versions of the same annotation — and complex annotation structure, such as partially overlapping units — for which the strictly hierarchical representation of annotations as inline XML is not sufficient.

Finally, among the three types of pointing mechanisms for stand-off annotations identified by Bański et al. (2016), we recommend to use references to a tokenization of the source materials (by representing tokens as `<tei:w>` elements for instance) if at all possible rather than character offset-based mechanisms (that make the manipulation of the XML serializations tedious and error-prone) or `<tei:anchor>` elements that require ad-hoc manipulation of the source material — negating most of the advantages of stand-off annotations. This is of course not an ideal choice, as it requires the existence of a pre-existing tokenization, but this is an assumption that is often already made by annotation softwares and it should not be too much of a burden in many cases, since current automatic tools can provide a good enough tokenization in most cases.

### 3.3.3 Application to ANCOR and DEMOCRAT

Our initial motivation by designing the XML-TEI URS format was to provide a single format for the two main coreference corpora for French: ANCOR and DEMOCRAT. As we mentioned in section 3.2.2, the annotation formats for previous versions of ANCOR were ad-hoc combinations of specialized formats for speech transcription and coreference annotations that were not easily reconcilable. Our solution was to develop a TEI-backed annotation format (section 3.3.2) that could be used to add coreference annotations to TEI representations of speech transcription. From there, using this annotation format for DEMOCRAT would be straightforward, since the source material for DEMOCRAT, written texts, were easy to represent in TEI.

#### Porting ANCOR to XML-TEI URS

The first step was to convert the inline annotated version of ANCOR (Désoyer et al. 2015b) and to port its speech transcription encoding to a TEI one. This already had several advantages, since the TEI offered richer representations than those available in the native Transcriber (Barras
et al. 1998) format originally used for ANCOR. For instance, utterances overlap are dealt with in
Transcriber by creating special overlap utterances, leading to a situation where a single phatic
'hm hm' - which occurs very frequently in the ANCOR - can effectively split a long monologue
in two as in example 24, where it even splits a mention ('Université d'Orléans-Tours') in two
parts.

(24) — [...] *c'est une place cette place que j'ai maintenant qui est ingénieur gestionnaire des campus de l'université d'Orléans-*

— hm hm

— *Tours parce qu'ils cherchaient quelqu'un de jeune dynamique [...]*

(It is a position that I have now which is engineer intendant of the Université d'Orléans-Tours since they were looking for someone young dynamic)

(ANCOR, Muzerelle et al. 2014)

The original annotation in ANCOR used a complex encoding (linking the two parts of the mention in a URS scheme) that introduced heterogeneity of representation for the mentions. Encoded in TEI, this is more easily represented as one single utterance for each speaker (with additional marks to indicate that an overlap occurs at some point), making the split mention contiguous again.

Then, in order to make the support of stand-off annotations easier, we went on to tokenize
the transcriptions. This was done automatically using UDPipe (Straka and Straková 2017) and
a small set of post-correction rules to ensure that this tokenization was compatible with the
existing mentions (the guidelines of ANCOR rule out sub-word mention boundaries and we
stayed true to them). Listing 1 gives a simplified example of the result of this conversion process.

From there, converting the inline annotations to stand-off was a relatively straightforward
matter of translating <anchor> elements to identifiers of the newly created <tei:w> elements.
Listing 2 is a simplified example of an annotation block representing two coreferent mentions
and their coreference chain (which also includes a third mention).

Syntactic analysis for ANCOR

*A more detailed account of the work described in this part can be found in Grobol et al. (2018b).*

As we mentioned in section 3.1.3, many coreference corpora include parallel syntactic annota-
tions, which helps both natural language processing systems and corpus linguistics research.
Accordingly, we wanted to explore these possibilities for ANCOR and integrate syntactic ana-
lysis in our own system (see section 6.4.2). Since human syntactic annotations were out of
reach considering our resources, we chose to try to integrate automatic syntactic parsing as a
substitute. The actual parsing was done using Dyalog-SRNN (Villemonte De La Clergerie et al.
2017) trained on the French part Universal Dependencies v2 (Nivre et al. 2016). Accordingly, the
syntactic annotations that we obtained were in dependency syntax, which is somewhat unusual
for a coreference corpus, but has the advantage of being supported by a large multilingual
corpus (Universal Dependencies) and being better suited than constituency syntax for syntactic
<figure><img src="image_3.png" /><figcaption>Listing 1: Speech transcription annotations in ANCOR-AS</figcaption></figure>

annotation of *spoken* French (Lacheret et al. 2014). Figure 3.1 gives an example of the syntactic
analysis of part of an utterance in ANCOR.

Dependency syntax translates straightforwardly in URS terms:

**Syntactic words** can be represented as URS units, which also allows grouping several orthographic tokens in a single syntactic word

**Dependencies** can be represented as URS directed relations

The only modification to our previous formulation of XML-TEI URS was the addition of `<tei:expan>` elements to represent the expansion of contractions such as ‘du’ that are single orthographic tokens into several syntactic words (here ‘de le’). Listing 3 gives an example of the encoding of the dependency tree from fig. 3.1 in XML-TEI URS.

URS annotations for DEMOCRAT

Having proof-tested XML-TEI URS annotations in ANCOR, we went on to propose it for the
release version of DEMOCRAT. This was done jointly with the integration of a URS annotation
plugin in the TXM platform (Heiden et al. 2010). This integration was relatively easy, since
the internal format of TXM was already built on TEI and it helped to develop its support for
Coreference resolution for spoken French

<standOff>
<annotation type="coreference">
<spanGrp type="unit" subtype="mention">
<span from="#u7-w76" to="#u7-w77" xml:id="m31"/>
<span from="#u7-w84" to="#u7-w85" xml:id="m32"/>
</spanGrp>
<linkGrp type="relation" subtype="coreference">
<link target="#m31 #m32" xml:id="r20"/>
</linkGrp>
<linkGrp type="schema" subtype="chain">
<link target="#m31 #m32 #m40" xml:id="c12"/>
</linkGrp>
</annotation>
</standOff>

<figure><img src="image_4.png" /><figcaption>Figure 3.1: Syntactic analysis (subtree) for “au moment où je me suis marié en juillet soixante-sept”</figcaption></figure>

parallel and versioned annotations. Conversely, the decoupling between annotations and source materials also allows a limited support for source versioning, since as long as the pointers supporting the annotations stay valid, it is not necessary to change the annotation.

Finally, by providing a single abstraction for a large class of annotations, this format also allows a uniform access to annotation in the form of the URS Query Language (URSQL) available in TXM since version 0.9 (Heiden 2019), which builds upon the concept of the GlozzQL (Mathet and Widlöcher 2011) language used in Glozz and allows complex combinations with the other specific querying languages available in TXM such as CQL (Evert and Hardie 2011), providing powerful interface for corpus linguistics inquiries (Quignard et al. 2018).
```xml
<standOff>
<annotation type="syntax">
<div type="tree" xml:id="tree10">
<div type="multiword-token">
<expan xml:id="tree10-w6-7" n="6-7" corresp="#u7-w76">
<w xml:id="u7-w76.1">à</w>
<w xml:id="u7-w76.2">le</w>
</expan>
</div>
<spanGrp type="unit" subtype="word">
[...]
<span target="#u7-w76.1" n="6" xml:id="tree10-w6" ana="#tree10-w6-fs"/>
<span target="#u7-w76.2" n="7" xml:id="tree10-w7" ana="#tree10-w7-fs"/>
<span target="#u7-w77" n="8" xml:id="tree10-w8" ana="#tree10-w8-fs"/>
[...]
</spanGrp>
<linkGrp type="relation" subtype="dependency">
[...]
<link target="#tree10-w8 #tree-10-w6" xml:id="tree10-d6" ana="#tree10-d6-fs"/>
<link target="#tree10-w8 #tree-10-w7" xml:id="tree10-d7" ana="#tree10-d7-fs"/>
<link target="#tree10-w3 #tree-10-w8" xml:id="tree10-d8" ana="#tree10-d8-fs"/>
[...]
</linkGrp>
<div type="dependency-fs">
[...]
<fs xml:id="tree10-d7-fs">
<f name="type"><symbol value="det"></symbol></f>
</fs>
<fs xml:id="tree10-d8-fs">
<f name="type"><symbol value="obl"></symbol></f>
</fs>
[...]
</div>
<div type="word-fs">
[...]
<fs xml:id="tree10-w6-fs">
<f name="upostag"><symbol value="DET"></symbol></f>
<f name="Definite"><symbol value="Def"></symbol></f>
[...]
</fs>
[...]
<div>
</div>
</annotation>
</standOff>
```

<caption>Listing 3: Syntactic annotations in ANCOR-AS corresponding to fig. 3.1</caption>
# Chapter 4

## Coreference resolution systems

In this chapter, we review the existing techniques, architectures and algorithms used in existing coreference resolution systems.

In section 4.1, we describe the input representations strategies for the three types of objects at the core of coreference resolution: mentions, mention pairs and mention sets.

Section 4.2 gives an overview of existing coreference resolution algorithms, sorted by their model of coreference (mention-pair and entity-mention) and their document processing strategies (discourse-ordered and global).

We will treat mention detection separately in section 4.3, since depending on the understanding of ‘mention’ in tasks definitions (see section 2.2) and resources (see section 3.1.2) it can be either a mostly syntactic task or so tightly tied to coreference resolution that it is a mere by-product of the coreference resolution algorithms used by end-to-end systems.

Finally, in section 4.4 gives a brief summary of the few existing works on coreference resolution for French.

### 4.1 Representations

A fundamental problem in natural language processing representing objects and concepts that are not always easy to define formally in logical or numerical forms that a machine is able to process. Even assuming that we are only concerned with unstructured text, it is possible to simply represent the inputs to natural language processing systems as streams of characters, but that would make their design overly complex for all but the simplest tasks. Consequently, the representations used in natural language processing are usually based on higher levels representations informed by linguistic theories — words, sentences... —, themselves usually represented as sets of *features*: symbolic or numerical properties that designed to be both sufficiently expressive and relevant for a given task.

Historically, features were defined directly by experts alongside the rules of the systems that used them. With the increasing availability of processing power and linguistic data (both raw
and annotated), the design of both<fnref n="1" /> features and rules tends to be delegated to machine learning algorithms, and the role of system designers has moved towards the design of machine learning architectures. This is not to say that expert knowledge does not play a role in this design process: on the contrary, it is critical to the design of architectures that are suitable interfaces between the linguistic data to be processed and its use in the subsequent parts of these systems. It is also currently irreplaceable in the quasi-craftsmanship process of training these systems.

For coreference resolution, the unavoidable objects are *mentions*, but most algorithms also use higher level representations of either mentions *pairs* or mentions *sets*, including their final targets: coreference chains — usually conflated with entities as is common in modelling coreference resolution for natural language processing. Some algorithms also call for representations of composite objects such as cluster-mention or cluster-cluster pairs. The representation strategies for these objects have mostly followed the same trends as in other areas of natural language processing, though perhaps with less inclination for generic end-to-end architectures than in e.g. machine translation or named entity recognition.

In this section, we give an overview of the representations used for mentions (section 4.1.1), mention pairs (section 4.1.2) and mention clusters (section 4.1.3) used in coreference resolution systems.

### 4.1.1 Mentions

Given the formulation of coreference resolution as an operation on the set of mentions in a document (see section 2.2), a coreference resolution system needs as a bare minimum a way to represent mentions. In most existing systems, the representation of a mention is *local* — it only depends on its immediate context — and *autonomous* — it does not depend on the existence and characteristics of other mentions. However, some recent works such as K. Lee et al. (2018) and Kantor and Globerson (2019) include additional information (on preceding mentions and preceding clusters respectively) that depends on other mentions.

We make the assumption that a coreference resolution system has at least access to mention boundaries, but not necessarily *gold* mention boundaries. In practice, mention detection modules are usually not perfect and their outputs will contain some errors — both false positives and false negatives — and in an end-to-end system, they all have to be represented. Thus, for the remaining of this section, we will call ‘mentions’ all the elements that serve as inputs to a coreference resolution algorithm such as those described in section 4.2. Furthermore, some recent works on end-to-end systems, starting with K. Lee et al. (2017) do not even use a dedicated mention detection module and for them, ‘mention’ will encompass most of the spans in a document, though they usually use some filtering heuristics to reduce the combinatorial explosion.

#### External features

In most cases, when mentions are provided by a mention detection system, this system itself builds upon the outputs of a processing pipeline. In the prototypical case of BART (Broscheit

<footnote n="1">¹Though not necessarily simultaneously: historically, this mostly concerned rules. Fully ‘end-to-end’ architectures that *do* only use characters stream-like inputs are a much more recent innovation.</footnote>
et al. 2010; Versley et al. 2008), for instance, the main<fnref n="2" /> mention detection pipeline includes a syntactic parser (from which POS tags can also be retrieved) and a named entities detector. In that case, this pipeline usually does not simply supply the mention boundaries, but returns mentions as objects with properties: intrinsic — such as a constituent type or a flexion— or structural — such as a position in a syntactic analysis or a semantic role.

We call these features *external features*, since they come from external resources that are usually not designed only for coreference resolution. We also include in these some basic features such as absolute positions in a document and character strings, that are used in similar ways but do not usually require extraneous processing as they are readily available in the raw source material.

In coreference resolution literature, most features are usually presented in terms of *agreements* and *distances* between their values for the mentions in a pair (see section 4.1.2), but defining them at the level of a single mention makes it clearer that they can also be used in other configurations, such as discourse-new detection or cluster-level representation (see section 4.1.3). Note however, that not all pair features can be meaningfully represented in this way, for instance the ‘Appositive Feature’ of Soon et al. (2001) models an actual linguistic relation and it makes more sense to consider it as a strictly pair-level feature.

Historically, the first widely used³ set of features comes from Soon et al. (2001), which famously defined a set of 12 pair-level features that translate to the following mention-level features categories:

1. Position of the mention in the document, implemented as a sentence index.

2. Internal morphosyntax: constituent type (pronoun or noun phrase), presence of a definite article or a demonstrative pronoun, number and grammatical gender.

3. Semantic class among ‘person’, ‘organization’, ‘location’, ‘date’, ‘time’ ‘money’ ‘percent’ and ‘object’. ‘Persons’ are also categorized in ‘male’ and ‘female’. This is deduced from the WordNet (Miller 1995) entry for the head word of the mention.

4. The content of the mention itself as a string of characters. Articles and demonstrative pronouns are ignored.

This feature set had the advantages of being robust, relatively easy to compute (considering the then-current state of the art for preprocessing modules) and sufficiently small to allow extensive experimentations with machine learning architectures.

From then to 2006, subsequent works used more or less similar sets of features, with the notable exception of Ng and Cardie (2002) who extended it by introducing more semantic information extracted from WordNet as well as more ad-hoc heuristics such as a boolean feature for ‘being part of a quoted string’ (and more pair features, see section 4.1.2).

<footnote n="2">²An alternative pipeline replaces the parser with the combination of a chunker and a POS-tagger.

³Earlier rule-based systems and domain-specific machine-learning systems such as RESOLVE (McCarthy and Lehnert 1995) or Aone and William (1995) use similar mention representations, but in a less systematic way, are harder to generalize to other domains and were less directly influential in subsequent works, though it is clear that they did have an influence on Soon et al. (2001).</footnote>
However, the addition of more features made the ensemble less easy to train, given the relatively
small size of the learning material (the MUC corpus, MUC Consortium 1995c, 1998), with the
need for non-negligible manual adjustments in order to actually improve over the Soon et al.
(2001) baseline. This last issue was made clearer by the extensive studies of Uryupina (2006,
2007) on much larger sets of features: if linguistically-motivated handcrafted features coming
from rich preprocessing pipelines do help, their efficiency seemed to be limited by the size (and
sometimes consistency) of MUC. This assessment is also consistent with the findings of Culotta
et al. (2007) and Bengtson and Roth (2008) whose systems obtained much better results with
features sets quite similar to that of Ng and Cardie (2002) and without the extensive manual
tuning needed for the latter, but using the much bigger ACE corpus (Doddington et al. 2004).

During this period, several works also showed an increased use of features associated with the
*head word* of a mention. While Soon et al. (2001) and Ng and Cardie (2002) only used it as a
way to probe WordNet, others such as Luo et al. (2004) and Uryupina (2003) started to use the
POS of the head as part of mentions representation and Poesio et al. (2005) and Uryupina (2004)
identified the crucial importance of head match as a pair feature for mentions-pair systems
— reasserting a tendency known since Vieira and Poesio (2000) for noun phrase coreference
detection.

This tendency is also confirmed by the quantitative study of Recasens and Hovy (2009), who
found head match to be the most relevant of the common features used in coreference resolution
even though it is irrelevant for pronominal reference. From a representation perspective, this
means that the head word of a mention, along with its features, provides a good approximation
of the mention as a whole — at least for most actual occurrences. Accordingly, all subsequent
works using external features use head words as a central part of their mentions representations.

From then on, the representation of mentions used in coreference resolution systems mostly
coalesced around the features used by Ng and Cardie (2002) (or subsets of them), including in
languages other than English with Broscheit et al. (2010) and Kopeć and Ogrodniczuk (2012)
using a compact set of such features for English, German, Italian and Czech. Attempts to expand
this feature set to include deeper linguistic insights were also made. For instance, Haghighi and
Klein (2009) propose to use heuristic estimations of salience and centering, inspired by their
use in early works such as Hobbs (1986). These were also part of the highly successful dcoref
system (H. Lee et al. (2013, 2011), also known as ‘Stanford’s sieve’ and ‘Stanford’s deterministic
coreference system’). However, Durrett and Klein (2013) shows that syntactic features such as
salience can be approximated using simpler features (see section 4.1.1) provided that enough
annotated data is available. Conversely, shallow semantic features such as those extracted
from WordNet only provided marginal improvement when used for system mentions. As a
result, their final system use a much shallower feature set, but relies instead on strong learning
algorithms and a large dataset, a method that later works tend to follow as it reduces the need
for costly expert rule handcrafting.<fnref n="4" />

In later years, with the rise of neural networks in natural language processing following Collobert

<footnote n="4">⁴It does enhance the need for annotated data and computational power, but at least for English, these are currently readily available.</footnote>
et al. (2011), works using external features such as Clark and Manning (2016a,b) and Wiseman et al. (2016, 2015) mostly stuck with the feature sets from Durrett and Klein (2013), occasionally including some features proposed by Recasens et al. (2013). The —mostly successful— strategy behind this choice is to leverage the greater capacities of neural networks architectures to learn higher level representations from low-level features rather than relying on rich linguistic features.

## Word-level features and compositions

The rich features described in section 4.1.1 have been shown to be efficient, especially in situations where annotated data is scarce, as in early efforts for English before the advent of large corpora or in current effort in lesser-resourced languages such as Basque (Soraluze et al. 2019). However, they all depend on the availability of a sufficiently reliable preprocessing pipeline for the target language *and domain*, whose quality strongly conditions that of the whole coreference resolution system.

When such a pipeline is not available, or when it is not accurate enough, the only features available are those that can be directly deduced from the contents of the mentions. As we mentioned in section 4.1.1, this content, represented as a string of characters, was already used in the representations of Soon et al. (2001) to fuel string-matching features (see section 4.1.2), and the content of the *head* word plays a central role in external features set (but depends on syntactic parsing). The first use of raw word-level features, however, came from Durrett and Klein (2013), who successfully introduced the contents of the first and last words of a mention and of the immediately surrounding words in their mentions representations.

When trained with enough data, these features can implicitly capture syntactic information in simple cases. For instance, in English, a subject NP is often followed by a verb and an object NP is often preceded by a verb and therefore in these cases, the words surrounding a mention can provide insights on its external syntactic features. Conversely, still in English, the first word of mention is often a good indicator of its definiteness—among other features—so a machine learning system can take advantage of the ‘first word’ feature to detect definiteness without help from a handwritten rule.

Still, no matter how useful these word-level features can be, Durrett and Klein (2013) and subsequent works using their feature set, such as Wiseman et al. (2015) or Clark and Manning (2015) were still relying on features targeting mentions' head words and were still dependent<fnref n="5" /> on syntactic parsing for their mention representations needs. The reason was that while words at mentions' boundaries are informative, in general, they do not capture any semantic information about the mention and little information about its internal syntactic structure. Conversely, the head word of a mention often⁶ carries the bulk of the semantic of the mention as a whole.

In that regard, the breakthrough is due to K. Lee et al. (2017): instead of using head words as

<footnote n="5">⁵A dependency that was not unjustified, since in an end-to-end setting, they also depended on syntactic parsing for mention detection in any case.

⁶But of course not always, consider for instance ‘the chair of the department of social sciences’ vs. ‘the chair in the kitchen’.</footnote>
rough summaries of mentions, they instead derive dense vector representations from the hidden
states of a recurrent neural network operating on the words of the document. More specifically,
given a mention, they apply a LSTM-based network (Hochreiter and Schmidhuber 1997) on the
sequence of the words in its sentence, and use as a representation of the mention the hidden
states of this network corresponding to the boundaries of this mention and a weighted average
of those corresponding to the inner words of the mentions as a ‘soft-head’ (see section 5.2.2 for a
more detailed description). These choices are reminiscent of the features used in previous works,
but they have the crucial advantage of depending on all the words in the sentence and in the
mention instead of a few heuristically chosen words. In fact, since this architecture is capable
of processing whole documents<sup>7</sup> word-by-word, the only theoretical limits to the quality of
its representations are the expressivity of its encoder (here the LSTM-based network) and the
quality of its training. Most importantly, these representations do not require preprocessing
beyond word segmentation, and are instead able to learn by themselves at least part of the
features that classical pipelines provide: an examination of the internals of the models by K. Lee
et al. (2017) suggest that the soft-head feature is often correlated with the actual syntactic head
of a mention.

Additionally, with neural network models also came another powerful tool: word embeddings (Bengio et al. 2003), which provide dense vector representations for *words*. While word embeddings are almost always an improvement over sparse word representations for uses in neural network models (Collobert and Weston 2008; Collobert et al. 2011), *pretrained* word embeddings are particularly interesting for coreference resolution since they encode at least some lexical semantic information (Mikolov et al. 2013), giving to the systems that use them some help in fighting Durrett and Klein (2013)'s 'uphill battles'. Pretrained word embeddings are a form of external knowledge, but one that is more easily generalizable to other languages and domains that dedicated preprocessing pipelines, since they only require the availability of raw documents — for instance using large web crawl corpora (Ortiz Suárez et al. 2019)—, with no need for expert annotations. Their first use for coreference resolution is due to Clark and Manning (2016b) who used Mikolov et al. (2013)'s original word2vec pretrained embeddings, both for unique words (boundaries and head) and for sets of words using the averages of the embeddings of the context words of the mention (with left and right context windows of size 5), the content of the mention the whole document. Nitoń et al. (2018) uses similar word embeddings-based representations with success for coreference resolution in Polish.

Finally, in the last two years, the takeover of *contextual* word embeddings — based on techniques not too dissimilar to the LSTM-based feature extractor of K. Lee et al. (2017) and pretrained on language modelling tasks — have provided coreference resolution systems such as Joshi et al. (2019a,b), Kantor and Globerson (2019) and K. Lee et al. (2018) with word representations that directly include sentence-level information, obviating even the need for further recurrent word encodings.

<footnote n="7">⁷the limitation to sentence-level processing is a practical choice to limit the computational complexity of the model, not a theoretical one</footnote>
## Bound features

The features defined in section 4.1.1 are autonomous, in that they do not depend on the existence or the characteristics of other mentions in the document. Indeed, historically, representations involving several mentions have often been formulated at the level of pairs (see section 4.1.2) or sets of mentions (see section 4.1.3) and designed in reference to a particular coreference resolution algorithm. But the focus put by the use of neural networks architectures on rich representations rather than sophisticated algorithm have made bound representations — representations that do depend on the other mentions in a document — desirable, as their use can counterbalance the limitations of the greedy antecedent-finding algorithms popular in these systems.

Specifically, the ‘higher order representations’ introduced by K. Lee et al. (2018) and used by Joshi et al. (2019a,b) enhance mention embeddings by summing them with an average of the vector representations all preceding mentions, weighted by their coreference likelihood as estimated by a first round of their antecedent scoring algorithm, a process that can be iterated. Assuming that the coreference likelihoods are correctly estimated and that the initial representations are relevant, this average provides an approximate representation of the mention’s antecedents.

Kantor and Globerson (2019) use a similar approach, but replace the mention pair formulation of K. Lee et al. (2018) by an entity-mention approach based on the fuzzy clustering proposed by P. Le and Titov (2017). In this formulation, fuzzy cluster representations are built recursively by estimating the degree of membership of every mention to the fuzzy clusters defined by the previous mentions initialized as singletons. In practice, this should be very close to what would be obtained with enough iterations of K. Lee et al. (2018)’s higher-order representations, but computationally more efficient.

### 4.1.2 Mention pairs

Mention pairs are critical objects in link-centric coreference resolution algorithms (section 4.2.2), but also in some entity-centric approaches such as Clark and Manning (2016b), where representations of cluster pairs is obtained by pooling the representations of the corresponding mention pairs. Historically, as we mentioned in section 4.1.1, mention pairs were actually the primary concept modelled for machine-learning coreference resolution systems, with mentions representations themselves being mostly latent. Conversely, in later works such as those following the approach of K. Lee et al. (2017), mention pairs are represented by a simple concatenation or pooling of the vector representations of the two mentions — leaving it to the antecedent scoring module to combine them in its internal layers to produce a score — but are not explicitly modelled, except for a single distance feature.

Except for systems that are more general anaphora resolvers, coreference resolution systems are only interested in mention pairs to determine if they are coreferent (or to compute a scalar coreference score), so the features used in their representations of mention pairs are *matching* features — boolean values indicating if the representations of the two mentions have the same value for a given feature — or *similarity* features — a degree of agreement between the two mentions on a given feature. These features are motivated by the (reasonable) assumption that
features of a mention are conditioned by properties of the corresponding entity, which implies
that coreferent mentions should have matching features. This is also supported by empirical
findings such as the preference for antecedent with identical syntactic functions (syntactic
parallelism, Smyth 1994; Stevenson et al. 1995)

Therefore, a simplistic model could decide that a pair is coreferent if and only if all the features
available for both mentions are matching. However, examples such as example 25 and corpus
studies such as Antoine (2004) show that grammatical gender for instance can be mismatched
between two coreferent mentions in at least some contexts, so that naïve ‘all must match’
heuristic is not sufficient in practice and combinations of features are needed.

(25) (a) # *At the farmhouse, the cowgirl left his lasso in the kitchen.*

(b) *At the Halloween party, the cowgirl left his lasso in the kitchen.* (Ackerman 2019)

Some systems also include *compatibility* and *preference* features, indicating if coreference
between the two mentions is possible according to a given heuristic and its order of priority
in case of ambiguity. These rules are usually derived from linguistic models of anaphora such
as government and binding theory (Chomsky 1981) or empirical observations such as the
preference for antecedent with subject roles. These features can also come from auxiliary tools,
for instance Ng and Cardie (2002) use the output of ‘a naïve pronoun resolution algorithm’ as a
pair feature.

Finally, most systems include *distance* features, integers or scalars encoding the distance between the two mentions. These might target various units, for instance Ng and Cardie (2002) include a distance in number of paragraphs, but the most common are distances in sentences, words and intervening mentions. These model to some extent the distance in discourse in terms of time or cognitive events, with the underlying supposition that discourse entities are stored in short-term memory, implying that coreference between distant mentions with no other coreferent intervening mentions follows different rules than coreference between close mentions. This property can be also be used to choose a single antecedent when several are equally likely, for example in the form of the ‘closest-first’ heuristic used by Soon et al. (2001).

4.1.3 Mention sets

As mention pairs for link-centric algorithms, mention *sets* lie at the core of most entity-centric
algorithms. They model not only whole coreference chains, but also partial sets of coreferent
mentions clustered together by a system as intermediary steps that can be seen as encoding the
partial knowledge that a system can have of the corresponding entity at a certain step in the
analysis of a document.

Accordingly, representations of mention sets that use explicit features tend to define them via
unions, intersections or averages of mention-level features, meant to represent properties of the
corresponding entities by inferring them from their known mentions. For instance, Luo et al.
(2004) represent sets of mentions using multi-valued features that are the sets of the values of
the corresponding mention-level features for the mentions that it contains allowing for lenient
matching with other mentions: for an entity-mention pair (e, m) with e = {c₁, ..., cₙ}, a given
feature is matching if there is at least one *i* such that the values of this feature are the same for *m* and *c*ᵢ. In contrast, Yang et al. (2004) and later works such as Culotta et al. (2007) are more strict, and derive their set-level features by taking majority values over the mentions in a set, often with a majority ratio indicating if a feature value is shared by none, some, most or all of the mentions. These features provide sensible approximations of entity properties, but they can be too rough in practice, for instance for entities with varying properties or in end-to-end settings, where spurious mentions can blur their accuracy.

Björkelund and J. Kuhn (2014) propose a more precise representation by considering not features *sets* but features *sequences*, e.g. a set containing a proper noun, then a pronoun, then a common noun, then a pronoun would have a feature value for ‘mention types’ of ‘NPCP’. By taking discourse order into account, these give a better accounting of the status of an entity, but the price of that precision is their far greater sparsity, making them hard to use in machine learning models.

Wiseman et al. (2016) tackle this sparsity issue elegantly, by using dense vector representations for clusters. Their system process documents left-to-right, in discourse order, and maintain a list of partial clusters. Every time a new mention is attached to an existing cluster, they derive a new representation for this cluster by applying an LSTM layer to the concatenation of its previous representation and the representation of the new mention (itself obtained from a neural network, see section 4.1.1). This approach relinquishes explicit control of clusters representations, but provides a relatively simple architecture for a system that process documents in their natural order and include effective clusters representations. A related<fnref n="8" /> approach is used in the recent work of Fei Liu et al. (2019), where entities representations are stored in slots of a memory network (Sukhbaatar et al. 2015; Weston et al. 2015) and updated by a recurrent neural network at each affectation of a mention to an existing partial cluster.

## 4.2 Coreference resolution algorithms

In this section, we describe the existing algorithms for coreference resolution, the task of partitioning the set *M* of the mentions in a document in coreference chains. Many solutions exist for this task, differing by their model of coreference and their approach of *M* as either a sequence or an unordered set. Historically, the choice of a specific algorithm is mostly empirical, with no clear ranking between the solutions and no definitive study of their respective merits. Indeed, since comparisons are usually done on different representations, different supporting architectures and different material restrictions (for instance on computing power), which makes it hard to determine the properties of these algorithms from works on systems that use them.

### 4.2.1 Ordering

An important distinction between coreference resolution algorithms, is between *document-order* and *global* algorithms. *Document-order* algorithms build coreference chains by processing

<footnote n="8">⁸Although their model of coreference differs significantly</footnote>
documents in linear reading order for written language or in utterance order<fnref n="9" /> for spoken language, effectively modelling M as sequence of mentions. *Global* algorithms do not take this order into account and process documents globally, modelling M as an *unordered* set of mentions. Of course, systems using global algorithms are not necessarily (and indeed not usually) order-agnostic, but they encode order at the *representation* level (see section 4.1) rather than in their document processing.

This distinction is mostly independent of the model(s) of coreference used by algorithms. Some models are more naturally suited for one approach or the other — for instance, mention-centric models are usually used in document-order — but all models can in principle be used with or without order. Rather, the choice between these alternative is a trade-off between the ability to take all the information contained in a document into account and the available processing resources.

Generally speaking, document-order algorithms tend to be easier to develop (and to train, for machine-learning systems) and are less demanding in resources, since they can usually be formulated in terms of mostly *local* decisions and operate greedily. They are also more appealing from a modelling point of view: human processing of documents is by necessity done in order<fnref n="10" /> and humans usually do succeed at coreference resolution, which implies that coreference chains in natural language documents must have structures that are compatible with and constrained by the structure of said documents. Therefore, a document-ordered algorithm, having direct access to this structure, could in principle use this structure to guide their operations. However, coreference resolution systems operate under more constraints than humans in terms of general language processing ability, world knowledge and explicit reasoning capabilities. Thus, a document-order algorithm might be unable to accurately process a document. In particular, compoundings of early errors make processing mentions near the end of a document particularly challenging.

Conversely, global algorithms can take into account more information, since they have access to the entirety of documents and can use this information to compensate for their lack of world knowledge. For instance, the grammatical gender of proper noun mentions is not necessarily easily accessible at the beginning of a document but might be deduced from later uses as in example 26, where without more context, ‘she’ might refer to ‘Jean’, but the later unambiguous reference of ‘he’ to ‘Jean’ makes far less likely.

(26) *Jean and Siobhán went to Orléans last year for an exchange program, and she was really happy about it. [...] Poor Jean, he was really unlucky.*

However, global processing of this kind is far more demanding in resources and more complex to design. As a result, it is not uncommon to see global algorithms performing worse than document-order ones, which can make better use of the available resources by using better representations or a richer model of coreference.

<footnote n="9">⁹Which might be problematic for settings with significant overlapping, but no such resource for coreference is available yet.</footnote>

<footnote n="10">¹⁰Although it can include some degree of delaying and backtracking. See for instance the results reported by Sturt and Lombardo (2005) for pronominal reference resolution.</footnote>
4.2.2 Link-centric algorithms

Link-centric or mention pair algorithms operate within what we called the link-centric model of coreference in section 2.2. In this model, the objects of interest are links — potentially oriented — between mentions, and the coreference resolution task is cast as the task of finding a graph structure Γ on M such that the connected components of Γ are coreference chains. In this setting, the coreference chains are latent structures and are not explicitly modelled.

Formally, almost<fnref n="11" /> all link-centric algorithms can be described by the two following elements<fnref n="12" />

* A pair scoring function $s: G \to V \subset \mathbb{R}$, where $G = \{\langle m, m' \rangle \subset M \mid m \neq m'\}$ is the set of unordered mention pairs

* A decoding procedure that maps the triplet $(M, G, s)$ to a graph $(M, \Gamma)$ where $\Gamma \subset G$, i.e. an edge selection heuristic

The decoding procedure is usually a heuristic optimization procedure operating on a constrained search space for $\Gamma$.

In that framework, coreference resolution is solved in two steps: first, scores are computed independently for all mention pairs, then a decoder selects the pairs in the output — either globally or in document order — to derive the system’s output. The two steps are usually partially independent, in the sense that different decoders can be used with the same scoring function and vice-versa.

The simplest realization of this framework is used for instance by McCarthy and Lehnert (1995), where mention are classified as either coreferent or non-coreferent and coreference chains are build transitively. In this case, $V = \{-1, 1\}$, $s(m, m') = 1$ if and only if *m* and *m'* are classified as coreferent (in McCarthy and Lehnert (1995), this was determined by a decision tree) and the decoding procedure globally selects all edges $\{m, m'\}$ such that $s(m, m') = 1$. In principle, this realization could yield perfectly accurate results, but it requires the classifier to have a perfect output, since every misclassification in the coreferent class will result in mistakenly merging two coreference chains. For MUC, which was the target of McCarthy and Lehnert (1995), this might yield good results, since the MUC metric is notably lenient with such overmerging (see section 2.4.2), but for most application this is an undesirable property, which is accordingly severely penalized by other metrics.

Antecedent-finding algorithms

Soon et al. (2001) also uses a decision tree classifier (albeit with different mention representations, see section 4.1.1), but proposes a different and important decoding procedure by selecting at most one antecedent per mention: a mention pair $\{m, m'\}$ such that $m'$ is before *m* in document order is included in $\Gamma$ if and only if *m'* is the closest<fnref n="13" /> mention such that $s(m, m') = 1$, the so-

<footnote n="11">¹¹To the best of our knowledge, only the twin-candidates model of Yang et al. (2003) requires a more complex formulation. Cai and Strube (2010) and Martschat et al. (2012) are formulated similarly but use higher level hypergraphs and multigraphs.</footnote>

<footnote n="12">¹²This formulation is inspired by Lassalle (2015).</footnote>

<footnote n="13">¹³This condition could also be included in s.</footnote>
called ‘closest-first’ decoding. This decoding is much safer than the transitive closure decoding
of McCarthy and Lehnert (1995): since every mention has at most one backward link, a spurious
{m, m'} link will only merge the chain of m' with the subsequent mentions of the chain of m
instead of its whole chain. However, a spurious link for a discourse-new mention will still
merge its whole chain.

This decoding procedure can be extended to the more general concept of *antecedent-finding algorithms*: algorithms where the search space for Γ is restricted to graphs where every mention is adjacent to at most one preceding mention in document order. It is easy to see that with this constraint, Γ is necessarily acyclic, but not necessarily connected<fnref n="14" /> and therefore has a forest structure. Accordingly, we call these graphs *discourse-order forests*.

Of course, due to the structure of the search space, antecedent-finding algorithms are naturally
discourse-ordered. While global alternatives such as an iterative search for a single coreferent
mention *m*′ for every mention in a document are theoretically conceivable, they don't really
add much power if used greedily. Instead, the corresponding global approaches of latent trees
usually uses more sophisticated decoding procedures (see 'Global decoding' below).

A limitation of the approach of Soon et al. (2001) is that it has to chose between two imperfect alternatives in a machine learning setting for the pairs {m, m'} used as positive examples for s(m, m') = 1:

* Use only pairs where *m*' is the closest antecedent of *m* as positive examples, which restricts the size of the training set but is consistent with the decoding heuristic

* Use all coreferent {m, m'} pairs, which gives more learning opportunity for the scoring function, but is inconsistent with the decoding, which only selects the nearest antecedent.

Soon et al. (2001) chose the first option with good results, but given the critical importance of
a large training set for this kind of systems — highlighted by Durrett and Klein (2013) —, this
choice is debatable.

Another obvious limitation of the closest-first heuristic is that it prevents easy resolution in
some cases. Consider for instance the case where two identical proper noun mentions N and
N' appear in document order with an intervening pronoun P compatible with N': N—P—N'. In
that case, if the classifier deems that {P, N} is coreferent, it will not even consider the {N, N'}
pair even if it far more likely to be coreferent.

A solution to these issues is proposed by Aone and William (1995) and Ng and Cardie (2002):
rather than arbitrarily select the closest antecedent, the authors propose instead to select the
antecedent for which their classifier has the most confidence, effectively switching from the
previous binary-valued s to a scalar-valued one. By keeping the condition that s(m, m') is set to
-1 if {m, m'} is classified as non-coreferent and to the confidence of the classifier else, this is
equivalent to searching the discourse order forest Γ such that Σ_{ {m,m'}∈Γ } s(m, m') is maximal. This

<footnote n="14"><sup>14</sup>And indeed, a connected Γ is usually a bad result for a coreference resolution system. Note however that optimization algorithms operating on tree structures can still be used by modelling discourse-new mentions as linked to a dummy ε mention instead of having no antecedent</footnote>