decoding procedure is known as ‘best-first’ clustering and tends to work better than closest-first
clustering.

However, there is still an issue with the formulation of Ng and Cardie (2002) in that s is not
directly optimized for in the training process, but is a by-product of training a binary classifier.
More precisely, a binary classifier assumes independence between samples and its confidence
score does not measure a degree of belonging to a given class. Rather, it is not unusual for
binary classifiers to err with a high confidence score on unseen samples. Therefore, using a
confidence score as an edge scoring function, if it does help in some cases, is not a good solution
in general.

Bengtson and Roth (2008) and Denis and Baldridge (2008) offers an alternative architecture
for best-first antecedent-finding, where s is directly learned.<fnref n="15" /> This allows to take the relative
scores of all the antecedent candidates of a mention into account, resulting in a more robust
scoring function, while still keeping it strictly pairwise. This approach has been adopted by
many subsequent works (Clark and Manning (2016a), Durrett et al. (2013), Joshi et al. (2019a,b),
K. Lee et al. (2017) and Wiseman et al. (2015) among others), with various implementation and
learning algorithms for s and various representations. Indeed, the case of K. Lee et al. (2017) in
particular, which uses a rather straightforward learning algorithm and almost no additional
knowledge suggests that despite its relative simplicity, this decoding procedure can be very
efficient when used with solid input representations and a powerful learner for s. However, the
architectures involved for these latter success are very demanding in data and in computing
power and might not be applicable to situations with more constrained resources.

Global decoding

The performances of the antecedent-finding models are closely tied to the accuracy of the edge
scoring function, and it is not always possible to improve it. Instead, another way to improve
link-centric algorithms is to change the constrained document-order decoding for a global
optimization procedure. By giving more power to the decoding procedure, these strategies
can take advantage of all the information provided by the scoring function and compensate its
eventual inaccuracies. As we mentioned in section 4.2.1, the price for these improvements is
the need for more complex implementations and a higher computational cost.

One way to use a global decoding with a pairwise scorer is to formulate decoding as an Integer
Linear Programming (ILP) problem, which returns a clustering with maximal total weight that
satisfies arbitrary linear constraints. This is the choice made by Denis and Baldridge (2009),
Klenner (2007), Roth and Yih (2007) and Uryupina (2010). The constraints can be formulated
to enforce desirable properties on the resulting graph, such as *transitivity*, which ensures that
the clusters generated by the decoding procedure do not include negative-scored links (i.e.
non-coreferent mention pairs). The main issue with ILP formulation is that solving an ILP
problem is NP-complete in general, and while Roth and Yih (2007) notes that ILP formulation of
coreference are usually of a lower class of complexity, in practice, they are still too expensive

<footnote n="15"><sup>15</sup>Using a log-linear model for Denis and Baldridge (2008) and an averaged perceptron for Bengtson and Roth (2008).</footnote>
for most uses (Lassalle 2015).

An alternative to ILP is the use of *latent tree* models (Björkelund and J. Kuhn 2014; E. Fernandes et al. 2012; Eraldo Rezende Fernandes et al. 2014; Lassalle and Denis 2015). Instead of considering all possible graphs in the search space for Γ, this formulation restricts it to the set of forest structures over M. This is a larger search space than in antecedent-finding algorithms — which considers only *discourse-order* forests — but it is still regular enough that polynomial optimization algorithms exist.<fnref n="16" /> Another advantage of latent trees algorithms over ILP is that the scoring function can be optimized directly for the targetted structures, for instance using the structured perceptron algorithm (Eraldo R. Fernandes and Brefeld 2011), where the scoring function in ILP was trained for an antecedent-finding objective, which does not ensure that it would perform correctly with an ILP decoder.

### 4.2.3 Entity-mention models

As we have seen in section 4.2.2, despite their relative simplicity, mention-pairs models can provide good performances for coreference resolution. Yet, they require a choice between two extreme alternatives: document-order algorithms have limited access to the information contained in documents and require very accurate input representations and strong learners, while global algorithms have to deal with large and complex graphs of mentions which results in greater complexity. Between these alternatives, a middle ground is to use an explicit modelling of mention sets, which allows taking into account information beyond mentions without having to deal with overly complex structures on M. As we have seen in section 4.1.3, modelling mention sets is usually thought of as approximating entities, using the properties of the mentions in a partial coreference chain as an approximation of the attributes of the corresponding entity.

#### Mention-centric algorithms

The most common family of algorithms using entity modelling are mention-centric entity-mention models. These models correspond to the discourse-oriented modelling of reference that we have described in section 2.1.1: reference can be thought of as a (possibly directed) relation between a mention and an entity in a discourse model, either already existing or summoned by the mention. Accordingly, mention-centric models proceed to build coreference chains by iteratively considering every mention *m* and either attaching it to an already existing chain or creating a new chain (*m*). These algorithms are usually run in document order, which is also consistent with their closeness to human models of coreference resolution, i.e. mentions are considered in document order and are attached to sets of preceding mentions. In principle, a global mention-centric model would be conceivable although harder to relate to theoretical principles, but no work has currently explored this possibility.

Conceptually, this is very close to antecedent-finding, except that the antecedent here is not a single mention, but a set of mentions carrying higher level information. Theoretically, this could have the advantage of reducing the number of alternative antecedents, however, many

<footnote n="16">¹⁶Such as Edmonds’ algorithm (Edmonds 1967) for directed forests or Kruskal’s algorithm (Kruskal 1956) for undirected forests.</footnote>
implementations still consider attachment to a cluster as an attachment to a specific mention in
a cluster and thus describe the whole procedure as antecedent mention finding. This distinction
does not make a fundamental difference, since it can be seen simply as a different procedure to
choose an antecedent cluster.

A clear distinction with antecedent-finding models is that while choosing an antecedent for a mention is independent of the choices for the other mentions, since the available mention sets available for attachment at any single time depend on all the previous decisions. This means that the result of a mention-centric algorithm depends on a chain of dependent attachment actions rather than on independent antecedent-finding, making learning the full problem intractable. In practice, as with other such problems in natural language processing, this is usually solved by using a Markovian model where the choice of action at each time step depends only on the current state (in that case the list of incomplete clusters) rather than the full sequence of actions. During learning, the attachment choice is usually presented with gold clusters from an oracle.<fnref n="17" /> At inference time, decoding is usually done either greedily or using a beam search.

Historically, the first mention-centric models came simultaneously from Luo et al. (2004) and Yang et al. (2004), with a similar formulation as an antecedent-finding procedure with cluster-level features using classifiers, Luo et al. (2004) with an interesting formulation of beam search using the Bell tree structure to model incremental partition building and Yang et al. (2004) with a more usual greedy decoding. Recasens and Hovy (2010) also uses a similar model (CISTELL) to investigate discrepancies in coreference resolution across corpora and metric bias. Rahman and Ng (2011) proposes a rather natural extension of the previous entity-mention models to act by *ranking* clusters, similarly to the move from mention-pair classification to antecedent scoring models.

Most recently, Wiseman et al. (2016) shows that using neural network architectures and dense representations could greatly improve the performances of mention-centric models, consistently with the improvements they provide to antecedent scoring models (Wiseman et al. 2015). The memory network architecture of Fei Liu et al. (2019) is also a form of entity-mention network which shows encouraging performances for pronominal reference resolution.

## Entity-centric algorithms

A dual approach to the mention-centric algorithms where coreference chains are built by attaching mentions and the base objects are *mentions* is to consider *coreference chains* as the base object and to build them by agglutinating partial clusters in multiple passes. This has the crucial advantage over other iterative methods that failing to detect that two mentions are coreferent at first can be recovered from at any time by merging their clusters, possibly using cluster-level information. In contrast, in antecedent-finding and mention-centric model, setting a mention as discourse-new is a final decision that can not be overridden by later information.

This approach was originally proposed by Raghunathan et al. (2010), using a rule-based system
in the so-called ‘multi-sieve’ approach also used in the later works of H. Lee et al. (2013, 2011,

<footnote n="17">¹⁷To our knowledge, while the bias induced by static oracles are well known for other tasks, dynamical oracles, used for example in syntactic parsing (Goldberg and Nivre 2012), have never been used for coreference.</footnote>
2017). In this formulation, a systems starts with a partition of M in singletons and applies successively a series of rules (handcrafted in H. Lee et al. (2013, 2011) and partially learned in H. Lee et al. (2017)) to merge existing clusters in document order.

A similar cluster agglomeration model is proposed by Stoyanov and Eisner (2012), but uses a reinforcement learning approach operating in global fashion: at every step of the algorithm, an agent observes the current clustering and emits an action to either merge two clusters or stop the process. This formulation allows performing ‘easy’ cluster merging first and delaying complex decisions until later, at which point the information contained in the partial clustering will hopefully make them easier. This approach is also used by Clark and Manning (2015, 2016b), which extract cluster-pair representations from auxiliary link-centric models and in the case of Clark and Manning (2016b), dense mention representations and a deep policy network.

## 4.3 Mention detection

Mention detection is a crucial part of an end-to-end coreference resolution system and the results on coreference resolution presented in the recent literature tend to report end-to-end results — following the standard tracks of the SemEval and CoNLL shared tasks (Pradhan et al. 2012, 2011; Recasens et al. 2010). In this end-to-end setting, the accuracy of mention detection is also crucial for the overall results, for instance, Ng (2008) reports far worse results for their unsupervised system when using system instead of gold mentions, an observation that is shared in the systematic study of Stoyanov et al. (2009).

Despite this, the attention received by the mention detection task has been mostly anecdotal: few works are specifically dedicated to mention detection, and it is rarely studied in details in works on end-to-end systems. This lack of interest might come in part from the particular status of mention detection in resources for English: among the corpora used in shared tasks, viz. MUC (Hirschman and Chinchor 1998; MUC Consortium 1995b), ACE (Doddington et al. 2004), Ontonotes (Pradhan et al. 2007a), SemEval (Recasens et al. 2010) and ARRAU (Poesio and Artstein 2008; Uryupina et al. 2019), only the last two includes annotations for all mentions including singletons. Furthermore, the English part of SemEval has not received as much attention as it could have, since it was mostly supplanted for English by the Ontonotes-based CoNLL corpora (see section 3.1.4), and the use of ARRAU in a shared task is very recent (Poesio et al. 2018).

Works with specific studies of mention detection have mostly been focused on ACE, which does include singletons, but only for the restricted class of entities that are annotated, and these works also include mention type classification, which makes it hard to assess their general results on untyped detection. For corpora that do not include singleton, mention detection is hard to study, since it is compounded with coreference resolution. Consequently, in corresponding works, mention detection is often a mere pre-filtering recall-oriented step and the techniques used to perform it are not particularly concerned with the precision of the detection, making simple heuristic mostly sufficient. In fact, the success of the span-based models initiated by K. Lee et al. (2017) suggests that for these resources, an explicit mention detection step is not an obligation.
It is hard to determine precisely the origin of mention detection in coreference resolution, with
early works on MUC using gold mentions or simply all NPs extracted from parse trees. Most of
the works that give any details on their procedure seem to use rule-based techniques such as
those reported by Uryupina (2010) or H. Lee et al. (2011): NPs, pronouns and named entities
are extracted, either using gold knowledge or from preprocessing pipelines, and a set of filters
is applied on them to filter out those that are known to be non-referential (which is usually
corpus-dependent, as noted by Stoyanov et al. (2010), whose RECONCILE system use different
sets of rules depending on the resource it is used on). When syntactic parsing is not available,
some works such as Soraluze et al. (2012, 2017) also report encouraging results by combining
lower-level preprocessing tools with extensive rule-based systems.

Beyond rule-based systems, Uryupina and Moschitti (2013) propose a machine learning system, still using syntactic parsing, but learning a noun-phrase classifier to filter out non-referential NPs. This allows extending to new languages more easily than when using language-specific rules, with a significant improvement over the all-NP baseline previously used by many systems for the Arabic and Chinese portions of Ontonotes.

More recently, the very promising work of B. Wang et al. (2018) describes a method for a full
end-to-end mention detection system in the nested named-entity recognition task that does
not rely on pre-existing parse trees, but learns instead a partial transition-based parser in a
semi-supervised setting where the only supervision is on the correct detection of mention spans.

## 4.4 Systems for French

Due to the recency of publicly available annotated resources for coreference resolution in French (see section 3.2), very few works exist on coreference resolution for French. Two rule-based systems, Longo (2013) and Trouilleux (2001), for narrow coreference resolution exist, but they suffer from a lack of resources, both for development and evaluation.

Such resources were first provided with the release of ANCOR (Muzerelle et al. 2013a, 2014),
but still, prior to this thesis, only two works studied coreference resolution, perhaps due to the
very specific nature of this corpus and the lack of suitable preprocessing tools.

Désoyer et al. (2015a) proposes the first coreference resolution system for French: CROC. It was
not meant as a full end-to-end system, since it only performed coreference resolution using both
gold mention spans and the gold mention features described in section 3.2.2. The coreference
resolution model use *mutatis mutandis* the same representations as Ng and Cardie (2002) and
the same closest-first antecedent-finding algorithm as Soon et al. (2001), supported by a SVM
classifier for their best model. Their results (reported in table 5.5, p 90) were very encouraging,
considering the relative simplicity of the system, which confirms the relevance of the gold
mention features included in ANCOR for coreference resolution.

The more recent work of Godbert and Favre (2017) is an attempt at an end-to-end coreference
resolution system, but was in this work only limited to pronominal reference and so-called
'direct anaphora' between NPs. In contrast with Désoyer et al. (2015a), it does not use a machine-
learning system, but a rule-based system on top of the MACAON pipeline (Nasr et al. 2011),
with a strong focus on pronouns and a rather simple heuristic for noun phrases.
# Chapter 5

## Automatic coreference resolution

Despite the large body of work in natural language processing for French in general, the specific task of coreference resolution has not received as much attention as it has in other well-resourced languages (such as Czech, Polish, German, Spanish or Catalan). Indeed, without any large-scale publicly available corpus with coreference annotations, until the publication of ANCOR (Muzerelle et al. 2014), early attempts (Longo 2013; Trouilleux 2001) at building a coreference resolution system were largely limited by either the scarcity of resources or their usage restrictions. Since then, some partial proof-of-concept systems (Désoyer et al. 2015a; Godbert and Favre 2017) have been produced and some experiments (Brassier et al. 2018) have been conducted using ANCOR, but there is yet no publicly available end-to-end coreference system for French.

In this chapter, we propose to address this lack, by proposing DeCOFre: an end-to-end corefer-ence resolution system leveraging state-of-the-art neural architectures and trained and evaluated on ANCOR.

In section 5.1, we list our requirements for an end-to-end coreference resolution system and justify our choices to designing a new system (instead of adapting an off-the-shelf system to French) and to use a two-step pipeline architecture as opposed to the fully end-to-end architecture used in the current state-of-the-art.

In section 5.2, we expose the structure of our system from a global point of view, describe our implementation in details and motivate our architectural choices.

In section 5.3, we describe the optimization process used to train the parameters of our system on ANCOR.

Finally, in section 5.4, we detail the experimental settings we used to assess the performances of our system and report some additional experimental results, meant as partial empirical validations for some technical choices described in the previous sections.
## 5.1 System scope

In designing our coreference resolution system, given our goal of end-to-end coreference resolution on ANCOR, in the configuration described in section 3.2.2 our main requirements for it were that

1. It had to be able to work directly on raw transcriptions and detect mentions by itself.

2. It had to treat singleton as chains.

3. It had to be *suitable for spontaneous spoken language*, which can be significantly different both from written language and from controlled and planned spoken language. The most inconvenient of these differences for us were the lack of clearly defined sentences boundaries and the presence of distractors such as disfluencies, incomplete words and phatic markers.

We also had some secondary requirements

4. The system had to be suitable for our data — namely ANCOR — with its specificities beyond language and genre. Most notably its great heterogeneities in terms of document sizes and natures (see section 3.2.2) was an issue.

5. The lack of pre-existing NLP resources for spoken French: at the time of writing, there are no publicly available systems for the usual preprocessing steps used in coreference resolution (POS-tagging, parsing, NER...) that reliably supports spontaneous spoken French. Moreover, systems designed for written French are of unreliable quality on spoken French and would require considerable post-correction work to reach the quality required for direct use (see Grobol et al. (2017b) and Tellier et al. (2014), ways of partially overcoming this issue are explored in chapter 6).

6. The limitations of our hardware in terms of memory and processing power. For instance, our early attempts to use K. Lee et al. (2017)'s E2E system directly on ANCOR simply crashed, having filled up all the memory available on our machines.

Considering these points, the publication of the E2E<fnref n="1" /> system K. Lee et al. (2017) was determinant, as it provided us with an architecture that met our requirements 1 and 5 by being able to take raw text as input, without relying on external resources while still reaching state-of-the-art quantitative results on the CoNLL-2012 benchmark.

However, while making E2E comply to 2 could have been done, for instance by using their *s*<sub>*m*</sub> score or a learned *s*(·, ε) score, as a discourse-new detector, the interaction between 6 and 4 prevented us to use it as a base system to build upon.

Indeed, since E2E always operate at the document level, whole documents have to be kept in memory during training and error backpropagation must take into account the operations performed on all of their text spans. To counter the prohibitively high computational and

<footnote n="1">¹For ‘end-to-end neural coreference resolution system’. The authors did not give a name to their system, but this abbreviation is becoming common in recent works.</footnote>
Coreference resolution for spoken French

memory cost of this approach, K. Lee et al. (2017) resort to a variety of aggressive pruning at
every step such as:

1. Truncating documents (to a maximum of 50 sentences)

2. Restricting the maximum span width and number of spans per word

3. Restricting the number of antecedent candidates

2 is particularly problematic as it is done *during training*, with the discarded spans contributing nothing to learning at the current step, and thus slowing down the training process. Furthermore, it implies that the training process must use whole documents as batches, preventing the use of common training techniques such as mini-batching or sample shuffling, which results in a greater reliance on the fine-tuning of the training hyperparameters.

At the root of these issues is the lack of an explicit mention detection step in E2E, which would
single-handedly:

- Reduce the cost of antecedent-finding

- At inference time, by considerably reducing the number of alternatives to consider

- At training time, by removing the reliance on heavy pruning and truncations

- Makes the system more modular, which would help for both downstream applications and evaluation

- Allows working with singleton entities

while making the whole architecture easier to implement.

Therefore, in order to develop a system for end-to-end coreference resolution for oral French,
our choice is to design a classical two-step pipeline for coreference resolution using and adapting
the neural building blocks and the span-based paradigm of K. Lee et al. (2017).

5.2 The DeCOFre system

5.2.1 General architecture

In its end-to-end inference mode, given a raw text seen as a sequence of tokens $T = (w_1, ..., w_n)$,
DeCOFre works according to fig. 5.1, which is essentially a combination of mention detec-
tion by span classification and antecedent scoring with greedy document-order decoding (see
section 4.2).

At the end of this process, the (M, A) graph has a forest structure, whose connected components
(viz. trees) are the inferred coreference chains.

5.2.2 Representing text spans

The core of DeCOFre is the *g* span representation function, implemented as a neural network
$N_g$. It is similar in its main architecture to the *g* representation from K. Lee et al. (2017), made
<table><caption>Figure 5.1: DeCOFre inference mode operating process</caption><tr><td>1</td><td>Let M = Ø</td><td>the set of the currently detected mentions</td></tr><tr><td>2</td><td>Let A = Ø</td><td>the set of the currently detected coreference links</td></tr><tr><td>3</td><td><b>For</b> i from 1 to n</td><td></td></tr><tr><td>4</td><td><b>For</b> j from i to n</td><td></td></tr><tr><td>5</td><td>Let m = (w<sub>i</sub>, ..., w<sub>j</sub>)</td><td>the (i, j) span</td></tr><tr><td>6</td><td>Compute g<sub>m</sub> = g(m, ψ(m))</td><td>its representation</td></tr><tr><td>7</td><td>Determine d(g<sub>m</sub>)</td><td>its class</td></tr><tr><td>8</td><td><b>If</b> d(g<sub>m</sub>) is a mention class</td><td></td></tr><tr><td>9</td><td>Compute s<sub>n</sub>(g<sub>m</sub>)</td><td>the confidence that m is discourse-new</td></tr><tr><td>10</td><td><b>For</b> c in M</td><td></td></tr><tr><td>11</td><td>Let a(m, c)</td><td>= the confidence that c is an antecedent of m</td></tr><tr><td>12</td><td>s<sub>a</sub>(g<sub>m</sub>, g<sub>c</sub>, φ(m, c))</td><td></td></tr><tr><td>13</td><td>Let c = arg max<sub>x∈M</sub> a(m, x)</td><td>the most likely antecedent</td></tr><tr><td>14</td><td><b>If</b> a(m, c) > s<sub>n</sub>(g<sub>m</sub>)</td><td></td></tr><tr><td>15</td><td>Add (m, c) to A</td><td></td></tr><tr><td>16</td><td>Add m to M</td><td></td></tr></table>

ψ and φ are additional features functions.

more general by removing the dependency on *a priori* sentence segmentations of documents and making the influence of the immediate context of text spans more direct. It also includes some improvements inspired by other recent works on neural architectures for natural language processing tasks.

## Input representations

As mentioned earlier, we consider documents as sequences of words (or tokens). In order to provide our neural architecture with efficiently usable representation of these, we use static *word embeddings* (Bengio et al. 2003). For this, we build a fixed dictionary mapping the words occurring in the considered training set, excluding hapax, which make up about 40 % of the total vocabulary — consistently with Zipf’s law (Zipf 1949). This has the double advantage of avoiding the addition of hard to train representations to the model and making it more robust to out-of-vocabulary words.

Formally, we note $e_w(w_i)$ the word vector corresponding to $w_i$.

However, there is information in inputs that is not easily accessible from word embeddings. Even in the case of perfectly standard language, typos and out-of-vocabulary words are all lost if we only rely on word embeddings. This issue is particularly salient for transcriptions of spontaneous speech, where incomplete, garbled and rare words are frequent, as in example 27.
<figure><img src="image_5.png" /><figcaption>Figure 5.2: Intrinsic words representations in DeCOFre</figcaption></figure>

(27) bah j'aime euh j'aime bien les disons les euh le ciné- le cinéma actue l euh le cinéma euh euh un petit symbolique

(well I like uh I rather like the say the uh fil- contemporary fil- uh films that are uh uh a small metaphorical)
(Muzerelle et al. 2014)

To address this, following Ling et al. (2015) we also use character-level representations of words via the $r_c$, function implemented as a recurrent neural network — in our case, as in dinarelli2019HybridNeuralNetworks; Dinarelli and Grobol (2018), a bidirectional GRU layer (Cho et al. 2014), which is theoretically less powerful than the LSTM layers used in some other works (Weiss et al. 2018) but seems sufficiently efficient in this case as poor man's morphological analyser.

Formally, a word $w_i$ made up of the characters $c_1, ..., c_n$, is represented at the character level by a vector $r_c(w)$

$$ 
\begin{gathered}
\mathbf{e}_c = (e_c(c_1), ..., e_c(c_n)) \\
\overrightarrow{\mathbf{h}}_c = \overrightarrow{\text{GRU}_c(\mathbf{e}_c)} \\
\overleftarrow{\mathbf{h}}_c = \overleftarrow{\text{GRU}_c(\mathbf{e}_c)} \\
r_c(w) = \text{FFNN}_c(\overrightarrow{\mathbf{h}}_c^n, \overleftarrow{\mathbf{h}}_c^1)
\end{gathered}
\qquad [5.1]
$$

Where $e_c(c_i)$ is a character embedding corresponding to $c_i$, FFNN_c is a feedforward layer and $\overrightarrow{\text{GRU}_c}$ (resp. $\overleftarrow{\text{GRU}_c}$) is the forward (resp. backward) direction of the bidirectional GRU layer.
As in Ballesteros et al. (2015) and Plank et al. (2016), where the best results for dependency parsing and POS-tagging respectively were obtained by using both word- and character-level information, we derive from this our final intrinsic word representations by concatenating word embeddings and character-level representations

$$e(w_i) = [e_w(w_i), r_c(w_i)] \qquad [5.2]$$

Contextual words representations

<figure><img src="image_6.png" /><figcaption>Figure 5.3: Contextual words representations in DeCOFre</figcaption></figure>

Since our objective is to build a span representation module, we could stop at intrinsic words representations. After all, as noted by Shen et al. (2018), simple pooling techniques work at least as well as more complex recurrent or convolutional architectures for a variety of semantic task across several datasets.<fnref n="2" /> For coreference specifically, Nitoń et al. (2018) shows that state-of-the-art results can be obtained for Polish in the gold mention setting with a simple average pooling mechanism on word embeddings (but using additional knowledge in the form of pair features).

However, there are reasons to think that what essentially amounts to a bag-of-words approach might be too limiting of the expressiveness of a span representation module. Indeed, research on distributed words and segment representation in the last years tend to show that representations able to take word order and word contexts into account are vastly more performant than the alternative for most NLP tasks (Devlin et al. 2018; Peters et al. 2018) including coreference resolution. (K. Lee et al. (2018), see also section 6.2.2). This is not to say that these findings are a definitive optimal solution to the problem of representing human languages in neural networks: even in more sophisticated—and indeed state-of-the-art for most natural language understanding tasks<fnref n="3" />—architectures that take word order into account, recent results (McCoy et al. 2019; Niven and Kao 2019) suggest that neural network systems might fall into simple heuristics that are too limited for reliable coreference resolution. That said, it seems safe to

<footnote n="2">²Even if these findings were limited to English and their applicability to other languages is not clear.</footnote>

<footnote n="3">³Though here, too, the reasons behind these results are still not completely clear.</footnote>
Coreference resolution for spoken French

assume that these representations should be more suitable for our tasks than order-agnostic
representations based purely on pooling word-level representations.

Considering this, we chose to imbue our word representations with contextual information, by augmenting them with the hidden states of a recurrent neural layer applied to each span, while taking into account its left and right contexts. In our baseline, this layer is a Bi-LSTM (Graves and Schmidhuber 2005; Schuster and Paliwal 1997), chosen for its superior computational capabilities (Weiss et al. 2018) over GRU.

Another possible choice would have been an attention-based architecture, with the most serious candidate being the Transformer (Vaswani et al. 2017), but we decided against it. This choice is motivated by their novelty and lack of applications beyond machine translation at the beginning of this work, while the efficiency of the LSTM units was supported by more empirical knowledge.<fnref n="4" /> Since then, other works suggest that attention-based sentence modelling architectures for span representation could be performant for coreference resolution (Joshi et al. 2019b; Webster et al. 2018) and one work in particular (Joshi et al. 2019a) that they could be far more efficient than previous approach, although in this case, as with most of the recent achievements in the wake of (Devlin et al. 2018), the relative contributions of architectures, data, training and model size are yet to be properly disentangled (Rogers 2019).

Specifically, given a text T = (w₁, ..., wₙ), to compute the representation of span (wᵢ, ..., wⱼ), we
truncate T to a fixed context window clipped at utterance boundariesS, formally

$$
S = (w_{\max(i-ctx_r, 1)}, \dots, w_i, \dots, w_j, \dots, w_{\min(j+ctx_r, n)}) \quad [5.3]
$$

where the positive integers ctx_l and ctx_r are hyperparameters of the model.

We then apply a Bi-LSTM layer on the context-free words representations **e** = (e(**w**))<sub>**w**∈S</sub> and
keep its hidden states **h** = (h<sub>k</sub>)<sub>k∈S</sub> computed as

$$
\begin{align*}
\overrightarrow{\mathbf{h}} &= \overrightarrow{\text{LSTM}}_{w}(\mathbf{e}) \\
\overleftarrow{\mathbf{h}} &= \overleftarrow{\text{LSTM}}_{w}(\mathbf{e}) \tag{5.4} \\
\mathbf{h} &= ([\overrightarrow{h_k}, \overleftarrow{h_k}])_{k}
\end{align*}
$$

where $\overrightarrow{\text{LSTM}}_w$ (resp. $\overleftarrow{\text{LSTM}}_w$) is the forward (resp. backward) direction of the Bi-LSTM layer.

Finally, to coax our model into learning better context-free representations instead of relying entirely on the recurrent layer, we add a skip connection around it by concatenating **h** with **e**

Our final contextual representation for a word **w**<sub>k</sub> is then

$$
t_k = [h_k, e(w_k)] \tag{5.5}
$$

Span embeddings

Span embeddings are derived from the contextual words representations in three different ways

<footnote n="4">⁴An inkling connection between these two families of architectures can be found in Levy et al. (2018), which could explain why both work so well on similar tasks</footnote>
<figure><img src="image_7.png" /><figcaption>Figure 5.4: Boundaries and context embeddings in DeCOFre</figcaption></figure>

**Boundaries** The simplest part of the span embeddings consists of the representations of the boundaries of the span, that is, the representations of its first and last words. This can be seen as a form of (gated) recurrent segment representation (Cho et al. 2014; Sutskever et al. 2014; Vinyals et al. 2015) generalized to the bidirectional case by considering the hidden states of the forward and directions for both the first and the last word of the segment instead of using only a single boundary and a single direction. This is mainly motivated by our objective of getting the mentions with their exact boundaries rather than a more relaxed target.

Consistently with K. Lee et al. (2017) we derive this representation from the concatenation of the representations of the boundary tokens. Formally, this is $b_{i,j} = [t_i, t_j]$.

We decided against the earlier LSTM-minus approach introduced by W. Wang and Chang (2016) and used by Cross and Huang (2016) and Stern et al. (2017), which yields representations of lower dimensions but would impose a much more constrained structure to the LSTM hidden states *and* the intrinsic word representations in our architecture.

<figure><img src="image_8.png" /><figcaption>Figure 5.5: Soft-head self-attention in DeCOFre</figcaption></figure>
**Soft-head** The second part of the span embeddings is a structured self-attentive embedding (Lin et al. 2017) of the span. Its use on spans rather than full sentences (as in the original from Lin et al. (2017)) was motivated by K. Lee et al. (2017) for its similarity to the syntactic head features used by previous systems, to which it provides a substitute in a context where syntactic parsing is either unavailable or unreliable.

Formally, we represent the span $(w_k)_{i \le k \le j}$ by a weighted average $s_{i,j}$ of the contextual representations $(t_k)_{i \le k \le j}$, where the weights $(\alpha_k)_{i \le k \le j}$ are themselves computed from $t_k$, that is

$$ (\alpha_k) = \underset{k}{\operatorname{softmax}}(\operatorname{FFNN}_{\alpha}(t_k)) \\ s_{i,j} = \sum_k \alpha_k t_k \qquad [5.6] $$

According to K. Lee et al. (2017) the maximal $\alpha_k$ is usually given to the syntactic head of a span or to its most semantically significant word in cases where the head does not bear a lot of meaning (as is the case for coordinations for example). In our case, since the syntax of spoken French tends to be more irregular,<fnref n="5" /> it is interesting to be able to attend to several words. In principle, this could be learned by FFNN$_\alpha$ in this formulation, but this is made unlikely by the high concentrating effect of the softmax normalization, which by design tends to concentrate the overwhelming majority of the weights on the maximal coordinate. To allow more expressivity, we follow the example of Lin et al. (2017) and concatenate several such self-attentive embeddings to obtain s. Formally,

$$ (\alpha_k^1) = \underset{k}{\operatorname{softmax}}(\operatorname{FFNN}_{\alpha}^1(t_k)) \\ \vdots \\ (\alpha_k^\tau) = \underset{k}{\operatorname{softmax}}(\operatorname{FFNN}_{\alpha}^\tau(t_k)) \\ s_{i,j} = \left[ \sum_k \alpha_k^1 t_k, \dots, \sum_k \alpha_k^\tau t_k \right] \qquad [5.7] $$

where the number of heads $\tau$ is an hyperparameter of the model.

**Context** Finally, we also use representations of the contexts of the spans, in the form of the final states of $\overleftarrow{\text{LSTM}}_w$ and $\overleftarrow{\text{LSTM}}_w$, that is $ct_{i,k} = [\overleftarrow{h_{n+p}}, \overleftarrow{h_{-l}}]$. Indeed, while information on the span's context outside $(w_k)_{i \le k \le j}$ might flow to $b_{i \le k \le j}$ and $s_{i \le k \le j}$ via the recurrent parts of $(t_k)_{i \le k \le j}$, it is more straightforward to provide it directly to the subsequent parts of our architecture, especially since in our implementation it comes with no additional computational cost.

The span embedding $\hat{g}_{i,j}$ is computed from the concatenation of all of these parts by a feedforward layer, that is

$$ \hat{g}_{i,j} = \text{FFNN}_o([b_{i,j}, s_{i,j}, ct_{i,j}]) \qquad [5.8] $$

**Final representation** To $\hat{g}$, which is entirely derived from the surface form of the spans without any supervision other than the architecture, we add some explicit features by concatenating some non-pretrained low dimensional feature embeddings to a vector $\psi$. For the

<footnote n="5">⁵Syntactic constructions are usually shallower than in written languages, but can be much wider and tends to use less explicit compositions, such as parataxis. (Lacheret et al. 2014)</footnote>
feature-poor model presented in this chapter, $\psi$ simply consists of the length of the span (see section 6.1).

The final representation of a span $g_m$ is then the concatenation of $\hat{g}(m)$ and the feature embedding $\psi(m)$, projected via a linear function $\pi_s$ to keep a constant dimension for the subsequent layers independently of the number of features used and of their embedding dimensions.

$$g_m = \pi_s([\hat{g}(m), \psi(m)]) \qquad [5.9]$$

### 5.2.3 Mentions detection

Among the different mention detection strategies described in section 4.3, the resource-poor setting of DeCOFre ruled out the syntax-based methods (due to the lack of reliability of the available parsers), and made the label+transform method proposed by Soraluze et al. (2012, 2017) very inconvenient. Therefore, we were left with a choice of a transition-based method (à la B. Wang et al. (2018)), which while very promising requires a complex machinery to train, or our final choice: a span based classifier.

This last method has the advantage of being very easy to implement, as it simply uses a neural classifier — a relatively well-known model — on top of the span embeddings described in section 5.2.2 that we intended to use for the coreference resolution subtask in any case. Therefore, in DeCOFre, mention detection is formulated as a classification task on the set of all text spans.

The theoretical complexity of this formulation for a document of $n$ words is $O(n^2)$, but in our case, since mentions can't cross utterance boundaries, it is reduced in average to $O(np)$ where $p$ is the average sentence length. This can be sufficiently small in some contexts, where utterances lengths are limited.

However, since the utterances in ANCOR can have a length of several hundreds of words (up to 542) and mentions are most a few words long (less than 5% consist of more than 4 words and the longest mention is a mere 17 words long), we add a further restriction of considering only spans of length inferior to an integer $l_{max}$, bringing the number of spans to consider in the worst case to $O(nl_{max})$.

In our implementation, the classification function $d$ is a feedforward neural network that returns non-normalized class scores. In principle, this could be done by a simple binary classifier, but since ANCOR provides mention types information (see section 3.2.2), we chose instead to classify the spans between pronouns, noun phrases and non-mentions.

Formally, this is then

$$d(g_m) = \arg \max(\text{FFNN}_d(g_m)) \qquad [5.10]$$

### 5.2.4 Coreference resolution

As seen in section 4.2, there are several competing algorithms to partition a set of mentions in coreference chains. Among these, link-centric algorithms have the advantage of being the most lightweight, since they only require representing two type of objects: mentions and mention
Coreference resolution for spoken French

pairs themselves, without having to deal with the extra level of entities. In our case it is even
most interesting, since it allows us to use a single *text span* representation module for both
mention detection and coreference resolution.

Following the previous work of Désoyer et al. (2015a), our preliminary experiments used mention-pair classification models, but their disappointing results prompted us to switch to an antecedent-scoring model.

As noted in section 4.2, the antecedent-scoring model is a notable improvement over mention pair classifiers, both in terms of empirical performance and in intuitive fitness to the task at hand, where the existence of a coreference link between a pair of mention is highly dependent on the other mentions in the same context (Denis and Baldridge 2007a, 2008), a fact that is hard to capture for a classification model.<fnref n="6" />

In line with most of the previous works on antecedent ranking, at inference time, given an antecedent score distribution for a mention, we simply greedily select the highest-scoring candidate as antecedent. This has proven to be good enough for a working system in practice and is less demanding in resources than more sophisticated alternatives.

Antecedent scoring

The antecedent scoring module uses a slightly modified version of the coarse-to-fine second-order inference mechanism from K. Lee et al. (2018). Instead of using a single antecedent score, it uses tow in a scaffold structure: a ‘coarse’” score — determined by a function cheap to evaluate but with limited representation power — and a ‘fine’ score — more precise but more expensive to compute — used only for the antecedent candidates with the K highest coarse score, where K is an hyperparameter.

In our case, the coarse score $s_a^b(m, c)$ for a mention $m$ and an antecedent candidate $c$ is computed using a biased bilinear layer$^7$ as

$$s_a^b(m, c) = {}^t g_m W_b g_c + b_b \qquad [5.11]$$

where $W_b$ and $b_b$ are learned parameters.

The choice of a biased bilinear function for this layer was mainly motivated by its computational efficiency (this operation is meant to be as fast as possible, since it will be applied to a large number of antecedent candidates) relative to its expressiveness (formally equivalent bilinear attention layers have proven their efficiency for a variety of tasks).

Let $M'$ be the set of the K antecedent candidates with the highest coarse score. For the elements of $M'$, we compute the fine score $s_a^#(m, c)$ by applying a feedforward network to the representations

<footnote n="6">⁶On the other hand, using this approach, it is not clear how to predict the type of anaphoric link (as annotated in ANCOR). However, if e.g. Roesiger et al. (2018) shows promising results for this kind of predictions, this is still very much uncharted territory for coreference, where such granularity is still lacking in most annotated resources and evaluation metrics.</footnote>

<footnote n="7">⁷Not to be confused with the *biaffine* layer introduced by Dozat and Manning (2017) and used for coreference resolution by R. Zhang et al. (2018)</footnote>
of *m* and *c* and a feature vector $\phi(m, c)$ (see chapter 6 for details on these features). Formally

$$s_a^\#(m, c) = \text{FFNN}_a([g_m, g_a, \phi(m, c)]) \cdot 1_{c \in M'} \quad [5.12]$$

The total antecedent score for (*m*, *c*) is then the sum of the coarse and fine scores

$$s_a(m, c) = s_a^b(m, c) + s_a^\#(m, c) \quad [5.13]$$

which allows training both the coarse and the fine score for the top candidates.

## Discourse-new score

While $s_a(m, c)$ represents the confidence that a given mention *a* is an antecedent of *m*, the discourse-new score $s_n(m)$ represents the confidence that *m* is the first mention of its chain, often improperly (Recasens 2010, p 24) called *anaphoricity score*. Since it is computed independently of the antecedent candidates' distribution, it depends only on the form of *m*, and an ideal model could use it to evaluate how well *m* fits the description of discourse-new mentions — for instance assigning a low $s_n$ if *m* is a pronoun and a higher $s_n$ if it is a definite description.

In practice, Vieira and Poesio (2000) findings suggest that even relatively local<fnref n="8" /> syntax-based heuristics can be reasonably accurate for discourse-new detection and both shallow (Denis and Baldridge 2007b; Lassalle and Denis 2015) and deep (Wiseman et al. 2015) learning approaches were found to benefit from learning an explicit ‘anaphoricity’ score.

K. Lee et al. (2017) chose to fix this score to 0, in order to ‘[remove] a spurious degree of freedom’. This does not reduce the expressivity of the model, which simple has to learn to affect negative antecedent scores to all antecedent candidates for mentions that resemble discourse-new mentions. However, this puts all the burden of identifying the common intrinsic characteristics of discourse-new mentions on the $s_a(m, c)$ score, thus spending part of the expressiveness of the $s_a$ function for things that do not depend on *c*.

In addition to this, since singleton mentions are always discourse-new, and given the high ratio of singletons entities, discourse-new mention are by far the most common mentions in our data (62.55% in the training set for instance). Therefore, leaving the detection of these mentions solely to the antecedent scoring module would make the majority class heuristic (in this case, always assigning a negative antecedent score to all the candidates) very easy to fall in.

Given these points, we elected not to use a constant discourse-new score, and to train a separate scoring layer

$$s_n(m) = \text{FFNN}_n(g_m) \quad [5.14]$$

## Mention embeddings refining

The antecedent-scoring procedure described so far now is purely *local* (Denis and Baldridge (2007b, 2008) and McCallum and Wellner (2004), see section 4.2 for details), since coreference

<footnote n="8">⁸Uryupina (2003) and Poesio et al. (2005) show even more encouraging results, but use non-local features.</footnote>
Coreference resolution for spoken French

chains are built greedily, with no possibility of global supervision. As mentioned earlier, in practice, this heuristic works reasonably well,<fnref n="9" /> but it can be too simplistic in the general case.

In that regard, the mention refining procedure proposed by K. Lee et al. (2018) is as a step in the direction of global coreference resolution that has the advantage of being both relatively cheap (as opposed to more involved earlier methods such as ILP decoding) while still being reasonably efficient.<fnref n="10" /> It consists in modifying (refining) the initial representation of spans by combining them with the representations of their antecedent candidates, using the aforementioned coarse score $s_a^b$ as a weighting function.

Formally, the refined representation of *m* is given by

$$g'_{m} = R\left(g_{m}, \sum_{c \in M} s_{a}^{b}(m, c)g_{c}\right) \qquad [5.15]$$

where R is an elementwise multiplicative sigmoid gate (Cho et al. 2014; Hochreiter and Schmidhuber 1997) and

$$\begin{aligned} \lambda(x, y) &= \sigma(W_R[x, y] + \beta_R) \\ R(x, y) &= (1 - \lambda(x, y))x + \lambda(x, y)y \end{aligned} \qquad [5.16]$$

Beyond the obvious parallel with the gating mechanisms in gated recurrent layers, this is also reminiscent of the method used to combine heterogeneous representations used in Balazs and Matsuo (2019) and Miyamoto and Cho (2016) (to combine word- and character-level representations of tokens in these cases).

Equation 5.12 then becomes

$$s_a^\#(m, c) = \text{FFNN}_a([g'_m, g_a, \phi(m, c)]) \cdot 1_{c \in M'} \qquad [5.17]$$

Note that since we do not proceed at the document level, refining only makes sense for the representation of *m* and not for its antecedent candidates (as we do not have access to their antecedent candidates at that time). Consequently, it would not make a lot of sense to reiterate the refining procedure (K. Lee et al. (2018) report a maximum return on investment for two consecutive refinements), since only the representation of *m* could change between the refining rounds.

## 5.3 Training

The inference algorithm presented in fig. 5.1 process a whole document in a single pass, iteratively detecting mentions and finding their antecedents, thus solving the coreference resolution task in an end-to-end setting. As shown by K. Lee et al. (2017), this kind of end-to-end processing can also be applied to the task of training a coreference resolution system. However, end-to-end

<footnote n="9">⁹Possibly because the repartition of complexity of coreference relation detection is heavily skewed, with a large proportion of naturally occurring coreferences being easily detected with simple heuristics (see chapter 2).</footnote>

<footnote n="10">¹⁰Regrettably, their reported experiments did not include an evaluation of the benefits of this technique independently of the use of contextual embeddings — whose impressive gains overshadow their other interesting contributions — but its positive effect is still clear.</footnote>
training comes with a high cost, both in terms of computations and memory usage. In our case, the separation of end-to-end coreference resolution in two components gives us a natural way to train our system at a lower cost: we can train mention detection and antecedent scoring independently, while still sharing a common span representation module.

### 5.3.1 Optimization algorithm

While second-order methods (Martens 2016) and evolutionary strategies (Gaier and Ha 2019) have been suggested to be more relevant in some contexts, at the time of writing, the overwhelming majority of neural network optimizations use variants of the Stochastic Gradient Descend (SGD) algorithm instead (LeCun et al. 2012). As with other hyperparameters, choosing among these variants can be a daunting task, with no clear answer on which one is the best suited for a given architecture. Exhaustive comparison is also far too costly to consider, particularly since most of these variants require further hyperparameter tuning to reach their full potential. In this context, Adam (Kingma and J. Ba 2014) has the advantage of having been empirically shown to be efficient for many neural network architectures and natural language processing tasks (notable examples being Devlin et al. (2018) and K. Lee et al. (2017)). It also requires very little tuning of hyperparameters (except for a learning rate schedule, see section 5.3.2). Recent advances (Dozat 2016; L. Liu et al. 2019; Reddi et al. 2018) have suggested that further modifications to Adam might make it even more efficient, but these have not yet received enough third-party attention to consider them clear successors.

Considering these points, for both mention detection and antecedent scoring, we optimize the trainable parameters of the architectures described in section 5.2.1 using the standard version of Adam from Kingma and J. Ba (2014), with a single modification: the weights of the embedding layers are only updated when their current gradient is non-null, instead of drifting with their current momentum (the so-called ‘sparse’ or ‘lazy’ version Adam used in PyTorch for sparse layers). Loss gradients with respect to the weights are computed by backpropagation (LeCun 1988; Rumelhart et al. 1986) using the automatic differentiation facilities of the Pytorch library (Paszke et al. 2017), see sections 5.3.4 and 5.3.5 for details on the loss functions considered.

### 5.3.2 Learning rate

Even when using adaptive optimization algorithms such as Adam — where the learning rate is dynamically adapted to the topology of the parameters search space —, designed to relieve practitioners from the hassle of fine-tuning yet another hyperparameter, the base learning rate still has a significant influence on the training process.<fnref n="11" /> Recent works (Loshchilov and Hutter 2019) go further and recommend the use of Adam with an explicit learning rate schedule (see also Devlin et al. (2018) for a notorious example). L. Liu et al. (2019) brings some theoretical justifications for this, in particular for the ubiquitous warmup strategy.

Given the lack of definitive answers regarding the best way to choose a learning rate, we experimented with some popular choices. In table 5.8, we compare the results of learning mention detection with a constant base learning rate for a few common values and with a

<footnote n="11">¹¹See for instance Karpathy and Johnson (2019).</footnote>
common schedule: a geometric step decay with warmup. These results are of course not to
be taken as a claim of exhaustivity, and further theoretical work in this area is still very much
needed.

5.3.3 Regularization

Among machine learning systems, neural networks occupy a special space with regard to the problem of overfitting: on one hand, the ratios between their number of parameters and the size of the datasets they are usually trained on are usually much higher than those of other systems, but on the other hand this does not seem to harm their generalization capacities.<fnref n="12" /> In fact, it seems that on the contrary, beyond a certain number of parameters, the classical U-shaped bias/variance risk curve starts decreasing again and adding more parameters yields architectures with more generalization power (Belkin et al. 2018). This has also been observed recently in experiments with Transformer-based language models (Devlin et al. 2018; Y. Liu et al. 2019; Radford et al. 2019; Shoeybi et al. 2019). Given this, the appropriateness of regularization for neural network architectures might be questioned. Indeed, findings from C. Zhang et al. (2016) suggest that:

*Explicit regularization may improve generalization performance, but is neither necessary nor by itself sufficient for controlling generalization error.* (C. Zhang et al. 2016)

However, it may still improve generalization performance and several such methods have been experimentally shown to not only do this, but also to stabilize and to speed up the learning process (see for instance J. L. Ba et al. (2016) and Ioffe and Szegedy (2015)). Given this, it seems sensible to at least try to apply some regularization to our base architecture, even if we could not afford an exhaustive exploration of the whole landscape of regularization strategies. We experiment with three well-known regularization techniques: dropout, weight decay and layer normalization.

Dropout

Dropout, first proposed by Hinton et al. (2012) is a regularization method that reduces the tendency of neural networks to overfit during training by randomly zeroing a certain amount of coordinates from the outputs of the input and hidden layers. Intuitively, this makes it harder for a system to remember a strict mapping from the training samples to the corresponding gold output (which would lead to very poor generalization) and forces it to learn more general structures. It can also be seen as an approximation of model ensemble averaging (Warde-Farley et al. 2013).

Several variations on dropout have been proposed, most notably DropConnect (Wan et al. 2013) which zeroes weights instead of activations ; Gaussian (Srivastava et al. 2014) and variational (Kingma et al. 2015) dropouts, which fuzz the activations with random noise instead of zeroing them ; recurrent dropout (Gal and Ghahramani 2015) and Zoneout (Krueger et al. 2016), where a single dropout mask is applied to each time step of a recurrent layer ; and curriculum dropout (Morerio et al. 2017), where the dropout rate *p* is gradually increased during training to accelerate

<footnote n="12">¹²For a comprehensive review on this, see Weng (2019)</footnote>
the learning process in early epochs while still preventing overfitting in the long run. However, while all of these variations reportedly improve the performances of some neural network architectures on specific benchmarks, there is yet no consensus on their benefits in general and their support in off-the-shelf neural network frameworks is still lacking.

For this reason, we limit our study to the original Bernoulli dropout as formulated by Srivastava et al. (2014). In this formulation, dropout is only applied during training and the activations are also scaled up by a factor of $\frac{1}{1-p}$ in order to prevent a spurious boost of the mean of their distribution at test time, which could hamper the operation in the next layer.

We apply dropout on the inputs of all the neural layers in our architecture, with the rates reported in section 5.4.1.

## Weight decay

Introduced by Hinton (1987) and formalized by Krogh and Hertz (1991), weight decay, like dropout, attempts to reduce overfitting by reducing the long-term influence of the specific noise of the training samples on the network weights through the introduction of a regularization term at each weight update. In the formulation of Hanson and Pratt (1988), shown by Loshchilov and Hutter (2019) to be more suitable in the general case than Krogh and Hertz (1991)'s,

$$ \theta_{t+1} = (1 - \lambda)\theta_t - \alpha_t \nabla f_t(\theta_t) \quad [5.18] $$

where $\theta_t$, $\alpha_t$ and $\nabla f_t(\theta_t)$ are respectively the weight vector, the learning rate and the error gradient at training step $t$ and $\lambda \in [0, 1]$ is the weight decay rate.

In table 5.9 we report the results of using weight decay with different decay rates, using the AdamW variant of Adam (Loshchilov and Hutter 2019), which gives a true weight decay instead of the L² regularization common in older implementations.

## Layer normalization

Layer normalization, first proposed J. L. Ba et al. (2016), is an improvement over older techniques such as batch normalization (Ioffe and Szegedy 2015) and weight normalization (Salimans and Kingma 2016). All of these techniques consist in normalizing the output of a neural layer, by shifting and scaling the resulting activations to make their empirical mean and standard deviation constant across the training dataset. Formally, given the output $x \in \mathbb{R}^d$ of a neural layer, the corresponding layer-normalised output $x'$ is

$$ x' = \gamma \frac{x - \mu}{\sqrt{\sigma^2 + \epsilon}} + \beta \quad [5.19] $$

where $\mu \in \mathbb{R}$ and $\sigma \in \mathbb{R}^+$ are respectively the empirical mean and the standard deviation of $x$, seen as an i.i.d. sample of some distribution, $\epsilon \in \mathbb{R}^+$ is a small positive number (typically $1 \times 10^{-5}$) meant for numerical stability and $\gamma \in \mathbb{R}^d$ and $\beta \in \mathbb{R}^d$ are learnable elementwise affine transform parameters.

This normalization aims at reducing internal covariate shift (Ioffe and Szegedy 2015). It is motivated by the observation that while training deep neural networks, the last layers are
significantly harder to train due to the fact that the distribution of their inputs (the outputs of previous layers) tend to vary significantly during training. This bears some similarities with the phenomenon known as covariate shift (Shimodaira 2000) where variations in the input distribution of a machine-learning system hampers the efficiency of the learning process.

The real impact of activation normalization on internal covariate shift is currently challenged. For instance, Santurkar et al. (2018) dispute both the claims that internal covariate shift hampers training and that batch normalization reduces it. The authors instead ascribe the improvements observed when using in batch normalization to its smoothing effect on the loss function as observed by the normalized layer.

However, empirical results tend to show that these normalization techniques have a positive impact on both learning speed and validation performance even for tasks and architectures not studied in the original works where these techniques were first introduced. Specifically, layer normalization, for its ease of use at both training and test time is at the core of the Transformer model (Vaswani et al. 2017) where it is used in the add-and-norm residual connection as a mechanism to regulate gradient explosion (Chen et al. 2018).

In our case, layer normalization is applied on the output of sequence to vector encoders, ie after the FFNN<sub>c</sub>, FFNN<sub>o</sub> and π<sub>s</sub> layers.

### 5.3.4 Mention detection

Since we cast mention detection as a classification problem, training the corresponding architecture is relatively straightforward, as neural classification is now a very common problem. Accordingly, we train the parameters of N<sub>g</sub> and d end-to-end, by minimizing the negative log-likelihood of the correct class in the softmax-normalized score distribution given by d (or, equivalently, the cross-entropy of this score distribution with respect to the corresponding punctual distribution). Formally, given the class score distribution v = FFNN<sub>d</sub>(g<sub>e</sub>) ∈ R<sup>c</sup> for a span e for which the correct class is y ∈ [0, c], this loss is

$$ \mathcal{L}_d(v, y) = -\log(\mathrm{softmax}(v)_y) \qquad [5.20] $$

which is simply averaged over individual samples in order to get a batch-level loss.

### Overcoming class imbalance

The biggest obstacle in considering mention detection as a classification task is the major<sup>13</sup> imbalance between mention and non-mention spans, making the majority class heuristic far more tempting for the system than in more balanced classification tasks. Moreover, even with the steps taken to reduce the complexity described in section 5.2.3, the size of the training set resulting from considering all spans is still considerable.

A natural way to solve both issues is to reduce the size of the training dataset by removing a certain number of non-mention spans from it, a procedure known as undersampling (Haixiang et al. 2017). According to Buda et al. (2018), undersampling does not necessarily have a positive

<footnote n="13">¹³Only about 1.81 % of all considered spans are mentions in the development dataset for instance.</footnote>
effect on learning (although the authors only considered balanced undersampling, which brings the minority and majority class to the same ratio) but it does not usually have a negative effect. Furthermore, the speed gain in our case was critical. We report the results of experimenting with various sampling ratios in table 5.10.

### 5.3.5 Antecedent scoring

Following K. Lee et al. (2017, 2018)<fnref n="14" />, we train our antecedent scoring module by minimizing the sum of the log-likelihood of all the correct antecedents for each mention.

Formally, given a mention *m* with antecedent candidates *c*₁, ..., *c*ₖ, *Y* = {*i* ∈ ₪₁, *n*} | *c*ᵢ is an antecedent of *m*} and *A* = [*s*₁(*m*), *s*₂(*m*, *c*₁), ..., *s*ₖ(*m*, *c*ₖ)], the per-sample loss is

$$ \mathcal{L}_a(A, Y) = - \log \sum_{y \in Y} \mathrm{softmax}(A)_y \quad [5.21] $$

as with mention detection, the batch-level loss is simply the (unweighted) average of all per-sample loss.

Due to our *local* resolution of coreference, we could not directly use the differentiable relaxed coreference metrics introduced by P. Le and Titov (2017) or the reinforcement learning approach of Clark and Manning (2016a) since they require document-level processing.

As noted in chapter 4, some works on coreference resolution using machine learning (and all of those using *neural* architectures) train their coreference resolution modules using predicted mentions. In our setting, this poses a number of issues, both from a technical and a methodological standpoint.

Firstly, if we did train our system using system mentions, we would have to choose a target antecedent distribution for spurious mentions resulting from precision errors. It is clear that this target distribution could not include any mention, leaving us with the choice of marking them as discourse-new, which they are not (contrarily to the no-singleton settings where spurious mentions can just be categorized as singletons to be discarded), leading our system to learn wrong outputs. Another option would be to consider only gold mentions but to allow system mention as antecedent candidates. Still, this would considerably complicate the training process and it is not clear that it would meaningfully improve the learning procedure.

Secondly, the best way to deal with recall errors — gold mentions missing in the system output — is also not clear: including them would not be consistent with the proposed ways to deal with precision errors, but ignoring them would remove precious from the already limited training dataset.

Lastly, even if we did devise a relevant scheme for introducing spurious mentions in the antecedent-finding learning phase, it would make the intrinsic performances of this module far harder to analyse, as they would in this case be dependent on the performances of the mention detection module.

<footnote n="14">¹⁴For this, we chose to trust their findings that using e.g. the previously popular weighted slack-rescaled margin loss (Wiseman et al. 2016, 2015) did not help training. This is also partly motivated by the need to tune yet more hyperparameters (the error cost weights) to use this family of loss.</footnote>
For all these reasons, following the example of Bengtson and Roth (2008), Culotta et al. (2007) and Rahman and Ng (2009) among others and unlike K. Lee et al. (2017) <fnref n="15" />, we train this module exclusively on gold mentions <fnref n="16" />.

Finally, in order to reduce the complexity of the training procedure, we only consider antecedent
candidates up to the 100 previous mentions, with the effect that coreference chains with mentions
further apart than this limit are effectively cut <fnref n="17" />. This can seem to be limiting, but we argue
that this simplification is reasonable. From a purely technical point of view, this concerns less
than 7% of all mentions while vastly simplifying our operations, given that the documents
we consider can include several thousands of mentions. This is also not inherently limiting at
inference time, since it can be easily disabled.

Moreover, from a theoretical point of view, in the discourse model framework that we adopt,
it is not clear that the relation between mentions that are very far apart are coreferent *stricto
sensu*, as it could be argued that they do not in fact refer to a single persistent entity in the
discourse space, but rather that the identity relation holds between distinct entities existing
in disjunct segments of the discourse (see chapter 2). That said, these matters fall well out of
the scope of the present work, and in the current state of our conventions for the evaluation
of coreference resolution, limiting the scope of antecedent-finding should be seen as a purely
technical simplification, which is in any case *not* reflected in the experimental results that we
report are of course given against the actual gold test clusters.

5.3.6 Spans representations

The span representation module N<sub>g</sub> is the centrepiece of our whole architecture and condition
most of its performances on both subtasks. As such, it is of critical importance to ensure that
the representations it provides to the rest of our system are as good as possible for these tasks.

It is a well-known fact that pre-training input representations on auxiliary, possibly simpler,
tasks is highly beneficial for subsequent, more involved, tasks (Devlin et al. (2018), Mikolov
et al. (2013) and Peters et al. (2018) are possibly the most notorious examples of such pretraining
in recent years, while Ruder (2019) provides a recent and comprehensive review). Since our
architecture already consists of two tasks and one of them — mention detection — is intuitively
simpler, it seemed natural to try to reuse the parameters learned in mention detection to initialize
N<sub>g</sub> for antecedent scoring.

However, while this *sequential* transfer learning procedure can be beneficial, it can also fail due
to the tendency of neural networks to *catastrophically forget* previous training (French 1999) in
some contexts.

<footnote n="15">¹⁵Although this characterization is somewhat misleading, since they do not actually train on mentions, gold or system, but on *all spans*.</footnote>

<footnote n="16">¹⁶Note, however, that this only concerns mentions boundaries, as we do not use any other gold features. In particular, unlike Bengtson and Roth (2008), we do not use gold mention types.</footnote>

<footnote n="17">¹⁷Note that this is fundamentally different from the hard antecedent candidates limit set by K. Lee et al. (2017) and lifted by K. Lee et al. (2018), as theirs was a mean to work with the high number of candidate *spans*. In their case, the reduction of number of candidate *mentions* resulted from the truncation of documents to 50 sentences.</footnote>
It is still not yet clear what kind of pretraining is beneficial to subsequent tasks and what
kind simply amounts to another random initialization, or worse, yields input representations
that are so unsuited for the subsequent task that it makes training the corresponding neural
network harder than random initialization. Therefore, to assess the applicability of sequential
transfer learning, we experiment with both the mention detection → antecedent scoring order
and the reverse antecedent scoring → mention detection. This allows us to report results for
independent training (by taking the result of the first step in both cases) and with both kinds of
pretraining (from the result of the second step)

In addition to sequential transfer learning, we also experiment with a variant of the hard
parameter sharing multi-task learning procedure described by Ruder (2017) and Søgaard and
Goldberg (2016) and used with success by Sanh et al. (2018) for several NLP tasks including
coreference detection (in the CoNLL-2012 shared task setting). It consists in training two or
more models that share some parameters in parallel by randomly sampling a model at each
training step, train on it on a single batch of the corresponding data and repeating this until
convergence. This is an easy way of training models that share some of their parts on different
tasks with almost no implementation costs and without requiring multi-task objectives, which
are notoriously hard to design properly.

In our case, the models are the coreference resolution and the antecedent scoring one and they
share the span embedding module. In this variant, instead of randomly sampling between the
models, we simply cycle through them, training for N steps on mention detection, then M steps
on antecedent scoring, evaluating each model on the development dataset after each epoch. As
in the sequential case, we stop training as soon as one of the models development performances
did not improve in the last *p* epochs.

5.4 Experiments

To assess the adequacy of our architecture, we train and evaluate it on ANCOR, using the
train/development/test split from Grobol (2019). The models for both mention detection and
antecedent scoring are trained end-to-end as described in section 5.3 until their performance on
the development data cease to improve, with an arbitrary patience of 5 epochs, which seems
to be a good compromise between giving the procedure unrestricted freedom to explore the
parameters space and limiting our computation budget. The performances that we report are
those of the dev-best model, both on development and test data.

The mention detection metrics are the usual precision, recall and F₁ score (see section 2.4.1).
For coreference, we report the CoNLL coreference metrics MUC, B³, CEAFₑ and MELA (*alias*
CoNLL), and BLANC (see section 2.4.2) on the system coreference clusters including singlerons.
These metrics are computed using scorch, as described in section 2.4.3. In particular, corpus-
level scores are computed by micro-averaging for all metrics using the number of mentions per
document as weights instead of using ad-hoc averaging schemes for MUC and BLANC, in order
to reach a better consistency across metrics and prevent an overly large influence of the largest
documents — which is particularly relevant for ANCOR, given the heterogeneity of the sizes of
its documents.
## 5.4.1 Hyperparameters

Since our goal was to assess the viability of this architecture for the particular case of spoken French more than pushing the boundaries of quantitative evaluations,<fnref n="18" /> the hyperparameters of both our architecture and its training setting were not extensively or systematically fine-tuned (however, see section 5.4.1 for experimental results on some hyperparameters), nor did we experiment with model ensembling or boosting methods.

### Input embeddings

Unless otherwise specified, all input embeddings are initialized randomly by sampling from the $N(0, 1)$ distribution and tuned during all the training phase.

The word embeddings $e_w$ are initialized using the FastText embeddings pretrained by (Grave et al. 2018) on a mixture of Wikipedia and Common Crawl data. The words for which no vector exists in these pretrained embeddings are kept (mostly to avoid discarding oral-specific tokens) with their default initializations. This default initialization is also used for the special tokens `<start>`, `<end>` (marking the beginnings and ends of utterances) and `<unk>` (for out-of-vocabulary words).

To take into account the casing information in the least disruptive way, when digitizing a word, we first look for an embedding with the same casing and fallback successively to looking for an embedding for its lowercasing and using the `<unk>` embedding. This is still not completely ideal, as it generate for instance two distinct embeddings for a word if it occurs more than twice at the beginning of a sentence, but it also let us keep distinct embeddings for proper noun homographs (such as *paris* vs. *Paris*) while requiring no systematic lexicon.

The character lexicon consists of all the characters encountered in the training dataset, with the addition of an `<unk>` character. The special tokens `<start>` are `<end>` treated as consisting of the single character `<no>`.

The dimensions and vocabulary sizes of all the input embeddings described in this chapter are reported in table 5.1, including those for the span and pair features (see )chapter 6).

<table><caption>Table 5.1: Embedding dimensions in DeCOFre baseline</caption><thead><tr><td>Feature</td><td>Symbol</td><td>Vocabulary size</td><td>Dimension</td></tr></thead><tbody><tr><td>Word</td><td>e<sub>w</sub></td><td>10 191</td><td>300</td></tr><tr><td>Character</td><td>e<sub>c</sub></td><td>83</td><td>50</td></tr><tr><td>Span length</td><td>φ</td><td>10</td><td>20</td></tr><tr><td>Words distance</td><td>ψ</td><td>10</td><td>20</td></tr><tr><td>Utterances distance</td><td>ψ</td><td>10</td><td>20</td></tr><tr><td>Mentions distance</td><td>ψ</td><td>10</td><td>20</td></tr><tr><td>Speaker agreement</td><td>ψ</td><td>2</td><td>2</td></tr><tr><td>Overlap</td><td>ψ</td><td>2</td><td>2</td></tr></tbody></table>

<footnote n="18">¹⁸Especially since none exists that we can meaningfully compare to.</footnote>
## Size restrictions

As mentioned in section 5.2.3, we only consider spans of at most 25 words to reduce the time and material requirements. This value is chosen arbitrarily to be superior to the length of the longest mention in ANCOR (17 words). The span contexts considered are of size 10 on both sides.

For antecedent-finding, as mentioned in section 5.3.5, only the 100 previous mentions are considered for antecedent scoring, and only the K = 25 best-scoring antecedents are kept for fine-scoring.

## Neural layers

Table 5.2 details the parameters of the feedforward layers used in our architecture. They all use at most one hidden layer, as in our experiments, increasing their depth did not seem to improve their usefulness, while it significantly increased the training time. This is not very surprising, since the global architecture in itself is already considerably deep. Some of them exhibit unusually large input sizes to deal with the fact that their inputs are concatenations of representations from several sources.

Our default non-linearity is the leaky rectifier<fnref n="19" />

$$ \text{LReLU}: \begin{array}{r} \mathbb{R} \rightarrow \mathbb{R} \\ x \mapsto \text{LReLU}(x) = \max(0, x) + 0.01 \min(x, 0) \end{array} \qquad [5.22] $$

a slightly modified version of the rectifier non-linearity (Nair and Hinton 2010) that does not suffer from the “dying ReLU” problem and generally performs well for many natural language processing tasks (Eger et al. 2018).

<table><caption>Table 5.2: Feedforward networks parameters in DeCOFre baseline</caption><thead><tr><th>Layer</th><th>Layers/Dimensions</th><th>Nonlinearities</th><th>Dropout</th></tr></thead><tbody><tr><td>FFNN<sub>c</sub></td><td>(300, 50)</td><td>None (Layer Norm)</td><td>0.6</td></tr><tr><td>FFNN<sub>α</sub><sup>i</sup></td><td>(950, 300, 1)</td><td>LReLU, None</td><td>0.2, 0.2</td></tr><tr><td>FFNN<sub>o</sub></td><td>(4400, 500, 500)</td><td>LReLU, LReLU (Layer Norm)</td><td>0.6, 0.2</td></tr><tr><td>π<sub>s</sub></td><td>(520, 500)</td><td>LReLU (Layer Norm)</td><td>None</td></tr><tr><td>FFNN<sub>d</sub></td><td>(500, 200, 3)</td><td>LReLU, None</td><td>0.6, 0.6</td></tr><tr><td>FFNN<sub>a</sub></td><td>(1064, 150, 1)</td><td>LReLU, None</td><td>0.4, 0.4</td></tr><tr><td>FFNN<sub>n</sub></td><td>(500, 150, 1)</td><td>LReLU, None</td><td>0.4, 0.4</td></tr></tbody></table>

* Layers/Dimension lists the dimensions of all layers, including input and output.

* Nonlinearities and Dropout rates are given for forward connection, with the convention that dropout and non-linearity are applied respectively before and after the corresponding connection.

<footnote n="19">¹⁹The notation (L)ReLU comes from the conflation of the name of activation — the (leaky) rectifier — and the designation of the neural units of which it is the non-linearity — (Leaky) Rectified Linear Units.</footnote>
Table 5.3 lists the hyperparameters for the recurrent layers, both of which are bidirectional (hence the 2x term for the dimension) and use the default Pytorch v1.4.0 bindings for the CuDNN library (Chetlur et al. 2014).

<table><caption>Table 5.3: Recurrent layers parameters in DeCOFre baseline</caption><thead><tr><th>Layer</th><th>Layers</th><th>Hidden dimension</th></tr></thead><tbody><tr><td>LSTM<sub>w</sub></td><td>2</td><td>2 × 300</td></tr><tr><td>GRU<sub>c</sub></td><td>1</td><td>2 × 150</td></tr></tbody></table>

## 5.4.2 Baseline results

### Mention detection

<table><caption>Table 5.4: Mention detection evaluation (ANCOR test)</caption><thead><tr><th>System</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td colspan="4">ANCOR</td></tr><tr><td>Grobol et al. (2017b)</td><td>57.28</td><td>77.07</td><td>65.72</td></tr><tr><td>Grobol (2019)</td><td>82.99</td><td>89.07</td><td>85.87</td></tr><tr><td>Our baseline (on test)</td><td>86.83</td><td>88.41</td><td>87.62</td></tr><tr><td colspan="4">Other corpora</td></tr><tr><td>Uryupina (2010) (SemEval)</td><td>79.8</td><td>76.4</td><td>78.1</td></tr><tr><td>Soraluze et al. (2017) (EPEC)</td><td>74.67</td><td>74.47</td><td>74.57</td></tr><tr><td>Poesio et al. (2018) (ARRAU)</td><td>79.33</td><td>86.16</td><td>82.60</td></tr></tbody></table>

Table 5.4 presents the results of our experiments with mention detection compared to the baseline of Grobol et al. (2017b) — which consists in merely extracting all the NP from the output of an off-the-shelf parser — and those reported in Grobol (2019) for an earlier state of DeCOFre. To give some sense of the performances that could be hoped for, given the lack of relevant work on ANCOR, we also include the performance of other mention detection systems evaluated on comparable corpora for other languages, but these results should of course not be taken as fully comparable with ours.

### Coreference resolution

Table 5.5 presents the performances of our system for coreference resolution and compare it with those of previous works. Note that we did not compare with the performances of systems designed for the CoNLL-2012 setting (and in particular those of K. Lee et al. (2017, 2018)) since there is in general no simple way to provide them with gold mentions<fnref n="20" /> at either training or

<footnote n="20">²⁰In the usual meaning used in this work and not in the ‘anaphoric gold mentions’ sense used in K. Lee et al. (2017).</footnote>
<table><caption>Table 5.5: Coreference resolution evaluation (ANCOR test)</caption><thead><tr><th rowspan="2">System</th><th colspan="3">MUC</th><th colspan="3">B<sup>3</sup></th><th colspan="3">CEAF<sub>e</sub></th><th>CoNLL</th><th colspan="3">BLANC</th></tr><tr><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>Avg.</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>Désoyer et al. (2015a)</td><td>—</td><td>—</td><td>63.5</td><td>—</td><td>—</td><td>83.8</td><td>—</td><td>—</td><td>79.0</td><td>75.3</td><td>—</td><td>—</td><td>67.4</td></tr><tr><td>Grobol (2019)</td><td>72.3</td><td>47.7</td><td>57.3</td><td>89.7</td><td>71.0</td><td>79.2</td><td>72.8</td><td>86.0</td><td>79.4</td><td>72.0</td><td>78.2</td><td>60.1</td><td>65.7</td></tr><tr><td>Our baseline</td><td>73.71</td><td>54.22</td><td>62.14</td><td>89.51</td><td>75.29</td><td>81.67</td><td>77.28</td><td>88.15</td><td>82.26</td><td>75.36</td><td>80.32</td><td>66.71</td><td>71.12</td></tr></tbody></table>

test time, nor to make them distinguish between singleton mention and non-mention spans without significantly modifying them.

As mentioned in chapter 4, the existing works on ANCOR have been developed in different paradigms and are not easily comparable to ours. This is particularly true for Désoyer et al. (2015a), which relies on gold features — including mention anaphoricity—, and as such was able to get high scores on all metrics with a relatively simple system. Their results should thus be considered as an assessment of the usefulness of these features rather than a realistic benchmark, especially since it only reports F₁ scores and used an undocumented dataset split, which limits the interpretability of their results

With this reserve, the results presented in table 5.5 prove that is possible to design an accurate coreference resolution system for spoken French without relying on specific resources or extensive linguistic knowledge — and indeed obtain better quantitative results than a system using gold features.

### 5.4.3 Transfer learning

As mentioned in section 5.3.6, there are several alternatives while training an architecture with shared parameters. We report here a comparison between the performances of our baseline model in some training configuration for both tasks: without pretraining, with pretraining on the other tasks and with quasi-simultaneous training by alternating training on 2 batches for mention detection and 1 batch for antecedent detection. Apart from this, all the settings of these experiments are the same as those of our baseline, whose configurations are marked with an asterisk (*).

<table><caption>Table 5.6: Influence of training setting on mention detection</caption><thead><tr><th colspan="4">ANCOR dev</th></tr><tr><th>Setting</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>*No pretraining</td><td>86.89</td><td>88.15</td><td>87.52</td></tr><tr><td>Pretraining</td><td>88.62</td><td>84.22</td><td>86.36</td></tr><tr><td>Simultaneous</td><td>87.23</td><td>86.47</td><td>86.85</td></tr></tbody></table>
Coreference resolution for spoken French

Table 5.6 compares the results between the three methods for mention detection, with a clear improvement for the case where the span embeddings were not pretrained using antecedent scoring. Simultaneous training does not seem to help either, which suggests that training on antecedent scoring actively degrades the relevance of the span embeddings for mention detection. These results are quite intuitive: during each epoch antecedent scoring training, a small number of spans (the gold mentions) are each seen by the network around 100 times (the maximum number of antecedents), with the consequence that the resulting spans representations are highly optimized for this specific category of spans.

<table><caption>Table 5.7: Influence of the training setting on coreference resolution</caption><thead><tr><th rowspan="2">System</th><th colspan="3">MUC</th><th colspan="3">B<sup>3</sup></th><th colspan="3">CEAF<sub>e</sub></th><th>CoNLL</th><th colspan="3">BLANC</th></tr><tr><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>Avg.</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>No pretraining</td><td>75.02</td><td>54.61</td><td>62.8</td><td>90.52</td><td>75.12</td><td>81.99</td><td>77.17</td><td>88.45</td><td>82.33</td><td>75.71</td><td>82.03</td><td>65.18</td><td>70.15</td></tr><tr><td>*Pretraining</td><td>73.71</td><td>54.22</td><td>62.14</td><td>89.51</td><td>75.29</td><td>81.67</td><td>77.28</td><td>88.15</td><td>82.26</td><td>75.36</td><td>80.32</td><td>66.71</td><td>71.12</td></tr><tr><td>Simultaneous</td><td>71.93</td><td>48.54</td><td>57.65</td><td>89.4</td><td>72.88</td><td>80.23</td><td>73.81</td><td>87.79</td><td>80.12</td><td>72.67</td><td>77.51</td><td>62.68</td><td>67.18</td></tr></tbody></table>

Table 5.7 compares the results for coreference resolution, with a much less clear situation. Here, the results are slightly better without pretraining for the CoNLL metrics and slightly worse for BLANC, due to the more balanced results when using pretraining. In the end, our choice for our baseline was a pretrained model, BLANC being more relevant for our data.

What is clear, however is the situation of the simultaneous training setting: for coreference resolution, too, it noticeably degrades the performances. In addition to these experiments, we tried some other simultaneous training configurations, with different task repartitions, with results that were similar to or worse than these, with significantly lower learning speed (which was to be expected) and a tendency to diverge in later training stages.

### 5.4.4 Hyperparameters influence

In addition to the baseline results, we report here the results of some of our experiments on choosing hyperparameters. Most of these experiments were done on only on the mention detection step of our model, due to the cost of training the antecedent scoring model. The settings used for our baseline results are marked with an asterisk (*).

In table 5.8, we study the influence of the base learning rate on the quality of the mention detection model by testing some common values with a constant learning rate and the default Adam base rate of $1 \times 10^{-3}$ with a warmup over the first 1000 batches and a multiplicative step decay of 0.7 per epoch, which corresponds to an average learning rate of approximately $3 \times 10^{-4}$ (the best constant rate in our experiments over the 10 first epochs, our upper limit for coreference resolution).

Consistently with the current understanding of the influence of learning rate for Adam (Karpathy and Johnson 2019), $3 \times 10^{-4}$ seems to be the safest alternative between the often too high $1 \times 10^{-3}$, which leads to unstable convergence on the training set and the too low $1 \times 10^{-4}$, which leads
<table><caption>Table 5.8: Influence of the learning rate schedule on mention detection</caption><thead><tr><th colspan="5">ANCOR dev</th></tr><tr><th>Schedule</th><th>Base LR</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>Constant</td><td>1 × 10<sup>-4</sup></td><td>84.58</td><td>87.23</td><td>85.88</td></tr><tr><td>Constant</td><td>3 × 10<sup>-4</sup></td><td>84.82</td><td>89.94</td><td>87.06</td></tr><tr><td>Constant</td><td>1 × 10<sup>-3</sup></td><td>83.66</td><td>89.52</td><td>86.49</td></tr><tr><td>*Step decay (0.7)</td><td>1 × 10<sup>-3</sup></td><td>86.89</td><td>88.15</td><td>87.52</td></tr></tbody></table>

<table><caption>Table 5.9: Influence of weight decay on mention detection performances</caption><thead><tr><th colspan="4">ANCOR dev</th></tr><tr><th>Decay rate</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>1 × 10<sup>-5</sup></td><td>87.27</td><td>83.65</td><td>85.42</td></tr><tr><td>*0</td><td>86.89</td><td>88.15</td><td>87.52</td></tr></tbody></table>

to a slow training process and degraded generalization capabilities, possibly due to overfitting.

However, it seems that decaying the learning rate during the training process is a better alternative. A tentative interpretation could be that a high learning rate in the first epoch allows a rapid exploration of the parameters space to find a promising region, while the lower values in the later epochs allow for more precise tuning within this region. That said, this is a very small sample of the possible choices for learning rate schedules, and many more experiments would be needed to confirm these findings and determine an optimal choice.

In table 5.9, we report the outcome of using weight decay regularization to train the mention detection module. Even with a relatively mild weight decay rate (1 × 10⁻⁵ for a base learning rate of 1 × 10⁻³), the negative impact on the performances is significant.

This might be a consequence of the sparsity of the training signal for mention (due to the aforementioned heavy imbalance between spans classes), our use of other regularization methods (Dropout and Layer Normalization) which already makes the learning process harder (if hopefully more robust in generalization), a combination of both or unrelated factors. In any case, this result was sufficiently bad to discourage us from further experiments in this direction.

## 5.5 Conclusion

In this chapter, we proposed an end-to-end coreference resolution system for oral French, taking into account the peculiarity of this domain (lack of dedicated resources, non-standard and irregular language), the specificities of the available data (document nature, size and characteristics of mentions and coreference chains) and our material resources. Our architecture
<table><caption>Table 5.10: Influence of undersampling rate on mention detection performances</caption><thead><tr><th rowspan="2">System</th><th colspan="3">ANCOR dev</th></tr><tr><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>90 %</td><td>81.50</td><td>90.12</td><td>85.60</td></tr><tr><td>*95 %</td><td>86.89</td><td>88.15</td><td>87.52</td></tr></tbody></table>

is an hybrid of recent end-to-end neural architectures — which gives us a relative independence from linguistic resources — and more classical pipelines — which allows us to use a less constrained definition of entities and significantly reduces the computational cost of training our system.

The quantitative results obtained by this system are on par or better than those of previous attempts on the same data, which proves that our design is at least sensible for this setting. However, there seems yet to be a large room for further improvements.

The success of deep neural networks in natural language processing are still relatively recent, and in many areas, the reasons for their efficiency are still unclear. There is also a clear lack of definitive answers on which techniques are optimal for a given task. In this work, we have explored but a small fraction of the alternatives the current state of the art proposes. In further works, a much more thorough review of our choices should be conducted to identify the parts of our architecture that are responsible for its good results, those that are responsible for its mistakes and ways to improve it, drawing inspiration from C. Zhang et al. (2019) for instance. To this effect, we have tried to suggest interesting alternatives to our choices along this chapter, but given the breadth of the field and its current rate of expansion, many others are bound to surface in the years to come, either from human design or automatic architecture search (Wong et al. 2018; Zoph and Quoc V. Le 2017).

Even with our current architecture, the size of the hyperparameters space is consequent — both for the neural networks themselves and their training process — and our explorations were limited by material and temporal constraints to the relatively safe region of common values and did not range very far even there. Further work should thus also be concerned by making more thorough assessments of the influence of these choice and systematically improving them (Bergstra et al. 2011). Real-world usage of our architecture could also benefit from pragmatic performance scraping techniques, such as boosting or model ensembling, which proved valuable for comparable works (K. Lee et al. 2017).

Finally, in line with the advantages we found in using a pipeline instead of a fully end-to-end system, it is likely that there are benefits to be reaped from reintroducing explicit knowledge in coreference resolution systems using techniques that have received little attention in the recent years compared to the rush towards improving neural architectures. For instance global decoding techniques, such as Integer Linear Programming (Denis and Baldridge 2007b, 2009) or
entity-mention models (which have seen some attention from Wiseman et al. (2016) and some
recent exploratory work from Fei Liu et al. (2019)) could prove beneficial.

Regarding mention detection, given its inherently syntactic nature, formulations closer to those
of successful syntactic parsing systems will probably be both more legitimate and more efficient,
given the very encouraging results in that regard from B. Wang et al. (2018). Orthogonally
to the question of integrating human knowledge of coreference at the architectural level, we
also give in chapter 6 some propositions of augmentations using explicit features from external
linguistic knowledge.
Chapter 6

Explicit knowledge augmentations
for coreference resolution systems

In chapter 5, we have shown that it is possible to develop a coreference resolution system for spoken French using almost no *a priori* knowledge, by relying instead on dense vector representations of inputs learned end-to-end by a neural network architecture. In this chapter, we present the results of our investigations on techniques to introduce explicit knowledge in this neural architecture and the benefits that such knowledge could bring to a modern coreference resolution system. Here, by 'explicit knowledge' we mean information that is not directly available using only coreference annotated resources, in which we include not only knowledge coming from external resources and preprocessing tools, but also information that comes from the application of *a priori* linguistic knowledge — e.g. a linguistic model of reference<fnref n="1" /> — to the raw data in a coreference corpus.

For this, we face two main issues: how to include such knowledge in a pre-existing neural
architecture, and how to cope with the lack of reliable knowledge and preprocessing tools for
spoken French, noted in section 5.1 and which led us to develop a knowledge-poor system in the
first place. These issues are not necessarily independent: since the knowledge that we would
like to incorporate in our system is not completely reliable, it is critical that it is included in
such way that our system has the opportunity to learn during its training when and how this
knowledge can be depended upon and when to ignore it.

In section 6.1, we study raw features — features that do not depend on external knowledge, including some that are already included in our baseline system, but that we did not describe in details in chapter 5.

In section 6.2, we experiment with weakly supervised knowledge — that depends on the avail-
ability of unannotated linguistic material in the target language.

Then in section 6.3, we study the role of shallow linguistic knowledge, by adding general-purpose
word-level linguistic information to the inputs of our system.

<footnote n="1">¹Even though the work presented here stays at a lower level of sophistication.</footnote>
Finally, in section 6.4, we work on the addition of structural linguistic knowledge — such as syntactic parsing or named entity information — that are directly linked to aspects of coreference resolution.

## 6.1 Raw features

We call *raw features*, all features that do not depend on resources beyond the available content in the raw source material. Given that our work is about speech transcriptions, this does not mean that this content is entirely void of annotations, since — as we noted in section 3.2.2 — accurate representation of spoken language in written transcription implies the presence of information beyond the words uttered. Accordingly, we also consider features that depend on speaker identification, utterances boundaries, etc. information as raw, in the same sense that features relying on punctuation would be for written language.

In all of this work, we also consider that word tokenization is given, not that it is a trivial problem in general, but because it is arbitrary and mention detection evaluation depends on it. Therefore, having our system perform word tokenization on its own could potentially add noise to the evaluation that would only be due to having different conventions for word boundaries.

### 6.1.1 Span features

At the span representation level, there are few useful features of this type: Clark and Manning (2016b) for instance uses the length of the mention in number of words, its absolute position in the document (in number of mentions), inclusion in another mention and its type (pronoun, noun, proper noun or list) and a general document genre embedding. Of these, those that depend on document-level context or pre-existing mention detection and classification are not relevant for span representation, since we obviously do not have access to detected mentions at this stage. Genre embeddings also seem relevant in corpora made of heterogeneous collections, but in the case of ANCOR, given the size distribution of the subcorpora and their relatively close language varieties, (see section 3.2.2), it would be hard to learn and not necessarily very useful. This only leaves the mentions' length, which could theoretically be learnt by LSTM<sub>w</sub> (Weiss et al. 2018), but providing it directly to the model is easy enough and relieves LSTM<sub>w</sub> of that burden. Raghunathan et al. (2010) also mention a character-based acronym-reconstruction feature, which is certainly useful in acronym-rich corpora, but of little relevance for ANCOR.

As in Clark and Manning (2016b) and K. Lee et al. (2017), we bin span length in the following buckets<fnref n="2" />: [0, 1[, [2, 3[, [3, 4[, [5, 7[, [8, 15[, [16, 31[, [32, 63[ and [64, +∞[, in order to densify the training signal somewhat, since there should be few differences in practice between a span of length 9 and one of length 10 for instance. The bin index is then embedded via a lookup in a table of end-to-end trained embeddings. The same procedure is used for all the integer features in our system and could also be a factor in the relevance of this feature: even though LSTM<sub>w</sub> can in principle recover the length of a span, deriving a useful dense representation without further supervision is less likely.

<footnote n="2">²Introduced by Clark and Manning (2016b) with no motivation for their boundaries, which are clearly [log₂(t)]</footnote>
<table><caption>Table 6.1: Influence of the length feature on mention detection (ANCOR dev)</caption><thead><tr><th>Setting</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>Baseline</td><td>86.89</td><td>88.15</td><td>87.52</td></tr><tr><td>-length</td><td>87.28</td><td>84.63</td><td>85.94</td></tr></tbody></table>

<table><caption>Table 6.2: Influence of the length feature on coreference resolution (ANCOR dev)</caption><thead><tr><th rowspan="2">System</th><th colspan="3">MUC</th><th colspan="3">B<sup>3</sup></th><th colspan="3">CEAF<sub>e</sub></th><th>CoNLL</th><th colspan="3">BLANC</th></tr><tr><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>Avg.</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>Baseline</td><td>73.60</td><td>53.54</td><td>61.74</td><td>89.40</td><td>74.67</td><td>81.32</td><td>76.12</td><td>88.07</td><td>81.59</td><td>74.88</td><td>79.61</td><td>64.54</td><td>69.46</td></tr><tr><td>-length</td><td>72.88</td><td>50.85</td><td>59.52</td><td>88.90</td><td>73.82</td><td>80.60</td><td>75.05</td><td>88.12</td><td>80.98</td><td>73.70</td><td>76.80</td><td>63.91</td><td>68.08</td></tr></tbody></table>

Table 6.1 compares the performance for mention detection with and without using the span length feature and table 6.2 those for coreference resolution. These results suggest that this feature is indeed a valuable addition to span representation, despite its seemingly redundant nature.

## 6.1.2 Pair features

As we saw in section 4.1.2, mention pairs features usually belong to one of the following categories: similarity and agreement features, distance features and compatibility features. Among these, most distance features are raw, since they depend on either basic segmentation or mention detection (which is available at this stage) ; similarity features between raw mention-level features are also obviously raw, but most compatibility features require information from further processing.

In our baseline system, we use the following features:

* Distance in number of words

* Distance in number of utterances

* Distance in number of mentions

* Speaker agreement

* Inclusion of the mention in the antecedent candidate (equivalent to Raghunathan et al. (2010)'s 'i-within-i' feature)

Since we only represent spans in their immediate context, these features are the only access that the antecedent scorer has to the document (and therefore discourse) structure, which is critical to most reference phenomena (see section 2.1.2). For instance, without access to at

<footnote>with a special case for 3.</footnote>
least one distance feature, pronouns would have to be attached by choosing randomly between the compatible antecedent candidates. Similarly, without a speaker agreement feature, there would be no way for our system to determine if two self-reference from a speaker can be coreferent. For these reasons, experimenting without these features is unnecessary, since there is no possibility of it resulting in anything but severe performance degradations.

A raw pair feature that is commonly used in existing systems and that is not strictly necessary for our system is string matching. String matching has been a staple of machine-learning systems for coreference resolution since Soon et al. (2001) and until very recently (Clark and Manning 2016b; H. Lee et al. 2017) it was included in almost all of them. Moreover, it is very often one of the most informative features: Soon et al. (2001) reports that only using string matching is already enough to obtain significantly better performances than the previous best machine learning system (RESOLVE, McCarthy and Lehnert 1995) and its ablation in Clark and Manning (2016b) results in the second-worse performance loss (behind length features).

It is quite clear why this information is so valuable: for noun phrases and close pronouns, it is very unlikely that two matching mentions are not coreferent, therefore, trusting this feature — with some restrictions on distance for pronouns — should almost never result in a mistake. On the other hand, even for systems that are able to use word-level representations, learning to match mention representations when they are matching *but do not appear in the same context* is highly non-trivial. For instance, in our implementation (see section 5.2.2), the only way to access this feature would be via the soft-head representation, and it would require attention weights to be independent of the recurrent representations of words. Predicting this feature is clearly not a reasonable investment should we force the network to learn it and it is very unlikely that it would be learned otherwise. Conversely, including this feature explicitly is not very expensive, as it is easily computed.

We experiment with a relaxed version of string matching. Instead of strict string matching as a boolean feature, we use two similarity measures, considering spans as bag of words: the Jaccard similarity coefficient $J(A, B)$ (Jaccard 1912) and the Szymkiewicz-Simpson overlap coefficient $O(A, B)$ (Simpson 1947; Szymkiewicz 1934):

$$ 
\begin{aligned}
J(A, B) &= \frac{|A \cap B|}{|A \cup B|} \\
O(A, B) &= \frac{|A \cap B|}{\min(|A|, |B|)}
\end{aligned}
\qquad [6.1] $$

These allow us to represent exact string matching $J(A, B) = 1$, strict inclusion $O(A, B) = 1$ but also relaxations of both cases, in order to take into account small amounts of modifications — for instance the addition of a modifier, changing an article from definite to indefinite or the inclusion of disfluencies —. This also allows us to delegate the task of choosing a sensibility threshold to the system instead of allowing only strict matches.

Table 6.3 shows the results of adding string matching features to our baseline system for coreference resolutions. The gains are obvious for all metrics and are easily interpretable as improvements in correctly predicting coreference, which results in recall gains for MUC, $B^3$
<table><caption>Table 6.3: Influence of string matching features on coreference resolution (ANCOR dev)</caption><thead><tr><th rowspan="2">System</th><th colspan="3">MUC</th><th colspan="3">B<sup>3</sup></th><th colspan="3">CEAF<sub>e</sub></th><th>CoNLL</th><th colspan="3">BLANC</th></tr><tr><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>Avg.</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>Baseline</td><td>73.60</td><td>53.54</td><td>61.74</td><td>89.40</td><td>74.67</td><td>81.32</td><td>76.12</td><td>88.07</td><td>81.59</td><td>74.88</td><td>79.61</td><td>64.54</td><td>69.46</td></tr><tr><td>+string</td><td>77.76</td><td>60.18</td><td>67.59</td><td>89.66</td><td>76.63</td><td>82.58</td><td>79.76</td><td>89.69</td><td>84.38</td><td>78.18</td><td>79.92</td><td>66.16</td><td>70.78</td></tr></tbody></table>

and BLANC (morally recall gains in coreference identification) and precision gains for CEAF<sub>e</sub>, indicating less oversplitting of coreference chains.

## 6.2 Semi-supervised knowledge

We call *semi-supervised knowledge* any information that can be obtained by processing unannotated linguistic data, usually in the target language or sometimes in closely related ones (e.g. written French wrt. spoken French). The term *semi-supervised* (Collobert and Weston (2008) among others) denotes the fact that this knowledge is often obtained from supervised learning methods, hence not strictly *unsupervised* but that supervision does not come from any extra analysis of the data. This kind of information is more easily available than knowledge that requires the development of specific tools, especially since the development of methods to extract it from web content (Grave et al. 2018; Mikolov et al. 2018), which provides very large scale repositories of such data — if not necessarily in clean or reliable forms.

Though the idea of using knowledge extracted from raw data in downstream task is older — it is found for instance in the concept of Brown Clusters (P. F. Brown et al. 1992)—, it is mostly with its integration in neural architectures in the form of pretrained word embeddings, championed by Collobert and Weston (2008), Maas et al. (2011), Mikolov (2012) and Turian et al. (2010) and popularized to omnipresence by Mikolov et al. (2013) that its relevance became evident. It is in this form that we include semi-supervised knowledge in our architecture, both as of *static* pretrained word embeddings (section 6.2.1) and in the form of the more recent *contextual* word embeddings (section 6.2.2). A form of unsupervised knowledge that we did not explore is the unsupervised morphological semantic segmentation proposed by Grönroos et al. (2014), to which the subword segmentation used in BERT (Devlin et al. 2018, see section 6.2.2) is analogue.

### 6.2.1 Static word embeddings

The hypothesis underlying most word embeddings pretraining is the *distributional semantics hypothesis* that ‘words which are similar in meaning occur in similar contexts’ (Rubenstein and Goodenough 1965), implying that semantic information about a word can be learned by observing its contexts of occurrence in a large enough corpus. In practice, the most successful implementations rely on some form of language modelling task, where a neural network is trained to predict the likelihood of occurrence of a word given a context (or more rarely the
opposite as with the continuous skip-gram model of Mikolov et al. (2013)), the word embeddings
are then extracted from the learned internal representation of words in the network.

While there are many variations on this simple idea, the best known are the word2vec models of Mikolov et al. (2013), the GloVe model of Pennington et al. (2014) — which combine a language modelling objective with a variant of earlier formulations based on the factorization of a co-occurrence matrix — and FastText (Bojanowski et al. 2017; Mikolov et al. 2018) — which applies Mikolov et al. (2013)'s skip-gram model to subword units. In practice, the respective merits of these models are hard to assess: while they are usually evaluated using word analogy or document classification tasks, few works have explored their influence on more involved downstream tasks or cross-domain usage. Furthermore, the availability of pretrained models for languages other than English is not very consistent and the models that do exist often suffer from a lack of extensive documentation of the data and configuration used, which can be an issue, considering the influence of the hyperparameter 'tricks' described by Mikolov et al. (2018).

In consequence, we did not experiment with alternative models and limited ourselves to FastText.
We use it with the pretrained embeddings released by Grave et al. (2018) for our baseline as
described in section 5.4.1. We do not use the sub-word representations provided by FastText,
since they would make the comparison between the two settings less clear. In both settings, the
word embeddings are fine-tuned during the training procedure, and the span representations
learned for mention detection are carried over for coreference resolution in the sequential mode
described in section 5.4.3.

Table 6.4 includes a comparison of the mention detection performances of our baseline and the same model, trained in the same setting, but without pretrained word embeddings; Table 6.5 includes the same comparison for coreference resolution. In both cases, the difference is not significant, though mention detection seems somewhat harder without pretrained embeddings. A tentative interpretation for these results could be that while the FastText embeddings help at the beginning of training, the genre difference between the data they were trained for (web text) and the spoken French of ANCOR is sufficient to allow randomly-initialized embeddings to reach similar performances using only the signal in ANCOR. This hypothesis is also consistent with the much slower convergence of the models without pretraining, which took in average 5 more epochs to overfit on the development set for mention detection and 10 more epochs for coreference resolution, an increase of about 50% in both cases.

6.2.2 Contextual word embeddings

Static word embeddings are usually represented by a mapping from a word *form* to a dense
vector, which implies that homographs are represented by the same vector while alternative
orthographies of the same word are not. This is symptomatic of larger issue: although they
rely on the assumed dependency between the meaning of a word and its context, static word
embeddings are — by definition — *context-agnostic* and a static word embedding is in effect a
mixture of the representations of all the meanings of the corresponding word form. This goes
beyond simple homography, since it also hampers the capacity of a system to take into account
e.g. figurative meanings of words.
While some earlier<fnref n="3" /> works such as Choi et al. (2017) and Frederick Liu et al. (2018) report efforts to address this issue at a downstream level, the first successful implementation of general-purpose⁴ context-dependent word embeddings is ELMo due to Peters et al. (2018). In spirit, it builds upon the same distributional semantics hypothesis as static word embeddings, and it is also implemented via a language modelling task, but instead of pooling word forms representations over all their occurrences, it instead proposes to keep a context-specific representation obtained by computing a weighted average of the internal states of stacked LSTM layers. These contextual representations give a clear advantage for many downstream tasks, as seen in the original article and more specifically in K. Lee et al. (2018), where merely replacing static word embeddings by ELMo representations improves the result of K. Lee et al. (2017)'s baseline by more than 3 F₁ percentage points for all the CoNLL metrics.

The cost for these impressive performance gains is a far worse computational cost, since it replaces a simple table lookup for static word embeddings by a sentence- or document-wise processing by a multilayer recurrent neural network. However, fine-tuning ELMo for downstream tasks only requires learning a very small number of parameters compared to fine-tuning static word embeddings, since the parameters of the recurrent layers themselves can be frozen and only the averaging weights have to be tuned. As a result, while using ELMo representations tends to be slower at inference time, in our experiments, it was not noticeably slower than the alternatives.

Despite its success and fast adoption by the natural language processing community, there have not been a lot of works using ELMo in coreference resolution, as it has soon been supplanted by BERT (Devlin et al. 2018) and its successors. In particular, alternative pretrained ELMo models are uncommon, especially for languages other than English. In our case, we experiment with an ELMo model trained on the French part of OSCAR (Ortiz Suárez et al. 2019) for Ortiz Suárez et al. (2020).⁵

BERT (Devlin et al. 2018) is a similar proposal, that replaces the bidirectional language model objective of ELMo by a masked language model, and its stacked LSTM architecture by a deep stack of Transformer blocks (Vaswani et al. 2017). Its main advantage over ELMo resides in this substitution, which allows a much more efficient training, since the feedforward architecture of Transformers is easily parallelizable, especially using dedicated hardware (Jouppi et al. 2017). This in turn allows for much larger models, trained on much larger datasets and extensive fine-tuning on multiple downstream tasks that would be impractical with ELMo. Consequently, the performances of all the currently published BERT-based models outperform the ELMo- and static embeddings-based alternatives, often by a fair margin, and coreference resolution is not different in that respect (Joshi et al. 2019b).

Due to the prohibitive cost of pre-training BERT, models for languages other than English are still not very common, but two pretrained models exist for French: CamemBERT (L. Martin et al. 2020) and FlauBERT (H. Le et al. 2020). We experiment with CamemBERT — which is also trained

<footnote n="3">³Or, to be precise, published simultaneously in the case of Frederick Liu et al. (2018).
⁴Dai and Quoc V Le (2015) proposed a very similar idea, but with task-dependent embeddings.
⁵Many thanks to Pedro Ortiz Suárez for giving us access to this model ahead of time.</footnote>
on the French part of OSCAR and is therefore more comparable with our ELMo representations – and the mutlilingual version of BERT provided by Devlin et al. (2018) (mBERT). Following Joshi et al. (2019b), we also made a small change in our architecture to accommodate for BERT subword tokenization: instead of using the bounding *words* in our spans representations, we use the bounding word *pieces*.

<table><caption>Table 6.4: Influence of word embeddings on mention detection (ANCOR dev)</caption><thead><tr><th>Setting</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>No pretraining</td><td>84.68</td><td>88.89</td><td>86.73</td></tr><tr><td>Baseline (FastText)</td><td>86.89</td><td>88.15</td><td>87.52</td></tr><tr><td>ELMo</td><td>89.67</td><td>91.96</td><td>90.80</td></tr><tr><td>mBERT</td><td>84.06</td><td>90.50</td><td>87.16</td></tr><tr><td>CamemBERT</td><td>88.47</td><td>93.24</td><td>90.79</td></tr></tbody></table>

<table><caption>Table 6.5: Influence of word embeddings on coreference resolution (ANCOR dev)</caption><thead><tr><th rowspan="2">System</th><th colspan="3">MUC</th><th colspan="3">B<sup>3</sup></th><th colspan="3">CEAF<sub>e</sub></th><th>CoNLL</th><th colspan="3">BLANC</th></tr><tr><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>Avg.</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>No pretraining</td><td>73.49</td><td>54.35</td><td>62.15</td><td>88.64</td><td>75.15</td><td>81.24</td><td>76.11</td><td>87.45</td><td>81.29</td><td>74.89</td><td>76.80</td><td>65.72</td><td>69.50</td></tr><tr><td>Baseline (FastText)</td><td>73.60</td><td>53.54</td><td>61.74</td><td>89.40</td><td>74.67</td><td>81.32</td><td>76.12</td><td>88.07</td><td>81.59</td><td>74.88</td><td>79.61</td><td>64.54</td><td>69.46</td></tr><tr><td>ELMo</td><td>76.33</td><td>60.21</td><td>67.06</td><td>88.67</td><td>77.30</td><td>82.53</td><td>79.24</td><td>88.53</td><td>83.56</td><td>77.72</td><td>78.90</td><td>67.84</td><td>71.74</td></tr><tr><td>mBERT</td><td>73.93</td><td>56.76</td><td>64.02</td><td>88.90</td><td>75.65</td><td>81.67</td><td>77.68</td><td>87.87</td><td>82.40</td><td>76.03</td><td>79.88</td><td>65.66</td><td>70.42</td></tr><tr><td>CamemBERT</td><td>76.20</td><td>60.61</td><td>67.32</td><td>88.52</td><td>77.40</td><td>82.53</td><td>79.40</td><td>88.47</td><td>83.63</td><td>77.83</td><td>79.27</td><td>67.94</td><td>71.96</td></tr><tr><td>ELMo+string</td><td>79.93</td><td>66.58</td><td>72.50</td><td>89.92</td><td>79.32</td><td>84.24</td><td>82.69</td><td>90.13</td><td>86.21</td><td>80.98</td><td>82.67</td><td>69.39</td><td>74.15</td></tr></tbody></table>

We report in table 6.4 a comparison between all the word embedding techniques that we have described in this section for mention detection, and in table 6.5 for coreference resolution. In all cases, contextual embeddings outperform static embeddings and non-pretrained embeddings, by a fair margin for the monolingual models and by a smaller one for mBERT (which actually perform worse, if not significantly so on mention detection).

Surprisingly, CamemBERT does not bring any significant improvement over ELMo, contrarily to the situation for English reported by Joshi et al. (2019b). An explanation for this might reside in their respective input representations: ELMo uses a character-level convolutional neural layer, which is robust to both out-of-vocabulary words and irregular spelling, while CamemBERT relies on subwords obtained by optimizing a byte pair encoding vocabulary (Sennrich et al. 2016) for its source corpus, in that case a web-based one. As a result, the CamemBERT tokenizing model is less robust to the specific spellings used in speech transcriptions, in particular for incomplete words: it tokenizes 'Je suis con-euh concentré' as 'Je', 'suis', 'con', '-', 'eu', 'h', 'con', 'cent', 'ré' for instance, which introduces a false homography between the 'eu' part of the interjection 'euh' and the 'eu' past participle form of 'avoir' (despite having had exposure to 'euh' in its training data), while ELMo correctly treats 'euh' as its own token.
Finally, we also include the result of augmenting our baseline with both ELMo representations and the string matching features described in section 6.1.2, which shows the same gain over simply using ELMo as using the string features does over the baseline. This suggests that the gain brought by ELMo representations is not simply due to the differences in their representations of inputs, but comes from additional knowledge learned from their pretraining for language modelling on a large corpus.

## 6.3 Shallow linguistic knowledge

We define *shallow linguistic knowledge* as knowledge that is ultimately derived from annotated resources and concerns the nature of linguistic material, but not its structure. In particular, we do not include knowledge that concerns segments not present in the raw material (i.e. words and utterances in the case of ANCOR) in this category. The most common types of shallow linguistic knowledge are word-level syntactic and lexical features, e.g. POS annotations, lemmas and morphological features, but semantic information such as sentiment polarity and sentence-level dialogue act features are also part of this category.

In our case, we experiment with two instances of shallow linguistic knowledge: POS tags and lemmas. These are two of the most common and simple shallow features found commonly in natural language processing systems, POS tags in particular are commonly used in existing coreference resolution systems, often as part of a syntactic parsing pipeline. POs tagging is also a relatively well understood task, and current automatic systems commonly reach human-level performances on this task in in-domain evaluations for well-resourced languages (Bohnet et al. 2018; Heinzerling and Strube 2019) although the performances for less regular genres such as social media texts tends to be somewhat worse (Godin 2019).

Lemmas are less universally used than POS features, since for morphologically-poor languages (e.g. English), they are not much more informative than a combination of POS and word embeddings, which are easier to obtain. Proper lemmatization of morphologically-rich languages and non-standard varieties, however, is still very much an open problem (Manjavacas et al. 2019) and is still relevant as an extreme version of word form normalization, which can have a non-negligible influence on downstream tasks (Straka and Straková 2019).

Specifically, in our experiments, we use the POS tags and lemmas provided by spaCy (Honnibal and Montani 2019) — precisely with the *fr_core_news_sm* version 2.2.5 model, trained on the Sequoia (Candito and Seddah 2012) and WikiNER (Nothman et al. 2013) corpora. In terms of performances, this model is somewhat behind the current state-of-the for French, however it has the advantage of being very fast and easy to integrate in a preprocessing pipeline. In any case, since there are currently no standard evaluation dataset for POS-tagging and lemmatization for spoken French, evaluating the relative performance of systems designed for written French on these auxiliary tasks would be out-of-scope for this work. A cursory exploration of the outputs of this model on ANCOR suggest that the POS and lemma, while obviously not perfectly accurate, were generally correct for parts with a low amount of disfluencies and low interactivities but more brittle in the most interactive parts. Conversely, although this model also provide additional morphological features (in the Universal Dependency (Nivre et al. 2016) formalism),
we did not experiment with these, as they were usually less reliable, harder to integrate in our
architecture (since they are in the form of variable-length multi-valued features) and since the
information they provide seemed too sparse to be useful.

In both cases, we augment our baseline architecture with these features by simply concatenating
their embeddings with the word representations e(w) given by our word-encoding layer (see
section 5.2.2). These embeddings are initialized randomly — in particular, lemma embeddings
do not share their weights with the word embeddings — and are of dimension 50 for the POS
and 150 for the lemmas.

<table><caption>Table 6.6: Influence of shallow linguistic knowledge on mention detection (ANCOR dev)</caption><thead><tr><th>Setting</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>Baseline</td><td>86.89</td><td>88.15</td><td>87.52</td></tr><tr><td>+POS</td><td>87.24</td><td>89.62</td><td>88.41</td></tr><tr><td>+lemmas</td><td>86.65</td><td>89.59</td><td>88.09X</td></tr></tbody></table>

<table><caption>Table 6.7: Influence of shallow linguistic knowledge on coreference resolution (ANCOR dev)</caption><thead><tr><th rowspan="2">System</th><th colspan="3">MUC</th><th colspan="3">B<sup>3</sup></th><th colspan="3">CEAF<sub>e</sub></th><th>CoNLL</th><th colspan="3">BLANC</th></tr><tr><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>Avg.</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>Baseline</td><td>73.60</td><td>53.54</td><td>61.74</td><td>89.40</td><td>74.67</td><td>81.32</td><td>76.12</td><td>88.07</td><td>81.59</td><td>74.88</td><td>79.61</td><td>64.54</td><td>69.46</td></tr><tr><td>+POS</td><td>70.95</td><td>50.24</td><td>58.61</td><td>89.22</td><td>73.42</td><td>80.48</td><td>74.83</td><td>87.55</td><td>80.62</td><td>73.24</td><td>79.05</td><td>63.18</td><td>68.11</td></tr><tr><td>+lemmas</td><td>74.09</td><td>50.40</td><td>59.71</td><td>90.14</td><td>73.73</td><td>81.06</td><td>74.61</td><td>88.50</td><td>80.90</td><td>73.89</td><td>78.69</td><td>63.97</td><td>68.66</td></tr></tbody></table>

We report in table 6.6 the comparisons between our baseline and its augmentation with shallow linguistic knowledge for mention detection and in table 6.7 for coreference resolution. For both tasks, the changes brought by these augmentations are not significant, and their interpretation can therefore only be speculative. However, the somewhat better results for mention detection with POS could be an indication that the span representation module can learn to use this information for mention detection, and conversely that learning to rely on it too much can reduce its effectiveness for coreference resolution, where POS are less relevant.

## 6.4 Structural linguistic knowledge

In opposition to *shallow* linguistic knowledge (section 6.3), we call *structural linguistic knowledge* any information that both ultimately derives from annotated resources and provide structural information on linguistic material. Structural information can be non-recursive, for segmentation information beyond those provided by the base material — as with chunking and named entity boundaries — or recursive, the most prominent form of which is syntactic parsing.

The main advantage of structural knowledge over the other types of information we experiment with in this chapter is that in many cases, the additional segmentation that it provided is correlated with mention boundaries: named entities are mentions, and in ANCOR non-nested
mentions are nominal chunks and all mentions are either pronouns or noun phrases. Consequently, successfully integrating this kind of knowledge as part of our mention detection architecture should relieve it of having to deal with the simple cases where these features are sufficient and let it concentrate instead on more complex cases.

However, given that the tools that we use to obtain this knowledge were not designed for spoken French, it is to be expected that the segmentations they provide will not be completely accurate, or even sufficiently accurate to use them as a resource for rule-based processing. Instead, we use it merely as an additional source of information, and include not only the exact boundaries that they provide but also features based on partial matching with span boundaries.

In this section, we report our experiments on integrating features obtained from a chunker, a
named entity detector and a syntactic parser. In all cases, we integrate these features in our
system in two ways: as span level features and as a guide for the undersampling performed on
the training set (see section 5.3.4).

6.4.1 Non-recursive structures

We experiment with two types of information on non-recursive structures: chunks and named entities.

Chunking, introduced by Abney (1992) as an intermediary step in the development of a syntactic parser consists in segmenting a sentence in non-overlapping, non-nested syntactic units, such as those in the following bracketing '[I begin] [with an intuition]: [when I read] [a sentence], [I read it][a chunk] [at a time]' (Abney 1992). While Abney (1992) does not describe it as an end it itself, chunking has subsequently become a very popular lightweight substitute for full syntactic parsing. Indeed, the non-recursive nature of chunking allows to cast it a sequence labelling task, for which efficient algorithms are known, in both rule-based and machine learning configurations (Sha and Pereira 2003).

For coreference as annotated in ANCOR, the chunks of interest are *noun* chunks, since their boundaries coincide with those of non-nested mentions. Works such as Broscheit et al. (2010) and Soraluze et al. (2012), operating in context where syntactic parsing is unavailable or unreliable have successfully used chunking in combination with handcrafted rules (in order to recover the nested structure of mentions) for mention detection.

We experiment with integrating the result of chunking in our system in two ways. First, every
span receives an additional feature depending on its overlapping with one of the nominal chunks
detected for its sentence, this feature can take one of the four following values:

**exact** if the span boundaries correspond exactly to those of a nominal chunk

**included** if the span is a proper subsequence of a nominal chunk

**outside** if the span does not intersect with any nominal chunk

**incompatible** if the span intersects with a nominal chunk, but neither is a subsequence of the other
Second, while undersampling for non-mention spans for the training set of our system, we prevent noun chunks from being discarded. The rationale behind this choice is that detected non-mention nominal chunks should be harder to distinguish from mention spans than any arbitrary span, since these spans should be more syntactically similar. Our hope is that leaning this harder task might make our mention detection module perform better overall.

As for named entities, the theoretical (section 2.3.1) and practical (section 4.1.1) links between
named entity recognition and coreference resolution are clear. Since named entities are a
specific class of mention with a specific mode of reference, having access to their boundaries
and their types should in principle be a valuable asset for mention detection, and possibly for
coreference resolution. Accordingly, we enhance our spans representations with a categorical
feature whose value for named entity spans is the type of the corresponding entity and None
for the other spans. Contrarily to chunks, in order to maximize the precision of this feature,
we only consider exactly matching spans and do not report partial overlaps, but we do prevent
erroneously detected named entity spans from being discarded during undersampling.

As in our experiments with shallow linguistic knowledge (section 6.3), we use spaCy for named
entity noun chunk detection, in the same configuration as previously. Other chunkers exist for
French, such as SEM (Dupont 2017; Tellier et al. 2012), but none for spoken French<fnref n="6" /> — hence
our choice of a somewhat worse preprocessing in terms of accuracy but one that more easily
integrable in our pipeline. The situation for named entity recognition is somewhat better, as
several named entity recognition systems were developed for broadcast spoken French on
the occasion of the ESTER 2 (Galliano et al. 2009) and ETAPE (Gravier et al. 2012) and in
independent works such as Hatmi (2014) and Zidouni et al. (2010), but none of these systems
was both publicly available and easily integrable in our pipeline.

Finally, similarly to word-level features both of these features are represented in our system
by concatenating a feature embedding (of dimension 16) to the spans representations. These
embeddings are also randomly initialized and learned during the training phase.

6.4.2 Syntactic parsing

Syntax, in one form of another is one of the oldest concerns in linguistics⁷ and the analytic tools
it provides are the foundation of many linguistic theories, including theories of anaphora, such
as the syntactic formulation of the government and binding theory of pronominal anaphora
(Chomsky 1981). It is thus only natural that syntactic parsing was a prerequisite to most of
the coreference resolution systems until very recently (see chapter 6). Indeed, in most models
of syntax and reference, mentions coincide with syntactic objects — usually constituents or
equivalents and their roles in the syntactic structure of sentences has a clear influence on the
reference identification process. Therefore, for mention detection, having access to constituents
allows for considerable simplifications, with relatively simple heuristics (see section 4.3) provid-
ing strong baseline strong baselines. As for coreference resolution, beyond the integration of

<footnote n="6">⁶Although the results of Tellier et al. (2013, 2014) suggest that a bootstrapping procedure could yield one with limited efforts, designing such a system is beyond the scope of this work.

⁷With roots as old as Pāṇini (–0349).</footnote>
syntactic constraints and preferences, many systems also use rules or features that relies on
the availability of mention heads (section 4.1.1), implicitly relying on the strong correlation
between the semantic content of the head of a constituent and that of the constituent itself.

In our case, we use the dependency syntax analyses (in the Universal Dependencies formalism) included in ANCOR-AS (Grobol et al. 2018b), obtained from the then-state-of-the-art parser for written French Dyalog-SRNN (Villemonte De La Clergerie et al. 2017). As equivalents to noun phrases, we extract noun-headed subtrees from these analyses, using simple heuristics to account for specific idiosyncrasies of the annotation scheme — most notably the choice of using the subject as the root of copular constructions.

As with chunks, in order to account for errors in these analyses, we also introduce a relaxation
in the matching between spans and subtrees: we define the distance between a span w<sub>j</sub>, ..., w<sub>i</sub>
and a subtree of maximal projection w<sub>k</sub>, ..., w<sub>ℓ</sub> as |i − k| + |j − ℓ| and associate every span with the
closest noun-headed subtree. Using this matching, we then enhance our spans representations
with the following features, also added as embeddings to span representations:

- Head word form (dimension 300)

• Head POS (dimension 50)

- Head dependency type (dimension 50)

- Head's governor word form (dimension 300)

- Head's governor POS (dimension 50)

- Distance to the corresponding span (dimension 20)

In addition to our usual setting for mention detection where all spans up to a certain length are considered as mention candidates, we also experiment with an alternative scheme where mention candidates are restricted to the span within a certain distance of their associated span. Unlike the restriction to undersampling that we use for chunks and named entities, this scheme is not only applied to the training set but also to the development and test set. This scheme thus serves as a filter that reduces the class imbalance for mention detection, by removing many spurious mention candidates — which should make the mention detection task easier while still leaving the possibility of recovering from minor parsing errors. However, for spans with severe parsing errors, mention spans could still be lost, depending on the maximal allowed distance between spans and subtrees. In our experiments, we chose to limit this distance to 3, which covers about 96.5 % of the mention spans. This alternative scheme is noted by ‘tree’ in table 6.8 and table 6.9, while our baseline scheme is noted by ‘span’.

6.4.3 Results

We report in table 6.8 the comparison of the structural linguistic features described in this
section. Clearly, in our experiments, adding information on noun chunk and on named entities
based on external preprocessing tools does not have a significant impact on mention detection.
These results suggest that either our system is able to learn implicitly the knowledge that the
<table><caption>Table 6.8: Influence of structural linguistic knowledge on mention detection (ANCOR dev)</caption><thead><tr><th>Setting</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>Baseline</td><td>86.89</td><td>88.15</td><td>87.52</td></tr><tr><td>+chunk</td><td>87.20</td><td>88.03</td><td>87.61</td></tr><tr><td>+NER</td><td>86.28</td><td>88.26</td><td>87.26</td></tr><tr><td>+parsing (tree)</td><td>91.32</td><td>83.76</td><td>87.38</td></tr><tr><td>+parsing (span)</td><td>86.59</td><td>90.49</td><td>88.21</td></tr></tbody></table>

tools provide using only the information latent in our training data, or alternatively, that these tools are too unreliable in this cross-domain setting to allow our system to learn from them.

The case of syntactic parsing is more interesting: in the tree-based setting, although the final $F_1$ is not significantly different from the baseline's, it comes from different scores, with both a significantly better precision and a significantly worse recall. This is consistent with our hypothesis that relaxed subtrees extracted from a syntactic analysis can be an interesting alternative to our baseline of using every spans, as it tends to give less imbalance between mentions and non-mentions. Conversely, it also confirms that the accuracy of the syntactic analysis used for this is critical, since it makes some mention spans irrecoverable, which can significantly increase the false negative rate.

The results in the span-based setting are better, and significantly so for recall, which suggests that there is indeed some knowledge captured by syntactic parsing that our system does not learn by itself and confirms our intuition that there should be a way to take advantage of external structured linguistic knowledge. The fact that these better results were obtained using knowledge from a state-of-the-art system rather than the 'good enough' tools used for the other features also tends to confirm that the cross-domain relevance of preprocessing tools are indeed correlated to their relative in-domain performances.

<table><caption>Table 6.9: Influence of structural linguistic knowledge on coreference resolution (ANCOR dev)</caption><thead><tr><th rowspan="2">System</th><th colspan="3">MUC</th><th colspan="3">B<sup>3</sup></th><th colspan="3">CEAF<sub>e</sub></th><th colspan="3">CoNLL</th><th colspan="3">BLANC</th></tr><tr><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th><th>Avg.</th><th>P</th><th>R</th><th>F</th><th>P</th><th>R</th><th>F</th></tr></thead><tbody><tr><td>Baseline</td><td>73.60</td><td>53.54</td><td>61.74</td><td>89.40</td><td>74.67</td><td>81.32</td><td>76.12</td><td>88.07</td><td>81.59</td><td>74.88</td><td>79.61</td><td>64.54</td><td>69.46</td><td>73.60</td><td>53.54</td><td>61.74</td></tr><tr><td>+chunks</td><td>72.46</td><td>51.94</td><td>60.19</td><td>88.26</td><td>74.29</td><td>80.61</td><td>75.67</td><td>87.97</td><td>81.28</td><td>74.03</td><td>75.92</td><td>64.66</td><td>68.43</td><td>72.46</td><td>51.94</td><td>60.19</td></tr><tr><td>+NER</td><td>74.19</td><td>47.60</td><td>57.69</td><td>90.93</td><td>72.37</td><td>80.52</td><td>73.29</td><td>88.70</td><td>80.19</td><td>72.80</td><td>80.44</td><td>62.16</td><td>67.36</td><td>74.19</td><td>47.60</td><td>57.69</td></tr><tr><td>+parsing</td><td>73.25</td><td>51.32</td><td>60.07</td><td>89.84</td><td>73.70</td><td>80.90</td><td>75.10</td><td>88.04</td><td>80.98</td><td>73.98</td><td>80.17</td><td>63.61</td><td>68.64</td><td>73.25</td><td>51.32</td><td>60.07</td></tr></tbody></table>

Table 6.9 gives a comparison of the influence of structural linguistic features on coreference resolution. As for mention detection, the differences with our baseline are not significant, suggesting that these features are not informative or reliable enough to be of use for our system.
This is not very surprising for chunking: since this evaluation is on gold mentions, the only information that noun chunk boundaries could bring to this task would be on cases where the chunker was consistently failing to detect some categories of chunks, which does not seem like a reasonable hypothesis. As for named entities their boundaries are not more relevant than those of noun chunks, and although their categorization might have been useful in other domains where named entities appear in coreference chains with other noun phrases, named entity mentions are probably too few — both in occurrences and in unique entities — in ANCOR for it to make a significant difference.

The result for syntactic parsing are more disappointing. Given the importance of head-matching features in other coreference resolution systems — including systems based on neural networks such as Clark and Manning (2016a) and Wiseman et al. (2015) — and the success of syntax-based heuristics in earlier works, one could have hoped to see more improvements by integrating explicit syntactic features in our system. However, the correlation noted by K. Lee et al. (2017) between the representations obtained via a soft-head attention mechanism trained for coreference resolution and the syntactic heads of mentions could suggest that this kind of information is recoverable without relying on a syntactic parser. In fact, it might actually be that the domination of the head in the semantic content of span and its importance in coreference resolution makes this knowledge inferrable using only coreference information.

## 6.5 Conclusion

In this chapter, we studied the impact of adding various forms of knowledge to the baseline system presented in chapter 5 on its performances. We find that in general, knowledge obtained from preprocessing tools — on which all coreference resolution systems were based until very recently — does not significantly improve nor degrades the performances of our system. Conversely, we find that knowledge derived from semi-supervised representation learning can greatly improve these performances, as does the addition of very simple string-matching features that do not require external resources.

However, these results come with some caveats: few preprocessing tools for spoken French are currently available — and almost none for the *spontaneous* genre used in ANCOR — which led us to use tools designed for *written* French instead. In consequence, although we had no way to quantify it, a degradation of their performances was to be expected, especially since the specific tools we chose were not specifically designed for cross-domain robustness either. From this, we can conclude that in order for explicit linguistic knowledge to be relevant in this setting, it has to be more accurate and that taking advantage of noisy inputs in neural architectures is not a trivial task. That said, the results reported in section 6.4.2 by using a syntactic parser to improve mention detection gives us hope that using more accurate preprocessing tools would indeed lead to real improvements and in the meantime, that they can be useful in specific contexts (in that case, for downstream applications where the precision of mention detection is more important than its recall).

The good results obtained by using semi-supervised knowledge is not very surprising, since they mostly confirm a trend observed in almost all areas of natural language processing — in
particular for the contextual input embeddings –, but it is interesting to see that contrarily
to knowledge extracted from preprocessing tools, the cross-domain setting is not an obstacle
here. This might be due to the fact that these representations were specifically designed and
trained to be used in neural architectures, with few constraints from linguistic theories, while
the information provided by preprocessing tools is necessarily geared toward specific models of
language that might not apply to other domains. Another explanation is that the usefulness of
these representations comes from the possibility of leveraging far more linguistic data than is
available for tools that need access to annotated corpora. Nevertheless, let us note that these
gains in performance are not free or even cheap: these representations learning techniques
*require* consequent amounts of data and computational power in order to reach these levels of
accuracy, which would be problematic for uses in lesser-resourced languages and for groups
with limited access to the necessary infrastructures.
# Chapter 7

## Conclusion and perspectives

In this work, we have dealt with coreference resolution applied to the case of spontaneous spoken French. Our main contribution to this question is DeCOFre, the first end-to-end coreference resolution for French, which also happens to be the first end-to-end coreference resolution system for the spontaneous spoken genre in any language. To overcome the scarcity of resources for spoken French, this system does not depend on external knowledge, and still obtains reasonably good results, quantitatively speaking. We also provide an assessment of the interest of introducing resources developed for written French in that system, and report that while resources built on unannotated corpora can improve on our baseline system, more traditional preprocessing tools have a very limited impact, stressing the importance of developing natural language processing systems that are robust in more genres and domains. Finally, we also propose a new standard way of representing annotated resources that is suitable for coreference corpora, a representation used in the recently released DEMOCRAT corpus (Landragin 2016) and an improved version of ANCOR (Muzerelle et al. 2013a, 2014), the resources on which this work is based.

In the short to immediate term, the clear priority for extending this work should be to reproduce our experiments on the DEMOCRAT corpus, which would make the respective influence of the genre and of the system architecture in DeCOFre performances clearer. To this end, continuations of our standardization effort for coreference corpora in French are underway, and a resource unifying DEMOCRAT and ANCOR will be available soon.

Developing tools and resources specific to spoken French would also make our results from chapter 6 more definitive. It would also be interesting to extend these experiments to other forms of external knowledge, and in particular knowledge based on the *actual* source material (i.e. the audio signal) rather than transcriptions. From a pure quantitative point of view, there is probably a large margin of improvement to be found by exploring the hyperparameter space of our system, as well as by experimenting with other coreference resolution algorithms. That said, if results from other fields in natural language processing are any indication, the easiest way to the raw quantitative performances of our system is probably to improve its input representations, for instance by pretraining contextual word embeddings on spoken French and fine-tuning them on tasks related to coreference resolution.
Although more research is needed in that direction, it seems clear from our experiments that including high-level linguistic knowledge in a neural coreference resolution system is not trivial, especially if this knowledge is unreliable: so far, the capacity of neural networks to make sense of information hidden in noisy inputs does not seem to be enough. However, despite the disappointing results of our experiments with joint learning, other works (such as Fei Liu et al. (2019), Sanh et al. (2018) and Swayamdipta et al. (2018)) suggest that joint learning of tasks of different level could help in that matter. The case of the Referential Reader (Fei Liu et al. 2019), of generative methods in neural parsing (Crabbé et al. 2019; Dyer et al. 2016) and of the various ways to learn representations from language modelling also suggest that semi-supervised tasks are a sensible way to obtain linguistic knowledge in forms that are well suited for neural architectures without requiring expensive resources. In the coming years, this could be a way to help fighting Durrett and Klein (2013)'s 'uphill battles'.

Finally, the successes of the transition-based mention detection system of B. Wang et al. (2018) and of the document-order processing of the Referential Reader (Fei Liu et al. 2019) suggest that the hypothetical document-reading device of Karttunen (1976) might indeed become a practical idea in the near future. More concretely: a multi-task system that would process a document in reading order, with explicit entity modelling and joint parsing and mention detection could be a way to improve on the current models in terms of sheer performance, interpretability and consistency with cognitive models of coreference.

Finally, drawing inspiration from recent advances in syntactic parsing (Meechan-Maddon and Nivre 2019; Smith et al. 2018), improvements in coreference resolution for lesser-resourced languages and genres could come from jointly learning on multilingual and multigenre datasets. Generally speaking, coreference resolution as a field would greatly benefit of moving toward unified frameworks and representations akin to those proposed by the Universal Dependencies project. A move in this direction was made on the occasion of the multilingual coreference detection shared task at SemEval 2010 (Recasens et al. 2010), which could serve as a starting point for a more unified understanding of coreference resolution across languages.
Bibliography

Abney, Steven P. (1992). 'Parsing By Chunks'. In: *Principle-Based Parsing: Computation and Psycholinguistics*. Ed. by Robert C. Berwick, Steven P. Abney and Carol Tenny. Studies in Linguistics and Philosophy. Dordrecht: Springer Netherlands, pp. 257-278. DOI: 10.1007/978-94-011-3474-3_10. URL: https://doi.org/10.1007/978-94-011-3474-3_10 (visited on 17/02/2020) (cit. on p. 105).

Ackerman, Lauren (2019). 'Syntactic and cognitive issues in investigating gendered coreference'. In: *Glossa*. DOI: 10.5334/gjgl.721. URL: https://eprint.ncl.ac.uk/248876 (visited on 23/01/2020) (cit. on p. 56).

Amsili, Pascal and Olga Seminck (2017). 'Schémas Winograd en français: une étude statistique et comportementale'. In: *Actes de la 24e Conférence sur le Traitement Automatique des Langues Naturelles*. TALN 2017. Orléans, France, June 2017. URL: https://hal.archives-ouvertes.fr/hal-01628342 (visited on 30/11/2019) (cit. on p. 18).

Antoine, Jean-Yves (2004). 'Résolutions des anaphores pronominales : quelques postulats du TALN mis à l'épreuve du dialogue oral finalisé'. In: *Actes de la 11ème Conférence sur le Traitement Automatique des Langues Naturelles*. TALN 2004. Fez, Morocco: Association pour le Traitement Automatique des Langues. URL: http://www.info.univ-tours.fr/~antoine/articles/2004_TALN.pdf (visited on 11/11/2019) (cit. on pp. 3, 17, 56).

Aone, Chinatsu and Scott William (1995). 'Evaluating Automated and Manual Acquisition of Anaphora Resolution Strategies'. In: *33rd Annual Meeting of the Association for Computational Linguistics*. ACL 1995. Cambridge, Massachusetts, USA: Association for Computational Linguistics, June 1995, pp. 122-129. DOI: 10.3115/981658.981675. URL: https://www.aclweb.org/anthology/P95-1017 (visited on 09/02/2020) (cit. on pp. 51, 60).

Asimov, Isaac (1950). *I, Robot*. Gnome (cit. on p. 1).

Ba, Jimmy Lei, Jamie Ryan Kiros and Geoffrey E. Hinton (2016). 'Layer Normalization'. In: *Barcelona, España, 8th Dec. 2016*. arXiv: 1607.06450. URL: http://arxiv.org/abs/1607.06450 (visited on 03/03/2019) (cit. on pp. 81, 82).

Bagga, Amit and Breck Baldwin (1998a). 'Algorithms for Scoring Coreference Chains'. In: *Proceedings of the First International Conference on Language Resources and Evaluation*. Workshop on Linguistics Coreference. Granada, España: European Language Resource Association, 25th May 1998, pp. 563-566. URL: Workshop%20on%20Linguistics%20Coreference (cit. on p. 23).

Bagga, Amit and Breck Baldwin (1998b). 'Entity-Based Cross-Document Coreferencing Using the Vector Space Model'. In: *Proceedings of COLING 1998: The 17th International Conference*
*on Computational Linguistics*. ACL-COLING 1998. Vol. 1. Montréal, Canada: Association for Computational Linguistics, 10th Aug. 1998. URL: https://www.aclweb.org/anthology/C98-1012 (visited on 30/11/2019) (cit. on p. 16).

Balazs, Jorge and Yutaka Matsuo (2019). ‘Gating Mechanisms for Combining Character and Word-level Word Representations: an Empirical Study’. In: *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics*. NAACL 2019 Student Research Workshop. Minneapolis, Minnesota, USA: Association for Computational Linguistics, June 2019, pp. 110–124. DOI: 10.18653/v1/N19-3016. URL: https://www.aclweb.org/anthology/N19-3016 (visited on 11/09/2019) (cit. on p. 79).

Baldwin, Breck, Tom Morton, Amit Bagga, Jason Baldridge, Raman Chandraseker, Alexis Di-mitriadis, Kieran Snyder and Magdalena Wolska (1998). ‘Description of the UPENN CAMP System as Used for Coreference’. In: *Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference Held in Fairfax, Virginia, April 29 - May 1, 1998*. MUC 1998. URL: https://www.aclweb.org/anthology/M98-1022 (visited on 19/12/2019) (cit. on p. 31).

Ballesteros, Miguel, Chris Dyer and Noah A. Smith (2015). ‘Improved Transition-based Parsing by Modeling Characters instead of Words with LSTMs’. In: *Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing*. EMNLP 2015. Lisbon, Portugal: Association for Computational Linguistics, pp. 349–359. DOI: 10.18653/v1/D15-1041. URL: http://aclweb.org/anthology/D15-1041 (visited on 03/09/2019) (cit. on p. 72).

Bański, Piotr, Bertrand Gaiffe, Patrice Lopez, Simon Meoni, Laurent Romary, Thomas Schmidt, Peter Stadler and Andreas Witt (2016). ‘Wake up, standOff!’ Sept. 2016. URL: https://hal.inria.fr/hal-01374102 (visited on 03/07/2017) (cit. on p. 44).

Barras, Claude, Edouard Geoffrois, Zhibiao Wu and Mark Liberman (1998). ‘Transcriber: a Free Tool for Segmenting, Labeling and Transcribing Speech’. In: *Proceedings of the First International Conference on Language Resources and Evaluation*. LREC 1998 (Granada, España). European Language Resources, May 1998. URL: http://trans.sourceforge.net/articles/Transcriber-LREC1998.pdf (visited on 06/07/2017) (cit. on pp. 39, 44).

Baude, Olivier and Céline Dugua (2011). ‘(Re)faire le corpus d’Orléans quarante ans après : quoi de neuf, linguiste ?’ In: *Corpus*. Varia 10.10 (1st Nov. 2011), pp. 99–118. URL: http://journals.openedition.org/corpus/2036 (visited on 10/01/2020) (cit. on p. 37).

Belkin, Mikhail, Daniel Hsu, Siyuan Ma and Soumik Mandal (2018). *Reconciling modern machine learning practice and the bias-variance trade-off*. 28th Dec. 2018. arXiv: 1812.11118 [cs, stat]. URL: http://arxiv.org/abs/1812.11118 (visited on 27/09/2019) (cit. on p. 81).

Bender, Emily M. (2019). ‘The #BenderRule: On Naming the Languages We Study and Why It Matters’. In: *The Gradient* (15th Sept. 2019). URL: https://thegridient.pub/the-benderrule-on-naming-the-languages-we-study-and-why-it-matters/ (visited on 20/01/2020) (cit. on pp. 26, 34).

Bengio, Yoshua, Réjean Ducharme, Pascal Vincent and Christian Janvin (2003). ‘A neural probabilistic language model’. In: *The Journal of Machine Learning Research* 3 (1st Mar. 2003), pp. 1137–1155. URL: http://dl.acm.org/citation.cfm?id=944919.944966 (visited on 27/09/2019) (cit. on pp. 54, 70).

Bengtson, Eric and Dan Roth (2008). ‘Understanding the Value of Features for Coreference Resolution’. In: *Proceedings of the 2008 Conference on Empirical Methods in Natural Language*
*Processing*. EMNLP 2008 (Honolulu, Hawai'i, USA). Association for Computational Linguistics, Oct. 2008, pp. 294–303. URL: https://www.aclweb.org/anthology/D08-1031 (visited on 29/01/2020) (cit. on pp. 52, 61, 85).

Bergstra, James S., Rémi Bardenet, Yoshua Bengio and Balázs Kégl (2011). ‘Algorithms for Hyper-Parameter Optimization’. In: *Advances in Neural Information Processing Systems*. NeurIPS 2011 (Granada, España). Ed. by J. Shawe-Taylor, R. S. Zemel, P. L. Bartlett, F. Pereira and K. Q. Weinberger. Vol. 24. Curran Associates, Inc., pp. 2546–2554. URL: http://papers.nips.cc/paper/4443-algorithms-for-hyper-parameter-optimization.pdf (visited on 06/10/2019) (cit. on p. 93).

Björkelund, Anders and Jonas Kuhn (2014). ‘Learning Structured Perceptrons for Coreference Resolution with Latent Antecedents and Non-local Features’. In: *Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics*. ACL 2014 (Baltimore, Maryland). Vol. 1. Association for Computational Linguistics, pp. 47–57. DOI: 10.3115/v1/P14-1005. URL: http://aclweb.org/anthology/P14-1005 (visited on 03/03/2019) (cit. on pp. 57, 62).

Bohnet, Bernd, Ryan McDonald, Gonçalo Simões, Daniel Andor, Emily Pitler and Joshua Maynez (2018). ‘Morphosyntactic Tagging with a Meta-BiLSTM Model over Context Sensitive Token Encodings’. In: *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics*. ACL 2018 (Melbourne, Australia). Vol. 1. Association for Computational Linguistics, July 2018, pp. 2642–2652. DOI: 10.18653/v1/P18-1246. URL: https://www.aclweb.org/anthology/P18-1246 (visited on 17/02/2020) (cit. on p. 103).

Bojanowski, Piotr, Edouard Grave, Armand Joulin and Tomas Mikolov (2017). ‘Enriching Word Vectors with Subword Information’. In: *Transactions of the Association for Computational Linguistics* 5, pp. 135–146. DOI: 10.1162/tacl_a_00051. URL: https://www.aclweb.org/anthology/Q17-1010 (visited on 16/02/2020) (cit. on p. 100).

Brassier, Maëlle, Alexis Puret, Augustin Voisin-Marras and Loïc Grobol (2018). ‘Classification par paires de mention pour la résolution des coréférences en français parlé interactif’. In: *Actes de la Conférence jointe CORIA-TALN-RJC 2018*. CORIA-TALN-RJC 2018. Rennes, France: Association pour le Traitement Automatique des Langues, 14th May 2018. URL: https://hal.inria.fr/hal-01821213/document (visited on 30/01/2019) (cit. on p. 67).

Broscheid, Samuel, Massimo Poesio, Simone Paolo Ponzetto, Kepa Joseba Rodriguez, Lorenza Romano, Olga Uryupina, Yannick Versley and Roberto Zanoli (2010). ‘BART: A Multilingual Anaphora Resolution System’. In: *Proceedings of the 5th International Workshop on Semantic Evaluation*. SemEval 2010 (Uppsala, Sverige). Association for Computational Linguistics, pp. 104–107. URL: http://dl.acm.org/citation.cfm?id=1859664.1859685 (visited on 29/03/2018) (cit. on pp. 50, 52, 105).

Brown, Gillian and George Yule (1983). *Discourse Analysis*. Cambridge University Press, 28th July 1983. 210 pp. Google Books: ZUnEAgAAQBAJ (cit. on p. 9).

Brown, Peter F., Vincent J. Della Pietra, Peter V. deSouza, Jenifer C. Lai and Robert L. Mercer (1992). ‘Class-Based n-gram Models of Natural Language’. In: *Computational Linguistics* 18.4, pp. 467–480. URL: https://www.aclweb.org/anthology/J92-4003 (visited on 16/02/2020) (cit. on p. 99).

Bruneseaux, Florence and Laurent Romary (1997). ‘Codage des références et coréférences dans les DHM’. In: *Proceedings of the Joint International Conference of the Association for Computers*
*and the Humanities and the Association for Literary & Linguistic Computing.* ACH-ALLC '97 (3rd-7th June 1997). Kingston, Ontario, Canada: Association for Computers and the Humanities. URL: https://hal.inria.fr/inria-00433399 (cit. on pp. 36, 43).

Buda, Mateusz, Atsuto Maki and Maciej A. Mazurowski (2018). 'A systematic study of the class imbalance problem in convolutional neural networks'. In: *Neural Networks* 106 (1st Oct. 2018), pp. 249-259. DOI: 10.1016/j.neunet.2018.07.011. URL: http://www.sciencedirect.com/science/article/pii/S0893608018302107 (visited on 21/09/2019) (cit. on p. 83).

Burnard, Lou (2014). *What is the Text Encoding Initiative?: How to add intelligent markup to digital resources*. Encyclopédie numérique. Marseille: OpenEdition Press, 11th Apr. 2014. 114 pp. URL: http://books.openedition.org/oep/426 (visited on 22/01/2020) (cit. on p. 43).

Cai, Jie and Michael Strube (2010). 'End-to-End Coreference Resolution via Hypergraph Partitioning'. In: *Proceedings of the 23rd International Conference on Computational Linguistics*. COLING 2010 (Beijing, China). Aug. 2010, pp. 143-151. URL: https://www.aclweb.org/anthology/C10-1017 (visited on 09/02/2020) (cit. on p. 59).

Calzolari, Nicoletta et al. (2010). 'The LREC Map of Language Resources and Technologies'. In: *Proceedings of the Seventh International Conference on Language Resources and Evaluation*. LREC 2010 (Valletta, Malta). European Language Resources Association, May 2010. URL: http://www.lrec-conf.org/proceedings/lrec2010/pdf/370_Paper.pdf (visited on 14/07/2020) (cit. on p. 35).

Candito, Marie and Djamé Seddah (2012). 'Le corpus Sequoia : annotation syntaxique et exploitation pour l'adaptation d'analyseur par pont lexical'. In: *Actes de la conférence conjointe JEP-TALN-RECITAL 2012*. TALN 2012 (Grenoble, France). Vol. 2. Association pour le Traitement Automatique des Langues, June 2012. URL: https://hal.inria.fr/hal-00698938 (visited on 17/02/2020) (cit. on p. 103).

Chen, Mia Xu et al. (2018). 'The Best of Both Worlds: Combining Recent Advances in Neural Machine Translation'. In: *Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics*. ACL 2018 (Melbourne, Australia). Vol. 1. Association for Computational Linguistics, pp. 76-86. URL: http://aclweb.org/anthology/P18-1008 (visited on 16/02/2019) (cit. on p. 83).

Chetlur, Sharan, Cliff Woolley, Philippe Vandermersch, Jonathan Cohen, John Tran, Bryan Catanzaro and Evan Shelhamer (2014). *cuDNN: Efficient Primitives for Deep Learning*. 3rd Oct. 2014. arXiv: 1410.0759 [cs]. URL: http://arxiv.org/abs/1410.0759 (visited on 30/09/2019) (cit. on p. 89).

Chinchor, Nancy (1992). 'MUC-4 Evaluation Metrics'. In: 4th Message Understanding Conference. McLean, Virginia, 16th June 1992. URL: https://www.aclweb.org/anthology/M92-1002 (visited on 30/11/2019) (cit. on p. 20).

Cho, Kyunghyun, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk and Yoshua Bengio (2014). 'Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation'. In: *Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing*. EMNLP 2014. Doha, Qatar: Association for Computational Linguistics, pp. 1724-1734. DOI: 10.3115/v1/D14-1179. URL: http://aclweb.org/anthology/D14-1179 (visited on 22/09/2019) (cit. on pp. 7, 71, 74, 79).
Choi, Heeyoul, Kyunghyun Cho and Yoshua Bengio (2017). 'Context-dependent word representation for neural machine translation'. In: *Computer Speech & Language* 45 (1st Sept. 2017), pp. 149–160. DOI: 10.1016/j.csl.2017.01.007. URL: http://www.sciencedirect.com/science/article/pii/S0885230816301024 (visited on 16/02/2020) (cit. on p. 101).

Chomsky, Noam (1981). *Lectures on Government and Binding: The Pisa Lectures*. Studies in Generative Grammar. Foris Publications (cit. on pp. 56, 106).

Clark, Kevin and Christopher D. Manning (2015). 'Entity-Centric Coreference Resolution with Model Stacking'. In: *Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing*. ACL-IJCNLP 2015 (Beijing, China), 26th–31st July 2015. Vol. 1. Association for Computational Linguistics, pp. 1405–1415. DOI: 10.3115/v1/P15-1136. URL: http://aclweb.org/anthology/P15-1136 (visited on 16/09/2019) (cit. on pp. 53, 64).

Clark, Kevin and Christopher D. Manning (2016a). 'Deep Reinforcement Learning for Mention-Ranking Coreference Models'. In: *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*. EMNLP 2016 (Austin, Texas, USA, 1st–4th Nov. 2016), pp. 2256–2262. URL: http://aclweb.org/anthology/D/D16/D16-1245.pdf (cit. on pp. 53, 61, 84, 109).

Clark, Kevin and Christopher D. Manning (2016b). 'Improving Coreference Resolution by Learning Entity-Level Distributed Representations'. In: *Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics*. ACL 2016 (Berlin, Deutschland, 7th–12th Aug. 2016). Vol. 1. Association for Computational Linguistics. URL: http://aclweb.org/anthology/P16/P16-1061.pdf (cit. on pp. 53–55, 64, 96, 98).

Clouzot, Catherine, Georges Antoniadis and Agnès Tutin (2000). 'Constitution and exploitation of an annotation system of electronic corpora: Toward automatic generation of understandable pronouns in French language'. In: *Natural Language Processing — NLP 2000*. Ed. by Dimitris N. Christodoulakis. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, pp. 242–251. DOI: 10.1007/3-540-45154-4_23 (cit. on p. 36).

Collobert, Ronan and Jason Weston (2008). 'A unified architecture for natural language processing: deep neural networks with multitask learning'. In: *Proceedings of the 25th international conference on Machine learning*. ICML 2008 (Helsinki, Suomi). ICML '08. Association for Computing Machinery, 5th July 2008, pp. 160–167. DOI: 10.1145/1390156.1390177. URL: https://doi.org/10.1145/1390156.1390177 (visited on 16/02/2020) (cit. on pp. 54, 99).

Collobert, Ronan, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu and Pavel P. Kuksa (2011). 'Natural Language Processing (almost) from Scratch'. In: *Journal of Machine Learning Research* 12 (12th Aug. 2011), 2493–2537. URL: https://dl.acm.org/doi/10.5555/1953048.2078186 (visited on 05/11/2016) (cit. on pp. 52, 54).

Cornuéjols, Antoine and Laurent Miclet (2010). *Apprentissage artificiel - Concepts et algorithmes*. 2nd ed. Eyrolles. URL: https://www.eyrolles.com/Sciences/Livre/apprentissage-artificiel-9782212124712 (visited on 13/01/2018) (cit. on p. 7).

Crabbé, Benoit, Murielle Fabre and Christophe Pallier (2019). 'Variable beam search for generative neural parsing and its relevance for the analysis of neuro-imaging signal'. In: *Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing*. EMNLP-IJCNLP 2019. Hong Kong, China: Association for Computational Linguistics, Nov. 2019, pp. 1150–1160. DOI:
10.18653/v1/D19-1106. URL: https://www.aclweb.org/anthology/D19-1106 (visited on 07/02/2020) (cit. on p. 112).

Cross, James and Liang Huang (2016). 'Span-Based Constituency Parsing with a Structure-Label System and Provably Optimal Dynamic Oracles'. In: *Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing*. EMNLP 2016 (Austin, Texas). Association for Computational Linguistics, pp. 1–11. DOI: 10.18653/v1/D16-1001. URL: http://aclweb.org/anthology/D16-1001 (visited on 02/03/2019) (cit. on pp. 4, 74).

Culotta, Aron, Michael Wick and Andrew McCallum (2007). 'First-Order Probabilistic Models for Coreference Resolution'. In: *Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference*. NAACL-HLT 2007. Rochester, New York: Association for Computational Linguistics, Apr. 2007, pp. 81–88. URL: https://www.aclweb.org/anthology/N07-1011 (visited on 29/09/2019) (cit. on pp. 52, 57, 85).

Dai, Andrew M and Quoc V Le (2015). 'Semi-supervised Sequence Learning'. In: *Advances in Neural Information Processing Systems*. NeurIPS 2015 (Montréal, Québec, Canada). Vol. 28. Curran Associates, Inc., pp. 3079–3087. URL: http://papers.nips.cc/paper/5949-semi-supervised-sequence-learning.pdf (visited on 16/02/2020) (cit. on p. 101).

Delaborde, Marine and Frédéric Landragin (2019). 'En quoi le pronom « on » a-t-il une valeur anaphorique ? Le cas des successions d'occurrences de « on »'. In: *Les cahiers de praxématique*. La gestion de l'anaphore en discours : complexités et enjeux 72 (June 2019), pp. 1–18. URL: https://hal.archives-ouvertes.fr/hal-02161902 (visited on 15/10/2019) (cit. on pp. 12, 20).

Denis, Pascal and Jason Baldridge (2007a). 'A Ranking Approach to Pronoun Resolution'. In: *Proceedings of the 20th International Joint Conference on Artificial Intelligence*. IJCAI 2007 (San Francisco, California, USA). Morgan Kaufmann, pp. 1588–1593. URL: http://dl.acm.org/citation.cfm?id=1625275.1625532 (visited on 19/07/2018) (cit. on p. 77).

Denis, Pascal and Jason Baldridge (2007b). 'Joint Determination of Anaphoricity and Coreference Resolution using Integer Programming'. In: *Human Language Technologies 2007: The Conference of the North American Chapter of the Association for Computational Linguistics; Proceedings of the Main Conference*. NAACL-HLT 2007 (Rochester, New York, USA). Association for Computational Linguistics, Apr. 2007, pp. 236–243. URL: https://www.aclweb.org/anthology/N07-1030 (visited on 26/09/2019) (cit. on pp. 78, 93).

Denis, Pascal and Jason Baldridge (2008). 'Specialized models and ranking for coreference resolution'. In: *Proceedings of the 2008 Conference on Empirical Methods in Natural Language Processing*. EMNLP 2008 (Honolulu, Hawai'i, USA). Association for Computational Linguistics, p. 660. DOI: 10.3115/1613715.1613797. URL: http://portal.acm.org/citation.cfm?doid=1613715.1613797 (visited on 26/09/2019) (cit. on pp. 61, 77, 78).

Denis, Pascal and Jason Baldridge (2009). 'Global joint models for coreference resolution and named entity classification'. In: *Procesamiento del lenguaje natural* 42 (Mar. 2009). URL: http://rua.ua.es/dspace/handle/10045/10549 (visited on 22/08/2019) (cit. on pp. 25, 61, 93).

Désoyer, Adèle, Frédéric Landragin and Isabelle Tellier (2015a). 'Apprentissage automatique d'un modèle de résolution de la coréférence à partir de données orales transcrites du français : le système CROC'. In: *Actes de la 22ème Conférence sur le Traitement Automatique des Langues*
*Naturelles*. TALN-RÉCITAL 2015 (Caen, France, 22nd–25th June 2015). June 2015, pp. 439–445.
URL: https://halshs.archives-ouvertes.fr/halshs-01162174 (visited on 31/01/2017)
(cit. on pp. 4, 65, 67, 77, 90).

Désoyer, Adèle, Frédéric Landragin, Isabelle Tellier, Anaïs Lefeuvre and Jean-Yves Antoine (2015b). 'Les coréférences à l'oral : une expérience d'apprentissage automatique sur le corpus ANCOR'. In: *Traitement Automatique des Langues*. Traitement automatique du langage parlé 55.2 (May 2015), pp. 97–121. URL: https://halshs.archives-ouvertes.fr/halshs-01153297 (visited on 31/01/2017) (cit. on pp. 39, 44).

Devlin, Jacob, Ming-Wei Chang, Kenton Lee and Kristina Toutanova (2018). *BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding*. 10th Oct. 2018. arXiv: 1810.04805 [cs]. URL: http://arxiv.org/abs/1810.04805 (visited on 16/02/2019) (cit. on pp. 4, 72, 73, 80, 81, 85, 99, 101, 102).

Dice, Lee R. (1945). 'Measures of the Amount of Ecologic Association Between Species'. In: *Ecology* 26.3, pp. 297–302. DOI: 10.2307/1932409. URL: https://esajournals.onlinelibrary.wiley.com/doi/abs/10.2307/1932409 (visited on 22/08/2019) (cit. on p. 24).

Diestel, Reinhard (2017). *Graph Theory*. 5th ed. Graduate Texts in Mathematics. Berlin Heidelberg: Springer-Verlag. DOI: 10.1007/978-3-662-53622-3. URL: https://www.springer.com/gp/book/9783662536216 (visited on 09/01/2020) (cit. on p. 7).

Dinarelli, Marco and Loïc Grobol (2018). 'Modélisation d'un contexte global d'étiquettes pour l'étiquetage de séquences dans les réseaux neuronaux récurrents'. In: *Actes de la onzième édition de la plate-forme Intelligence Artificielle*. Journée commune AFIA-ATALA sur le Traitement Automatique des Langues et l'Intelligence Artificielle pendant la onzième édition de la plate-forme Intelligence Artificielle. PFIA 2018. Nancy, France: As, 6th July 2018. URL: https://hal.archives-ouvertes.fr/hal-02002111/document (visited on 01/02/2019) (cit. on p. 71).

Doddington, George, Alexis Mitchell, Mark Przybocki, Lance Ramshaw, Stephanie Strassel and Ralph Weischedel (2004). 'The Automatic Content Extraction (ACE) Program : Tasks, Data, and Evaluation'. In: *Proceedings of the Fourth International Conference on Language Resources and Evaluation*. LREC-2004. Lisbon, Portugal: European Language Resources Association, May 2004. URL: http://www.lrec-conf.org/proceedings/lrec2004/pdf/5.pdf (visited on 06/07/2017) (cit. on pp. 14, 18, 30–32, 52, 64).

Dozat, Timothy (2016). 'Incorporating Nesterov Momentum into Adam'. In: Fourth International Conference on Learning Representations. San Juan, Puerto Rico, 2nd May 2016. URL: https://openreview.net/forum?id=OM0jvwB8jIp57ZJjtNEZ (visited on 19/08/2019) (cit. on p. 80).

Dozat, Timothy and Christopher D. Manning (2017). 'Deep Biaffine Attention for Neural Dependency Parsing'. In: *5th International Conference on Learning Representations*. ICLR 2017. Toulon, France: OpenReview.net, 24th Apr. 2017. URL: https://openreview.net/forum?id=Hk95PK9le (visited on 24/07/2019) (cit. on p. 77).

Dupont, Yoann (2017). 'La structuration dans les entités nommées'. PhD Thesis. Université Sorbonne Paris Cité, 23rd Nov. 2017. URL: http://www.theses.fr/2017USPCA100 (visited on 06/08/2019) (cit. on pp. 16, 106).

Durrett, Greg, David Hall and Dan Klein (2013). 'Decentralized Entity-Level Modeling for Coreference Resolution'. In: Proceedings of the 51st Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers). Aug. 2013, pp. 114–124. URL: https://www.aclweb.org/anthology/P13-1012/ (visited on 04/08/2019) (cit. on p. 61).

Durrett, Greg and Dan Klein (2013). 'Easy Victories and Uphill Battles in Coreference Resolution'. In: *Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing*. EMNLP 2013. Association for Computational Linguistics, pp. 1971–1982. URL: http://aclweb.org/anthology/D13-1203 (visited on 03/03/2019) (cit. on pp. 52–54, 60, 112).

Dyer, Chris, Adhiguna Kuncoro, Miguel Ballesteros and Noah A. Smith (2016). 'Recurrent Neural Network Grammars'. In: *Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*. NAACL: HLT 2016. NAACL: HLT 2016. San Diego, California, USA: Association for Computational Linguistics, pp. 199–209. DOI: 10.18653/v1/N16-1024. URL: http://aclweb.org/anthology/N16-1024 (visited on 12/03/2019) (cit. on p. 112).

Edmonds, Jack (1967). 'Optimum branchings'. In: *Journal of Research of the national Bureau of Standards B* 71.4, pp. 233–240 (cit. on p. 62).

Eger, Steffen, Paul Youssef and Iryna Gurevych (2018). 'Is it Time to Swish? Comparing Deep Learning Activation Functions Across NLP tasks'. In: 2018 Conference on Empirical Methods in Natural Language Processing. Brussels, Belgium: Association for Computational Linguistics, pp. 4415–4424. DOI: 10.18653/v1/D18-1472. URL: http://aclweb.org/anthology/D18-1472 (visited on 30/09/2019) (cit. on p. 88).

Ehrmann, Maud (2008). 'Named entities, from Linguistics to NLP: Theoretical status and disambiguation methods'. Theses. Université Paris Diderot, June 2008. URL: https://hal.archives-ouvertes.fr/tel-01639190 (visited on 21/11/2019) (cit. on pp. 15, 16).

Eshkol-Taravella, Iris, Olivier Baude, Denis Maurel, Linda Hriba, Céline Dugua and Isabelle Tellier (2011). 'Un grand corpus oral « disponible » : le corpus d'Orléans 1 1968-2012'. In: *Traitement Automatique des Langues*. Ressources Linguistiques Libres 53.2, pp. 17–46. URL: https://halshs.archives-ouvertes.fr/halshs-01163053 (visited on 08/01/2020) (cit. on p. 36).

Evert, Stefan and Andrew Hardie (2011). 'Twenty-first century Corpus Workbench: Updating a query architecture for the new millennium'. In: *Proceedings of the Corpus Linguistics 2011 conference*. CL2011. Birmingham, United Kingdom (cit. on p. 47).

Fauconnier, Gilles, George Lakoff and Eve Sweester (1994). *Mental spaces: aspects of meaning construction in natural language*. 2nd ed. Cambridge University Press (cit. on p. 9).

Fernandes, Eraldo, Cícero dos Santos and Ruy Milidiú (2012). 'Latent Structure Perceptron with Feature Induction for Unrestricted Coreference Resolution'. In: *Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning*. EMNLP-CoNLL 2012 (XXX, XXXX (Jeju Island, Republic of Korea)). Association for Computational Linguistics, July 2012, pp. 41–48. URL: https://www.aclweb.org/anthology/W12-4502 (visited on 09/02/2020) (cit. on p. 62).

Fernandes, Eraldo R. and Ulf Brefeld (2011). 'Learning from Partially Annotated Sequences'. In: *Machine Learning and Knowledge Discovery in Databases*. Ed. by Dimitrios Gunopulos, Thomas Hofmann, Donato Malerba and Michalis Vazirgiannis. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer, pp. 407–422. DOI: 10.1007/978-3-642-23780-5_36 (cit. on p. 62).
Fernandes, Eraldo Rezende, Cícero Nogueira dos Santos and Ruy Luiz Milidiú (2014). 'Latent Trees for Coreference Resolution'. In: *Computational Linguistics* 40.4 (Dec. 2014), pp. 801–835. DOI: 10.1162/COLI_a_00200. URL: https://www.aclweb.org/anthology/J14-4004 (visited on 09/02/2020) (cit. on p. 62).

Finkel, Jenny Rose and Christopher D. Manning (2008). 'Enforcing Transitivity in Coreference Resolution'. In: *Proceedings of the 46th Annual Meeting of the Association for Computational Linguistics on Human Language Technologies*. HLT 2008 (Columbus, Ohio, USA). Association for Computational Linguistics, June 2008, pp. 45–48. URL: https://www.aclweb.org/anthology/P08-2012 (visited on 30/11/2019) (cit. on p. 23).

Finkel, Jenny Rose and Christopher D. Manning (2009). 'Joint Parsing and Named Entity Recognition'. In: *Proceedings of Human Language Technologies: The 2009 Annual Conference of the North American Chapter of the Association for Computational Linguistics*. HLR: NAACL 2009 (Boulder, Colorado, USA). Association for Computational Linguistics, pp. 326–334. URL: http://dl.acm.org/citation.cfm?id=1620754.1620802 (visited on 26/03/2019) (cit. on p. 33).

Flaubert, Gustave (1910). *Salammbô*. Paris, France: Louis Conard. URL: https://fr.wikisource.org/wiki/Salammb%C3%B4 (cit. on p. 22).

Fort, Karën, Adeline Nazarenko and Ris Claire (2011). 'Corpus Linguistics for the Annotation Manager'. In: Corpus Linguistics 2011. Birmingham, United Kingdom, 20th July 2011. URL: https://hal.archives-ouvertes.fr/hal-00641571 (visited on 10/12/2019) (cit. on p. 30).

Fort, Karën, Adeline Nazarenko and Sophie Rosset (2012). 'Modeling the Complexity of Manual Annotation Tasks: a Grid of Analysis'. In: *Proceedings of COLING 2012*. COLING 2012. Mumbai, India: The COLING 2012 Organizing Committee, Dec. 2012, pp. 895–910. URL: https://www.aclweb.org/anthology/C12-1055 (visited on 20/12/2019) (cit. on p. 31).

French, Robert Matthew (1999). 'Catastrophic forgetting in connectionist networks'. In: *Trends in Cognitive Sciences* 3.4 (Apr. 1999), pp. 128–135. pmid: 10322466 (cit. on p. 85).

Gaier, Adam and David Ha (2019). *Weight Agnostic Neural Networks*. 10th June 2019. arXiv: 1906.04358 [cs, stat]. URL: http://arxiv.org/abs/1906.04358 (visited on 28/09/2019) (cit. on p. 80).

Gal, Yarin and Zoubin Ghahramani (2015). *A Theoretically Grounded Application of Dropout in Recurrent Neural Networks*. 16th Dec. 2015. arXiv: 1512.05287 [stat]. URL: http://arxiv.org/abs/1512.05287 (visited on 21/02/2019) (cit. on p. 81).

Galliano, Sylvain, Guillaume Gravier and Laura Chaubard (2009). 'The ESTER 2 evaluation campaign for the rich transcription of French radio broadcasts'. In: *Proceedings of the 10th Annual Conference of the International Speech Communication Association*. InterSpeech 2009 (Brighton, United Kingdom). International Speech Communication Association (cit. on p. 106).

Gardent, Claire and Hélène Manuélian (2005). 'Création d'un corpus annoté pour le traitement des descriptions définies'. In: *Trajetoy Automatique des Langues*. Modèles et algorithmes pour la résolution d'anaphores 1.46, pp. 115–140. URL: https://members.loria.fr/CGardent/publis/tal05-dede.pdf (visited on 07/01/2020) (cit. on p. 36).

Gillick, Daniel, Benoit Favre, Dilek Hakkani-Tür, Berndt Bohnet, Yang Liu and Shasha Xie (2009). 'The ICSI/UTD Summarization System at TAC 2009'. In: *Proceedings of the Text Analysis*
*Conference workshop*. TAC 2009 (Gaithersburg, Maryland, USA). URL: https://hal-amu.archives-ouvertes.fr/hal-01194277 (visited on 20/01/2020) (cit. on p. 33).

Godbert, Elisabeth and Benoît Favre (2017). ‘Détection de coréférences de bout en bout en français’. In: *Actes de la 24e Conférence sur le Traitement Automatique des Langues Naturelles. TALN 2017*. Orléans, France: Association pour le Traitement Automatique des Langues, June 2017. URL: https://hal.archives-ouvertes.fr/hal-01687116 (cit. on pp. 65, 67).

Godin, Frédéric (2019). ‘Improving and Interpreting Neural Networks for Word-Level Prediction Tasks in Natural Language Processing’. PhD Thesis. Gent, België: Universiteit Gent, July 2019. 214 pp. URL: https://fredericgodin.com/wp-content/uploads/2019/07/phd_frederic_godin_book.pdf (visited on 17/02/2020) (cit. on p. 103).

Goldberg, Yoav and Joakim Nivre (2012). ‘A Dynamic Oracle for Arc-Eager Dependency Parsing’. In: *Proceedings of the 24th International Conference on Computational Linguistics*. COLING 2012 (Mumbai, India). Dec. 2012, pp. 959–976. URL: https://www.aclweb.org/anthology/C12-1059 (visited on 10/02/2020) (cit. on p. 63).

Grave, Edouard, Piotr Bojanowski, Prakhar Gupta, Armand Joulin and Tomáš Mikolov (2018). ‘Learning Word Vectors for 157 Languages’. In: *Proceedings of the 11th International Conference on Language Resources and Evaluation*. LREC 2018 (Miyazaki, Japan). European Language Resource Association. URL: http://aclweb.org/anthology/L18-1550 (visited on 03/03/2019) (cit. on pp. 87, 99, 100).

Graves, Alex and Jürgen Schmidhuber (2005). ‘Framewise phoneme classification with bidirectional LSTM and other neural network architectures’. In: *Neural Networks* 18.5-6, pp. 602–610. DOI: 10.1016/j.neunet.2005.06.042. pmid: 16112549 (cit. on pp. 6, 73).

Gravier, Guillaume, Gilles Adda, Niklas Paulson, Matthieu Carré, Aude Giraudel and Olivier Galibert (2012). ‘The ETAPE corpus for the evaluation of speech-based TV content processing in the French language’. In: *LREC - Eighth international conference on Language Resources and Evaluation*. Turkey, na. URL: https://hal.archives-ouvertes.fr/hal-00712591 (visited on 18/02/2020) (cit. on p. 106).

Grice, Herbert Paul (1989). *Studies in the Way of Words*. Harvard University Press. 406 pp. (cit. on p. 10).

Grobol, Loïc (2019). ‘Neural Coreference Resolution with Limited Lexical Context and Explicit Mention Detection for Oral French’. In: *Proceedings of the Second Workshop on Computational Models of Reference, Anaphora and Coreference*. CRAC 2019 (Minneapolis, Minnesota). June 2019, pp. 8–14. URL: https://www.aclweb.org/anthology/papers/W/W19/W19-2802/ (visited on 24/06/2019) (cit. on pp. 86, 89, 90).

Grobol, Loïc, Frédéric Landragin and Serge Heiden (2017a). ‘Interoperable annotation of (co)references in the Democrat project’. In: *Proceedings of the thirteenth Joint ISO-ACL Workshop on Interoperable Semantic Annotation*. ISA13 (Montpellier, France). Ed. by Harry Bunt. ACL Special Interest Group on Computational Semantics (SIGSEM) and ISO TC 37/SC 4 (Language Resources) WG 2, Sept. 2017. URL: https://hal.archives-ouvertes.fr/hal-01583527 (cit. on p. 41).

Grobol, Loïc, Frédéric Landragin and Serge Heiden (2018a). ‘XML-TEI-URS: using a TEI format for annotated linguistic resources’. In: *CLARIN Annual Conference 2018* (Pisa, Italia). Oct.
2018. URL: https://hal.archives-ouvertes.fr/hal-01827563 (visited on 30/01/2019)
(cit. on p. 41).

Grobol, Loïc, Isabelle Tellier, Éric Villemonte De La Clergerie, Marco Dinarelli and Frédéric Landragin (2017b). 'Apports des analyses syntaxiques pour la détection automatique de mentions dans un corpus de français oral'. In: *Actes de la 24e Conférence sur le Traitement Automatique des Langues Naturelles*. TALN 2017 (Orléans, France). Association pour le Traitement Automatique des Langues, June 2017. URL: https://hal.inria.fr/hal-01558711 (cit. on pp. 68, 89).

Grobol, Loïc, Isabelle Tellier, Éric Villemonte De La Clergerie, Marco Dinarelli and Frédéric Landragin (2018b). 'ANCOR-AS: Enriching the ANCOR Corpus with Syntactic Annotations'. In: *Proceedings of the 11th edition of the Language Resources and Evaluation Conference*. LREC 2018 (Miyazaki, Japan). European Language Resource Association, May 2018. URL: https://hal.inria.fr/hal-01744572 (cit. on pp. 41, 45, 107).

Grönroos, Stig-Arne, Sami Virpioja, Peter Smit and Mikko Kurimo (2014). 'Morfessor Flat-Cat: An HMM-Based Method for Unsupervised and Semi-Supervised Learning of Morphology'. In: *Proceedings of COLING 2014, the 25th International Conference on Computational Linguistics: Technical Papers*. COLING 2014 (Baile Átha Cliath, Éire). Dublin City University and Association for Computational Linguistics, Aug. 2014, pp. 1177-1185. URL: https://www.aclweb.org/anthology/C14-1111 (visited on 16/02/2020) (cit. on p. 99).

Habert, Benoît (2000). 'Détournements d'annotation : armer la main et le regard'. In: *Corpus: méthodologie et applications linguistiques*. Ed. by Mireille Bilger. 1 vols. Paris, France: Honoré Champion, pp. 106-120 (cit. on p. 30).

Hachey, Ben, Will Radford, Joel Nothman, Matthew Honnibal and James R. Curran (2013). 'Evaluating Entity Linking with Wikipedia'. In: *Artificial Intelligence*. Artificial Intelligence, Wikipedia and Semi-Structured Resources 194 (1st Jan. 2013), pp. 130-150. DOI: 10.1016/j.artint.2012.04.005. URL: http://www.sciencedirect.com/science/article/pii/S0004370212000446 (visited on 30/11/2019) (cit. on p. 16).

Haghighi, Aria and Dan Klein (2009). 'Simple Coreference Resolution with Rich Syntactic and Semantic Features'. In: *Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing*. EMNLP 2009 (Singapore). Association for Computational Linguistics, Aug. 2009, pp. 1152-1161. URL: https://www.aclweb.org/anthology/D09-1120 (visited on 30/01/2020) (cit. on p. 52).

Haixiang, Guo, Li Yijing, Jennifer Shang, Gu Mingyun, Huang Yuanyue and Gong Bing (2017). 'Learning from class-imbalanced data: Review of methods and applications'. In: *Expert Systems with Applications* 73 (1st May 2017), pp. 220-239. DOI: 10.1016/j.eswa.2016.12.035. URL: http://www.sciencedirect.com/science/article/pii/S0957417416307175 (visited on 21/09/2019) (cit. on p. 83).

Hanson, Stephen José and Lorien Y. Pratt (1988). 'Comparing Biases for Minimal Network Construction with Back-propagation'. In: *Proceedings of the 1st International Conference on Neural Information Processing Systems*. NIPS'88. Cambridge, MA, USA: MIT Press, pp. 177-185. URL: http://dl.acm.org/citation.cfm?id=2969735.2969756 (visited on 06/09/2019) (cit. on p. 82).
Hatmi, Mohamed (2014). 'Named entity recognition in multimodal documents'. Theses. Université de Nantes, Jan. 2014. URL: https://hal.archives-ouvertes.fr/tel-01154811 (visited on 18/02/2020) (cit. on p. 106).

Heiden, Serge (2019). *Manuel de TXM, Extension Annotation URS (Unité-Relation-Schéma) version 1.0*. Manual. 3rd July 2019. DOI: 10.5281/zenodo.3267345. URL: https://zenodo.org/record/3267345 (visited on 23/01/2020) (cit. on p. 47).

Heiden, Serge, Jean-Philippe Magué and Bénédicte Pincemin (2010). 'TXM : Une plateforme logicielle open-source pour la textométrie - conception et développement'. In: *10th International Conference on the Statistical Analysis of Textual Data*. JADT 2010. Ed. by Luca Giuliano Sergio Bolasco Isabella Chiari. Vol. 2. Roma, Italia: Edizioni Universitarie di Lettere Economia Diritto, May 2010, pp. 1021-1032. URL: https://halshs.archives-ouvertes.fr/halshs-00549779 (visited on 06/07/2017) (cit. on pp. 40, 46).

Heinzerling, Benjamin and Michael Strube (2019). 'Sequence Tagging with Contextual and Non-Contextual Subword Representations: A Multilingual Evaluation'. In: *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*. ACL 2019 (Firenze, Italia). Florence, Italy: Association for Computational Linguistics, July 2019, pp. 273-291. DOI: 10.18653/v1/P19-1027. URL: https://www.aclweb.org/anthology/P19-1027 (visited on 17/02/2020) (cit. on p. 103).

Hendrickx, Iris, Gosse Bouma, Walter Daelemans and Véronique Hoste (2013). 'COREA: Coreference Resolution for Extracting Answers for Dutch'. In: *Essential Speech and Language Technology for Dutch: Results by the STEVIN programme*. Ed. by Peter Spyns and Jan Odijk. Theory and Applications of Natural Language Processing. Berlin, Heidelberg: Springer, pp. 115-128. DOI: 10.1007/978-3-642-30910-6_7. URL: https://doi.org/10.1007/978-3-642-30910-6_7 (visited on 19/01/2020) (cit. on p. 33).

Hinrichs, Erhard, Sandra Kübler, Karin Naumann, Heike Telljohann and Julia Trushkina (2004). 'Recent developments in linguistic annotations of the TüBa-D/Z treebank'. In: *Proceedings of the Third Workshop on Treebanks and Linguistic Theories*. TLT 2004 (10th-11th Dec. 2004). Ed. by Sarah Kübler, Joakim Nivre, Erhard Hinrichs and Wolger Wunsch. Tübingen, Deutschland, pp. 51-62 (cit. on p. 32).

Hinton, Geoffrey E. (1987). 'Learning translation invariant recognition in a massively parallel networks'. In: *PARLE Parallel Architectures and Languages Europe*. Ed. by J. W. de Bakker, A. J. Nijman and P. C. Treleaven. Lecture Notes in Computer Science. Springer Berlin Heidelberg, pp. 1-13 (cit. on p. 82).

Hinton, Geoffrey E., Nitish Srivastava, Alex Krizhevsky, Ilya Sutskever and Ruslan R. Salakhutdinov (2012). *Improving neural networks by preventing co-adaptation of feature detectors*. 3rd July 2012. arXiv: 1207.0580 [cs]. URL: http://arxiv.org/abs/1207.0580 (visited on 06/09/2019) (cit. on p. 81).

Hirschman, Lynette and Nancy Chinchor (1998). 'Appendix F: MUC-7 Coreference Task Definition (version 3.0)'. In: *Seventh Message Understanding Conference (MUC-7): Proceedings of a Conference*. Fairfax, Virginia, 29th Apr. 1998. URL: https://www.aclweb.org/anthology/M98-1029 (visited on 04/08/2019) (cit. on pp. 21, 64).
Hirst, G. (1981). *Anaphora in Natural Language Understanding: A Survey*. Lecture Notes in Computer Science. Berlin Heidelberg: Springer-Verlag. DOI: 10.1007/3-540-10858-0. URL: https://www.springer.com/fr/book/9783540108580 (visited on 24/11/2019) (cit. on p. 19).

Hobbs, Jerry R. (1986). 'Resolving Pronoun References'. In: *Readings in Natural Language Processing*. Ed. by Barbara J. Grosz, Karen Sparck-Jones and Bonnie Lynn Webber. San Francisco, California, USA: Morgan Kaufmann, pp. 339–352. URL: http://dl.acm.org/citation.cfm?id=21922.24343 (cit. on pp. 2, 29, 52).

Hochreiter, Sepp and Jürgen Schmidhuber (1997). 'Long Short-Term Memory'. In: *Neural Computation* 9.8 (1st Nov. 1997), pp. 1735–1780. DOI: 10.1162/neco.1997.9.8.1735. URL: https://www.mitpressjournals.org/doi/10.1162/neco.1997.9.8.1735 (visited on 13/09/2019) (cit. on pp. 6, 54, 79).

Honnibal, Matthew and Ines Montani (2019). *spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing*. Version 2.2.5. Explosion. URL: https://spacy.io (cit. on p. 103).

Hovy, Eduard, Mitchell Marcus, Martha Palmer, Lance Ramshaw and Ralph Weischedel (2006). 'OntoNotes: The 90% Solution'. In: *Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics*. HLT NAACL 06. Vol. Short papers. New York City, USA: Association for Computational Linguistics, June 2006, pp. 57–60. URL: https://www.aclweb.org/anthology/N06-2015 (visited on 18/12/2019) (cit. on pp. 5, 14, 30–33, 41).

Ioffe, Sergey and Christian Szegedy (2015). *Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift*. 10th Feb. 2015. arXiv: 1502.03167 [cs]. URL: http://arxiv.org/abs/1502.03167 (visited on 10/09/2019) (cit. on pp. 81, 82).

Isard, Amy, David McKelvie, Andreas Mengel and Morten Baun Møller (2000). 'The MATE Workbench: A Tool for Annotating XML Corpora'. In: *Content-Based Multimedia Information Access*. RIAO 00 (Paris, France). Vol. 1. Centre de hautes études internationales d'informatique documentaire, pp. 411–425. URL: http://dl.acm.org/citation.cfm?id=2835865.2835908 (cit. on p. 40).

ISO/TC 37/SC 4/WG 2 (2017). *ISO AWI 24617-9 Language resource management – Part 9 Semantic annotation framework (SemAF)*. Reference. Geneva, Switzerland: International Organization for Standardization. URL: https://www.iso.org/standard/69658.html (cit. on p. 44).

Jaccard, Paul (1912). 'The Distribution of the Flora in the Alpine Zone'. In: *New Phytologist* 11.2, pp. 37–50. DOI: 10.1111/j.1469-8137.1912.tb05611.x. URL: https://nph.onlinelibrary.wiley.com/doi/abs/10.1111/j.1469-8137.1912.tb05611.x (visited on 16/02/2020) (cit. on p. 98).

Joshi, Mandar, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer and Omer Levy (2019a). *SpanBERT: Improving Pre-training by Representing and Predicting Spans*. 24th July 2019. arXiv: 1907.10529 [cs]. URL: http://arxiv.org/abs/1907.10529 (visited on 25/07/2019) (cit. on pp. 54, 55, 61, 73).

Joshi, Mandar, Omer Levy, Daniel S. Weld and Luke Zettlemoyer (2019b). *BERT for Coreference Resolution: Baselines and Analysis*. 24th Aug. 2019. arXiv: 1908.09091 [cs]. URL: http://arxiv.org/abs/1908.09091 (visited on 01/09/2019) (cit. on pp. 54, 55, 61, 73, 101, 102).