Abstract: Coreference evaluation metrics are hard to optimize directly as they are non-differentiable functions, not easily decomposable into elementary decisions. Consequently, most approaches optimize objectives only indirectly related to the end goal, resulting in suboptimal performance. Instead, we propose a differentiable relaxation that lends itself to gradient-based optimisation, thus bypassing the need for reinforcement learning or heuristic modification of cross-entropy. We show that by modifying the training objective of a competitive neural coreference system, we obtain a substantial gain in performance. This suggests that our approach can be regarded as a viable alternative to using reinforcement learning or more computationally expensive imitation learning.

Introduction: Coreference resolution is the task of identifying all mentions which refer to the same entity in a document. It has been shown beneficial in many natural language processing (NLP) applications, including question answering BIBREF0 and information extraction BIBREF1 , and often regarded as a prerequisite to any text understanding task. Coreference resolution can be regarded as a clustering problem: each cluster corresponds to a single entity and consists of all its mentions in a given text. Consequently, it is natural to evaluate predicted clusters by comparing them with the ones annotated by human experts, and this is exactly what the standard metrics (e.g., MUC, B INLINEFORM0 , CEAF) do. In contrast, most state-of-the-art systems are optimized to make individual co-reference decisions, and such losses are only indirectly related to the metrics. One way to deal with this challenge is to optimize directly the non-differentiable metrics using reinforcement learning (RL), for example, relying on the REINFORCE policy gradient algorithm BIBREF2 . However, this approach has not been very successful, which, as suggested by clark-manning:2016:EMNLP2016, is possibly due to the discrepancy between sampling decisions at training time and choosing the highest ranking ones at test time. A more successful alternative is using a `roll-out' stage to associate cost with possible decisions, as in clark-manning:2016:EMNLP2016, but it is computationally expensive. Imitation learning BIBREF3 , BIBREF4 , though also exploiting metrics, requires access to an expert policy, with exact policies not directly computable for the metrics of interest. In this work, we aim at combining the best of both worlds by proposing a simple method that can turn popular coreference evaluation metrics into differentiable functions of model parameters. As we show, this function can be computed recursively using scores of individual local decisions, resulting in a simple and efficient estimation procedure. The key idea is to replace non-differentiable indicator functions (e.g. the member function INLINEFORM0 ) with the corresponding posterior probabilities ( INLINEFORM1 ) computed by the model. Consequently, non-differentiable functions used within the metrics (e.g. the set size function INLINEFORM2 ) become differentiable ( INLINEFORM3 ). Though we assume that the scores of the underlying statistical model can be used to define a probability model, we show that this is not a serious limitation. Specifically, as a baseline we use a probabilistic version of the neural mention-ranking model of P15-1137, which on its own outperforms the original one and achieves similar performance to its global version BIBREF5 . Importantly when we use the introduced differentiable relaxations in training, we observe a substantial gain in performance over our probabilistic baseline. Interestingly, the absolute improvement (+0.52) is higher than the one reported in clark-manning:2016:EMNLP2016 using RL (+0.05) and the one using reward rescaling (+0.37). This suggests that our method provides a viable alternative to using RL and reward rescaling. The outline of our paper is as follows: we introduce our neural resolver baseline and the B INLINEFORM0 and LEA metrics in Section SECREF2 . Our method to turn a mention ranking resolver into an entity-centric resolver is presented in Section SECREF3 , and the proposed differentiable relaxations in Section SECREF4 . Section SECREF5 shows our experimental results.

Question: How long is the training dataset?
Answer: 3492