Package: scorch
Scorch¹This is an alternative implementation of the coreference scorer for the CoNLL-2011/2012 shared tasks
on coreference resolution.It aims to be more straightforward than thereference implementation, while
maintaining as much compatibility with it as possible.The implementations of the various scores are as close as possible from the formulas used byPradhan et al. (2014), with the edge cases for BLANC taken fromRecasens and Hovy (2011).1.Scorer forcoreferencechains.Usagescorchgold.jsonsys.jsonout.txtInstallFrom the cheeseshoppython3-mpipinstall--userscorchOr directly from gitpython3-mpipinstall--usergit+https://github.com/LoicGrobol/scorch.gitFormatsSingle documentThe input files should be JSON files with a"type"key at top-levelIf"type"is"graph", then top-level should have at top-levelA"mentions"key containing a list of all mention identifiersA"links"key containing a list of pairs of corefering mention identifiersIf"type"is"clusters", then top-level should have a"clusters"key containing a mapping
from clusters ids to cluster contents (as lists of mention identifiers).Of course the system and gold files should use the same set of mention identifiers for the mentions
they have in common.For convenience, theconll.pyconverts CoNLL-2012 files to this format.Multiple documentsIf the inputs are directories, files with the same base name (excluding extension) as those present
in the sys directory are expected to be present in the gold directory, with exactly one gold file
for each sys file.
In that case, the output scores will be the micro-average of the individual files
scores, ie their arithmetic means weighted by the relative numbers ofGold mentions for RecallSystem mentions for PrecisionThe sum of the previous two for F₁This is different from the reference interpretation whereMUCweighting ignores mentions in singleton entitiesThis should not make any difference for the CoNLL-2012 dataset, since singleton entities are not
annotated.For datasets with singletons, the shortcomings of MUC are well known, so this score shouldn't
matter muchBLANCis calculated by micro-averaging coreference and non-coreference separately, using
the number of links as weights instead of the number of mentions.This is roughly equivalent to weighting coreference scores per document by their number of
non-singleton clusters and non-coreference scores by the square of their number of mentions.
This give disproportionate importance to large documents, which is not desirable
in heterogenous corporaThe CoNLL average score is the arithmetic mean of the global MUC, B³ and CEAFₑ F₁ scores.SourcesScoring Coreference Partitions of Predicted Mentions: A Reference
Implementation.Sameer Pradhan, Xiaoqiang Luo, Marta Recasens, Eduard Hovy, Vincent Ng and
Michael Strube.Proceedings of the 52nd Annual Meeting of the Association for Computational
Linguistics, Baltimore, MD, June 2014. (pdf)BLANC: Implementing the Rand Index for Coreference Evaluation.Marta Recasens and Eduard Hovy In:Natural Language Engineering17 (4). Cambridge University
Press, 2011.
(pdf)An Extension of BLANC to System Mentions.Xiaoqiang Luo, Sameer
Pradhan, Marta Recasens and Eduard Hovy.Proceedings of the 52nd Annual Meeting of the
Association for Computational Linguistics, Baltimore, MD, June 2014.
(pdf)The version of the Kuhn-Munkres algorithm used for the CEAF scores usesscipy.optimize.linear_sum_assignment,
with$-ϕ_n$as cost function.LicenseUnless otherwise specified (seebelow), the following licence (the
so-called “MIT License”) applies to all the files in this repository.
See alsoLICENSE.md.Copyright 2018 Loïc Grobol <loic.grobol@gmail.com>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and
associated documentation files (the "Software"), to deal in the Software without restriction,
including without limitation the rights to use, copy, modify, merge, publish, distribute,
sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or
substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT
NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT
OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.License exceptionsThe reference scorer testcases located intests/fixtures/conll/are distributed under theCreative
Commons Attribution ShareAlike 4.0 International
LicenseCopyright© 2011- Sameer Pradhan pradhan <at> cemantix.orgAuthorsEmili Sapena, Universitat Politècnica de Catalunya,http://www.lsi.upc.edu/~esapena, esapena
<at> lsi.upc.eduSameer Pradhan,http://cemantix.org, pradhan <at> cemantix.orgSebastian Martschat, sebastian.martschat <at> h-its.orgXiaoqiang Luo, xql <at> google.comOriginhttp://conll.github.io/reference-coreference-scorersThese files are taken verbatim from the 8.0.1 of the official CoNLL scorer athttps://github.com/conll/reference-coreference-scorers/releases