SlowGuess/ACL_2020_OCR huggingface dataset:
Method To evaluate the distance metric learnt by the model we use the silhouette coefficient (Rousseeuw, 1987), which is commonly used for intrinsic clustering evaluation. The silhouette coefficient for each pronoun  $\mathbf{x}$  is defined as in Eq. (9), where  $a$  is the mean distance between  $\mathbf{x}$  and all other items in the same chain, and  $b$  is the mean distance between  $\mathbf{x}$  and all other items in the closest chain (measured in the learnt space, not in terms of linear distance). Its range is  $[-1, 1]$ , with 1 corresponding to the pronoun being much closer to the other pronouns in its chain, 0 being borderline (equally close to the two compared chains), and -1 being much closer to the pronouns in the other chain. The average silhouette coefficient is used as an overall measure of clustering quality. A score below 0.25 is usually deemed a null result (Kaufman and Rousseeuw, 1990).
$$
s = \frac {b - a}{\operatorname* {m a x} (a , b)} \tag {9}
$$
The probe model is trained for 50 epochs, keeping the model at the best validation epoch, i.e., where the silhouette score over the validation data is highest.
In addition to the trained probe model, we provide the results on global entity discrimination for the unsupervised baseline which computes the cosine similarity between the non-transformed hidden representations of the language models, similarly to the first probe task.
Results and Discussion All the obtained values are well below 0.25. Table 6 contains the results for all the datapoints as well as divided into easy and difficult documents. In easy documents, all the chains have different pronouns, so they can be distinguished by the token form only. Difficult documents contain confusable chains, that is,
there are at least two different chains which share the same pronoun. Coefficients are a bit higher for easy documents, but still very low, and, for complex documents, they are virtually zero. Moreover, the supervised models performs marginally better than the cosine baselines, but clearly do not learn any reliable information.
LSTM	Transformer
N	unsup	sup	unsup	sup
all	1142	-0.09	0.02	-0.08	0.03
easy	194	0.12	0.14	0.13	0.16
diff	948	-0.13	-0.007	-0.13	0.01
Table 6: Results for the second probe task (average silhouette coefficient).
Indeed, the average distances within and across chains seem to confirm these results. If models were capturing global entity-related properties in their mention representations, we would expect pronouns with the same form but in different chains to be further away than pronouns (of any form) that belong to the same chain; instead, they are at the same distance (average cosines of 0.75 / 0.76 for Transformer, 0.74 / 0.73 for LSTM, respectively).
We conclude that the models' sensitivity to whether two identical pronouns belong to the same chain or not only shows if linear distance is factored out (as in the main text). If it is not, as in the current experiment, the models fail completely at distinguishing entities.