SlowGuess/NAACL_2022_OCR huggingface dataset:
train our model from scratch on the AIDA-CoNLL training set and evaluate on the development set, keeping hyperparameters constant. Surprisingly, the performance of our model is strong in this limited data setting, which means that our model is not dependent on a large set of training examples when there is a small amount of annotated in-domain data. Note that for "w/o 128 standard relations" we collapse all standard relations into the OTHER special relation; and for "w/o RE transformer" we replaced the RE transformer with a single bilinear layer. Our results (Table 7) indicate that all aspects
Table 6: Our model hyperparameters.
Method	AIDA
KBED	94.37
w/o KB	92.20
w/o SAME AS relation	93.65
w/o OTHER relation	94.23
w/o 128 standard relations (collapsed)	93.67
w/o RE transformer	94.06
w/o weighting facts by entity scores	94.23
w/o weighting facts by relation scores	93.44
w/o reflexive RE	93.84
w/o entity descriptions	93.89
w/o entity types	93.47
w/o entity priors	93.63
w/o task hidden layers	94.07
w/o negative relation scores	94.19
with Wikipedia ED pre-training	95.58
Table 7: ED F1 score on AIDA-CoNLL development split for model ablations trained from scratch on AIDA-CoNLL training split using the standard CoNLL candidates (Hoffart et al., 2011). The result is in red when the performance drops by more than 0.7.
of our model that we measured have a positive impact on performance. Interestingly, the KB module (+2.2 F1) has a greater impact than the entity description (+0.48 F1) and entity typing (+0.9 F1) modules despite weaker performance when used on its own (Table 1). This implies there is less overlap between examples where KB module performs well, and the other modules perform well. We observe, the SAME AS relation improves performance by 0.72 F1, which demonstrates that using coreference improves ED. Finally, we find that when the KB module has greater control over how to weight KB facts (based on the context) it leads to better results, for example if we collapse all standard relations into a single relation our performance drops by 0.7 F1.
D Doc-level RE results on DOCRED
To verify the performance of our document-level RE architecture introduced in Section 3.5, we present results of models trained and evaluated on the DOCRED dataset (Yao et al., 2019). Our baseline implementation uses roberta-base as an encoder and a bilinear output layer. We show two variants in Table 8, a bilinear layer with input dimension 128 and with input dimension 256, which give an F1 score of 57.8 and 58.4 respectively. This compares to a score of 59.5 for an equivalent baseline implemented in (Xu et al., 2021). The difference is explained by our baseline not giving the model access to the gold coreference information, which is allowed in the DOCRED task but which we exclude as it will not be available for our entity linking task.
F1	Train time (seconds per epoch)
Baseline (bilinear)- SSAN imp.	59.5	-
SSAN (roberta-base)	60.9	-
Baseline (bilinear-128)	57.8	155.7
Baseline (bilinear-256)	58.4	343.2
Coarse to fine (2 layers)	60.4	100.6
Coarse to fine (4 layers)	61.2	106.2
Our coarse-to-fine approach, with 4 "fine" transformer layers, pushes the dev-level F1 up by 2.8 F1 to 61.2. This puts it slightly above the roberta-base version of the current state-of-the-art model, SSAN (Xu et al., 2021), which scores 60.9, and additionally has access to the gold coreference labels in the embedding layer of the model. This validates that our document-level RE architecture is capable of producing accurate relation predictions, which we see in the main results table (Table 1) also translates into stronger ED performance.
By avoiding the bilinear layer, our implementation is also faster to train, achieving 106.2 seconds per epoch on the DOCRED dataset on a single Tesla V100 GPU, compared to 155.7 seconds for the baseline model with a 128-dimension bilinear layer, and 343.2 seconds for the more accurate baseline model with a 256 dimension bilinear layer.
We present the topic, number of documents and number of mentions for each dataset used for evaluation (Table 9). The datasets used cover a variety
of sources including wikipedia text, news articles, web text and tweets. Note that the performance of the model outside these domains may be significantly different.
Table 8: Document-level relation extraction F1 scores on the DOCRED dev dataset.
Topic	Num docs	Num Mentions
AIDA	news	231	4464
MSNBC	news	20	656
AQUAINT	news	50	743
ACE2004	news	57	259
CWEB	web	320	11154
WIKI	Wikipedia	320	6821
ShadowLink-ALL	web	2712	2712
Table 9: Dataset statistics for entity disambiguation datasets.
E.2 ShadowLink Full Text versions
The authors of Provatorova et al. (2021) kindly provided us with the full documents from which the shorter text snippets (usually one or two sentences) in the ShadowLink dataset were sourced. We were able to match 596 of the 904 examples in the SHADOW split to its corresponding document, and 530 out of the 904 examples in the TOP split. As some full articles were extremely long we limited the document-length to 10000 characters, centred around the single annotated entity. To validate that the subset of examples we were able to match to full documents were representative of the original dataset splits, we ran our model on the sentence-level versions of these subsets, achieving 47.7 on the SHADOW split (comparable to 47.6 in Table 2) and 63.9 on the TOP split (comparable to 64.2 in Table 2).
F Additional relation analysis
To expand on the analysis in Section 6.1 we also include the number of gold and predicted relations in documents in the AIDA dataset (Table 10). The first clear difference is that there is a far higher count of gold SAME AS facts in the AIDA dataset, which is potentially explained by pages on Wikipedia generally having hyperlinks for the first mention of an entity only.
It is also interesting to note that there are lower recall numbers for the AIDA dataset relative to WIKI (Table 5), indicating that the RE module may have "overfit" in some sense to the Wikipedia style of article, and may be less effective on AIDA style news articles.
Gold	Predicted	Recall
SAME AS	4410	14765	0.31
OTHER	3169	5501	0.05
country	3066	728	0.06
member of sports team	880	2653	0.44
country of citizenship	861	150	0.02
shares border with	628	294	0.01
member of	385	669	0.08
league	380	416	0.23
located in the a.t.e	297	1199	0.09
contains a.t.e	261	11	0.01
headquarters location	252	1924	0.15
country for sport	175	130	0.00
has part	126	532	0.07
place of birth	123	365	0.11
sport	103	32	0.00
G Inference Speed and Scalability
We measure the time taken to run inference on the AIDA-CoNLL test dataset and compare it to SOTA baselines. Table 11 shows the results alongside the average ED performance on the 6 standard ED datasets (used in Table 1). Our model is an order of magnitude faster than the baselines with comparable ED performance.
Table 10: Analysis of relation predictions for AIDA dataset with threshold 0.5.
Method	Time taken (s)	Avg. ED F1
Cao et al. (2020)	2100	88.7
Wu et al. (2020) bi-encoder	93	80.4
Wu et al. (2020) cross-encoder	917	87.2
Orr et al. (2021)	438	77.6
KBED	96	90.0
w/o KB	15	88.6
Table 11: Time taken in seconds for EL inference on AIDA-CoNLL test dataset.
The most computationally expensive part of our model (accounting for approximately  $80\%$  of the inference and training time) is computing the KB score due to the large number of pairwise interactions present in documents. The hyperparameter for coarse-to-fine relation extraction can be lowered to trade-off computation cost with ED performance by reducing the number of pairwise interactions. Alternatively, as computation of the initial entity score  $\psi_{\mathrm{a}}$  is computationally cheap relative to the KB score  $\psi_{\mathrm{b}}$ , candidate entities with low initial entity scores can be pruned to further increase training and/or inference speed. These approaches would also allow scaling of the initial number of candidate entities to more than the 30 used for inference in this paper, if the use case required it.