SlowGuess/ACL_2020_OCR huggingface dataset:
Leakage
Removal	Model	Model
Selection	MRR	HITS@1	HITS@10	HITS@50
THOROUGH	COMPLEX-UNI	ALL	2.2	0.8	4.7	10.2
COMPLEX-UNI	MENTION	2.2	0.9	4.7	10.3
COMPLEX-UNI	LINKED	2.2	0.9	4.7	10.3
DISTMULT-LSTM	ALL	3.2	1.7	5.9	11.6
DISTMULT-LSTM	MENTION	3.3	1.8	5.9	12.2
DISTMULT-LSTM	LINKED	3.3	1.8	5.9	12.2
COMPLEX-LSTM-XL	ALL	3.3	1.8	5.8	12.0
COMPLEX-LSTM-XL	MENTION	3.6	1.9	6.6	13.9
COMPLEX-LSTM-XL	LINKED	3.6	1.9	6.6	13.9
COMPLEX-LSTM	ALL	2.7	1.5	4.7	9.1
COMPLEX-LSTM	MENTION	3.8	2.1	7.1	14.1
COMPLEX-LSTM	LINKED	3.9	2.1	7.0	14.6
Table 5: Additional Test results. Comparing DISTMULT-LSTM, COMPLEX-LSTM-XL with embedding size 768, COMPLEX-UNI with uni-gram pooling as composition function. Model selection on VALID, VALID-LINKED and VALID-MENTION, models trained on THOROUGH; Results in percent.
and  $L^{kj}$  is computed likewise. With batch negative examples the mentions/entities appear in expectation proportional to their frequency in the training data as a "negative example".
C.2 Training settings
We used Adagrad with mini batches (Duchi et al., 2011) with batch size 4096. The token embeddings were initialized with the Glorot initialization (Glorot and Bengio, 2010). One epoch takes  $\approx 50$  min with a TitanXp/1080Ti. We performed a grid search over the following hyperparameters: entity and relation token embedding sizes [256, 512], drop-out after the composition function  $f$  and  $g$  [0.0, 0.1], learning rate [0.05, 0.1, 0.2] and weight decay  $[10^{-6}, 10^{-10}]$ . We trained the models for 10 epochs and selected the hyperparameters, which achieved the best MRR with mention ranking on VALID-LINKED. We trained the final models for up to 100 epochs but did early stopping if no improvement occurred within 10 epochs.
D Performance Metrics
Denote by  $\mathcal{M}(\mathcal{E})$  all mentions from the dataset. Denote by  $\mathcal{Q}$  the set of all questions generated from the evaluation data. Given a question  $q_{t}\in \mathcal{Q}$ , we rank all  $m\in \mathcal{M}(\mathcal{E})$  by the scores  $s(i,k,m)$  (or  $s(m,k,j)$  for  $q_{h}\in \mathcal{Q}$ ), then filter the raw rank according to either the entity-ranking protocol or the mention-ranking protocol. Finally, we record the positions of the correct answers in the filtered ranking.
MRR is defined as follows: For each question  $q \in \mathcal{Q}$ , let  $\mathrm{RR}_q$  be the filtered reciprocal rank of the top-ranked correct answer. MRR is the micro-average over  $\{\mathrm{RR}_q \mid q \in \mathcal{Q}\}$ . HITS@k is the proportion of the questions where at least one correct mention appears in the top  $k$  positions of the filtered ranking.
Tab. 5 provides results for other models and hyperparameters. The COMPLEX-LSTM results from the Sec. 6 are given at the bottom for comparison. COMPLEX-LSTM-XL has a larger embedding size of 768, which did not help to improve the results. COMPLEX-UNI is the ComplEx model with the uni-gram pooling composition function, i.e., averaging the token embeddings. Compared to COMPLEX-LSTM it shows that LSTM as a composition function did yield better results. DISTMULT-LSTM is the DistMult relational model (Yang et al., 2015) with an LSTM as composition function, which did not improve over COMPLEX-LSTM. In Summary, the results support the hyperparameters, model and composition function chosen for the experiments in Sec. 6. Overall, we observed that model selection based on VALID-ALL seems to have a higher variance because the model selected for COMPLEX-LSTM with VALID-ALL is outperformed by other models, whereas COMPLEX-LSTM performed best for models selected with VALID-MENTION and VALID-LINKED.