SlowGuess/NAACL_2021_OCR huggingface dataset:
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vassilis Plachouras, Tim Roektaschel, et al. 2020. Kilt: a benchmark for knowledge intensive language tasks. arXiv preprint arXiv:2009.02252.
Ankit Singh Rawat, Jiecao Chen, Felix Xinnan X Yu, Ananda Theertha Suresh, and Sanjiv Kumar. 2019. Sampled softmax with random fourier features. In Advances in Neural Information Processing Systems, pages 13857-13867.
Joshua David Robinson, Ching-Yao Chuang, Suvrit Sra, and Stefanie Jegelka. 2021. Contrastive learning with hard negative samples. In International Conference on Learning Representations.
Karl Stratos. 2019. Noise contrastive estimation. http://karlstratos.com/notes/nce.pdf. Unpublished technical note. Accessed: April 8, 2021.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. 2020. What makes for good views for contrastive learning. arXiv preprint arXiv:2005.10243.
Ledell Wu, Fabio Petroni, Martin Josifoski, Sebastian Riedel, and Luke Zettlemoyer. 2020. Scalable zero-shot entity linking with dense entity retrieval. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6397-6407.
Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2021. Approximate nearest neighbor negative contrastive learning for dense text retrieval. In International Conference on Learning Representations.
A Percentage of Hard Negatives
We show top-64 validation recalls with varying values of the hard negative percentage  $p$  in training below:
Mixed-p (%)	DUAL	MULTI-8	SOM
0 (Random)	91.08	91.13	92.51
25	92.18	92.74	94.13
50	91.75	92.76	94.66
75	92.24	93.41	94.37
100 (Hard)	92.05	93.27	94.54
The presence of hard negatives is clearly helpful, but the exact choice of  $p > 0$  is not as important. We choose  $p = 50$  because we find that the presence of some random negatives often gives slight yet consistent improvement.
B Reranking Experiments
We show the normalized and unnormalized accuracy of a reranker as we change the architecture while holding the retriever fixed:
Model	Normalized	Unnormalized
Val	Test	Val	Test
DUAL	60.43	62.49	54.87	54.73
POLY-16	60.37	60.98	54.82	53.37
POLY-64	60.80	61.88	55.20	54.15
POLY-128	60.60	62.72	55.03	54.92
MULTI-8	61.56	62.65	55.90	54.87
MULTI-64	61.94	62.94	56.23	55.15
MULTI-128	61.67	62.95	55.98	55.17
SOM	65.38	65.24	59.35	57.04
GENPOLY-128	65.89	64.98	59.82	56.82
JOINT	76.17	74.90	69.14	65.42
Logeswaran et al.	76.06	75.06	-	55.08
Wu et al.	78.24	76.58	-	-
JOINT (ours)	78.82	77.09	58.77	56.56
GENPOLY- $m$  denotes a generalized version of the poly-encoder in which we use  $m$  leftmost entity embeddings rather than one:  $s_{\theta}(x,y) = 1_m^\top F_{1:m}(y)^\top C_m(x,y)$ . We use a trained dual encoder with  $91.93\%$  and  $83.48\%$  validation/test recalls as a fixed retriever. The accuracy increases with the complexity of the reranker. The dual encoder and the poly-encoder are comparable, but the multi-vector, the sum-of-max, and the generalized poly-encoder achieve substantially higher accuracies. Not surprisingly, the joint encoder achieves the best performance. We additionally show reranking results using the BM25 candidates provided in the Zeshel dataset for comparison with existing results. Our implementation of JOINT with BERT-base obtains comparable accuracies.
C Bias Experiments on Zeshel
We consider the dual encoder  $s_{\theta}(x,y) = E_1(x)^\top F_1(y)$  where  $E$  and  $F$  are parameterized by BERT-bases. We randomly sample 64 mentions, yielding a total of 128 entities: 64 referenced by the mentions, and 64 whose descriptions contain these mentions. We consider these 128 entities to constitute the entirety of the label space  $\mathcal{V}$ . On the 64 mentions, we estimate  $J_{\mathrm{CE}}(\theta)$  by normalizing over the 128 entities; we estimate  $J_{\mathrm{HARD}}(\theta)$  by normalizing over  $K = 8$  candidates where 7 are drawn from a negative distribution: either random, hard, or mixed. Instead of a single-sample estimate as in (11), we draw negative examples 500 times and average the result. We estimate the bias  $b(\theta) \in \mathbb{R}^d$  by taking a difference between these two estimates and report the norm below:
Negatives	||b(θCE)||	||b(θRAND)||
Random	16.33	166.38
Hard	0.68	0.09
Mixed-50	1.20	0.90
We consider two parameter locations.  $\theta_{\mathrm{CE}}$  is obtained by minimizing the cross-entropy loss (92.19% accuracy).  $\theta_{\mathrm{RAND}}$  is obtained by NCE with random negatives (60% accuracy). The bias is drastically smaller when negative examples are drawn from the model instead of randomly. Mixed negatives yield comparably small biases. With random negatives, the bias is much larger at  $\theta_{\mathrm{RAND}}$  since  $\nabla J_{\mathrm{CE}}(\theta_{\mathrm{RAND}})$  is large. In contrast, hard and mixed negatives again yield small biases.