SlowGuess/NAACL_2021_OCR huggingface dataset:
There is a great deal of recent work on unsupervised contrastive learning of image embeddings in computer vision (Oord et al., 2018; Hjelm et al., 2019; Chen et al., 2020, inter alia). Here,  $s_{\theta}(x,y) = E_{\theta}(x)^{\top}F_{\theta}(y)$  is a similarity score between images, and  $E_{\theta}$  or  $F_{\theta}$  is used to produce useful image representations for downstream tasks. The model is again learned by (4) where  $(x,y_1)$  are two random corruptions of the same image and  $y_{2:K}$  are different images. Robinson et al. (2021) propose a hard negative distribution in this setting and analyze the behavior of learned embeddings under that distribution. In contrast, our setting is large-scale supervised classification, such as entity linking, and our analysis is concerned with NCE with general hard negative distributions.
In a recent work, Xiong et al. (2021) consider contrastive learning for text retrieval with hard negatives obtained globally from the whole data with asynchronous updates, as we do in our experiments. They use the framework of importance sampling to argue that hard negatives yield gradients with larger norm, thus smaller variance and faster convergence. However, their argument does not imply our theorems. They also assume a pairwise loss, excluding non-pairwise losses such as (4).
We now study empirical aspects of the hard-negative NCE (Section 3) and the spectrum of score functions (Section 4). Our main testbed is Zeshel (Logeswaran et al., 2019), a challenging dataset for zero-shot entity linking. We also present complementary experiments on AIDA CoNLL-YAGO (Hoffart et al., 2011).
6.1 Task
Zeshel contains 16 domains (fictional worlds like Star Wars) partitioned to 8 training and 4 validation and test domains. Each domain has tens of thousands of entities along with their textual descriptions, which contain references to other entities in the domain and double as labeled mentions. The input  $x$  is a contextual mention and the label  $y$  is the description of the referenced entity. A score function  $s_{\theta}(x,y)$  is learned in the training domains and applied to a new domain for classification and retrieval. Thus the model must read descriptions of unseen entities and still make correct predictions.
We follow prior works and report micro-averaged top-64 recall and macro-averaged accuracy for evaluation. The original Zeshel paper (Logeswaran et al., 2019) distinguishes normalized vs unnormalized accuracy. Normalized accuracy assumes the presence of an external retriever and considers a mention only if its gold entity is included in top-64 candidates from the retriever. In this case, the problem is reduced to reranking and a computationally expensive joint encoder can be used. Unnormalized accuracy considers all mentions. Our goal is to improve unnormalized accuracy.
Logeswaran et al. (2019) use BM25 for retrieval, which upper bounds unnormalized accuracy by its poor recall (first row of Table 1). Wu et al. (2020) propose a two-stage approach in which a dual encoder is trained by hard-negative NCE and held fixed, then a BERT-based joint encoder is trained to rerank the candidates retrieved by the dual encoder. This approach gives considerable improvement in unnormalized accuracy, primarily due to the better recall of a trained dual encoder over BM25 (second row of Table 1). We show that we can further push the recall by optimizing the choice of hard negatives and architectures.
6.2 Architectures
We represent  $x$  and  $y$  as length-128 wordpiece sequences where the leftmost token is the special symbol [CLS]; we mark the boundaries of a mention span in  $x$  with special symbols. We use two independent BERT-bases to calculate mention embeddings  $E(x) \in \mathbb{R}^{768 \times 128}$  and entity embeddings  $F(y) \in \mathbb{R}^{768 \times 128}$ , where the columns  $E_t(x)$ ,  $F_t(y)$  are contextual embeddings of the  $t$ -th tokens.
Retriever. The retriever defines  $s_{\theta}(x,y)$ , the score between a mention  $x$  and an entity  $y$ , by one of the architectures described in Section 4.2:
$$
E _ {1} (x) ^ {\top} F _ {1} (y) \qquad (\text {D U A L})
$$
$$
F _ {1} (y) ^ {\top} C _ {m} (x, y) \quad (\text {P O L Y -} m)
$$
$$
\max  _ {t = 1} ^ {m} E _ {1} (x) ^ {\top} F _ {t} (y) \quad (\text {M U L T I -} m)
$$
$$
\sum_ {t = 1} ^ {1 2 8} \max  _ {t ^ {\prime} = 1} ^ {1 2 8} E _ {t} (x) ^ {\top} F _ {t ^ {\prime}} (y) \quad (\text {S O M})
$$
denoting the dual encoder, the poly-encoder (Humeau et al., 2020), the multi-vector encoder (Luan et al., 2020), and the sum-of-max encoder (Khattab and Zaharia, 2020). These architectures are sufficiently efficient to calculate  $s_{\theta}(x,y)$  for all entities  $y$  in training domains for each mention  $x$ . This efficiency is necessary for sampling hard negatives during training and retrieving candidates at test time.
Reranker. The reranker defines  $s_{\theta}(x,y) = w^{\top}E_{1}(x,y) + b$  where  $E(x,y) \in \mathbb{R}^{H \times 256}$  is BERT (either base  $H = 768$  or large  $H = 1024$ ) embeddings of the concatenation of  $x$  and  $y$  separated by the special symbol [SEP], and  $w, b$  are parameters of a linear layer. We denote this encoder by JOINT.
6.3 Optimization
Training a retriever. A retriever is trained by minimizing an empirical estimate of the hard-negative NCE loss (5),
$$
\widehat {J} _ {\text {H A R D}} (\theta) = - \frac {1}{N} \sum_ {i = 1} ^ {N} \log \frac {\exp \left(s _ {\theta} \left(x _ {i} , y _ {i , 1}\right)\right)}{\sum_ {k ^ {\prime} = 1} ^ {K} \exp \left(s _ {\theta} \left(x _ {i} , y _ {i , k ^ {\prime}}\right)\right)} \tag {11}
$$
where  $(x_{1},y_{1,1})\ldots (x_{N},y_{N,1})$  denote  $N$  mentionentity pairs in training data, and  $y_{i,2}\dots y_{i,K}\sim h(\cdot |x_i,y_{i,1})$  are  $K - 1$  negative entities for the ith mention. We vary the choice of negatives as follows.
- Random: The negatives are sampled uniformly at random from all entities in training data.
2Our code is available at: https://github.com/WenzhengZhang/hard-nce-el.
Figure 1: Synthetic experiments. We use a feedforward network to estimate the population distribution by minimizing sampled cross entropy in each step ( $x$ -axis). We show the NCE loss (left) and the norm of the gradient bias (right) using hard vs random negatives.
- Hard: The negatives are sampled from (8) each epoch. That is, in the beginning of each training pass, for each  $i$  we sample entities  $y_{i,2} \ldots y_{i,K}$  from  $\mathcal{Y} \setminus \{y_{i,1}\}$  without replacement with probabilities proportional to  $\exp(s_{\theta}(x_i, y_{i,k}))$ . This is slightly different from, and simpler than, the original hard negative mining strategy of Gillick et al. (2019) which pretrains the model using random negatives then greedily adds negative entities that score higher than the gold.
- Mixed- $p$ :  $p$  percent of the negatives are hard, the rest are random. Previous works have shown that such a combination of random and hard negatives can be effective. We find the performance is not sensitive to the value of  $p$  (Appendix A).
We experimented with in-batch sampling as done in previous works (e.g., Gillick et al. (2019)), but found sampling from all training data to be as effective and more straightforward (e.g., the number of random negatives is explicitly unrelated to the batch size). We use  $K = 64$  in all experiments.
Training a reranker. We use JOINT only for reranking by minimizing (11) with top-63 negatives given by a fixed retriever, where we vary the choice of retriever. We also investigate other architectures for reranking such as the poly-encoder and the sum-of-max encoder, but we find the full cross attention of JOINT to be indispensable. Details of reranking experiments can be found in Appendix B.
Other details. All models are trained up to 4 epochs using Adam. We tune the learning rate over  $\{5\mathrm{e} - 5,2\mathrm{e} - 5,1\mathrm{e} - 5\}$  on validation data. We use the training batch size of 4 mentions for all models
Model	Negatives	Val	Test
BM25	-	76.22	69.13
Wu et al. (2020)	Mixed (10 hard)	91.44	82.06
DUAL	Random	91.08	81.80
Hard	91.99	84.87
Mixed-50	91.75	84.16
DUAL-(10)	Hard	91.57	83.08
POLY-16	Random	91.05	81.73
Hard	92.08	84.07
Mixed-50	92.18	84.34
MULTI-8	Random	91.13	82.44
Hard	92.35	84.94
Mixed-50	92.76	84.11
SOM	Random	92.51	87.62
Hard	94.49	88.68
Mixed-50	94.66	89.62
Table 1: Top-64 recalls over different choices of architecture and negative examples for a retriever trained by NCE. Wu et al. (2020) train a dual encoder by NCE with 10 hard negatives. DUAL-(10) is DUAL trained with the score-adjusted loss (10).
except for JOINT, for which we use 2. Training time is roughly half a day on a single NVIDIA A100 GPU for all models, except the SOM retriever which takes 1-2 days.
6.4 Bias
We conduct experiments on synthetic data to empirically validate our bias analysis in Section 3.1. Analogous experiments on Zeshel with similar findings can be found in Appendix C.
We construct a population distribution over 1000 labels with small entropy to represent the peaky conditional label distribution  $\mathbf{pop}(y|x)$ . We use a feedforward network with one ReLU layer to estimate this distribution by minimizing the empirical cross-entropy loss based on 128 iid samples per update. At each update, we compute cross-entropy (2) exactly, and estimate NCE (5) with 4 negative samples by Monte Carlo (10 simulations).
Figure 1 plots the value of the loss function (left) and the norm of the gradient bias (right) across updates. We first observe that hard NCE yields an accurate estimate of cross entropy even with 4 samples. In contrast, random NCE quickly converges to zero, reflecting the fact that the model can trivially discriminate between the gold and random labels. We next observe that the bias of the gradient of hard NCE vanishes as the model distribution converges to the population distribution, which supports our analysis that the bias becomes smaller as the model improves. The bias remains nonzero for random NCE.
Model	Retriever	Negatives	Joint Reranker	Unnormalized
Val	Test
Logeswaran et al. (2019)	BM25	-	base	-	55.08
Logeswaran et al. (2019)+DAP	BM25	-	base	-	55.88
Wu et al. (2020)	DUAL (base)	Mixed (10 hard)	base	-	61.34
Wu et al. (2020)	DUAL (base)	Mixed (10 hard)	large	-	63.03
Ours	DUAL (base)	Hard	base	69.14	65.42
DUAL (base)	Hard	large	68.31	65.32
SOM (base)	Hard	base	69.19	66.67
SOM (base)	Hard	large	70.08	65.95
SOM (base)	Mixed-50	base	69.22	65.37
SOM (base)	Mixed-50	large	70.28	67.14
Table 2: Unnormalized accuracies with two-stage training. DAP refers to domain adaptive pre-training on source and target domains.
Mention	... his temporary usurpation of the Imperial throne by invading and seized control of the Battlespire, the purpose of this being to cripple the capacity of the Imperial College of Battlemages, which presented a threat to Tharn's power as Emperor. Mehrunes Dagon was responsible for the destruction of Mournhold at the end of the First Era, and apparently also ...
Random	1. Mehrunes Dagon is one of the seventeen Daedric Princes of Oblivion and the primary antagonist of ...2. Daedric Forces of Destruction were Mehrunes Dagon's personal army, hailing from his realm of Oblivion, the Deadlands. ...3. Weir Gate is a device used to travel to Battlespire from Tamriel. During the Invasion of the Battlespire, Mehrunes Dagon's forces ...4. Jagar Tharn was an Imperial Battlemage and personal adviser to Emperor Uriel Septim VII. Tharn used the Staff of Chaos ...5. House Sotha was one of the minor Houses of Vvardenfell until its destruction by Mehrunes Dagon in the times of Indoril Nerevar. ...6. Imperial Battlespire was an academy for training of the Battlemages of the Imperial Legion. The Battlespire was moored in ...
Hard	1. Fall of Ald'ruhn was a battle during the Oblivion Crisis. It is one of the winning battles invading in the name of Mehrunes Dagon ...2. Daedric Forces of Destruction were Mehrunes Dagon's personal army, hailing from his realm of Oblivion, the Deadlands. ...3. House Sotha was one of the minor Houses of Vvardenfell until its destruction by Mehrunes Dagon in the times of Indoril Nerevar. ...✓ 4. Sack of Mournhold was an event that occurred during the First Era. It was caused by the Dunmer witch Turala Skeffington ...5. Mehrunes Dagon of the House of Troubles is a Tribunal Temple quest, available to the Nerevarine in ...6. Oblivion Crisis, also known as the Great Anguish to the Altmer or the Time of Gates by Mankar Camoran, was a period of major turmoil ...
Table 3: A retrieval example with hard negative training on Zeshel. We use a SOM retriever trained with random vs hard negatives (92.51 vs 94.66 in top-64 validation recall). We show a validation mention (destruction) whose gold entity is retrieved by the hard-negative model but not by the random-negative model. Top entities are shown for each model (title boldfaced); the correct entity is Sack of Mournhold (checkmarked).
6.5 Retrieval
Table 1 shows the top-64 recall (i.e., the percentage of mentions whose gold entity is included in the 64 entities with highest scores under a retriever trained by (5)) as we vary architectures and negative examples. We observe that hard and mixed negative examples always yield sizable improvements over random negatives, for all architectures. Our dual encoder substantially outperforms the previous dual encoder recall by Wu et al. (2020), likely due to better optimization such as global vs in-batch random negatives and the proportion of hard negatives. We also train a dual encoder with the bias-corrected loss (10) and find that this does not improve recall. The poly-encoder and the multi-vector models are comparable to but do not improve over the dual encoder. However, the sum-of-max encoder delivers a decisive improvement, especially with hard negatives, pushing the test recall to above  $89\%$ . Based on this finding, we use DUAL and SOM for retrieval in later experiments.
6.6 Results
We show our main results in Table 2. Following Wu et al. (2020), we do two-stage training in which we train a DUAL or SOM retriever with hard-negative NCE and train a JOINT reranker to rerank its top-64 candidates. All our models outperform the previous best accuracy of  $63.03\%$  by Wu et al. (2020). In fact, our dual encoder retriever using a BERT-base reranker outperforms the dual encoder retriever using a BERT-large reranker  $(65.42\%$  vs  $63.03\%)$ . We obtain a clear improvement by switching the retriever from dual encoder to sum-of-max due to its high recall (Table 1). Using a sum-of-max retriever trained with mixed negatives and a BERT-large reranker gives the best result  $67.14\%$ .
To better understand practical implications of hard negative mining, we compare a SOM retriever trained on Zeshel with random vs hard negatives (92.51 vs 94.66 in top-64 validation recall). The
Model	Accuracy
BLINK without finetuning	80.27
BLINK with finetuning	81.54
DUAL with p = 0	82.40
DUAL with p = 50	88.01
MULTI-2 with p = 50	88.39
MULTI-3 with p = 50	87.94
Table 4: Test accuracies on AIDA CoNLL-YAGO. BLINK refers to the two-stage model of Wu et al. (2020) pretrained on Wikipedia. All our models are initialized from the BLINK dual encoder and finetuned using all 5.9 million Wikipedia entities as candidates.
mention categories most frequently improved are Low Overlap (174 mentions) and Multiple Categories (76 mentions) (see Logeswaran et al. (2019) for the definition of these categories), indicating that hard negative mining makes the model less reliant on string matching. A typical example of improvement is shown in Table 3. The random-negative model retrieves person, device, or institution entities because they have more string overlap (e.g. “Mehrunes Dagon”, “Battlespire”, and “Tharn”). In contrast, the hard-negative model appears to better understand that the mention is referring to a chaotic event like the Fall of Ald’ruhn, Sack of Mournhold, and Oblivion Crisis and rely less on string matching. We hypothesize that this happens because string matching is sufficient to make a correct prediction during training if negative examples are random, but insufficient when they are hard.
To examine the effect of encoder architecture, we also compare a DUAL vs SOM retriever both trained with mixed negatives (91.75 vs 94.66 in top-64 validation recall). The mention categories most frequently improved are again Low Overlap (335 mentions) and Multiple Categories (41 mentions). This indicates that cross attention likewise helps the model less dependent on simple string matching, presumably by allowing for a more expressive class of score functions.
6.8 Results on AIDA
We complement our results on Zeshel with additional experiments on AIDA. We use BLINK, a Wikipedia-pretrained two-stage model (a dual encoder retriever pipelined with a joint reranker, both based on BERT) made available by Wu et al.
(2020).<sup>3</sup> We extract the dual encoder module from BLINK and finetune it on AIDA using the training portion. During finetuning, we use all 5.9 million Wikipedia entities as candidates to be consistent with prior work. Because of the large scale of the knowledge base we do not consider SOM and focus on the MULTI- $m$  retriever (DUAL is a special case with  $m = 1$ ). At test time, all models consider all Wikipedia entities as candidates. For both AIDA and the Wikipedia dump, we use the version prepared by the KILT benchmark (Petroni et al., 2020).
Table 4 shows the results. Since Wu et al. (2020) do not report AIDA results, we take the performance of BLINK without and with finetuning from their GitHub repository and the KILT leaderboard. We obtain substantially higher accuracy by mixed-negative training even without reranking. There is no significant improvement from using  $m > 1$  in the multi-vector encoder on this task.
Hard negatives can often improve NCE in practice, substantially so for entity linking (Gillick et al., 2019), but are used without justification. We have formalized the role of hard negatives in quantifying the bias of the gradient of the contrastive loss with respect to the gradient of the full cross-entropy loss. By jointly optimizing the choice of hard negatives and architectures, we have obtained new state-of-the-art results on the challenging Zeshel dataset (Logeswaran et al., 2019).
This work was supported by the Google Faculty Research Awards Program. We thank Ledell Wu for many clarifications on the BLINK paper.
Yoshua Bengio and Jean-Sébastien Senécal. 2008. Adaptive importance sampling to accelerate training of a neural probabilistic language model. IEEE Transactions on Neural Networks, 19(4):713-722.
Guy Blanc and Steffen Rendle. 2018. Adaptive sampled softmax with kernel based sampling. In
3https://github.com/facebookresearch/ BLINK
4https://ai.facebook.com/tools/kilt/ (as of April 8, 2021)
<sup>5</sup>We find that reranking does not improve accuracy on this task, likely because the task does not require as much reading comprehension as Zeshel.
International Conference on Machine Learning, pages 590-599.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Thibault Févry, Nicholas FitzGerald, Livio Baldini Soares, and Tom Kwiatkowski. 2020. Empirical evaluation of pretraining strategies for supervised entity linking. In _Automated Knowledge Base Construction_.
Octavian-Eugen Ganea and Thomas Hofmann. 2017. Deep joint entity disambiguation with local neural attention. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2619-2629, Copenhagen, Denmark. Association for Computational Linguistics.
Daniel Gillick, Sayali Kulkarni, Larry Lansing, Alessandro Presta, Jason Baldridge, Eugene Ie, and Diego Garcia-Olano. 2019. Learning dense representations for entity retrieval. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL), pages 528-537, Hong Kong, China. Association for Computational Linguistics.
Nitish Gupta, Sameer Singh, and Dan Roth. 2017. Entity linking via joint encoding of types, descriptions, and context. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2681-2690, Copenhagen, Denmark. Association for Computational Linguistics.
Michael U Gutmann and Aapo Hyvarinen. 2012. Noise-contrastive estimation of unnormalized statistical models, with applications to natural image statistics. The journal of machine learning research, 13(1):307-361.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrievalaugmented language model pre-training. arXiv preprint arXiv:2002.08909.
R Devon Hjelm, Alex Fedorov, Samuel Lavoie-Marchildon, Karan Grewal, Phil Bachman, Adam Trischler, and Yoshua Bengio. 2019. Learning deep representations by mutual information estimation and maximization. In International Conference on Learning Representations.
Johannes Hoffart, Mohamed Amir Yosef, Ilaria Bordino, Hagen Fürstenau, Manfred Pinkal, Marc Spaniol, Bilyana Taneva, Stefan Thater, and Gerhard Weikum. 2011. Robust disambiguation of named entities in text. In Proceedings of the 2011 Conference on Empirical Methods in Natural Language Processing, pages 782-792.
Po-Sen Huang, Xiaodong He, Jianfeng Gao, Li Deng, Alex Acero, and Larry Heck. 2013. Learning deep structured semantic models for web search using clickthrough data. In Proceedings of the 22nd ACM international conference on Information & Knowledge Management, pages 2333-2338.
Samuel Humeau, Kurt Shuster, Marie-Anne Lachaux, and Jason Weston. 2020. Poly-encoders: Architectures and pre-training strategies for fast and accurate multi-sentence scoring. In International Conference on Learning Representations.
Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Ledell Wu, Sergey Edunov, Danqi Chen, and Wentau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.
Omar Khattab and Matei Zaharia. 2020. Colbert: Efficient and effective passage search via contextualized late interaction over BERT. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, SIGIR 2020, Virtual Event, China, July 25-30, 2020, pages 39-48. ACM.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096.
Lajanugen Logeswaran, Ming-Wei Chang, Kenton Lee, Kristina Toutanova, Jacob Devlin, and Honglak Lee. 2019. Zero-shot entity linking by reading entity descriptions. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3449-3460.
Yi Luan, Jacob Eisenstein, Kristina Toutanova, and Michael Collins. 2020. Sparse, dense, and attentional representations for text retrieval. arXiv preprint arXiv:2005.00181.
Zhuang Ma and Michael Collins. 2018. Noise contrastive estimation and negative sampling for conditional models: Consistency and statistical efficiency. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3698-3707, Brussels, Belgium. Association for Computational Linguistics.
Andriy Mnih and Yee Whye Teh. 2012. A fast and simple algorithm for training neural probabilistic language models. In Proceedings of the 29th International Coherence on International Conference on Machine Learning, pages 419-426.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748.