SlowGuess/NAACL_2021_OCR huggingface dataset:
{wenzheng.zhang, karl.stratos}@rutgers.edu
The choice of negative examples is important in noise contrastive estimation. Recent works find that hard negatives—highest-scoring incorrect examples under the model—are effective in practice, but they are used without a formal justification. We develop analytical tools to understand the role of hard negatives. Specifically, we view the contrastive loss as a biased estimator of the gradient of the cross-entropy loss, and show both theoretically and empirically that setting the negative distribution to be the model distribution results in bias reduction. We also derive a general form of the score function that unifies various architectures used in text retrieval. By combining hard negatives with appropriate score functions, we obtain strong results on the challenging task of zero-shot entity linking.
Noise contrastive estimation (NCE) is a widely used approach to large-scale classification and retrieval. It estimates a score function of input-label pairs by a sampled softmax objective: given a correct pair  $(x, y_1)$ , choose negative examples  $y_2 \ldots y_K$  and maximize the probability of  $(x, y_1)$  in a softmax over the scores of  $(x, y_1) \ldots (x, y_K)$ . NCE has been successful in many applications, including information retrieval (Huang et al., 2013), entity linking (Gillick et al., 2019), and open-domain question answering (Karpukhin et al., 2020).
It is well known that making negatives "hard" can be empirically beneficial. For example, Gillick et al. (2019) propose a hard negative mining strategy in which highest-scoring incorrect labels under the current model are chosen as negatives. Some works even manually include difficult examples based on external information such as a ranking function (Karpukhin et al., 2020) or a knowledge base (Févry et al., 2020).
While it is intuitive that such hard negatives help improve the final model by making the learning task more challenging, they are often used without a formal justification. Existing theoretical results in contrastive learning are not suitable for understanding hard negatives since they focus on unconditional negative distributions (Gutmann and Hyvarinen, 2012; Mnih and Teh, 2012; Ma and Collins, 2018; Tian et al., 2020) or consider a modified loss divergent from practice (Bengio and Senécal, 2008).
In this work, we develop analytical tools to understand the role of hard negatives. We formalize hard-negative NCE with a realistic loss (5) using a general conditional negative distribution, and view it as a biased estimator of the gradient of the cross-entropy loss. We give a simple analysis of the bias (Theorem 3.1). We then consider setting the negative distribution to be the model distribution, which recovers the hard negative mining strategy of Gillick et al. (2019), and show that it yields an unbiased gradient estimator when the model is optimal (Theorem 3.2). We complement the gradient-based perspective with an adversarial formulation (Theorem 3.3).
The choice of architecture to parametrize the score function is another key element in NCE. There is a surge of interest in developing efficient cross-attentional architectures (Humeau et al., 2020; Khattab and Zaharia, 2020; Luan et al., 2020), but they often address different tasks and lack direct comparisons. We give a single algebraic form of the score function (9) that subsumes and generalizes these works, and directly compare a spectrum of architectures it induces.
We present experiments on the challenging task of zero-shot entity linking (Logeswaran et al., 2019). We calculate empirical estimates of the bias of the gradient estimator to verify our analysis, and systematically explore the joint space of negative examples and architectures. We have clear
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 1090-1101 June 6-11, 2021. ©2021 Association for Computational Linguistics
practical recommendations: (i) hard negative mining always improves performance for all architectures, and (ii) the sum-of-max encoder (Khattab and Zaharia, 2020) yields the best recall in entity retrieval. Our final model combines the sum-of-max retriever with a BERT-based joint reranker to achieve  $67.1\%$  unnormalized accuracy: a  $4.1\%$  absolute improvement over Wu et al. (2020). We also present complementary experiments on AIDA CoNLL-YAGO (Hoffart et al., 2011) in which we finetune a Wikipedia-pretrained dual encoder with hard-negative NCE and show a  $6\%$  absolute improvement in accuracy.
2 Review of NCE
Let  $\mathcal{X}$  and  $\mathcal{Y}$  denote input and label spaces. We assume  $|\mathcal{Y}| < \infty$  for simplicity. Let pop denote a joint population distribution over  $\mathcal{X} \times \mathcal{Y}$ . We define a score function  $s_{\theta}: \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  differentiable in  $\theta \in \mathbb{R}^d$ . Given sampling access to pop, we wish to estimate  $\theta$  such that the classifier  $x \mapsto \arg \max_{y \in \mathcal{Y}} s_{\theta}(x, y)$  (breaking ties arbitrarily) has the optimal expected zero-one loss. We can reduce the problem to conditional density estimation. Given  $x \in \mathcal{X}$ , define
$$
p _ {\theta} (y | x) = \frac {\exp \left(s _ {\theta} (x , y)\right)}{\sum_ {y ^ {\prime} \in \mathcal {Y}} \exp \left(s _ {\theta} \left(x , y ^ {\prime}\right)\right)} \tag {1}
$$
for all  $y\in \mathcal{V}$  .Let  $\theta^{*}$  denote a minimizer of the cross-entropy loss:
$$
J _ {\mathrm {C E}} (\theta) = \underset {(x, y) \sim \mathbf {p o p}} {\mathbf {E}} [ - \log p _ {\theta} (y | x) ] \tag {2}
$$
If the score function is sufficiently expressive,  $\theta^{*}$  satisfies  $p_{\theta^{*}}(y|x) = \mathbf{pop}(y|x)$  by the usual property of cross entropy. This implies that  $s_{\theta^{*}}$  can be used as an optimal classifier.
The cross-entropy loss is difficult to optimize when  $\mathcal{V}$  is large since the normalization term in (1) is expensive to calculate. In NCE, we dodge this difficulty by subsampling. Given  $x\in \mathcal{X}$  and any  $K$  labels  $y_{1:K} = (y_1\dots y_K)\in \mathcal{Y}^K$ , define
$$
\pi_ {\theta} (k | x, y _ {1: K}) = \frac {\exp \left(s _ {\theta} \left(x , y _ {k}\right)\right)}{\sum_ {k ^ {\prime} = 1} ^ {K} \exp \left(s _ {\theta} \left(x , y _ {k ^ {\prime}}\right)\right)} \tag {3}
$$
for all  $1 \leq k \leq K$ . When  $K \ll |\mathcal{Y}|$ , (3) is significantly cheaper to calculate than (1). Given  $K \geq 2$ , we define
$$
J _ {\mathrm {N C E}} (\theta) = \underset { \begin{array}{c} (x, y _ {1}) \sim \mathbf {p o p} \\ y _ {2: K} \sim q ^ {K - 1} \end{array} } {\mathbf {E}} [ - \log \pi_ {\theta} (1 | x, y _ {1: K}) ] \quad (4)
$$
where  $y_{2:K} \in \mathcal{Y}^{K-1}$  are negative examples drawn iid from some "noise" distribution  $q$  over  $\mathcal{V}$ . Popular choices of  $q$  include the uniform distribution  $q(y) = 1 / |\mathcal{V}|$  and the population marginal  $q(y) = \mathbf{pop}(y)$ .
The NCE loss (4) has been studied extensively. An optimal classifier can be extracted from a minimizer of  $J_{\mathrm{NCE}}$  (Ma and Collins, 2018); minimizing  $J_{\mathrm{NCE}}$  can be seen as maximizing a lower bound on the mutual information between  $(x,y)\sim \mathbf{pop}$  if  $q$  is the population marginal (Oord et al., 2018). We refer to Stratos (2019) for an overview. However, most of these results focus on unconditional negative examples and do not address hard negatives, which are clearly conditional. We now focus on conditional negative distributions, which are more suitable for describing hard negatives.
3 Hard Negatives in NCE
Given  $K\geq 2$  , we define
$$
J _ {\text {H A R D}} (\theta) = \underset { \begin{array}{c} (x, y _ {1}) \sim \mathbf {p o p} \\ y _ {2: K} \sim h (\cdot | x, y _ {1}) \end{array} } {\mathbf {E}} [ - \log \pi_ {\theta} (1 | x, y _ {1: K}) ] \tag {5}
$$
where  $y_{2:K} \in \mathcal{Y}^{K-1}$  are negative examples drawn from a conditional distribution  $h(\cdot|x, y_1)$  given  $(x, y_1) \sim \mathbf{pop}$ . Note that we do not assume  $y_{2:K}$  are iid. While simple, this objective captures the essence of using hard negatives in NCE, since the negative examples can arbitrarily condition on the input and the gold (e.g., to be wrong but difficult to distinguish from the gold) and be correlated (e.g., to avoid duplicates).
We give two interpretations of optimizing  $J_{\mathrm{HARD}}$ . First, we show that the gradient of  $J_{\mathrm{HARD}}$  is a biased estimator of the gradient of the cross-entropy loss  $J_{\mathrm{CE}}$ . Thus optimizing  $J_{\mathrm{HARD}}$  approximates optimizing  $J_{\mathrm{CE}}$  when we use a gradient-based method, where the error depends on the choice of  $h(\cdot | x, y_1)$ . Second, we show that the hard negative mining strategy can be recovered by considering an adversarial setting in which  $h(\cdot | x, y_1)$  is learned to maximize the loss.
3.1 Gradient Estimation
We assume an arbitrary choice of  $h(\cdot |x,y_1)$  and  $K\geq 2$  . Denote the bias at  $\theta \in \mathbb{R}^d$  by
$$
b (\theta) = \nabla J _ {\mathrm {C E}} (\theta) - \nabla J _ {\mathrm {H A R D}} (\theta)
$$
To analyze the bias, the following quantity will be important. For  $x \in \mathcal{X}$  define
$$
\gamma_ {\theta} (y | x) = \Pr_ { \begin{array}{c} y _ {1} \sim \mathbf {p o p} (\cdot | x) \\ y _ {2: K} \sim h (\cdot | x, y _ {1}) \\ k \sim \pi_ {\theta} (\cdot | x, y _ {1: K}) \end{array} } \left(y _ {k} = y\right) \tag {6}
$$
for all  $y \in \mathcal{Y}$ . That is,  $\gamma_{\theta}(y|x)$  is the probability that  $y$  is included as a candidate (either as the gold or a negative) and then selected by the NCE discriminator (3).
Theorem 3.1. For all  $i = 1\ldots d$
$$
b _ {i} (\theta) = \underset {x \sim \mathbf {p o p}} {\mathbf {E}} \left[ \sum_ {y \in \mathcal {Y}} \epsilon_ {\theta} (y | x) \frac {\partial s _ {\theta} (x , y)}{\partial \theta_ {i}} \right]
$$
where  $\epsilon_{\theta}(y|x) = p_{\theta}(y|x) - \gamma_{\theta}(y|x)$ .
Proof. Fix any  $x \in \mathcal{X}$  and let  $J_{\mathrm{CE}}^x (\theta)$  and  $J_{\mathrm{HARD}}^x (\theta)$  denote  $J_{\mathrm{CE}}(\theta)$  and  $J_{\mathrm{HARD}}(\theta)$  conditioned on  $x$ . The difference  $J_{\mathrm{CE}}^x (\theta) - J_{\mathrm{HARD}}^x (\theta)$  is
$$
\log Z _ {\theta} (x) - \underset { \begin{array}{c} y _ {1}: K \sim h (\cdot | x, y _ {1}) \\ y _ {2: K} \end{array} } {\mathbf {E}} [ \log Z _ {\theta} (x, y _ {1: K}) ] \quad (7)
$$
where we define  $Z_{\theta}(x) = \sum_{y^{\prime}\in \mathcal{Y}}\exp (s_{\theta}(x,y^{\prime}))$  and  $Z_{\theta}(x,y_{1:K}) = \sum_{k = 1}^{K}\exp (s_{\theta}(x,y_k))$ . For any  $(\tilde{x},\tilde{y})$ , the partial derivative of (7) with respect to  $s_\theta (\tilde{x},\tilde{y})$  is given by  $\left[[x = \tilde{x} ]\right]p_{\theta}(\tilde{y} |x) - \left[[x = \tilde{x} ]\right]\gamma_{\theta}(\tilde{y} |x)$  where  $[A]]$  is the indicator function that takes the value 1 if  $A$  is true and 0 otherwise. Taking an expectation of their difference over  $x\sim$  pop gives the partial derivative of  $b(\theta) = J_{\mathrm{CE}}(\theta) - J_{\mathrm{HARD}}(\theta)$  with respect to  $s_\theta (\tilde{x},\tilde{y})$  ..  $\mathbf{pop}(\tilde{x})(p_{\theta}(\tilde{y} |\tilde{x}) - \gamma_{\theta}(\tilde{y} |\tilde{x}))$  . The statement follows from the chain rule:
$$
b _ {i} (\theta) = \sum_ {x \in \mathcal {X}, y \in \mathcal {Y}} \frac {\partial b (\theta)}{\partial s _ {\theta} (x , y)} \frac {\partial s _ {\theta} (x , y)}{\partial \theta_ {i}}
$$
Theorem 3.1 states that the bias vanishes if  $\gamma_{\theta}(y|x)$  matches  $p_{\theta}(y|x)$ . Hard negative mining can be seen as an attempt to minimize the bias by defining  $h(\cdot |x,y_1)$  in terms of  $p_{\theta}$ . Specifically, we define
$$
\begin{array}{l} h (y _ {2: K} | x, y _ {1}) \\ \propto \left[ \left| \left\{y _ {1} \dots y _ {K} \right\} \right| = K \right] \prod_ {k = 2} ^ {K} p _ {\theta} \left(y _ {k} \mid x\right) \tag {8} \\ \end{array}
$$
Thus  $h(\cdot |x,y_1)$  has support only on  $y_{2:K}\in \mathcal{Y}^{K - 1}$  that are distinct and do not contain the gold. Greedy
sampling from  $h(\cdot | x, y_1)$  corresponds to taking  $K - 1$  incorrect label types with highest scores. This coincides with the hard negative mining strategy of Gillick et al. (2019).
The absence of duplicates in  $y_{1:K}$  ensures  $J_{\mathrm{CE}}(\theta) = J_{\mathrm{HARD}}(\theta)$  if  $K = |\mathcal{V}|$ . This is consistent with (but does not imply) Theorem 3.1 since in this case  $\gamma_{\theta}(y|x) = p_{\theta}(y|x)$ . For general  $K < |\mathcal{V}|$ , Theorem 3.1 still gives a precise bias term. To gain a better insight into its behavior, it is helpful to consider a heuristic approximation given by
$$
\gamma_ {\theta} (y | x) \approx \frac {p _ {\theta} (y | x) \exp {(s _ {\theta} (x , y))}}{N _ {\theta} (x)}
$$
where  $N_{\theta}(x) = \sum_{y^{\prime}\in \mathcal{Y}}p_{\theta}(y^{\prime}|x)\exp (s_{\theta}(x,y^{\prime}))$  Plugging this approximation in Theorem 3.1 we have a simpler equation
$$
b _ {i} (\theta) \approx \mathbf {E} _ {(x, y) \sim \mathbf {p o p}} \left[ (1 - \delta_ {\theta} (x, y)) \frac {\partial s _ {\theta} (x , y)}{\partial \theta_ {i}} \right]
$$
where  $\delta_{\theta}(x,y) = \exp (s_{\theta}(x,y)) / N_{\theta}(x)$ . The expression suggests that the bias becomes smaller as the model improves since  $p_{\theta}(\cdot |x)\approx \mathbf{pop}(\cdot |x)$  implies  $\delta_{\theta}(x,y)\approx 1$  where  $(x,y)\sim \mathbf{pop}$ .
We can formalize the heuristic argument to prove a desirable property of (5): the gradient is unbiased if  $\theta$  satisfies  $p_{\theta}(y|x) = \mathbf{pop}(y|x)$ , assuming iid hard negatives.
Theorem 3.2. Assume  $K \geq 2$  and the distribution  $h(y_{2:K}|x,y_1) = \prod_{k=2}^{K} p_{\theta}(y_k|x)$  in (5). If  $p_{\theta}(y|x) = \mathbf{pop}(y|x)$ , then  $\nabla J_{\mathrm{HARD}}(\theta) = \nabla J_{\mathrm{CE}}(\theta)$ .
Proof. Since  $\mathbf{pop}(y|x) = \exp (s_{\theta}(x,y)) / Z_{\theta}(x)$ , the probability  $\gamma_{\theta}(y|x)$  in (6) is
$$
\begin{array}{l} \sum_ {y _ {1: K} \in \mathcal {Y} ^ {K}} \prod_ {k = 1} ^ {K} \frac {\exp (s _ {\theta} (x , y _ {k}))}{Z _ {\theta} (x)} \frac {\exp (s _ {\theta} (x , y))}{Z _ {\theta} (x , y _ {1 : K})} \\ = \frac {\exp \left(s _ {\theta} (x , y)\right)}{Z _ {\theta} (x)} \sum_ {y _ {1: K} \in \mathcal {Y} ^ {K}} \frac {\prod_ {k = 1} ^ {K} \exp \left(s _ {\theta} (x , y _ {k})\right)}{Z _ {\theta} (x , y _ {1: K})} \\ \end{array}
$$
The sum marginalizes a product distribution over  $y_{1:K}$ , thus equals one. Hence  $\gamma_{\theta}(y|x) = p_{\theta}(y|x)$ . The statement follows from Theorem 3.1.
<sup>1</sup>We can rewrite  $\gamma_{\theta}(y|x)$  as
$$
\mathbf{E}_{\substack{y_{1}\sim \mathbf{pop}(\cdot |x)\\ y_{2};K\sim h(\cdot |x,y_{1})}}\left[\frac{\mathrm{count}_{y_{1:K}}(y)\exp(s_{\theta}(x,y))}{\sum_{y^{\prime}\in\mathcal{Y}}\mathrm{count}_{y_{1:K}}(y^{\prime})\exp(s_{\theta}(x,y^{\prime}))}\right]
$$
where  $\mathrm{count}_{y_{1:K}}(y)$  is the number of times  $y$  appears in  $y_{1:K}$ . The approximation uses  $\mathrm{count}_{y_{1:K}}(y) \approx p_{\theta}(y|x)$  under (8).
The proof exploits the fact that negative examples are drawn from the model and does not generally hold for other negative distributions (e.g., uniformly random). We empirically verify that hard negatives indeed yield a drastically smaller bias compared to random negatives (Section 6.4).
3.2 Adversarial Learning
We complement the bias-based view of hard negatives with an adversarial view. We generalize (5) and define
$$
J_{\mathrm{ADV}}(\theta ,h) = \underset { \begin{array}{c}(x,y_{1})\sim \mathbf{pop}\\ y_{2:K}\sim h(\cdot |x,y_{1}) \end{array} }{\mathbf{E}}[-\log \pi_{\theta}(1|x,y_{1:K})]
$$
where we additionally consider the choice of a hard-negative distribution. The premise of adversarial learning is that it is beneficial for  $\theta$  to consider the worst-case scenario when minimizing this loss. This motivates a nested optimization problem:
$$
\min  _ {\theta \in \mathbb {R} ^ {d}} \max  _ {h \in \mathcal {H}} J _ {\mathrm {A D V}} (\theta , h)
$$
where  $\mathcal{H}$  denotes the class of conditional distributions over  $S\subset \mathcal{V}$  satisfying  $|S\cup \{y_1\} | = K$
Theorem 3.3. Fix  $\theta \in \mathbb{R}^d$ . For any  $(x,y_1)$ , pick
$$
\tilde{y}_{2:K}\in \operatorname *{arg  max}_{\substack{y_{2:K}\in \mathcal{Y}^{K - 1}:\\ |\{y_{1}\ldots y_{K}\} | = K}}\sum_{k = 2}^{K}s_{\theta}(x,y_{k})
$$
breaking ties arbitrarily, and define the point-mass distribution over  $\mathcal{Y}^{K - 1}$
$$
\tilde {h} \left(y _ {2: K} \mid x, y _ {1}\right) = \left[ \left[ y _ {k} = \tilde {y} _ {k} \forall k = 2 \dots K \right] \right]
$$
Then  $\tilde{h}\in \arg \max_{h\in \mathcal{H}}J_{\mathrm{ADV}}(\theta ,h)$
Proof.  $\max_{h\in \mathcal{H}}J_{\mathrm{ADV}}(\theta ,h)$  is equivalent to
$$
\max_{h\in \mathcal{H}}\underset { \begin{array}{c}(x,y_{1})\sim \mathbf{pop}\\ y_{2:K}\sim h(\cdot |x,y_{1}) \end{array} }{\mathbf{E}}\left[\log \sum_{k = 1}^{K}\exp \left(s_{\theta}(x,y_{k})\right)\right]
$$
The expression inside the expectation is maximized by  $\tilde{y}_{2:K}$  by the monotonicity of log and exp, subject to the constraint that  $|\{y_{1}\ldots y_{K}\} | = K$ .  $\tilde{h}\in \mathcal{H}$  achieves this maximum.
4 Score Function
Along with the choice of negatives, the choice of the score function  $s_{\theta} : \mathcal{X} \times \mathcal{Y} \to \mathbb{R}$  is a critical component of NCE in practice. There is a clear tradeoff between performance and efficiency in modeling the cross interaction between the input-label
pair  $(x, y)$ . This trade-off spurred many recent works to propose various architectures in search of a sweet spot (Humeau et al., 2020; Luan et al., 2020), but they are developed in isolation of one another and difficult to compare. In this section, we give a general algebraic form of the score function that subsumes many of the existing works as special cases.
4.1 General Form
We focus on the standard setting in NLP in which  $x \in \mathcal{V}^T$  and  $y \in \mathcal{V}^{T'}$  are sequences of tokens in a vocabulary  $\mathcal{V}$ . Let  $E(x) \in \mathbb{R}^{H \times T}$  and  $F(y) \in \mathbb{R}^{H \times T'}$  denote their encodings, typically obtained from the final layers of separate pretrained transformers like BERT (Devlin et al., 2019). We follow the convention popularized by BERT and assume the first token is a special symbol (i.e., [CLS]), so that  $E_1(x)$  and  $F_1(y)$  represent single-vector summaries of  $x$  and  $y$ . We have the following design choices:
- Direction: If  $x \to y$ , define the query  $Q = E(x)$  and key  $K = F(y)$ . If  $y \to x$ , define the query  $Q = F(y)$  and key  $K = E(x)$ .
- Reduction: Given integers  $m, m'$ , reduce the number of columns in  $Q$  and  $K$  to obtain  $Q_{m} \in \mathbb{R}^{H \times m}$  and  $K_{m'} \in \mathbb{R}^{H \times m'}$ . We can simply select leftmost columns, or introduce an additional layer to perform the reduction.
- Attention: Choose a column-wise attention Attn:  $A \mapsto \bar{A}$  either Soft or Hard. If Soft,  $\bar{A}_t = \operatorname{softmax}(A_t)$  where the subscript denotes the column index. If Hard,  $\bar{A}_t$  is a vector of zeros with exactly one 1 at index  $\arg \max_i [A_t]_i$ .
Given the design choices, we define the score of  $(x,y)$  as
$$
s _ {\theta} (x, y) = 1 _ {m} ^ {\top} Q _ {m} ^ {\top} K _ {m ^ {\prime}} \operatorname {A t t n} \left(K _ {m ^ {\prime}} ^ {\top} Q _ {m}\right) \tag {9}
$$
where  $1_{m}$  is a vector of  $m$  1s that aggregates query scores. Note that the query embeddings  $Q_{m}$  double as the value embeddings. The parameter vector  $\theta \in \mathbb{R}^d$  denotes the parameters of the encoders  $E,F$  and the optional reduction layer.
4.2 Examples
Dual encoder. Choose either direction  $x \to y$  or  $y \to x$ . Select the leftmost  $m = m' = 1$  vectors in  $Q$  and  $K$  as the query and key. The choice of attention has no effect. This recovers the standard dual
encoder used in many retrieval problems (Gupta et al., 2017; Lee et al., 2019; Logeswaran et al., 2019; Wu et al., 2020; Karpukhin et al., 2020; Guu et al., 2020):  $s_{\theta}(x,y) = E_1(x)^\top F_1(y)$ .
Poly-encoder. Choose the direction  $y \to x$ . Select the leftmost  $m = 1$  vector in  $F(y)$  as the query. Choose an integer  $m'$  and compute  $K_{m'} = E(x)\mathrm{Soft}(E(x)^\top O)$  where  $O \in \mathbb{R}^{H \times m'}$  is a learnable parameter ("code" embeddings). Choose soft attention. This recovers the poly-encoder (Humeau et al., 2020):  $s_\theta(x,y) = F_1(y)^\top C_{m'}(x,y)$  where  $C_{m'}(x,y) = K_{m'}\mathrm{Soft}\left(K_{m'}^\top F_1(y)\right)$ . Similar architectures without length reduction have been used in previous works, for instance the neural attention model of Ganea and Hofmann (2017).
Sum-of-max. Choose the direction  $x \to y$ . Select all  $m = T$  and  $m' = T'$  vectors in  $E(x)$  and  $F(y)$  as the query and key. Choose  $\mathrm{Attn} = \mathrm{Hard}$ . This recovers the sum-of-max encoder (aka., Colbert) (Khattab and Zaharia, 2020):  $s_{\theta}(x,y) = \sum_{t=1}^{T} \max_{t'=1}^{T'} E_t(x)^\top F_{t'}(y)$ .
Multi-vector. Choose the direction  $x \to y$ . Select the leftmost  $m = 1$  and  $m' = 8$  vectors in  $E(x)$  and  $F(y)$  as the query and key. Choose  $\mathrm{Attn} = \mathrm{Hard}$ . This recovers the multi-vector encoder (Luan et al., 2020):  $s_{\theta}(x, y) = \max_{t' = 1}^{m'} E_1(x)^\top F_{t'}(y)$ . It reduces computation to fast dot products over cached embeddings, but is less expressive than the sum-of-max.
The abstraction (9) is useful because it generates a spectrum of architectures as well as unifying existing ones. For instance, it is natural to ask if we can further improve the poly-encoder by using  $m > 1$  query vectors. We explore these questions in experiments.
We discuss related work to better contextualize our contributions. There is a body of work on developing unbiased estimators of the population distribution by modifying NCE. The modifications include learning the normalization term as a model parameter (Gutmann and Hyvarinen, 2012; Mnih and Teh, 2012) and using a bias-corrected score function (Ma and Collins, 2018). However, they assume unconditional negative distributions and do not explain the benefit of hard negatives in NCE (Gillick et al., 2019; Wu et al., 2020; Karpukhin et al., 2020; Févry et al., 2020). In contrast, we
directly consider the hard-negative NCE loss used in practice (5), and justify it as a biased estimator of the gradient of the cross-entropy loss.
Our work is closely related to prior works on estimating the gradient of the cross-entropy loss, again by modifying NCE. They assume the following loss (Bengio and Senecal, 2008), which we will denote by  $J_{\mathrm{PRIOR}}(\theta)$ :
$$
\underset {y _ {2: K} \sim \nu (\cdot | x, y _ {1}) ^ {K}} {\mathbf {E}} \left[ - \log \frac {\exp \left(\bar {s} _ {\theta} (x , y _ {1} , y _ {1})\right)}{\sum_ {k = 1} ^ {K} \exp \left(\bar {s} _ {\theta} (x , y _ {1} , y _ {k})\right)} \right] \tag {10}
$$
Here,  $\nu (\cdot |x,y_1)$  is a conditional distribution over  $\mathcal{V}\backslash \{y_1\}$ , and  $\bar{s}_{\theta}(x,y^{\prime},y)$  is equal to  $s_{\theta}(x,y)$  if  $y = y^{\prime}$  and  $s_\theta (x,y) - \log ((K - 1)\nu (y|x,y_1))$  otherwise. It can be shown that  $\nabla J_{\mathrm{PRIOR}}(\theta) = \nabla J_{\mathrm{CE}}(\theta)$  iff  $\nu (y|x,y_1)\propto \exp (s_\theta (x,y))$  for all  $y\in \mathcal{V}\backslash \{y_1\}$  (Blanc and Rendle, 2018). However, (10) requires adjusting the score function and iid negative examples, thus less aligned with practice than (5). The bias analysis of  $\nabla J_{\mathrm{PRIOR}}(\theta)$  for general  $\nu (\cdot |x,y_1)$  is also significantly more complicated than Theorem 3.1 (Rawat et al., 2019).