Title: EXTREME CLASSIFICATION VIA ADVERSARIAL SOFTMAX APPROXIMATION

Abstract: Training a classifier over a large number of classes, known as 'extreme classification', has become a topic of major interest with applications in technology, science, and e-commerce. Traditional softmax regression induces a gradient cost proportional to the number of classes C, which often is prohibitively expensive. A popular scalable softmax approximation relies on uniform negative sampling, which suffers from slow convergence due a poor signal-to-noise ratio. In this paper, we propose a simple training method for drastically enhancing the gradient signal by drawing negative samples from an adversarial model that mimics the data distribution. Our contributions are three-fold: (i) an adversarial sampling mechanism that produces negative samples at a cost only logarithmic in C, thus still resulting in cheap gradient updates; (ii) a mathematical proof that this adversarial sampling minimizes the gradient variance while any bias due to non-uniform sampling can be removed; (iii) experimental results on large scale data sets that show a reduction of the training time by an order of magnitude relative to several competitive baselines.

1. INTRODUCTION:

In many problems in science, healthcare, or e-commerce, one is interested in training classifiers over an enormous number of classes: a problem known as 'extreme classification' (Agrawal et al., 2013; Jain et al., 2016; Prabhu & Varma, 2014; Siblini et al., 2018) . For softmax (aka multinomial) regression, each gradient step incurs a cost proportional to the number of classes C. As this may be prohibitively expensive for large C, recent research has explored more scalable softmax approximations which circumvent the linear scaling in C. Progress in accelerating the training procedure and thereby scaling up extreme classification promises to dramatically improve, e.g., advertising (Prabhu et al., 2018) , recommender systems, ranking algorithms (Bhatia et al., 2015; Jain et al., 2016) , and medical diagnostics (Bengio et al., 2019; Lippert et al., 2017; Baumel et al., 2018) While scalable softmax approximations have been proposed, each one has its drawbacks. The most popular approach due to its simplicity is 'negative sampling' (Mnih & Hinton, 2009; Mikolov et al., 2013) , which turns the problem into a binary classification between so-called 'positive samples' from the data set and 'negative samples' that are drawn at random from some (usually uniform) distribution over the class labels. While negative sampling makes the updates cheaper since computing the gradient no longer scales with C, it induces additional gradient noise that leads to a poor signalto-noise ratio of the stochastic gradient estimate. Improving the signal-to-noise ratio in negative sampling while still enabling cheap gradients would dramatically enhance the speed of convergence.

In this paper, we present an algorithm that inherits the cheap gradient updates from negative sampling while still preserving much of the gradient signal of the original softmax regression problem. Our approach rests on the insight that the signal-to-noise ratio in negative sampling is poor since there is no association between input features and their artificial labels. If negative samples were harder to discriminate from positive ones, a learning algorithm would obtain a better gradient signal close to the optimum. Here, we make these arguments mathematically rigorous and propose a non-uniform sampling scheme for scalably approximating a softmax classification scheme. Instead of sampling labels uniformly, our algorithm uses an adversarial auxiliary model to draw 'fake' labels that are more realistic by taking the input features of the data into account. We prove that such procedure reduces the gradient noise of the algorithm, and in fact minimizes the gradient variance in the limit where the auxiliary model optimally mimics the data distribution.

A useful adversarial model should require only little overhead to be fitted to the data, and it needs to be able to generate negative samples quickly in order to enable inexpensive gradient updates. We propose a probabilistic version of a decision tree that has these properties. As a side result of our approach, we show how such an auxiliary model can be constructed and efficiently trained. Since it is almost hyperparameter-free, it does not cause extra complications when tuning models.

The final problem that we tackle is to remove the bias that the auxiliary model causes relative to our original softmax classification. Negative sampling is typically described as a softmax approximation; however, only uniform negative sampling correctly approximates the softmax. In this paper, we show that the bias due to non-uniform negative sampling can be easily removed at test time.

The stucture of our paper reflects our main contributions as follows:

1. We present a new scalable softmax approximation (Section 2). We show that non-uniform sampling from an auxiliary model can improve the signal-to-noise ratio. The best performance is achieved when this sampling mechanism is adversarial, i.e., when it generates fake labels that are hard to discriminate from the true ones. To allow for efficient training, such adversarial samples need to be generated at a rate sublinear (e.g., logarithmic) in C.

2. We design a new, simple adversarial auxiliary model that satisfies the above requirements (Section 3). The model is based on a probabilistic version of a decision tree. It can be efficiently pre-trained and included into our approach, and requires only minimal tuning.

3. We present mathematical proofs that (i) the best signal-to-noise ratio in the gradient is obtained if the auxiliary model best reflects the true dependencies between input features and labels, and that (ii) the involved bias to the softmax approximation can be exactly quantified and cheaply removed at test time (Section 4).

4. We present experiments on two classification data sets that show that our method outperforms all baselines by at least one order of magnitude in training speed (Section 5).

We discuss related work in Section 6 and summarize our approach in Section 7.

2. AN ADVERSARIAL SOFTMAX APPROXIMATION:

We propose an efficient algorithm to train a classifier over a large set of classes, using an asymptotic equivalence between softmax classification and negative sampling (Subsection 2.1). To speed up convergence, we generalize this equivalence to model-based negative sampling in Subsection 2.2.

2.1. ASYMPTOTIC EQUIVALENCE OF SOFTMAX CLASSIFICATION AND NEGATIVE SAMPLING:

Softmax Classification (Notation). We consider a training data set x,θ) .

D = {(x i , y i )} i=1:N of N data points with K-dimensional feature vectors x i ∈ R K . Each

(θ) = (x,y)∈D -ξ y (x, θ) + log y ∈Y e ξ y (

(1)

While the first term encourages high scores ξ y (x, θ) for the correct labels y, the second term encourages low scores for all labels y ∈ Y, thus preventing degenerate solutions that set all scores to infinity. with the sigmoid function σ(z) = 1/(1 + e -z ). Here, we used the same score functions ξ y as in Eq. 1 but introduced different model parameters φ so that we can distinguish the two models. Gradient steps for neg.sampl (φ) cost only O(K) time as there is no sum over all labels y ∈ Y.

Asymptotic Equivalence. The models in Eqs. 1 and Eq. 2 are exactly equivalent in the nonparametric limit, i.e., if the function class x → ξ y (x, θ) is flexible enough to map x to any possible score. A further requirement is that p n in Eq. 2 is the uniform distribution over Y. If both conditions hold, it follows that if θ * and φ * minimize Eq. 1 and Eq. 2, respectively, they learn identical scores,

EQUATION

)

As a consequence, one is free to choose the loss function that is easier to minimize. While gradient steps are cheaper by a factor of O(C) for negative sampling, the randomly drawn negative samples increase the variance of the stochastic gradient estimator and worsen the signal-to-noise ratio of the gradient, slowing-down convergence. The next section combines the strengths of both approaches.

2.2. ADVERSARIAL NEGATIVE SAMPLING:

Overview. We propose a generalized variant of negative sampling that reduces the gradient noise.

The main idea is to train with negative samples y that are hard to distinguish from positive samples. We draw y from a conditional noise distribution p n (y |x) using an auxiliary model. This introduces a bias, which we remove at prediction time. In summary our proposed approach consists of three steps:

1. Parameterize the noise distribution p n (y |x) by an auxiliary model and fit it to the data set.

2. Train a classifier via negative sampling (Eq. 2) using adversarial negative samples from the auxiliary model fitted in Step 1 above. For our proposed auxiliary model, drawing a negative sample costs only O(k log C) time with some k < K, i.e., it is sublinear in C. 3. The resulting model has a bias. When making predictions, remove the bias by mapping it to an unbiased softmax classifier using the generalized asymptotic equivalence in Eq. 5 below.

We elaborate on the above Step 1 in Section 3. In the present section, we focus instead on Step 2 and its dependency on the choice of noise distribution p n , and on the bias removal (Step 3).

Why Adversarial Noise Improves Learning. We first provide some intuition why uniform negative sampling is not optimal, and how sampling from a non-uniform noise distribution may improve the gradient signal. We argue that the poor gradient signal is caused by the fact that negative samples are too easy to distinguish from positive samples. A data set with many categories is typically comprised of several hierarchical clusters, with large clusters of generic concepts and small sub-clusters of specialized concepts. When drawing negative samples uniformly across the data set, the correct label will likely belong to a different generic concept than the negative sample. For example, an image classifier will therefore quickly learn to distinguish, e.g., dogs from bicycles, but since negative samples from the same cluster are rare, it takes much longer to learn the differences between a Boston Terrier and a French Bulldog. The model quickly learns to assign very low scores ξ y (x, φ) 0 to such 'obviously wrong' labels, making their contribution to the gradient exponentially small,

EQUATION

A similar vanishing gradient problem was pointed out for word embeddings by Chen et al. (2018) .

Here, the vanishing gradient is due to different word frequencies, and a popular solution is therefore to draw negative samples from a nonuniform but unconditional noise distribution p n (y ) based on the empirical word frequencies (Mikolov et al., 2013) . This introduces a bias which does not matter for word embeddings since the focus is not on classification but rather on learning useful representations.

Going beyond frequency-adjusted negative sampling, we show that one can drastically improve the procedure by generating negative samples from an auxiliary model. We therefore propose to generate negative samples y ∼ p n (y |x) conditioned on the input feature x. This has the advantage that the distribution of negative samples can be made much more similar to the distribution of positive samples, leading to a better signal-to-noise ratio. One consequence is that the introduced bias can no longer be ignored, which is what we address next.

Bias Removal. Negative sampling with a nonuniform noise distribution introduces a bias. For a given input feature vector x, labels y with a high noise probability p n (y |x) are frequently drawn as negative samples, causing the model to learn a low score ξ y (x, φ * ). Conversely, a low p n (y |x) leads to an inflated score ξ y (x, φ * ). It turns out that this bias can be easily quantified via a generalization of Eq. 3. We prove in Theorem 1 (Section 4) that in the nonparametric limit for arbitrary p n (y |x),

ξ y (x, θ * ) = ξ y (x, φ * ) + log p n (y|x) + const.

(nonparametric limit and arbitrary p n ).

(5)

Eq. 5 is an asymptotic equivalence between softmax classification (Eq. 1) and generalized negative sampling (Eq. 2). While strict equality holds only in the nonparametric limit, many models are flexible enough that Eq. 5 holds approximately in practice. Eq. 5 allows us to make unbiased predictions by mapping biased negative sampling scores ξ y (x, φ * ) to unbiased softmax scores ξ y (x, θ * ). There is no need to solve for the corresponding model parameters θ * , the scores ξ y (x, θ * ) suffice for predictions.

Regularization. In practice, softmax classification typically requires a regularizer with some strength λ > 0 to prevent overfitting. With the asymptotic equivalence in Eq. 5, regularizing the softmax scores ξ y (x, θ) is similar to regularizing ξ y (x, φ) + log p n (y|x) in the proposed generalized negative sampling method. We thus propose to use the following regularized variant of Eq. 2,

(reg.) neg.sampl. (φ) = 1 N (x,y)∈D -log σ(ξ y (x, φ)) + λ ξ y (x, φ) + log p n (y|x) 2 (6) -log σ(-ξ y (x, φ)) + λ ξ y (x, φ) + log p n (y |x) 2 ; y ∼ p n (y |x).

Comparison to GANs. The use of adversarial negative samples, i.e., negative samples that are designed to 'confuse' the logistic regression in Eq. 2, bears some resemblance to generative adversarial networks (GANs) (Goodfellow et al., 2014) . The crucial difference is that GANs are generative models, whereas we train a discriminative model over a discrete label space Y. The 'generator' p n in our setup only needs to find a rough approximation of the (conditional) label distribution because the final predictive scores in Eq. 5 combine the 'generator scores' log p n (y|x) with the more expressive 'discriminator scores' ξ y (x, φ * ). This allows us to use a very restrictive but efficient generator model (see Section 3 below) that we can keep constant while training the discriminator. By contrast, the focus in GANs is on finding the best possible generator, which requires concurrent training of a generator and a discriminator via a potentially unstable nested min-max optimization.

3. CONDITIONAL GENERATION OF ADVERSARIAL SAMPLES:

Having proposed a general approach for improved negative sampling with an adversarial auxiliary model p n (Section 2), we now describe a simple construction for such a model that satisfies all requirements. The model is essentially a probabilistic version of a decision tree which is able to conditionally generate negative samples by ancestral sampling. Readers who prefer to proceed can skip this section without loosing the main thread of the paper.

Our auxiliary model has the following properties: (i) it can be efficiently fitted to the training data D requiring minimal hyperparameter tuning and subleading computational overhead over the training of the main model; (ii) drawing negative samples y ∼ p n (y |x) scales only as O(log |Y|), thus improving over the linear scaling of the softmax loss function (Eq. 1); and (iii) the log likelihood log p n (y|x) can be evaluated explicitly so that we can apply the bias removal in Eq. 5. Satisfying requirements (i) and (ii) on model efficiency comes at the cost of some model performance. This is an acceptable trade-off since the performance of p n affects only the quality of negative samples.

Model. Our auxiliary model for p n is inspired by the Hierarchical Softmax model due to Morin & Bengio (2005) . It is a balanced probabilistic binary decision tree, where each leaf node is mapped uniquely to a label y ∈ Y. A decision tree imposes a hierarchical structure on Y, which can impede performance if it does not reflect any semantic structure in Y. Morin & Bengio (2005) rely on an explicitly provided semantic hierarchical structure, or 'ontology'. Since an ontology is often not available, we instead construct a hierarchical structure in a data driven way. Our method has some similarity to the approach by Mnih & Hinton (2009) , but it is more principled in that we fit both the model parameters and the hierarchical structure by maximizing a single log likelihood function.

To sample from the model, one walks from the tree's root to some leaf. 

Greedy Model Fitting. We maximize the likelihood L in Eq. 7 over (i) the model parameters w ν and b ν of all nodes ν, and (ii) the hierarchical structure, i.e., the mapping between labels y and leaf nodes. The latter involves an exponentially large search space, making exact maximization intractable. We use a greedy approximation where we recursively split the label set Y into halves and associate each node ν with a subset Y ν ⊆ Y. We start at the root node ν 0 with Y ν0 = Y and finishing at the leaves with a single label per leaf. For each node ν, we maximize the terms in L that depend on w ν and b ν . These terms correspond to data points with a label y ∈ Y ν , leading to the objective

EQUATION

We alternate between a continuous maximization of L ν over w ν and b ν , and a discrete maximization over the binary indicators ζ y ∈ {±1} that define how we split Y ν into two equally sized halves. The continuous optimization is over a convex function and it converges quickly to machine precision with Newton ascent, which is free of hyperparameters like learning rates. For the discrete optimization, we note that changing ξ y for any y ∈ Y ν from -1 to 1 (or from 1 to -1) increases (or decreases) L ν by

EQUATION

Here, the sums over D y run over all data points in D with label y, and the second equality is an algebraic identity of the sigmoid function. We maximize L ν over all ζ y under the boundary condition that the split be into equally sized halves by setting ζ y ← 1 for the half of y ∈ Y ν with largest ∆ y and ζ y ← -1 for the other half. If this changes any ζ y then we switch back to the continuous optimization. Otherwise, we have reached a local optimum for node ν, and we proceed to the next node.

Technical Details. In the interest of clarity, the above description left out the following details. Most importantly, to prioritize efficiency over accuracy, we preprocess the feature vectors x and project them to a smaller space R k with k < K using principal component analysis (PCA). Sampling from p n thus costs only O(k log |Y|) time. This dimensionality reduction only affects the quality of negative samples. The main model (Eq. 2) still operates on the full feature space R K . Second, we add a quadratic regularizer -λ n (||w ν || 2 + b 2 ν ) to L ν , with strength λ n set by cross validation. Third, we introduce uninhabited padding labels if |Y| is not a power of two. We ensure that p n (ỹ|x) = 0 for all padding labels ỹ by setting b ν to a very large positive or negative value if either of ν's children contains only padding labels. Finally, we initialize the optimization with b ν ← 0 and by setting w ν ∈ R k to the dominant eigenvector of the covariance matrix of the set of vectors { x∈Dy x} y∈Yν .

4. THEORETICAL ASPECTS:

We formalize and prove the two main premises of the algorithm proposed in Section 2.2. Theorem 1 below states the equivalence between softmax classification and negative sampling (Eq. 5), and Theorem 2 formalizes the claim that adversarial negative samples maximize the signal-to-noise ratio.

Theorem 1. In the nonparametric limit, the optimal model parameters θ * and φ * that minimize softmax (θ) in Eq. 1 and neg.sampl. (φ) in Eq. 2, respectively, satisfy Eq. 5 for all x in the data set and all y ∈ Y. Here, the "const." term on the right-hand side of Eq. 5 is independent of y.

Proof. Minimizing softmax (θ) fits the maximum likelihood estimate of a model with likelihood p θ (y|x) = e ξy(x,θ) /Z θ (x) with normalization Z θ (x) = y ∈Y e ξ y (x,θ) . In the nonparametric limit, the score functions ξ y (x, θ) are arbitrarily flexible, allowing for a perfect fit, thus p D (y|x) = p θ * (y|x) = e ξy(x,θ * ) /Z θ * (x) (nonparametric limit). ( 10)

Similarly, neg.sampl. (φ) is the maximum likelihood objective of a binomial model that discriminates positive from negative samples. The nonparametric limit admits again a perfect fit so that the learned ratio of positive rate σ(ξ y (x, φ)) to negative rate σ(-ξ y (x, φ)) equals the empirical ratio,

EQUATION

)

where we used the identity σ(z)/σ(-z) = e z . Inserting Eq. 10 for p D (y|x) and taking the logarithm leads to Eq. 5. Here, the "const." term works out to log Z θ * (x), which is indeed independent of y.

Signal-to-Noise Ratio. In preparation for Theorem 2 below, we define a quantitative measure for the signal-to-noise ratio (SNR) in stochastic gradient descent (SGD). In the vicinity of the minimum φ * of a loss function (φ), the gradient g ≈ H (φ -φ * ) is approximately proportional to the Hessian H of at φ * . SGD estimates g via stochastic gradient estimates ĝ, whose noise is measured by the covariance matrix Cov [ĝ, ĝ] . Thus, the eigenvalues {η i } of the matrix A := H Cov[ĝ, ĝ] -1 measure the SNR along different directions in parameter space. We define an overall scalar SNR η as

EQUATION

Here, we sum over the inverses 1/η i rather than η i so that η ≤ min i η i and thus maximizing η encourages large values for all η i . The definition in Eq. 12 has the useful property that η is invariant under arbitrary invertible reparameterization of φ. Expressing φ in terms of new model parameters φ maps H to J H J and Cov[ĝ, ĝ] to J Cov[ĝ, ĝ]J, where J := ∂φ/∂φ is the Jacobian. Inserting into Eq. 12 and using the cyclic property of the trace, Tr[P Q] = Tr[QP ], all Jacobians cancel. Theorem 2. For negative sampling (Eq. 2) in the nonparametric limit, the signal-to-noise ratio η defined in Eq. 12 is maximal if p n = p D , i.e., for adversarial negative samples. Proof. In the nonparametric limit, the scores ξ y (x, φ) can be regarded as independent variables for all x and y. We therefore treat the scores directly as model parameters, using the invariance of η under reparameterization. Using only Eq. 2, Eq. 11, and properties of the σ-function, we show in Appendix A.1 that the Hessian of the loss function is diagonal in this coordinate system, and given by

EQUATION

and that the noise covariance matrix is block diagonal,

EQUATION

where α x,: ≡ (α x,y ) y∈Y denotes a |Y|-dimensional column vector. Thus, the trace in Eq. 12 is

EQUATION

We thus have to maximize y∈Y α x,y for each x in the training set. We find from Eq. 13 and Eq. 11,

EQUATION

EQUATION

with f (z) := 1/(1 + 1/z). Using Jensen's inequality for the concave function f , we find that the right-hand side of Eq. 16 has the upper bound f E pn(y|x) [p D (y|x)/p n (y|x)] = f (1) = 1 2 , which it reaches precisely if the argument of f in Eq. 16 is a constant, i.e., iff p n (y|x) = p D (y|x) ∀y ∈ Y. 

5. RESULTS:

We evaluated the proposed adversarial negative sampling method on two established benchmarks by comparing speed of convergence and predictive performance against five different baselines.

Datasets, Preprocessing and Model. We used the Wikipedia-500K and Amazon-670K data sets from the Extreme Classification Repository (Bhatia et al.) with K = 512-dimensional XML-CNN features (Liu et al., 2017) downloaded from (Saxena) . As oth data sets contain multiple labels per data point we follow the approach in (Ruiz et al., 2018) and keep only the first label for each data point. Table 1 shows the resulting sizes. We fit a liner model with scores ξ y (x, φ) = x w y + b y , where the model parameters φ are the weight vectors w y ∈ R K and biases b y ∈ R for each label y.

Baselines. We compare our proposed method to five baselines: (i) standard negative sampling with a uniform noise distribution; (ii) negative sampling with an unconditional noise distribution p n (y ) set to the empirical label frequencies; (iii) noise contrastive estimation (NCE, see below); (iv) 'Augment and Reduce ' (Ruiz et al., 2018) ; and (v) 'One vs. Each' (Titsias, 2016) . We do not compare to full softmax classification, which would be unfeasible on the large data sets (see Table 1 ; a single epoch of optimizing the full softmax loss would scale as O(N CK)). However, we provide additional results that compare softmax against negative sampling on a smaller data set in Appendix A.2.

NCE (Gutmann & Hyvärinen, 2010) is sometimes used as a synonym for negative sampling in the literature, but the original proposal of NCE is more general and allows for a nonuniform base distribution. We use our trained auxiliary model (Section 3) for the base distribution of NCE. Compared to our proposed method, NCE uses the base distribution only during training and not for predictions. Therefore, NCE has to re-learn everything that is already captured by the base distribution. This is less of an issue in the original setup for which NCE was proposed, namely unsupervised density estimation over a continuous space. By contrast, training a supervised classifier effectively means training a separate model for each label y ∈ Y, which is expensive if Y is large. Thus, having to re-learn what the base distribution already captures is potentially wasteful.

Hyperparameters. We tuned the hyperparameters for each method individually using the validation set. Table 1 shows the resulting hyperparameters. For the proposed method and baselines (i)-(iii) we used an Adagrad optimizer (Duchi et al., 2011 ) and considered learning rates ρ ∈ {0.0003, 0.001, 0.003, 0.01, 0.03} and regularizer strengths (see Eq. 6) λ ∈ {10 -5 , 3 × 10 -5 , . . . , 0.03}. For 'Augment and Reduce' and 'One vs. Each' we used the implementation published by the authors (Ruiz), and tuned the learning rate ρ and prior variance σ 2 0 . For the auxiliary model, we used a feature dimension of k = 16 and regularizer strength λ n = 0.1 for both data sets.

Results. Figure 1 shows our results on the Wikipedia-500K data set (left two plots) and the Amazon-670K data set (right two plots). For each data set, we plot the the predictive log likelihood per test data point (first and third plot) and the predictive accuracy (second and fourth plot). The green curve in each plot shows our proposed adversarial negative sampling methods. Both our method and NCE (orange) start slightly shifted to the right to account for the time to fit the auxiliary model.

Our main observation is that the proposed method converges orders of magnitude faster and reaches better accuracies (second and third plot in Figure 1 ) than all baselines. On the (smaller) Amazon-670K data set, standard uniform and frequency based negative sampling reach a slightly higher predictive [Gutmann et al., 2010] uniform neg. sampl. A&R [Ruiz et al., 2018] OvE [Titsias, 2016] Figure 1 : Learning curves for our proposed adversarial negative sampling method (green) and for five different baselines on two large data sets (see Table 1 ). log likelihood, but our method performs considerably better in terms of predictive accuracy on both data sets. This may be understood as the predictive accuracy is very sensitive to the precise scores of the highest ranked labels, as a small change in these scores can affect which label is ranked highest. With adversarial negative sampling, the training procedure focuses on getting the scores of the highest ranked labels right, thus improving in particular the predictive accuracy.

6. RELATED WORK:

Efficient Evaluation of the Softmax Loss Function. Methods to speed up evaluation of Eq. 1 include augmenting the model by adding auxiliary latent variables that can be marginalized over analytically (Galy-Fajou et al., 2019; Wenzel et al., 2019; Ruiz et al., 2018; Titsias, 2016) . More closely related to our work are methods based on negative sampling (Mnih & Hinton, 2009; Mikolov et al., 2013) and noise contrastive estimation (Gutmann & Hyvärinen, 2010) . Generalizations of negative sampling to non-uniform noise distributions have been proposed, e.g., in (Zhang & Zweigenbaum, 2018; Chen et al., 2018; Wang et al., 2014; Gutmann & Hyvärinen, 2010) . Our method differs from these proposals by drawing the negative samples from a conditional distribution that takes the input feature into account, and by requiring the model to learn only correlations that are not already captured by the noise distribution. We further derive the optimal distribution for negative samples, and we propose an efficient way to approximate it via an auxiliary model. Adversarial training (Miyato et al., 2017 ) is a popular method for training deep generative models (Tu, 2007; Goodfellow et al., 2014) . By contrast, our method trains a discriminative model over a discrete set of labels (see also our comparison to GANs at the end of Section 2.2).

A different sampling-based approximation of softmax classification is 'sampled softmax' (Bengio et al., 2003) . It directly approximates the sum over classes y in the loss (Eq. 1) by sampling, which is biased even for a uniform sampling distribution. A nonuniform sampling distribution can remove or reduce the bias (Bengio & Senécal, 2008; Blanc & Rendle, 2018; Rawat et al., 2019) . By contrast, our method uses negative sampling, and it uses a nonuniform distribution to reduce the gradient variance.

Decision Trees. Decision trees (Somvanshi & Chavan, 2016) are popular in the extreme classification literature (Agrawal et al., 2013; Jain et al., 2016; Prabhu & Varma, 2014; Siblini et al., 2018; Weston et al., 2013; Bhatia et al., 2015; Jasinska et al., 2016) . Our proposed method employs a probabilistic decision tree that is similar to Hierarchical Softmax (Morin & Bengio, 2005; Mikolov et al., 2013) . While decision trees allow for efficient training and sampling in O(log C) time, their hierarchical architecture imposes a structural bias. Our proposed method trains a more expressive model without such a structural bias on top of the decision tree to correct for any structural bias.

7. CONCLUSIONS:

We proposed a simple method to train a classifier over a large set of labels. Our method is based on a scalable approximation to the softmax loss function via a generalized form of negative sampling. By generating adversarial negative samples from an auxiliary model, we proved that we maximize the signal-to-noise ratio of the stochastic gradient estimate. We further show that, while the auxiliary model introduces a bias, we can remove the bias at test time. We believe that due to its simplicity, our method can be widely used, and we publish the code1 of both the main and the auxiliary model.

For both methods, we tested the same hyperparameter combinations as in Section 5 on the validation set using early stopping. For softmax, we extended the range of tested learning rates up to ρ = 10 as higher learning rates turned out to perform better in this method (this can be understood due to the low gradient noise). The optimal hyperparameters for softmax turned out to be a learning rate of ρ = 0.3 and regularization strength λ = 3 × 10 -4 . For negative sampling, we found ρ = 3 × 10 -3 and λ = 3 × 10 -4 .

Results. We evaluated the predictive accuracy for both methods. With the full softmax method, we obtain 33.6% correct predictions on the test set, whereas the predictive accuracy drops to 26.4% with negative sampling. This suggests that, when possible, minimizing the full softmax loss function should be preferred. However, in many cases, the softmax loss function is too expensive.

:

https://github.com/mandt-lab/adversarial-negative-sampling

Figures and Tables:

At each node ν, one makes a binary decision ζ ∈ {±1} whether to continue to the right child (ζ = 1) or to the left child (ζ = -1). Given a feature vector x, we model the likelihood of these decisions as σ ζ(w ν x + b ν ) , where the weight vector w ν and the scalar bias b ν are model parameters associated with node ν. Denoting the unique path π y from the root node ν 0 to the leaf node associated with label y as a sequence of nodes and binary decisions, π y ≡ (ν 0 , ζ 0 ), (ν 1 , ζ 1 ), . . . , the log likelihood of the training set D is thus L := (x,y)∈D log p n (y|x) = (x,y)∈D (ν,ζ)∈πy log σ ζ(w ν x + b ν ) .

data point has a single label y i ∈ Y from a discrete label set Y. A softmax classifier is defined by a set of functions {ξ y } y∈Y that map a feature vector x and model parameters θ to a score ξ y (x, θ) ∈ R for each label y. Its loss function is softmax

Table content: <table/>

Unfortunately, the sum over y ∈ Y makes gradient based minimization of softmax (θ) expensive if the label set Y is large. Assuming that evaluating a single score ξ y (x, θ) takes O(K) time, each gradient step costs O(KC), where C = |Y| is the size of the label set.

Table content: <table><tr><td colspan="4">from the training set and constructs negative samples (x, y ) by drawing random labels y from some</td></tr><tr><td colspan="4">noise distribution p n . One then trains a logistic regression by minimizing the stochastic loss function</td></tr><tr><td>neg.sampl. (φ) =</td><td>-log σ(ξ y (x, φ)) -log σ(-ξ y (x, φ))</td><td>where</td><td>y ∼ p n (2)</td></tr><tr><td>(x,y)∈D</td><td/><td/><td/></tr><tr><td colspan="4">Negative Sampling. Negative sampling turns classification over a large label set Y into binary</td></tr><tr><td colspan="4">classification between so-called positive and negative samples. One draws positive samples (x, y)</td></tr></table>

Sizes of data sets and hyperparameters. N = number of training points; C = number of categories (after preprocessing); ρ = learning rate; λ = regularizer; σ 2 0 = prior variance.

Table content: <table><tr><td/><td>Size of</td><td colspan="2">adv. neg. s. uniform ∝ freq.</td><td/><td/><td/></tr><tr><td>Data set</td><td>data set</td><td>(proposed) neg. s.</td><td>neg. s.</td><td>NCE</td><td>A&amp;R</td><td>OVE</td></tr><tr><td>Wikipedia-500K</td><td colspan="6">N =1,646,302 ρ=0.01 ρ=0.001 ρ=0.003 ρ=0.01 ρ=0.03 ρ=0.02 C=217,240 λ=0.001 λ=0.0001 λ=10 -5 λ=0.003 σ 2 0 =0.1 σ 2 0 =1</td></tr><tr><td>Amazon-670K</td><td>N =490,449 C=213,874</td><td colspan="5">ρ=0.01 ρ=0.01 ρ=0.003 ρ=0.01 ρ=0.1 ρ=0.03 λ=0.001 λ=0.0003 λ=10 -5 λ=0.001 σ 2 0 =10 σ 2 0 =10</td></tr></table>

Peer review (iclr2020):
This work addresses the problem of training softmax classifiers when the number of classes is extreme. The authors improve the negative sampling method which is based on reducing the multi-class problem to a binary class problem by introducing randomly chosen labels in the training. Their idea is generating the fake labels nonuniformly from an adversarial model (a decision tree). They show convincing results of improved learning rate. The work is very technical in nature, but the proposal is presented in detail and in a didactic way with appropriate connections to alternative methods, so that it may be useful for the non-expert (as me). That is the reason why I recommend to accept this work: even not being an expert I found the paper educative in introducing the problem and interesting in explaining the proposal.