A Truly Constant-time Distribution-aware Negative Sampling

1 INTRODUCTION . Neural Networks ( NN ) have successfully pushed the boundaries of many application tasks , such as image or text classification ( Wang et al. , 2017 ; Yao et al. , 2019 ) , speech recognition ( Dong et al. , 2018 ) and recommendation systems ( Zhang et al. , 2015 ; Medini et al. , 2019 ) . Many hard AI problems are currently modeled as massive multiclass or multilabel problems leading to a drastic improvement over prior work . For example , popular NLP models predicts the best word , given the full context observed so far . Such models are becoming the state-of-the-art . Recommendation systems and related Information Retrieval ( IR ) problems are classical examples of machine learning with outrageously large outputs ( Medini et al. , 2019 ; Jain et al. , 2019 ) . In IR , given the user query , the task is to predict few relevant documents ( or products ) from among hundreds of millions possible documents , a typical machine learning problem with massive output space . Owing to the significance of the problem , machine learning with large output space or alternatively also known as extreme classification is a field in itself ( Bengio et al. , 2019 ) . A large number of classes naturally brings a new set of computational and memory challenge . Fortunately , with access to our powerful Graphic Processing Unit ( GPU ) ( Owens et al. , 2008 ) , training processes of large models have been accelerated heavily . That is because GPUs have a unique advantage for matrix multiplication , which usually requires a cubic time algebraic operation ( O ( N3 ) ) and is the major and costly building block of NN computations . However , the number of concurrent operations required in large matrix multiplications for classification with extensive number of classes has reached a limit for further speedups even using GPUs . 1.1 NEGATIVE SAMPLING . The common approach to address this challenge is known as negative sampling ( Pennington et al. , 2014 ; Jean et al. , 2014 ; Rawat et al. , 2019 ; Mikolov et al. , 2013b ) . In Negative Sampling , we only sample a small subset of classes for each input and compute the softmax and cross-entropy function . This subset usually includes the positive ( true ) and a small set of negative ( false ) classes . Negative sampling scales down the computations in the most cumbersome last layer , thereby making training efficient . However , approximating full-softmax with small sub-sample results in poor convergence if the negative samples are not chosen appropriately . For instance , let us take the example of a recommendation system ( predicting products relevant to a query ) with a large number of products . If the input query is ‘ Nike Running Shoes ’ , the true loss concentrates on the specific small number of confusing ( ’ hard ’ ) negative classes like ‘ Adidas Running Shoes ’ . Since the number of classes is huge , random sampling is unlikely to identify this hard negative class . Other heuristics like frequent class sampling as negative samples are also unlikely to find these hard negatives most of the time . Clearly , without discriminating between closely related negative samples , the classifier can not achieve good accuracy . Our experiments on recommendations datasets clearly indicate this sub-optimality of current negative sampling heuristics . If there exists a way to sample the subset of confusing classes from the skewed distribution , the training progress would be largely accelerated . However , as evident from the example , such ground-truth distribution depends on the input sample and current model parameters . Moreover , this distribution varies significantly as training progresses . Consider the same query ’ Nike Running Shoes ’ , initially , when the network has not learned anything and has random weights , all classes are equally confusing . Thus , uniform sampling is optimal initially as the network has just started to learn . As the training progresses , the network ’ s belief starts getting more concentrated on a few classes ; at this time , a negative sample of say ’ baby toys ’ is not at all useful because the network has already learned to tell them apart . The sampling distribution keeps changing , often drastically , as the training progresses . To the best of our knowledge , there does not exist any statistical sampling scheme and implementation for adaptive Negative Sampling , where the cost of maintaining and updating the distribution , per iteration , is O ( 1 ) ( independent of the number of classes ) . This is because the input , current true class , and parameters update all the sampling weights in every iteration . It is widely assumed that there is no such sampling scheme , and hence several heuristic alternatives are proposed . The first set of alternatives use a static distribution . The most popular ones , implemented in TensorFlow , assume a static distribution such as the distribution based on the frequency of classes . Uniform sampling is another popular choice . Learning-based alternatives are also proposed ( Bamler & Mandt , 2020 ) , where a machine learning generator predicts ( or generates ) the negative samples . The sampler is solving the same hard problem , prediction over a large number of classes , as a sub-routine . Most importantly , since the sampling distribution for the same data point shifts drastically throughout training , ML models are likely to suffer . Negative sampling alternatives try to balance the sampling cost with quality . So far , negative sampling methods , other than the ones based on static sampling , have failed to demonstrate any training time improvements over the optimized full softmax implementation over GPUs . Static sampling strategies are known to be fast but lead to poor accuracy . With current strategies , the cost of improving the quality with current alternatives does not seem worth it over the GPU acceleration of softmax . In this paper , we change this . Our work provides a truly constant time adaptive sampling scheme utilizing the recent advances in Locality Sensitive Sampling ( Charikar & Siminelakis , 2017 ; Spring & Shrivastava , 2017a ) . More impressively , we provide an efficient implementation of our proposal on CPU , which outperforms TensorFlow ’ s implementation of softmax and other negative sampling strategies on some of the best available GPUs ( V100 ) in terms of wall-clock training time . Summary of Contributions : 1 ) We propose two efficient schemes for sampling ‘ hard ’ negatives where the negative sampling distribution provably adapts to changing parameters and the data instance . Furthermore , the sampling cost is provably constant ( independent of the number of classes ) . 2 ) We show that our technique is not only provably adaptive but also practical . We provide an efficient CPU implementation , in C++ , of our negative sampling approach . We demonstrate the effectiveness of a truly constant time negative sampler by showing that our implementation significantly outperforms standard TensorFlow on V100 GPU in-wall clock speed , while retaining the accuracy . 3 ) We provide a rigorous evaluation of our proposal with its efficient implementation against full softmax and popular approximations like sampled softmax , frequency-based sampled softmax , top-K activation softmax , differentiation softmax ( D-Softmax ) , and Noise Contrastive Estimation ( NCE ) . We report the time-wise and iteration-wise precision on large recommendation datasets like Amazon670K , WikiLSH-325K , and popular natural language processing dataset Text8 corpus . 1.2 LSH BASED HASH TABLES . In this section , we briefly describe the recent development of using locality sensitive hashing for sampling and estimation ( Chen et al. , 2019b ; Spring & Shrivastava , 2017a ; Charikar & Siminelakis , 2017 ; Spring & Shrivastava , 2017b ) . Locality Sensitive Hashing ( Indyk & Motwani , 1998 ; Indyk & Woodruff , 2006 ) is a widely used paradigm for large scale similarity search and nearest neighbor search . LSH is a family of hash functions with a unique property that vectors ‘ close ’ wrt some distance metric are more likely to have the same hash code as opposed to vectors that are ‘ far ’ from each other . Formally , one sufficient condition for a hash family H to be a LSH family is that the collision probability PrH ( h ( x ) = h ( y ) ) is a monotonically increasing function of the similarity : PrH ( h ( x ) = h ( y ) ) = f ( Sim ( x , y ) ) , ( 1 ) where f is a monotonically increasing function . The idea is to use the hash value of x , i.e. , h ( x ) , to generate key of x in the hash table . We first initialize L hash tables by constructing a meta-LSH hash function usingK independent hash functions for each of them . For details , see ( Andoni & Indyk , 2004 ) . There are three major steps : Pre-processing Phase : Given a dataset of size n , we first insert all the data points into the hash tables using the meta-LSH formed by concatenating K independent LSH hash functions . We only store the index/pointer of the data point in the hash tables instead of the entire vector . The cost of addition is K × L hash computations followed by L insertions in the buckets . Query Phase : During the query phase , we use the same meta-LSH hash to compute the hash codes for the query . Then we probe the corresponding bucket of each table and retrieve samples from it . The union of candidates from all hash tables constitute the samples for the particular query . Update Phase : If an existing element in the database is updated , we can delete it from the hash table and re-add it . The cost is equivalent to twice the insertion cost of an element which is 2×K × L. 1.3 ADAPTIVE SAMPLING VIEW OF LSH Denote pqx be the probability of retrieving x from the datasets , when queried with a given query q . In ( Indyk & Motwani , 1998 ) , it was shown that for ( K , L ) parametrized LSH algorithm the precise form of pqx = 1− ( 1−αK ) L , where α is the collision probability of query q and x under the given LSH function , i.e . α = PrH ( h ( x ) = h ( q ) ) . pqx is monotonic in α which is further monotonic in the similarity between query q and the data element x . Note the similarity measure is dependent on the LSH function in use . See ( Spring & Shrivastava , 2017a ) for details . Constant Time Sampling : It should be noted that the cost of sampling is the cost of querying , which is only K × L. This sampling cost is independent of the number of elements in the data . Clearly , the probability pqx is dependent on the query , and every element x in the data has a different sampling probability . Thus , even though our sampling scheme induces n different sampling probabilities every time the query q is changed , the sampling cost is independent of n , and in-fact is constant if K and L are small constants . All this is assuming one O ( n ) time preprocessing . This efficient sampling view of LSH has been used in a wide range of applications , such as deep neural networks ( Spring & Shrivastava , 2017b ; Chen et al. , 2019a ) , kernel density estimation ( Charikar & Siminelakis , 2017 ) , record linkage ( Chen et al. , 2018 ) , and optimization ( Chen et al. , 2019c ) . Recent advances in fast inner product search using asymmetric LSH has made it possible to sample large inner products ( Shrivastava & Li , 2014 ) . Effectively , given a query q , it is possible to sample an element x from the database with probability proportional to a monotonic function of inner product f ( qTx ) . Here f is a monotonically increasing function .

Summary: When the number of classes is very large, calculating softmax for classification (e.g., in backpropagation) is computationally costly. Approaches based on negative sampling have been used in literature to alleviate this problem. However, most of existing approaches are (argued to be) either inaccurate or computationally costly. This paper proposes to use the well-known LSH (locality sensitive hashing) method to address this problem. In particular, two variants, LSH label and LSH Embedding are showed to speed up the training in terms of time needed to converge compared with a number of baseline methods over three large scale datasets.