LSH Microbatches for Stochastic Gradients:  Value in Rearrangement

1 INTRODUCTION . Metric embeddings of entities that are trained to capture example associations are common representations that also allow for inference of associations not present in the data . Embeddings are used in complex learning tasks or directly applied for similarity and recommendations tasks . Example usage domains includes embeddings of text document from occurrences of words Berry et al . ( 1995 ) ; Dumais ( 1995 ) ; Deerwester et al . ( 1990 ) , users and videos from watch or ratings Koren et al . ( 2009 ) , words from co-occurrence frequencies in a corpus Mikolov et al . ( 2013 ) , and nodes in a graph from co-occurrence in short random walks Perozzi et al . ( 2014 ) . The example associations may involve entities of the same type ( word co-occurrences , video co-watch , social ) or different types ( such as users and products ) and often are distilled by reweighing frequencies of raw interactions Salton & Buckley ( 1988 ) ; Deerwester et al . ( 1990 ) ; Mikolov et al . ( 2013 ) ; Pennington et al . ( 2014 ) . Embeddings are computed by minimizing a loss objective of the form of a sum over example associations . The optimization starts with random initialization followed by gradient updates . In modern applications , the objective can have billions of terms or more , and the de facto method at such scale is stochastic gradient descent ( SGD ) Robbins & Siegmund ( 1971 ) ; Koren ( 2008 ) ; Salakhutdinov et al . ( 2007 ) ; Gemulla et al . ( 2011 ) ; Mikolov et al . ( 2013 ) . The terms ( examples ) are randomly grouped into minibatches . Gradient updates , that are equal in expectation to the full gradient , are then computed sequentially for each minibatch . SGD is much more efficient than working with full gradients and the minibatch size determines the amount of concurrency . There are numerous tunable parameters and methods aimed to improve quality and speed of convergence . Some notable recent work includes per-parameter tuning of the learning rate Duchi et al . ( 2011 ) and altering the distribution of training examples by gradient magnitude Alain et al . ( 2015 ) , negatives selection with triplet loss Schroff et al . ( 2015 ) , clustering Fu & Zhang ( 2017 ) and diversity criteria Zhang et al . ( 2017 ) . In this work we introduce principled schemes that control the arrangement of examples into minibatches . Note that arrangement tuning is separate and orthogonal to optimizations knobs that alter the distribution of training examples , learning rate , or minibatch size . The baseline practice of independent arrangements places examples into minibatches independently at random . This practice is supported by classic SGD convergence analysis and has the upside of controlling the variance of the stochastic gradients . We make a novel case here for an antithesis of independent arrangements which we term coordinated arrangements . Coordinated arrangements are much more likely to place corresponding associations in the same minibatch . We show that coordination offers different upsides : At the micro level , updates are more effective in pulling vectors of similar entities closer . At a macro level , the examples in small fractions of epochs encode ( in expectation ) the similarity structure in the full set of example associations whereas independent arrangement disperse that information . This “ self similarity ” of the training sequence effectively allows a single epoch to act as multiple passes . We specify coordinated arrangements through a distribution on randomized subsets of associations which we refer to as microbatches . Basic coordinated microbatches co-place corresponding associations to the maximum extent possible while adhering to the marginal distribution of training examples . Locality Sensitive Hashing ( LSH ) maps allow for refining our microbatches so that corresponding associations are more likely to be co-placed when the overall similarity of the entities is higher . The LSH maps we apply leverage some available coarse proxy of entity similarity . A readily available first-order proxy is the similarity of the sparse association vectors . Another proxy is an embedding obtained by a weaker model . We present efficient generators of basic and refined microbatches for both LSH functions . Finally , microbatches are independently grouped into minibatches of desired size , which allows us to retain the traditional advantages of independent arrangements at the microbatch level . This design enables us to tune the microbatches to the problem and even the stage of training . We compare the effectiveness of different arrangements through experiments on synthetic stochastic block matrices and on recommendation data sets . The stochastic block data , with its simplicity and symmetry , allows us to factor out the potential effect of other optimizations . We learn that basic coordination is always beneficial earlier in training whereas LSH refinements are more effective later on . We obtain consistent 3 % -30 % reduction in training with respect to the independent arrangements baseline that holds across other training hyperparameters . The paper is organized as follows . Section 2 presents necessary background on the loss objective we use in our experiments and working with minibatches with one-sided gradient updates . In Section 3 we define LSH microbatches and coordinated minibatch arrangements . In We report our experiment results comparing different arrangement methods on stochastic blocks and on recommendation data sets in Section 4 . In Section 5 we examine properties of coordinated arrangements that are helpful for faster convergence . We conclude in Section 6 . 2 PRELIMINARIES . Our data has the form of associations between a set F of focus entities and a set C of context entities . In practice , the focus and context entities could be the same type or even two representations of the same set ( say words or images ) or different types ( users and videos ) . We use κij as the association strength between focus i and context j . In practice , the association strength can be derived from frequencies in the raw data or from an associated value ( for example , numeric rating or watch time ) . An embedding is a set of vectors fi , cj ∈ < d that is trained to minimize a loss objective that encourages fi · cj to be larger when κij is larger . For concreteness , we focus here on Skip Gram with Negative Sampling ( SGNS ) loss Mikolov et al . ( 2013 ) . Examples of positive associations ( i , j ) are drawn with probability proportional to κij . Random associations are then used as negative examples Hu et al . ( 2008 ) : Each positive example ( i , j ) is matched with a set of negative examples of i with random context entities and similarly j with random focus entities . The negative examples provide an antigravity effect that prevents all embeddings from collapsing into the same vector . We use the notation nij for respective weights of negative associations . The SGNS objective is designed to maximize the log likelihood of these examples . The probability of positive and negative examples are respectively modeled using pij = σ ( fi · cj ) = 1 1 + exp ( −fi · cj ) and 1− pij = σ ( −fi · cj ) = 1 1 + exp ( fi · cj ) . The likelihood function , which we seek to maximize , can then be expressed as Πij p κij ij Πij ( 1 − pij ) nij . We equivalently can minimize the negated log likelihood that turns the objective into a sum : L : = ∑ ij κij log pij + ∑ ij nij log ( 1− pij ) . The optimization is performed by random initialization of the embedding vectors followed by stochastic gradient updates . The stochastic gradients are computed for minibatches of examples that include b positive examples , where ( i , j ) appears with frequency κij/|κ|1 and corresponding sets of negative examples . 2.1 ONE-SIDED UPDATES . We work with one-sided updates , where each minibatch updates only its focus or only its context embedding vectors , and accordingly say that minibatches are designated for focus or context updates . One-sided updates are related to alternating minimization Csiszar & Tusnády ( 1984 ) and to decomposition-coordination approaches Cohen ( 1980 ) . For our purposes , one-sided updates facilitate our coordinated arrangements and also allow more precise matching of corresponding sets of negative examples to positive ones . In a focus-updating minibatch , we will generate a random set of λ context vectors C ′ and for each positive example ( i , j ) we generate λ negative examples ( i , j′ ) for j′ ∈ C ′ . The focus embedding fi is updated to be closer to cj but at the same time repealed ( in expectation ) from C ′ context vectors . With applicable learning rate η , the update has the form : fi ← fi − η∇fi log σ ( fi · cj ) + ∑ j′∈C′ log σ ( −fi · cj′ ) . Symmetrically , for a context-updating minibatch we use a random set of focus vectors F ′ as our negative examples and for each positive example ( i , j ) we perform the update cj ← cj − η∇cj ( log σ ( fi · cj ) + ∑ i′∈F ′ log σ ( −fi′ · cj ) ) . 3 MINIBATCH ARRANGEMENT SCHEMES . Minibatch arrangement schemes determine how examples are organized into minibatches of specified size parameter b . At the core of each arrangement scheme is a distribution B over subsets of positive examples which we call microbatches . Our microbatch distributions have the property that the marginal probability of each example ( i , j ) is always equal to κij/||κ||1 . However , subset probabilities vary between schemes and within a scheme we generally will have different distributions Bf for focus and Bc for context designations . Minibatches are obtained from microbatches as specified in Algorithm 1 for focus updates ( a symmetric construction applies to context updates ) . The input is a microbatch distribution Bf , minibatch size parameter b , and a parameter λ that determines the ratio of negative to positive training examples . We draw independent microbatches until we have a total of b or more positive examples . We then draw λ random contexts C ′ and generate λ negative examples ( i , j′ ) for j′ ∈ C ′ for each positive example ( i , j ) . When training , we alternate between focus and context updating minibatches . Algorithm 1 : Minibatch construction ( Focus updates ) Input : Bf , b , λ // Microbatch distribution , size , negative sampling P , N ← ∅ repeat X ∼ Bf ; P ← P ∪X until |P | ≥ b C′ ← λ contexts selected uniformly at random foreach example pair ( i , j ) ∈ P do foreach j′ ∈ C′ do N ← N ∪ { ( i , j′ ) } return P ∪N The baseline independent arrangement method ( IND ) can be placed in this framework using microbatches that consist of a single positive example ( i , j ) selected with probability κij/||κ||1 ( see Algorithm 2 ) . With coordinated arrangements , the microbatch distribution depends on designation . Algorithm 3 generates basic coordinated microbatches ( COO ) for focus-updates . These microbatches have the form of a set of positive examples with a shared context . Basic microbatches have the property that if κij ≤ κi′j and the positive example ( i , j ) is included in a basic microbatch then the microbatch would also include the positive example ( i′ , j ) . It is instructive to consider the special case of κ with all positive entries being equal : Basic microbatches with focus designation have the form of some context j , and all ( i , j ) with positive κij . Our basic microbatches maximize the co-placement probability of two examples with a shared context while respecting the marginal probabilities . A sym- metric construction applies to context-update microbatches that maximize the co-placement of two examples with a shared focus . We establish that our microbatch generator respects the marginal probabilities , that is , example ( i , j ) is included with probability ∝ κij : Lemma 3.1 . A positive example ( i , j ) is included in a basic coordinated microbatch ( Algorithm 3 ) with focus designation with probability κij/M and in a microbatch with context designation with probability κij/N , where M : = ∑ hMh with Mh : = maxi κih for context h and N : = ∑ hNh with Nh : = maxj κhj for focus h. Proof . Consider focus updates ( apply a symmetric argument for context updates ) . The example ( i , j ) is selected if first context j is selected , which happens with probability Mj/M and then we have u ≤ κij/Mj for independent u ∼ U [ 0 , 1 ] , which happens with probability κij/Mj . Combining , the probability that ( i , j ) is selected is the product of the probabilities of these two events which is κij/M . We preprocess κ and precompute the per-context maxima so that we can efficiently draw a random context with probability proportional to the column maxima . We also generate an index for efficient retrieval , for context j and a threshold value T , of all entries i with κij ≥ T . The preprocessing is linear in the sparsity of κ . Note that it is typical in applications to preprocess the data , in particular , to obtain κ by aggregating and reweighing the frequencies of associations in the raw data . The additional overhead imposed by our preprocessing is minimal . We can observe that given this preprocessing , the microbatch generator is very efficient : It draws a context according to the distribution ( which is O ( 1 ) operation ) and then uses T to index into the start point in the sorted “ column . ”

Summary: The paper presents a method for improving the convergence rate of Stochastic Gradient Descent for learning embeddings by grouping similar training samples together. The basic idea is that gradients computed on a batch of highly associated samples encode related information in a single update that independent samples might take multiple updates to capture. These structured minibatches are constructed by independently combining subsets of positive examples called “microbatches”. Two methods are presented for constructing these microbaches; first by grouping positive examples by shared context (called “basic” microbatches), second by applying Locality Sensitive Hashing to further partition the microbatches into groups that are more likely to contain similar examples.