Breaking the Linear Iteration Cost Barrier for Some Well-known Conditional Gradient Methods Using MaxIP Data-structures

1 Introduction . Conditional gradient methods ( CGM ) , such as Frank-Wolfe and its variants , are well-known optimization approaches that have been extensively used in modern machine learning . For example , CGM has been applied to kernel methods [ 1 , 2 ] , structural learning [ 3 ] and online learning [ 4 , 5 , 6 ] . Running Time Acceleration in Optimization : Recent years have witnessed the success of largescale machine learning models trained on vast amounts of data . In this learning paradigm , the computational overhead of most successful models is dominated by the optimization process [ 7 , 8 ] . Therefore , reducing the running time of the optimization algorithm is of practical importance . The total running time in optimization can be decomposed into two components : ( 1 ) the number of iterations towards convergence , ( 2 ) the cost spent in each iteration . Reducing the number of iterations requires a better understanding of the geometric proprieties of the problem at hand and the invention of better potential functions to analyze the progress of the algorithm [ 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . Reducing the cost spent per iteration usually boils down to designing problem-specific discrete data-structures . In the last few years , we have seen a remarkable growth of using data-structures to reduce iteration cost [ 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 ] . MaxIP Data-structures for Iteration Cost Reduction : A well-known strategy in optimization , with CGM , is to perform a greedy search over the weight vectors [ 9 , 10 , 13 , 16 , 25 ] or training 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . samples [ 26 , 27 ] in each iteration . In this situation , the cost spent in each iteration is linear in the number of parameters . In practical machine learning , recent works [ 28 , 29 , 30 , 31 , 32 ] formulate this linear cost in iterative algorithms as an approximate maximum inner product search problem ( MaxIP ) . They speed up the amortized cost per iteration via efficient data-structures from recent advances in approximate MaxIP [ 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ] . In approximate MaxIP data-structures , locality sensitive hashing ( LSH ) achieves promising performance with efficient random projection based preprocessing strategies [ 33 , 34 , 35 , 36 ] . Such techniques are widely used in practice for cost reduction in optimization . [ 28 ] proposes an LSH based gradient sampling approach that reduces the total empirical running time of the adaptive gradient descent . [ 29 ] formulates the forward propagation of deep neural network as a MaxIP problem and uses LSH to select a subset of neurons for backpropagation . Therefore , the total running time of neural network training could be reduced to sublinear in the number of neurons . [ 31 ] extends this idea with system-level design for further acceleration , and [ 30 ] modifies the LSH with learning and achieves promising acceleration in attention-based language models . [ 32 ] formulates the greedy step in iterative machine teaching ( IMT ) as a MaxIP problem and scale IMT to large datasets with LSH . Challenges of Sublinear Iteration Cost CGM : Despite the practical success of cost-efficient iterative algorithms with approximate MaxIP data-structure , the theoretical analysis of its combination with CGM is not well-understood . In this paper , we focus on this combination and target answering the following questions : ( 1 ) how to transform the iteration step of CGM algorithms into an approximate MaxIP problem ? ( 2 ) how does the approximate error in MaxIP affect CGM in the total number of iterations towards convergence ? ( 3 ) how to adapt approximate MaxIP data structure for iterative CGM algorithms ? Our Contributions : We propose a theoretical formulation for combining approximate MaxIP and convergence guarantees of CGM . In particular , we start with the popular Frank-Wolfe algorithm over the convex hull where the direction search in each iteration is a MaxIP problem . Next , we propose a sublinear iteration cost Frank-Wolfe algorithm using LSH type MaxIP data-structures . We then analyze the trade-off of approximate MaxIP and its effect on the number of iterations needed by CGM to converge . We show that the approximation error caused by LSH only leads to a constant multiplicative factor increase in the number of iterations . As a result , we retain the sub-linearly of LSH , with respect to the number of parameters , and at the same time retain the same asymptotic convergence as CGMs . We summarize our complete contributions as follows . • We give the first theoretical CGM formulation that achieves provable sublinear time cost per iteration . We also extend this result into Frank-Wolfe algorithm , Herding algorithm , and policy gradient method . • We propose a pair of efficient transformations that formulates the direction search in FrankWolfe algorithm as a projected approximate MaxIP problem . • We present the theoretical results that the proposed sublinear Frank-Wolfe algorithm asymptotically preserves the same order in the number of iterations towards convergence . Furthermore , we analyze the trade-offs between the saving in iteration cost and the increase in the number of iterations to accelerate total running time . • We identify the problems of LSH type approximate MaxIP for cost reduction in the popular CGM methods and propose corresponding solutions . The following sections are organized as below : Section 2 introduces the related works on datastructures and optimization , Section 3 introduces our algorithm associated with the main statements convergence , Section 4 provides the proof sketch of the main statements , Section 5 presents the societal impact and Section 6 concludes the paper . 2 Related work . 2.1 Maximum Inner Product Search for Machine Learning . Maximum Inner Product Search ( MaxIP ) is a fundamental problem with applications in machine learning . Given a query x ∈ Rd and an n-vector dataset Y ⊂ Rd , MaxIP targets at searching for y ∈ Y that maximizes the inner product x > y . The naive MaxIP solution takes O ( dn ) by comparing x with each y ∈ Y . To accelerate this procedure , various algorithms are proposed to reduce the running time of MaxIP [ 33 , 34 , 36 , 35 , 37 , 38 , 43 , 44 , 39 , 45 , 40 , 41 , 42 ] . We could categorize the MaxIP approaches into two categories : reduction methods and non-reduction methods . The reduction methods use transformations that transform approximate MaxIP into approximate nearest neighbor search ( ANN ) and solve it with ANN data-structures . One of the popular data-structure is locality sensitive hashing [ 46 , 47 ] . Definition 2.1 ( Locality Sensitive Hashing ) . Let c > 1 denote a parameter . Let p1 , p2 ∈ ( 0 , 1 ) denote two parameters and p1 > p2 . We say a function family H is ( r , c · r , p1 , p2 ) -sensitive if and only if , for any vectors x , y ∈ Rd , for any h chosen uniformly at random fromH , we have : • if ‖x− y‖2 ≤ r , then Prh∼H [ h ( x ) = h ( y ) ] ≥ p1 , • if ‖x− y‖2 ≥ c · r , then Prh∼H [ h ( x ) = h ( y ) ] ≤ p2 . Here we define the LSH functions for euclidean distance . LSH functions could be used for search in cosine [ 48 , 49 ] or Jaccard similarity [ 50 , 51 ] . [ 33 ] first shows that MaxIP could be solved by ` 2 LSH and asymmetric transformations . After that , [ 34 , 36 , 35 , 43 ] propose a series of methods to solve MaxIP via LSH functions for other distance measures . Besides LSH , graph-based ANN approaches [ 38 ] could also be used after reduction . On the other hand , the non-reduction method directly builds data-structures for approximate MaxIP . [ 37 , 42 ] use quantization to approximate the inner product distance and build codebooks for efficient approximate MaxIP . [ 38 , 44 ] propose a greedy algorithm for approximate MaxIP under computation budgets . [ 39 , 40 , 41 ] directly construct navigable graphs that achieve state-of-the-art empirical performance . Recently , there is a remarkable growth in applying data-structures for machine learning [ 52 , 53 , 54 ] . Following the paradigm , approximate MaxIP data-structures have been applied to overcome the efficiency bottleneck of various machine learning algorithms . [ 38 ] formulates the inference of a neural network with a wide output layer as a MaxIP problem and uses a graph-based approach to reduce the inference time . In the same inference task , [ 55 ] proposes a learnable LSH data-structure that further improves the inference efficiency with less energy consumption . In neural network training , [ 29 , 30 , 31 ] uses approximate MaxIP to retrieve interested neurons for backpropagation . In this way , the computation overhead of gradient updates in neural networks could be reduced . In linear regression and classification models , [ 28 ] uses approximate MaxIP data-structures to retrieve the samples with large gradient norm and perform standard gradient descent , which improves the total running time for stochastic gradient descent . [ 32 ] proposes a scalable machine teaching algorithm that enables iterative teaching in large-scale datasets . In bandit problem , [ 56 ] proposes an LSH based algorithm that solves linear bandits problem with sublinear time complexity . Despite the promising empirical results , there is little theoretical analysis on approximate MaxIP for machine learning . We summarize the major reasons as : ( 1 ) Besides LSH , the other approximate MaxIP data-structures do not provide theoretical guarantees on time and space complexity . ( 2 ) Current approaches treat data-structures and learning dynamics separately . There is no joint analysis on the effect of approximate MaxIP for machine learning . 2.2 Projection-free Optimization . Frank-Wolfe algorithm [ 25 ] is a projection-free optimization method with wide applications in convex [ 9 , 10 ] and non-convex optimizations [ 11 , 12 ] . The procedure of Frank-Wolfe algorithm could be summarized as two steps : ( 1 ) given the gradient , find a vector in the feasible domain that has the maximum inner product , ( 2 ) update the current weight with the retrieved vector . Formally , given a function g : Rd → R over a convex set S , starting from an initial weight w0 , the Frank-Wolfe algorithm updates the weight with learning rate η follows : st ← arg min s∈S 〈s , ∇g ( wt ) 〉 wt+1 ← ( 1− ηt ) · wt + ηt · st . Previous literature focuses on reducing the number of iterations for Frank-Wolfe algorithm over specific domains such as ` p balls [ 9 , 10 , 13 , 14 ] . The cost reduction in the iterative procedure of Frank-Wolfe algorithm is hardly discussed except [ 57 ] . In this work , we focus on the Frank-Wolfe algorithm over the convex hull of a finite feasible set . This formulation is more general and it includes recent Frank-Wolfe applications in probabilistic modeling [ 1 , 2 ] , structural learning [ 3 ] and policy optimization [ 5 ] .

Summary: The paper proposes a strategy to reduce the per-iteration cost of Frank-Wolfe, Herding and policy gradient methods. The algorithm is based on maximum inner product search (MIPS), equipped with proper projection and data transformations. These techniques transforms the problem into an ANN problem, where LSH is applied to achieve improved sublinear complexity. Theorems with rigorous statements on the improvement in complexity are provided on three algorithms, with clearly sketched proof to follow. Practical tricks using quantization and the utility-efficiency tradeoff is also discussed.