Breaking the Linear Iteration Cost Barrier for Some Well-known Conditional Gradient Methods Using MaxIP Data-structures

1 Introduction . Conditional gradient methods ( CGM ) , such as Frank-Wolfe and its variants , are well-known optimization approaches that have been extensively used in modern machine learning . For example , CGM has been applied to kernel methods [ 1 , 2 ] , structural learning [ 3 ] and online learning [ 4 , 5 , 6 ] . Running Time Acceleration in Optimization : Recent years have witnessed the success of largescale machine learning models trained on vast amounts of data . In this learning paradigm , the computational overhead of most successful models is dominated by the optimization process [ 7 , 8 ] . Therefore , reducing the running time of the optimization algorithm is of practical importance . The total running time in optimization can be decomposed into two components : ( 1 ) the number of iterations towards convergence , ( 2 ) the cost spent in each iteration . Reducing the number of iterations requires a better understanding of the geometric proprieties of the problem at hand and the invention of better potential functions to analyze the progress of the algorithm [ 9 , 10 , 11 , 12 , 13 , 14 , 15 ] . Reducing the cost spent per iteration usually boils down to designing problem-specific discrete data-structures . In the last few years , we have seen a remarkable growth of using data-structures to reduce iteration cost [ 16 , 17 , 18 , 19 , 20 , 21 , 22 , 23 , 24 ] . MaxIP Data-structures for Iteration Cost Reduction : A well-known strategy in optimization , with CGM , is to perform a greedy search over the weight vectors [ 9 , 10 , 13 , 16 , 25 ] or training 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . samples [ 26 , 27 ] in each iteration . In this situation , the cost spent in each iteration is linear in the number of parameters . In practical machine learning , recent works [ 28 , 29 , 30 , 31 , 32 ] formulate this linear cost in iterative algorithms as an approximate maximum inner product search problem ( MaxIP ) . They speed up the amortized cost per iteration via efficient data-structures from recent advances in approximate MaxIP [ 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ] . In approximate MaxIP data-structures , locality sensitive hashing ( LSH ) achieves promising performance with efficient random projection based preprocessing strategies [ 33 , 34 , 35 , 36 ] . Such techniques are widely used in practice for cost reduction in optimization . [ 28 ] proposes an LSH based gradient sampling approach that reduces the total empirical running time of the adaptive gradient descent . [ 29 ] formulates the forward propagation of deep neural network as a MaxIP problem and uses LSH to select a subset of neurons for backpropagation . Therefore , the total running time of neural network training could be reduced to sublinear in the number of neurons . [ 31 ] extends this idea with system-level design for further acceleration , and [ 30 ] modifies the LSH with learning and achieves promising acceleration in attention-based language models . [ 32 ] formulates the greedy step in iterative machine teaching ( IMT ) as a MaxIP problem and scale IMT to large datasets with LSH . Challenges of Sublinear Iteration Cost CGM : Despite the practical success of cost-efficient iterative algorithms with approximate MaxIP data-structure , the theoretical analysis of its combination with CGM is not well-understood . In this paper , we focus on this combination and target answering the following questions : ( 1 ) how to transform the iteration step of CGM algorithms into an approximate MaxIP problem ? ( 2 ) how does the approximate error in MaxIP affect CGM in the total number of iterations towards convergence ? ( 3 ) how to adapt approximate MaxIP data structure for iterative CGM algorithms ? Our Contributions : We propose a theoretical formulation for combining approximate MaxIP and convergence guarantees of CGM . In particular , we start with the popular Frank-Wolfe algorithm over the convex hull where the direction search in each iteration is a MaxIP problem . Next , we propose a sublinear iteration cost Frank-Wolfe algorithm using LSH type MaxIP data-structures . We then analyze the trade-off of approximate MaxIP and its effect on the number of iterations needed by CGM to converge . We show that the approximation error caused by LSH only leads to a constant multiplicative factor increase in the number of iterations . As a result , we retain the sub-linearly of LSH , with respect to the number of parameters , and at the same time retain the same asymptotic convergence as CGMs . We summarize our complete contributions as follows . • We give the first theoretical CGM formulation that achieves provable sublinear time cost per iteration . We also extend this result into Frank-Wolfe algorithm , Herding algorithm , and policy gradient method . • We propose a pair of efficient transformations that formulates the direction search in FrankWolfe algorithm as a projected approximate MaxIP problem . • We present the theoretical results that the proposed sublinear Frank-Wolfe algorithm asymptotically preserves the same order in the number of iterations towards convergence . Furthermore , we analyze the trade-offs between the saving in iteration cost and the increase in the number of iterations to accelerate total running time . • We identify the problems of LSH type approximate MaxIP for cost reduction in the popular CGM methods and propose corresponding solutions . The following sections are organized as below : Section 2 introduces the related works on datastructures and optimization , Section 3 introduces our algorithm associated with the main statements convergence , Section 4 provides the proof sketch of the main statements , Section 5 presents the societal impact and Section 6 concludes the paper . 2 Related work . 2.1 Maximum Inner Product Search for Machine Learning . Maximum Inner Product Search ( MaxIP ) is a fundamental problem with applications in machine learning . Given a query x ∈ Rd and an n-vector dataset Y ⊂ Rd , MaxIP targets at searching for y ∈ Y that maximizes the inner product x > y . The naive MaxIP solution takes O ( dn ) by comparing x with each y ∈ Y . To accelerate this procedure , various algorithms are proposed to reduce the running time of MaxIP [ 33 , 34 , 36 , 35 , 37 , 38 , 43 , 44 , 39 , 45 , 40 , 41 , 42 ] . We could categorize the MaxIP approaches into two categories : reduction methods and non-reduction methods . The reduction methods use transformations that transform approximate MaxIP into approximate nearest neighbor search ( ANN ) and solve it with ANN data-structures . One of the popular data-structure is locality sensitive hashing [ 46 , 47 ] . Definition 2.1 ( Locality Sensitive Hashing ) . Let c > 1 denote a parameter . Let p1 , p2 ∈ ( 0 , 1 ) denote two parameters and p1 > p2 . We say a function family H is ( r , c · r , p1 , p2 ) -sensitive if and only if , for any vectors x , y ∈ Rd , for any h chosen uniformly at random fromH , we have : • if ‖x− y‖2 ≤ r , then Prh∼H [ h ( x ) = h ( y ) ] ≥ p1 , • if ‖x− y‖2 ≥ c · r , then Prh∼H [ h ( x ) = h ( y ) ] ≤ p2 . Here we define the LSH functions for euclidean distance . LSH functions could be used for search in cosine [ 48 , 49 ] or Jaccard similarity [ 50 , 51 ] . [ 33 ] first shows that MaxIP could be solved by ` 2 LSH and asymmetric transformations . After that , [ 34 , 36 , 35 , 43 ] propose a series of methods to solve MaxIP via LSH functions for other distance measures . Besides LSH , graph-based ANN approaches [ 38 ] could also be used after reduction . On the other hand , the non-reduction method directly builds data-structures for approximate MaxIP . [ 37 , 42 ] use quantization to approximate the inner product distance and build codebooks for efficient approximate MaxIP . [ 38 , 44 ] propose a greedy algorithm for approximate MaxIP under computation budgets . [ 39 , 40 , 41 ] directly construct navigable graphs that achieve state-of-the-art empirical performance . Recently , there is a remarkable growth in applying data-structures for machine learning [ 52 , 53 , 54 ] . Following the paradigm , approximate MaxIP data-structures have been applied to overcome the efficiency bottleneck of various machine learning algorithms . [ 38 ] formulates the inference of a neural network with a wide output layer as a MaxIP problem and uses a graph-based approach to reduce the inference time . In the same inference task , [ 55 ] proposes a learnable LSH data-structure that further improves the inference efficiency with less energy consumption . In neural network training , [ 29 , 30 , 31 ] uses approximate MaxIP to retrieve interested neurons for backpropagation . In this way , the computation overhead of gradient updates in neural networks could be reduced . In linear regression and classification models , [ 28 ] uses approximate MaxIP data-structures to retrieve the samples with large gradient norm and perform standard gradient descent , which improves the total running time for stochastic gradient descent . [ 32 ] proposes a scalable machine teaching algorithm that enables iterative teaching in large-scale datasets . In bandit problem , [ 56 ] proposes an LSH based algorithm that solves linear bandits problem with sublinear time complexity . Despite the promising empirical results , there is little theoretical analysis on approximate MaxIP for machine learning . We summarize the major reasons as : ( 1 ) Besides LSH , the other approximate MaxIP data-structures do not provide theoretical guarantees on time and space complexity . ( 2 ) Current approaches treat data-structures and learning dynamics separately . There is no joint analysis on the effect of approximate MaxIP for machine learning . 2.2 Projection-free Optimization . Frank-Wolfe algorithm [ 25 ] is a projection-free optimization method with wide applications in convex [ 9 , 10 ] and non-convex optimizations [ 11 , 12 ] . The procedure of Frank-Wolfe algorithm could be summarized as two steps : ( 1 ) given the gradient , find a vector in the feasible domain that has the maximum inner product , ( 2 ) update the current weight with the retrieved vector . Formally , given a function g : Rd → R over a convex set S , starting from an initial weight w0 , the Frank-Wolfe algorithm updates the weight with learning rate η follows : st ← arg min s∈S 〈s , ∇g ( wt ) 〉 wt+1 ← ( 1− ηt ) · wt + ηt · st . Previous literature focuses on reducing the number of iterations for Frank-Wolfe algorithm over specific domains such as ` p balls [ 9 , 10 , 13 , 14 ] . The cost reduction in the iterative procedure of Frank-Wolfe algorithm is hardly discussed except [ 57 ] . In this work , we focus on the Frank-Wolfe algorithm over the convex hull of a finite feasible set . This formulation is more general and it includes recent Frank-Wolfe applications in probabilistic modeling [ 1 , 2 ] , structural learning [ 3 ] and policy optimization [ 5 ] .

Summary: This paper focuses on the iterative conditional gradient methods and proposes a way to improve the per-iteration cost of such methods without significantly improving the number of iterations needed for convergence. The paper presents a way to pose the computational bottleneck in each iteration as a maximum inner product search (MaxIP) problem and leverages existing LSH-based approximate MaxIP schemes to theoretically improve the per-iteration cost while also quantifying the convergence rate of such a conditional gradient method that introduces approximation in each iteration. Specifically, the conditional gradient method involved in the Frank-Wolfe scheme, the kernel herding scheme and the Policy gradient scheme are studied.