Distribution Compression in Near-Linear Time

√ n points with Õ ( 1/ √ n ) distributional discrepancy to P. Unfortu- nately , these same algorithms suffer from quadratic or super-quadratic runtime in the sample size n. To address this deficiency , we introduce a simple metaprocedure—Compress++—for speeding up any input thinning algorithm while suffering at most a factor of four in error . When combined with the quadratictime kernel halving and kernel thinning algorithms of Dwivedi and Mackey ( 2021 ) , Compress++ delivers √ n points with O ( √ log n/n ) integration error and better-than-Monte-Carlo maximum mean discrepancy in O ( n log2 n ) time and O ( √ n log2 ( n ) ) space . Moreover , Compress++ enjoys the same near-linear runtime given any quadratic-time input and reduces the runtime of super-quadratic algorithms by a square-root factor . In our benchmarks with high-dimensional Monte Carlo samples and long-running Markov chains targeting challenging differential equation posteriors , Compress++ matches or nearly matches the accuracy of its input algorithm in orders of magnitude less time . 1 INTRODUCTION . Distribution compression—constructing a concise summary of a probability distribution—is at the heart of many learning and inference tasks . For example , in Monte Carlo integration and Bayesian inference , n representative points are sampled i.i.d . or from a Markov chain to approximate expectations and quantify uncertainty under an intractable ( posterior ) distribution ( Robert & Casella , 1999 ) . However , these standard sampling strategies represent a bottleneck in computationally-demanding settings due to their slow root-n Monte Carlo error rate . For instance , the Monte Carlo estimate Pnf , 1n ∑n i=1 f ( xi ) of an unknown expectation Pf , EX∼P [ f ( X ) ] based on n i.i.d . points has Θ ( n− 1 2 ) integration error |Pnf − Pf | , requiring n = 10000 points for 1 % relative error and n = 106 points for 0.1 % error . Such bloated sample representations preclude downstream applications with critically expensive function evaluations like computational cardiology , where a 1000-CPU-hour tissue or organ simulation is required for each sample point ( Niederer et al. , 2011 ; Augustin et al. , 2016 ; Strocchi et al. , 2020 ) , or expert data annotation which can be monetarily expensive Beaugnon et al . ( 2017 ) , and is sometimes coined as the human bottleneck of machine learning . To restore the feasibility of such critically expensive tasks , it is common to thin down the initial sequence of points to a produce a much smaller coreset . The standard thinning approach , select every t-th sample point ( Owen , 2017 ) , is simple to implement but often leads to an substantial increase in error : e.g. , standard thinning n points from a fast-mixing Markov chain yields Ω ( n− 1 4 ) error when n 1 2 points are returned . Recently , Dwivedi & Mackey ( 2021b ; a ) introduced a more effective alternative , kernel thinning , that provides near optimal Õ ( n− 12 ) error when compressing n points down to size n 1 2 . While practical for moderate sample sizes , the runtime of this algorithm scales quadratically with the input size n , making its execution prohibitive for very large input sizes . Our goal is to significantly improve the runtime of such compression algorithms while providing comparable error guarantees . Problem setup Given a sequence Sin of n input points summarizing a target distribution P , our aim is to identify a high quality coreset Sout of size √ n in time nearly linear in n. We measure coreset quality via its integration error |Pf −Poutf | , |Pf − 1|Sout| ∑ x∈Sout f ( x ) | for functions f in the reproducing kernel Hilbert space ( RKHS ) H induced by a given kernel k ( Berlinet & Thomas-Agnan , 2011 ) . We consider both single function error and kernel maximum mean discrepancy ( MMD , Gretton et al. , 2012 ) , the worst-case integration error over the unit RKHS norm ball : MMDk ( P , Pout ) , sup‖f‖k≤1|Pf − Poutf | = ‖Pk− Poutk‖k . Our contributions We introduce a new simple meta procedure—COMPRESS++—that significantly speeds up a generic thinning algorithm while simultaneously inheriting the error guarantees of its input up to a factor of four . A direct application of COMPRESS++ to kernel thinning improves its quadratic O ( n2 ) runtime to near linear O ( n log2 n ) time while maintaining the error guarantees up to a factor four . Since the Õ ( n− 12 ) KT MMD guarantees of Dwivedi & Mackey ( 2021b ) match the Ω ( n− 1 2 ) minimax lower bounds of Tolstikhin et al . ( 2017 ) ; Phillips & Tai ( 2020 ) up to factors of √ log ( n ) and constants depending on d , KT-COMPRESS++ also provides near-optimal MMD compression for a wide range of kernels and distributions P. Moreover , the practical gains from applying COMPRESS++ are substantial : KT thins 65 , 000 points in 10 dimensions in 20m , while KTCOMPRESS++ needs only 1.5m ; KT takes more than a day to thin 250 , 000 points in 100 dimensions , while KT-COMPRESS++ takes less than an hour ( a 32× speed-up ) . COMPRESS++ can also be directly combined with any thinning algorithm , even those that have suboptimal guarantees but often perform well in practice , like kernel herding ( Chen et al. , 2010 ) , support points ( Mak & Joseph , 2018 ) , Stein points ( MCMC ) ( Chen et al. , 2018 ; 2019 ) , and Stein thinning ( Riabiz et al. , 2020a ) , all of which run in Ω ( n2 ) time . As a demonstration , we combine COMPRESS++ with the popular kernel herding algorithm and observe 10-60× speed-ups . In all of our experiments , COMPRESS++ leads to minimal loss in accuracy and , surprisingly , even improves upon herding accuracy for lower-dimensional problems . Overview of COMPRESS++ To define COMPRESS++ , we first introduce COMPRESS : A simple and elegant recursive strategy takes in a halving algorithm and provides an intermediate thinned coreset in significantly faster runtime . COMPRESS divides the coreset into four parts , recursively applies COMPRESS to each , combines the output of each and halves the resulting coreset to get the output . Then COMPRESS++ with a thinning algorithm ALG is defined as follows : In stage one , use COMPRESS with ALG instantiated for 2-thinning , to obtain an intermediate coreset of size 2g √ n for a suitable parameter g , and then in stage two apply ALG directly to further thin it down to √ n points . In this manner , ALG is ever applied to coresets of size 2g √ n or smaller thereby improving the runtime . The parameter g is chosen based on the run-time and known guarantees of the underlying thinning algorithm , but a default choice of 4 can be shown to be optimal in several theoretical settings , and it provided competitive performance across all our experiments . Overall , with n input points and a thinning algorithm with runtime nα , COMPRESS++ uses a outputs √ n points in time Õ ( nα/2 ) with errors similar to those provided by the input thinning algorithm . Notation We write SALG for the coreset outputted by an algorithm ALG and extend our MMD definition for point sequences ( S1 , S2 ) with empirical distributions ( Q1 , Q2 ) via MMDk ( S1 , S2 ) , MMDk ( Q1 , Q2 ) and MMDk ( P , S1 ) , MMDk ( P , Q1 ) . We use a - b and a % b to mean a = O ( b ) and a = Ω ( b ) and use -k to denote that the underlying constants depend on k. Basic definitions We use H to denote an inner-product space endowed with the inner-product 〈· , ·〉H . We start with the definition of sub-gamma random variables ( Boucheron et al. , 2013 ) . Definition 1 ( f -Sub-Gamma ) A random variable X is said to be sub-gamma on the right with parameters ( σ2 , c ) denoted by Γ+ ( σ2 , c ) if for all 0 < λ < 1c , we have E [ exp ( λX ) ] ≤ exp ( λ2σ2 2 ( 1−cλ ) ) . For any f ∈ H , we say that a random variable G on H is f -sub-gamma on the right , denoted by Γf+ ( σ2 , c ) , if the random variable 〈f , G〉H ∈ Γ+ ( σ2 , c ) . As a reminder , we note that c = 0 yields a sub-Gaussian tails on the right . For X ∈ Γ+ ( σ2 , c ) , Boucheron et al . ( 2013 , Section 2.4 ) shows that for t ≥ 0 , and δ ∈ ( 0 , 1 ] , we have P [ X > √ 2σ2t+ ct ] ≤ e−t , or equivalently P [ X ≤ √ 2σ2 log ( 1δ ) + c log ( 1 δ ) ] ≥ 1− δ . ( 1 ) A property that we frequently use is that the sub-gamma property is suitably closed under multiplication and addition of sub-gamma random variables . A discussion is deferred to App . A . Definition 2 ( Thinning and halving algorithms ) Consider an algorithm ALG that takes as input a point sequence Sin of size n and returns a ( possibly random ) subsequence SALG ⊂ Sin of size nout . We say that ALG is αn-thinning if nout = n/αn whenever n/αn ∈ N. When αn = 2 , we say that ALG is a Halving algorithm , and when αn = √ n , we say ALG is root-thinning . For a thinning algorithm ALG , we associate a kernel discrepancy embedding that measures the approximation quality for the n input points Sin provided by the nout points SALG output by ALG : ψALG ( Sin ) , ∑ x∈Sin k ( x , · ) − n nout ∑ x∈SALG k ( x , · ) . ( 2 ) Letting PS denote the empirical distribution of S , the reproducing property of k yieldsthat PSinf − PSALGf = 1n 〈f , ψALG〉k , for any f ∈ H , and MMD ( Sin , SALG ) = 1 n‖ψALG‖k . ( 3 ) Next , we define the notion of a sub-gamma thinning algorithm via the object ψALG ( 2 ) . Definition 3 ( Sub-Gamma Thinning Algorithm ) Given the functions f ∈ H , σ : N→ R+ and c : N→ R+ , an αn-thinning algorithm ALG is called f -sub-gamma on the right with parameters ( σ2 , c ) , if ψALG ( Sin ) ∈ Γf+ ( σ2 ( n ) , c ( n ) ) conditioned on Sin of size n. When the algorithm is both Γf+ and Γ −f + -sub-gamma , ( 1 ) and ( 3 ) immmediately imply a high probability tail bound on the integration error |PSinf − PSALGf | . 2 COMPRESS . We now introduce our first meta procedure COMPRESS ( which is also a building block for COMPRESS++ ) . COMPRESS takes as input a halving algorithm HALVE and a oversampling factor g , and then given any input of size n , it outputs at thinned coreset of size 2g √ n. The algorithm is extremely simple to implement : first , divide the input sequence into four subsequences of size n4 ( in any manner the user chooses ) ; second , recursively call COMPRESS on each subsequence to produce four coresets of size 2g−1 √ n ; finally , call HALVE on the concatenation of those coresets to produce the final output of size 2g √ n. We denote this algorithm as COMPRESS ( HALVE , g ) , and describe it formally in Alg . 1 . COMPRESS can also be implemented in a streaming fashion but we defer the discussion to App . G. We remind that COMPRESS with g = 0 is a root thinning algorithm . Algorithm 1 : COMPRESS Input : Halving algorithm HALVE , oversampling factor g , point sequence Sin of size n if n = 4g then return Sin else Partition Sin into four subsets { Si } 4i=1 each of size n/4 for i = 1 , 2 , 3 , 4 do S̃i ← COMPRESS ( Si , HALVE , g ) // return coresets of size 2g · √ n 4 end S̃ ← CONCATENATE ( S̃1 , S̃2 , S̃3 , S̃4 ) // coreset of size 2 · 2g · √ n S ← HALVE ( k , S̃ ) // coreset of size 2g √ n return S end

Summary: This paper gives a meta algorithm for speeding up coreset constructing algorithms for the distribution compression problem.  The benefit of this meta algorithm is that its running time is faster by a square-root factor (e.g. quadratic to linear) while keeping the error rate roughly the same: only a factor of 4 worse.  The method is very simple: split the input into four pieces of size n/4 each, recursively solve each of them, then combine all four answers into a set and take a coreset of this set.  The speed-up in running time is immediate (just because of how recursive formulas work) and the error bounds are also rather clear.