Distribution Compression in Near-Linear Time

√ n points with Õ ( 1/ √ n ) distributional discrepancy to P. Unfortu- nately , these same algorithms suffer from quadratic or super-quadratic runtime in the sample size n. To address this deficiency , we introduce a simple metaprocedure—Compress++—for speeding up any input thinning algorithm while suffering at most a factor of four in error . When combined with the quadratictime kernel halving and kernel thinning algorithms of Dwivedi and Mackey ( 2021 ) , Compress++ delivers √ n points with O ( √ log n/n ) integration error and better-than-Monte-Carlo maximum mean discrepancy in O ( n log2 n ) time and O ( √ n log2 ( n ) ) space . Moreover , Compress++ enjoys the same near-linear runtime given any quadratic-time input and reduces the runtime of super-quadratic algorithms by a square-root factor . In our benchmarks with high-dimensional Monte Carlo samples and long-running Markov chains targeting challenging differential equation posteriors , Compress++ matches or nearly matches the accuracy of its input algorithm in orders of magnitude less time . 1 INTRODUCTION . Distribution compression—constructing a concise summary of a probability distribution—is at the heart of many learning and inference tasks . For example , in Monte Carlo integration and Bayesian inference , n representative points are sampled i.i.d . or from a Markov chain to approximate expectations and quantify uncertainty under an intractable ( posterior ) distribution ( Robert & Casella , 1999 ) . However , these standard sampling strategies represent a bottleneck in computationally-demanding settings due to their slow root-n Monte Carlo error rate . For instance , the Monte Carlo estimate Pnf , 1n ∑n i=1 f ( xi ) of an unknown expectation Pf , EX∼P [ f ( X ) ] based on n i.i.d . points has Θ ( n− 1 2 ) integration error |Pnf − Pf | , requiring n = 10000 points for 1 % relative error and n = 106 points for 0.1 % error . Such bloated sample representations preclude downstream applications with critically expensive function evaluations like computational cardiology , where a 1000-CPU-hour tissue or organ simulation is required for each sample point ( Niederer et al. , 2011 ; Augustin et al. , 2016 ; Strocchi et al. , 2020 ) , or expert data annotation which can be monetarily expensive Beaugnon et al . ( 2017 ) , and is sometimes coined as the human bottleneck of machine learning . To restore the feasibility of such critically expensive tasks , it is common to thin down the initial sequence of points to a produce a much smaller coreset . The standard thinning approach , select every t-th sample point ( Owen , 2017 ) , is simple to implement but often leads to an substantial increase in error : e.g. , standard thinning n points from a fast-mixing Markov chain yields Ω ( n− 1 4 ) error when n 1 2 points are returned . Recently , Dwivedi & Mackey ( 2021b ; a ) introduced a more effective alternative , kernel thinning , that provides near optimal Õ ( n− 12 ) error when compressing n points down to size n 1 2 . While practical for moderate sample sizes , the runtime of this algorithm scales quadratically with the input size n , making its execution prohibitive for very large input sizes . Our goal is to significantly improve the runtime of such compression algorithms while providing comparable error guarantees . Problem setup Given a sequence Sin of n input points summarizing a target distribution P , our aim is to identify a high quality coreset Sout of size √ n in time nearly linear in n. We measure coreset quality via its integration error |Pf −Poutf | , |Pf − 1|Sout| ∑ x∈Sout f ( x ) | for functions f in the reproducing kernel Hilbert space ( RKHS ) H induced by a given kernel k ( Berlinet & Thomas-Agnan , 2011 ) . We consider both single function error and kernel maximum mean discrepancy ( MMD , Gretton et al. , 2012 ) , the worst-case integration error over the unit RKHS norm ball : MMDk ( P , Pout ) , sup‖f‖k≤1|Pf − Poutf | = ‖Pk− Poutk‖k . Our contributions We introduce a new simple meta procedure—COMPRESS++—that significantly speeds up a generic thinning algorithm while simultaneously inheriting the error guarantees of its input up to a factor of four . A direct application of COMPRESS++ to kernel thinning improves its quadratic O ( n2 ) runtime to near linear O ( n log2 n ) time while maintaining the error guarantees up to a factor four . Since the Õ ( n− 12 ) KT MMD guarantees of Dwivedi & Mackey ( 2021b ) match the Ω ( n− 1 2 ) minimax lower bounds of Tolstikhin et al . ( 2017 ) ; Phillips & Tai ( 2020 ) up to factors of √ log ( n ) and constants depending on d , KT-COMPRESS++ also provides near-optimal MMD compression for a wide range of kernels and distributions P. Moreover , the practical gains from applying COMPRESS++ are substantial : KT thins 65 , 000 points in 10 dimensions in 20m , while KTCOMPRESS++ needs only 1.5m ; KT takes more than a day to thin 250 , 000 points in 100 dimensions , while KT-COMPRESS++ takes less than an hour ( a 32× speed-up ) . COMPRESS++ can also be directly combined with any thinning algorithm , even those that have suboptimal guarantees but often perform well in practice , like kernel herding ( Chen et al. , 2010 ) , support points ( Mak & Joseph , 2018 ) , Stein points ( MCMC ) ( Chen et al. , 2018 ; 2019 ) , and Stein thinning ( Riabiz et al. , 2020a ) , all of which run in Ω ( n2 ) time . As a demonstration , we combine COMPRESS++ with the popular kernel herding algorithm and observe 10-60× speed-ups . In all of our experiments , COMPRESS++ leads to minimal loss in accuracy and , surprisingly , even improves upon herding accuracy for lower-dimensional problems . Overview of COMPRESS++ To define COMPRESS++ , we first introduce COMPRESS : A simple and elegant recursive strategy takes in a halving algorithm and provides an intermediate thinned coreset in significantly faster runtime . COMPRESS divides the coreset into four parts , recursively applies COMPRESS to each , combines the output of each and halves the resulting coreset to get the output . Then COMPRESS++ with a thinning algorithm ALG is defined as follows : In stage one , use COMPRESS with ALG instantiated for 2-thinning , to obtain an intermediate coreset of size 2g √ n for a suitable parameter g , and then in stage two apply ALG directly to further thin it down to √ n points . In this manner , ALG is ever applied to coresets of size 2g √ n or smaller thereby improving the runtime . The parameter g is chosen based on the run-time and known guarantees of the underlying thinning algorithm , but a default choice of 4 can be shown to be optimal in several theoretical settings , and it provided competitive performance across all our experiments . Overall , with n input points and a thinning algorithm with runtime nα , COMPRESS++ uses a outputs √ n points in time Õ ( nα/2 ) with errors similar to those provided by the input thinning algorithm . Notation We write SALG for the coreset outputted by an algorithm ALG and extend our MMD definition for point sequences ( S1 , S2 ) with empirical distributions ( Q1 , Q2 ) via MMDk ( S1 , S2 ) , MMDk ( Q1 , Q2 ) and MMDk ( P , S1 ) , MMDk ( P , Q1 ) . We use a - b and a % b to mean a = O ( b ) and a = Ω ( b ) and use -k to denote that the underlying constants depend on k. Basic definitions We use H to denote an inner-product space endowed with the inner-product 〈· , ·〉H . We start with the definition of sub-gamma random variables ( Boucheron et al. , 2013 ) . Definition 1 ( f -Sub-Gamma ) A random variable X is said to be sub-gamma on the right with parameters ( σ2 , c ) denoted by Γ+ ( σ2 , c ) if for all 0 < λ < 1c , we have E [ exp ( λX ) ] ≤ exp ( λ2σ2 2 ( 1−cλ ) ) . For any f ∈ H , we say that a random variable G on H is f -sub-gamma on the right , denoted by Γf+ ( σ2 , c ) , if the random variable 〈f , G〉H ∈ Γ+ ( σ2 , c ) . As a reminder , we note that c = 0 yields a sub-Gaussian tails on the right . For X ∈ Γ+ ( σ2 , c ) , Boucheron et al . ( 2013 , Section 2.4 ) shows that for t ≥ 0 , and δ ∈ ( 0 , 1 ] , we have P [ X > √ 2σ2t+ ct ] ≤ e−t , or equivalently P [ X ≤ √ 2σ2 log ( 1δ ) + c log ( 1 δ ) ] ≥ 1− δ . ( 1 ) A property that we frequently use is that the sub-gamma property is suitably closed under multiplication and addition of sub-gamma random variables . A discussion is deferred to App . A . Definition 2 ( Thinning and halving algorithms ) Consider an algorithm ALG that takes as input a point sequence Sin of size n and returns a ( possibly random ) subsequence SALG ⊂ Sin of size nout . We say that ALG is αn-thinning if nout = n/αn whenever n/αn ∈ N. When αn = 2 , we say that ALG is a Halving algorithm , and when αn = √ n , we say ALG is root-thinning . For a thinning algorithm ALG , we associate a kernel discrepancy embedding that measures the approximation quality for the n input points Sin provided by the nout points SALG output by ALG : ψALG ( Sin ) , ∑ x∈Sin k ( x , · ) − n nout ∑ x∈SALG k ( x , · ) . ( 2 ) Letting PS denote the empirical distribution of S , the reproducing property of k yieldsthat PSinf − PSALGf = 1n 〈f , ψALG〉k , for any f ∈ H , and MMD ( Sin , SALG ) = 1 n‖ψALG‖k . ( 3 ) Next , we define the notion of a sub-gamma thinning algorithm via the object ψALG ( 2 ) . Definition 3 ( Sub-Gamma Thinning Algorithm ) Given the functions f ∈ H , σ : N→ R+ and c : N→ R+ , an αn-thinning algorithm ALG is called f -sub-gamma on the right with parameters ( σ2 , c ) , if ψALG ( Sin ) ∈ Γf+ ( σ2 ( n ) , c ( n ) ) conditioned on Sin of size n. When the algorithm is both Γf+ and Γ −f + -sub-gamma , ( 1 ) and ( 3 ) immmediately imply a high probability tail bound on the integration error |PSinf − PSALGf | . 2 COMPRESS . We now introduce our first meta procedure COMPRESS ( which is also a building block for COMPRESS++ ) . COMPRESS takes as input a halving algorithm HALVE and a oversampling factor g , and then given any input of size n , it outputs at thinned coreset of size 2g √ n. The algorithm is extremely simple to implement : first , divide the input sequence into four subsequences of size n4 ( in any manner the user chooses ) ; second , recursively call COMPRESS on each subsequence to produce four coresets of size 2g−1 √ n ; finally , call HALVE on the concatenation of those coresets to produce the final output of size 2g √ n. We denote this algorithm as COMPRESS ( HALVE , g ) , and describe it formally in Alg . 1 . COMPRESS can also be implemented in a streaming fashion but we defer the discussion to App . G. We remind that COMPRESS with g = 0 is a root thinning algorithm . Algorithm 1 : COMPRESS Input : Halving algorithm HALVE , oversampling factor g , point sequence Sin of size n if n = 4g then return Sin else Partition Sin into four subsets { Si } 4i=1 each of size n/4 for i = 1 , 2 , 3 , 4 do S̃i ← COMPRESS ( Si , HALVE , g ) // return coresets of size 2g · √ n 4 end S̃ ← CONCATENATE ( S̃1 , S̃2 , S̃3 , S̃4 ) // coreset of size 2 · 2g · √ n S ← HALVE ( k , S̃ ) // coreset of size 2g √ n return S end

Summary: The paper introduces meta algorithms "Compress" and "Compress++" that take existing thinning algorithm as a subroutine and improves on their runtime while incurring marginally more error.   The algorithm "Compress" runs in recursive fashion: divides input set into four parts, runs Compress on each part independently, combines the resulting sets and halves the combined set using "HALVE" algorithm. As an example, the authors demonstrate that one can use KT-SPLIT as HALVE algorithm and get faster runtimes to compress albeit suffering extra error (upto log factors). Compress ensures that KT-SPLIT runs on sets of much smaller size.   They build on Compress to obtain better error rates while sacrificing on runtimes (upto log factors). For this, they use the thinning algorithm on much smaller set obtained after running Compress. The runtimes are quite easy to derive based on recursion of algorithms. They use sub-gamma property to prove the error guarantees of the algorithms. They demonstrate the faster runtimes of their algorithms using high dimensional Monte Carlo samples.