An Efficient Protocol for Distributed Column Subset Selection in the Entrywise $\ell_p$ Norm

1 p− 1 2 poly ( log nd ) -approximation to the best possible column subset . A key ingredient in our proof is the reduction to the ` p,2-norm , which corresponds to the p-norm of the vector of Euclidean norms of each of the columns of A . This enables us to use strong coreset constructions for Euclidean norms , which previously had not been used in this context . This naturally also allows us to implement our algorithm in the popular streaming model of computation . We further propose a greedy algorithm for selecting columns , which can be used by the coordinator , and show the first provable guarantees for a greedy algorithm for the ` 1,2 norm . Finally , we implement our protocol and give significant practical advantages on real-world data analysis tasks . 1 INTRODUCTION . Column Subset Selection ( k-CSS ) is a widely studied approach for rank-k approximation and feature selection . In k-CSS , one seeks a small subset U ∈ Rd×k of k columns of a data matrix A ∈ Rd×n , typically n d , for which there is a right factor V such that |UV −A| is small under some norm | · | . k-CSS is a special case of low rank approximation for which the left factor is an actual subset of columns . The main advantage of k-CSS over general low rank approximation is that the resulting factorization is more interpretable , as columns correspond to actual features while general low rank approximation takes linear combinations of such features . In addition , k-CSS preserves the sparsity of the data matrix A. k-CSS has been extensively studied in the Frobenius norm ( Guruswami & Sinop , 2012 ; Boutsidis et al. , 2014 ; Boutsidis & Woodruff , 2017 ; Boutsidis et al. , 2008 ) and operator norms ( Halko et al. , 2011 ; Woodruff , 2014 ) . A number of recent works ( Song et al. , 2017 ; Chierichetti et al. , 2017 ; Dan et al. , 2019 ; Ban et al. , 2019 ; Mahankali & Woodruff , 2020 ) studied this problem in the ` p norm ( k-CSSp ) for 1 ≤ p < 2 . The ` 1 norm is less sensitive to outliers , and better at handling missing data and non-Gaussian noise , than the Frobenius norm ( Song et al. , 2017 ) . Specifically , the ` 1 norm leads to improved performance in many real-world applications , such as structure-from-motion ( Ke & Kanade , 2005 ) and image denoising ( Yu et al. , 2012 ) . Distributed low-rank approximation arises naturally when a dataset is too large to store on one machine , takes prohibitively long time for a single machine to compute a rank-k approximation , or is collected simultaneously on multiple machines . Despite the flurry of recent work on k-CSSp , this problem remains largely unexplored in the distributed setting . This should be contrasted to Frobenius norm column subset selection and low rank approximation , for which a number of results in the distributed model are known , see , e.g. , Altschuler et al . ( 2016 ) ; Balcan et al . ( 2015 ; 2016 ) ; Boutsidis et al . ( 2016 ) . We consider a widely applicable model in the distributed setting , where s servers communicate to a central coordinator via 2-way channels . This model can simulate arbitrary point-to-point communication by having the coordinator forward a message from one server to another ; this increases the total communication by a factor of 2 and an additive log s bits per message to identify the destination server . We consider the column partition model , in which each column of A ∈ Rd×n is held by exactly one server . The column partition model is widely-studied and arises naturally in many real world scenarios such as federated learning ( Farahat et al. , 2013 ; Altschuler et al. , 2016 ; Liang et al. , 2014 ) . In the column partition model , we typically have n d , i.e. , A has many more columns than rows . Hence , we desire a protocol for distributed k-CSSp that has a communication cost that is only logarithmic in the large dimension n , as well as fast running time . In addition , it is important that our protocol only uses a small constant number of communication rounds ( meaning back-and-forth exchanges between servers and the coordinator ) . Indeed , otherwise , the servers and coordinator would need to interact more , making the protocol sensitive to failures in the machines , e.g. , if they go offline . Further , a 1-round protocol can naturally be adapted to an single pass streaming algorithm when we consider applications with limited memory and access to the data . In fact , our protocol can be easily extended to yield such a streaming algorithm 1 . In the following , we denote Ai∗ and A∗j as the i-th row and j-th column of A respectively , for i ∈ [ d ] , j ∈ [ n ] . We denote AT as the subset of columns of A with indices in T ⊆ [ n ] . The entrywise ` p-norm of A is |A|p = ( ∑d i=1 ∑n j=1 |Aij |p ) 1 p . The ` p,2 norm is defined as |A|p,2 = ( ∑d j=1 |A∗j | p 2 ) 1 p . We consider 1 ≤ p < 2 . We denote the best rank-k approximation error for A in ` p norm by OPT : = minrank-kAk |A−Ak|p . Given an integer k > 0 , we say U ∈ Rd×k , V ∈ Rk×n are the left and right factors of a rank-k factorization for A in the ` p norm with approximation factor α if |UV −A|p ≤ α ·OPT . Since general rank-k approximation in ` 1 norm is NP hard ( Gillis & Vavasis , 2015 ) , we follow previous work and consider bi-criteria k-CSS algorithms which obtain polynomial running time . Instead of outputting exactly k columns , such algorithms return a subset of Õ ( k ) columns of A , suppressing logarithmic factors in k or n. It is known that the best approximation factor to OPT that can be obtained through the span of a column subset of size Õ ( k ) is Ω ( k1/2−γ ) for p = 1 ( Song et al. , 2017 ) and Ω ( k1/p−1/2−γ ) for p ∈ ( 1 , 2 ) ( Mahankali & Woodruff , 2020 ) , where γ is an arbitrarily small constant . 1We provide a detailed analysis of our streaming algorithm in Appendix E. 1.1 PREVIOUS APPROACHES TO k-CSSp IN THE DISTRIBUTED SETTING If one only wants to obtain a good left factor U , and not necessarily a column subset of A , in the column partition model , one could simply sketch the columns of Ai by applying an oblivious sketching matrix S on each server . Each server sends Ai · S to the coordinator . The coordinator obtains U = AS as a column-wise concatenation of AiS ’ s . Song et al . ( 2017 ) showed that AS achieves an Õ ( √ k ) approximation to OPT , and this protocol only requires Õ ( sdk ) communication , O ( 1 ) rounds and polynomial running time . However , while AS is a good left factor , it does not correspond to an actual subset of columns of A . Obtaining a subset of columns that approximates A well with respect to the p-norm in a distributed setting is non-trivial . One approach due to Song et al . ( 2017 ) is to take the matrix AS described above , sample rows according to the Lewis weights ( Cohen & Peng , 2015 ) of AS to get a right factor V , which is in the row span of A , and then use the Lewis weights of V to in turn sample columns of A . Unfortunately , this protocol only achieves a loose Õ ( k3/2 ) approximation to OPT ( Song et al. , 2017 ) . Moreover , it is not known how to do Lewis weight sampling in a distributed setting . Alternatively , one could adapt existing single-machine k-CSSp algorithms to the distributed setting under the column partition model . Existing works on polynomial time k-CSSp ( Chierichetti et al. , 2017 ; Song et al. , 2019b ; Dan et al. , 2019 ; Mahankali & Woodruff , 2020 ) give bi-criteria algorithms , and are based on a recursive framework with multiple rounds , which is as follows : in each round , Õ ( k ) columns are selected uniformly at random , and with high probability , the selected columns can provide a good approximation to a constant fraction of all columns of A . Among the remaining columns that are not well approximated , Õ ( k ) columns are recursively selected until all columns of A are well approximated , resulting in a total of O ( log n ) rounds . A naı̈ve extension of this bi-criteria k-CSSp framework to a distributed protocol requires O ( log n ) rounds , as in each round , the servers and the coordinator need to communicate with each other in order to find the columns that are covered well and select from the remaining unselected columns . To reduce this to a single round , one might consider running the O ( log n ) round selection procedure on the coordinator only . In order to do this , the coordinator needs to first collect all columns of A from the servers , but directly communicating all columns is prohibitive . Alternatively , one could first apply k-CSSp on Ai to obtain factors Ui and Vi on each server , and then send the coordinator all of the Ui and Vi . The coordinator then column-wise stacks the UiVi to obtain U · V and selects Õ ( k ) columns from U · V . Even though this protocol applies to all p ≥ 1 , it achieves a loose O ( k2 ) approximation to OPT and requires a prohibitive O ( n+ d ) communication cost2 . One could instead try to just communicate the matrices Ui to the coordinator , which results in much less communication , but this no longer gives a good approximation . Indeed , while each Ui serves as a good approximation locally , there may be columns that are locally not important , but become globally important when all of the matrices Ai are put together . What is really needed here is a small coreset Ci for each Ai so that if one concatenates all of the Ci to obtain C , any good column subset of the coreset C corresponds to a good column subset for A . Unfortunately , coresets for the entrywise ` p-norm are not known to exist . 1.2 OUR CONTRIBUTIONS . Our Distributed Protocol We overcome these problems and propose the first efficient protocol for distributed k-CSSp ( 1 ≤ p < 2 ) in the column partition model that selects Õ ( k ) columns of A achieving an Õ ( k1/p−1/2 ) -approximation to the best possible subset of columns and requires only Õ ( sdk ) communication cost , 1 round and polynomial time . Figure 1 gives an overview of the protocol . We note that our subset of columns does not necessarily achieve an Õ ( k1/p−1/2 ) -approximation to OPT itself , although it does achieve such an approximation to the best possible subset of columns . Using the fact that there always exists a subset of columns providing an Õ ( k1/p−1/2 ) -approximation to OPT ( Song et al. , 2017 ) , we conclude that our subset of columns achieves an Õ ( k2/p−1 ) -approximation to OPT . Recently , and independently of our work , Mahankali & Woodruff ( 2020 ) show how to obtain a subset of columns achieving an Õ ( k1/p−1/2 ) -approximation to OPT itself ; however , such a subset is found by uniformly sampling columns in O ( log n ) adaptive rounds using the recursive sampling 2We give this protocol and the analysis in Appendix A framework above , and is inherently hard to implement in a distributed setting with fewer rounds . In contrast , our protocol achieves 1 round of communication , which is optimal . We make use of a strong coreset , i.e. , a sampled and reweighted subset of columns of each Ai that approximates the cost of all potential left factors of Ai , by first embedding all subspaces spanned by any subset of Õ ( k ) columns of A from ` p-space to Euclidean space , to bypass the lack of strong coresets for the ` p norm . We denote this new norm as ` p,2 norm , which is the sum of the p-th powers of the ` 2 norms of the columns . To reduce the error incurred by switching to the ` p,2-norm , we reduce the row dimension of A by left-multiplying by an oblivious sketching matrix S shared across servers , resulting in an overall approximation factor of only Õ ( k1/p−1/2 ) . Afterwards , each server sends its strong coreset to the coordinator . The coordinator , upon receiving the coresets from each server , runs an O ( 1 ) -approximate bi-criteria k-CSSp,2 algorithm to select the final column subset , giving an overall Õ ( k1/p−1/2 ) approximation to the best column subset . We introduce several new technical ideas in the analysis of our protocol . Our work is the first to apply a combination of oblivious sketching in the p-norm via p stable random variables and strong coresets in the ` p,2 norm ( Sohler & Woodruff , 2018 ; Huang & Vishnoi , 2020 ) to distributed k-CSS . Furthermore , to show that our oblivious sketching step only increases the final approximation error by a logarithmic factor , we combine a net argument with a union bound over all possible subspaces spanned by column subsets of A of size Õ ( k ) . Previous arguments involving sketching , such as those by Song et al . ( 2017 ) ; Ban et al . ( 2019 ) ; Mahankali & Woodruff ( 2020 ) , only consider a single subspace at a time . Theoretical Guarantees and Empirical Benefits for Greedy k-CSS1,2 We also propose a greedy algorithm to select columns in the k-CSS1,2 step of our protocol , and show the first additive error guarantee compared to the best possible subset AS of columns , i.e. , our cost is at most ( 1 − ) minV |ASV − A|1,2 + |A|1,2 . Similar error guarantees were known for the Frobenius norm ( Altschuler et al. , 2016 ) , though nothing was known for the ` 1,2 norm . We also implement our protocol and experiment with distributed k-CSS1 on various real-world datasets . We compare the O ( 1 ) -approximate bi-criteria k-CSS1,2 and the greedy k-CSS1,2 as different possible subroutines in our protocol , and show that greedy k-CSS1,2 yields an improvement in practice .

Summary: This paper focused $\ell_p$-norm error analysis of distributed column subset selection problems. The proposed distributed algorithm guarantee an $\tilde O(k^{1/p-1/2} )$-approximation to the best subset of columns with $\tilde O(sdk)$ communication cost per round and polynomial time.  The lower bound analysis show that such communication complexity is near optimal. The design and analysis of the algorithm is based on f oblivious sketching and strong coresets, which leads to the overall approximation factor is only $\tilde O(k^{1/p-1/2} )$. The authors also extended the main idea to address $\ell_{1,2}$ norm error.