Support Recovery of Sparse Signals from a Mixture of Linear Measurements

1 Introduction . In the support recovery problem , widely studied in the literature of compressed sensing [ 6 , 2 , 35 ] , the objective is to recover the support ( positions of nonzero coordinates ) of a sparse vector from minimal number of ( noisy ) linear measurements . The support recovery problem is also extensively studied under the 1-bit compressed sensing model where measurements are further quantized and just the signs of the linear measurements are provided [ 22 , 24 , 1 ] . In a recent line of work that started with [ 46 ] , a generalization of the sparse recovery problem is considered [ 29 , 31 , 21 , 11 ] , where instead of one sparse vector , multiple unknown sparse vectors are to be recovered . However any attempt to obtain a measurement ( linear or 1-bit ) from the vectors results in a mixture model , where a vector from the unknown set is picked uniformly to generate the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . response . Due to the asynchronicity of the measurements , this set of problems pose fundamentally different challenges than recovery of a single sparse vector . This line of work also connects the mixture of simple learning models that have been studied extensively in the past few decades , with mixtures of linear regression model being more widely studied [ 13 , 12 , 23 , 27 , 39 , 41 , 44 , 45 , 48 ] than mixture of linear classifiers [ 43 , 38 ] . Such mixture models , that assume the training data to come from multiple models , are good approximators of a function [ 4 , 25 ] and have numerous applications in modeling heterogeneous settings such as machine translation [ 30 ] , behavioral health [ 15 ] , medicine [ 5 ] , object recognition [ 33 ] etc . The mixture of sparse recovery models of [ 46 ] and followup works can be framed as a Mixture of Linear Classifiers ( MLC ) or a Mixture of Linear Regressions ( MLR ) problems . The statistical model in MLC is the following : there exists ` unknown hyperplanes with normal vectors v1 , v2 , . . . , v ` and for a particular feature vector , the label ( response ) is generated stochastically by selecting one of the unknown hyperplanes at random and then returning the side of the chosen hyperplane on which the feature vector lies . In MLR , the statistical model again assumes the presence of ` unknown hyperplanes with normals v1 , v2 , . . . , v ` and for a particular feature vector , the response is stochastically generated by selecting one of the unknown hyperplanes at random and then returning the projection of the feature vector to the chosen hyperplane . In order to make these models more general , we can assume that the responses are corrupted by noise . The overarching goal for both the MLR and MLC is to learn the ` unknown hyperplanes as accurately as possible , using the least number of noisy responses . Sparsity , incorporated into the MLC and MLR problems , is also a common assumption that represents redundant features and lower dimensionality of the models [ 47 , 36 , 10 ] . In this work , we tackle the problem of support recovery of sparse vectors for both MLR and MLC model in the active query based setting of [ 46 , 29 , 31 , 21 ] . Our goal is to recover the support of all the unknown sparse vectors ( hyperplane normals ) with minimum number of measurements . 1.1 Formal Problem Statement and Relevant Works . In both the problems below , let V be a set of ` unknown vectors v1 , v2 , . . . , v ` 2 Rn such that vi 0 k for all i 2 [ ` ] ⌘ { 1 , 2 , . . . , ` } . Mixtures of Sparse Linear Classifiers ( MLC ) . Let sign : R ! { 1 , +1 } be the sign function that takes a real number and returns its sign . We consider MLC label queries O : Rn ! { 1 , +1 } that takes as input a query vector x 2 Rn and returns sign ( hx , vi ) · ( 1 2Z ) where v is sampled uniformly at random from V and Z ⇠ Ber ( ⌘ ) , the noise , is a Bernoulli random variable that is 1 with probability ⌘ and 0 with probability 1 ⌘ . In this problem , our objective is to recover the support of all the unknown vectors in V using minimum number of label queries . The only relevant work in this setting is [ 21 ] which provided results for both support recovery and approximate recovery of the unknown vectors . However , the results of [ 21 ] are valid only under the restrictive assumption that the support of any unknown vector is not contained within the union of the supports of the other unknown vectors . In this work , we generalize the techniques of [ 21 ] for support recovery of the unknown vectors and get rid of the restrictive assumption . We further improve the generalized result in a wide regime by demonstrating a new low-rank tensor decomposition based algorithm for support recovery . Mixtures of Sparse Linear Regressions ( MLR ) . In this setting , we have an MLR label map O : Rn ! R that takes as input a query x 2 Rn and returns as output the quantity hx , vi+ Z where v is sampled uniformly at random from V and Z ⇠ N ( 0 , 2 ) is a zero-mean Gaussian random variable with variance 2 . For our MLR results to hold we further assume that the minimum magnitude of any non-zero entry of any unknown vector in V is known to be at least , i.e. , mini2 [ ` ] minj2 [ n ] : vij 6=0 |v i j | . Note that , because of the additive noise , a result for MLC setting can not be transformed into a result in MLR setting ( i.e. , MLC response is not simple quantization of MLR ) . It is possible to increase the ` 2 norm of the queries arbitrarily so that the noise becomes inconsequential . To avoid this we use the following definition of signal to noise ratio . Suppose the algorithm designs the ith query vector by first choosing a distribution Qi and subsequently sampling a query vector xi ⇠ Qi . The signal to noise ratio is defined as follows : SNR = max i min j2 [ ` ] Exi⇠Qi |hxi , vji|2 EZ2 . ( 1 ) Our objective in this setting is to recover the support of all unknown vectors v1 , v2 , . . . , v ` 2 Rn while minimizing the number of queries for a fixed SNR . The most relevant works in this setting would be [ 46 ] , [ 29 ] and [ 31 ] , all of which were concerned with approximately recovering the k-sparse unknown vectors v1 , v2 , . . . , v ` i.e . computing estimates v̂1 , v̂2 , . . . , v̂ ` such that for some precision parameter > 0 , kvi v̂ ( i ) k2 O ( ) for all i 2 [ ` ] for some permutation : [ ` ] ! [ ` ] . While approximate recovery of vectors can also be translated into support recovery , the results of [ 46 ] and [ 29 ] are valid only under the restrictive assumption that the sparse vectors all belong to some scaled integer lattice . The results of [ 31 ] does not have any restriction , but it holds only when ` = 2 . However , note that in this special case i.e . when ` = 2 , [ 31 ] provides a query complexity guarantee that is linear in the sparsity k. On the other hand , our query complexity guarantees ( see Section 2.3 ) have a polynomial dependence on k ( with a larger degree ) implying that in the regime when ` = 2 and k is large , the guarantees of [ 31 ] are better . Here we provide results for support recovery of any number of unknown vectors that do not have any of the aforementioned restrictions and also have a polynomial dependence on the noise variance , sparsity and a near polynomial dependence on the number of unknown vectors . 1.2 Other Related Work . Learning the unknown vectors in the MLR setting is a generalization of the compressed sensing problem [ 8 , 16 ] where the objective is to learn a single unknown k-sparse vector ( ` = 1 ) with minimum number of noisy linear measurements . Support recovery is a well-studied area within this literature [ 6 , 2 , 35 ] . Similarly , learning the unknown vectors in the MLC setting is a generalization of the 1-bit compressed sensing problem where the objective is to learn a single unknown k-sparse vector ( ` = 1 ) with minimum number of linear measurements quantized to only 1-bit . Support recovery of the sparse vector from 1-bit measurements has also been widely studied [ 1 , 22 , 19 , 24 ] . The major building block of one of our two algorithms is low-rank tensor decomposition also known as Canonical Polyadic ( CP ) decomposition . Tensor decomposition has been widely used in parameter estimation in mixture models and latent variable models . We refer the reader to [ 34 ] and the references therein for a detailed survey . Our other algorithm makes use of combinatorial structures such as a general class of Union Free Families ( UFF ) , see , [ 42 ] , to recover the support . UFFs have been previously used in [ 1 ] and [ 21 ] for support recovery in linear classifiers . Organization . The rest of the paper is organized as follows . In Section 2 , we gave the necessary backgrounds , and described our techniques and main results , namely , Theorems 1 , 3 , 4 , and Corollary 1 . In Section 3 , we provided the detailed proofs of Theorem 1 ( Section 3.1 ) , Theorem 3 ( Section 3.2 ) and Theorem 4 ( Section 3.3 ) while deferring the proof of Theorem 2 to Appendix D. In Appendix A , we provided the necessary background on families of sets and finally , in Appendix B , we gave the details of a Lemma that is an integral component of the proofs of our main Theorems . 2 Our Techniques and Results . 2.1 Preliminaries . Notations : Let round : R ! Z denote a function that returns the closest integer to a given real input . Let 1n denote a length n vector of all 1 ’ s . We will write [ n ] to denote the set { 1 , . . . , n } and let P ( [ n ] ) be the power set of [ n ] . For a vector v 2 Rn , let vi denote its i-th coordinate for any i 2 [ n ] . We will use supp ( v ) ✓ [ n ] to denote the support of the vector v , i.e , the set of indices with non-zero entries in v. We will abuse notations a bit , and also sometimes use supp ( v ) to denote the binary indicator vector of length n that takes 1 at index i if and only if vi 6= 0 . For a vector v 2 Rn and subset S ✓ [ n ] of indices , let v|S 2 R|S| denote the vector v restricted to the indices in S. Finally , let f : P ( [ n ] ) ⇥ { 0 , 1 } n ! { 0 , 1 } n be a function that takes a binary vector v 2 { 0 , 1 } n and a subset S ✓ [ n ] as input and returns another binary vector v0 such that the indices of v corresponding to the the set S are flipped i.e . v0i = vi 1 if i 2 S and v0i = vi otherwise . Tensor Decomposition : Consider a tensor A of order w 2 N , w > 2 on Rn which is denoted by A 2 Rn ⌦ Rn ⌦ · · · ⌦ Rn ( w times ) . Let us denote by Ai1 , i2 , ... , iw where i1 , i2 , . . . , iw 2 [ n ] , the element in A whose location along the jth dimension is ij i.e . there are ij 1 elements along the jth dimension before Ai1 , i2 , ... , iw . Notice that this indexing protocol uniquely determines the element within the tensor . For a detailed review of tensors , we defer the reader to [ 28 ] . In this work , we are interested in low rank decomposition of tensors . A tensor A can be described as a rank-1 symmetric tensor if it can be expressed as A = z⌦ z⌦ · · ·⌦ z| { z } w times for some z 2 Rn i.e . Ai1 , i2 , ... , iw = Qw j=1 zij . A tensor A that can be expressed as a sum of R rank-1 symmetric tensors is defined as a rank R symmetric tensor . For such a rank R tensor A provided as input , we are concerned with the problem of unique decomposition of A into a sum of R rank-1 symmetric tensors ; such a decomposition is also known as a Canonical Polyadic ( CP ) decomposition . Below , we show a result due to [ 40 ] describing the sufficient conditions ( Kruskal ’ s result ) for the unique CP decomposition of a rank R tensor A : Lemma 1 ( Unique CP decomposition [ 40 ] ) . Suppose A is the sum of R rank-one tensors i.e . A = RX r=1 zr ⌦ zr ⌦ · · ·⌦ zr| { z } w times . and further , the Kruskal Rank of the n⇥R matrix whose columns are formed by z1 , z2 , . . . , zR is J . Then , if wJ 2R+ ( w 1 ) , then the CP decomposition is unique and we can recover the vectors z1 , z2 , . . . , zR up to permutations . There exist many different techniques for CP decomposition of a tensor but the most well-studied ones are Jennrich ’ s Algorithm ( see Section 3.3 , [ 32 ] ) and the Alternating Least Squares ( ALS ) algorithm [ 28 ] . Among these , Jennrich ’ s algorithm ( see Section E for more details ) is efficient and recovers the latent rank-1 tensors uniquely but it works only for tensors of order 3 when the underlying vectors z1 , z2 , . . . , zR are linearly independent ( See Theorem 3.3.2 , [ 32 ] ) ; this is a stronger condition than what we obtain from Lemma 1 for w = 3 . On the other hand , the ALS algorithm is an iterative algorithm which is easy to implement for tensors of any order but unfortunately , it takes many iterations to converge and furthermore , it is not guaranteed to converge to the correct solution . Jennrich ’ s algorithm also has the additional advantage that it will throw an error if its sufficient condition for unique CP decomposition is not satisfied . This property will turn out to be useful for the problem that we study in this work . Finally , notice that if A is the weighted sum of R rank-1 tensors i.e. , A = RX r=1 r z r ⌦ zr ⌦ · · ·⌦ zr| { z } w times . then we can rewrite A = PR r=1 y r ⌦ yr ⌦ · · ·⌦ yr where yr = 1/wr zr . If { yr } Rr=1 satisfies the conditions of Lemma 1 and if it is known that { zr } Rr=1 are binary vectors , then we can still recover zr by first computing yr and then taking its support for all r 2 [ R ] . Subsequently , notice that we can also recover { r } Rr=1 . As we discussed , for tensors of order w > 3 , there is no known efficient algorithm that can recover the correct solution even if its existence and uniqueness is known . Due to this limitation , it was necessary in prior works using low rank decomposition of tensors that the unknown parameter vectors are linearly independent [ 9 , 3 ] since tensors of order > 3 could not be used . However , if it is known apriori that the vectors { zr } Rr=1 are binary and the coefficients { r } Rr=1 are positive integers bounded from above by some C > 0 , then we can exhaustively search over all possibilities ( O ( C2n ) of them ) to find the unique decomposition even in higher order tensors . The set of possible solutions can be reduced significantly if the unknown vectors are known to be sparse as is true in our setting .

Summary: This work studies support recovery of a collection of multiple, unknown sparse vectors using a small number of noisy linear or 1-bit observations where the observations are constructed from random mixtures of the unknown sparse vectors. In other words, for each measurement, one of the multiple, unknown sparse vectors is chosen uniformly at random to generate the response and we aim to learn the support of all of these unknown vectors from the mixed measurements using the least amount of measurements as possible. In particular, the work considers sample complexity of recovery in both models and provides efficient algorithms to perform the recovery. Previous work in the area considered either approximate recovery under restrictive assumptions or support recovery only for the case where there are \ell = 2 unknown sparse vectors (of course, the \ell = 1 case has been studied extensively). This work considers support recovery for \ell >= 2 unknown sparse vectors.