Support Recovery of Sparse Signals from a Mixture of Linear Measurements

1 Introduction . In the support recovery problem , widely studied in the literature of compressed sensing [ 6 , 2 , 35 ] , the objective is to recover the support ( positions of nonzero coordinates ) of a sparse vector from minimal number of ( noisy ) linear measurements . The support recovery problem is also extensively studied under the 1-bit compressed sensing model where measurements are further quantized and just the signs of the linear measurements are provided [ 22 , 24 , 1 ] . In a recent line of work that started with [ 46 ] , a generalization of the sparse recovery problem is considered [ 29 , 31 , 21 , 11 ] , where instead of one sparse vector , multiple unknown sparse vectors are to be recovered . However any attempt to obtain a measurement ( linear or 1-bit ) from the vectors results in a mixture model , where a vector from the unknown set is picked uniformly to generate the 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . response . Due to the asynchronicity of the measurements , this set of problems pose fundamentally different challenges than recovery of a single sparse vector . This line of work also connects the mixture of simple learning models that have been studied extensively in the past few decades , with mixtures of linear regression model being more widely studied [ 13 , 12 , 23 , 27 , 39 , 41 , 44 , 45 , 48 ] than mixture of linear classifiers [ 43 , 38 ] . Such mixture models , that assume the training data to come from multiple models , are good approximators of a function [ 4 , 25 ] and have numerous applications in modeling heterogeneous settings such as machine translation [ 30 ] , behavioral health [ 15 ] , medicine [ 5 ] , object recognition [ 33 ] etc . The mixture of sparse recovery models of [ 46 ] and followup works can be framed as a Mixture of Linear Classifiers ( MLC ) or a Mixture of Linear Regressions ( MLR ) problems . The statistical model in MLC is the following : there exists ` unknown hyperplanes with normal vectors v1 , v2 , . . . , v ` and for a particular feature vector , the label ( response ) is generated stochastically by selecting one of the unknown hyperplanes at random and then returning the side of the chosen hyperplane on which the feature vector lies . In MLR , the statistical model again assumes the presence of ` unknown hyperplanes with normals v1 , v2 , . . . , v ` and for a particular feature vector , the response is stochastically generated by selecting one of the unknown hyperplanes at random and then returning the projection of the feature vector to the chosen hyperplane . In order to make these models more general , we can assume that the responses are corrupted by noise . The overarching goal for both the MLR and MLC is to learn the ` unknown hyperplanes as accurately as possible , using the least number of noisy responses . Sparsity , incorporated into the MLC and MLR problems , is also a common assumption that represents redundant features and lower dimensionality of the models [ 47 , 36 , 10 ] . In this work , we tackle the problem of support recovery of sparse vectors for both MLR and MLC model in the active query based setting of [ 46 , 29 , 31 , 21 ] . Our goal is to recover the support of all the unknown sparse vectors ( hyperplane normals ) with minimum number of measurements . 1.1 Formal Problem Statement and Relevant Works . In both the problems below , let V be a set of ` unknown vectors v1 , v2 , . . . , v ` 2 Rn such that vi 0 k for all i 2 [ ` ] ⌘ { 1 , 2 , . . . , ` } . Mixtures of Sparse Linear Classifiers ( MLC ) . Let sign : R ! { 1 , +1 } be the sign function that takes a real number and returns its sign . We consider MLC label queries O : Rn ! { 1 , +1 } that takes as input a query vector x 2 Rn and returns sign ( hx , vi ) · ( 1 2Z ) where v is sampled uniformly at random from V and Z ⇠ Ber ( ⌘ ) , the noise , is a Bernoulli random variable that is 1 with probability ⌘ and 0 with probability 1 ⌘ . In this problem , our objective is to recover the support of all the unknown vectors in V using minimum number of label queries . The only relevant work in this setting is [ 21 ] which provided results for both support recovery and approximate recovery of the unknown vectors . However , the results of [ 21 ] are valid only under the restrictive assumption that the support of any unknown vector is not contained within the union of the supports of the other unknown vectors . In this work , we generalize the techniques of [ 21 ] for support recovery of the unknown vectors and get rid of the restrictive assumption . We further improve the generalized result in a wide regime by demonstrating a new low-rank tensor decomposition based algorithm for support recovery . Mixtures of Sparse Linear Regressions ( MLR ) . In this setting , we have an MLR label map O : Rn ! R that takes as input a query x 2 Rn and returns as output the quantity hx , vi+ Z where v is sampled uniformly at random from V and Z ⇠ N ( 0 , 2 ) is a zero-mean Gaussian random variable with variance 2 . For our MLR results to hold we further assume that the minimum magnitude of any non-zero entry of any unknown vector in V is known to be at least , i.e. , mini2 [ ` ] minj2 [ n ] : vij 6=0 |v i j | . Note that , because of the additive noise , a result for MLC setting can not be transformed into a result in MLR setting ( i.e. , MLC response is not simple quantization of MLR ) . It is possible to increase the ` 2 norm of the queries arbitrarily so that the noise becomes inconsequential . To avoid this we use the following definition of signal to noise ratio . Suppose the algorithm designs the ith query vector by first choosing a distribution Qi and subsequently sampling a query vector xi ⇠ Qi . The signal to noise ratio is defined as follows : SNR = max i min j2 [ ` ] Exi⇠Qi |hxi , vji|2 EZ2 . ( 1 ) Our objective in this setting is to recover the support of all unknown vectors v1 , v2 , . . . , v ` 2 Rn while minimizing the number of queries for a fixed SNR . The most relevant works in this setting would be [ 46 ] , [ 29 ] and [ 31 ] , all of which were concerned with approximately recovering the k-sparse unknown vectors v1 , v2 , . . . , v ` i.e . computing estimates v̂1 , v̂2 , . . . , v̂ ` such that for some precision parameter > 0 , kvi v̂ ( i ) k2 O ( ) for all i 2 [ ` ] for some permutation : [ ` ] ! [ ` ] . While approximate recovery of vectors can also be translated into support recovery , the results of [ 46 ] and [ 29 ] are valid only under the restrictive assumption that the sparse vectors all belong to some scaled integer lattice . The results of [ 31 ] does not have any restriction , but it holds only when ` = 2 . However , note that in this special case i.e . when ` = 2 , [ 31 ] provides a query complexity guarantee that is linear in the sparsity k. On the other hand , our query complexity guarantees ( see Section 2.3 ) have a polynomial dependence on k ( with a larger degree ) implying that in the regime when ` = 2 and k is large , the guarantees of [ 31 ] are better . Here we provide results for support recovery of any number of unknown vectors that do not have any of the aforementioned restrictions and also have a polynomial dependence on the noise variance , sparsity and a near polynomial dependence on the number of unknown vectors . 1.2 Other Related Work . Learning the unknown vectors in the MLR setting is a generalization of the compressed sensing problem [ 8 , 16 ] where the objective is to learn a single unknown k-sparse vector ( ` = 1 ) with minimum number of noisy linear measurements . Support recovery is a well-studied area within this literature [ 6 , 2 , 35 ] . Similarly , learning the unknown vectors in the MLC setting is a generalization of the 1-bit compressed sensing problem where the objective is to learn a single unknown k-sparse vector ( ` = 1 ) with minimum number of linear measurements quantized to only 1-bit . Support recovery of the sparse vector from 1-bit measurements has also been widely studied [ 1 , 22 , 19 , 24 ] . The major building block of one of our two algorithms is low-rank tensor decomposition also known as Canonical Polyadic ( CP ) decomposition . Tensor decomposition has been widely used in parameter estimation in mixture models and latent variable models . We refer the reader to [ 34 ] and the references therein for a detailed survey . Our other algorithm makes use of combinatorial structures such as a general class of Union Free Families ( UFF ) , see , [ 42 ] , to recover the support . UFFs have been previously used in [ 1 ] and [ 21 ] for support recovery in linear classifiers . Organization . The rest of the paper is organized as follows . In Section 2 , we gave the necessary backgrounds , and described our techniques and main results , namely , Theorems 1 , 3 , 4 , and Corollary 1 . In Section 3 , we provided the detailed proofs of Theorem 1 ( Section 3.1 ) , Theorem 3 ( Section 3.2 ) and Theorem 4 ( Section 3.3 ) while deferring the proof of Theorem 2 to Appendix D. In Appendix A , we provided the necessary background on families of sets and finally , in Appendix B , we gave the details of a Lemma that is an integral component of the proofs of our main Theorems . 2 Our Techniques and Results . 2.1 Preliminaries . Notations : Let round : R ! Z denote a function that returns the closest integer to a given real input . Let 1n denote a length n vector of all 1 ’ s . We will write [ n ] to denote the set { 1 , . . . , n } and let P ( [ n ] ) be the power set of [ n ] . For a vector v 2 Rn , let vi denote its i-th coordinate for any i 2 [ n ] . We will use supp ( v ) ✓ [ n ] to denote the support of the vector v , i.e , the set of indices with non-zero entries in v. We will abuse notations a bit , and also sometimes use supp ( v ) to denote the binary indicator vector of length n that takes 1 at index i if and only if vi 6= 0 . For a vector v 2 Rn and subset S ✓ [ n ] of indices , let v|S 2 R|S| denote the vector v restricted to the indices in S. Finally , let f : P ( [ n ] ) ⇥ { 0 , 1 } n ! { 0 , 1 } n be a function that takes a binary vector v 2 { 0 , 1 } n and a subset S ✓ [ n ] as input and returns another binary vector v0 such that the indices of v corresponding to the the set S are flipped i.e . v0i = vi 1 if i 2 S and v0i = vi otherwise . Tensor Decomposition : Consider a tensor A of order w 2 N , w > 2 on Rn which is denoted by A 2 Rn ⌦ Rn ⌦ · · · ⌦ Rn ( w times ) . Let us denote by Ai1 , i2 , ... , iw where i1 , i2 , . . . , iw 2 [ n ] , the element in A whose location along the jth dimension is ij i.e . there are ij 1 elements along the jth dimension before Ai1 , i2 , ... , iw . Notice that this indexing protocol uniquely determines the element within the tensor . For a detailed review of tensors , we defer the reader to [ 28 ] . In this work , we are interested in low rank decomposition of tensors . A tensor A can be described as a rank-1 symmetric tensor if it can be expressed as A = z⌦ z⌦ · · ·⌦ z| { z } w times for some z 2 Rn i.e . Ai1 , i2 , ... , iw = Qw j=1 zij . A tensor A that can be expressed as a sum of R rank-1 symmetric tensors is defined as a rank R symmetric tensor . For such a rank R tensor A provided as input , we are concerned with the problem of unique decomposition of A into a sum of R rank-1 symmetric tensors ; such a decomposition is also known as a Canonical Polyadic ( CP ) decomposition . Below , we show a result due to [ 40 ] describing the sufficient conditions ( Kruskal ’ s result ) for the unique CP decomposition of a rank R tensor A : Lemma 1 ( Unique CP decomposition [ 40 ] ) . Suppose A is the sum of R rank-one tensors i.e . A = RX r=1 zr ⌦ zr ⌦ · · ·⌦ zr| { z } w times . and further , the Kruskal Rank of the n⇥R matrix whose columns are formed by z1 , z2 , . . . , zR is J . Then , if wJ 2R+ ( w 1 ) , then the CP decomposition is unique and we can recover the vectors z1 , z2 , . . . , zR up to permutations . There exist many different techniques for CP decomposition of a tensor but the most well-studied ones are Jennrich ’ s Algorithm ( see Section 3.3 , [ 32 ] ) and the Alternating Least Squares ( ALS ) algorithm [ 28 ] . Among these , Jennrich ’ s algorithm ( see Section E for more details ) is efficient and recovers the latent rank-1 tensors uniquely but it works only for tensors of order 3 when the underlying vectors z1 , z2 , . . . , zR are linearly independent ( See Theorem 3.3.2 , [ 32 ] ) ; this is a stronger condition than what we obtain from Lemma 1 for w = 3 . On the other hand , the ALS algorithm is an iterative algorithm which is easy to implement for tensors of any order but unfortunately , it takes many iterations to converge and furthermore , it is not guaranteed to converge to the correct solution . Jennrich ’ s algorithm also has the additional advantage that it will throw an error if its sufficient condition for unique CP decomposition is not satisfied . This property will turn out to be useful for the problem that we study in this work . Finally , notice that if A is the weighted sum of R rank-1 tensors i.e. , A = RX r=1 r z r ⌦ zr ⌦ · · ·⌦ zr| { z } w times . then we can rewrite A = PR r=1 y r ⌦ yr ⌦ · · ·⌦ yr where yr = 1/wr zr . If { yr } Rr=1 satisfies the conditions of Lemma 1 and if it is known that { zr } Rr=1 are binary vectors , then we can still recover zr by first computing yr and then taking its support for all r 2 [ R ] . Subsequently , notice that we can also recover { r } Rr=1 . As we discussed , for tensors of order w > 3 , there is no known efficient algorithm that can recover the correct solution even if its existence and uniqueness is known . Due to this limitation , it was necessary in prior works using low rank decomposition of tensors that the unknown parameter vectors are linearly independent [ 9 , 3 ] since tensors of order > 3 could not be used . However , if it is known apriori that the vectors { zr } Rr=1 are binary and the coefficients { r } Rr=1 are positive integers bounded from above by some C > 0 , then we can exhaustively search over all possibilities ( O ( C2n ) of them ) to find the unique decomposition even in higher order tensors . The set of possible solutions can be reduced significantly if the unknown vectors are known to be sparse as is true in our setting .

Summary: This paper considers a support-recovery problem for a mixture of sparse linear classifier and regressions with $l \ge 2$ number of components in the active-learning setting, i.e., where we can design input-vectors (or queries, as few as possible). For sparse linear classifiers, unlike in previous work [19] where supports of any two unknown vectors are disjoint, the authors consider a few milder assumptions on the support (Definition 1 and 2). For sparse linear regressions, they do not impose target vectors to be on some scaled integer lattice [27, 44] (but instead we need minimum magnitude $\delta > 0$ for any non-zero entry, and some high SNR condition).   In order to recover all supports, three assumptions are considered for the support-matrix: (1) p-identifiable, if some size $p$ subset of support locations uniquely decides an unknown vector, then the authors can devise some combinatorial approach extending [19], (2) flip-independent, which essentially is required for some 3rd-order (support) tensor uniquely decomposable, (3) more general case where we deal with higher-order (support) tensor decomposition which cannot be done in polynomial-time in general but for full-rank 3rd order tensors.   The heart of the proposed algorithms for all three cases is to compute $|occ(C,a) := \{v \in \mathcal{V} | \ supp(v)|_C = a \}|$ for all possible combination of $C$ and $a$ of a fixed length $s \ge 1$ (Lemma 2). It seems quite non-trivial to compute all these $\Omega(n^s)$ quantities using only $O(log n)$-queries. Equipped with this powerful lemma, subsequent procedures seem to be nice combinatorial arguments adjusted to different assumptions. Overall algorithms and proofs are quite non-trivial.