Support Recovery of Sparse Signals from a Mixture of Linear Measurements

Abstract: Soumyabrata Pal, Arya Mazumdar, Venkata Gandikota

Reviews and author responses:

 I'd like to thank the authors for answering my questions. I will keep my original score.

 The paper addresses a generalized problem in 1-bit compressed sensing, considered in several papers [9, 19, 27, 29]. The problem is to recovery the supports of ALL of l unknown k-sparse vectors from 1-bit noisy measurements based on a sufficient number of query vectors (See Sec. 1.1). The problem is addressed for both mixture of linear classifiers (MLC) and mixture of linear regressions (MLR). 

Corollary 1 provides sufficient conditions for the number of queries to recover the supports with asymptotically probability 1. The corollary follows immediately from Theorems 1 and 2 based on the notion of an identifiable set of l sparse vectors in Definition 1. Theorem 3 postulates a flip-independent set of l sparse vectors in Definition 2 to use Jenerich’s algorithm for the canonical polyadic (CP) decomposition of a tensor. Theorem 4 postulates that the set of sparse vectors has a known Kruskal rank r to guarantee the uniqueness of the CP decomposition. Theorem 3 provides a tighter sufficient condition than Corollary 1 for l>4 while Theorem 4 is tighter than Theorem 2 for (2l-1)<(r-1)(log l + 2), which holds for small l.  
  Originality

 The main originality is in Lemma 2, claiming the existence of an algorithm to count the size of the set occ(C, a) of unknown vectors of which corresponding supports have a substring a at positions indexed by C for all C and a (See Sec. 2.2). In fact, Theorem 1 is based on Algorithm 1, which utilizes the identifiability and the size |occ(,)|. Theorems 3 and 4 also use the size |occ(,)|. As a result, these theorems claim similar orders in l, k, and n, which originate from Lemma 2. 
 The proof of Lemma 2 in the supplementary material is based on two subroutines to count the sizes |\cup_i occ((i),1)| and |occ((i),1)|. The former is counted via Algorithm 4 using a cover free family (CFF) [24] of queries in Definition 7 while the latter is computed via Algorithm 5 using a robust union free family (RUFF) [16] of queries in Definition 6. Lemma 2 is proven by combining the two algorithms. 

Quality

The tightness of Corollary 1 and Theorems 3, 4 (or essentially Lemma 2) is not discussed. Compare them with state-of-the-art results and clarify whether the bounds are tight. 

Clarity

 The paper is well organized. The main idea of the proof strategy is outlined in Sec. 2.2. While Lemma 2 is the main result, the proof of Lemma 2 is technically complicated. Thus, the proof is presented in the supplementary material. The proofs of Theorems 1, 3, and 4 are easy to follow since the proof strategy has been clarified in Sec. 2.2. Nonetheless, I have several comments for a further improvement.

--In Sec. 2.1, the authors regard that the unique CP decomposition indicates trivially correct support recovery and therefore present no explanation. An addition of a brief explanation for this point should improve the readability in understanding Theorems 3 and 4.  

--it would be better to clarify the meaning of “occ” in Sec. 2.2. In the definition of occ(C, a), clarify that supp(v^i) represents the binary indicator vector of length n. 

--The authors should be mathematically careful in the descriptions after “if A is p-identifiable, then the set of distinct support vectors have a unique sub-string of length at most p,” in terms of some or all. In my understanding, “for each unknown vector” in line 195 should be “for some unknown vector.” Is “unique” correct in line 196. I do not understand why the uniqueness is required. The first j in line 197 should be all j/in C\backslash supp(v). Thus, “for some j” in line 198 is not appropriate.  

Significance 

 In the proof of Lemma 2, the construction of CFF is information-theoretic (not constructive). As a result, the authors cannot simulate the algorithms used in the proofs. I believe that NeurIPS requires not only theoretical results but also practical algorithms unless the theoretical results are extremely novel results. 

--It would be better to present a practical construction of CFFs even if the construction is suboptimal.
  As pointed out in Significance, the paper is information-theoretic and cannot simulate the algorithms numerically. As a result, the paper provides no negative social impact. 

 We thank the reviewer for increasing the score to 7. We will add the numerical results in the supplementary material in the subsequent version.

 I'd like to thank the authors for addressing my concerns, and to keep my original score.

 Thanks to the authors for their careful response to my review. They have addressed my comments appropriately. Thus, I agree with the evaluation of the other reviewers and am going to increase the rating to 7. If the paper is accepted, I would like to suggest an addition of the numerical results into the supplementary materials strongly. 

 **"--In Sec. 2.1, the authors regard that the unique CP decomposition indicates trivially correct support recovery and therefore present no explanation. An addition of a brief explanation for this point should improve the readability in understanding Theorems 3 and 4. "**

We thank the reviewer for the suggestion. We will include further details in Sec 2.1. We remark that the suggested details are provided in the proofs of Theorem 3, and 4 in Sec 3.2 and Sec 3.3 respectively. In essence, if a tensor $A$ is low rank i.e. it can be written as a sum of R rank 1 tensors (as described in Lemma 1), then by a CP decomposition, we obtain a set of R rank 1 tensors whose sum is $A$. Uniqueness of this CP decomposition implies that there is no ambiguity; there can only be a single solution and we obtain the R latent rank 1 tensors and the corresponding vectors.

**"it would be better to clarify the meaning of “occ” in Sec. 2.2. In the definition of occ(C, a), clarify that supp(v^i) represents the binary indicator vector of length n. "**

Occ(C, a) has been explained in Sec 2.2, line 165 - the first time it was defined. The definition is reiterated in the beginning of Sec 3 (line 295). In line 117, when we introduce the notations, we specify that “supp(v^i) represents the binary indicator vector of length n.” We will try to make these definitions more prominent in the final manuscript.  

"**--The authors should be mathematically careful in the descriptions after “if A is p-identifiable, then the set of distinct support vectors have a unique sub-string of length at most p,” in terms of some or all. In my understanding, “for each unknown vector” in line 195 should be “for some unknown vector.” Is “unique” correct in line 196. I do not understand why uniqueness is required. The first j in line 197 should be all j/in C\backslash supp(v). Thus, “for some j” in line 198 is not appropriate.**"
 
We thank the author for catching this error. We will definitely fix this in the final version. The corrected version should be: “If $\mathbf{A}$ is $p$-identifiable, then there exists an unknown vector $\mathbf{v}$ , a set of coordinates $C \subset [n], |C| \le p$, and substring $a \in \\{ 0,1 \\}^{|C|}$, such that $\mathsf{occ}(C, a) = \mathbf{v}$ . Therefore we can identify the $\mathsf{supp}(v)$ by identifying all $j \in [n] \setminus C$ such that $|\mathsf{occ}(C \cup {j}, (a,1))|  = 1$ . We can then recurse by removing the column in $\mathbf{A}$ corresponding to $\mathbf{v}$ and updating the $\mathsf{occ}(\cdot)$ values”  We also refer the reviewer to the rigorous proof of Theorem 1 (page 7) where this aforementioned error does not creep in and therefore, we hope that it will clarify any doubts in this approach.

**"In the proof of Lemma 2, the construction of CFF is information-theoretic (not constructive). As a result, the authors cannot simulate the algorithms used in the proofs. I believe that NeurIPS requires not only theoretical results but also practical algorithms unless the theoretical results are extremely novel results. "**

 In [35, 18, 19], the authors show that random Bernoulli matrices with sufficient rows satisfy the condition of a CFF with very high probability. Therefore randomized constructions are very efficient and practical. This can be easily used in the algorithms. 
Note that the paper contains information about feasibility of the algorithms, in terms of both sample complexity and computational complexity. Please see Remark 5 (line 288-293) for a summary.

We also demonstrate a proof-of-concept experiment to validate Algorithm 1. We set $\ell=3$ i.e. we have 3 unknown vectors of dimension 60. For each of the first two vectors, we randomly choose 5 indices to be their support along with the constraint that their supports intersect on 2 indices. We choose the third vector so that its support is the union of the supports of the other two unknown vectors. Note that with such a choice of the unknown vectors, the separability assumption in [19]  (the support of any unknown vector is not contained in the union of the support of the other unknown vectors) no longer holds true. Let $T$ be the number of times each query is repeated to estimate the $\mathsf{nzcount}(\cdot)$ of the query. For each value of $T \in \{5,10,15,20,25,30,35,40,45,50\}$, we run our algorithm $50$ times and compute the fraction of times (let’s call this accuracy) the support of the unknown vectors are recovered exactly by using Algorithm 1 with $p=2$. In this experiment, the union-free families are simulated by just obtaining a random design which works with high probability. We obtain the following result (here T can be a proxy for the total number of measurements, as the latter grows linearly with T):

| T          | Accuracy (Fraction of times support is recovered exactly)| 
| ---------- | ---------- |
| 5          | 0.02
| 10        | 0.16
| 15        | 0.3
| 20        | 0.38
| 25        | 0.34
| 30        | 0.52
| 35        | 0.8
| 40        | 0.86
| 45        | 0.88
| 50        | 0.9

It is evident that the accuracy increases with the number of times a particular vector is repeatedly queried. This shows that the algorithm indeed works for a set of unknown vectors that for which support recovery was not possible using techniques of [19].


Note that the main reason we did not simulate the algorithms is that we do not believe it will add much value to the paper, and may even take focus away from the sample complexity results.  However, we can definitely add such simulations along the lines mentioned above if the reviewer suggests so.

**"It would be better to present a practical construction of CFFs even if the construction is suboptimal.**"

While the randomized construction of CFFs is practical and will work with high probability, it is also possible to have a strongly explicit construction with a concatenated code with outer Reed-Solomon code and an inner constant weight code (this is quite standard in the Group Testing literature, albeit slightly suboptimal), eg. see Section 3 of [a].
[a]: Stinson and Wei, Generalized cover-free families, Discrete Math., 2004.


 **"One small question concerns Theorem 2 and Corollary 1. In Theorem 2, it says that all binary matrices with distinct columns are p-identifiable for $p \le \log \ell$. But Corollary 1 seems to drop the requirement for the support matrix to have distinct columns, as does the phrase "unconditional" given before Corollary 1. A similar confusion is caused by the sentences around line 372, which state, "we will assume that all of the unknown vectors have distinct supports...However, this assumption is not necessary and all our algorithms work even when the supports are not distinct." In summary, it is not quite clear which results require distinct supports and which do not."**

Note that in Definition 1 (similarly in Def 2, and Def 3) we call the set of supports to be $p$-identifiable if the binary matrix obtained after removing the duplicate columns from the support matrix of the unknown vectors is $p$-identifiable. All the results hold even when the supports are not all distinct. From Corollary 1, we get that any (multi) set of supports is $\log \ell$- identifiable. Therefore the result is unconditional. We make the assumption that all supports are distinct only to ease the high level proof presentation. All the presented algorithms, however, also compute the multiplicity with which each distinct support occurs. We also refer the reader to the proofs of Theorems 1,3 and 4 for further details. 

**"Although I understand that the primary contribution is theoretical, it would have been nice to see whether the proposed algorithms yield better empirical results than those of the prior work [44,27,29]."**

Following the suggestion of the reviewer, we demonstrate a proof-of-concept experiment to validate Algorithm 1. We will compare with the results in [19] since we resolve a problem left open in [19]. We set $\ell=3$ i.e. we have 3 unknown vectors of dimension 60. For each of the first two vectors, we randomly choose 5 indices to be their support along with the constraint that their supports intersect on 2 indices. We choose the third vector so that its support is the union of the supports of the other two unknown vectors. Note that with such a choice of the unknown vectors, the separability assumption in [19]  (the support of any unknown vector is not contained in the union of the support of the other unknown vectors) no longer holds true. Let $T$ be the number of times each query is repeated to estimate the $\mathsf{nzcount}(\cdot)$ of the query. For each value of $T \in \{5,10,15,20,25,30,35,40,45,50\}$, we run our algorithm $50$ times and compute the fraction of times (let’s call this accuracy) the support of the unknown vectors are recovered exactly by using Algorithm 1 with $p=2$. In this experiment, the union-free families are simulated by just obtaining a random design which works with high probability. We obtain the following result (here T can be a proxy for the total number of measurements, as the latter grows linearly with T):

| T          | Accuracy (Fraction of times support is recovered exactly)| 
| ---------- | ---------- |
| 5          | 0.02
| 10        | 0.16
| 15        | 0.3
| 20        | 0.38
| 25        | 0.34
| 30        | 0.52
| 35        | 0.8
| 40        | 0.86
| 45        | 0.88
| 50        | 0.9

It is evident that the accuracy increases with the number of times a particular vector is repeatedly queried. This shows that the algorithm indeed works for a set of unknown vectors that for which support recovery was not possible using techniques of [19].



 **"I hope the authors can address the following concerns:
Line 16 claims "a number of measurements ... quasi-polynomial in \ell ". However, the query complexity in Theorem 4 involves (lk)^(w+1) where . Doesn't this imply an exponential dependence on \ell?"**

From Corollary 1  we know that the query complexity in the worst case would be $O(\ell^{\log \ell})$ since any support set of $\ell$ unknown vectors is $\log \ell$- identifiable. One would therefore use the algorithm from Theorem $4$ only if $w < \log \ell$. Hence, this implies a quasi-polynomial dependence on $\ell$. 

**"Line 165: when defining "occ", explaining what these letters stand for might be helpful for understanding."**

The word “occ” was meant to denote the set of unknown vectors where the argument substring occurs in coordinates indexed by C. It is a short form of occurence. We will make this clear in the subsequent version.
 
**"Line 283 claims "without any assumptions"; it might be fairer to say "with mild assumptions", since the three conditions on the support matrix (p-identifiable, flip-independent, Kruskal rank) are indeed assumptions."**

Theorems 1, 3, and 4 indeed require certain assumptions on the support matrix. However, from Theorem 2, we know that the support set of any set of $\ell$ unknown vectors is $\log \ell$ - identifiable. Therefore, Corollary 1 gives worst case support recovery guarantees without any assumptions. Hence Theorem 1,3 and 4 can only provide improved results as compared to Corollary 1 when the respective assumptions are satisfied.


The authors consider the problem of recovering the support of multiple sparse vectors from a noisy mixture of linear measurements (i.e., mixed linear regression), and from a noisy mixture of signed measurements (i.e., mixed linear classification). That is, each measurement corresponds to one of the unknown sparse vectors, but which vector is unknown.  For these problems, they propose three algorithms, each of which can be applied to either mixed regression or mixed classification. Furthermore, they provide guarantees on the number of measurements that needs to be collected to recover the support with high probability.  The guarantees for their first algorithm hold when the support matrix is p-identifiable, but worst-case guarantees that do not require p-identifiability are also given. Stronger guarantees hold for their second algorithm in the case that the unknown supports are flip-independent. Finally, guarantees hold for the third algorithm when the support matrix has small Kruksal rank.   The theoretical contributions seem to be very strong, although I must admit that this work is outside my area of expertise.

One small question concerns Theorem 2 and Corollary 1.  In Theorem 2, it says that all binary matrices with distinct columns are p-identifiable for p <= log ell.  But Corollary 1 seems to drop the requirement for the support matrix to have distinct columns, as does the phrase "unconditional" given before Corollary 1.  A similar confusion is caused by the sentences around line 372, which state, "we will assume that all of the unknown vectors have distinct supports...However, this assumption is not necessary and all our algorithms work even when the supports are not distinct."  In summary, it is not quite clear which results require distinct supports and which do not.

Although I understand that the primary contribution is theoretical, it would have been nice to see whether the proposed algorithms yield better empirical results than those of the prior work [44,27,29]. I do not see any potential for negative societal impact of this work.

This paper studies the problem of recovering the supports of multiple sparse signals from a mixture of measurements, in both "mixture of linear regressions" and "mixture of linear classifiers" settings. It proposes three algorithms, each with theoretical guarantees under different technical assumptions.  This work makes good theoretical contributions to understanding and solving the problem of support recovery in the mixture models. It identifies three key assumptions (which are milder than those in the existing literature) that enables support recovery with near-polynomial query complexity, though some of the algorithms are computationally inefficient. The comparison with prior work is rather complete. The overall writing is quite polished, and the paper's structure is clean. The technical parts are quite involved and out of my expertise, so I cannot comment much on it.

I hope the authors can address the following concerns:
* Line 16 claims "a number of measurements ... quasi-polynomial in $l$". However, the query complexity in Theorem 4 involves $(lk)^{w+1}$, where $w \propto l$. Doesn't this imply an exponential dependence on $l$?
* Line 165: when defining "occ", explaining what these letters stand for might be helpful for understanding.
* Line 283 claims "without any assumptions"; it might be fairer to say "with mild assumptions", since the three conditions on support matrix ($p$-identifiable, flip-independent, Kruskal rank) are indeed assumptions. N/A

Meta-review: All reviewers have very positively received this paper; all marks match to a clear accept, that I will follow. The concerns by the reviewers seem to have been fully answered. Some reviewers pointed out the lack of numerical experiments, then provided by the authors answers. I believe that, as promised to some reviewers, the authors should definitely add these experiments, possibly in an appendix. Additionally, it seems that some definitions could be slightly more furnished to improve readability. I'm confident that the authors will implement these few improvements for the final version as suggested by the reviewers. Given the overall quality of redaction of the paper and the originality of the results, I recommend acceptance for a poster.

Decision: accept