Spectral Group Lasso for Selecting Factors Hidden in Plain Sight

Authors: Naixin GUO, Zhixin Zhou, Arash Amini, Guanhao Feng
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: variable selection, group lasso, factor model

Abstract:
We decide to withdraw our work from the conference.

Decision: reject
Reviewer scores: 5, 6, 3, 5

Reviews and discussion:
<thread>
creation date:
1730376629671

thread_type:
official_review

<thread_object>
creator: reviewer_qz4c

<content>
summary:
The manuscript introduces two new approaches for selecting factor variables in high-dimensional settings: Directed Group Lasso (DGL) and Spectral Group Lasso (SGL). Both approaches are based on the group lasso regression approach. The methods aim to identify factors spanning linear subspaces close to the space spanned by the full candidate factors. The SGL approach also utilises the dimensionality reduction of Principal Component Analysis (PCA) to provide better computational efficiency over DGL. The methods are developed theoretically and applied to both synthetic and real data.

strengths:
The paper is generally well-written and easy to follow. The authors provide a nice introduction to the problem. The manuscript also clearly motivates the problem. Specific strengths are:

1. The manuscript contains detailed and extensive theoretical results, providing strong theoretical guarantees. This is particularly important under high-dimensional settings, as such results are not always trivial to derive. 
2. The theoretical results are described very well. Each theorem and proposition is described in terms of its broader importance and the purpose for it being used. This is very important in theoretical papers, to allow the reader to follow without getting bogged down by extensive theory.
3. The assumptions are presented. 
4. The manuscript uses an interesting model (the group lasso) and finds intriguing further applications for it.

weaknesses:
My main concerns are regarding the empirical results. I'm not convinced by both the scope and outcome of the results. Specifically:

1. Both the synthetic and real data studies are very limited in scope. In particular, the real data results should be in the main report, and not the appendix. For any method development manuscript, I feel that the real data results can be the most insightful, because the conditions under which the theoretical guarantees are derived are often not present in practice, so it is vital to observe the performance of such methods when this happens.
2. I would have liked to see a direct comparison to other existing methods. Currently, the results shown do not provide compelling evidence for the usefulness of the methods. 
3. More discussion around the following is needed: computational cost/complexity (including empirical results, such as timing tests), limitations of the approaches, and reproducibility.


Suggestions:

1. I would be careful with the wording in some places around the group lasso. In particular the first line of the conclusion makes it sound as if the manuscript introduces the group lasso as a method. 
2. I would perhaps move some of the theoretical results to the appendix to make room for more empirical results.

questions:
1. This question comes from studying the group lasso under the general linear regression case in high-dimensional settings. I've found the group lasso to often be quite limiting in its performance. In particular, applying the group lasso causes full groups to be selected, which often includes noisy variables. As such, I have often found the sparse-group lasso [1] to perform much better. Can you draw any parallels between your work and this? Would there be any use for the sparse-group lasso in your work?
2. Can you summarise the key challenges associated with the development of your methods?

Refs:

[1] Simon, N., et al. 'A Sparse-Group Lasso'
</content>
</thread_object>

</thread>
<thread>
creation date:
1730722459163

thread_type:
official_review

<thread_object>
creator: reviewer_6yfg

<content>
summary:
The paper presents a Group Lasso Approach (GL) for variable selection in high-dimensional data, introducing Directed Group Lasso (DGL) and Spectral Group Lasso (SGL) methods. DGL identifies factors that span a linear subspace, and SGL incorporates PCA in its first step. Both methods are analyzed theoretically. The authors also claim to verify theoretical claims empirically using the F1 score metric for support recovery. For detailed comments, please see the weakness and questions section.

strengths:
1) Proposed an interesting problem
2) Proposed interesting algorithms for the problems 
3) Theoretically rigorous analysis of the proposed methods

weaknesses:
1) The problem is not well motivated with respect to utility for the ML community 
2) The paper seems limited to analyzing two methods proposed by authors for a new problem. The authors should mention their contributions in comparison to existing literature more clearly. 
3) Assuming no error in $X_S$ seems to be a very restrictive assumption.  
4) The assumption in Eq 13 made in Lemma 1 seems theoretically motivated to prove the lemma and is not justified/motivated from a practitioner's perspective. I have some comments for Eq 14 in Theorem 1 and Eq 17 in Theorem 2. 
5) For example, kindly see the mutual incoherence assumption made in the standard support recovery paper: [https://ieeexplore.ieee.org/document/4839045](https://ieeexplore.ieee.org/document/4839045). One can get some intuition about the problem from such an assumption on mutual incoherence but not from the assumptions made in the submission.  
6) The authors discuss support recovery in theory but use the F1 score in their experiments. It would be easier to verify the theoretical claims if the authors directly compute the empirical support recovery probability. For example, please refer to experiments of [https://ieeexplore.ieee.org/document/4839045](https://ieeexplore.ieee.org/document/4839045). 
 
Minor comments
1) Section 3.1 seems to be slightly verbose. I think it can be put in the Appendix. 
2) The authors should provide reference for Wedin’s theorem in line 368 
3) Theorem 1 and Theorem 2 seem to be a corollary of Lemma 1 and not novel theorems.

Typos: 
1. Line 190,192, 357: Equation is typed twice 
2. Various typos in Eq 13. 2-> $\inf$ should be in subscript for the norm? 
3. Line 343: Problem in equation 5
4. Line 397: I think the authors mean the right-hand side instead of the left-hand side. 
5. There are also some grammatical errors in the submission.

questions:
1) What do the authors mean by factors hidden in plain sight? 
2) What is the research the authors are referring to in line 41? They should cite relevant literature. 
3) In line 211, the authors claim their method DGL is computationally expensive when p is large. What is the computational complexity? 
4) Is the SGL method less computationally expensive as compared to DGL method? It seems the SGL method has an additional step of PCA whose computational complexity is O(n^3). So, what is the motivation for doing SGL?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730732973400

thread_type:
official_review

<thread_object>
creator: reviewer_7szu

<content>
summary:
This paper presents an approach for variable selection in an unsupervised learning setting (i.e., no respondent variable). It was assumed that redundant variables can be represented by linear combinations of true factors. Under such assumption, two optimization problems were formulated, with one targeting the original data matrix and the other one targeting the data matrix after dimension reduction. Theoretical results were provided to show the property of the proposed optimization problems. Empirical study was performed using synthetic data, mainly for demonstration purpose.

strengths:
- The paper is well written and easy to follow in general. 

- As far as I know, the proposed formulation of feature selection in an unsupervised setting is new. My biggest concern is its practical relevance (see weakness).

- Theoretical results are provided to demonstrate the property of the formulated optimization problems.

weaknesses:
- The practical significance is unclear. There is neither discussion of examples of practical applications that motivate the formulated learning problem nor empirical studies on real-world data. 

- In their derivation, it is assumed the data on true factors are noisy free. However, measurements in real world applications are noisy. Could the authors clarify how noisy measurements among true factors may affect the solution obtained by the proposed algorithms. 

- The formulated learning problem is very related to sparse subspace clustering, both involving sparse self-representation learning, which should be discussed and related. 

- In both proposed algorithms, there is a hyperparameter \tau. But I did not find the authors discuss how to determine a value to use for \tau and the impact of different choices.

questions:
See weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1731063867404

thread_type:
official_review

<thread_object>
creator: reviewer_pkuf

<content>
summary:
In this paper, the authors study the problem of identifying the factors in a dataset. They propose a strategy based on structural equation modeling, incorporating the group lasso penalty. They study two variants: one with the given input matrix X, and the other with the left singular vectors matrix truncated to the top-k vectors. The authors, through theorems 1,2 derive the conditions under which the proposed formulation recovers the true support and the underlying model parameters. The authors have illustrated the performance of the proposed approach over simulated data settings.

strengths:
- Overall the proposed approach, the use of Group Lasso for identifying factors through structural equation modeling, is easy to follow. 
- The proposed approach is also supported with the theoretical statements on the support recovery. 
- The application of group lasso for this problem appears to have been the first attempt for this problem (to the knowledge of the reviewer).

weaknesses:
The main weakness seems to be in the presentation of the paper in the broader context of the problem. There are not much discussion related to the SEM approaches for the factor analysis, and how the presented strategy is significant in the context of the existing approaches. For this reason, it becomes difficult to assess the significance of the results in the broader context. 

The novelty of the theoretical results are not very clear, once the problem is posed as a version of the group lasso regularized regresison, for which there exists well understood results on consistency and model recovery. 

Minor: The writing may be improved in some sections
- The discussion leading up to the main problem formulation line 198 might be derived in a more rigorous way. 
- Line 189, "Assuming the permutation matrix P = I, equation equation 4 simplifies to " -> there is no P in equation 4

questions:
In line 180, we make a structural assumption on the true model, to have an identity, non-identity \tilde{B}^*, and 0 components. Is that structural assumption being used anywhere in the solution? Curious to understand if the fitted model values follow the underlying assumption.
</content>
</thread_object>

</thread>


Metareview: