Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposed a novel optimization algorithm for the Group Elastic Net problem, with additional application to the function-on-scalar regression problem. By exploiting the sparsity structure of the Augmented Lagrangian, the new algorithm significantly reduces the computational load. Experiments also showed supreme results, comparing with multi-task elastic net procedures in sklearn and glmnet. The paper is well written and easy to follow.
main_review: The following contributions from the paper are quite novel:
* Exploiting the analytical form of the Fenchel conjugate of the group elastic net regularization.
* Using semi-smooth Newton method to solve the dual variable in the subproblem.
* Applying the algorithm to function-on-scalar regression problem.

This novel optimization algorithm proposed in this paper can be very valuable to the machine learning practitioners. 
limitations_and_societal_impact: Here are a few minor concerns I have for the paper:

1. Table I only showed the speedup compared with sklearn and glmnet, but no comparison in solution accuracy is provided along with it. The speedup is meaningful only if the solution accuracy is comparable to other methods. A better way to compare different algorithms is to show the accuracy (or error) along with the time spent in the computation. 

2. The discussion on the convergence rate could be more rigorous. For example, the super-linear convergence rate is for the semi-smooth Newton subproblem, and I did not see any strict theorem or proof of convergence rate for the whole algorithm.

3. The augmented Lagrange algorithm converges as the \sigma variable goes to infinity. How is this increase in \sigma scheduled in the algorithm? Will different scheduling scheme impact the convergence property?
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper adds to the various existing solvers for group elastic net regression. Here, a grouping of input variables is jointly penalized with both L1 and L2 penalties in order to yield an estimate of sparse penalized regression coefficients. The newly developed optimization algorithm — semi-smooth newton augmented langrangian — incorporates group structure and leverages sparsity to solve the optimization problem an order of magnitude faster than off-the-shelf methods from python sklearn and R glmnet. The method is further extended to function-on-scalar regression by leveraging functional principal components analysis. This extension allows for analyzing real data where the target is functional curves, in this case growth curves of childhood obesity development. In the INSIGHT study, one single nucleotide polymorphism was found to be significantly associated with childhood obesity, a locus occurring in a gene that has been previously associated with anthropometric-related phenotypes.
main_review: The authors present a novel algorithm for fast optimization of group elastic net regression. This is further extended to function-on-scalar regression to increase the relevance of the method. The motivation is clear and the paper is well-organized and clearly written. The experiments are clearly laid out and easy to follow. The empirical results comparing computational efficiency are strong, and it seems the new methodology here could become a new baseline, given its comparison to already highly adopted packages. Theoretical results further confirm the efficiency of the proposed method. The applications to a childhood obesity GWAS provide a novel perspective on traditional GWAS framework, and while the result of confirming a previously discovered significant association is encouraging, there are some areas that could be improved to increase impact.

First, GWAS is not typically considered with penalized regression due to identifiability issues. For this reason, it may be more applicable to consider this methodology for prediction rather than hypothesis testing, which is difficult to interpret in this setting. The signal-to-noise ratio in GWAS is often very low (as defined by the overall variance explained or variance explained per SNP), and it appears the current methodology might struggle in this setting as well. In particular, there is only a single SNP that’s identified by the proposed method, and the $c_\lambda$ parameter is effectively at its maximum, so the model is being constrained as heavily as possible. It might be more fruitful to consider applications such as polygenic risk scores (where prediction is considered), or to look into gene expression experiments with measurements taken over time in order to increase the sample size. The sample size here (n=210) is bound to be underpowered to detect associations.

limitations_and_societal_impact: The authors have addressed limitations of their method. Societal impacts of the GWAS application should be discussed a bit more thoroughly considering this methodology paves the way for more function-on-scalar type GWAS to be investigated, and the current assumptions of the method should be made clear for those interested in hypothesis testing.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The main contribution of the paper is two-fold.
- First, the paper proposed a fast algorithm to learn the regression coefficients with the group elastic net regularization. Suppose that the dataset has k-dimensional response variables, and therefore the regression coefficients B has the form $B\in R^{p\times k}$ (p: number of features). Then the group elastic net regularization enhances the sparsity of features for all k response variables (i.e., some rows in B are to be zero).
- Secondly, the paper proved that it can be extended to function-on-scalar setup, that is, we assume a regression problem where the response variable is a functional object of summing $k$ basis functions. (Therefore each element in $B\in R^{p\times k}$ is a functional object.)

main_review: The proposed method is to efficiently learn a sparse function-on-scalar model, and (although it is not specifically stated in the paper) a sparse multitask model. It is theoretically proved that, even if the number of features is huge, the main computational cost depends on the number of only a part of features ($|\mathcal{J}|$ in the paper), which is of value.

Suggestions:

-   The improvements from the conventional method is better to be stated in Section 1. For example, the paper stated three conventional methods in line 152 and their limitation for our problem setup, but the overview of such discussions is better to be appeared in Section 1 to emphasize how the proposed method achieved the improvements.
-   The paper mainly consider a fast method to learn a sparse function-on-scalar model, but it solves a sparse multitask model (multiple response variables) as a building block. So, in order to clarify the advantage of the proposed method, it is better to discuss the relationships to conventional learning methods for sparse multitask model. For example, the following paper discusses the group lasso (instead of group elastic net) regularization for multitask learning:
    -   Karim Lounici, Massimiliano Pontil, Alexandre B. Tsybakov and Sara A. van de Geer, "Taking Advantage of Sparsity in Multi-Task Learning". COLT2009.

Questions:

-   Suppose that we consider a model with ordinary response variable (i.e., non-multitask, non-function-on-scalar). Then, does the proposed method work efficiently compared to conventional elastic net learning methods? In other words, how the multitask setup and/or function-on-scalar setup is employed for efficient computation?
-   In Section 3.1, the definition of $\hat{X}_\mathcal{J}$ is not so obvious: since $T\in R^{p\times k}$, $\mathcal{J}$ must be $\mathcal{J}\subseteq\{1, 2, \dots, p\}$ and therefore inappropriate as a subscript for $\hat{X}\in R^{nk\times pk}$. Does it mean that $\hat{X}_\mathcal{J} = \hat{\mathcal{X}}$ where $\mathcal{X} = X_\mathcal{J}$?

limitations_and_societal_impact: As stated in the main review, the proposed method can largely reduce the learning cost both theoretically and experimentally. So it seems to be of use.

The limitation of the method is discussed in Section 5. As possible extensions of the method, the authors consider (1) different number of coefficients in a group, and (2) use for other functional models than function-on-scalar model.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a new group elastic net algorithm solver for linear regression problem with multivariate response variable. The paper then claims its application on function-on-scalar.
main_review: I am not an expert in optimization, but I feel that this paper is strikingly similar to [1], which aims to solve a similar elastic net problem but with univariate response variable. While the paper extends the univariate case to multivariate case, I think this extension is not very complicated. Most of methods and results have similar corresponding in [1]. At least for me, this paper seems trivial. I think that the authors should give a thorough related work section and explain what their main contributions are.

Besides, I feel that the application on function-on-scalar regression is also a trivial application. The contribution of using FPCA is overly stated, since it seems very natural and is not the only choice. In fact, by using any function basis and doing dimension reduction via projection, we can always easily convert a function-on-scalar problem to a linear regression problem with multivariate response.

The paper is technically sound. The results are well proved.

The paper is also clearly written and well-organized.

In terms of significance, I think the paper has its own value in application perspective. Since the problem solved by the paper can appear in many places, an efficient implementation will be valuable. However, my main concern is about its limited originality as mentioned above. I would suggest this paper to be submitted to some statistics software focused conferences/journals rather than methodology/theory focused ones like NeurIPS. If the authors insist, it is better to explain its novelty in terms of methods or theoretical analysis.

limitations_and_societal_impact: The authors claim that they do not see any potential for adverse societal impacts. Since this paper is mainly theoretical, I agree with the authors' claim.
review_title: null

Meta-review: Three reviewers indicated acceptance, only one reviewer had concerns about the novelty with respect to reference [1]. It turned out, however, that [1] is basically an unpublished pre-print of this paper, and in their rebuttal, the authors could convincingly show that even with respect to [1], the paper contains some important extensions. So I recommend acceptance of this paper.   
Decision: Accept (Poster)