Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Paper proposes a new generic algorithm for solving sparse optimization problems with very large number of variables, with convex or non-convex penalties (regularization). Their algorithm 'skglm' uses working-set strategy on the outer loop and coordinate descent with Anderson acceleration in the inner loop. Working set strategy enables scaling to millions of variables.
strengths_and_weaknesses: Strengths
- broad area of application for the proposed algorithm, surely relevant to the community
- work is placed well into the literature
- algorithm shows promising experimental results
- code is available and should be easy to use

Weaknesses
- I find the paper overly technical and hard to read. For example, would be good to include definitions of MCP and SCAD in the paper, as especially MCP is referred to many times. Similarly, it would be good to include the definition or intuition of Anderson acceleration in the paper itself.
- The intuition, story, about how how authors arrived to the solution could be clearer. As the proposed algorithm seems combination of existing methods, what is the great insight, research contribution, that leads to this solution?
- all plots are way too small and hard to understand. Consider different ways to present the results in a clearer way.


As a summary, the results look promising, even though I am not an expert of this particular area. By making the paper more readable, self-contained and intuitive, it would be more impactful. Consider limiting the other packages you compare to (you can include in the appendix) and highlighting the experiments where the algorithm particularly beats the competition,

questions: - in the ablation study you note that the Anderson acceleration does not help unless working set strategy is used. This seems counterintuitive, isn't working set strategy quite orthogonal technique? Do you have any insight on this.
- 
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper proposed a new algorithm to estimate generalized linear model with separable penalties. The algorithm is based on coordinate descent, working sets and Anderson acceleration. It is implemented in a new flexible sklearn compatible package. The numerical experiments show time improvement compared to existing methods.
strengths_and_weaknesses: The authors proposed a generic algorithm that uses coordinate descent, working sets and Anderson acceleration for generalized linear model with sparsity-induced separable penalties (which could be nonconvex). For a certain class of nonconvex penalties, they provided the theoretical analysis for the convergence, support identification, and local convergence rate of the proposed algorithm under certain regularity conditions. They implemented the algorithm in a flexible way that is compatible with scikit-learn package, which can also accommodate other penalties. They also perform extensive numerical experiments for both convex ($\ell_1$, $\ell_1-\ell_2$) and nonconvex penalties (MCP), on huge-scale real datasets synthetic. They also conduct ablation study on working sets and Anderson acceleration. The paper is well written. 
The following are some questions/comments:
1. There are many assumptions and propositions in the paper, and the important propositions 10 and 13 uses multiple assumptions that may not hold in reality. Some of these assumptions like convergence to a critical point may be standard for the convergence analysis. But some of them such as Assumptions 11 and 12 might be too strong and may not hold for a general formulation, or even MCP. I think it is better to illustrate how strong these assumptions are. For example, for Proposition 13, Assumptions 1,2, and 8 and convergence towards a critical point might be standard; and piecewise quadratic holds for MCP regression; what about assumptions 9, 11 and 12 for $\hat\beta$? Because I do not think Assumption 11 and 12 will hold for any $\hat\beta$. Especially when $|\hat\beta_j|=\gamma\lambda$, Assumption 11 will not hold?
2. Nonmonotonicity of the curves: when I read the figures for the first time, I noticed this nonmonotonicity in terms of the duality gap vs time curve. After carefully reading through the paper, I found that the authors explained it in line 219 on page 7 as well as in the last paragraph of supplement. It is probably better to point out this to reader and refer to this paragraph in some of these plot captions, so as to improve readability. Furthermore, I am wondering if it is possible to fix different levels of duality gaps and plot the median time of achieve this duality gap with the package to ensure the curve’s monotonicity?
3. For the MCP plot (Figure 5), how is optimality condition in the y-axis of the bottom panel defined?

questions: See "Strengths and Weaknesses"
limitations: See "Strengths and Weaknesses"
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a fast co-ordinate descent algorithm for sparse linear models with \alpha-semi-convex penalties. The algorithm includes two key step, one is to introduce a score to rank the variables and to get a working set; the other is to user Anderson acceleration in the inner-loop. The authors give a series of theoretical guarantees. The numerical experiments show the effectiveness of the proposed algorithm. 
strengths_and_weaknesses: I am not an expert of optimization theory and I might not be authoritative enough to evaluate the theoretical contributions of the paper. 

Pros:
1. the theoretical results seem rich and solid.

Cons:
1.  the algorithmic contributions are not clearly articulated. It seems working set and Anderson acceleration are known ideas, right? How novel is it to apply them in the co-ordinate descent framework? Is there any other CD algorithm that uses the two ideas? Also, is the proposed score function brand-new? The authors should clarify their contributions in the two critical steps. 
2. The paper does not introduce the details about Anderson acceleration, not even including a brief summary. 
3. Line 93, why n_k is bounded by 2 | gsupp(\beta^t)|? The authors do not explain the motivation and intuition. Can we change the factor 2? Since the size of the working set is important for the efficiency, the authors should give a clarification about the size choice. 
4. The authors should be aware of Bayesian sparse learning works, such as based on the spike-and-slab prior and horse-shoe prior. They are proven to be much more effective than L1/L1 +L2, and can  scale up to a large number of features/variables. They can provide uncertainty quantification, and well support non-convex loss functions. See the reference algorithms in [1][2]

[1] Ghosh, Soumya, Jiayu Yao, and Finale Doshi-Velez. "Model Selection in Bayesian Neural Networks via Horseshoe Priors." J. Mach. Learn. Res. 20.182 (2019): 1-46.
[2] Fang, S., Zhe, S., Lee, K. C., Zhang, K., & Neville, J. (2020, November). Online bayesian sparse learning with spike and slab priors. In 2020 IEEE International Conference on Data Mining (ICDM) (pp. 142-151). IEEE.
questions: See above 
limitations: N/A
review_title: null

Meta-review: metareview: The paper proposed a fast coordinate descent algorithm for sparse linear models with \alpha-semi-convex penalties. The algorithm includes two key steps:  to introduce a score to rank the variables for obtaining a working set, and to use Anderson acceleration in the inner-loop. The theoretical analysis and the numerical experiments show the effectiveness of the proposed algorithm. The reviewers raised several concerns, which should be addressed in the final version.
confidence: Certain
Decision: Accept