Title: Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: convex optimization

coordinate descent

generalized linear models
Abstract: We present a novel enhanced cyclic coordinate descent (ECCD) framework for solving generalized linear models with elastic net constraints that reduces training time in comparison to existing state-of-the-art methods.
We redesign the CD method by performing a Taylor expansion around the current iterate to avoid nonlinear operations arising in the gradient computation.
By introducing this approximation we are able to unroll the vector recurrences occurring in the CD method and reformulate the resulting computations into more efficient batched computations.
We show empirically that the recurrence can be unrolled by a tunable integer parameter, $s$, such that $s > 1$ yields performance improvements without affecting convergence, whereas $s = 1$ yields the original CD method.
A key advantage of ECCD is that it avoids the convergence delay and numerical instability exhibited by block coordinate descent.
Finally, we implement our proposed method in C++ using Eigen to accelerate linear algebra computations.
Comparison of our method against existing state-of-the-art solvers show consistent performance improvements of $3\times$ in average for regularization path variant on diverse benchmark datasets. Our implementation is available at [https://github.com/Yixiao-Wang-Stats/ECCD](https://github.com/Yixiao-Wang-Stats/ECCD).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces enhanced cyclic coordinate descent (ECCD), a variant of coordinate descent for fitting $\\ell_1+\\ell_2$ generalized linear models (GLMs). The algorithm approximates gradients via  Taylor expansion and updates $s$ coordinates as a block. Standard coordinate descent is recovered when $s=1$. The advantage of the new algorithm is a computational speedup on standard coordinate descent, which the paper demonstrates theoretically and empirically.\n\nSTRENGTHS AND WEAKNESSES:\nGiven the importance of $\\ell_1+\\ell_2$ GLMs (evidenced by the sheer popularity of glmnet), this direction of research is attention worthy, and I personally like the proposed algorithm. However, I have a few major concerns that in my opinion prevent the paper from being accepted in its current form:\n\n1. It is clear that the method typically provides a speedup over existing methods. However, all the speedups are on very small wall clock times (mostly less than one second). Are such speedups practically meaningful in the vast majority of use cases? In my opinion, a more compelling story would be speedups in situations where existing methods take, e.g., 1, 10, or 100 minutes to run. This issue is my most serious concern.\n\n2. The main limitation of the method is that the optimal block size $s$ is unknown in practice. The authors provide a theoretical choice of $s$, but it depends on unknown quantities and hence is not applicable in practice. If one tries to tune the block size the method is possibly no longer fastest. The authors give $s=8$ as a rule of thumb, but this choice seems a rough approximation in general. The paper should acknowledge this drawback clearly.\n\n3. Do the baselines all use the same convergence criterion? Looking at glmnet, its convergence criterion seems to be based on the change in objective values. Meanwhile, convergence in ncvreg is based on the change in coefficients. Is it really meaningful to compare runtimes on algorithms that do not share the same stopping rule?\n\n4. There is no code available for (a) the method or (b) the experiments. The author's justification is that it is straightforward to replicate the results, but I disagree given that much of the code is in C++. The lack of code severely limits the papers potential impact in my opinion. Why would someone go to the effort of implementing this method when they can use glmnet, which for most practical purposes runs in the same time?\n\nQUESTIONS:\nSee above\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an enhanced cyclic coordinate descent (ECCD) algorithm for generalized linear models with elastic net penalty. By using Taylor expansion to approximate gradients and unrolling vector recurrences into batched computations, ECCD avoids the unstable behavior of the regular BCD algorithm and successfully enhances computation. Implemented in C++ with RcppEigen, ECCD generally achieves 1–3 times speedup over the baselines on LIBSVM benchmark datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nOverall, the paper is well-organized and well-motivated. The mathematical derivation looks solid, and the experiments are comprehensive.\n\n**Weaknesses**\n\n- The authors omit some recent important libraries, e.g., skglm [1] and abess [2].\n\n- The result (14) in Theorem 1 is somewhat over-simplified. In the worst case where $s=p$, the approximation error can be expected to be excessively large, whereas $s=1$ yields zero error. However, the trade not reflected in the current results.\n\n- Numerical results for the lambda path (a typical case) are overstated. For example, the 13× improvement only appears in the airway dataset with one \\(\\alpha\\). Notably, airway has an extremely small sample size ($n=8$), which is atypical for real-world data analysis. In most regular datasets, the improvement ranges from 1.0 to 3.0.\n\n- I am a little confused about why biglasso is so slow. As shown on this page, it is significantly faster than glmnet in memory-intensive cases. Additionally:\n    + How many cores did biglasso use? Since the hardware has 64 cores, using 16 cores could be considered.\n    + I found the current datasets are either large $p$ or large $n$? What is the performance when both $n$ and $p$ is large (e.g., 10k).\n\n- ECCD uses a diagonal Hessian for approximation, while glmnet computes the exact Hessian matrix by default. Would glmnet perform similarly if it used an upper bound on the Hessian?\n\n- Typos in line 414: min{n, , p} --> min{n, p}\n\n[1] Quentin Bertrand, Quentin Klopfenstein, Pierre-Antoine Bannier, Gauthier Gidel, and Mathurin Massias. 2022. Beyond L1: faster and better sparse models with skglm. NeurIPS '22. Curran Associates Inc., Red Hook, NY, USA, Article 2823, 38950–38965.\n\n[2] Zhu Jin, Xueqin Wang, Liyuan Hu, Junhao Huang, Kangkang Jiang, Yanhang Zhang, Shiyun Lin, and Junxian Zhu. \"abess: A Fast Best-Subset Selection Library in Python and R.\" Journal of Machine Learning Research 23, no. 202 (2022): 1-7.\n\nQUESTIONS:\n- Can this method be extended to (Group) SCAD and MCP, when the group of large size?\n\nLIMITATIONS:\nThis work focuses on optimization algorithms for elastic net in generalized linear models, without targeting specific application domains or sensitive data scenarios. Therefore, it does not have direct negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Enhanced Cyclic Coordinate Descent, an efficient block-wise optimization method for fitting Elastic Net penalized generalized linear models. ECCD improves upon naive block coordinate descent by incorporating second-order Taylor corrections to stabilize updates within each block. The authors provide theoretical convergence guarantees and complexity analysis, and demonstrate through experiments on synthetic and real-world datasets that ECCD achieves faster convergence than existing methods.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The proposed ECCD method improves over naive block coordinate descent in GLMs by introducing a Taylor-approximated correction, resulting in a stable and efficient algorithm.\n\n2. The paper provides both theoretical convergence guarantees and practical complexity analysis, offering a clear justification for the expected computational benefits.\n\n3. The experimental results are well-structured, demonstrating ECCD’s stability and speed advantages across many standard datasets, and showing that it achieves the same solution quality as established solvers like glmnet.\n\n**Weaknesses**\n1. The space complexity analysis in Theorem 3 states an overall $\\mathcal{O}(np)$ bound, dominated by the design matrix $X \\in \\mathbb{R}^{n \\times p}$. However, this masks the additional memory overhead introduced specifically by ECCD. A clearer breakdown of the extra storage costs incurred by ECCD, excluding the fixed cost of storing $X$, would provide better insight into the practical memory demands of the algorithm.\n\n2. While the paper provides extensive experiments measuring actual running time, it does not report experimental results on memory usage. Given that the space complexity analysis in Theorem 3 abstracts away the extra storage introduced by ECCD, it would be valuable to include experimental comparisons of memory consumption across methods to more clearly demonstrate the practical efficiency of ECCD relative to baseline algorithms.\n\n3. Minor issues: \n- There is a dangling sentence in the caption of Figure 1.\n\nQUESTIONS:\n1. Could the authors provide a more detailed theoretical analysis of the space complexity of ECCD, isolating the additional memory terms introduced by the algorithm (beyond storing the design matrix $X$)? It would also be helpful to include corresponding empirical results comparing memory usage across methods.\n\n2. In Section 6.1, the authors evaluate ECCD with block sizes $s$ ranging from 2 to 32 on the duke dataset. However, according to Table 6, the number of features is $p = 7{,}129$, which makes even the largest tested block size relatively small. Would it be more informative to explore larger values of $s$ to better assess the performance?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}