Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models
Authors: Yixiao Wang
Zishan Shao
Ting Jiang
Aditya Devarakonda
Venue: NeurIPS 2025 poster ()
Keywords: convex optimization
coordinate descent
generalized linear models
Primary area: optimization
TL;DR: Developed improvements to coordinate descent algorithm for generalized linear models

Abstract: We present a novel enhanced cyclic coordinate descent (ECCD) framework for solving generalized linear models with elastic net constraints that reduces training time in comparison to existing state-of-the-art methods.
We redesign the CD method by performing a Taylor expansion around the current iterate to avoid nonlinear operations arising in the gradient computation.
By introducing this approximation we are able to unroll the vector recurrences occurring in the CD method and reformulate the resulting computations into more efficient batched computations.
We show empirically that the recurrence can be unrolled by a tunable integer parameter, $s$, such that $s > 1$ yields performance improvements without affecting convergence, whereas $s = 1$ yields the original CD method.
A key advantage of ECCD is that it avoids the convergence delay and numerical instability exhibited by block coordinate descent.
Finally, we implement our proposed method in C++ using Eigen to accelerate linear algebra computations.
Comparison of our method against existing state-of-the-art solvers show consistent performance improvements of $3\times$ in average for regularization path variant on diverse benchmark datasets. Our implementation is available at [https://github.com/Yixiao-Wang-Stats/ECCD](https://github.com/Yixiao-Wang-Stats/ECCD).

Enhanced Cyclic Coordinate Descent Methods for Elastic Net Penalized Linear Models
Yixiao Wang"
Zishan Shao"
Department of Statistical Science
Department of Statistical Science
Duke University yw6760duke edu
Duke University zs89@duke. edu
Ting Jiang Department of Electrical & Computer Engineering
Aditya Devarakonda
Department of Computer Science Wake Forest University devaraa@wfu, edu
Duke University tj tj1470duke. edu
Abstract
We present novel enhanced eyclic coordinate descent (ECCD) framework for solving generalized linear models with elastic net constraints that reduces training time in comparison existing state-of-the-art methods. We redesign the CD method by performing a Taylor expansion around the current iterate to avoid nonlinear operations arising in the gradient computation. By introducing this approximation we are able to unroll the vector recurrences occurring in the CD method and reformulate the resulting computations into more efficient batched computations. We show empirically that the recurrence can be unrolled by a tunable integer parameter, 8, such that s > yields performance improvements without affecting convergence, whereas key advantage ECCD is that it avoids the convergence delay and numerical instability exhibited by block coordinate descent. Finally, we implement our proposed method in C++ using Eigen to accelerate linear algebra computations. Comparison of our method against existing state-of-the-an solvers show consistent performance improvements of 3x average for regularization path variant on diverse benchmark datasets. Our implementation is available at httttt:///giiii com/ Com/Yixiao-o Llang-Stats/BCo
1 yields the original CD method. A
Introduction
Generalized linear model (GLM) is cornerstone of modern machine learning and statistics with applications demanding both variable selection and regularization. Among the various penalties, the elastic-net [ which combines & and lg regularization, has attracted significant interest because not only promotes sparsity but also alleviates some of the limitations inherent in using either penalty alone, especially
sichddimnnsional settings. The coordinate descent (CD) algorithm has become popular method for optimizing such models due to its simplicity and effectiveness, typically updating a subset of model parameters at time 2
Despite the widespread success of coordinate-we wiiie updates, block generalization for training GLMs have encountered numerical stability issues especially for large block sizes. Existing approaches
'Equal contribution.
"Corresonndenc:: yu676@duke edu
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

utilize second-order approximations to solve nest penalized GLMs. However, utilizing BCD to train such models has resulted in deteriorating accuracy as the block size increases.
We introduce the enhanced cyclic coordinate descent (ECCD) method for elastic-net penalized GLMs which performs recurrence unrolling of the single coordinate descent update. We couple the recurrence unrolling with a second-order correction in the gradient computation in order to reduce the frequency of expensive nonlinear link function evaluations. The correction term implicitly incorporates updates from one coordinate update to the next, which yields accuracy improvements over classical BCD applied to elastic-net penalized GLMs. The main contributions of this paper are:
1. Derivation of a coordinate descent update strategy that improves accuracy of block coordinate updates over the classical BCD method for elastic-net penalized GLMs.
2. Theoretical analysis to bound the approximation error and theoretically optimal choice of block size for our update strategy.
3. Experimental evaluation of single and sequential C++ implementation of our approach compared against state-of-the-art GLM solvers which shows average 3 speedup on path fits across diverse benchmark datasets.
Coefficient P
0.5
0.0
0.0
8.4
0.6
9:8
1.0
0.0
8.4
0.6
9.8
1.0
A/Amax
== A/Amax
-0.5
-1.0
-1.0
(a) glmnet
(b) ECCD (Ours)
Figure 1: Coefficient trajectories 35 for the diabetes dataset over 100 values of the regularization (see Table Appendix for dataset dimensions). ECCD achieves identical solutions and induced sparsity patterns to GL.Mne.
2 Related Work
Generalized Linear Models In the context of high- high-dimensional inference, GLMs have been extensively studied and optimized for scalable regularization-pale computation. The seminal glmnet framework [2] introduced eyclic coordinate descent with warm starts and inexact line search- trading off formal convergence guarantees to efficiently trace elastic-net paths for linear, logistic, and multinomial objectives. Building nonconvex penalty theory, novreg K extends these ideas to MCP and SCAD penalties. while biglasso 4 leverages memoryymmppiin and sparse linear algebra handle datasets that exceed RAM capacity. For structured sparsity, grpnet [
implements block coordinate updates for group lasso, and adelie [6] employs an ADMM scheme to accelerate group-penne proup-penalized regressions. To reduce per-iteration overhead, many methods integrate feature- screening techniques: "safe rules" (7) guarantee no loss of optimality by preemptively discarding inactive features, and "strong rules" [ perform aggressive but heuristic pruning with minimal KKT checks. These screening strategies have become staples in modern regularization-pal toolkits, substantially lowering the computational barrier to solving large-scale GLMs.
s-Step and Block Coordinate Methods communication in Krylov methods 9 10
The s-step paradigm, originally developed to reduce IT
[2 [4. The proposed s-step Krylov methods were later stabilized and implemented in distributed -ererbytddmmmory systems for multigrid applications
[5] 16 17 I8). More recently, these ideas have been extended to CD solvers and generalized
23 24 Block generalizations of CD methods have also been developed with recent work establishing convergence guarantees for distributed optimization of strongly convex objectives in the deterministic and stochastic block update settings 255 26 Our ECCD algorithm builds directly on these prior results by generalizing the s-step technique to the class of GLMs. Unlike BCD applied to GLMs, ECCD couples the s-step technique
to nonlinear convex optimization [19 20
21 22

---

with a Taylor expansion of the gradient computation to ensure numerical stability for block sizes larger than is feasible through the use of BCD. We also show that our method yields speedups over existing state-of-the-art GLM solvers.
3
Preliminary
3.1 Generalized Linear Model
Type
Distribution
Link
B|y X
F(0)
F'(0)
(*)
d(r)
Gaussian
Identity
o²
Bernoulli
Bernoulli(p)
Logit
1+e-0
log 1 ea))
14
U++
1
Poisson
Poisson(X (X)
Log
e
e
1
Gamma
Gamma(a, 3)
Inverse
log(0)
Table 1:
Common GLMs and their properties. The functions F(0). (O),), and F0) are the cumulant and its derivatives, and d(r)) is the dispersion parameter.
We consider a GLM with mean response: E[y (X where X E R"XP is the design matrix, B E IR is the coefficient vector, and response variable Y follows an exponential-family distribution with probability mass/density function parameterized by and
invertible link function. We assume the
T(v) T()) F(0) d(T)
( 10, c(u,,))
T) exp
(I)
where d(r)) is the dispersion parameter, b(0) is the natural parameter, T(y) is sufficient statistic of y and F()) the cumulant- geellaattg function (CGF). As demonstrated in Table I the specific choice of distribution determines the form of the cumulant function F(0). Throughout, we assume that the first two derivatives F, F²(-) and (-). exist. the canonical form, where b(0) and (②) y. one has: E[Y] = F0),,, Var(Y) (0) d(T). We solve B by maximizing the log- log-likelihood
n y: F(0.) d(T)
((3))
C(u,,),,
0, = x B
where x. is the i-th of design matrix X. We define gj and hy as the first and second derivatives of the log-likelihood with respect to By. respectively:
08
(O.) 1/4 d(T)
0l
O.) d(T)
9j
03,
Fejs
h
033
zny.
3.2 Traditional Coordinate Descent for Elastic Net Regression
We begin with the classical CD method for clastic net-regularized generalized linear models, as developed in prior work (2), which accelerates updates via first-order Taylor expansion, bypassing line search at the expense formal convergence guarantees (,,27] One can write the objective as
((B) - --((())) > " || + a111111).
where -E( is the loss function, which is the negative log-likelihood for GLMs, and À > 0 and a E [0, 1] control the overall magnitude and the relative weighting of the lz tridge-like) and L (lasso-like) penalties, respectively. We solve å = arg ming C(B) to obtain the regularized estimator.
In cyclic coordinate descent setting, we update each coefficient B, while fixing the others. Using second-order Taylor expansion around l(()) the value of 33 at iteration f, one obtains the approximate local objective:
((B(00) + (3, - )
(3, - //0)18] + +X 83 + +alB,))
min 85
(5)
n
3

---

where 9j and h are the first- and second-ords order derivatives evaluated at ( Setting the gradient with respect 8, to zero yields
aC(3) 05, n
1 1 n (5, A(1 a)) + do sign(B,) 0.
(6)
Solving for 8g gives the coordinate descent update:
5 99: 11 + X(1 a)
h, 8//0 da
36-11)
(7)
where 7) denotes the soft-thresholding operator defined by
7 if 0 andd y < |=|,
S(2
2 7 = < 0 and 7 < |=|
y 2 |
3.3 Efficient Screening and Warm Start Strategy
To accelerate sparse estimation in GLMs, prior works 12 8 have introduced complementary tech- niques that reduce computational overhead while ensure exact solutions. In this work, we adopt a path- wise CD framework over decreasing sequence of regularization parameters λ > X2 ... AK. where A Amax = max, |X] (y g)//a, and Aι E>max with E [10 10 10--).. When computing stand-alone regularized solution, these strategies can also be applied to single À value.
At each ÅR. active sel A& S (1,..,,p) The set initially includes all previously nonzero coordinates and any that violate the Karush-Kuhu KithhhKKhn-TTckkee Tucker (KKT) conditions. We then iterate: update 33 for As. perform a full KKT check, augment Ar with any newly violating indices, and repeat until convergence. By cycling only over Aş. each iteration costs Axl) instead of O(np), yielding substantial speedups in high-dimensional settings.
start optimization using the solution at λ and restrict update to reduced
To further reduce the size of the initial working set, the sequential strong rule 7 is employed at each dy_ In logistic regression, the rule suggests discarding feature j from the working set if
X [((=1111 2 - A6-10, (k-1) = y -((k-)) ((h-1) = (F(X3(A-11) Bo). (8)
Although this is heuristic, performing a post-convergence KKT check guarantees recovery of any missed variables. Warm starts, strong-rule screening, and active-set cycling with KKT correction together form robust, scalable backbone for both single-A and pathwise GLM solvers.
Methodology
4.1 Block Coordinate Descent
n) because the l penalty encourages sparse solutions. A natural generalization of CD is BCD, in which update a block of coordinates jointly at each iteration. However, we observe that naive BCD often diverges for moderate to large block sizes, shown in Figure 2a (see Table 3 for quantitative results).
GLMs are well-suited for high-dimenssonall regimes (p >>
Each coordinate handles the intermediate value 0; x; B. In classical CD, 6; is recomputed before every single- coordinate update, ensuring that both the gradient gj and curvature hy reflect the most recent B3. In contrast, BCD recomputes 0; only once per block, reusing stale values of yj and h across updates. This delayed refresh can introduce significant numerical instability when is large. For completeness, defer the full derivation of the BCD update rule to Appendix A.I
block remains the standard CD update (Eq. Z).. but BCD differs in how

---

101
10a
0.12
Indices
Rolative Solution Enror
OGF Gradient
107-
10a3
Time (records)
0.10
Matrices
Construct
km]
0.06
Block Coef
10-4
Threshold
S-32
5-4
4
0.00
Updata
Active Set
5
1
10-42
km3
Threshold
0.00
200
400
600
800
200
400
600
800
36
32
Epochs
Epochs
Block Size (5)
(a) BCD
(b) ECCD (Ours)
(c) Time Composition
Figure 2: Results on the duke dataset with a = 0.5 and c 0.01. (a) Convergence of BCD exhibits numerical instability as block size s increases. (b) Convergence of ECCD remains stable even for = 32. The convergence threshold is set to 1e-10. (c) Breakdown of ECCD's runtime components.
4.2 Enhanced Cyclic Coordinate Descent
To stabilize block-wise CD updates, we propose Enhanced Cyclic Coordinate Descent (ECCD). ECCD incorporates second-order approximation corrections through Taylor expansion within each block, compensating for the error that accumulates when multiple coordinates share the same imput to the link function. This targeted correction restores the descent property and empirical convergence even for large block sizes, as shown in Figure 2b
We now derive ECCD. Let I p be the block size, and consider the k-th block. For each coordinate in this block, indexed by l E (1,2......8 we update ,3 as
g(0-1)0te (((-11)
Σ Ag(h-1)0++ ((k-1))+ti 1=0
where ( is the standard basis vector corresponding to the ((k 1)8 i)-th coordinate in the descent step, and A (, 1)s++ 11)+4 is the scalar change at iteration (I: - 1)s + i. Equivalently, we can write ej, with j, = ((k- 1)s + (mod) p) to emphasize the cyclic ordering of the coordinates.
Throughout our paper, we assume the design matrix X is standardized: each column is centered and scaled so | Let 3o be the intercept. For the gradient term F'(a((
X Xg(k--1)--
take a first-order Taylor expansion at 36 g(k-1) + (()(=-1)= and denote the linearized term by Fg.
('( 1)ste 1, +xx = F' (1 1n X(B(--100) F. (36
11ste
8-1 r" (2(t-1))011 x @ (X ((0-1) Ag(h-11)-- (10)
For numerator, we derive the coefficient update A for each coordinate je = ((k: 1)8
() pm without considering the & -penallyy Using the approximation F. we obtain:
o( ((k- 1)o+t) dz
y T X 110- e(x-1) x F3 1n + +X (xB(k-1)en
1jstt
n d(T)
- A(1 a) / 1)0+1
T X e 1)o4t - XT F- ('( 1, 1)(
d(T
(ee-1)st,TT x
FM (F + x)g(k-1)s) @ (X el elt ()5tt)ss))) g 1
1=0
A - a) 8// (1-
(11)
5

---

Algorithm ECCD for GLMs with Elastic Net Penalty (Active-Set Version)
Require: X E R y R" B E RP, 10 R, >> 0, a E [0, H E N, tol > 0.
active set A S {I, .I. F(-): CGF, d(r): dispersion parameter
1: for outer = 1,.... . H/ do
2: Intercept Update: (6 t Po (t-1)
+ 11
3: ECCD Update:
4: for block = 1,,.......
[|///]] do mod |A|
5: k (iter-
6: I. = (Cay: Car-1:..- .... Canta-1)
7:
At (X.11, [1) VF, B t (X.L)) diag( P Io)
8:
for l ,...... do
9:
Øj. nd(T)
y Xej, A
Σ 1=0
B A 1(1-1)
A(1 -a)0//-
10:
Vi nd(T))
Bee X(1 -a) a))
0/0 s a(t-1) +
Pje do
11:
Vi Vi
12:
end for end for
14: Convergence check: Apply Algorithm 3
if converged, break
16: end for
Output: 3. 90
Here we use ((k-1)s 1)s and a(*-1)) the Taylor expansion for the first E - 1 updates within the same de and $6 (k-1))++ remain unchanged within the block.
cycle, since B/ 1)s+t
(k-1))+E
Next, we handle the denominator by freezing the Hessian term across the block, as in classical BCD. Specifically, we hold the second-order derivative F-)-) constant for all updates, avoiding repeated evaluation of this expensive term and thereby reducing computational overhead:
di (kk-1))
e(k-1)o+e,T X F" (X o (X (( + A(1 a).
d(T)) Hence, the interim update is
$ ((k-1)=+1)
A A (lb--1) 1)s+e
(12)
( 1))=)
Incorporating the & f1-penalty leads soft-thresholding step. Thus, the updated coefficient:
1k-1))stt 1)ste
s
g(t-1 1)s + +1 1)stE
A
(13)
2y ((((-1))++))
where S(z, 7) is the soft-thresholding operator. This completes one ECCD iteration for the coordinate je. By iterating over all cyclic order (modulo p) for l {l..... s), and then repeating over blocks, we obtain the full Enhanced Cyclic Coordinate Descent procedure.
For clarity, summarize the procedure in Algorithm omitting the full matrix derivation. The complete matrix-form representation and additional algorithmic variants are deferred to Appendix A.2
Theoretical Analysis
We first bound the approximation error and then show that it yields sub-linear time while preserving (mp) memory footprint, All proofs, including auxiliary lemmas and technical details, can be found
in Appendix A.3 A.4
A.5

---

Below, we state a simplified version of Theorem I which characterizes the leading-orde order approxima- tion error introduced by the Taylor-expanded updates.
Theorem (Taylor-Aprrxximate Update Error Bound), Let the true coordinate update at iteration
(k - 1)s + be given by A := 65 /55, and let A3 == $5.115 be the corresponding approximate update computed using first-order Taylor expansion. error is bounded as:
expartatt Then, the Taylor approximation
A3
A SCI(X,,,
+ S.
(14)
The denominator error is of order 0(143113) ): the overall update enror reusing the same W3, across block of s coordinates is controlled to O | ( preventing error accumulation across iterations.
Theorems 2 and demoontrate that, in comparison to the classical coordinate descent method, our enhanced scheme achieves reduced time complexity without incurring additional space overhead.
Theorem 2 (Time Complexity Analysis). Consider single epoch of the enhanced cyclic coordinate descent (ECCD) algorithm, defined as one full pass over all p features. Let C denote the compu- tational cost of evaluating the mean link function F.) once, so that applying
the full linear predictor X3 R" requires (nC)) operations. Then, by choosing the block size as s /C, the computational cost per epoch reduced from the standard O (mpC') to O)pvVC).
This complexity gain becomes particularly significant when C' large (i.c., when the cost of evaluating the link function is high; see Section D for further discussion). In the special case s = 1, ECCD reduces standard coordinate descent with complexity OnpC'). Therefore, the square-root choice achieves an asymptotical acceleration factor of O(VC). Empirically, moderate block sizes (e.g.
8 E {8, 16)) often provide favorable runtime/accuracy trade-offs on medium-scale datasets.
Theorem 3 (Space Complexity Analysis). Let s S min(n. p). The enhanced cyclic coordinate descent (ECCD) algorithm can be implemented with space complexity of (mp). which is dominated by the storage of the design matrix X E R=xp.
When the block size is chosen as 8 = VC which is optimal from a time complexity perspec- this remains substantially smaller than min(n, (2, p) for most practical settings, since C' denotes a constant representing the cost of evaluating the mean link function F'(-). As a result, the overall memory footprint of ECCD remains (mp)), matching that of classical coordinate descent methods.
6 Experiments
Unless otherwise noted, all experiments were executed on single Cray EX node using a single OS process; the only exception BIGLASSO, which was run multithreaded configuration with thread counts as reported in Tables 8 and 9 Each node was equipped with an AMD EPYC 7763 ("Milan"") CPU (64 physical cores, 2.45 GHz base frequency, 256 MiB L3 cache) and 512 GiB of DDR4-3200 memory. For scalability experiments on dense datasets (Table 5), we used an AMD) EPYC 7352 CPU (24 physical cores, 2.30 GHz, 128 MiB 1.3 cache).
Our implementation is based on R v4.3.1, utilizing the following key packages: glmnet v4.1-8, Matrix v1.7-1, stringr vl.5.1, ReppEigen v0.3.4.0.2, and Rcpp v..0.14. All experiments were performed in double precision, with machine epsilon Emach 2.220446 x 10
10-16 1116 on our platform.
Table 2: Relative differences of the objective function on duke.
Dataset &
Relative Differences A,
s=
8 16
$ 32
duke
4.06 10
4.15
10 8
8.12
10 8.90
10
2.74
10 10
0.2 1.84 x 10-8
1.93 x 10
1.05
10-7 2.30
10-7
1.96
10
0.5 1.23 10
1.56 10
7.31
10-7
1.18
10
1.18
10
0.8 3.50 x 10-9 1.19 x 10-8
5.86 x
10-7
2,51 x 10-6
2.51
x
10
1.0 2.99 x 10
2.12 x 10-7
4.78
10-7
4.78 x 10-7
4.78
10
7

---

Speeddu (x)
gimnet
(Ours)
novreg
biglasso
australia
diabetes
prostate
Figure 3:
Comparison runtimes across six benchmark datasets (a = 0.5). Speedups are normalized so that glmnet is 1.0x. ECCD (s 8) consistently achieves the highest acceleration on both datasets.
6.1 Numerical Experiments
We evaluated ECCD against classical cyclic coordinate descent solver for elastic-net regureuularrzzedd logistic regression on standard LIBSVM benchmarks (Table[15). Unless otherwise stated, all runs were initialized at the zero vector, O = O. ECCD employs an active-set strategy [2], strong- rule screening [8), and periodic Marush-Kuun-Tuucee (KKT) checks to prune inactive coordinates. Performance results are averaged over 100 independent trials. Details of the synthetic-dar generation protocol are provided in Appendix
Convergence on ECCD We evaluate ECCD on the duke dataset using the X-path implementation.
We compare block sizes E (2, 8, 16, 32) against the standard coordinate update 1), as shown in Table The differential maximum relative difference A, ||obj(s) obj( / (loj(( [1] across all and is 2.5 10-6, For mixing ratios a E {0.1, 0.2). A, 10-7. Thus, even for large blocks, ECCD preserves solution quality up to numerical precision.
Table 3: Relative differences of the objective function duke, comparing ECCD and BCD.
À = 0.1
À = 0.01
À = 0.001
Alg
Rel. Diff
Alg
Rel. Diff
Alg
Rel.
Diff
8
FCCD
1.14 x 10
ECCD
1.19 x 10-7
8
ECCD
1.69 10
BCD
7.50 10
BCD
1,28 x10
BCD
5.28 100
16
ECCD
5.38 x 10
16
ECCD
1.41 x 10-7
16
ECCD
9.92
10-7
BCD
1.27 10
BCD
5,53 x10
BCD
1.04 x10
32
ECCD
8.41 10
32
ECCD
1.26 10-7
32
ECCD
4.20 10
BCD
2.53 x10
BCD
1.03 10
BCD
5.79 x00
Stability of Optimized ECCD with Comparison to BCD Table shows that ECCD retains numerical stability across all block sizes and regularization strengths the largest relative discrepancy from the 1 objective is 9.9 10-? (at 16, À 0.001). In contrast, BCD's
remain small for = 0.1 but then explode once S 0.01 for s 2 8 (reaching on the order of 10 at À 0.01,
8) and diverge catastrophically at A 0.001 (on the order of 10 10" This pattern matches the theory: BCD's 's fixed-Hessian updates accumulate curvature approximation error in the denominator, which becomes ill-conditioned for small À and large s, whereas ECCD's Tayler-based correction term in the numerator controls that error and preserves stability.
6.2 Performance Experiments
Block Size Selection Strategy in Practice While our theoretical analysis suggests that the optimal block size is s* VC. practical implementation requires careful estimation of the hidden constant C. which captures the relative cost of link function evaluations versus basic linear algebra operations. Empirically, our profiling across datasets revealed that a slightly larger block size, particularly consistently delivered near-optimal speedups while preserving stability. Please refer to Appendix D for detailed analysis on the selection strategy of block size s.
Performance Experiment with Single Table demonstrates that tuning the block size yields noticeable benefits, especially high-dimensional cases (p >>> n). ECCD achieves up to 4.19x speedups glmnet duke, with comparable gains leu -5.37x) and colon-cances
8

---

Table 4: Logistic Peformance: Time (s), prediction error, and speedup under different a.
Fllenane Method
U.1
a-wBb 0.2
055
08
1,0
Time:
Rei. Dill
Specshep
Time
Rel. bail
Spacdep
Time
Rel Dilf
Spezilep
Tim:
Rel. Dill
Spectap
Time
Ret bifl Spunlop
phnnzt
6-400-11
1000
4770-00
1.00x
5:108-11
1.00
4 -9le-4
1,00x
485504
100%
neving
8:20e-11
5.5te-01
0.08
6.80-01
5.30o01
0.0x
5.800-01
5.33c-01
0.00%
5.5 5.55c-01
5:47011
0.09%
5.45-011
561:-011
0.89%
duke
liglaso
2:37e401
2.6e 01
0.2X
2 2224
01
1.9ted1
0.25%
2.26e-01
5.11e-12
0.33%
2.36 01
7.35e-03
0.31%
2:59e-01
8.75e-05
0.19%
Ours to 85
3.00-412
7:90--00
1888
400--0
6.77--06
133
1.74-412
5.25--00
1 1.55e-12
5.3e-000
3.16x
2:09-002
5:176-05
2.57%
Ouzs (a 16)
3.48-022
8.73 06
1.4K
248c-2
05
2
L.79-02
7.052 06
2
1
7.30e:06
3,49%
1.7le42
2.05-005
2.89
plmnet
3.06c-02
100%
2:30=002
1,0x
2.33c-02
100X
1.92--02
1,00x
2.1c002
L00K
nevrsg
7.4c.01
5.05: 01
0.0K
5.49:001
4.9ke-01
0.64
4.70e00
482:
0.05%
4.21c
5:00e:01
0.8%
3.85c-01
5.14-00
0.05
colon
liplaao
1.90-411
2.5 2.96--1
0.5
19
1.58-01
0.12x
1.56-01
5866-11
0.13x
1 188--1
4.77-64
0.10x
3.01001
8144-06
0.10x
Oure (x
3.10-02
38
0.9
2:30--2
4.55-05
1.05x
1.34-401
3.96-006
188
1.0c-02
4.30-06
1.90x
143c-02
6.17:-06
1.1x
3.15-41
2:12-10
0.4fe:05
140%
1.34e-03
3.00-100
6.9ee000
1,56x
1:5le-0
358
1.34x
plaeet
5836-02
100x
5.170-02
1.00x
4.3c-01
1.00
<.73-02
1.00x
4.50-02
1.0x
5.86-111
030X
4.02-00
5:7ee01
0.11
4.55e001
5:50e-01
0.1x
01
5:48e-01
0.12x
4:00-001
5:17e01
lighaso
2.34-411
3.2te-01
0.36
19e-01
2.31e01
0.36x
1.12-411
1M
0.33x
21Me-01
1.Me-001
0.35x
1.316001
1389--01
0.21
3.33g
214c
1542-005
1.5ee002
6.102
3.16x
9:33x
343x
1.97e
9.13;
0.
1.00
1877
2.49e
0.04e
2.08
1.5ee
9,49
3.16x
1.54e
3,01
1.40e
2
glnast
4355
1,00>
1.67-003
1,00x
1.36e-03
100x
nevreg
5.4e
2.54
5:42=
2.53e.01
082
3.36c
3,55e
001x
5.35e.01
3.62
0.01
166--01
89
1676-01
507002
1.16-402
1.36-08
0.6x
1.50-01
9.36--05
7.22e40
2162-08
2.25c-08
2.262-05
0.5le45
0.99
1.5g.04
8016441
1.56-08
7.40-01
1.56-04
plaast
7.54c403
7:13-00
100x
7,00-003
1.00x
7.06:40
100x
7.40-03
100%
noving
higlose
513--00
1.33-00
3.N6-04
to -
4376413
2176-05
1.53K
5.0le:0
6.94e00
1,37%
6.10e-03
2755-106
1.16x
1.33e:05
1.17%
6334e
2576-05
1.23
glmsst
7.4te412
100x
7.33e-12
1,00x
6:59e-01
1,0XX
6.33:
1,00x
6.
100K
nevreg
2.55e100
4.00 01
008
208e100
4335e:0
0.4%
1.8ee100
5055--01
1,74e100
50re-01
0.4x
1.0k-100
5115e-0l
014
prootate liplaao
3.26-001
196:-01
0.3x
3.00-01
1.3e00
0.34x
2.96-001
1.0c-002
0.22
2.4--01
1.19-63
0.3x
2.9k-01
2.0:-06
0.21
0.13c42
283s-0
0.2
7.09=022
166c-05
1.83x
5:20e-01
6.376-07
1.7X
4.53-022
1.92-66
1,4xx
4.34-02
2.5c-06
1.50x
Ours (4 = 16)
8.36-41
373--00
0.0
6:0-00
2.46-08
1.81
5.00600
1166-06
1.32
4.54-40
3.16-06
1.56x
4.00-02
3660-06
1.54x
(up to 6.34x). For sample-dominated (n >> p) scenarios, improvements are modest (2.xx-4.0x on diabetes-scall. and 2.8x-3.4X on australian), and ECCD is slower at larger À values for phishing, recovering to 2.19x at the smallest penalty. This highlights fundamental trade-off: block updates strongly accelerate computations in high-dimensional, strongly regularized problems, but yield smaller or even negative gains otherwise.
Performance Experiment with the X Path Table demonstrates that the ECCD outperforms baseline GLM colvers-includig the ofte-oo-thhe-nn glmmet-acros diverse benchmark datasets with single thread under default setups of g glmnet. ECCD consistently matches glmnet in predictive accuracy with relative f2-difference of objective values below 10-5 for every dataset across the regularization path. In terms of runtime efficiency, ECCD) demonstrates improvements, notably attaining up to 3.5x speedup over glmnet. For the lower dimensional benchmarks (e.g., australia i and di tiabete), our method shows consistent gains of 1.1-1.5xx The best speedup (3.49x) ) is observed with block size s = 16 at a = 0.8 in duke dataset. Notably, in the regularization-pee path setting, the combination of the strong rule and warm starts substantially accelerates convergence 2 which makes our relative speedup appear less pronounced than in the full-model scenario.
Scalability We evaluate GLMNET and ECCD under identical, single- settings. 3 We fix tol = 1e-7, precompute length- 100 À path in R, and pass the same sequence to both solvers with Aimin/Amas = 0.01 when n < p and 10 otherwise (nlambda = 100, 1 lmmdda. min. ratio ifelse (nobs < nvars, 0.01, 1e-4)). Inputs are standardized in R and in-solver stan- dardization is disabled (standardize = FALSE); for GLMNET we use type logistic = "modified-newtae" " with intercept TRUE. Timing is performed on Cray cluster across three regimes (n < n.n. > p. and n = p). ECCD additionally reports the selected block size s (Table
Under matched tolerance, À path, and preprocessing, ECCD is consistently faster than GLMNET across all regimes (Table 5). The largest relative gains occur in the wide setting (n > p), where strong-rule screening plus periodic KKT checks quickly concentrate work on a small active set and block updates improve cache locality; here speedups peak around 8x mid-size problems and remain above 2x even at the largest tall instance (1 000 000 x 5000). For square problems (n = p). improvements are steady at roughly 3-4x across sizes up 50 000 50 000. When p > n. speedups are more modest (about 1. 1.1-1.9x), reflecting slower active-set contraction when many features are weakly informative. In absolute terms, ECCD can reduce wall-clock time from minutes tens of
All runs pinned to one thread eliminate BLAS/OpenMP variability.

---

Table Wall-clock time across regimes. ECCD uses the indicated block size s. Speedup GLMNET time ECCD time.
Regime
Datasel (n1 x p)
Time (s)
Speedup (x)
GLMNET
ECCD
P
100 1 000 000
6.348
3.330
16
1.91
n < p
500 x 1 1000 000
133.606
113.281
1.18
< P
500 10000 000
389.342
346.456
32
1.12
n n<p
10000 x 50 000
127.511
83.490
1.53
P
10000 x 50
0.193
0.093
2.07
n>1
10000 x 500
3.776
1.080
3.50
n n>p P
100000 X 5000
158.968
19.653
8.09
n> P
1 000 5000
2082.622
1027.311
2.03
=p P
5000 x 5000
35.061
8.171
4.29
n =
10000 x 10000
126.171
30.896
4.08
n = P
200000 x 20000
661.032
213.412
3.10
P
40000 X 40 000
3369.816
986.692
3.42
n = P
50000 x 00000
4956.082
1466.968
3.38
seconds mid-size wide data (e.g., 10000 5000), while using small block sizes (s {4,8, 16, 32)) that balance per-iteration cost and parallel efficiency.
Core memory results. Empirically, the measurements align with Theorem 3 with the active-set cap. ECCD's residual memory beyond storing X is small and essentially insensitive to the block size the square case n=p=5000 (Table 2T), raising up to 4096 increases peak resident set size (RSS) by only ~ 39 MiB over baseline and then plateaus, consistent with the warm-start cap = VIAl In wide (n=100, p=100,000) and tall (n=100,000, p=100) regimes (Tables 22 23 peak RSS is effectively flat (variation X 40 MiB), indicating that neither ((ns) nor O(s terms dominate. By contrast, removing the cap exposes the anticipated Oss²) growth: for n=10, 50000,
peak RSS increases by ~ 129 MiB reaches 4096 (Table 24). Overall, the overhead scales as (s/(np)) s/p in high dimensions which is negligible under the capping rule.
Conclusion
In summary, ECCD batches multiple coordinate descent updates with tunable block size and Taylor correction, which cuts per-epoch cost without affecting convergence. Our analysis shows that setting x* = yields an ((VC) speedup, and experiments demonstrate up to 3x gains on logistic and Poisson tasks. ECCD integrates warm-start and screening rules, matches leading solvers in accuracy, and naturally extends to other GLMs, structured penalties, and distributed or asynchronous settings.
8 Limitations
While ECCD achieves strong results, selecting an appropriate block size remains nontrivial: perfor- mance is robust across a range of s, but the exact optimum is hard to pinpoint.
9
Acknowledgment
This work was supported by the US Department of Energy, Office of Science, Advanced Scientific Computing Research program under award DE-SC20023 This research used resources of the National Energy Research Scientific Computing Center (NERSC), U.S. Department of Energy Office of Science User Facility located at Lawrence Berkeley National Laboratory, operated under Contract No. DE-ACO2-05CHII 1231 using NERSC award ASCR-ERCAP0024 This work also used computational resources provided by the Wake Forest University (WFU) High Performance Computing Facility for testing and debugging.
10

---

References
[1] Hui Zou and Trevor Hastie. Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301-320, 2005.
[2] Jerome Friedman, Trevor Hastie, and Rob Tibshirani. Regularization paths for generalized linear models via coordinate descent. Journal of statistical software, 33(1):1, 2010.
[3] Patrick Breheny and Jian Huang. Coordinate descent algorithms for nonconvex penalized regression, with applications to biological feature selection. Annals of Applied Statistics, 5(1):232-253, 2011.
[4] Yaohui Zeng and Patrick Breheny. The biglasso Package: A Memory- and Computation- Efficient Solver for Lasso Model Fitting with Big Data in R. The R Journal, 12(2):6-19, 2021.
[5] Nathaniel E. Helwig. grpnet: Group Elastic Net Regularized GLMs and GAMs, 2025. R package version 1.0.
[6] James Yang and Trevor Hastie. fast and scalable pathwise-solver for group lasso and elastic net penalized regression via block-coordinate descent. arXiv preprini urXiv:2405.0663), 2024.
[7] Laurent El Ghaoui, Vivian Viallon, and Tarek Rabbani. Safe feature elimination for the lasso and sparse supervised learning problems. arXiv preprint arXiv:1009.42119 2010.
[8] Robert Tibshirani, Jacob Bien, Jerome Friedman, Trevor Hastie, Noah Simon, Jonathan Taylor, and Ryan J Tibshirani. Strong rules for discarding predictors in lasso-type problems. Journal of the Royal Statistical Society Series B: Statistical Methodology, 74(2):245-266) 2012.
[9] A. T. Chronopoulos. s-step iterative methods for (ymmerymmttric (in)definite linear systems. SIAM Journal on Numerical Analysis, 286))1776-1189, 1991.
[10] Anthony Chronopoulos. class of parallel iterative methods implemented on multiprocessors. PhD thesis, University of Illinois at Otbana-Champaign, 1987.
[11] Anthony T. Chronopoulos and C. William Gear. On the efficient implementation of precondi- tioned s-step conjugate gradient methods multiprocessors with memory hierarchy. Parallel computing, 11(1):37-53, 1989. Citation Key: chronopoulos89.
[12] SK Kim and AT Chronopoulos. An efficient nonsymmetric lanezos method parallel vector computers. Journal of Computational and Applied Mathematics, 42(3):357-374, 1992.
[13] James Demmel, Mark Hoemmen, Marghoob Mohiyuddin. and Katherine Yelick. Avoiding communication in sparse matrix computations. In 2008 IEEE International Symposium on Parallel and Distributed Processing, pages 1-12, Miami, FL, USA, 2008. IEEE.
[14] Mark Hoemmen. Communication-avvidiiie Krylov subspace methods. PhD thesis, University of California, Berkeley, 2010.
[15] Erin C Carson. The adaptive s-step conjugate gradient method. SIAM Journal Matrix Analysis and Applications, 39(3):1318 1138, 2018.
[16] Erin Carson. Communication-avvidiii Krylov subspace methods in theory and practice, PhD thesis, EECS Department, University of California, Berkeley, August 2015.
[17] E. Carson, N. Knight, and Demmel. Avoiding communication in nonsymmetric basezd krylov subspace methods. SIAM Journal on Scientific Computing, 35(5): S61, 2013.
[18] Samuel Williams, Mike Lijewski, Ann Almgren, Brian Van Straalen, Erin Carson, Nicholas Knight, and James Demmel. s-step krylov subspace methods as bottom solvers for geometric multigrid. In 2014 IEEE 28th International Parallel and Distributed Processing Symposium, pages 1149-11 1158, Piscataway, NJ, USA, 2014. IEEE

---

[19] Saeed Soori, Aditya Devarakonda, Zachary Blanco, James Demmel, Mert Gurbuzbalaban, and Maryam Mehri Dehnavi. Reducing communication in proximal newton methods for sparse least squares problems. In Proceedings of the 47th International Conference on Parallel Processing, ICPP '18, New York, NY, USA, 2018. Association for Computing Machinery.
[20] A. Devarakonda. Avoiding Communication in First Order Methods for Optimization. PhD thesis, UC Berkeley, 2018. ProQuest ID: Devarakonda, berkeley 0028E 18070. Merritt ID: ark:/13030m5662133
[21] Aditya Devarakonda, Kimon Fountoulakis, James Demmel, and Michael W. Mahoney, Avoid- ing synchronization in first-order methods for sparse convex optimization. In 2018 IEEE International Purallel and Distributed Processing Symposium (IPDPS), pages 409-418, 2018.
[22] Aditya Devarakonda, Kimon Fountoulakis, James Demmel, and Michael W. Mahoney. Avoiding synchronization in first-order methods for sparse convex optimization. In 2018 IEEE Interna- tional Parallel and Distributed Processing Symposium (IPDPS), pages 409-418 Piscataway,
NJ, USA, 2018. IEEE.
[23] Aditya Devarakonda and James Demmel. Avoiding communication in logistic regression. In 2020 IEEE 27th International Conference on High Performance Computing, Data, and Analytics (HiPC), pages 91-100, Piscataway, NJ, USA, 2020. IEEE.
[24] Zeyuan Allen Zhu, Weizhu Chen, Gang Wang, Chenguang Zhu, and Zheng Chen. P-packSVM. Parallel primal gradient descent kernel svm. In Proceedings of the 9th IEEE international conference on data mining, ICDM '09, page 677-686, Piscataway, NJ, USA, 2009. IEEE Computer Society.
[25] Peter Richtárik and Martin Takáč. Iteration complexity of randomized block-coordinate descent methods for minimizing composite function. Mathematical Programming, 144(1):1-38, 2014.
[26] Yu Nesterov. Efficiency of coordinate descent methods on huge-scale optimization problems. SIAM Journal on Optimization, 22(2):341-33) 2012.
[27] S. Lee, H. Lee, P. Abbeel, and A. Ng. Efficient 11 logistic regression. In Proceedings of the 21st National Conference Artificial Intelligence (AAAI), pages 401-408, Boston, MA, USA, 2006.
[28] Tyler Johnson and Carlos Guestrin. Blitz: A principled meta-algorithm for scaling sparse optimization. In International Conference on Machine Learning, pages 1171-1179. PMLR, 2015.
[29] Mathurin Massias, Alexandre Gramfort, and Joseph Salmon. Celer: a fast solver for the lasso with dual extrapolation. International Conference on Machine Learning, pages 3315-3322 PMLR, 2018
[30] Yaohui Zeng and Patrick Breheny. The biglasso package: A memory-and computatiin-eefficeentt solver for lasso model fitting with big data in r. arXiv preprint arXiv: (.K1.0/700.093666 2017.
[31] Patrick Breheny and Jian Huang. Group descent algorithms for nonconvex penalized linear and logistic regression models with grouped predictors. Statistics and computing, 25(2):173-187, 2015.
[32] Thomas Moreau, Mathurin Massias, Alexandre Gramfort, Pierre Ablin, Pntrre-Antoine Bannier, Benjamin Charlier, Mathieu Dagréou, Tom Dupre Tour, Ghislain Durif, Cassio F Dantas, et al. Benchopt: Reproducible, efficient and collaborative optimization benchmarks. Advances in Neural Information Processing Systems, 35:25404 25421, 2022.
[33] Chih- Chung Chang and Chih-Jen Lin. LIBSVM: library for support vector machines. ACM Transactions on Intelligent Systems and Technology, 2:27:1-27:27, 2011.
12

---

A Theoretical Analysis and Matrix Derivations
A.1 Update Rule for Block Coordinate Descent (BCD)
We set the block size equal to 8, where S p. Then the objective function t-th iteration is given by:
LC ((((ok)))))
1 ((B(1) + V (blooc) (Blosock)
-3(() (black))
+ (P(book)) B (block)) H ((bloo)) (. (O(lloc)) (chock))
+ 2 2 || 112 + ax |11|3+/00ock Il.)
(15)
where V (blockk) is the gradient of the loss function - E(3) with respect to ((btock) at 3(t), While H (bbock) is the second-order derivative (block of the Hessian matrix).
From the definitions of the first-order derivative and the second-order derivative in we can express the gradient vector V (block) and the Hessian matrix H (bboock) as follows:
91s
Fi
9iz
(y (O)))
Ti.ja
V (block)
(16)
d(r)
95
x i
0
a
(0.)
H ((block)
(17)
d(T)
To do block coordinate descend, ,ee take the derivative of objective function with respect to ((ttock))
we obtain:
OL (()oc)))
1
V (block)) + ((block) (B(Mkock)
B(0) (black))
0 (Bock)
n
A(1 () + aa
(18)
Setting the gradient with respect to block B(block) to zero and solving for the update rule:
(block)
(H(btock)) + A(1 - -a- (bicc)))))
(19)
((t) L_ V ((block)) H ((btook) (block) >a sign sign( (stcc)) )
(20)
Then, for each coordinate j in this block, the update rule is:
B0411 5( s(50) 1 1800 2a
(21)
n h3 + A(1 a1
The biggest difference from the coordinate descent (CD) method is that the parameter A; in both gj and hy is only updated once per block update, rather than at every coordinate update.
A.2 Matrix Form Representation
In this section, we illustrate the Enhanced Cyclic Coordinate Descent updates using a matrix- based formulation. We begin by introducing notation for the coordinate roordinate-indexing scheme, the relevant design submatrices, and the first- and second-order derivative vectors:
II, = (e 1)s+ e(k-1)sth 1)s+1, .... eko-1) E RPx*
(22)
X T[ = (X ((k-1)er X (((k-1)ott:... ..., X Xess-1) Raxs
(23)
VF = F'(8o X x1g((t111 1)8) Baxl
(24)
2 F = F"(oo + -X ((A-11))) E Bax
(25)
13

---

We first update the intercept term So by subtracting the mean difference between y and the current predicted mean FOD. Specifically,
TT VF)
Bo - Bo
(26)
1 V2F
We next define the following quantities for the weighted linear regression associated with the current block of coordinates:
(xI,) VF E Rex1 B= (X diag( (XT.)) E RR*
(27)
where diag(V22)) the diagonal matrix formed by the second-order derivative vector V2 For the l-th update within the current cycle (0 I & S 8), the numerator of (g(k-1)ete (i.e., the update to the ((k- 1)s + E)-th coordinate) is given by
(-1
y' X 1)-4E nd(r) Ae n n((T)
Σ 2=0
Be Agtti A(1 a) 1) ((k-1))
(28)
n d(())
and the denominator is
1 n d(()) Bee A(1 a).
(29)
To incorporate the fi-penalyy, we apply a softtthresholding step:
Snumeralor, Xx)
1/4 (k--))
(30)
denominator
where -(::)) the softtthreholdinng operator. This completes the matrix-based update for the ((k: - 1)8 + E)-th coordinate in the ECCD procedure. We now summarize the matrix-based updates described above into a formal algorithm (Algorithm 5) for solving GL.Ms with elastic net penalty at fixed regularization level X. In particular, the block-wise coefficient updates correspond to the matrix formulation derived in this Section.
More detailed algorithmic variants, including full matrix-based block update procedure, are provided in Algorithm I addition, we implement pathwise regularization following and summarize the full procedure in Algorithm 6 All settings and subroutines follow established conventions in prior work, and are carefully adapted to fit within the proposed ECCD framework.
A.3 Proof of Theorem
Theorem I (Taylor-Approximaat Update Error Bound). Let the true coordinate update at iteration (k 1)s +l, which corresponds to the éth coordinate in the kth block, be defined as A 05./Vie- The corresponding approximate update, computed using Taylor expansion, is given by A3 = /b,
(A1) Let 0max == || los denote the maximum coordinate update within the block.
(A2) The coefficients are bounded as |8yl < B, for all coordinate j. which holds under regular- ization.
(A3) The residual vector y (k- 1)a+e = F' (z - y satisfies (x((-1))+4 S R.
Then, the Taylor approximation error is bounded as:
M3 0max Loc
AB Artr
<
(31)
v12 min d(T))
R
M2 d(r)
+ A(1 (a)-aa) Vrl 0max Loo
+ +11 - -BBBBBB+
+1(1-a))
(32)
Vri(T)
14

---

Proof. For notational simplicity, we write j in place of je. unless otherwise stated. Since the (with Mmnin X(1 o)) as valid lower bound), we
denominator terms satisfy V3 bh 2 timin begin with:
Φ
Φ
A3j
(33)
10j
4/
1
1 + +0; 5.
(34)
f/i
(1051
180g 1
1/51 + 10b - (5.1
(35)
where
0j nd(T) X.J (F(2(t-11)) 1)07 + (z(h 00 - -A(1 - a)) -(1--)),ft
(36)
क5 nd(T)
) X.) F" (F"(x( -1) -A(1 -a),s(
(37)
33 = nd(T)
X. diag (2((2()))))) X., + +(1 -a),
(38)
5, nd(())
1
X. diag ((((t(t-)) X. + 1(1-a),
(39)
where (1) X 1))
We now estimate each component. By Taylor expansion, we have: F((z(t-I)ote) = (1(((-11))) + (x-(1))))) o 8 + R. where Vi E (1,2,.... n). Rd = (E))0 ,for some J, and EiD Agtti. X.
& E [2g
(111)) =i (t-1))++),,
1)s
For all updated coordinates j. we use Loe to denote the infinity norm bound of each column of X, i.e., || & Loo. Then for z in (k- 1) + s-th update, by using Assumption (A2) we obtain the
bound:
p
(max (m fJ
||z (k- 1)s 11000 = =11_a
F.j
1//h 1 10 S
11((h12)1112
(40)
j=l
S Tooo 1|1(k=1)0112 S L.opB.
(41)
Therefore, we assume the function F(-) determined by link function in the GLM is three-times differentiable, with [F(3) (z)l < remainder:
Ma and [=)( < M2 for all z. Next, we bound the Taylor
M2
M2
| S M3. 2
R
Σ
84
||51
|8112
||81|2 100 (42)
4
From the definition of and by using Assumption
(AI) we have:
|1
A A
|| J. Iloo S se. 2.-0mxx Loo
(43)
||01|2 5
Σ
A/ghti. || se.
1--00ax 0max vn.
(44)
Therefore:
M3 |R]22 <
L 0max Vn. E. m Loo = M3 l B vn. Loo.
(45)
2
2
15

---

Since the data has been standardized so that || 1l2 = Vn, it follows that:
1
R I
nd(r) ||x 1112 || S
1
M3
l -6maxx
Loo.
(46)
nd(1) r)
2d(r)
Next, for the denominator difference 1:5 - 1ly1 we again use the fact that the data has been standard- ized:
16) U5 ndd(T
1. (zi (=((=11)+t)) (z) (z!h-110e)
xzg =
M3 nd((T)
Σ i=1
16a1
-2
M3
Ms d(T)
|101|00
8. 2-0max Loo.
(47)
d(T)
Moreover, applying Assumptions (A2) and (A3) we obtain:
18,1 = rud(T) x.
[nd (F (2( (F y) A(1 - a).
R (Td(T)
+ x(1 - a)B.
(48)
And:
M2
|//1 s d(r)
+ A(1 - aa).
(49)
Putting all together:
R +11 +X(L-)BB
Mr t- 6max Loo
A
A Aggu g
S Uim
Vnd(r)
d(T)
[M2
M2
+ +1(1- -a)) +x(1-a)
M3
l2 B vn- Ls0)
+
(50)
d(T))
2d((T)
M3 L- 8max Lix
R
1
M2 X(1 a(l-a)) -i)) .Vnl mmax Loo
X(1 a
+
22 d(T)
Vri((T)
2
d(r)
(51)
This completes the proof.
A.4 Proof of Theorem
Theorem (Time Complexity Analysis) Consider single epoch of the enhanced cyclic coordinate descent (ECCD) algorithm, defined as one full pass over all features. Let C denote the compu- tational cost of evaluating the mean link function FF(() once, that applying predictor B e R" requires (nC') operations. Then, by choosing the block size computational cost per epoch is reduced from the standard OmppC)) to OnpyVC).
the full linear
VC, the
Proof. During epoch the algorithm processes the p coordinates in [p/s] blocks of size at most For each block we (i) extract an n x s slice X, (ii) update the working response F + 30). and (iii) solve tiny weighted least-squares system of dimension
First we calculate the cost per block. By extracting the slice and forming the weighted Gram quantities requires ((ns) and O(ns²) operations, respectively; the latter dominates when s > 1. Recomputing or refreshing the V and 2 link function contributes further (nC)). Hence,, single block incurs AnC + ns²) work.
Aggregating over all blocks, we multiply by the number of blocks to obtain the per-epoch cost:
( + ns²])
nppC
+ Anps).
Applying the arithnetic-goometric mean inequality to the two terms, we have
npC + nps 2 2npVC.
(52)
16

---

This lower bound is attained when s" = VC, yielding the minimal total cost (npyvC).
Comparison with the one-at-a-time update. When 1 the second term collapses to (mp), leaving the classical (npC') bound. Thus, the square-root choice reduces the asymptotic cost by factor of vC, an advantage that widens as the link evaluation becomes more expensive.
A.5 Proof of Theorem
3
Theorem Space Complexity Analysis) Let I min(n, p}. The enhanced eyclic coordinate descent (ECCD) algorithm can be implemented with a space complexity of (mp)), which is dominated by the storage of the design matrix X "K=
Proof. The principal memory allocation is the dense design matrix X, whose cost is (mi)) and, as we shall see, dominates all subsidiary structures. The response vector y R and the current iterate B E Rº contribute only (n) and Op)), respectively. At each sweep the algorithm selects an index set of size and manipulates the corresponding indicator matrix L,, E IRxx(s) Because I. is stored as an index list rather than full column slice, its cost merely (s)) Multiplying X by I, materialises an n X submatrix; even if this object is formed explicitly (the pessimistic scenario), it incurs O (ns) additional space. Element wise logistic weights such as (-) and F-) are cached as length-n vectors, yielding another On)) Finally, the blockwise normal equations manipulate a tiny x Gram matrix together with its right-hand side, which together add (ss²).
Collecting the above, the aggregate per-epoch requirement is
((p)) + +()) + Opp) + O(ns) + O(s²).
(53)
Because & < min(n,,]; we have both ns S np and s2 S np, rendering every term other than the first negligible in the asymptotic sense. Consequently, the complexity of the working memory of the ECCD routine (np)) no additional allocation beyond the data matrix alters the leading order. D
B
Algorithm
B.1
Deviance Computation and Stopping Rules
Deviance is likelihood hheeihood-based measure used to quantify the goodness-of-fit in generalized linear models. Formally, it is defined as:
Deviance = - ((
Esal -
where ((33 is the log- likellhodd under the fitted model, and laa is the log lig-liielhood of a saturated model (which perfectly fits the data). The null deviance is similarly defined by replacing the fitted model with the intercept-only model. We present the specific formulas for logistic regression and Poisson regression.
Logistic Regression. For binary responses y E {0, 1} and predicted probabilities p; ( (3), the log-likelihood is:
((3)
Ui log Pi + (1 log(1 p.)]
(54)
The saturated log-likelihood is:
fast
yy log y: + +(1 y:) log(1 - -y)]
(55)
with the convention that 0 log 0 == 0. The log- likelihoodd intercept only model uses T.
Euull
lys log + (1 - y4) log(1 7)
Σ
Vi-
(56)
im
17

---

Algorithm 2 Computation of Null Deviance for Generalized Linear Models
Require: y E ER, GLM family, F().d(1)
1: y f- ->n 1= Vs 0g + VF-1(g (1)
2: Culll
[t 0g - [(0)] /d(T)
3: last f- -i=1
1=1 M B5 =( ] /d(r)
where F"((ttt) yi
4: NullDeviance < -2 (faull sal)
5: return NullDeevance
Algorithm 3 Deviance-Based Block Convergence Check
Require: Current ( previous ((1-1), F(.), X, tol, block sizeblz, dev_ 0 computed using
Algorithm 2
Au $ Po 1 La (
F2F ((0))
Aù: Levet
2: for each active coordinate j do
3: A, (800) -3/ 3
Inj (E; F x.! ((0) A}: Level max ( (Iroeet In}
4: end for
5: if Lo < tol dev, or Levet < tol devy blz then
6: return Converged
7: else
8: return Not Converged
9: end if
Poisson Regression. For count data y; E N and = exp(2; B), the log-likelihood is:
(33)
Us log b4 14 1k--lon log ya!]
(57)
The saturated le-likkllihod is:
Sat
[ys log y: - log ys!]
(58)
and log-likelihood of intercept only model uses overall mean y.
fuull Σ 1y log ū log vr!
(59)
The computation of the null deviance for generalized linear models is detailed in Algorithm 2 with all notation consistent with that used throughout this paper.
For the stopping rule, we follow 8 and monitor the maximum change in deviance, normalized by the null deviance. For our ECCD algorithm, further incorporate the block size into the threshold, in order to account for the variation in deviance magnitude induced by different block sizes.
The logelikel is approximated via second-order Taylor expansion, which allows us to reuse the first- and corond-order quantities already computed during optimization. The detailed procedure is shown in Algorithm
B.2 Baseline: Cyelic Coordinate Descent for GLMs with Elastic Net
As a baseline, we implement the classical cyclic coordinate descent (CCD) algorithm under the generalized linear model (GLM) framework with elastic net regularization, as proposed by Friedman et al. 22. The algorithm performs coordinate-wise updates in cyclic fashion, equivalent to applying FCCD with fixed block size of one, minimizing a penalized approximation the negative log- likelihood in each step. The full procedure is summarized in Algorithm 4
18

---

Algorithm 4 Cyclic Coordinate Descent for GLMs with Elastic Net Penalty (Baseline)
Require: X E R E YER, 300 E RR, _ E R. X a 10. H E N, tol > 0,
F(.)
1: for 1 H do
2: 7 + (1111) (B VF VF()). PF+WVFF
3: (Intercept update):
(y VF) 1, V2F
0 + P0
4: (Coordinate updates):
5: for k 0, .....P p 1 do
6: 2h f Xexi A t T4
VF: t Ik F
S ( Ib A B Aa
(1)
n
7:
B 1 σ))
8: end for
9: (Convergence check): Apply Algorithm
10: if converged, break
11: end for
12: Output: 3(0, (t))
B.3 ECCD for GLMs: Single-Step and Pathwise Algorithms
In this section, we introduce our proposed Enhanced Cyclic Coordinate Descent (ECCD) algorithm for solving generalized linear models (GLMs) with elastic net regularization. We present three variants:
Algorithm implements ECCD with fixed active set.
Algorithm solves the penalized GLM for single regularization level.
Algorithm extends ECCD to a pathwise setting with sequential strong rules for active set screening.
Our method improves computational efficiency by grouping updates in blocks and reusing gradi- ent/Hessian terms via second-order Taylor expansion. minor but effective optimization we adopt to only recompute the log-likelihood gradient, FF, when there is non-zero update in 3, avoiding unnecessary recomputation epochs with no coefficient changes.
For other standard components such as warm-start initialization, early stopping criteria, and X-grid construction, we closely follow established practices in the paper
19

---

Algorithm 5 ECCD With Single À
1: // STRONG-RULE ACTIVE SET
2: COMPUTE qo. Bo. A, null deviance
3: SET & x
4: while Not Converged do
5: Reset epoch loss, has_inc
6: for all Block B C do
7: if has
First Block then p (== + Bo)
8:
with_intercen then update 5g Update 3B (Apply Algorithm
9:
10:
+ c-hange in BB, 2 + + Xgδ; Update has inc, epoch_ loss end for
11:
12:
if First Epoch or Apply Algorithm then
13:
Check the KKT condition and update A (see Section 3.3 and Algorithm 6 for details). No Change in AA then Break 16: end if
14:
15:
end if
18:
end while
19:
RETURN B, 80
Algorithm 6 Pathwise Enhanced Cyclic Coordinate Descent (ECCD) with Sequential Strong Rule
Require: X E R=xp, y R g(0) E R.P o
R, {)1 x2 > Aq);
10. 1]. 8, H E N, tol, ne nemlimit, raq_max, sml > O, F", F" for log- null deviance devu-
Ensure: Estimated 3,30 and deviance for each A
1: Set gradient: r + VF(3))) X grad
grad
2: Compute: \ + max,
=
3: for k ......g do
4: Run Algorithm] with block updates on current active set As, reusing (3, Bo) and A + A from previous A (warm start).
5: for outer = 1...., H do
6: (Strong Rule Screening):
S - [j:1x](x VF(1)) 2 a(2>x - Ax-1)}
7: Let A₁ + Sz Ufj B, *0} 0}
8: Run Algorithm I on block updates with predictors in A4. current À Ak
9:
(KKT Violation Check): For & Ag. add back if:
xx] (y VF + Bo)) > ad
Update A& and continue outer loop if any violations
11: end for
Compute deviance:
deva + -2 (2((),3)) lsa))
13: Optional Early Stop if:
(c) dev devi
(a) AA nemlimit,
(b) R² rsq_max,
aml
2 devo
14: Warm start next A&+1 using current (B,30)
end for
16: return B, So. [deviance]
20

---

Table 6: Runtimes on GLMNET default path (Amax + max x10 or 10-2; 100 points). ECCD single-thread; BIGLASSO uses the listed thread count.
Dataset
GLMNET [s]
ECCD [s]
8
BIGLASSO [s] (threads)
Duke
0.117
0.012
0.215(8)
10,000 x 10,000
44.204
31.644
8
122. 114 (64)
100 x 100,000
0,937
0,640
4
1.198 16)
100,000 x 100
1.862
0.767
8
4.372(2)
Table 7: Runtimes on BIGLASSO default path (Amax 7 Amax x 10-1; 100 points). single-thread: BIGLASSO uses the listed thread count.
ECCD is
Dataset
GLMNET [s]
ECCD [s]
BIGLASSO [s] (threads)
Duke
0.076
0.010
8
0.197(8)
10,000 x 10,000
4.115
2.250
4
1.419(64)
100 100,000
0.942
0.709
4
0.890(8)
100,000 100
1.033
0.484
16
1.519(64)
C Benchmarking Experiments with Additional Baselines
While the most direct baseline the ECCD the GLMNet that explore path-wis, solution with single-thread focus. There are many popular softwares that involves solving the LASSO, ElasticNet problem for regularized logistic problems, despite differences solution type (i.e. single lambda fit), convergence criterion 28, 29], and system-level design (30 4 Yet we still select some of the method
serves as baselines for comparison because we expect the performance be consistent across various of domains. Below we perform intensive comparison with other baselines:
C.1 BigLasso
is a parallel coordinate descent solver that targets the same objective and design GLMNET; in the single-thread limit it reproduces GLMNET's solution. Motivated by the reviewer's 's suggestion, we benchmarked BIGLASSO on a Cray EX cluster using {2, 4, 8, 16, 32, 64) threads and compared against single-thread ECCD. We report two settings: (i) GLMNET's default path (Amax + Amax x 10-4 or 10-2, 100 points), where ECCD remains faster than BIGLASSO when the latter
BIGLASSO 31 30
up 64 threads (Table 6): and (ii) BIGLASSO's shorter default path (Amax + Amax x 10- 100 points), where ECCD is still faster on three of four datasets while BIGLASSO wins on the 10k 10k case (Table 7 These results indicate that ECCD's pathwise, active-set strategy provides a strong sequential baseline; parallelizing ECCD is straightforward future work and would further widen the gap.
C.2 SKGLM
We include a comparison against SKGLM, a Python library that provides pathwise coordinate descent primarily for linear models and exposes logistic regression as single-X solver the time of our experiments. Because path solver for logistic regression is not available in SKGLM, a direct path-to-path comparison with ECCD is not possible. To enable controlled evaluation, we reuse the X-sequence generated for ECCD and invoke SKGLM's 's logistic solver at those values, matching preprocessing (standardization) and error tolerances. Runs are conducted under the same BenchOpt protocol in Section C.4 wall-clock time is reported.
Table 10 summarizes the results. ECCD attains substantially lower runtimes across all datasets. On Duke, ECCD completes in 0.013 versus 1.375 s for SKGLM (=106x faster); on Diabetes the gap widens to 0.002 versus 6.155 (==3078x). The advantage persists in large synthetic regimes: for tall design (n= 100, p=100,000) ECCD achieves 1.426s versus 131.340s (#=2xx;; for wide design (n=100,000 00,000,,00000000 p=100), 1.032 versus 213.012s (x= These findings are consistent with the expected behavior of pathwise methods: ECCD leverages warm starts, strong rules, and block updates with active-set screening to amortize work along the sequence, whereas repeatedly solving single points lacks these path-specific accelerations.
21

---

Table 8:
Table A
BigLasso sweep grouped as 1/2/4/8 and 16/32/64/128. Times in seconds.
BIGLASSO (threads)
BIGLASSO (threads)
Dataset
GLMNet
ECCD
1/21/8
16/32/61/128
Duke
1.375
0.012
0.330 / 0.265 0.226 0.215
0.221 /0.228/ 0.277 /0.293
10k x 10k
43.807
32.477
127.226
123.642 123.034/ 122.464
123.462 126.189 124.638 123.583
100 x 100k
131.340
0.640
2.919 1.555 1.340/ 1.245
1.198 1.253 / 1.306/ 1.251
100k 100
213.012
0.767
4.501 /43372 4.415 4.518
4,822 4,995 4,928 5.038
Table 9: Table B
Baglasso sweep grouped as 1/2/4/8 and 16/32/64/128. Times seconds.
BIGLASSO (threads)
BIGLASSO (threads)
Dataset
GLMNet
ECCD
1/2//8
16/32/64/128
Duke
0.076
0.010
0.230 / 0.289 / 0.203 0.19
0.206 0.211 /0.215 0.280
10k x 10k
4.196
2.263
9.944 / 5,557 3.176 2.287
2,017 2.119/ 1.902 / 1.729
x 100k
0.942
0.709
2.923 / 1.281 / 0.980 0.890
0.913/0.922 /0997 / 1.011
100k 100
1.033
0.484
2.066 1.744/ 1.553/ 1.519
1.632 1.778 1815 1.659
Taken together, the results indicate that ECCD provides strong sequential baseline for logistic regression paths-even against highly optimized Python implementations, while remaining competitive individual À values We emphasize that this comparison isolates the benefit of path-aware computations: SKGLM is competitive for single-point fits, but without logistic path solver it incurs near-linear cost the number of À values.
C.3
ABESS
We also compare against ABESS, which solves best-subset selection with an explicit to constraint-a different objective from the f1/fl-regularized problems (Lasso/Elastic Net) considered here-so direct pathwise equivalence does not hold. Nevertheless, using the correlated synthetic generator from Section D with p {0.1, 0.9}. we benchmark small (n=20, p= 2000), moderate (n=1000, p=1000; n=1000, p0 100), and large (n= 100,000,0000 p= p=100) regimes. Table shows that ECCD consistently fastest across settings, including high-correlation cases; ABESS slows substantially as grows, while ECCD maintains sub-second to single-second times under identical preprocessing and tolerance. These findings reinforce that ECCD retains its advantage even in regimes where b--based methods are often promoted.
C.4 More Experiment on Skglm Family with BenchOPT for Comparison
Recently the BenchOpt (32 emerges as an strong baseline for generalized linear models with convex/nonconvex regularization. Thus we augmented our baselines with the skgim family-SKGLM CELER, and BLITZI I BenchOpt. These packages are strong sparse estimators at single regularization level but differ from ECCD/GLMNET in two substantive respects: (i) they are primarily engineered for single-point fitting rather than full regularization paths; and (ii) their model coverage differs (e.g., Elastic Net and/or GLM path solvers are not uniformly exposed). To ensure fairness and reproducibility, we therefore report results in two regimes: single-A versus sequential 100-> path.
Setups. All experiments
BenchOpt with package defaults unless otherwise celer wwg.soiiieggggesssiiiiii.
noted: skglm. blitzli. 0litz11.LogfeeProo.... LogRegProblem (). Single-X evaluations use three levels A/ A/Amax {0.5, 0.1, 0.05); path experiments use a 100-İ sequence. We retain each solver's default stopping rule and enable warm starts wherever available (BLITZI.1 reuses previous solutions; CELER and SKGLM expose warm starts). ECCD and GLMNET use warm starts and strong rules, which are known to be decisive for
(), and
'On n=100, p= 100,000 with {0.1, 0.9). data generation process runs exceeded 12 hours without completion cluster.
22

---

Table 10: ECCD vs, SKGLM (logistic; times in seconds). ECCD uses block size s-4.
Dataset
SKGLM [s]
ECCD [s]
Duke
1.375
0.013
Diahetes
6.155
0.002
100, p=1000000
131.340
1.426
= 100,000, p=100
213.012
1.032
Table 11: ECCD vs. GLMNET vs. ABESS (times seconds) on correlated synthetic data and Duke.
ECCD uses block size 8=4.
Dataset
GLMNET [s]
ABESS [s]
ECCD [s]
n=20, p=2000, p=0.1
0.014
0.011
0.002
n=20,
p=2000, p=0.9
0.018
0.011
0.002
n=2000, p-20, p=0.1
0.030
0.426
0.013
n=2000, p=20, p=0.9
0.047
0.493
0.027
n=100, p=1000, p=0.1
0.028
0.026
0.005
n=100, p=1000, p=0.9
0.066
0.028
0.006
n= 1000, p-100, P=0.1
0.021
0.131
0.007
1000,
p-100,
p=9.9
0.026
0.128
0.006
100000000 p=100, p=0.1
2.384
904.368
1,426
n= n= 100,000, p=100, p=0.9
3.840
931-303
1.032
Duke
0.022
0.025
0.013
path efficiency. Inputs are standardized upstream of the solvers; timings are wall-clock seconds. A dash (-) denotes unsupported model/setting combinations (e.g., Elastic Net for ICCELE).
Findings. (i) In the single-X regime, Python sparse solvers are competitive and can be fastest on individual instances; ECCD remains consistently strong across datasets (Tables 12-13). (ii) In the 100-À path regime, ECCD/GLMNET are consistently faster-often by one to two orders of magnitude- because
combination warm starts and strong rules mitigates the near-linear cost growth with the number of À values (Table 14). (iii) These trends robust across real (Duke, Leu, Diabetes) and synthetic (tall Synl and wide Syn2) settings. ECCD's s block updates with active-set screening yield stable wins in path mode while remaining competitive in single-point mode.
Takeaways. (A) In single-À mode, SKGLM/BLITZLI I are competitive on small dense problems and sometimes match beat GLMNET; ECCD remains in the same ballpark (often faster) across real and synthetic datasets. (B) In path mode, ECCD and GLMNET dominate across all datasets because strong rules prune inactive coordinates early, the effective work per À shrinks markedly. Warm starts alone (as IKGLM/CELER/BLITEEE 1) reduce constants but still scale roughly with the number of À values. (C) Elastic Net support is uneven in the Python sparse packages; results shown reflect what each library exposes under BenchOpt. Overall, these experiments corroborate our central claim: ECCD provides pathwise solver that is both competitive single points and substantially faster full paths-the regime most relevant to model selection and real workflows.
D Additional Experiment Result
Synthetic data generation. We generate datasets with n observations and p predictors by first drawing rows of the design matrix X E Raxp i.i.d. from zero-mean Gaussian with an equi- correlated covariance,
X, ~ (O, E):
1, Ejx- = p (j k),
which imposes a controlled, tunable correlation among features (the special case = recovers independence). We highlight this family because it explicitly produces correlated designs and is therefore relevant stress test for the ABESS baseline. To induce sparsity, we construct 3* E IP by
'Objective values (not shown) matched within 10 across methods in our runs, except that BLITZL] occasionally reported slightly different objective. We followed the BenchOpt reference code verbatim.
23

---

Table 12: Single-A runtime (s), Lasso. Each cell lists A/Amax E {0.5, 0.1, 0.05) from top bottom.
Dataset
SKGLM
CELER
BLITZLI
GLMNET
ECCD
(0.5/0.1/0.05)
(0.5/0.1/0.05)
(0.5/0.1/0.05) (0.5/0.1/0.05)
(0.5/0.1/0.05)
Duke
1.8 X 10 3
1.6 X 10
2.3 10
5.5 X 10
2.5 X 10
2.3 x 10
3.0 X 10
5.6 x 10 3
7.8 x 10 3
5.0 X 10
2.3 10
3.5 10
6.5 10-3
1.4x 10-2
6.2 X 10 10
Leu
2.2 10
1.2 X 10
2.1 10
1021100>
2.1 10
x x 10
2.3 x 10
4.4 10
x
4.3 X 10
x 10-3
2.2 10
5.7 10 10-3
x.7
4.7
Diabetes
1,6 10
3.6 X 10
8.0 10
13xx10
4.0 10
x x 10
4.9 x 10
6.0 10
1
4.7 10
1.6 x06
5.0 X 10
5.0 10
1.8 10
5.6 10
Synl (100x 100k)
3.8 x 10
1.1 10
1.0 10
x.0
6.2 10
x.7x1007
3.2 X 10
2.2 10-1
10
1
7.7 10
3.7 10
3.4 10
2.7x
x0-
8.8
Syn2 (100kx 100)
2.9 x 10
6.8 x 10
1,8 10
1.1x 111110-1 10-1
6.2 10
9.0 x 10
5.4 10
10
1.1 10
6.4 10
2.5 x 10
5.6 10
2.9 10
1.3 X 10
8.4 X 10
Table 13: Single-A runtime (s), Elastic Net (a = 0.5). Each cell lists X/Amax E {0.5, 0.1, 0.05) from top to bottom. unsupported.
Dataset
SKGLM (0.5/0.1/0.05)
CELER
BLITZL1 (0.5/0.1/0.05)
GLMNET
ECCD (0.5/0.1/0.0)
(0.5/0.1/0.05)
Duke
2.3 x 10 3
5.1 X 10
2.4 X 10
2.6 x
8.0
4.5 X 10
2.4 10-3
1.1 10
6.7 10
Leu
2.2 x 10
4.8 x 10
2.2 10
2.2 10
x
4.0 10
2.3 x 10
9 10
5.2 X 10
Diabetes
1.6 x 10
1.1x 3
6.0 X 10
1.6 10-3
1,5 10-3
10 10
1.5 x 10-3
1.3 x 10-3
5.7 X 10
Synl (100x LOOK)
6.0 x 10
1.0 x.0x10 10
6.1 10 -
3.9 x 10 10
1.3 10
7.6 X 10
4.0 X 10
105
1.0 X 10
Syn2 (100kx 100)
2.6 X 10-1
1.0 X 10-1
5.6 X 10
3.5 X 10-1
1.1 10-1
6.5 X 10-2
2.4 x 10
1.2 x 10
8.3 X 10
selecting indices uniformly random and assigning them values drawn uniformly from fixed interval (c.g., [1, 2]), leaving all other entries zero. Given X and f8, we form the linear predictor n X33 convert to success probabilities via the logistic link
1
p;
i=
and finally sample the binary response as yt Bernoulli(p)).
Experiment Dataset Summary. We evaluate ECCD and GLMNet on diverse set of eight publicly available benchmarks drawn from applications in genomics, credit modeling, biomedical studies, and web security. These include a couple of small-sample, high-feature gene-expression sets, an RNA-seq dataset with an extreme feature-to-sample ratio, several moderate-scale classification tasks (e.g. credit approval and medical diagnostics), and larger web phishing collection. By covering a broad range of dimensionalities and aspect ratios, this suite stresses both numerical stability and computational performance under the varied conditions practitioners commonly face.

---

Table 14: Sequential 100-A path runtime (seconds) via BenchOpt. ECCD entries include the block size s used. dash (-) indicates unsupported.
Duke
Leu
Diabetes
Syn1
Syn2
Method
Model
Lasso
ENet
Lasso
ENet
Lasso
ENet
Lasso
ENet
Lasso
ENet
SKGLM
Lasso/ENet
2.8c-1
2.9c-1
5.5
7.2
1,4e-1
1.4e-1
6.2
4.1
5.3e1
1
Lasso
4.1
2.4
3.0e-1
2.2e1
5.6e1
BLITZL
Lasso
4.5c-1
2.3c-1
4.1c-2
1.5e1
GLMNET
Lasso/ENet
2.1e-2
4.9e-2
2.0e-2
5.9e-2
6.0e-3
6.0e-3
4.7e-1
4.8e-1
1.7
2.0
1.3e-2
2.2e-2
1.0e-2
1.8e-2
4.0e-3
4.0e-3
2.7e-1
2.9e-1
1.6
1.6
ECCD
Lasso/ENet
(s=32)
(s=8)
(s=16)
(s=16)
(s=8)
(s=8)
(s=32)
(s=32)
(sm4)
(s=4)
Table 15: Summary of datasets used for numerical tests.
Filename # Observations (n)
# Features (p)
duke
44
7.129
leukemia
38
7,129
colon-cancer
63
2,000
diabete-scale
768
8
australian
690
14
airway
8
63679
prostate
102
6033
phishing
11,055
68
Implementation details and baselines. ensure a fair comparison with the state-of-the-art, we align our path-tracking experiments exactly to GLMNet's settings. In particular, we use GLM- Net's default convergence threshold (thresh 10-2), intercept fitting, and grid of 100 As with Amin/Amax 10- All input features pre-standardized by and we disable GLMNet's internal standardization to avoid double-scaling. For the single lambda case, we simply set the lambda parameter euqals given lambda input, and set the path parameter equals false.
While we also benchmark biglasso (v1.6.1, GPL-3 license) and nevreg (v3.15.0, GPL-3 license), GLMNet remains the de facto standard for elastic-net penalized GLMs, so we continue to report GLMNet''s solutions as our canonical baseline. GLMNet (v4.1-8, GPL-2.0 license) remains the most widely used coordinate-descent engine for elastic-net elastic-net-eenalizee generalized linear models and enforces a stricter stopping rule than these alternatives. We ran all methods with their latest CRAN versions, always matching sparsity parameters a and, where possible, using their fastest safe defaults.
Estimating the per-operation cost C. To understand the relative cost of estimating the expensive first and second order derivative of the CGF (e.g. sigmoid) versus simple arithmetic update, measure the time per call and form the ratio
C time VF(x))
time( +
We perform one complementary experiments at the vector level (to mimic typical linear-algebra kernels) and report the result on HPC node.
All performance experiments were carried out on a Cray compute node (Intel Xeon 6248R @ 3,0 GHz, 192 GB RAM) running SUSE Linux Enterprise Server 15 SP5 with the vendor-provided multi-threaded LibSCI BLAS/LAPACK (libsci Loosci_guu__a gn___nn__________ _____22_m___ 1333__pp__0__ LAPACK 10. We used R 4.3.1 (built under glibe) with C++ extensions compiled via Repp 1.0.10 and ReppEigen 0.3.3 under -03 -march -ffast-matie Vectorized kernels leveraged Eigen's 's dense operations (on dimensions up to 100), while scalar tests invoked plain C++ loops for exp and addition.
Using a microbenchmark harness R, we call two routines in each iteration: (1) sigmoid on length-100 Eigen vector, (2) pure elementwise addition of a constant. We fix 10 iterations and repeat each measurement 100 times. This setup approximates the hotspot cost in block- kernels on perfomeeffrm system.
25

---

Table 16:
ECCD convergence on Duke under adaptive vs, fixed block sizes s. Entries are relative objective differences (smaller is better).
Block size
Rel. Obj. Diff (Adaptive)
Rel. Obj. Diff (Fixed)
1.38 10-4
1.38 10-4
16
8.025 x 10
9.810 x 10
64
8.674 10
1.598 10
256
8.674 x 10-5
1.240 x 10
2048
8.674 10
1.218 10
7129
8.674 x 10-5
1.470 x 10
Across both setups we observe that computing the sigmoid is roughly 13x ~ 17x more expensive than simple add, depending on argument range and vector-width effects.
Block Size Selection Strategy in Practice While our theoretical analysis suggests that the optimal block size 8* vC, practical implementation requires careful estimation of the hidden constant C, which captures the relative cost of link function evaluations versus basic linear algebra operations. To empirically estimate C, we benchmarked the cluster environment under two settings: (i) fully optimized Eigen-based matrix operations, and (ii) vanilla implementations involving repeated sig- moid evaluations and summations. Across both settings, we consistently observed that, in logistic regression, sigmoid function evaluations were in average 13x to 17x more expensive than basic matrix-vector multiplications. This implies theoretical optimal block size of s* VC 4. However, in practice, additional considerations arise. Notably, ECCD avoids redundant sigmoid function updates when coefficient increments negligible, which slightly decouples actual update frequency from worst-case complexity bounds. Moreover, secondary factors-incllding the need to explicitly compute the second derivative of the sigmoid function, hardware-level vectorization effects, and dynamic fluctuations in the active set size-can further shift the empirical optimal block size. Our profiling across datasets revealed that slightly larger block size, particularly 8, consistently delivered near-optimal speedups while preserving stability across various scaled tasks. Therefore, although identifying the precise optimum can be challenging. s=8 emerges robust default that achieves near-optimal speedups and beats most baselines across varied scales.
3.005
Gradient Natrices
(rnconae)
0. 020
Matrices
0.10
Matrices Block Coef Update Active Sat
(2ccona))
6.125
Block Coef
0.08
Block Caef
0. 015
Updwle
Update
Active Set
0.075
Active Set
010
0.66
0.02
0.025
16
52
16
52
16
Block: Size 15)
Block Size (5)
Block (5)
(a) A = 0.1
(b) X = 0.01
(c) A = 0.001
Figure 4: ECCD Time Decomposition Plot (Duke)
Convergence of ECCD with Different Blocksize Selection. To illustrate the convergence behavior of ECCD under different blocksize selection, extended ECCD to block sizes up = p 7129 on the Duke dataset (the maximum possible blocksize for this dataset); results in T6ble[6]]hoos stable convergence even
p. underscoring the robustness of second-order Taylor correction and its advantage over na've block coordinate descent (cf. main-paper Table 3). Column 2 reports relative objective differences under adaptive active-set scheme: updates restricted to currently nonzero coefficients, KKT violations trigger working-set expansion, and when > |A| we cap the working block by setting + ||A to avoid the s² memory/compute spike. Column 3 gives the same metric with fixed s (no resizing), providing direct comparison. Across all 8, the adaptive variant matches improves the objective and remains numerically stable through with our theory, the practically optimal is a small constant determined by link-function cost, so our original range E [2, 32] is both relevant and justified; the large-s study is included for completeness.
7129. Consistent
26

---

Empirical Cost Breakdown Figure 4 reports the per-epoch wall-clock composition of ECCD on the Duke dataset for three regularization strengths. For the baseline update schedule (s 1). evaluation of the cumulant-generating -uuntttiinnnontiiinnnn (CGF) gradient (pink) and matrix-update operations (purple) together consume over 80% of total runtime; coefficient-batch updates (dark blue) and active-set maintenance (teal) also contribute nontrivially. As the block size grows, we observe:
A roughly 1/s reduction in CGF-gradient cost, confirming that ECCD's block-amortizatiin
defers expensive derivative evaluations until after each batch of s coordinate updates. costs (dark purple and dark blue), since these are still performed once per block.
A relative increase in the share of
and coefficient-
Matrix-update overhead and active-set maintenance become nearly negligible at larger validating that their amortization is effective.
Across all three À settings, the absolute time spent on each component drops by approximately 75% when moving from 8 = 1 to s = 32, aligning tightly with our theoretical (mpp/ + p) complexity for sigmoid-eve readd-vvllattondnnneeeee dominated workloads. ECCD thus shifts the bottleneck away from per-coordinate derivative calls toward bloek-level matrix operations, unlocking substantial runtime savings.
Memory complexity. Theorem 3 provides a detailed space space- complexity analysis of ECCD. Here we extend our experiments to verify this claim, explicitly accounting for the additional memory required by ECCD beyond the storage cost of the design matrix. Concretely, ECCD allocates ((ns) memory for the working block Xs and (s²2) for its Gram submatrix, where the ideal block size is a small constant that approximately independent of and p. For comparison, GLMNET corresponds to the from the standpoint of asymptotic memory. Consequently, in high-dimensional regimes with p >> s, the relative overhead of ECCD scales as ns/ ans/(pp) = s/p + and is therefore negligible. This behavior is consistent with our empirical observations.
special case
In all experiments we report resident set size (RSS) and track peak usage during a regularization path. To better isolate ECCD's overhead, we distinguish (i) the baseline RSS at process start, (ii) the peak RSS during optimization, and (iii) the end RSS. Because absolute baselines can vary slightly across runs and environments, the s=1 row in each table serves as the reference point within that experiment.
With the active-step capping mechanism, the extra storage in ECCD is dominated by s vectors of length n (the working block Xx) and an s Gram submatrix. In the extreme where approaches the size of the active set, the Oss²) term can dominate. To prevent this, we cap the working block by dynamically setting < |VIAi once the active set reaches size |.A]. This ensures that further increases in |.|| do not cause quadratic growth memory, keeping ECCD's overhead negligible relative to the design-matrix storage. Without the active-set cap, the (ss)) term becomes dominant for large s, and peak RSS rises accordingly (Table 24 This confirms the necessity and effectiveness of the capping rule in practice.
Using the solution at the previous À as warm start, peak RSS grows only until s reaches VIA max along the path, after which peak memory plateaus even as the nominal increases. For example, when n=p=5000 (Table 2T). the bascline RSS is approximately 755-757 MiB; increasing s up to 4096 adds only ~ 39 MiB over the s=1 case, indicating that the effective cap suppresses 8 growth. In contrast, if the cap is removed (Table 24h, large s values trigger the (ss2) regime and peak memory rises substantially.
When >> n and the active-set cap is in effect, neither O(ns) nor O(s²) dominates, and peak RSS is effectively flat increases (Table 22). Conversely, in n >> p settings observe similar stability in
peak RSS with respect to s (Table 23). Together these results indicate that ECCD's memory footprint closely tracks the design-matris storage across aspect ratios.
indicating that is effectively constrained by the active-set upper bound and that ECCD's overhead is negligible at medium scale. For GLMNET, we report peak RSS (excluding baseline) measured via the peakRAM R package. These figures offer a reference for the additional memory footprint of coordinate wise updates (s=1) across aspect ratios (Table 26)
On the Duke dataset, memory usage is essentially flat across all 8 (Table 25
BBeaasse GLMNET is implemented in optimized C++, direct in process instrumentation of its internal allocations is impractical: the reported numbers provide consistent external estimate across problem sizes.
27

---

Table 17: Performance Experiment for Single À Fits (a: 0.5)
Benchmark
Lambbda Ratio
ECCD Optimal
ECCD
GLMNet
ECCD
Dataset
Time (8) (best s)
Time (8)
Time (S)
Objective
Rel. Diff
Speedup
0.800
7.30c-04 (at 521
LU3c-03
6.036-03
0.611
3.09c-06
8.23x
0.500
3.18e-03 (al 8 16)
1.01e-02
7.05e-03
0.409
1.04e-04
2.22x
0.200
4.40c-03 (at
1.76c-02
6.87e-03
0.175
4.78c-05
1.56x
0.100
5.23e-03 (al 32)
2.18e 02
03
0.0891
4.06e-04
1.71x
16)
9.76e-03
0.050
6.93e-03 (at 16)
3.20c-02
1.10c-02
0.0456
2.20c-04
1.59
duke
0.020
8.216-03 (at # 32)
4.37c-02
1.926-02
0.0189
2.71c-04
2.33x
0.010
1.07e-02 (at 4 16)
4.84c-02
2,51c-02
9.69c-03
1.124-04
2.35
0.008
1.176-02 (at 16)
6.04c-02
2.676-02
7.83c-03
2.13c-05
2.29x
0.005
1.22e-02
16)
6.35e-02
3,35e-02
4.98e-03
2.09c-04
2.75x
0.002
1.32e
(at s 32)
8.10e
4.83e
2.0fe
9.86c
3.65
0.001
1,37e-02
32)
9.02c-02
5.72e-02
1.05e-03
3,99c-03
4.19x
0,800
5,20c-04 (at 8)
8.90c-04
5,206-03
0,304
1.02-066
10.0x
03
321
0.78e-03
0.200
3.68e 03 (al
16)
1. 1.34 02
6.96e 03
0.128
1.43e 14
1,89X
32)
8.07e-03
2.45c-05
2.00
0.080
4.33e 03 (al
32)
2.00e 02
8.75e
0.0516
1.63e 05
2.02X
0.34c-03
32)
3.
1.12e-02
8.44--05
2.58
leu
0.020
6.78c-03 (at
16)
8.45c-02
1.65c-02
0.0140
2.520-04
2.13x
0.010
1.90-022 (at
4
3,22e-01
1.02c-01
6.47e-03
4,92c-04
5
0.008
2. 2.30e
8)
3.92e
1.16e-01
5.81e:03
4.12c-04
5.00X
0.005
02 (at
16)
4.66c-01
1.37e-01
3.200-03
7.15c-04
4.50x
0.002
4.10e 02
32)
5.17e-01
1.71e.01
1.36e 03
1.92c-04
4.20X
0.001
0. (at
32)
1.31e-11
5.54e-02
7.86f--44
1.11e-03
4.83
2.00c.04
3.30e-04
1,52c-06
0.500
1.27e
(at
8)
3.72e-03
2.54e
0.428
3. 3.18c-00 05
2.00
03
16)
03
0.100
3.58c
(at
4)
1.17e-02
4.9
0.124
7.56c 16
1.37X
0.080
4,25c.003
4)
1.176-02
4.856-03
1.976-05
1.14X
0.050
3.66c-03
(at
8)
1.53c-02
6.42c-03
0.0645
8.67c-05
1.75x
0.020
7.26e-03
(al
4)
2.35e-02
1.16e-02
0.0269
1,35e-04
1.60X
0.010
6335c-03
8)
3.44c-02
2.17e-02
0.0139
9.66c-05
3.11X
0.008
0,59:-03
(al
8)
3.48--02
2.54e-02
0.0113
2.92e-05
3,34X
0.005
03
16)
3.77e
0,95e-02
0.00721
4.48c 05
3.85
0.002
8, 8.72e-03
(at
16)
4.43-022
4,74e-02
0.00301
5. 5,55c-04
5.43
0.001
32)
5,23c-02
6.06c-02
0.00155
1.18c-03
6.34X
0.8
3.0c-05 (al 2)
0.0c-05
1.726-03
0.624
2.24e-10
57.3x
6.c
9.0e
9.5X
0.2
2.6e-44 (at I)
2.6c-04
1.40c-03
0,512
5.766-06
5.4X
4.8444
5.6c-04
1,936-03
0.490
4.03--07
4.0%
0.08
4.9-044 (at
I)
4.9c-04
1.06c-03
0.485
2.220-066
2.2X
4.0c044
6,2c-04
1.026-03
0.478
5,590-06
2.6X
diabetee-scale
0.02
4.2c-04 (at
5.2c-04
1.82e-03
0.472
1.73e-08
4.3X
0.01
4.9c
(al s
5.5c-04
1.27e
0.471
1.02e
2.6%
0.088
3.4e4M4
6.0c-04
1.03e-03
0.471
1.00e-07
3.0X
0.005
4.6e
(at
0.6e-04
1.64e 03
0.471
8.44e
3.6%
0.002
5.4e4M4
7.4c-04
1.699-03
0.471
1.39c-08
0.001
5.26-44 (at
2)
6.4c-04
1.68c-03
0.471
8. 8,54c-09
3.2x
4.00e-05
2.37e-10
0.500
1,30e-04 (al
2)
6.30e 04
0.0011
0.556
2.40-
8.46X
3. 10e-4
4)
5.23x
0.100
3, 30c-04 (at
14)
6.80c-04
0.0011
0.366
1. 1.266-05
3.42X
0.080
4, 4.200004 (at
4)
9,70c-04
0.0011
0.354
1.25c-06
2.69x
0.050
2,80c-44
14)
7.50c-44
0.0012
0.336
4.70c-07
4.18x
australian
0.020
4.20c 04 (at s
14)
1,02e
0.0013
0.320
1.32e 07
3.14X
0.010
4,90--44 (at
141
1.07e-03
0.0016
0.313
5.38c-07
3,33x
0.008
5.00e 04 (at s
14)
1,28e 03
0.0016
0.312
2.16e 06
3.16x
0.005
5.00e-04
1.21e-03
0.0019
0,310
14e4-0
3.72
0.002
4, 9.90~044 (at = 9
I. 1.08-033
0.0016
0,308
2. 2.86c-07
3.35 X
2
0.0017
0.8
0XX026 (al
16)
0X0028
0.0143
0.625
L60E
5.5X
1.30-100
1.9x
0.2
0.0273 (at 8)
0.0382
0.0335
0.339
1.39c-8
1.23x
0.0461 [at 6 4)
0.0816
0.0271
0.258
4.53e-8
0.59X
0.08
0,0515 (at
0.1
0.0457
0.240
6.20c-8
0.89%
phishing
0.0569 (at 8 32)
0.143
0.0652
0.210
1.05e-7
1.15x
0.02
0,285
0.665
0.196
0.174
8
0.69X
0.01
0,942 (at 4)
1.62
0.505
0.157
4.6e-8
0.54
0.008
0.870
0.651
0.153
1.36e-7
0.75x
0.005
1.13 (at
1.85
1.19
0.148
1.77e-8
1.05x
0.002
421
4.63
2.95
0.144
2.19
0.70x
0.001
2,44 (at
7.79
5.33
0.142
9.75e-10
2.19x
28

---

Regularized Poisson Regression We also evaluated our method on Poisson regression. As given in the tables 27] on the Bioconductor "airway" dataset (8 samples: 4 control, dexamethasone; ~64,000 features), our implementation reduces runtime from 0.621 (glmnet's is runtime) to 0.0402 (ECCD's optimal runtime), corresponding
13.0x speedup across all selected a values. On the Duke dataset, runtime decreases from 0.068 to 0.029 (a 2.3x speedup). These results demonstrate that the favorable scaling our approach extends beyond logistic models to other generalized linear model link functions, while preserving solution quality, More details are available in table 20
29

---

Table 18: Time (s) and relative prediction error under different a (logistic regression w. intercept).
Filename
Method
0.1
0.2
0.5
=
0.8
1
Time
Rel. Diff
Time
Rel. Diff
Time
Rel. Diff
Time
Rel. Diff
Time
Rel. Diff
gimnet
6.40c-02
5.72c-02
5.30c-02
4.89c-02
4.95c-02
nevreg
8.29e-01
5.37e-01
6.88e-01
5.33e-01
5.80e-01
5.33e-01
5.55e-01
5.47e-01
5.45e-01
5.63e-01
biglasso
2.37e-01
2.98c-01
2.29c-01
1.91c-11
2.26e-01
5.11e-02
2.28c-01
7.35e-03
2.59c-01
8.75e-06
8 = 5.56c--02
5.44e-06
3.81e-02
6.41c-06
2.27e-02
5,50c-06
1.62c-02
5.50c-06
5.68e-02
4.82c-06
8 = 5.75c-02
5,47c-06
3.68e-02
6.33e-06
2.28c-02
5,27c-06
1.60c-02
5,40c-06
3.74c-02
4,45e-06
8 = 3,97e-02
6.02e-06
2.83e-02
6.14e-06
1.86e-02
5.40e-06
1.4Se-02
5.56e-06
2.60e-02
3.52e-06
duke
$ 3,40e-02
7.91e-06
2,46e-02
6.77e-06
1.74e-02
5.28e-06
1.55e-02
5.31e-06
2.09e-02
5.12e-06
16
3,48c-02
8.73e-06
2.48c-02
8.04c-06
1.79c-02
7.05e-06
1.40c-02
7.30e-06
1.71e-02
2.09c-05
32
3.69e-02
2.48e-05
2.67e-02
8.99e-06
1.87e-02
9.15e-06
1.59e-02
7.30e-06
1.75e-02
2.09e-05
64
0.00c-02
3,89c-05
2.73c-02
1.46e-05
1.86-02
9.15c-06
1.74c-02
7.30c-06
2.08c-02
2.09c-05
glmnet
3.06e-02
2.32e-02
2.33e-02
1.92e-02
2.01e-02
nevreg
7.44c-01
5.05c-01
5,49c-01
4.96c-01
4.70c-01
4.82c-01
4.21c-01
5.06c-01
3.88c-01
5.14c-01
biglasso
1.99e-01
2.56e-01
1.91e-01
1.58e-01
1.86e-01
3.86e-02
1.88e-01
4.77e-03
2.01e-01
8.14e-06
s
7.04e-02
2.79c-06
4.10e-02
4.33e-06
1.95e-02
3,55e-06
1.52e-02
4.26e-06
5,99e-02
8.17e-06
s
6.86e-02
2.62e-06
4.04e-02
4.38e-06
2.02e-02
3.44e-06
1.35e-02
4.24e-06
4.01e-02
7.78e-06
colon-cancer
s
8
3.10e-02
4.12c-02
3,87e-06
3.21e-06
2.20e-02
2.64c-02
4.55e-06
3.92c-06
1.40c-02
1.24e-02
3.92c-06
3,96e-06
1.01e-02
1.02c-02
3.78c-06
4.32e-06
2.06c-02
1.43e-02
6.17e-06
7.17c-06
16
3.18e-02
6.95e-06
2.12e-02
5,46e-06
1.34e-02
3.66e-06
1.03e-02
6.94c-06
1.50e-02
3,98e-06
8 = 32
3.53c-02
2.15e-05
2.35c-02
8.12c-06
1.42c-02
4.83e-06
9.99c-03
7.61c-06
1.33e-02
3. 3.98c-06
8 = 64
3.76e-02
2.98e-05
2.35e-02
5.81e-06
1.41e-02
4.83e-06
1.04e-02
7.61e-06
1.34e-02
3.98e-06
glmnet
5.83e-02
5.17e-02
4.93e-02
4.75e-02
4.80e-02
nevreg biglasso
5.86e-01
5.73e-01
4.62e-01
5.74e-01
4.55e-01
5.50e-01
3.94e-01
5.48c-01
4.00e-01
5.11e-01
2.24c-01
3.29c-01
2.19c-01
2.51c-01
2.12c-01
1.99c-01
2.09c-01
1.89c-01
2.27e-01
1.89c-01
s ==
5,54c-02
9,46e-06
4.10c-02
4.33c-06
1.88c-02
8.66c-06
1.51c-02
1.23c-05
3.18c-02
1.90c-06
8 == 5.44e-02
1.10e-05
3.68e-02
4.38c-06
1.85e-02
8.27e-06
1.55e-02
1.10e-05
2.65e-02
1.73e-06
8 == 4
3.87e-02
1.49e-05
2.73e-02
1.13e-05
1.64--02
8,29e-06
1.39e-02
9.38e-06
2.08e-02
3,57e-06
leukemia
8 = 3,33c-02
2.14c-05
2.45c-02
1.42c-05
1.56c-02
6.10c-06
1.57c-02
9.32c-06
1.97c-02
9.13e-06
8 = 16
3.11e-02
1.10e-04
2.49e-02
2.04e-05
1.56e-02
9.49e-06
1.58e-02
1.84e-05
1.98e-02
1.40e-05
$ 32
3.20e-02
3.56e-04
2.80e-02
3.75e-05
1.59e-02
7.77e-06
1.62e-02
1.84e-05
1.97e-02
1.40e-05
8 64
3.43e-02
6.95e-04
2.62c-02
3.73e-05
1.58c-02
7.77e-06
1.58c-02
1.84c-05
1.99c-02
1.40c-05
gimnet
8.65c-03
8.10c-03
7.67c-03
7.42c-03
7.36c-03
nevreg
5.43e-01
2.54e-01
5.42e-01
2.93e-01
5.49e-01
3.36e-01
5.40e-01
3.53e-01
5.35e-01
3.62e-01
biglasso
1.60-001
8.97e-02
1.67e-01
5.07e-02
1.65e-01
1.16e-02
1.65e-01
1.33e-03
1.59e-01
8,59e-05
s
1.39e-02
1.06e-09
1.39e-02
2.22e-08
1.32e-02
2.26e-06
1.31e-02
3.51e-05
1.35e-02
1.53e-04
$
1.37c-02
1.05e-09
1.29c-02
2.22c-08
1.29c-02
2.26c-06
1.33e-02
3.51e-05
1.31e-02
1.53e-04
australian
s
7.69c-03
2.15e-08
7.59e-03
2.22e-08
7.65e-03
2.26e-06
7.10e-03
3.51e-05
7.63e-03
1.53e-04
s
7.22e-03
2.16e-08
7.52e-03
2.25e-08
7.12e-03
2.26e-06
7.53e-03
3.51e-05
7.79e-03
1.53e-04
16
8.03c-03
2.15c-08
7.64c-03
2.25c-08
7.27c-03
2.26c-06
7.48c-03
3.51e-05
7.80c-03
1.53e-04
32
0.00e-03
2.15e-08
7.67e-03
2.25e-08
7.32e-03
2.26e-06
7.57e-03
3.51e-05
7.83e-03
1.53e-04
8 64
8.09c-03
2.15e-08
7.66e-03
2.25e-08
7.48e-03
2.26c-06
7.52e-03
3.51e-05
7.79e-03
1.53e-04
glmnet
7.54e-03
7.13e-03
7.09e-03
7.06e-03
0.49e-03
nevreg
2.71c-01
3.17e-02
2.76c-01
1.52c-02
2.88c-01
2.48c-03
2.86c-01
2.29c-04
3.02c-01
1.31c-10
biglasso 1.82c-01
3.17e-02
1.62e-01
1.52e-02
1.61e-01
2.48e-03
1.62e-01
2.30e-04
1.76e-01
3.17e-05
==1
8.68e-03
4.18e-06
9.38c-03
2.27c-07
9.58e-03
2.75e-06
9.63e-03
1.32c-05
9.68e-03
2.87e-05
s == 8.76e-03
4.19e-06
9.75e-03
6.93e-07
1.02e-02
2.75e-06
9.99e-03
1.32e-05
9,88e-03
2.87e-05
8 == 4
6,25c-03
4.19c-06
6,97c-03
6,93c-07
7.06c-03
2.75c-06
6.86c-03
1.32c-05
6.97c-03
2.87c-05
diabetes_scale
s ==
4.95e-03
2.17e-05
5.63e-03
6.94c-07
6.07e-03
2.75e-06
6.09e-03
1.32e-05
6.28e-03
2.87e-05
s 16
4.97e-03
2.17e-05
5.61e-03
6.94e-07
6.10e-03
2,75e-06
6.06e-03
1.32e-05
6.14e-03
2.87e-05
8 = 32
4.92c-03
2.17c-05
5.68c-03
6.94c-07
6.14c-03
2.75c-06
6.1-c-03
1.32c-05
6.20c-03
2.87c005
8 = 64
4.94e-03
2.17e-05
5.57e-03
6.94e-07
6.12e-03
2.75e-06
6. 6.e-0303
1.32e-05
6.18e-03
2.87e-05
gimnet
7.48e-02
7.33e-02
6.59e-02
6,53e-02
6.21e-02
nevreg
2.53e+00
4.40e-01
2,08e+00
4.33e-01
1.84e+00
5.06e-01
1.74e+00
5.09e-01
1.69++00
5.15e-01
biglasso
3.26e-01
1.96e-01
3.03e-01
1.03e-01
2.96c-01
1.98e-02
2.94e-01
2.19e-03
2.98e-01
2.09e-06
s ==:
3.24c-01
8.42c-07
2.65e-01
3.30c-07
1.61c-01
8.80c-07
1.25e-01
2.24c-06
1.13c-01
5.27c-06
8 = 2.15e-01
1.30e-06
1.74e-01
6.27e-07
1.14c-01
6.77e-07
9.03e-02
2.11e-06
8.32e-02
5.08e-06
s ==
1.16e-01
1.90e-06
9,45e-02
1.05e-06
6.25e-02
4.42e-07
5.56e-02
1.94e-06
5.04e-02
4.39e-06
prostate
$ 9.12c-02
2.83e-06
7.09c-02
1.69c-06
5.20c-02
6.37c-07
4.53c-02
1.92e-06
4.14c-02
2.54c-06
s 16
8.28c-02
3.73e-06
6.60e-02
2.44e-06
5.00e-02
1.16e-06
4.24e-02
3.15e-06
4.02e-02
3.69e-06
32
8.68e-02
5.10e-06
6.80e-02
3,48e-06
5.01e-02
1.56e-06
4.38e-02
3.84e-06
3.89e-02
5,06e-06
64
9.38c-02
5.91e-06
7.11
3.34e-06
5.15c-02
1.56c-06
4,39c-02
3.84e-06
3.88c-02
5.06c-06
30

---

Table 19: Time (s) and relative prediction error under different a (logistic regression w/o intercept).
Filename
Method
0.1
0.2
0.5
0.8
Time
Rel. Diff
Time
Rel. Diff
Time
Rel. Diff
Time
Rel. Diff
Time
Rel. Diff
glmnet
6.06e-02
0.00e+00
5.26e-02
0.00e+00
5.24e-02
0.00e+00
4.74e-02
0.00e+00
5.17e-02
0.00e+00
8 = 1
8.89c-02
2.63e-06
8.00c-02
1.08c-06
4.42c-02
1.88c-06
2.67c-02
3.02c-06
2.40c-02
1.71c-05
8 = 2 6,88e-02
3.91e-06
6.47e-02
2.40e-06
3.58e-02
1.18e-06
2.53e-02
2.77e-06
2.26e-02
1.63e-05
8
4.07e-02
5.78e-06
3.38e-02
3.17e-06
2.37e-02
1.13e-06
1.89e-02
2.23e-06
1.77e-02
1.36c-05
s
3.42e-02
6.96e-06
2.88e-02
5.35e-06
2.04e-02
1.75e-06
1.67e-02
1.44e-06
1.56e-02
1.19e-05
duke
8
16
3,32c-02
1.02c-05
2.63c-02
6.78c-06
2.09c-02
2.59c-06
1.73e-02
1.94c-06
1.68c-02
7.79c-06
32
3.54e-02
1.24e-05
2.74e-02
1.03e-05
2.16e-02
3.66e-06
1.75e-02
2.30e-06
1.55e-02
7.79e-06
glmnet
3.02e-02
2.47e-02
2.17e-02
1.93e-02
1.79c-02
1
8,73e-02
3.36e-07
6.57e-02
5.79e-07
3.82e-02
1.59e-06
2.98e-02
2.25e-06
2.59e-02
6.98e-06
8
7,51c-02
4.13c-07
5,88c-02
5.14c-07
3.33e-02
1.42c-06
2.84c-02
2.12c-06
2.75c-02
6.76c-06
8
4.26e-02
6.64e-07
3.12e-02
5.11e-07
1.93e-02
1.13e-06
1.60e-02
1.73e-06
1.81e-02
6.46c-06
$
3.03e-02
1.14e-06
2.30e-02
9.15e-07
1.5le-22
8.11e-07
1.27e-02
1.44e-06
1.26e-02
5.75e-06
16
2.88c-02
1.96c-06
2.14c-02
1.74e-06
1.47e-02
1.52e-06
1.19c-02
3.57c-06
1.13e-02
1.46c-05
colon-canced
32
3.18c-02
2.47c-06
2.27c-02
2.40c-06
1.64--02
2.63e-06
1.26c-02
4.12c-06
1.12e-02
1,46c-05
64
3.53e-02
2.50e-06
2.31e-02
2.30e-06
1.63e-02
2.63e-06
1.26e-02
4.12e-06
1.12e-02
1.46e-05
glmnet
5.66c-02
4.92c-02
4.56c-02
4.62c-02
4.43e-02
$ 1
8.33e-02
3.76e-06
6.48e-02
2.52e-06
4.02e-02
9.42e-07
2.85e-02
1.80e-06
1.67e-02
2.92e-06
$
6.7le-02
5.62e-06
5,27c-02
3,47c-06
3,26c-02
1.45c-06
2.46c-02
1.83c-06
1.64c-02
2.83c-06
4.52e-02
7.49e-06
3.35e-02
5.08e-06
2.31e-02
2.75e-06
1.84e-02
1.97e-06
1.38c-02
4.66c-06
8
3.51e-02
9.32e-06
2.70e-02
7.12e-06
1889-02
3.25e-06
1.64e-02
2.42e-06
1.34e-02
7.6le-06
16
3.53c-02
1.19c-05
2.59c-02
8.81c-06
1.94c-02
3.62c-06
1.73e-02
3.77c-06
1.35e-02
8.61e-06
leukemia
32
3.86e-02
1.42e-05
2.66e-02
1.15e-05
2.05e-02
4.08e-06
1.72e-02
3.77e-06
1.35e-02
8.61e-06
= 64
4.47c-02
1.70e-05
2.82c-02
1.04e-05
2.06c-02
4.08e-06
1.71e-02
3.77c-06
1.36c-02
8.61c-06
glmnet
8.04c-03
7.94c-03
7.56c-03
7.41c-03
7.55c-03
1
1.20e-02
3.11e-10
1.13e-02
4.71e-10
1.19e-02
1.39e-09
1.31e-02
3.59e-09
1.41e-02
6.85e-09
1.19c-02
3,07e-10
1.13e-02
4.71e-10
1.16e-02
1.39e-09
1.25e-02
3.58e-09
1.39c-02
6.85e-09
4
6.33e-03
3.13e-10
6.29c-03
3.73e-10
6.97e-03
1.37e-09
7.30c-03
3.68e-09
7.53e-03
7.62e-09
australian
8
6.39e-03
3.20e-10
6.38e-03
5.00e-10
6.81e-03
2.42e-09
6.87e-03
1.53e-08
7.18e-03
3.40e-08
16
6.20c-03
2.88c-10
6.37c-03
4.99e-10
6.92c-03
2.42c-09
6.85c-03
1.53e-08
7.16c-03
3, 3.40e-08
8 32
6.30e-03
2.88e-10
6.37e-03
4.99e-10
6.89e-03
2.42e-09
6.79e-03
1.53e-08
7.15c-03
3.40e-08
8 64
6,34e-03
2.88e-10
6.43e-03
4.99e-10
6.85e-03
2.42c-09
6.82e-03
1.53e-08
7.15e-03
3,40e-08
glmnet
7.33c-03
6,95c-03
6.96c-03
6,95c-03
6,94c-03
1
6.67e-03
9.50e-12
7.20e-03
1.87e-11
7.83e-03
6.20e-11
8.09e-03
8.30e-1
8.17e-03
1.04e-10
2
6.68e-03
7.74e-12
7.20e-03
1.29e-11
8.00e-03
5.57e-11
8.30e-03
7.62e-11
8.22e-03
9,63e-11
8 4
4.68c-03
5.52c-11
4.94c-03
1.33e-10
5.13e-03
4.70c-10
5.22c-03
1.63c-09
5.23c-03
1.93e-09
8 = 8
4.35e-03
4.35e-10
4.62e-03
7.66e-10
4.97e-03
1.27e-09
5.01e-03
1.75e-09
5.06e-03
9,96e-10
$ 16
4.32e-03
4.35c-10
4.57c-03
7.66e-10
4.91e-03
1.27c-09
4.98e-03
1.75e-09
5.07e-03
9.96e-10
diabetes_scale
8 32
4.28c-03
4.35e-10
4.64c-03
7.66e-10
4.94e-03
1.27e-09
4.97e-03
1.75c-09
5.05e-03
9.96c-10
s 64
4.33e-03
4.35e-10
4.58e-03
7.66e-10
4.95e-03
1.27e-09
5.01e-03
1.75e-09
5.01e-03
9,96e-10
glmnet 6.52e-02
5.96e-02
5.69e-02
5.23e-02
5.52e-02
8 = 1
1.67e-01
6.72e-07
1.31e-01
3.37e-07
8.34e-02
7.16c-07
6.36e-02
1.25c-06
5.89c-02
2.92c-06
8 2
1.45-001
1.12e-06
1.17e-01
5.64e-07
7.75e-02
6.24e-07
6.11e-02
1.17e-06
5.60e-02
2.71e-06
8
8.18c-02
1.71c-06
6.56c-02
1.07c-06
4.86c-02
4.64c-07
4.11c-02
1.09c-06
3.69c-02
2.28c-06
8
7.17e-02
2.69e-06
5.52e-02
1.65e-06
4.32e-02
5.58e-07
3.69e-02
1.45e-06
3.33e-02
1.61e-06
16
7.27e-02
3,48e-06
5.69e-02
2.49e-06
4.34e-02
1.60e-06
3.68e-02
3.62e-06
3.30e-02
2.63e-06
prostate
32
7,93c-02
4.55e-06
6.15e-02
3.31e-06
4.72c-02
1.73e-06
3.92e-02
2.85c-06
3.45e-02
4,98c-06
64
8.75e-02
5.07e-06
6.18e-02
3.56e-06
4.74e-02
1.73e-06
3.90e-02
2.85e-06
3.45e-02
4,98e-06
31

---

Table 20: Poisson Regression: Time (s) and relative error under different a.
Filename
Method
ak = 0.1
0.2
=
0.5
0.8 a =
=
Time
Rel. Diff
Time
Rel. Diff'
Time
Rel. Diff
Time
Rel. Diff
Time
Rel. Diff
glmnet
6.864c-02
5.81c-02
5.18c-02
4.92c-02
4.67c-2
8
3.39e-01
1.95e-06
2.53e-01
1.91e-06
1.23e-01
1.46e-06
9.15e-02
3.31e-06
7.67e-02
3.77e-06
$
2.30c-01
1.94c-06
1.54c-01
2.25e-06
8.88e-02
1.63e-06
6.68e-02
3,99c-06
5,57c-02
4,58e-06
duke
8
1.23e-01
2.21e-06
8.89c-02
2.26ic-06
5.15c-02
2.14c-06
3.84e-02
3.73e-06
3.68e-02
5.87e-06
s
7.4le-02
3.66e-06
5.42e-02
2.63e-06
3.43e-02
2.00e-06
2.85e-02
3.52e-06
2.83e-02
3.05e-06
8 16
5.66c-02
1.34c-05
4.59c-02
4.29c-06
2.89c-02
2.81c-06
2.60c-02
3,44e-06
2.54e-2
3.02e-06
32
4.88e-02
2.37e-05
4.11e-02
9.55e-06
2.77e-02
4.85e-06
2.61e-02
3.44e-06
2.68e-02
3.02e-06
glmnet
5.63e-02
5.40c-02
4.96c-02
4.49c-02
4.58c-02
8
2.42c-01
1.24c-06
1.52c-02
1.14c-06
9.15c-02
1.34c-06
6,57c-02
1.40c-06
5.95c-02
9.74c-07
8
1.65e-01
1.26e-06
9.80e-02
1.32e-06
6.67e-02
1.64e-06
5.19e-02
1.24e-06
4.92e-02
1.07e-06
leukemia
8
8.90e-02
1.70e-06
5.75e-02
1.61e-06
3.85e-02
1.35e-06
3.23e-02
1.13e-06
3.00e-02
1.37e-06
5.67e-02
2.12c-06
3.97e-02
2.86c-06
2.93e-02
7.51c-07
2.54c-02
1.33e-06
2.21c-02
1.68e-06
16
4.64e-02
2.68e-06
3.22e-02
2.03e-06
2.40e-02
1.16e-06
2.18e-02
1.14e-06
2.07e-02
1.42e-06
32
4.22c-02
3.64c-06
3.12e-02
1,42e-05
2.42c-02
8.39c-07
2.35e-02
1.12e-06
2.06c-02
1.42c-06
glmnet
5.82e-01
5,79e-01
5.85e-01
5.91e-01
6.04e-01
8
1.63e-01
9.76c-07
0.131c-01
1.30c-06
1.14c-01
2.41c-06
1.02c-01
2.87c-06
9.56-02
8.90c-05
8
1.28e-01
1.06e-06
1.12e-01
1.32e-06
9.91e-02
2.25e-06
9.78e-02
2.71e-06
9.29e-02
8.89e-05
airway
8
1.34-001
1.02e-06
1.21e-01
1.59c-06
1.16e-01
2.02e-06
1.13e-01
2.99e-06
1.1le-02
8.90e-05
8.39e-02
9.65e-07
9.51e-02
1.02e-06
9.21e-02
2.66e-06
9.04e-02
5,41e-06
8.82e-02
8.91e-05
16
8.09c-02
9.19c-07
5.32c-02
1.52c-06
5.03c-02
3.04-066
4.86c-02
4.83e-06
4.64c-02
8.85e-05
8 32
7.87e-02
1.03e-06
7.29e-02
1.53e-06
7.05e-02
3.53e-06
7.12e-02
4.83e-06
6.74e-02
8.85e-05
Table 21: Memory vs. block size with active-set cap (m=p=5000). The s-1 row provides the per-experiment baseline.
RSS start (MiB)
RSS max (MiB)
RSS end (MiB)
754.7
757.0
757.0
2
754.7
759.3
759.3
754.7
762.4
762.4
8
754.7
768.5
768.5
16
754.7
780.7
780.7
32
754.7
795.9
788.3
64
754.7
795.9
788.3
1024
754.7
795,9
788.3
4096
754.7
795,9
788.3
Table 22: Memory vs. block size with cap, wide case (n=100, p=000,0000100,000, Peak memory remains
effectively constant across
5
RSS start (MiB)
RSS max (MiB)
RSS end (MiB)
755.3
756.1
741.6
2
755.3
756.1
741.6
4
755.3
756.
741.6
8
755.3
756.1
741.6
16
755.3
756.
741.6
32
755.3
756.1
741.6
64
755.3
756.
741.6
128
755.3
756.1
741.6
256
755.3
756.
741.6
512
755.3
756.1
741.6
1024
755.3
756.1
741.6
2048
755.3
756.1
741.6
4096
755.3
756.
741.6
32

---

Table 23:
Memory vs. block size with cap, tall case (n=100,000, p=100).
Time (s)
RSS start (MiB)
RSS max (MiB)
RSS end (MiB)
1 1.810
754.7
757.0
757.0
2
2.149
754.7
759.3
759.3
4
1.633
754.7
762.4
762.4
8
2.130
754.7
768.5
768.5
16
2.332
754.7
780.7
780.7
32
2.921
754.7
795.9
788.3
64
2.982
754.7
795.9
788.3
128
2.544
754.7
795.9
788.3
256
2.542
754.7
795.9
788.3
512
2.499
754.7
795.9
788.3
1024
2.786
754.7
795.9
788.3
2048
2,656
754.7
795.9
788.3
4096
2.508
754.7
795.9
788.3
Table 24: Memory vs. block size without cap (n=10, p-5000); the O(8²) term dominates for large
RSS start (MiB)
RSS max (MiB)
RSS end (MiB)
467.7
467.7
466.7
2
467.7
467.7
466.7
467.7
467.7
466.7
8
467.7
467.7
466.7
16
467.7
467.7
466.7
32
467.7
467.7
466.7
64
467.7
467.7
466.7
128
467.7
467.8
466.8
256
467.7
468.2
467.2
512
467.7
469.8
468.8
1024
467.7
475.8
474.9
2048
467.7
500.2
499.2
4096
467.7
596.8
596.0
Table 25: Memory vs. block size on the Duke dataset.
8
RSS start (MiB)
RSS max (MiB)
RSS end (MiB)
518.3
518.3
517.2
2
518.3
518.3
517.1
4
518.3
518.3
517.1
518.3
518.3
517.1
16
518.3
518.3
517.2
32
518.3
518.3
517.2
64
518.3
518.3
517.2
128
518.3
518.3
$17.2
256
518.3
518.3
517.2
512
518.3
518.3
517.2
1024
518.3
518.3
517.2
2048
518.3
518.3
517.2
4096
518.3
518.3
517.2
33

---

Table 26: GLMNET peak memory (excluding baseline),
Dataset
Memory (MiB)
N= N000, p=100,000
351.10
N = 1000,000, p=100
132.43
==5000, p=5000
322.50
N= 10, p=5000
19.47
Table 27: Poisson Performance: Time (s), relative error, and speedup under different a.
Elasame
Mechad
0.1
0.2
0.5
Time
Rel. Deff
Dift
Speedup
Time
Rel.
Speedup
Rel.
Time
Dift
glmnet
3.99=01
2.55-00
1.33-001
1.46:06
3.31es06
dukk
7.41e002
3.66c-06
093
5.42e-12
2.33-066
1,07
3.43-405
2.00:066
1.51
2.85e-02
3.52e-06
1.75
288e-022
3015e-06
1.65
10
S666-00
1.54--0
1.21
90--1
4.200405
1.27
2.89-402
2.51c-06
1.79
2,50-402
1.440006
1.89
3.54--00
3.06-166
1.84
plenet
S.636400
1.00
5400-07
1.00
4.96--2
1.00
4.49px00
1.00
58e-00
1.00
leukenia
2.42e
1.24e
012
1.1e
6.57
1,40
(12
9:74
18
4.646-00
2.686-06
3.22-00
2.0ee466
1.68
2.40-002
1.16606
2.18
1.14-00
2.0e-000
1.42e006
glanet
5.82-01
1.00
5.79c001
1,00 x
5.85c
1,00
5.91--01
1.00
6.0c-001
1,00
stroxy
8.39 02
9u0de 07
6.9
9.51e 02
1.2e
6,09
9.21e 02
2.56s 06
5,35
9dMe 02
5.41e 06
6.54
8.82 02
8.91e
6.85
10
8.00-00
9.10c-07
7.19
5.32-112
1.52-06
10.9
5.03--02
3.04--06
11.6
4 4.86x00
4.830006
12.2
64c-02
8.85--t5
13.0
34

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction state that the paper introduces an enhanced cyclic coordinate descent (ECCD) framework for elastic-net penalized GLMs, which achieves significant speedups over existing solvers while preserving convergence and accuracy. These claims are fully substantiated in the body: Section introduces the ECCD method with detailed derivation. Section (rriiionnpreeeettt presents theoretical guarantees, and Section shows empirical speedups of up to 13.0x across logistic and Poisson regression tasks without loss of accuracy. The limitations of block size and performance in owwdimensiional settings acknowledged in Section 6.2 ensuring that the claims reflect realistic expectations and generalization boundaries.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Section 6.2 discusses the practical trade-offs of block sizes and shows that ECCD may not yield speedups in low-dimensional or weakly regularized settings (e.g.. phishing dataset, high
Justification:
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they seale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
35

---

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [Yes]
Justification: The paper provides formal statements and corresponding proofs for all theoret- ical results. The main theoretical [Ssult-Theorm]]] quantifies the Taylor approximation emror and includes complete set of assumptions (bounded coefficients, residual norms, etc.) and a rigorous bound (Eq. 15 in the main text and extended in Eq. 32-33 in Appendix A.3). Supporting results (Theorems and 3 time and space complexity) are also clearly stated with derivations or intuitive justifications. Proofs for all claims are provided in the appendix with appropriate notation, bounding arguments, and references prior methods, thereby meeting the NeurIPS standard for completeness and correctness.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides detailed algorithmic descriptions sufficient for reproduc- tion. Algorithm 1 (p. 6) describes the ECCD procedure in active-set form, while Appendix A.2 offers a naurix-based variant with explicit update rules. Experimental settings, including datasets (e.g., LIBSVM, Bioconductor "airway"), regularization paths, convergence thresh- olds, block sizes, and machine specifications (Cray EX node with AMD EPYC 7763) are fully disclosed in Section Additionally, implementation details such as Taylor expansion approximations and screening rules are clearly explained. While code is not provided, the level of detail enables reproduction of both theoretical and empirical results that support the paper's main claims.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same
36

---

dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: All datasets used (e.g., LIBSVM benchmarks, Bioconductor "airway") are publicly available, and teh data is currently opensourced with link in abstract section. Read- ers can now directly access the authors' code with automated setup instructions. Meanwhile, the ECCD includes all details for reproduction, and the readers were able to reimplement ECCD from the detailed pseudocode and experimental descriptions.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
haaii///uuides//deeuubbi.i.e son.soooiisssiiooooocoooo. for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
7/
CodeeSumiiss iss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
37

---

Answer: [Yes]
Justification: Section[deescriin the datasets used (LIBSVM benchmarks and Bioconductor airway), the regularization path setup (Amax emmax with {10-1, 10-- TI = 100), convergence tolerance (10-7 on deviance change), block-size grid (s E (1, 2, 8, 16.32)) and screening/active-stt strategy (strong rules and KKT checks). Implementation details include R v4.3.1, gimnet v4.1-8, ReppEigen, and machine specs (Cray EX with AMD EPYC 7763). These choices fully specify the experimental protocol and hyperparameters needed to reproduce the main results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: While our core scalability experiment results are presented as multiple-run averaged values, reported results are presented without accompanying error bars, confidence intervals, or statistical tests. The paper does not describe repeated trials, variance sources, or the method used to quantify experimental variability, Consequently, statistical significance the observed performance differences is not assessed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall
run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: S reports that all experiments ran on single Cray EX node equipped with detailed information of computation resources. Individual runtimes for each dataset and configuration are listed in Tables 4 27 enabling reproduction of both resource setup
and expected execution times.
38

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics hattps://neurip:: httppuulicc/hhhcsuuudeeeee
Answer: [Yes]
Justification: The work focuses on algorithmic development for generalized linear models and does not involve human or animal subjects, sensitive data, or applications with potential for misuse. There are no conflicts of interest ethical concerns raised by the experiments or methods. All authors have reviewed and comply with the NeurIPS Code of Ethics. Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: This work is foundational research optimization algorithms for generalized linear models and does not target a specific application domain, sensitive data, or deployment scenario. As such, there are no direct societal
or negative-to discuss.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
39

---

11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not release any high-risk models, sensitive datasets, or genera- tive systems. It focuses algorithmic improvements for standard generalized linear models using public, non-sensitive benchmark data, so no additional safeguards are necessary.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question:
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [No]
Justification: The paper cites all external tools and datasets used (e.g., glmnet 2 LIBSVM (33), including package versions, but it does not state the license names URLs for these assets, nor confirm compliance with their terms of use.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URI
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, saperswithcode com /atasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- se-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not introduce or release any new datasets, code repositories, or models; it focuses the ECCD algorithm and evaluates on existing public benchmarks.
Guidelines:
The answer NA means that the paper does not release new assets.
40

---

Researchers should communicate the details of the dataset/code/mmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to amonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The work is purely algorithmic and empirical on public standard datasets; it does not involve any human participants or crowdsourcing.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve any experiments with human subjects or crowd- sourcing; focuses entirely algorithmic development and evaluation using public bench- mark datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the L.M used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core contributions focus on the development and analysis of the ECCD optimization algorithm for generalized linear models and do not involve any use of large language models.
41

---

Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy for what should or should not be described.
42


Reviews:
[{"review_id": "7IuBBe2Ph9", "reviewer": "Reviewer_uQeX", "summary": "This paper proposes Enhanced Cyclic Coordinate Descent, an efficient block-wise optimization method for fitting Elastic Net penalized generalized linear models. ECCD improves upon naive block coordinate descent by incorporating second-order Taylor corrections to stabilize updates within each block. The authors provide theoretical convergence guarantees and complexity analysis, and demonstrate through experiments on synthetic and real-world datasets that ECCD achieves faster convergence than existing methods.", "questions": "1. Could the authors provide a more detailed theoretical analysis of the space complexity of ECCD, isolating the additional memory terms introduced by the algorithm (beyond storing the design matrix $X$)? It would also be helpful to include corresponding empirical results comparing memory usage across methods.\n\n2. In Section 6.1, the authors evaluate ECCD with block sizes $s$ ranging from 2 to 32 on the duke dataset. However, according to Table 6, the number of features is $p = 7{,}129$, which makes even the largest tested block size relatively small. Would it be more informative to explore larger values of $s$ to better assess the performance?", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n1. The proposed ECCD method improves over naive block coordinate descent in GLMs by introducing a Taylor-approximated correction, resulting in a stable and efficient algorithm.\n\n2. The paper provides both theoretical convergence guarantees and practical complexity analysis, offering a clear justification for the expected computational benefits.\n\n3. The experimental results are well-structured, demonstrating ECCD’s stability and speed advantages across many standard datasets, and showing that it achieves the same solution quality as established solvers like glmnet.\n\n**Weaknesses**\n1. The space complexity analysis in Theorem 3 states an overall $\\mathcal{O}(np)$ bound, dominated by the design matrix $X \\in \\mathbb{R}^{n \\times p}$. However, this masks the additional memory overhead introduced specifically by ECCD. A clearer breakdown of the extra storage costs incurred by ECCD, excluding the fixed cost of storing $X$, would provide better insight into the practical memory demands of the algorithm.\n\n2. While the paper provides extensive experiments measuring actual running time, it does not report experimental results on memory usage. Given that the space complexity analysis in Theorem 3 abstracts away the extra storage introduced by ECCD, it would be valuable to include experimental comparisons of memory consumption across methods to more clearly demonstrate the practical efficiency of ECCD relative to baseline algorithms.\n\n3. Minor issues: \n- There is a dangling sentence in the caption of Figure 1.", "was_revised": true, "final_justification": "My main concerns were related to the lack of explicit storage analysis and the relatively small block sizes $s$ used in the Duke dataset experiment. The authors have addressed both points well: they provided a clearer theoretical explanation of ECCD's memory costs and supplemented the paper with comprehensive experimental results. Additionally, the extended experiments with larger block sizes offer a more complete picture of ECCD's behavior under aggressive partitioning. These clarifications and additions adequately resolve my concerns. I will maintain my recommendation for acceptance.", "rebuttal": "Thank you for your time and detailed comments. We provide responses to weaknesses and questions below. We provide benchmarks on representative datasets (real and synthetic), where the data generation process for synthetic data is described in the response to Reviewer NJYD.\n\n### Weakness\n\n**W1.** *Theorem 3 (Appendix A.5)* provides space complexity analysis which includes low order terms, including the additional memory required by ECCD beyond the storage cost of the design matrix. Specifically, ECCD allocates $\\mathcal{O}(ns)$ memory for the working buffer of the active block $X_s$, and $\\mathcal{O}(s^2)$ for the corresponding Gram matrix. These bounds can be specialized to glmnet by setting $s = 1$. The additional memory overhead of ECCD remains negligible in high-dimensional settings, with relative cost $\\frac{ns}{np} \\to 0$ as $p \\gg s$. This is also empirically supported, as discussed in our response to the second question.\n\nImportantly, the ideal value of $s$ is a small constant that is approximately independent of $n$ and $p$, and reflects the per-coordinate cost of evaluating the link function, as characterized in Theorem 2. Thus, assuming $p \\gg s$ is both theoretically justified and reasonable in practice. If desirable, we can incorporate Theorem 3 from Appendix A.5 into the main text.\n\n\n\n**W2.** To address this, we have rerun our benchmarks and now include peak RSS (resident set size) measurements for ECCD as the block size $s$ varies. These experiments serve to empirically validate our theoretical space-complexity analysis in Theorem 3 (Appendix A.5).\n\nIn our analysis, the extra storage required by ECCD consists of $s$ vectors of length $n$ (the working block $X_s$) plus the $s\\times s$ Gram matrix, $X_s^\\intercal X_s$. In extreme case where $s \\approx p$, the Gram matrix storage cost ($O(s^2)$ term) could dominate; however, since we incorporate the active-set method similar to glmnet, this cost is smaller than storing $X$ and the active block $X_s$, itself. Our approach for $s$ is to bound it to $\\sqrt{|\\mathcal{A}|}$, where $\\mathcal{A}$ is the active set in the current iteration. This ensures that further increases in $s$ do not increase memory usage. The memory overhead of ECCD remains negligible compared to the cost of storing the original data matrix.\n\nFor lambda path, by reusing the previous lambda solution as a warm start along the path, the RSS memory grows only until $s$ reaches square root of the largest active set in the lambda path.\n\nTable 1 shows Memory vs. $s$ experiments for a synthetic $n=p=5000$ matrix. We instrumented our C++ code to obtain memory stats via `getrusage()`. The baseline RSS is $754.7$ MiB for $s = 1$ (i.e. equivalent to glmnet); At the largest value of $s=4096$, peak RSS increases by $41.2$ MiB when compared to $s = 1$. These results demonstrate that ECCD's memory footprint increases slowly with the primary cost being the original design matrix storage, even for large block sizes. If $s$ is allowed to exceed the active-set upper bound, the $O(s^2)$ term can dominate and increase memory usage substantially. We show experiments without active set to illustrate the memory trade off.\n\n**Table 1: Memory vs. block size ($n=p=5000$, with active-set constraint on $s$, baseline RSS = 754.7 MiB)**\n\n|s|Rss start (MiB)|Rss max (MiB)|Rss end (MiB)|\n|-|-|-|-|\n|1|754.7|757.0|757.0|\n|2|754.7|759.3|759.3|\n|4|754.7|762.4|762.4|\n|8|754.7|768.5|768.5|\n|16|754.7|780.7|780.7|\n|32|754.7|795.9|788.3|\n|64|754.7|795.9|788.3|\n|1024|754.7|795.9|788.3|\n|4096|754.7|795.9|788.3|\n\n\nTable 1 summarize the case where the block is capped at the squared root of active set size $|\\mathcal{A}|$, so memory usage rises only modestly with $s$ and plateaus once $s$ exceeds the $\\sqrt{|\\mathcal{A}|}$ upperbound when $n$ is moderately significant.\n\n**Table 2: Memory vs. block size (\\$n=p=5000\\$, without active-set constraint on $s$, baseline RSS = 809.6 MiB)**\n\n|$s$|Rss start (MiB)|Rss max (MiB)|Rss end (MiB)|\n|-|-|-|-|\n|1|1000.4|1000.4|1000.4|\n|2|1000.4|1000.4|1000.4|\n|4|1000.4|1000.4|1000.4|\n|8|1000.4|1000.4|1000.4|\n|16|1000.4|1000.4|1000.4|\n|32|1000.4|1000.4|1000.4|\n|1024|1000.4|1079.3|1079.3|\n|2048|1000.4|1142.5|1142.5|\n|4096|1000.4|1316.5|1316.5|\n\nIn Table 2, we removed the active set cap $|\\mathcal{A}|$. Without the active set cap $|\\mathcal{A}|$, the $\\mathcal{O}(s^2)$ component dominates for large $s$, leading to a substantial rise in peak memory as block size increases. At $s=4096$ peak RSS is 1316.5 MiB, compared to 1000.4 MiB at $s = 1$ (an increase of $316.1$ MiB). Without active set, the storage cost of $X_s$ and the Gram matrix $X_s^\\intercal X_s$ are not negligible. However, this is an impractical setting for performance reasons as glmnet incorporates active set to improve performance/efficiency. Similarly, the fastest variant of ECCD is one where active set is included.\n\n**Table 3: glmnet peak memory**\n\n|Dataset|Memory (MiB)|Base Memory (MiB)|\n|-|-|-|\n|n = 100, p = 100000|953|601.9|\n|n = 100000, p = 100|804.7333|672.3|\n|n = 5000, p = 5000|1076.9|754.7|\n|n = 10, p = 5000|487.16667|467.7|\n\n\nIn Table 3 shows peak RSS memory for glmnet using **peakRAM** R package. Because glmnet is implemented in optimized C++, we did not directly instrument the C++ code similar to ECCD, as this would require source code changes and building glmnet from the instrumented source code. We provide Table 3 to give a general sense in memory usage of glmnet (without instrumented C++ code). ECCD with $s = 1$ is equivalent to glmnet, so we expect the actual peak RSS memory to be similar to $s = 1$. Overall, these results confirm that ECCD's practical memory overhead is negligible—even for large $s$—with the largest memory cost arising from storing the original data matrix.\n\nWe provide additional RSS memory usage below under different settings:\n\n**Table 4: Memory vs. block size ($n=100$, $p=100\\,000$; with active-set constraint on $s$, baseline RSS = 601.9 MiB)**\n|s|Rss start (MiB)|Rss max (MiB)|Rss end (MiB)|\n|-|-|-|-|\n|1|755.3|756.1|741.6|\n|2|755.3|756.1|741.6|\n|4|755.3|756.1|741.6|\n|8|755.3|756.1|741.6|\n|16|755.3|756.1|741.6|\n|32|755.3|756.1|741.6|\n|64|755.3|756.1|741.6|\n|128|755.3|756.1|741.6|\n|256|755.3|756.1|741.6|\n|512|755.3|756.1|741.6|\n|1024|755.3|756.1|741.6|\n|2048|755.3|756.1|741.6|\n|4096|755.3|756.1|741.6|\n\nTable 4 shows RSS memory for $p \\gg n$ with active set.\n\n**Table 5: Memory vs. block size ($n = 100000$, $p = 100$, with active-set constraint on $s$, baseline RSS = 672.3 MiB)**\n|$s$|Time (s)|RSS start (MiB)|RSS max (MiB)|RSS end (MiB)|\n|-|-|-|-|-|\n|1|1.810|754.7|757.0|757.0|\n|2|2.149|754.7|759.3|759.3|\n|4|1.633|754.7|762.4|762.4|\n|8|2.130|754.7|768.5|768.5|\n|16|2.332|754.7|780.7|780.7|\n|32|2.921|754.7|795.9|788.3|\n|64|2.982|754.7|795.9|788.3|\n|128|2.544|754.7|795.9|788.3|\n|256|2.542|754.7|795.9|788.3|\n|512|2.499|754.7|795.9|788.3|\n|1024|2.786|754.7|795.9|788.3|\n|2048|2.656|754.7|795.9|788.3|\n|4096|2.508|754.7|795.9|788.3|\n\nTable 5 shows RSS memory for $n \\gg p$ with active set.\n\n**Table 6: Memory vs. block size (Duke dataset, with active-set constraint on $s$, base = 510.5 MiB)**\n|s|Rss start (MiB)|Rss max (MiB)|Rss end (MiB)|\n|-|-|-|-|\n|1|518.3|518.3|517.2|\n|2|518.3|518.3|517.1|\n|4|518.3|518.3|517.1|\n|8|518.3|518.3|517.1|\n|16|518.3|518.3|517.2|\n|32|518.3|518.3|517.2|\n|64|518.3|518.3|517.2|\n|128|518.3|518.3|517.2|\n|256|518.3|518.3|517.2|\n|512|518.3|518.3|517.2|\n|1024|518.3|518.3|517.2|\n|2048|518.3|518.3|517.2|\n|4096|518.3|518.3|517.2|\n\n\nTable 6, shows RSS memory for the Duke dataset with active set. Tables 4 - 6 illustrate that memory usage for large $s$ is not significant when using the active set method.\n\n\n**W3.** This error has been fixed.\n\n### Questions:\n\n**Q1.** Please see our responses under Weakness 1 & 2 which includes a discussion of other memory costs (low order terms) from Theorem 3 Appendix A.5 and additional experiments to support our analysis.\n\n\n**Q2.** We appreciate the reviewer's suggestion to evaluate larger block sizes. In response, we conducted additional experiments on the Duke dataset with block sizes up to $s = p = 7129$; the results are summarized in Table 8 below.\n\nECCD remains stable and convergent even when $s = p$, underscoring the robustness of our second‑order Taylor correction and its advantage over naive BCD (see Table 3 in the main paper).\n\n**Interpretation of Table 8.**\n\n* **Column 2 (Active Set)**: We employ an active‑set strategy where updates are restricted to variables with nonzero coefficients, and use KKT conditions to govern active set expansion. When the block size $s$ exceeds the active-set size $|\\mathcal{A}|$, we dynamically set $s \\leftarrow \\sqrt{|\\mathcal{A}|}$ to improve efficiency.\n* **Column 3 (No Active Set)**: We turn the active-set feature off so that the block size, $s$, remains fixed at the initial setting. We report accuracy using relative objective difference.\n\nThe optimal setting for $s$ is typically a small constant determined by the cost of the link function and hardware features (e.g. DRAM memory bandwidth/latency, cache size, processor speed) and, as illustrated in Table 8, not due to numerical stability or accuracy. Our experiments in the main paper yielded $s \\in [2, 32]$ as the most performant, so the additional memory footprint is bounded with performance (and not accuracy) being the upper bound on the best value of $s$. The large‑$s$ results will be included in the camera-ready version for completeness.\n\n**Table 8. ECCD Convergence Behavior under Fixed and Adaptive Block Sizes ($s$) on the Duke Dataset**\n\n|Block Size (s)|Rel. Obj. Diff (Active Set)|Rel. Obj. Diff (No Active Set)|\n|-|-|-|\n|1|1.38e-04|1.38e-04|\n|16|8.025e-05|9.810e-05|\n|64|8.674e-05|1.598e-04|\n|256|8.674e-05|1.240e-04|\n|2048|8.674e-05|1.218e-04|\n|7129|8.674e-05|1.470e-04|"}, {"review_id": "5j1sz1uFxN", "reviewer": "Reviewer_ARDF", "summary": "This paper proposes an enhanced cyclic coordinate descent (ECCD) algorithm for generalized linear models with elastic net penalty. By using Taylor expansion to approximate gradients and unrolling vector recurrences into batched computations, ECCD avoids the unstable behavior of the regular BCD algorithm and successfully enhances computation. Implemented in C++ with RcppEigen, ECCD generally achieves 1–3 times speedup over the baselines on LIBSVM benchmark datasets.", "questions": "- Can this method be extended to (Group) SCAD and MCP, when the group of large size?", "limitations": "This work focuses on optimization algorithms for elastic net in generalized linear models, without targeting specific application domains or sensitive data scenarios. Therefore, it does not have direct negative societal impacts.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nOverall, the paper is well-organized and well-motivated. The mathematical derivation looks solid, and the experiments are comprehensive.\n\n**Weaknesses**\n\n- The authors omit some recent important libraries, e.g., skglm [1] and abess [2].\n\n- The result (14) in Theorem 1 is somewhat over-simplified. In the worst case where $s=p$, the approximation error can be expected to be excessively large, whereas $s=1$ yields zero error. However, the trade not reflected in the current results.\n\n- Numerical results for the lambda path (a typical case) are overstated. For example, the 13× improvement only appears in the airway dataset with one \\(\\alpha\\). Notably, airway has an extremely small sample size ($n=8$), which is atypical for real-world data analysis. In most regular datasets, the improvement ranges from 1.0 to 3.0.\n\n- I am a little confused about why biglasso is so slow. As shown on this page, it is significantly faster than glmnet in memory-intensive cases. Additionally:\n    + How many cores did biglasso use? Since the hardware has 64 cores, using 16 cores could be considered.\n    + I found the current datasets are either large $p$ or large $n$? What is the performance when both $n$ and $p$ is large (e.g., 10k).\n\n- ECCD uses a diagonal Hessian for approximation, while glmnet computes the exact Hessian matrix by default. Would glmnet perform similarly if it used an upper bound on the Hessian?\n\n- Typos in line 414: min{n, , p} --> min{n, p}\n\n[1] Quentin Bertrand, Quentin Klopfenstein, Pierre-Antoine Bannier, Gauthier Gidel, and Mathurin Massias. 2022. Beyond L1: faster and better sparse models with skglm. NeurIPS '22. Curran Associates Inc., Red Hook, NY, USA, Article 2823, 38950–38965.\n\n[2] Zhu Jin, Xueqin Wang, Liyuan Hu, Junhao Huang, Kangkang Jiang, Yanhang Zhang, Shiyun Lin, and Junxian Zhu. \"abess: A Fast Best-Subset Selection Library in Python and R.\" Journal of Machine Learning Research 23, no. 202 (2022): 1-7.", "was_revised": true, "final_justification": "The authors have added some necessary benchmarks including additional baselines and setting. Although I still feel the theoretical results is not tight, these comprehensive experiments already shown the potential of the method. Therefore, I increase the score  of this paper.", "rebuttal": "Thank you for your time and detailed comments. We provide responses to weaknesses and questions below. We provide benchmarks on representative datasets (real and synthetic), where the data generation process for synthetic data is described in the response to Reviewer NJYD.\n\nWe also appreciate the reviewer for pointing out the references, we will add the below citations to the paper.\n\n**References**\n* [1] J. Qian et al., *skglm: Scikit-Learn compatible sparse-GLM solver*, 2023.\n* [2] Y. Wang et al., *abess: Best-subset Selection Library*, 2023.\n* [3] Breheny & Huang., *Group descent algorithms for nonconvex penalized linear\nand logistic regression models with grouped predictors*, 2015.\n\n\n### Weaknesses\n\n**W1.**\nWe thank the reviewer for pointing out **skglm** and **abess**.  Below, we explain why direct comparison is limited with empirical experiment results.\n\n1. **skglm**\n\n   * Implements pathwise coordinate-descent for linear regression in Python.\n   * **Does not support pathwise GLMs** (logistic, Poisson, etc.), so it is not directly comparable to ECCD in those settings.\n   * We benchmarked ECCD against the single-point logistic regression algorithm from skglm, evaluating both methods along the penalty path derived from ECCD on the Duke dataset, with default error tolerance matched to that used by ECCD. When $s=4$, ECCD ran in 0.013 s versus skglm's 1.375 s.\n   * More results are presented in Table 1.\n2. **abess**\n   \n   * The **best-subset selection** problem is characterized by an explicit $\\ell_0$ constraint, which directly limits the number of nonzero coefficients. fundamentally different from the $\\ell_1$ or $\\ell_2$ regularized problems (Lasso, Elastic Net) we study.\n   * abess's claimed advantage arises in high-correlation regimes.  We tested under $\\rho=0.9$ for both small $(n=20,p=2000)$ and moderate $(n=100,p=1000)$ settings and found abess is still slower than ECCD.\n   * Under the exact setting, we found that Abess perform poorly on relatively large dataset (i.e. n = 1000)\n   * On larger-scale datasets (e.g., $n=100000$), Abess's performance degrades substantially, yielding slower runtimes than ECCD under identical conditions.\n\n\n**Table 1: ECCD vs. skglm (time in seconds)**\n\n| Dataset              |  skglm  | ECCD (s = 4) |\n| :------------------- | :-----: | :----------: |\n| **Duke**             |  1.375  |     0.013    |\n| **Diabetes**         |  6.155  |     0.002    |\n| n = 100, p = 100000  | 131.340 |     1.426    |\n| n = 100000, p = 100  | 213.012 |     1.032    |\n\n\n**Table 2: ECCD vs. glmnet vs. abess (time in seconds)**\n\n|Dataset|glmnet|abess|ECCD(s=4)|\n|:------|:----:|:---:|:-------:|\n|n=20,p=2000,ρ=0.1|0.014|0.011|0.002|\n|n=20,p=2000,ρ=0.9|0.018|0.011|0.002|\n|n=2000,p=20,ρ=0.1|0.030|0.426|0.013|\n|n=2000,p=20,ρ=0.9|0.047|0.493|0.027|\n|n=100,p=1000,ρ=0.1|0.028|0.026|0.005|\n|n=100,p=1000,ρ=0.9|0.066|0.028|0.006|\n|n=1000,p=100,ρ=0.1|0.021|0.131|0.007|\n|n=1000,p=100,ρ=0.9|0.026|0.128|0.006|\n|n=100000,p=100,ρ=0.1|2.384|904.368|1.426|\n|n=100000,p=100,ρ=0.9|3.840|931.303|1.032|\n|**Duke**|0.022|0.025|0.013|\n\n\nNOTE: we attempt to run n = 100, p = 100000 with rho = 0.1, 0.9 on cluster, but the data generation process took over 12 hours.\n\n\n\n**W2.** Thank you for pointing this out. Equation (14) in Theorem 1 is indeed a simplified bound derived from a Taylor expansion. It is primarily intended to offer theoretical intuition rather than a tight bound in all scenario. Nevertheless, our algorithm is designed to ensure that the relative approximation error remains small in practice. Specifically, we adopt a convergence criterion based on the relative change in the objective function, which is identical to the stopping rule used in glmnet. Regardless of the block size ($s$), the algorithm proceeds until the objective value achieves the specified threshold. This design guarantees that the final objective value remains consistently low for both small and large $s$, which accounts for the stable performance across block sizes observed in our experiments.\n\n\n\n**W3.** We thank the reviewer for this observation and will modify our claims in the abstract and paper body to reflect the more realistic speedups of $1 - 3\\times$.\n\n\n\n**W4.1.** To investigate, we ran BigLasso on one node of the Cray EX Cluster used in the main paper. We experimented with using 2, 4, 8, 16, 32, 64, and 128 threads, and compared against our single-thread ECCD.  Across a range of datasets and both full and short $\\lambda$-paths, ECCD's active-set screening and path-wise updates yielded consistent performance advantage even versus BigLasso's multi-threaded solver.  We report representative results below.\n\n**Table 3: Runtime on glmnet's default path ($\\lambda_{\\rm max} \\to \\lambda_{\\rm max} \\times 10^{-4}$ or $10^{-2}$, 100 points)**\n\n|Dataset|glmnet|ECCD(s)|BigLasso|\n|:------|:----:|:-----:|:------:|\n|Duke|0.117s|0.012s(s=8)|0.215s(8 threads)|\n|10000$\\times$10000|44.204s|31.644s(s=8)|122.114s(64 threads)|\n|100$\\times$100000|0.937s|0.640s(s=4)|1.198s(16 threads)|\n|100000$\\times$100|1.862s|0.767s(s=8)|4.372s(2 threads)|\n\n\n\n**Table 4: Runtime on BigLasso's default path ($\\lambda_{\\rm max} \\to \\lambda_{\\rm max} \\times 10^{-1}$, 100 points)**\n\n|Dataset|glmnet|ECCD(s)|BigLasso|\n|:------|:----:|:-----:|:------:|\n|Duke|0.076s|0.010s(s=8)|0.197s(8 threads)|\n|10000$\\times$10000|4.115s|2.250s(s=4)|1.419s(64 threads)|\n|100$\\times$100000|0.942s|0.709s(s=4)|0.890s(8 threads)|\n|100000$\\times$100|1.033s|0.484s(s=16)|1.519s(64 threads)|\n\n\n**Key observations:**\n\n1. **Sequential ECCD outperforms** BigLasso even when it is parallelized using 2–64 threads.\n2. **glmnet's default path** incurs more work, yet ECCD's optimized screening keeps it faster.\n3. **BigLasso's default path** still favor ECCD for moderate block sizes ($s\\le16$).\n\nThis confirms that ECCD's pathwise, active-set strategy provides a fundamental performance advantage. We plan to address parallel algorithms design for ECCD in shared- and distributed-memory environments in future work. Once parallelized, we believe that ECCD will continue to outperform BigLASSO.\n\n\n\n**W4.2.**\nWe include additional experiments when $n$ and $p$ are large (e.g.  $10^4$) below, comparing three solvers (glmnet, glmnet with Modified Newton, and ECCD) on total runtime:\n\nAs $n=p$ increases from 5 000 to 70 000:\n\n* **glmnet** grows by roughly $30.79\\times$ (15.0 s $\\to$ 461.6 s),\n* **glmnet w. Modified Newton** grows by roughly $6.73\\times$ (57.2 s $\\to$ 384.8 s), and\n* **ECCD** grows by roughly $18.7\\times$ (13.8 s $\\to$ 258.3 s).\n\nMoreover, we observed the relative speedup:\n\n* At $n=p=40,000$, ECCD (69.4 s) is $\\sim1.87\\times$ faster than glmnet (129.9 s) and outperform modified case (195.2 s).\n* Even at the largest size, $n=p=70\\,000$, ECCD (258.3 s) maintains a $\\sim1.5\\times$ speedup over glmnet (384.8 s) and a $\\sim1.8\\times$ advantage over Modified Newton (461.6 s).\n\nThese results show that ECCD's active-set screening and block-coordinate updates scale much more efficiently in large, dimension-balanced problems. \n\nAll scaling experiments reported here were conducted on an AMD EPYC 7352 CPU using a single thread, as noted in our response to Reviewer NJYD's W1.\n\n**Table 5: Synthetic $n = p$ Benchmarks on Cluster**\n\n| Dataset (n = p) | glmnet (Newton) (s) | glmnet (Modified Newton) (s) | ECCD Time (s) (s) | glmnet / ECCD | modified / ECCD |\n|:----------:|:-----------:|:-------------------------:|:------------:|:--------:|:------------:|\n|5000×5000|14.991|57.162|13.785(4)|1.09×|4.14×|\n|10000×10000|75.791|225.034|56.939(8)|1.33×|3.95×|\n|20000×20000|29.347|50.391|15.889(4)|1.85×|3.17×|\n|40000×40000|129.915|195.170|69.432(8)|1.87×|2.81×|\n|50000×50000|244.209|241.476|102.168(16)|2.39×|2.36×|\n|70000×70000|461.645|384.758|258.302(32)|1.79×|1.49×|\n\n\n**W5.** \n\n**ECCD's approach:** ECCD does not rely on a diagonal approximation of the Hessian. As shown in Equation (11) of the main paper, we incorporate *all entries of the true Hessian* to form a second‑order approximation of the loss. This full‑matrix update more faithfully captures curvature, yielding more accurate coordinate steps and leads to faster convergence overall.\n\n**glmnet's options:** The `glmnet` package does offer a diagonal Hessian variant (e.g. via `type.logistic = \"modified-newton\"` for logistic regression)—which is claimed in the documentation to be a more efficient method. However, we evaluated this variant on large-scale datasets, and our experiments (see Table 5 above) show that when both $n$ and $p$ are large (e.g. $10^4$), this diagonal variant suffers noticeable degradation in both runtime and solution quality. By contrast, ECCD's full Hessian updates remain robust and efficient.\n\n**W6.** Thanks for point this out, we have corrected this.\n\n\n### Questions\n\n**Q1.** Yes, ECCD can be naturally extended to other penalties that are compatible with coordinate descent based optimization. And prior work by Breheny and Huang (2015) [3] has shown that both SCAD and MCP can be efficiently optimized using coordinate descent frameworks.\n\nWhile these penalties primarily address nonconvex regularization, our method is designed to handle the *nonlinearity in the loss function*. ECCD focuses on accelerating coordinate descent by accurately approximating the nonlinear loss via second-order updates, and is agnostic to the specific form of the penalty.\n\nTherefore, as long as the penalty admits coordinate-wise or block-wise updates, our ECCD framework can be applied. In our work, we adopt the $\\ell_1$ penalty for simplicity and clarity of exposition, but the method naturally generalizes to other penalties."}, {"review_id": "rUxcsNmDns", "reviewer": "Reviewer_NJYD", "summary": "The paper introduces enhanced cyclic coordinate descent (ECCD), a variant of coordinate descent for fitting $\\ell_1+\\ell_2$ generalized linear models (GLMs). The algorithm approximates gradients via  Taylor expansion and updates $s$ coordinates as a block. Standard coordinate descent is recovered when $s=1$. The advantage of the new algorithm is a computational speedup on standard coordinate descent, which the paper demonstrates theoretically and empirically.", "questions": "See above", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Given the importance of $\\ell_1+\\ell_2$ GLMs (evidenced by the sheer popularity of glmnet), this direction of research is attention worthy, and I personally like the proposed algorithm. However, I have a few major concerns that in my opinion prevent the paper from being accepted in its current form:\n\n1. It is clear that the method typically provides a speedup over existing methods. However, all the speedups are on very small wall clock times (mostly less than one second). Are such speedups practically meaningful in the vast majority of use cases? In my opinion, a more compelling story would be speedups in situations where existing methods take, e.g., 1, 10, or 100 minutes to run. This issue is my most serious concern.\n\n2. The main limitation of the method is that the optimal block size $s$ is unknown in practice. The authors provide a theoretical choice of $s$, but it depends on unknown quantities and hence is not applicable in practice. If one tries to tune the block size the method is possibly no longer fastest. The authors give $s=8$ as a rule of thumb, but this choice seems a rough approximation in general. The paper should acknowledge this drawback clearly.\n\n3. Do the baselines all use the same convergence criterion? Looking at glmnet, its convergence criterion seems to be based on the change in objective values. Meanwhile, convergence in ncvreg is based on the change in coefficients. Is it really meaningful to compare runtimes on algorithms that do not share the same stopping rule?\n\n4. There is no code available for (a) the method or (b) the experiments. The author's justification is that it is straightforward to replicate the results, but I disagree given that much of the code is in C++. The lack of code severely limits the papers potential impact in my opinion. Why would someone go to the effort of implementing this method when they can use glmnet, which for most practical purposes runs in the same time?", "was_revised": true, "final_justification": "The new runtimes reported in the rebuttal address my most critical concern of the paper about demonstrating meaningful speedups.", "rebuttal": "## Reviewer 3\n\nThank you for your time and detailed comments. We provide responses to weaknesses and questions below.\n\n\n**W1.**\n\nWe thank the reviewer for suggesting large-scale benchmarks. To ensure a fair comparison, we ran both glmnet and ECCD with **identical convergence criteria and path settings**:\n\n* **Tolerance:** `tol = 1e-7`.\n* **Lambda path:** `lambda.min.ratio = if n < p then 0.1 else 1e-4`, `nlambda = 100`; 0.1 used as \\$p\\$ is extremely large in this response's big dataset experiment.\n* **Standardization:** `standardize = TRUE`.\n* **Logistic Hessian:** `type.logistic = \"newton\"` (exact Hessian) and `type.logistic = \"modified-newton\"` (diagonal approximation) both tested, with `intercept = TRUE`, as GLMNet suggests the latter can be faster.\n\n**Synthetic Datasets Generation Process**\n\nThe synthetic datasets are generated through the following steps. We begin by sampling a design matrix $X \\in \\mathbb{R}^{n \\times p}$ from a multivariate normal distribution. Next, we impose sparsity on the true coefficient vector $\\boldsymbol{\\beta}$ with the intercept $\\beta_0$, so the dataset fit in with elastic net problems. Then, the linear predictor is computed by adding Gaussian noise $\\boldsymbol{\\varepsilon}$ drawn from a multivariate normal distribution:\n$$\n\\boldsymbol{\\eta} = X\\boldsymbol{\\beta} + \\beta_0 + \\boldsymbol{\\varepsilon}.\n$$\nFinally, each $\\eta_i$ is transformed using the logistic link function: $p_i = \\frac{1}{1 + e^{-\\eta_i}},$\nand the binary response variables are sampled as $y_i \\sim \\mathrm{Bernoulli}(p_i)$.\n\nUsing these settings, we ran wall-clock timings on an AMD EPYC 7352 CPU across three synthetic regimes where the number of observations is less than, greater than, or equal to the number of features.  The results (reported in the main text) demonstrate ECCD's consistent speedups even when GLMNet runs for minutes or longer.\n\n| Regime    | Dataset ($n \\times p$)     | glmnet (Newton) (s) | glmnet (Modified Newton) (s) | ECCD (s) (block $s$) | Speedup (Newton) | Speedup (Modified) |\n|-----------|----------------------------|----------------------|-------------------------------|-----------------------|-------------------|---------------------|\n| **$n < p$** | 1,000 $\\times$ 1,000,000        |           79.597            |                85.067               |           76.485 ($16$)            |          1.05 $\\times$         |            1.11 $\\times$         |\n|           | 1,000 $\\times$ 10,000,000        |          1107.444            |                788.616              |           798.350 ($32$)            |        1.39 $\\times$           |            0.99 $\\times$         |\n|           | 5,000 $\\times$ 1,000,000        |           474.164            |                377.016               |           215.748 ($8$)            |        2.20 $\\times$           |           1.75 $\\times$          |\n|           | 10,000 $\\times$ 50,000        |          41.905            |                49.236               |           23.131 ($8$)            |        1.81 $\\times$           |          2.13 $\\times$           |\n| **$n > p$** | 10,000 $\\times$ 50           |           0.227           |              0.213                 |            0.149 ($4$)           |          1.52 $\\times$         |           1.43 $\\times$          |\n|           | 10,000 $\\times$ 500           |          2.220            |              3.155                 |           1.959 ($4$)            |          1.12 $\\times$         |          1.61 $\\times$           |\n|           | 10,000 $\\times$ 5,000         |          63.261            |              257.35                 |            36.129 ($8$)           |        1.75 $\\times$           |          7.12 $\\times$           |\n|           | 100,000 $\\times$ 5,000      |         495.636             |              727.013                 |          315.341 ($4$)             |          1.57 $\\times$         |         2.31 $\\times$            |\n|           | 1,000,000 $\\times$ 5,000      |         2633.412             |              2668.265                 |          1632.232 ($4$)             |   1.61 $\\times$ | 1.63 $\\times$\n| **$n = p$** | 5,000 $\\times$ 5,000          |           14.991           |               57.162                |        13.785 ($4$)               |        1.09 $\\times$           |           4.14 $\\times$          |\n|           | 10,000 $\\times$ 10,000        |           75.791           |            225.034                   |            56.939 ($8$)           |         1.33 $\\times$          |          3.95 $\\times$           |\n|           | 20,000 $\\times$ 20,000        |            29.347          |                 50.391              |         15.889 ($4$)              |           1.85 $\\times$        |           3.17 $\\times$          |\n|           | 40,000 $\\times$ 40,000        |            129.915          |              195.170                 |          69.432 ($8$)             |         1.87 $\\times$          |         2.81 $\\times$            |\n|           | 50,000 $\\times$ 50,000        |            244.209          |             241.476                  |          102.168 ($16$)                                |       2.39 $\\times$    | 2.36 $\\times$          \n|           | 70,000 $\\times$ 70,000        |            461.645          |             384.758                  |          258.302 ($32$)             |        1.79 $\\times$ | 1.49 $\\times$\n\n**Interpretation:**\n\n* **Feature-dominated ($n<p$)**: ECCD cuts runtime by $\\sim\\\\!1.8\\text{–}2.2\\times$ versus glmnet's Newton solver and show competitive, even superior performance for majority cases, versus glmnet's modified.Newton solver, even when $p$ reaches millions (e.g.  $5\\,000\\times1\\,000\\,000$).\n\n* **Sample-dominated ($n>p$)**: Speedups range from $\\sim \\\\!1.5\\times$ on small-$p$ problems (e.g. $10\\,000\\times50$) up to $\\sim \\\\!1.8\\times$ versus Newton on moderate-$p$ ($10\\,000\\times5\\,000$), and as much as $\\sim \\\\!7\\times$ versus modified.Newton at that scale.\n\n* **Balanced ($n=p$)**: ECCD consistently achieves $\\sim \\\\!3\\text{–}4\\times$ faster fits than glmnet, e.g. cutting the 5 000$\\times$5 000 case from 57.2 s down to 13.8 s, and still delivering a 1.5-1.8$\\times$ speedup at $70\\,000\\times 70\\,000$.\n\nECCD scales gracefully across all GLM regimes—feature-dominated, sample-dominated, and balanced—outperforming both glmnet's default cyclic solver and a Modified-Newton variant on very large problems (up to $70{,}000\\times70{,}000$ and $1{,}000{,}000\\times5{,}000$), shaving off tens to hundreds of seconds on a single thread.\n\nWith ECCD, the $1{,}000{,}000\\times5{,}000$ problem runtime drops from 2 633.4 s ($\\approx$43.9 min) to 1 632.2 s ($\\approx$27.2 min), saving 1 001.2 s ($\\approx$16.7 min). This dramatic reduction directly addresses real-world GLM runtime concerns. A shared-memory parallel version of ECCD—currently in development—will drive even greater speedups and make even larger datasets tractable.\n\nWe will include these results in the revised manuscript to highlight ECCD's impact on large-scale GLM problems.\n\n\n\n\n\n\n**W2.**\nWe agree that selecting $s$ requires tuning based on system-specific (i.e. hardware parameters) and data-specific (i.e. condition number, sparsity, dimensions) properties. In the paper, we will add a clear disclaimer that *practical tuning is necessary*. Theorem 3 suggests an asymptotically optimal $s$, however, it is possible to utilize a more detailed performance model which incorporate hardware/data properties (e.g. memory bandwidth/latency, cache size, processor speed, sparsity, dimensions) and obtain a better estimate for $s$. We will incorporate such a performance model in our planned future work on shared-memory parallelization of ECCD.\n\n\n\n\n\n**W3.**\nWe acknowledge that the baselines use different convergence criteria as stated by the reviewer. We chose to align our criteria with `glmnet` as minimizing the objective is our primary goal. For other baselines, we follow their default settings. While the stopping rules differ, they are closely related; under a second-order Taylor approximation, objective decrease is roughly proportional to the squared coefficient change. Thus, the comparison remains fair and meaningful. Our main goal with this paper is to illustrate that a numerically-stable block coordinate update is feasible for GLM problems through algorithmic redesign. We empirically show performance improvements over our target baseline, but also other packages (in these discussions and in the main paper).\n\n\n\n**W4.**\nWe recognize that *public code release* is essential for reproducibility and community adoption. We will:\n* *Open-source ECCD's implementation* (R and C++), along with all scripts needed to reproduce the experiments of the main paper, appendices, and reviewer discussions.\n* *Hosting*: The code repository will be hosted on GitHub with detailed installation and benchmarking instructions.\n\nThis will ensure that users can easily deploy ECCD and verify our results."}]

Author rebuttal:

Meta-review: The paper proposes an improved version of coordinate descent for elastic net regularized linear problems (both regression and classification). Though a great variety of solvers have been proposed in the last decade, there seem to be some numerical advantages to use the method, at least in the sequential setup (computing a full regularization path, which is often needed for hyperparameter tuning). The authors did not submit their implementation, but claimed they would release it, which may be a nice addition to the list of existing toolboxes.
Decision: Accept (poster)