Dual Feature Reduction for the Sparse-group Lasso and its Adaptive Variant
Authors: Fabio Feser
Marina Evangelou
Venue: ICML 2025 poster (2025)
Keywords: penalized regression
screening rules
karush–kuhn–tucker
lasso
high-dimensional
sparse-group
feature reduction
Primary area: general_machine_learning->scalable_algorithms
TL;DR: Feature reduction approach for the sparse-group lasso and adaptive sparse-group lasso using strong screening rules for both variables and groups. The two layers of screening allow the fitting time of the models to be greatly reduced.

Abstract: The sparse-group lasso performs both variable and group selection, simultaneously using the strengths of the lasso and group lasso. It has found widespread use in genetics, a field that regularly involves the analysis of high-dimensional data, due to its sparse-group penalty, which allows it to utilize grouping information. However, the sparse-group lasso can be computationally expensive, due to the added shrinkage complexity, and its additional hyperparameter that needs tuning. This paper presents a novel feature reduction method, Dual Feature Reduction (DFR), that uses strong screening rules for the sparse-group lasso and the adaptive sparse-group lasso to reduce their input space before optimization, without affecting solution optimality. DFR applies two layers of screening through the application of dual norms and subdifferentials. Through synthetic and real data studies, it is shown that DFR drastically reduces the computational cost under many different scenarios.

Dual Feature Reduction for the Sparse-group Lasso and its Adaptive Variant
Fabio Feser'
Marina Evangelou
Abstract
In genetics, these methods help identify genes associated with disease outcomes. As genes are naturally found in groups (pathways), group selection approaches have been proposed, that allow grouping information to be used, such as the group lasso (Yuan & Lin, 2006), group SLOOPE (Brzyski et al., 2019), and group SCAD (Guo et al., 2015). Applying only group shrinkage can harm convergence and prediction, as all variables in an active group are retained, including noise variables (Simon et al., 2013; Feser & Evan- gelou, 2023).
The sparse-group lasso performs both variable and group selection, simultancously using the strengths of the lasso and group lasso. It has found widespread use in genetics, field that reg- ularly involves the analysis of high- -ish-siiensi data, due to its sparse-group penalty, which al- lows it to utilize grouping information. However, the sparse-group lasso can be computationally ex- pensive, due to the added shrinkage complexity, and its additional hyperparameter that needs tun- ing. This paper presents a novel feature reduction method, Dual Feature Reduction (DFR), that uses strong screening rules for the searse-group lasso and the adaptive pparse-rroup lasso to reduce their imput space before optimization, without affecting solution optimality. DFR applies two layers of screening through the application of dual norms and subdifferentials. Through synthetic and real data studies, it is shown that DFR drastically duces the computational cost under many differ- ent scenarios.
This limitation led to the development of sparse-group mod- els, such as the Sparse- Lasso (SGL) (Simon et al., 2013) and Sparse-group SLOPE (SGS) (Feser & Evangelou, 2023). These models can shrink variables in active groups by applying shrinkage on both variables and groups to yield bi-level selection. SGL has found increased popularity in applications in the machine learning (Vidyasagar, 2014; Yo- gatama & Smith, 2014) and healthcare (Peng et al., 2010; Simon et al., 2013; Fang et al., 2015) communities. Sparse- group models have been shown to have tangible benefits by consistently outperforming the lasso and group lasso in selection and prediction tasks (Simon al., 2013; Feser Evangelou, 2023).
Suppose the variables sit in grouping structure, with dis- joint sets of variables ,I...,, Gm of sizes Pl... PPn- Then, SGL is a convex combination of the lasso and group lasso (Simon et al., 2013):
1. Introduction
High-dimensional datasets, where the number of features (p) is far greater than the number of observations (n) in a matrix X E R=xp, are becoming increasingly common with the increased rate of data collection. To handle this, shrinkage methods, such as the lasso (Tibshirani, 1996), elastic-net (Zou& & Hastie, 2005), and SLOPE (Bogdan et al., 2015) have been proposed and found increased use in the machine learning community (Alaoui & Mahoney, 2015; Michoel, 2018; Lemhadri et al., 2021; Thompson et al., 2023). These methods shrink estimates towards zero during optimization, enabling variable selection, to identify which features, 8 IR", have an association with the response y E R
(A) E arg ( + s111)
(1)
BER
where || = al| + (1 - a) Σ g=1
v PPgl (
(2)
such that f is differentiable and convex loss function, À 0 defines the level of shrinkage, ((() RPs is the vector of coefficients in group g. and j0, 1]. SGL has been extended to have adaptive shrinkage through the adap- tive sparse-group lasso (aSGL) (Poignard, 2020; Mendez- Civieta et al., 2021) (Section 3.3).
Department of Mathematics, Imperial College London, Lon- don, UK. Correspondence to: Fabio Feser <ff120@ uck.c.u>>.
1.1. Feature Reduction Approaches for the Sparse-group Lasso
Proceedings of the 4nnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
The strengths of SGL come with increased computational cost, due the additional shrinkage and the tuning of two

---

Dual Feature Reduction for SGL
hyperparameters. Typically, a is set subjectively (Simon et al. (2013) suggest a = 0.95) and À is tuned using eross- validation along path X .... ... 2 A 2 O, Algorithms such as the Least Angle Regression (LARS) approach cal- culate solutions for all possible values of X, but are very sensitive to multicollinearit and scale quadratically, render- ing their use in high-dimennional settings limited (Efron et al., 2004). Instead, feature reduction techniques, in- cluding screening rules, can help ease the cost of fitting a model along path by discarding features before optimiza- tion that would have been inactive at the optimal solution. Whilst methods exist to discard observations (Shibagaki et al., 2016; Zhang et al., 2017), the focus of this paper is high-dimensional settings, in which discarding features is more impactful on computational savings.
15.0
Improvementt Factor
12.6
10.0
7.5
0.0
0.1
0.3
0.5
0.7
0.9
a
DFR-SGL.
DFR-aSGL
GAP dynamic
DFR BCD
spariegl
GAP sequential
The improvement factor (high is best), which measures by how many orders the screening has improved fitting time, shown for different SGL screening methods, as a function of a, with 95% confidence intervals. The data was generated under a linear model with even groups of sizes 20 (Section 4.1). The two GAP methods have very similar values so jitter was added.
Figure 1:
Feature reduction techniques either exact or heuristic. Exact methods strictly discard only inactive features but are conservative, while heuristic methods discard more features at the risk of violations. These violations are countered by checking the Karush-Kuhn-Tuuee Tucker (KKT) conditions (Kuhn& & Tucker, 1950) and adding any offending features back into the optimization. Heuristic rules discard signifi- cantly more variables than exact rules, providing large com- putational savings (Tibshirani et al., 2010).
(Ndiaye al., 2015), and sparsegl (heuristic) (Liang et al., 2022), which applies only group-level reduction. Additional speed-up attempts include using approximate bounds for inactive conditions (Ida et al., 2019) and a heuristic screen- ing rule limited to multi-response Cox modeling (Li et al., 2022).
Most exact methods follow the seminal Safe Feature Elimi- nation (SAFH) framework (El Ghaoui et al., 2010), which has been applicd to the group lasso (Bonnefoy et al., 2015) and SGL (Ndiaye et al., 20) Other exact examples include the dome test (Xiang & Ramadge, 2012). Dual Polytope Projections (DPP) (Wang et al., 2013), and Slores (Wang et al., 2014). The strong rule by Tibshirani et al. (2010) provides framework for applying heuristic reduc- tion with single separable norms, which has been extended to non-separable (Larsson et al., 2020) and aparse-group norms (Feser & Evangelou, 2025). Other examples include Sure Independence Screening (SIS) (Fan & Lv, 2008) and the Hessian rule (Larsson & Wallin, 2024).
1.2. Contributions
In this paper, we propose a new feature reduction method for SGL and adaptive SGL, Dual Feature Reduction (DFR), which is based on the strong rule (Tibshirani et al., 2010) and the sparse-group screening framework (Feser & Evangelou, 2025). DFR introduces the first bi-level strong screening rules for SGL and the first screening rules for aSGL.
DFR applies two layers of screening, discarding inactive groups and inactive variables within active groups. By re- ducing the input dimensionality before optimization, DFR enables expanded tuming regimes to be performed, including concurrent tuning of X and a. These benefits are achieved without affecting solution optimality. The computational efficiency of DFR increases the accessibility of SGL and aSGL models, encouraging wider adoption across fields.
Aside from the exact and heuristic categories, feature reduc- tion techniques tend to follow three forms: static, where the feature reduction occurs only once at the start (El Ghaoui et al., 2010; Xiang et al., 2011; Xiang & Ramadge, 2012), dynamic, where reduction occurs iteratively (Bonnefoy et al.,, 2015), and sequential, where information from the previous solution is used (Tibshirani et al., 2010; Larsson et al., 2020; Larsson & Wallin, 2024; Feser & Evangelou, 2025).
The GAP safe rules for SGL require computation of safe regions, which includes a radius and center, and the dual norm, as well as iterative screening and fitting. In contrast, DFR needs only the dual norm and screens only once at each path point, making it considerably less expensive, as evidenced by our results (Figure 1).
An exact reduction method for SGL, called GAP safe, was proposed by Ndiaye et al. (2016a) using the SAFE frame- work. GAP safe uses the duality gap to create feasible regions where the active variables sit and applies reduction on the groups and variables. Other reduction methods for SGL include Two-layer Feature Reduction (TLFre) (exact) (Wang& & Ye, 2014), though it was shown not to be exact
DFR is described for SGL in Section 3 and then extended to aSGL in Section 3.3. The proofs of the results presented in these sections are provided in Appendix A. for SGL and
2

---

Dual Feature Reduction for SGL
Appendix B.2 for aSGL. DFR is applied synthetic and real data in Sections 4 and 5, where it is found be the state-of-the-art screening approach for SGL, outperforming other existing methods, while still achieving the optimal solution.
3.1. Sparse-group Lasso
3.1.1. GROUP REDUCTION
To generate candidate group set, the KKT stationarity conditions (Kuhn & Tucker, 1950) are used, providing con- ditions for an inactive group. For SGL, they are given by, for group E [mn] at Aa+1 (using the 6-norm representation in Equation 3)
2. Preliminaries
2.1. Problem Statement
O E V V (B(A++1)) + Ty 1
(6)
SGL is trained along path of parameters À1 2 ... 2 A 0. The objective is to use the solution A to generate set of candidate variables C, C [p] := (I...., p). that is a superset of the (unknown) set of active variables at λ given by A.(--1)) == (i E [p] (Ax--1) * 0}. The optimization at 1 (Equation 1) is then calculated using only (o If the candidate set small proportion of the total input space, large computational savings are expected.
where g. = dual norm of the e-norm at A++1- The subgradient for an inactive group g (at zero) can be expressed by the unit ball of the dual norm (Schneider & Tardivel, 2022):
Il? 7, is the subgradient of the
0g = 0| {xr R |,lle =
Plugging the unit ball into Equation 6 and applying the e-norm, the subgradient can be canceled out, so the KKT conditions can be written as
To generate the candidate variable set, we first generate candidate group set (Section 3.1.1), which is then used as a basis for constructing the candidate variable set (Section 3.1.2). This done using the dual norm of SGL.
| ( = =Tg>++1160 gottle,, S TgAR+1
(7)
If the gradient at A were available, we could exactly identify the active groups (Proposition 3.1).
2.2. Dual Norm
Proposition 3.1 (Theoretical SGL group screening). For SGL applied with any A set,
DFR requires evaluating the dual norm of SGL., defined as ||=|اي == sup{= Jax || S 1]. The SGL norm can
[[ 11, the candidate group
expressed in terms of the dual of the e-norm (Ndiaye al., 2016a),
Cg(Ax+1)= to E [m] 1V V > TgAetzh. is such that C,(Ae+1) = Ag(Ax+1) = to E [m] (Xe+1)l + 0}.
| sgl
Ty
(3)
However, this is not possible in practice, approxima- tion Mg is required such that
where T, = a + (1 a) The e-norm, ||r||,, applied to a group g E [m]] is defined as the unique nonnegative solution q the equation (Burdakov, 1988)
IV (3( S SMg.
(8)
Then, the screening rule tests whether Mg S AR+1+1 this found to be true, it can be concluded that Equation holds and the group must inactive. An approximation can be found by assuming that the gradient is Lipschitz function of d4+1 with respect to the c-norm,
Po Σ
(|0:) - (1 - (n) = (c,9)², c = Tg a
(4)
Tg
Using this, by Ndiaye et al. (2016a), the dual norm of SGL applied to a group E [m] can be formulated as
| 1 (3(x+ V vf S Tg/AAt1 - Aul. (9)
which is a similar assumption to that used in the lasso strong rule (Tibshirani et al., 2010). Using the reverse triangle inequality gives
||<(
lgl
max TØ 11 m
(5)
|| [ S [V + Tg(Ai AR+1):
3. Dual Feature Reduction
m:M,
DFR is first derived for SGL (Section 3.1) and then extended to aSGL (Section 3.3). DFR summarised for SGL and aSGL in Table A1.
yielding a suitable approximation Mg Therefore, the strong group screening rule for SGL (Proposition 3.2) can
3

---

Dual Feature Reduction for SGL
be formulated by plugging Mg into Equation 8: discard group g [m] if
To derive Proposition 3.3, knowledge of Ag(Ax+1) is required, but this is unknown. By Proposition 3.2, (x(1=- C Cg (x((+1)), and so the candidate set is used in practice for applying Proposition 3.3. That is, the vari- able screening rule is applied to the candidate group set. Any violations caused by this replacement are checked for by the KKT checks, which are performed any case (for any strong rule).
VV ( (2 Ak)-
(10)
Since the Lipschitz assumption can fail, KKT checks (Sec-
tion 3.1.3) are performed to prevent violations.
Proposition 3.2 (DFR-SGL group screening). For SGL
applied with any 1 h E I 11, ussuming that
3.1.3. KKT CHECKKS
(Vo(((( V vof((())) S &TgAA+1-- dnl. for all g E [ml, then the candidate group set,
The screening rules of DFR use several Lipschitz assump- tions (Propositions 3.2 and 3.3), as well as replacing the active group set by the candidate group set for the variable screening step (Section 3.1.2). When these assumptions fail, the screening rules can incorrectly discard active variables. To protect against this, the KKT conditions are checked for each variable after screening. A KKT violation occurs for variable E 9g if
Co(A++1) = Eg E [m]] IV [Vgf( > (27x+++ Ab]b,
is such that (x++++1) C (g(++t).
3.1.2. VARIABLE REDUCTION
IS IS(V. AR+ <))) Ak+1a, (13)
Group screening reduces the input dimensionality, but fur- ther reduction is possible by applying a second screening layer to the variables in the candidate groups. For an inac- tive variable, i & ( Gg, the KKT conditions (by Equation 1)
where Stu, b) sign(a)((al b)+ is the soft-thresholding operator (see Appendix A.2 for the derivation). A violating variable is added back into the optimization procedure (see Section 3.2).
1 u
(1())
i (11) where & and v are the subgradients of the é and l2 norms respectively. For an active group, the subgradient of the le norm is given by (10) /1,//11)11 which vanishes for inactive variable. So,
3.1.4. PATH START
When fitting SGL along path of values, è 2 ... X 2 0, A is often chosen be the exact point at which the first predictor becomes non-zero. By Ndiaye et al. (2016b) and using the dual norm from Equation 5, this value is given by
- -V.f( E A4t10T4
A = [
= max g=1....m
[V
Vv.( S A4t1a,
(12)
3.2. Algorithm
where $0 +i++1 {z E R: |x| S 1). As before, knowledge of the gradient would lead to exact support recovery (Propo- sition I). Equation 12 is similar to the strong screening rule for the lasso (Tibshirani et al., 2010), scaled by a. Hence, using a scaled version Lipschitz assumption for the lasso, the variable screening rule for SGL is formulated in Proposition 3.3.
The DFR algorithm is based on the sparse-group strong screening framework, proposed by Feser & Evangelou (2025), and is shown in Algorithm A1. The algorithm has the following key steps for Ak+1
1. Group screening: find C, (A++1) using Proposition 3.2.
Proposition 3.3 (DFR-SGL variable screening). For SGI. applied with any Ax+1 1, & 1), assuming that
2. Variable screening: find C, Co (A++1) using Proposition
3.3 for i E gg Ao(A)) E Cy (N++1)..
(+((((AE+1) v.f(((A))) S a(Ax= -x-1),
3. Optimization: Compute 3o. (Ax+ using the optimiza- tion set O = (o(++1)) U Aox)). Perform KKT checks to identify any violations (Section 3.1.3) and add offending variables into O,,. Repeat this step until no violations.
for all i E for E Ag(Ax-1), 1), then the candidate vari-
able set,
Co(A++1) E for E Ag(A+t1):
(vI((((A)) > a(2)x+1- - Ab)}.
The two main computational costs the algorithm are the calculation of the solution, 39, and the evaluation of the
such that A (A++1) C (c

---

Dual Feature Reduction for SGL
e-norm. The former depends on the fitting algorithm, this framework is applicable for any SGL fitting algorithm, with proximal and descent algorithms typically having com- plexities of O(tp²). for t iterations (Zhao & Huo, 2023). The latter has a worst-case cost of Op2 log log (Ndiaye et al., 2016a). Tables A3 and A4 in the Appendix provide a computational breakdown of the components of DFR.
Setting
a &iijeg (4jj Vy1Bil U113311
7/ al1o(001|1
+ (1 -a)aa v Ps
||(0011
2 = (1 (l-a)Way PP
Y9
allows Equation 16 to be written in terms of the e-norm
3.3. Adaptive Sparse-group Lasso
| asg)
Σ Yo 7o1.8 (1 112
(1)
The Adaptive Sparse-groue group Lasso (aSGL) applies adaptive shrinkage a sparse-group setting, achieving the oracle property in a asympooayy framework, and has the norm (Poignard, 2020; Mendez- Civieet--Cviieeeee et al., 2021)
Through this connection, the aSGL norm admits a direct link to SGL. (Equation 3), allowing the DFR-SGL. rules to be used for aSGL by replacing To with 7o and Eg with c in the e-norm (Appendices B.2.1 and B.2.2). The connection is further sharpened theoretically: Lemma 3.5 guarantees that the "/y term in Equation 15 will always exist, even under inactive groups Lemma 3.6 guarantees that the represen- tation of aSGL as the e-norm correctly reduces to the SGL c-norm representation under constant weights.
|| lagi Σ
e a)
Σ w Py
Pg Bl9) 1|2, (14)
where U; and Wy are adaptive weights (described in Ap- pendix B.3). aSGL has a less straightforward connection to the e-norm that allows for the derivation of screening rules (Proposition 3.4).
Proposition 3.4. The aSGL. norm (Equation 14) can be expressed as the e-norm by
Lemma 3.5. Under an inactive group g & Ag, i.e. Bla) = 0, the 7o term in Equation 15 exists.
Lemma 3.6. Under v = 1 and w = 1 in Equation 15, for each g [m], 7/g = Tg and cg Eg-
| laggt
Σ 9=1
7 liy where
(15)
Algorithm A1 is also applicable for aSGL., using the corre- sponding aSGL equations (Algorithm A2).
7/ = ||lا(6ا||
1
|| 1|1 WE
+ (1 - a)uu, Pg:
3.3.1. KKT CHECKS
'g 16-1111 (1 ax) ())gyVP
The KKT checks for aSGL are also similar to those for SGL (Section 3.1.3): a KKT violation occurs for a variable E 9, if
Proof of Proposition 3.4. Splitting up the summation term in the variable norm yields
(((V.(((AA++)) A4+1 aa)--a))yypp /PW A8414;a.
(17)
3.3.2. PATH START
gml EE
To find the path start for aSGL, the dual norm cannot be used, since all groups are zero at this point. As a result, Y exists only in limit for ((3) = 0 (Lemma 3.5), A solution can instead be found using a similar approach to that of Simon et al. (2013) for SGL, where the point is found by solving the piecewise quadratic, for each g E [m],
Eiseg, i 2/13,1
a
||300)112
||2
||000||11
1/360)|1
g
This allows the aSGL norm to be written in terms of the groups
Is
py a) = 0, (T) (x()Ty/nn y/n, Xg a) || -
[10t0111 Enjes 1ets uy1Bal || 1
|| asgl
Σ
all 11
where (9) E RXXP, is the design matrix for only group g and p(9) E R contains the penalty weights for the vari- ables in group g. Then, choosing A1 = max, Ag gives the path start point.
+ (1- +1-a)wy p 132
(16)
5

---

Dual Feature Reduction for SGL
Improvement factor
201
01.20
Inpu proporton
0.2-
20
0.20
201
0.30
Input
0.00
0.3
0.5
0.7
Signal strength
Signal strength
DSCL
GAP dynamic
SFR-SGL.BE BCD
spunseg
GAP sequemiaal
DFR-SGL
DFR-aSGL
GAP dynamic
DFL-SGL BCD
sparsegl
GAP sequential
Figure 2: The imput proportion for the screening methods applied to synthetic data, as a function of o, with 95% confidence intervals.
Figure 3: The improvement factor (left) and input proportion (right) for the screening methods applied to synthetic data, function of the signal strength, with 95% confidence intervals.
4. Numerical Results
In this section, the efficiency and robustness of DFR is evaluated through the analysis of synthetic data that capture different data characteristics. As the purpose of screening rules is to reduce the dimensionality of the input space and, as result, reduce the computational cost, the following two metrics are used for evaluation:
4.1. Analysis of Synthetic Data
The synthetic data was generated using linear model, y X + e, where X ~ NO,, E 12000 with noise ~ (O( 1) and where 3 is a sparse vector with the signal sampled from (0,0 4) (signal strength of zero). For X, correlation was applied inside each group, such that E,,- = p 0.3 for i and j in the same group. The variables were placed in even groups of sizes 20, with 0.2 group sparsity proportion (20% of the groups are active) and 0.2 variable sparsity proportion inside the active groups. The models were fit along a 50-Iength path, starting at λ₁ = Amax (as defined by Section 3.1.4 for SGL and Section 3.3.2 for aSGL), and terminating at 0.1A1. Each simulation case was repeated 100 times and the results are averaged across these repetitions, unless otherwise stated. Simulation and model implementation information can be found in Table D.2.
Improvement factor screen time / screen time, quantifies by how many orders the screening has im- proved the fitting time (high is best).
Input proportion Op/p, measures how much of the input space was used in the optimization (low is best).
DFR is compared with the existing SGL screening rules sparsegl (Liang et al., 2022) and GAP safe (Ndiaye al., 6a). sparsegl, in contrast to DFR, performs only sin- gle layer of group screening. This rule is also based the strong framework of Tibshirani et al. (2010), but uses different Lipschitz assumption, which applies only the (2 group norm, rather than the full SGL. norm (as is the case for DFR). On the other hand, GAP safe is an exact feature reduction method for SGL that can be implemented dynamically or sequentially under linear regression (Ndiaye al., 2a). GAP safe has many different implementation forms, and we are presenting the two versions that provided the best results in our studies. Appendix C provides detailed descriptions of these two methods with Table A showing a summary of all rules considered.
Comparison to GAP Safe Rules Comparing DFR the GAP safe rules, under both varying and signal strength, it is evident that the improvement factor is significantly supe- rior for DFR compared to both the dynamic and sequential GAP rules (Figures 1 and 3). In fact, although the input pro- portion of DFR and GAP safe are of similar levels (Figures and 3), the cost of calculating safe regions appears to nul- lify any gain in dimensionality reduction. This comparison shows that the two reduction approaches (heuristic vs exact) arrive at very similar results (the screened sets), but DFR achieves this with greater computational efficiency.
For all values of a, DFR considerably reduces the input space, with the screening efficiency a linearly decreasing function (Figure 2). Under values of close to zero, SGL is forced to pick more variables within a group as active, limiting the potential reduction of the input space. In such scenarios, the second screening layer becomes less crucial, evidenced by the similar performances of all approaches. Approaching the commonly used value a = 0.95 shows the clear strengths of DFR. The screening methods are all relatively unaffected by the signal strength (Figure 3).
Throughout the analyses, the SGL optimization for DFR and sparsegl is performed using the Adaptive Three Operator Splitting (ATOS) (Pedregosa & Gidel, 2018) algorithm, it can be easily adapted for use with different sparse-group penalties (by simply swapping out the proximal operators), providing flexibility. However, DFR can be used with any fit- ting algorithm, including Block Coordinate Descent (BCD) (Qin et al., 2013).
6

---

Dual Feature Reduction for SGL
Inpert proportion
20
0.4
Improvement
10
Inpur
0.2
0.0
2000
4000
8020
03
0.1
0-1
Dimensiomility (p)
Dimensionulity (g)
Sparsity progortion
Correlation (p)
DFR-SGL
DFR-aSGL
spansegl
DFR-SGL
DFR-aSGL
sparseg]
Figure 4: The improvement factor (left) and imput proportion (right) for the strong rules applied to synthetic data, as function of with 95% confidence intervals.
Figure 6: The input proportion for the strong rules applied to synthetic data, as function of the sparsity proportion (left) and data correlation (right), with 95% confidence intervals.
20
Improvement factoor
Linca:
Lugoots
Inprovenent
10-
0.9
Sparsity proportion
Correlation (e)
DER-SGL
DER-
spanseg
Figure 7: The improvement factor of DFR-aSGL under different weights b2 b2, shown for the linear (left) and logistic (right) models, with 95% confidence intervals.
Figure 5: The improvement factor for the strong rules ap- plied to synthetic data, as a function of the sparsity propor- tion (left) and data correlation (right), with 95% confidence intervals.
p. few large groups dominate, limiting the potential screen out groups, while for large many small groups reduce the impact of group screening.
Comparison to BCD The GAP safe rules are imple- mented using BCD while DFR uses ATOS. In Figures and 3, DFR has also been implemented using BCD, demon- strating similar performance to ATOS, showing that the choice of fitting algorithm has little impact on the gain from screening.
Additional explorations increased dimensionality in- cluded the analysis of interaction terms (Table and Ap- pendix D.5). In this setting, all possible interactions of order and within each group were included. DFR pro- interactions, vides large computational savings when fitting
especially compared to sparsegl, which under order in- teractions provides only marginal improvements. These savings make more feasible for sparse-groue models to be used in interaction detection problems. Such challenges are frequently seen in the field of genetics, where gene-gene and gene- aene-environment relationships useful discoveries (D' Angelo et al., 2009; Zemlianskaia et al., 2022).
Increasing Dimensionality The benefits of DFR over sparsegl are observed under varying p (Figure 4), peaking around 1000. With groups of fixed sizes of 20, their relative size proportion to changes, suggesting an optimal grouping regime around p = 1000 for screening. For small
Table The improvement factor for the strong rules applied to synthetic interaction data under the linear model, with standard errors. The parameters of the data were
Robustness For the remainder of the section, the variables placed in 22 uneven groups of sizes in [3, 100) to gain additional insights into the robustness of DFR. Earlier, DFR was found to be robust under varying signal strength and a. We further observe that DFR is also robust to the data- generating parameters of signal sparsity (variable and group sparsity proportions varied together) and group correlation in X (Figures 5 and 6).
p 400, 80, and 52 groups of sizes in [3, 15). The interaction input dimensionality was po, = 2111 and Po. 7338, with no interaction hierarchy imposed. The sparsity proportion of interaction variables was set to 0.3 (with the same signal as the marginal effects).
Interaction
A clear benefit of DFR sparsegl is observed under sparse signals. Screening rules generally have greater im- pact as the signal becomes sparser. However, once the signal saturates, their effectiveness declines, leading to similar per- formance across approaches. Under varying correlation, DFR is more successful reducing the input space when
Method
Order 2
Order 3
DFR-aSGL
137.3 + 12.0
54.0 + 10.7 23.6
DFR-SGL
44.3 + 2,4
sparsegl
7.4 0.9
1.2

---

Dual Feature Reduction for SGL
Impovveeet (logo)
Expat propottion
8.4
breal
adctotia
0.01
DHR- SGL DFR- aSGL,
sparneg!
Inga propenties
schortz
colinc
0.1
breal
scheetz
trust experts adenomn
celiae
Tumoou
Lincer
Logisti
Figure 8: The improvement factor (logı0 scale) of the strong rules applied to the six real datasets, split by model type.
1.0
0.5
0.2
A/Arss
(loga)
DFR-SGL
DFR-
spanagg
Figure 10: The input proportion as function of the shrink- age path for the strong rules applied to the real datasets.
DER-SGL DER aSGL, sparseg!
brcal
scheetz experss adenoma
celac
tumour
Logistic
quent. Note that DFR-aSGL violations refer variable ones, and sparsegl to group ones, making more likely to have a variable violation. The elevated number of KKT violations for sparsegl suggests that the group Lipschitz assumption of DFR-SGL is more robust.
Figure 9: The input proportion (log10 scale) of the strong rules applied to the six real datasets, split by model type.
One possible explanation for the increased KKT violations of DFR-aSGL lies the role of the Lipschitz assumptions. Unlike SGL, the adaptive penalties in aSGL introduce addi- tional dependencies on hyperparameters into the I. Lipschitz assumptions. This extra dependence on hyperparameters has the potential to lead to violations.
compared to sparsegl, especially under minor correlation. Under higher correlation, the models become less sparse, again resulting reduced screening importance.
Similar the with even groups, for uneven groups, DFR remains relatively unaffected by the signal strength and the choice of a, consistently achieving effective reduction. (Figures A2 and A3).
5. Real Data Analysis
Hyperparameter Tuning and Volss-aaii The per- formance DFR-aSGL was found be robust under differ- ent values of hyperparameters b and by (Figure 7), which are used to define the adaptive weights (Appendix B.3).
The efficiency of DFR is further evaluated through the anal- ysis of six real datasets with different characteristies, includ- ing response type and dimensionality. Three of the datasets, brcal, scheetz, and Irust-experis, have continuous responses, so are fit using an SGL linear model. The former two were also analyzed with regards to screening rules in Larsson Wallin (2024), and the latter in Liang al. (2022).
The efficiency and robustness of DFR across different hyper- parameters (a, b1, b2) make it promising tool for enabling approaches like cross-validation (CV) to tune all SGL. and aSGL hyperparameters, which is rarely done in practice. Applying DFR with CV yields substantial computational savings (Table A1l). These findings highlight DFR's value in facilitating expanded tuning regimes for SGL and aSGL.
The other three datasets, adenoma, celiac, and fumour, have binary responses, an SGL logistic model is used. The trust-experts dataset is low-dimensional, and the other five are high-dimensional. The models were fit along a 100- length path, terminating at 0.2X1, where A₁ generates the null model. More information on the datasets and model implementation is provided in Table D.2 and Appendix E.
Logistic Model DFR is also effective and robust for logis- tic models (Figures A7, A8, A9, and A10; see Appendix D.6 for further results and description of the data generation).
For all datasets, DFR outperforms sparsegl reducing the computational cost (Figure 8) and input dimension (Figure 9), as well as keeping the input proportion low along the whole path (Figure 10). Despite being most useful for high- dimensional data, even in the case of -iw-imensiooal data (frust-experts), DFR improves fitting time.
KKT Violations KKT violations for DFR are very rare. Across all experiments with linear models, DFR-SGL had only a single KKT violation (Table A9). Violations were more common for DFR-aSGI. and sparsegl, but still infre-
8

---

Dual Feature Reduction for SGL
DFR-aSGL performs very well for scheetz and adenoma, improving the computational cost by over 600 times. For the scheetz, dataset, the aSGL model had more difficulty con- verging without screening compared to SGL, so DFR-aSCR offered a greater advantage over DFR-SGI. For adenoma, the active set for aSGL was smaller (Table A18), due to the increased penalization that comes with the adaptivity. How- ever, despite the advantage of smaller active set, we do still observe that DFR-aSGL was more efficient at reducing the optimization set, with respect to the active set.
lighting the benefit of bi-level screening.
Several assumptions are required to perform two layers of feature reduction for DFR. Propositions 3.2 and 3.3 use Lipschitz assumptions which are consistent with the strong framework (Tibshirani et al., 2010). Any breach of assumptions is guarded against by KKT checks. Only single KKT violation occurred for SGL across all our simulations and only very infrequently for aSGL. These assumptions are a limitation of any strong rule, although DFR carries additional assumptions over other strong rules, which are necessary for the second layer of screening.
Limitations
DFR is observed to aid in mitigating convergence issues for both SGL and aSGL (Table A19). Across all datasets, DFR encountered no failed convergences. In contrast, sparsegl did not converge at several path points for both adenoma and scheetz. As sparsegl only screens groups, when a group enters the optimization set, sparsegl is forced to fit with the full group, which can contain noise variables. Apply- ing no screening led to SGL not converging for adenoma, scheetz, and tumour. By drastically reducing the input space, convergence issues caused by large datasets are resolved, improving both computational cost and solution optimality.
Code DFR implemented in the dfr R package (Feser, 2024), available on CRAN.
Impact Statement
This paper aims to advance Machine Learning while en- suring no disadvantage to anyone. The proposed screening rules do not alter the solution but improve the accessibility of SGL and aSGL for researchers with limited computa- tional resources.
6. Discussion
A novel feature reduction method for the sparse-group lasso and adaptive sparse- group lasso, called Dual Feature Re- duction (DFR), has been introduced, derived using the dual norms of SGL and aSGL. DFR introduces the first bi-level strong screening rules for SGL and the first screening rules for aSGL. By applying two layers of reduction, DFR ef- fectively reduces input dimensionality for optimisation and is computationally simpler than the GAP safe rules, which require iterative screening and fitting. In contrast, DFR screens only once per path point, so that it adds minimal computational overhead.
Acknowledgements
We would like to thank the anonymous reviewers for their valuable comments. This work was supported by the Engi- neering and Physical Sciences Research Council (EPSRC) through the Modern Statisties and Statistical Machine Learn- ing (StatML) CDT programme, grant no. EP/S023151/1.
References
Alaoui, A. and Mahoney, M. W. Fast Randomized Ker- nel Ridge Regression with Statistical Guarantees. In Advances in Neural Information Processing Systems, vol- ume 28. Curran Associates, Inc., 2015.
DFR first applies group-level screening, discarding inac- tive groups, followed by variable-level reduction, where inactive variables in active groups removed. By dis- carding variables that are inactive at the optimal solution, DFR achieves significant computational savings, enabling the SGL family of models to scale more efficiently with increasing dimensionality and handle larger, more complex datasets. This gain comes at no cost, as the optimal solution is still achieved (Appendices D.4, D.6, and E.2). In fact, by reducing the input, instances were observed where DFR helped SGL. and aSGL overcome convergence issues.
Bauschke, H. H. and Combettes, P. Convex Analysis and Monotone Operator Theory in Hilbert Spaces. Springer International Publishing, Cham, 2017. ISBN 978-3-319- 48310-8. doi: 10. 10.10077977-3-319-488
Bogdan, M., van den Berg, E., Sabatti, C., Su, W., and Candès, E. J. SLOPE-Adapte convex optimization. The Annals of Applied Statistics, 9 (3), 2015, ISSN 1932-6157. doi: 10. 10.121//5--0AA544..
variable selection via
DFR proved robust across different data and model parame- ters, achieving drastic feature reduction under all scenarios considered. This consistently translated into large compu- tational savings across both synthetic and real data. DFR outperformed all other screening approaches, establishing it as the state-of-the-art screening method for SGL and high-
Bonnefoy, A., Emiya, V., Ralaivola, I.., and Gribonval, R. Dynamic Screening: Accelerating First-Order Al- gorithms for the Lasso and Group-Lasso. JEEE Trans- actions on Signal Processing, 63(19):5121-5132 doi: 10.1 SP.09......1......55......

---

wtt.d//iorrg1111111jj.... Dual Feature Reduction for SGL
Brzyski, D., Gossmann, A., Su, W., and Bogdan, M. Group SLOPE Adaptive Selection of Groups of Predictors. Journal of the American Statistical Association, 1 114(525); 419 433, 2019. doi: 10. 00.1080/01221440 112221149922..111 1269.
2025. URL URL www.hhttps:/pprcceedingge ml prr.ssss//sspss/////// #258/feser2555 html.
Guo, X., Zhang, H., Wang, Y., and Wu, J.-L. Model selection and estimation in high dimensional regres- sion models with group SCAD. Statistics & Proba- bility Letters 103:86-92, 2015. ISSN 01677152. doi: 10.10166f.s1..2115.444
Burdakov, O. A new vector norm for nonlinear curve fitting and some other optimization problems. In Mathematische Optimierung - Theorie und Anwendungen, volume 33 of Int. Wiss. Kolloq. Fortragsreihe, pp. 15-17, 1988.
Heap, G. A., Trynka, G., Jansen, R. C., Bruinenberg, M., Swertz, M. A., Dinesen, L. C., Hunt, A., Wijmenga. C., vanHeel, D. A., and Franke, I.. Complex nature of SNP genotype effects on gene expression in primary human leucocytes. BMC Medical Genomics, 2(1):1, 2009, ISSN 1755-8794, doi: 10.1186/1755-8794-
D' Angelo, G. M., Rao. D., and Gu, C. C. Combining least absolute shrinkage and selection operator (LASSO) and cemcipal-componenn analysis for detection of gene-gene interactions in genome-wide association studies. BMC Proceedings, 3(57):562, 2009. ISSN 1753-6561. doi: 10.1 10.1188//753--66-3335777
Ida, Y., Fujiwara, Y., and Kashima, H. Fast Sparse Group Lasso. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Efron, B., Hastie, T., Johnstone, I., and Tibshirani, R. Least angle regression. The Annals of Statistics, 32(2):407-499 2004. doi: 10. 00.1214/0000533644000000000000
Kuhn, H. W. and Tucker, A. W. Nonlinear programming. In Proceedings of Second Berkeley Symposium on Mathe- matical Statistics and Probability, pp- 481-492, Berkeley, Los Angeles, USA, 1950. University of California Press.
El Ghaoui, L., Viallon, V., and Rabbani, T. Safe Feature Elimination in Sparse Supervised Learning. Technical Report UCB/EECS-2011 1261000,,,100 EECS Department, Uni- versity of California, Berkeley, 2010.
Larsson, J. and Wallin, J. The Hessian screening rule. In Proceedings of the 36th International Conference on Neu- ral Information Processing Systems. Curran Associates Inc., 2024. ISBN 9781713871088.
Ellsworth, K. A., Eckloff, B. W., Li, L.,, Moon, I., Fridley, B. L., Jenkins, G. D., Carlson, E., Brisbin, A., Abo, R., Bamlet, W., Petersen, G., Wieben, E. D., and Wang. L. Contribution of FKBP5 Genetic Variation to Gemcitabine Treatment and Survival in Pancreatic Adenocarcinoma. PLoS ONE, 8(8): 2013. ISSN 1932-6203. doi: 10.137 N0.137//jommall.poe000000000 007021...
Larsson, J., Bogdan, M., and Wallin, J. The Strong Screen- ing Rule for SLOPE. In Advances in Neural Information Processing Systems, volume 33, pp. 14592-14603, Cur- ran Associates, Inc., 2020.
Fan, J. and Lv, J. Sure independence screening for ultrahigh dimensional feature space. Journal of the Royal Statis- tical Society: Series B (Statistical Methodology), 70(5): 849-911, 11 2008. doi: 2008.00
Lemhadri, I., Ruan, F., Abraham, L., and Tibshirani, R. Las- soNet: Neural Network with Feature Sparsity. Journal of Machine Learning Research, 22(127):1-29, 2021.
Li, L.. Zhang, J..W., Jenkins, G., Xie, F., Carlson, E. E., Fri- dley, B. L.., Bamlet, W. R., Petersen, G. M., McWilliams, R. R., and Wang, L. Genetic variations associated with gemeitabine treatment outcome in pancreatic cancer. Pharmacogenetics and Genomics, 26(12):577-537, 2016. ISSN 1744-6872. doi: 10.1997/FFC.0000000000000
Fang, K., Wang, X., Zhang, S., Zhu, J., and Ma, S. Bi- level variable selection via adaptive sparse group Lasso. Journal of Statistical Computation and Simulation, 85 (13):2750-2760, 2015. ISSN 15635163. doi: 10.1080/ 00949655 201.99555..01.9882411
Feser, F. dfr, 2024. URL orq/paackage-dfr ORCID: 000000007-3 -927..
Li, R., Chang, C., Justesen, J. M., Tanigawa, Y., Qian, J., Hastie, T., Rivas, M. A., and Tibshirani, R. Fast Lasso method for large-scale and shthhighh-dimesiinnall Cox model with applications UK Biobank. Biostatistics, 23(2):522-540, 2022. ISSN 1465-4644. doi: 10.1093/ biostatistics/kxaa0333
Feser, F. and Evangelou, M. Sparse-group SLOPE: adap- tive bi-level selection with FDR-control. arXiv preprini arXiv:2305.09466, 2023.
Feser, F. and Evangelou, M. Strong Screening Rules for Group-based SLOPE Models. In Proceedings of The 28th International Conference on Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pp. 352-360. PMLR, 03-05 May
Liang, X., Cohen, A., Heinsfeld, A. S., Pestilli, F., and McDonald, D. J. sparsegl: An R Package for Estimating Sparse Group Lasso. arXiv preprint arXiv:2208.n 02942, 2022.
10

---

Dual Feature Reduction for SGL
Mendez-Civieta, A., Aguilera-Morillo, M. C., and Lillo, R. E. Adaptive sparse group LASSO in quantile re- gression. Advances in Data Analysis and Classifica- tion, 15(3):547-573, 2021. ISSN 1862-5347. doi:
Sabates-Bellver, J., Van der Flier, L. G., de Palo, M., Catta- neo, E.,, Maake, C.,, Rehrauer, H., Laczko, E., Kurowski, M. A., Bujnicki, J. M., Menigatti, M., Luz, J., Ranalli, T. V., Gomes, V., Pastorelli, A.. Faggiani, R., Anti, M., Jirieny, J., Clevers, H., and Marra, G. Transcriptome Pro- file of Human Colorectal Adenomas. Molecular Cancer Research, 5(12): 2007. ISSN 1541-7786. doi: 10. 10.1155//511-778MMMCC
10.1007/s1133-00200000
Michoel, T. Analytic solution and stationary phase approxi- mation for the Bayesian lasso and elastic net. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Salomon, J. A., Reinhart, A., Bilinski, A., Chua, E. J., La Motte-Kerr, W., Rönn, M. M., Reitsma, M. B., Morris, K. A., LaRocca, S., Farag, T. H., Kreuter, F,, Rosenfeld, R., and Tibshirani, R. J. The US COVID-19 Trends and Impact Survey: Continuous real-time measurement of COVID-19 symptoms, risks, protective behaviors, testing, and vaccination. Proceedings of the Nutional Academy of Sciences, 118(51), 2021. ISSN 0027-8424. doi: 10. 10.1073/pnas.21 21145511 18.4118.
National Cancer Institute. The Cancer Genome At- las Program. https://wwa www.er..cc..eoccoo abbot-nnet/orgaaiaat..omm http://///n/////eeeeeee/// research/ prruuturaa-ggnomicce
1988.
Ndiaye, E., Fercoq, O., Gramfort, A., and Salmon, J. GAP Safe screening rules for sparse multi-task and multi-class models. Advances in Neural Information Processing Systems, 3015-January:811-8111 889, 2015. ISSN 10495258.
Scheetz, T. E., Kim, K.-Y. A., Swiderski, R. E., Philp, A. R., Braun, T. A., Knudtson, K. L., Dorrance, A. M., DiBona, G. F., Huang, J., Casavant, T. L., Sheffield, V. C., and Stone, E. M. Regulation of gene expres- sion in the mammalian eye and its relevance to eye dis- ease. Proceedings of the National Academy of Sciences, 103(39):14429-14433 2006. ISSN 0027-8424. doi: 10. 10.1077//pnas.0002556101.3
Ndiaye, E., Fercoq, O., Gramfort, A., and Salmon, J. GAP Safe Screening Rules for Sparse-Group Lasso. In Ad- vances in Neural Information Processing Systems, vol- ume 29, Curran Associates, Inc., 2016a.
Ndiaye, E., Fercoq, O., Gramfort, A., and Salmon, J. Gap Safe screening rules for sparsity enforcing penalties. Jour- nal of Machine Learning Research, 18, 2016b. ISSN 15337928.
Schneider, U. and Tardivel, P. The Geometry of Uniqueness, Sparsity and Clustering in Penalized Estimation. Journal of Machine Learning Research, 23:1-36, 2022.
Pedregosa, F. and Gidel, G. Adaptive Three Operator Split- ting. In Proceedings of the 35th International Conference on Machine Learning, volume 80 of Proceedings of Ma- chine Learning Research, pp. 4085 4094, PMLR, 2018.
Shibagaki, A., Karasuyama, M., Hatano, K., and Takeuchi, I. Simultaneous Safe Screening of Features and Sam- ples in Doubly Sparse Modeling. In Proceedings of The 33rd International Conference on Machine Learning, vol- ume 48 of P Proceedings of Machine Learning Research, pp. 1577-1586. PMLR, 2016.
Pei, H., Li, L., Fridley, B. L., Jenkins, G. D., Kalari, K. R., Lingle, W., Petersen, G., Lou, Z., and Wang, FKBP51 Affects Cancer Cell Response Chemotherapy by Nega- tively Regulating Akt. Cancer Cell, 16(3):259-20) 266, 2009. doi: 10. 11101666.ccr2009007.000
ISSN 15356108.
Simon, N., Friedman, J., Hastie, T., and Tibshirani, R. A Sparse-Groud Lasso. Journal of Computational and Graphical Statistics, 22(2):231-245 2013. ISSN 106]- 8600. doi: 10.1080/ 00880/106/ 166 36000.201.688
Peng. J., Zhu, J., Bergamaschi, A., Han, W., Noh, D.-Y., Pollack, J. R., and Wang, P. Regularized multivariate regression for identifying master predictors with applica- tion to integrative genomics study of breast cancer. The Annals of Applied Statistics, 4(1), 2010. ISSN 1932-6157. doi: 1211//0-AAAAS
Thompson, R., Dezfouli, A., and Kohn, R. The Contextual Lasso: Sparse Linear Models via Deep Neural Networks. In Advances in Neural Information Processing Systems, volume 36, pp. 19940-19961. Curran Associates, Inc., 2023.
Poignard, B. Asymptotic theory of the adaptive Sparse Group Lasso. Annals of the Institute of Statistical Math- ematics, 72(1):297-328, 2020. ISSN 0020-3157. doi: 10.11077/104443-011406666.......
Tibshirani, R. Regression Shrinkage and Selection Via the Lasso. Journal of the Royal Statistical Society: Series (Methodological), 58(1)2277-28, 1996. ISSN 00359246.
Qin, Z., Scheinberg, K., and Goldfarb, D. Efficient block- coordinate descent algorithms for the Group Lasso. Math- ematical Programming Computation, 5:143-169, 6 2013. ISSN 1867-2949. doi: 15007//12322001-00000
doi:
Tibshirani, R., Bien, J., Friedman, J., Hastie, T., Simon, N., Taylor, J., and Tibshirani, R. J. Strong rules for discarding

---

Dual Feature Reduction for SGL
predictors in lasso-type problems. Journal of the Royal Statistical Society. Series B: Statistical Methodology, 74 (2):245-266, 2010. ISSN 13697412. doi: 10.1111/j. 1467-9868. 1...00444..
Zhang, W., Hong, B., Liu, W., Ye, J., Cai, D., He, X., and Wang, J. Scaling Up Sparse Support Vector Ma- chines by Simultancous Feature and Sample Reduction. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 4016-4025. PMLR, 2017.
Vidyasagar, M. Machine learning methods in the compu- tational biology of cancer. Proceedings of the Royal Society A: Mathematical,, Physical and Engineering Sci- ences, 470(2177:201444 40081, 2014. ISSN 1364-5021. doi: 10.10988ssa.2211..088..
Zhao, Y. and Huo, X. A survey of numerical algorithms that can solve the Lasso problems. WIREs Computa- tional Statistics, (54))e16022 2023. doi: https://doi.org/ 10.1002/wics.1622
Wang, and Ye, Two-Layer Feature Reduction for Sparse-Group Lasso via Decomposition of Convex Sets. Advances in Neural Information Processing Systems, 3: 2132-2140, 2014. ISSN 10495258.
Zou, H. and Hastie, T. Regularization and Variable Selec- tion Via the Elastic Net. Journal of the Royal Statistical Society Series B: Statistical Methodology, 67(2):301-320, 2005, ISSN 7412- doi: 00503.x.
Wang, J., Zhou, J., Wonka, P., and Ye, J. Lasso screening rules via dual Polytope Projection. In Proceedings of the 26th International Conference on Neural Information Processing Systems, volume 1, pp. 1070-1078. Curran Associates Inc., 2013.
Wang, J., Zhou, J., Liu, J., Wonka, P., and Ye, J. A Safe Screening Rule for Sparse Logistic Regression. In Ad- vances in Neural Information Processing Systems, vol- ume 27. Curran Associates, Inc., 2014.
Xiang, Z., Xu, H., and Ramadge. J. Learning Sparse Rep- resentations of High Dimensional Data on Large Scale Dictionaries. In Advances in Neural Information Process- ing Systems, volume 24. Curran Associates, Inc., 2011.
Xiang, Z. J. and Ramadge, P. J. Fast lasso screening tests based on correlations. In 2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2137-2140, 2012. doi: 10.1109/CASSP.
2012.6288334.
Yogatama, D. and Smith, N. A. Linguistic Structured Spar- sity in Text Categorization. In Proceedings of the 52nd Annual Meeting of the Association for Computational Lin- guistics (Volume 1: Long Papers), pp. 786-796, Strouds- burg, PA, USA, 2014. Association for Computational Linguisties. doi: 10.31 1077
Yuan, M. and Lin, Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49-67, 2006. ISSN 1369-7412. doi: 10.1111fj. 0ww.998.8..000.000.50..
Zemlianskaia, N., Gauderman, W. J., and Lewinger, J. P. A Scalable Hierarchical Lasso for GevenEnnironment Interactions. Journal of Computational and Graphical Statistics, 3104):1091-1103, 2022. ISSN 1061-8600, doi: 10.0/00/161 8600.2022..033111...
12

---

Dual Feature Reduction for the Sparse-group Lasso and its Adaptive Variant: Supplementary Materials
A. Sparse-group Lasso
A.1. Theory
A. GROUP REDUCTION
Proof of Proposition 3.1. To prove the two sets are equivalent, we need prove that for any g [m]] and k E 1]. = Cg (Ax+1)). We instead prove the contrapositive: g & Cy(++)))
9 E Ag (x++2)
Ag Ag ()k+1), So,
(x441)
| (8(A Me,,, I TgAR+1,
by definition of the candidate set
=
V Vg (((AAtt) ETgA++100 1 g.++
as Oo g.++1 = for E Rp ||llee S 1}
E Vof(3(A + 1TA 00 g.++1
g & (1(+++1),
by the KKT conditions (Equation 6).
Proof of Proposition 3.2. To prove the candidate set is a superset of the active set, we need to prove that for any E [m]] and E |[- - 1], g E (Ax+1) = E (A(+))). We instead prove the contrapositive: g & Cg(Ax+1) = & A, (A++1).
First, we rewrite the Lipschitz assumption as (using the reverse triangle inequality)
[Vof(3(A (V (+(((()++))) Vof(B(Ae))le,, S SggAAk+1 Anl
+ + 1V.
- Aul.
(18)
Now, 9 & (A6++)),
IV (B(A))() S (2(2E111 Au).
Plugging this into Equation 18 yields
I 1)) S T, A6) TolMu-1 Aal
| (V,(((+++))))lle S AgA
(Bf(((A)+1)) E TgAx++1 00 g.++11
A g,++1 {xr Ip ||ælle, S 1}
O E ( + Tg 0
g & Ay(A++1),
by the KKT conditions (Equation 6)
A.1.2. VARIABLE REDUCTION
Proposition A.1 (Theoretical SGL variable screening). For SGL. applied with any A1-1, k: E 1 - 1), the candidate variable set,
(o(A)))) = Ei for Ag(A+t1): (Vf(((()++))) > \4t10),
such that C. (X++1) A.(++))).
13

---

Dual Feature Reduction for SGL
Proof of Proposition A.I. The proof strategy is similar to that of Proposition 3.1. To prove the two sets are equivalent, need prove that for any i gg such that g Ag. and [ 11- Au(Aet) b i i C We instead prove
the contrapositive: i Co(+++1) è i q Av (A++1). So,
i # (x(+11) = V.f((((A++))) S Abt10, -vrf((AA++1) E A2+10 (.41
by definition of the candidate set
as $ i = {x R: |
for iE gg. Ag
0 e vef(((N)t))) + A411000 1,41 i i (x(++)));
by the KKT conditions (Equation 11).
Proof of Proposition 3.3. The proof strategy is similar to that of Proposition 3.2. To prove the candidate set a superset of the active set, we need to prove that for any E gg such that g Ag. and & [ 1], A.x(++1)) = E (o++1))): We instead prove the contrapositive: i & (Ax+1) (using the reverse triangle inequality)
i i A. First, rewrite the Lipschitz assumption as
vrf(3(A++))) (V. + + -Ad-
(19)
Now, as i & (A
& a(2 - An).
Plugging this into Equation 19 yields
V (+f(((A S ad+t1
viS(3(AE+1) E ado+ D Pik+i, as $ Pi = (x R |
O v.f(3(A++1) + act+1 1 1
& Ao(2+t)),
by the KKT conditions (Equation 6).
A.2. KKT Checks
To determine whether variable E Gg has been correctly discarded, the KKT stationarity conditions checked. Equation 11 describes the condition under which variable i E Go is inactive. Without specifying whether the group y is inactive, this can be rewritten as (by the definition of $o.+.))
(9) V (+f(((AA+)) + A - ( i S \bt10,
(20)
where v/22, fa E RvP | S }} is the subgradient of the l2 norm. To satisfy Equation 20, the unknown subdifferential, (1)) 1,6+1: is taken to be the minimum possible value. For E (0)) k+1 we have that
| <
/Pgl|=012 S VP'
||||| I Vpg by the inequality ||^||| S <Pyl
. < Pg
Hence, the values in the subdifferential are bounded by vPg. We consider the following scenarios for Equation 20:
1. v.f(3()())) A1 (x) /pgg: choose v. -/pg-
2. v.f(3(A(+1) <-hu+i ap choose 24 VPg.
3. v.f(3(((1)) E -1-AR+1 (1- A+11 a) /pol: choose y4 [1+1(1-a)
v.f(E(Ae 1))
14

---

Dual Feature Reduction for SGL
This allows Equation 20 to be expressed using the soft-thresholding operator
SS (V (p) S As+1a.
similar derivation can be found in Simon et (2013) derive conditions to check whether a group is active for SGL.
A.3. Algorithm
Algorithm A1 Dual Feature Reduction (DFR) for SGL.
Input: (A1,...,AA) Ar) R', X E R=xp, R" a E [0,
compute ( using Equation
for k = to 1 do
Cy(Ax+1) candidate groups from Proposition 3.2 C. < candidate variables from Proposition 3.3 for E 9g An(As) Cg(Ae+1)
O, + u.
Optimization set
compute (x(A++1). Ou, using Equation 1
KG, + variable KKT violations for i O,, using Equation 13
KKT check
while card(CC,) > 0 do
O 0. UK, compute 3, (.x+1),. OO, using Equation Krp variable KKT violations for & O2 using Equation end while end for
Optimization set
KKT check
Output: (al( Bagl (1)) TRP
A.4. Reduction to (Adaptive) Lasso and (Adaptive) Group Lasso
Under 1, SGL reduces to the lasso. In this case, group screening occurs and the variable screening rule reduces to the lasso strong rule (Tibshirani et al., 2010):
( S 2A++1 - AR-
Under 0, SGL reduces to the group lasso. Under this scenario, the group screening reduces to the group lasso strong rule (Tibshirani et al., 2010):
| (B,x((())))ll S (2 An),
and no variable sereening is performed. For aSGL, the rules reduce to the adaptive lasso and adaptive group lasso:
Adaptive lasso: [V.((((Ae)) & uz (2 - -xa) Adaptive group lasso: (3,((() " Wg Ax)
= B((A+t1) = 0. B(s) (Xx+11) O,
where 6,, denotes the e-norm under e = 1 (Equations 4 and 15).
15

---

Dual Feature Reduction for SGL
B. Adaptive Sparse-group Lasso
B.1. Derivation the Connection to e-norm
Full proof of Proposition 3.4. The adaptive SGL is given by
P
m
| asgl
0 +
a)
Σ
W/ v Pa ( 1|2.
The aim is to link this norm to the e-norm, in a similar way to SGL:
||3| sgl Σ (a + (1 (ll--))/Pgll a))P)))1330111
Splitting up the summation term in the adaptive lasso norm yields
0 =
1
1=1
9=1
m
02
|3
Σ
Bg 1/1361
9=1
jEG,
iEG
iJEDg.ifj
TN
Ei Eigeiii 1391361
V
|3,1
Dieg, 13:1
Σ 1,3cl
JESy
iEG
iEG
Σ Eijesooity 17518c Eieg, 1.36|
|B
Σ Uj
9=1
1
jEg,
Eijegsnity 2y13c |1360011 l|1
||ا(و) 112
||u|s)|12
Hence
P
m
|8 asgl a Σ
B:|B.|| + (1 a) Σ
wy v Pg 11000111
8=1
[|1601]1 Ei U313811
= Σ
a a11366) 11 + (1 -a) 3 v Po (1 112
(21)
|
g=1
Setting
a Σ Ei.jeg, .sj 13188
7g = =alloba))
+ (1 -a) PPg
1|
simplifies Equation 21
( Y/ (1 a)wl / Pp
(1 (1-a) 1-)aa)yy Pp
|
langl
|18600 +
[1,(00) 12
(22)
7g
79
g=1
Setting
(1 a) Wy
Yi
allows Equation 22 to be written in terms of the e-norm
m
||3 aspl
Ta (1 <0)118(0111 + +
Σ
7a |3(00 11:
16

---

Dual Feature Reduction for SGL
B.1.1. LEMMA PROOFS FOR THE CONNECTION THE e-NORM
Proof of Lemma 3.5. Under 3(9) 0 for a group g Ag the middle % becomes
Eisegmatty 1515,1
aPp, 1)
Po
lim 8(97++
||
Uis
Pg
so that Ng still exists. This can be observed by using L' L'Hôpital's 's rule and noting that for E Gg.
0
2,13.1
Σ
(1 11 1.
08;
03,
171
Proof of Lemma 3.6. Under v = 1 and w = 1, note that
Σ Dijeo, i4t) 131361
= +(1 (11- -(1-a)VPs
7g =
Py
(3(0) 11
(Pg 1)113601111
0:
Py
| 11
a)/Ps
+ a)) Pg gg-
To understand the cross summation term, note that we are summing over each 33 term Pg - times, as the matching indices are removed, that (for ease of notation, we consider 91 so that the indexing here is reset from 1)
Σ iJE
1513.1 102 + ... + +1.111pp + 1,2/01 +... +182/pp +... 18p Pp
(pi - 1)1.811 +... (pr - 1)18p1l by setting "y = 1, Vý 91, for SGL
(pI 1) Σ
|8&1 (pi - 1)11,(1111
RES
Hence, using wy = and Tg = a + (1 - (x) VPg
(1 - a a))
Pg
(1 PP Pa
Tg
6
6g-
Yg
Tg
Tg
B.2. Theory
B.2.1. GROUP SCREENING
To derive the group screening rule for aSGL, we compare the formulations of SGL and aSGL in terms of the e-norm (Equations 3 and 15):
m
m
|
( Ii,, ||8
Σ
g lr;
Therefore, the derivation for the group screening rule for aSGL is identical to that of SGL (Section 3.1.1) replacing Ty with Ng and lle, with || lleg. The group screening rule is given by: discard a group if
I [v,(((W))) S (2 >x),
(23)
and is formalized in Propositions B.1 and B.2.
17

---

Dual Feature Reduction for SGL
Proposition B.1 (Theoretical aSGL group screening). For aSGL applied with any A k 11, the candidate group
set,
Cg(A++1)= = t0 E [m] I1 [Vg/((()++)))]] > (g
such that (x(A)+1) = (A(1++)).
Proof. The proof is identical to that of Proposition 3.1 replacing Tg with To and || II with || llg (see Appendix A.1.1). Proposition B.2 (DFR-aSGL group screening). For aSGL applied with any A k E II - 1}, assuming that
- - -o S 7g11441 - -xb
for all g E [m]) then the candidate group set,
C,(A++1) = fg E [m] [VV > 122xA-11 - Ad1}
such that Ag ( (2(
Proof. The proof is identical to that of Proposition 3.2 replacing Tg with 7g and II, with || -lll, (see Appendix A.1.1).
B.2.2. VARIABLE SCREENING
The construction of the variable screening rule for aSGL is very similar that SGL (Section 3.1.2). The KKT stationary conditions for aSGL for inactive variable in active group (in comparison to Equation 12 for SGL)
-V, (V.(((()++)) E A 22,41
Therefore, the derivation of the rule is identical, replacing a with ongy The variable screening rule given by: discard variable
V v.I(((Ae)) S o2; AR).
(24)
and is formalized in Propositions B.3 and B.4.
Proposition B.3 (Theoretical aSGL variable screening). For aSGL applied with any A4t1, k II 1], the candidate variable set,
(o(A+))) E for E Ag(A+1)) VV. Ant1auih
such that C. (Ax+1) A.(A++1).
Proof. The proof is identical to that of Proposition A. replacing a with ov; (see Appendix A.1.2).
Proposition B.4 (DFR-aSGL variable screening). For aSGL applied with any Act1, k E Il - 1], assuming that
- - ae - det1).
for all for g E (Ax+1)), then the variable candidate set,
(2++1) = E for E Ag(A++1) (+ a0,(222+1 Ac)],
such that Ao(A++)) Co(A++)).
Proof. The proof identical that of Proposition 3.3 replacing a with on; (see Appendix A.1.2).
18

---

Dual Feature Reduction for SGL
B.3. Choice of Adaptive Weights
The adaptive weights chosen according Mendez-Civieta al. (2021) as
Wg
91
1
ba
where q1 is the first principal component from performing principal component analysis on and b by are chosen by the user, often in the range [0, 2]. The weights shown for bj 0.1 in Figure Al.
1.4
Weight
Weight
12
100
200
400
600
800
1000
10
15
20
Variable index
Group index
Figure A1: The weights, (v, w), for aSGL, used in Figure 5 (right), where p 1000, 200, m 22, 0.3, 61 0.1, and 0.95.
B.4. Algorithm
Algorithm A2 Dual Feature Reduction (DFR) for aSGL
Input: (Ag...., A²) RR X IT*P, E R" [0, 1]
compute (A1) using Equation 1, replacing the SGL norm with Equation for = to do
(x(+1+1) candidate groups from Equation 23
(x(++))) candidate variables from Equation 24 for i Gy \ Ao Cg(Ax41)
O, U.
Optimization set
compute (2x+1)), O., using Equation replacing the SGL norm with Equation 14 K,, +- variable KKT violations for i O.. using Equation 17
KKT check
while card(AC..) > 0 do
O - -OOUKG compute 3, (Ax4t))) E O., using Equation 1, replacing the SGL norm with Equation 14 Kc, f- variable KKT violations for & O., using Equation 17
Optimization set
KKT check
end while
end for
Output: Busgl (Az)....... ..., Busel (Nt) RP
19

---

Dual Feature Reduction for SGL
C. Competitive Feature Reduction Approaches
Table A1: summary of the four screening rules for SGL considered.
Rules (discard if true)
Method
Type
Variable
Group
DFR-aSGL
Heuristic
V.f(3(Au)) & ar, (2Ax+1 - AR)
IV o Me, S 12 Ax)
DFR-SGL
Heuristic
V v.J(B(>=)) S. 5a(2AR+1 -
IV ( S Tg(22)11 As) |[(V (B((a))) /Pp(1 a2) A1)
sparsegl
Heuristic
GAP safe
Exact
X, +r||X <T
To (1 (1-a) (1-1))
sparsegl sparsegl is a screening rule proposed by Liang et al. (2022) and performs a single layer of group sereening. The rule based on the strong screening framework (Tibshirani et al., 2010) and the first order condition derived Simon et al. (2013), i.e., that group g E [m] inactive if
|S(V ( A & vP aa)x+1
As the gradient at l + is not available, the following Lipschitz assumption on the & norm is used:
Axa))) ( Ax+1a) - ((V,,( A2a)]l2 S vPy(1 a)|Ax+1 - Apl-
This leads the sparsegl screening rule (via the triangle inequality): discard group y if
||S(V, (((e)) Apa)ll2 S v - (2 Aς).
This screening rule uses a different Lipschitz assumption the group-level (DFR: Equation 9), which in turn leads to different group-level rule (DFR: Equation 10). Our Lipschitz assumption is more consistent with the work of Tibshirani al. (2010), as the assumption is with regards the dual norm of the full SGL norm, rather than just the group component.
GAP Safe An exact feature reduction method for SGL was proposed in Ndiaye al. (2016a) under linear regression. The approach makes use of the subdifferential inclusion equation of Fermat's 's rule (Bauschke & Combettes, 2017):
X + (A.ll)) E 0|| |laa (10.1
where Ô is the solution the dual formulation of Equation Using this, exact (theoretical) rules are derived determine which variables and groups inactive the optimal solution. The rules are theoretical they rely on A which is not available in practice. Instead, safe region is constructed that contains the optimal dual solution; in Ndiaye et al. (201 6a) it is taken as sphere, but other regions can also be used (such as domes). Due to the strict requirements on these safe regions, the reduction is generally more conservative.
The safe sphere defined as B(O,,7) r) with center θ₂ and radius r. An ideal region would be such that is small and the center is close H-ll, Using this safe region, the GAP safe rules at d&+1 are derived as, for a variable and group 9.
Variable screening: |X] O2l + <T 3((A++1) 0.
Group screening: Ty < (1- (1-a)
(X++1) = O,
where
T ||S(X) 0. a)ll + +r| if |X, Oc1oo >a, (1Ix, l +r|xX,l11 a)+, otherwise.
The center O₂ and the radius derived using the duality gap and are calculated at iteration t in an iterative algorithm as
-3 (x(3))) max ( 1X (y XB(00 1lgg)
2P 2P Pheiio (3()) - D D (0,)
(r O.)
20

---

Dual Feature Reduction for SGL
where aa and D the primal and dual objectives, and Peg is the primal value iteration L. The radius and center expensive evaluate, calculated only every 10 iterations (Ndiaye et al., 20)
The above formulation combines both dynamic and sequential screening. The method can also be implemented using just sequential screening, in which the primal values used in the calculation of the center and radius are from Ag-
For both the GAP safe rules and DER, theoretically it would be possible to exactly identify the active sets, but both instead require approximations. While GAP safe has different implementations, we present the best performing versions in our studies.
D. Synthetic Data Analysis
This section complements Section 4.1 by providing further information about the simulation set-up and additional results for the synthetic data. Additional tables and figures are provided that further showcase the effectiveness of DFR, including under logistic model (Appendix D.6).
Notes: The GAP safe methods are only applicable under linear regression and caused computational issues under uneven groups. Given their poor performance in the simulations considered (Figures 1, and 3), they excluded from the remaining simulations. Additionally, note that an intercept only applied for linear models. Applying intercept centers the response data, which is not applicable to binary response. As we are interested in computational cost, not predictive performance, adding an intercept to logistic models provides no benefit.
D.1. Metrics
The following metrics are shown in the tables in the Appendix:
Ags Ag: the number of active variables/groups.
Cr, Cy: the number of variables/groups in the candidate sets.
Oy: the number of variables/groups used in the optimization process. As per Algorithms A1 and A2, O, U. However, Og not produced as O Cg U Ag. Instead, O, are the groups for which there variables present in O, to give measure of the number groups used in the optimization.
Ky, Kg: the number of variable/group KKT violations. DFR only checks for variable violations and sparsegl only checks for group violations.
O. / A. and Og / Ag: the proportion of variables/groups used the optimization against the number active. Defines how efficient the rules are. A low value best.
O₂ p and 0g m: the variable/group input proportion, as defined in Section 4. A low value is best.
& distance to no screen: & distance from the fitted values obtained with screening to without.
IF: the improvement factor, as defined in Section 4.
21

---

Dual Feature Reduction for SGL
D.2. Setup
Table A2: Default model, data, and algorithm parameters for the synthetic and real data analyses.
Category
Parameter
Values
Synthetic
Real
Model
0.95
0.95
b1 b2 (aSGL only)
0.1
0.1
Path length (l)
50
100
Path termination (At)
0.1A1
0.2X1
Path shape
Log-linear
Log-linear
Data
P 7
1000
200
m (uneven cases)
22
m (even cases)
50
Group sizes (uneven cases)
[3, 100)
Group sizes (even cases)
20
Signal / (signal strength of zero)
N(0, 4)
Variable sparsity
0.2
Group sparsity
0.2
Correlation (p)
0.3
Noise (c)
A( 1)
Algorithm (ATOS/BCD)
Maximum iterations
5000
10000
Backtracking (ATOS only)
0.7
0.7
Maximum backtracking iterations (ATOS only)
100
100
Convergence tolerance
10-
10-5
Standardization
E2
Intercept
Yes for linear Yes
Yes for linear
Warm starts
Yes
22

---

Dual Feature Reduction for SGL
D.3. Runtime Breakdown
The runtime breakdowns for two cases are presented to illustrate the computational cost of screening.
Table A3: Figure runtime breakdown.
Table A4: scheetz runtime breakdown (Figure 8).
Component
DFR-SGL
DFR-aSGL
Component
DFR-SGL.
DFR-aSGL
Fitting algorithm
88%
86%
Fitting algorithm
77%
65%
6-norm evaluation
3.9%
3.6%
6-norm evaluation
0.46%
0.57%
Group screening
3.9%
3.6%
Group screening
0,46%
0.58%
Variable screening
0.01%
0.01%
Variable screening
0.01%
0.02%
KKT checks 0.6%
0.6%
KKT checks
0.2%
0.3%
D.4. Additional Results for the Linear Model
Improvemest factor
12.5
7.5
5.0
0.0
0.1
Signal strength
DFR-SGL
DFR-aSGL
sporsegl
Figure A2: The improvement factor for the strong rules applied synthetic data, under the linear model, function of the signal strength (left) and (right), with 95% confidence intervals.
0.30
lapur pooortion
0.25
0.15
0.10
0.00
0.1
0.3
0.5
0.7
0.9
Signal strength
DFR-SGL
DFR-aSGL
sparsegl
Figure A3: The input proportion for the strong rules applied synthetic data, under the linear model, as function of the signal strength (left) and (right), with 95% confidence intervals.
Inpor proportion
Linear
Logistic
0.06
0.05
0.08
0.03
Figure A4: The input proportion of DFR-aSGL. under different weights by b22 shown for the linear (left) and logistic (right) models, with 95% confidence intervals.
23

---

Dual Feature Reduction for SGL
1.4.1. TABLES GAP SAFE SIMULATIONS
Table A5: Group screening metrics corresponding to the GAP safe simulations (Figures 1, 2, and 3) averaged over all cases and path points, shown with standard errors.
CARDINALITY
INPUT PROPORTION
METHOD
Ag
C,
O₂
Kg
Og / A
Og /
DFR-ASGL
7.86 0.02
10.00
10.00 +000000 0.03
1.1551 +
0.2001 + 10
DFR-SGL
8,16 0.02
10.47
1.02
10.47 0.02
1.1937 1
0.2094 1 10
DFR-SGL BCD
8.45
0..02
10.07 = 0.02
1.1893 8 10
+ 0.2015
SPARSEGI.
8.16
0.004
11.72 + 0.04
3 x 10
+2x 10-5
1.5837 0.0111
0.2344
10
GAP SEQUENTIAL
8,55 0.02
10.20
10.20 = 0.02
1.1934 0.0018
0.2040 + 4
GAP DYNAMIC
8.55 0.02
10.20 0.002
10.20 + 0.02
1.0117
0.2039 t 4 10
Table A6: Variable screening metrics corresponding to the GAP safe simulations (Figures 1, 2, and 3) averaged over all cases and path points, shown with standard errors.
CARDINALITY
INPUT PROPORTION
METHOD
A
C.
O,
K.
O, / A.
O
DFR-ASGL
61.77 + 0.22
36.70
0.34
95.99
0.42
0.0092 = x 10
1.3464 0.0011
+ 4 10
DFR-SGL.
65.37 1 0.22
38.30 -: 0.34
101.07
1 0.43
0 0:
1.3576 1: 0.0011
0.1011 + 10
DFR-SGL BCD
72.69 + 0.25
23.35 0.09
93.11 + 0.32
0.0250 + 0.0054
1.3281 + 0.0011
0.0931 + 3 10
SPARSEGL
65.41 1 0.22
234.39 0.700
234.39 1 0.70
11.5604 1 0.2001
0.2444177 10
GAP SEQUENTIAL
73.01 + 0.25
93.85 0.33
93.85 + 0.33
1.3292 + 0.0022
0.0938 + 10
GAP DYNAMIC
73.01
+ 0.25
93.38 0.33
93.38 + 0.33
1.2927 0.0020
0.0934 + 3 10
Table A7: Model fitting metrics corresponding to the GAP safe simulations (Figures 1, 2, and 3) averaged over all cases and path points, shown with standard errors. The timing results are the average time taken to evaluate the full path on a dataset.
TIMINGS
ITERATIONS
6 DISTANCE
FAILED CONVERGENCE
METHOD
No SCREEN (S)
SCREEN (8)
No SCREEN
SCREEN
TO NO SCREEN
No SCREEN
SCREEN
DFR-ASGL
659.89
154.92
271.60
174.55
4 10
43 10
0
0
DFR-SGI.
685.49 6.27
157.1 1 2.24
286.13 2.04
185.47
2.43
4 10
13x 10
0::0
0
DFR-SGL BCD
131.46 1.12
32.02
0.55
5.55 + 0.08
252.26
1.94
139.39
1.15
2 10
+1 10
0
0
6.27
278.18
0.10
GAP SEQUENTIAL
0.11
10-3
0,11 :
10
0.98 1 0.01
GAP DYNAMIC
0.11 #3 10-3
0.11
x
10
1.00
D.4.2. TABLES FOR OTHER SIMULATIONS
Table A8: Group screening metrics corresponding to the other linear model simulations (Figures 4, 6, A2, and A3 and Table 1) averaged over all cases and path points, shown with standard errors.
CARDINALITY
INPUT PROPORTION
METHOD
A
C
O
K
O, / A,
DFR-ASGL
8.46 0.03
10.55 + 0.04
10.54 4 0.04
3 x 10
0.1968 + 4 10
DFR-SGL
8.51 0.03
11.16 0.04
11.16 + 0.04
1.1792 6 10
0.2084 4 10
SPARSEGL
8.51 0.03
10.25 0.04
10.25 + 0.04
8
1.2191 0.0026
0.2083 x 10
24

---

Dual Feature Reduction for SGL
Table A9: Varible screening metrics corresponding to the other linear model simulations (Figures 4, 5, 6, A2, and A3 and Table 1) averaged over all cases and path points, shown with standard errors.
CARDINALITY
INPUT PROPORTION
METHOD
A.
O,
O, / A.
O p
DFR-ASGL
50.95 + 0.13
29.90 = 0.15
78.92 + 0.22
0.0297 4 10
1,4805 = 0.0011
0.0634 + 2 10
DFR-SGL
54.40
+ 0.13
32.78 0.15
85.16 + 0.22
4 x 10
14x 10
1.5057 0.0014
0.0676 2 10
SPARSEGL
54.42
0.13
331.75
331.75 + 0.86
11.4845 0.0882
0.2205 10
Table A10: Model fitting metrics corresponding to the other linear model simulations (Figures 4, 5, 6, A2, and A3 and Table 1) averaged over all cases and path points, shown with standard errors. The timing results are the average time taken to evaluate the full path on a dataset.
TIMINGS
ITERATIONS
E, DISTANCE
FAILED CONVERGENCE
METHOD
No SCREEN (S)
SCREEN (8)
IF
No SCREEN
SCREEN
TO NO SCREEN
No SCREEN
SCREEN
DFR-ASGL
723.99 1: 34.04
77.00 = 2.23
11.19 0.41
0.41
415.87 + 7.41
227.93 3.51
x 15 x 10"7
0
0
DFR-SGL
251.59 + 3.93
51.73 = 0.89
9.15
113.10
240.78 + 3.78
6 10100"4:0 10
0
0
SPARSEGL
251.59 3.93
138.18 = 5.62
3.59 =
413.10
353.66 5.79
x +5x
0
0
D.4.3. CROSS-VALIDATION
Table A1l: The improvement factor for the strong rules applied to synthetic data, under the linear and logistic models, with 10-fold CV, with standard errors.
Method
Linear
Logistic
DFR-aSGL.
3.9 0.2
2.3 0.1
DFR-SGL sparsegl
2.6 0.1 0.3
2.1 0.2
0.1
D.5. Interaction Models
Order
0.15
Input proportion
Order
0.10
0.05
0.00
DGL.
DFR-aSGL
sparsegl
Figure A5: The input proportion for the strong rules applied synthetic interaction data, under the linear model, with standard errors. The parameters of the data were set p 400, 80, and m 52 groups of sizes in [3, 15). The interaction input dimensionality was PO, 2111 and po, 7338, with no interaction hierarchy imposed. The sparsity proportion of interaction variables was set to 0.3 (with the same signal as the marginal effects).
25

---

Dual Feature Reduction for SGL
D.6. Results for the Logistic Model
The data input components X, B, and for the logistic model were generated for the linear models. The class probabilities for the response were calculated using oXX E), where is the sigmoid function.
Table A12: The improvement factor for the strong rules applied synthetic interaction data, under the logistic model, with standard errors. The parameters of the data were set p 400, n. 80, and 52 groups of sizes in [3, 15]. The interaction input dimensionality was PO₂ 2111 and po; = 7338, with no interaction hierarchy imposed. The sparsity proportion of interaction variables was set to 0.3 (with the same signal as the marginal effects).
Interaction
Method
Order
Order
DFR-aSGL
6.7 0.4
12.2 + 0.4
DFR-SGL
5.8 0.2
8.3 + 0.4
sparsegl
1.0 0.1
2.1 + 0.3
Order
Input proportion
Order
0.20
0.15
0.10
0.03
0.00
DER-SCIL SCL
DGG-
sparsegl
Figure A6: The input proportion for the strong rules applied to synthetic interaction data, under the logistic model, with standard errors. The parameters of the data were as p 400, n 80, and m 52 groups of sizes in [3, 15). The interaction input dimensionality was Po, 2111 and Po, 7338, with no interaction hierarchy imposed. The sparsity proportion of interaction variables was set to 0.3 (with the same signal as the marginal effects).
Imprevement Sactor
0.2
0.3
0.4 0.0
0.2
0.6
0.8
Sparsity propection
Correlation (p)
SGL
DFR.
sparsegl
Figure A7: The improvement factor for the strong rules applied to synthetic data, under the logistic model, as function of the sparsity proportion (left) and data correlation (right), with 95% confidence intervals.
26

---

Dual Feature Reduction for SGL
0.8
Topat proportion
0.0
0.1
0.2
0.4 Correlatinn (o)
Sparsity proportion
DFR-SGL
DFR-aSGL
sparsegl
Figure A8: The input proportion for the strong rules applied to synthetic data, under the logistic model, function of the sparsity proportion (left) and data correlation (right), with 95% confidence intervals.
Improveneet Factoo
Signal
SGL
DFR-aSGL
sparsegl
Figure A9: The improvement factor for the strong rules applied synthetic data, under the logistic model, as function of the signal strength (left) and (right), with 95% confidence intervals.
0.5
Togus proportion
0.4
0.3
0.2
0.1
Signal strength
DFR-SGL
DFR-aSGL
sparsog]
Figure A 10: The input proportion for the strong rules applied to synthetic data, under the logistic model, function of the signal strength (left) and a (right), with 95% confidence intervals.
1. TABLES FOR LOGISTIC SIMULATIONS
Table A13: Group screening metrics corresponding to the logistic model simulations (Figures A7, A8, A9, and A10 and
Table A12) averaged over all cases and path points, shown with standard errors.
CARDINALITY
INPUT PROPORTION
METHOD
A
C
Kgg
O A,,
0 /
DFR-ASGL
8.08 0.01
8.55 + 0.02
8.55 + 0.02
1.0292 + x 10
0.3528 + 6 x 10
DFR-SGL
8.56 0.02
9.30 + 0.02
9.30 0.02
1.0554 + 10"*
0.3823 +: x 10
SPARSEGL
8.56 0.02
8.87 + 0.02
8.87 0.02
5 10
2 10
1.0808 + 0.0029
0.3697 + 7 10
27

---

Dual Feature Reduction for SGL
Table A14: Variable screening metrics corresponding to the logistic model simulations (Figures A7, A8, A9, and A 10 and Table A12) averaged over all cases and path points, shown with standard errors.
CARDINALITY
INPUT PROPORTION
METHOD
A.
O,
K.
O, / A..
O. p
DFR-ASGL
79.76 + 0.24
36.27
0.10
111.79 + 0.32
0.0155 13 10
1.5263 = 0.0012
0.1075 + 3 10
DFR-SGL
84.23
+ 0.25
40.13
0.12
120.84 + 0.34
9 47 10
1.5503 0.0013
0.1154 + 3 10
SPARSEGL
84.26
0.25
445.21 1.05
445.21 + 1.05
11.6349 0.1015
0.4004
Table A15: Model fitting metrics corresponding to the logistic model simulations (Figures A7, A8, A9, and A10 and Table A12) averaged over all cases and path points, shown with standard errors. The timing results are the average time taken to evaluate the full path on a dataset.
TIMINGS
ITERATIONS
& DISTANCE
FAILED CONVERGENCE
METHOD
No SCREEN (S)
SCREEN (S)
IF
No SCREEN
SCREON
TO NO SCREEN
No SCREEN
SCREEN
DFR-ASGL
132.10 7.21
34.75 4: 0.66
3.37 0.05
125.73 2.17
78.95 = 1.10
9 10
10 4 x
10 11
0
0::
DFR SGL
103.16 +: 3.37
34.27 +: 0.56
3.43 0.05
141.42 2.92
87.88 1.30
x 10
4
12
0
0::0
SPARSEGI.
109.93
10
0
28

---

Dual Feature Reduction for SGL
E. Real Data Analysis
E.1. Data Deseription
brcal: Gene expression data for breast cancer tissue samples.
Response (continuous): Gene expression measurements for the BRCAI gene.
Data matrix: Gene expression measurements for the other genes.
Grouping structure: Variables are grouped via singular value decomposition.
scheetz: Gene expression data in the mammalian eye.
Response (continuous): Gene expression measurements for the Trim32 gene.
Data matrix: Gene expression measurements for the other genes.
Grouping structure: Variables are grouped via singular value decomposition.
trust-experts: Survey response data as to how much participants trust experts (e.g., doctors, nurses, scientists) to provide COVID-19 news and information.
Response (continuous):
The trust level of each participant.
Data matrix: Contingency table including factors about participants (e.g., age, gender, ethnicity).
Grouping structure: The factor levels grouped into their original factors.
adenoma: Transcriptome profile data to identify the formation of colorectal adenomas, which are the predominate cause of colorectal cancers.
Response (binary): Labels classifying whether the sample came from an adenoma or normal mucosa.
Data matrix: Transcriptome profile measurements.
Grouping structure:
Genes were assigned to pathways from all nine gene sets on the Molecular Signature
Database.
celiac: Gene expression data of primary leucocytes to identify celiac disease.
Response (binary): Labels classifying patients into whether they have celiac disease.
Data matrix: Gene expression measurements from the primary leucocytes.
Grouping structure: Genes were mapped to pathways from all nine Molecular Signature Database gene sets.
tumour: Gene expression data of pancreative cancer samples to identify tumorous tissue.
Response (binary): Labels classifying whether samples are from normal or tumour tissue.
Data matrix: Gene expression measurements.
Grouping structure: Genes were mapped to pathways from all nine Molecular Signature Database gene sets.!
Table A16: Dataset information for the six datasets used in the real data analysis.
Dataset
p
n
m
Group sizes
Type
Source
brcal
17322
536
243
[1, 6505]
Linear
(National Cancer Institute, 1988)²
scheetz
18975
120
85
[1, 6274]
Lincar
(Schcetz et al., 2006)²
trust-experts
101
9759
7
[4, 51]
Linear
(Salomon et al., 2021)*
adenoma
18559
64
313
[1, 741
Logistic
(Sabates-Bellver Bellver et al., 2007)
celiac
14657
132
276
[1, 617
Logistic
(Heap et al., 2009>
tumour 18559
52
313
[41
Logistic
(Pei et al., 2009; Ellsworth et al., 2013; Li et al., 2016)*
E.2. Additional Results for the Real Data
2 "downloaded on 08/2024 from https:://owwbiiosttttto wtt.:isstttttttttaaa.o..o github. ht/dattt jsp. 'downloaded on 08/2024 from https: com/da jwodoo/////ooooooooo jncdon/spasseggllll sparsse1,
'downloaded on 08/2024 from gsea-n.siddb
'downloaded on 08/2024 from https://www.ndt nlm. nih. gov/
29

---

Dual Feature Reduction for SGL
Table A17: Group sereening metries corresponding to the real data studies (Figures 8, 9, and 10) averaged over all path points, shown with standard emors.
Og m
0.0046 +3x 10
0.0430 + 0.0022
0.0274 + 0.0017
0.1044 1 0.0085
0.0799 i 0.0074
0.0700 1 0.0067
0.0171 + 0.0013
0.0284 + 0.0025
0.0257 + 0.0022
0.0282 + 0.0019
0.0101 + 0.0014
0.0086 0.0012
0.4877 + 0.0118
0.4820 + 0.0113
0.4719 + 0.0127
0.0159 11 10-3
0.0281 + 0.0012
0.0216 1 10-4
INPUTT PROPORTION
0 Ag
1.0053 0.0053
4.6121 + 0.2003
0.0942
1.4412 1
0.0276
1.2415 0.0213
1.0439 + 0.0121
1.2023 + 0.0218
1.1062 0.0176
1.0515 + 0.0121
1.3537 0.0531
0.0486
1.0000 + 0.0000
1.0185 + 0.0117
1.0000 + 0.0000
1.2228 + 0.0240
1.8253 + 0.0357
1.4276 + 0.0351
2.6141
1.4367
1.1829
Kg
0
0
0
0
0
0
Og
1.42 + 0.08
13.46 + 0.68
8.59 + 0.53
28 28.82 1222222 2.35
22.04 2.03
19.32 1 1.84
4.16 + 0.33
6.90 1 0.60
6.25 + 0.55
2.39 + 0.16
0.86 + 0.12
0.73 + 0.10
3.41 - 0.08
3.37 + 0.08
3.30 + 0.09
4.98 0.30
8.80 + 0.38
6.77 6 : 0.29
CARDINALITY
Cg
1.38 + 0.08
13. 23.46 + 0.68
8.59 0.53
29.94 1 2.45
22.04 + 2.03
19.32 1.84
0.33
6.90 1 0.60
6.25 0.55
0.16
0.86 + 0.12
0.73 - 0.10
0.09
3.37 + 0.08
0.04 0.04 + 0.02
4.96 0.30
8.80 + 0.38
6.77 0.29
4.17
2.37
3.37
Ag
1.41 - 0.08
3.30 + 0.21
3.32 + 0.21
19.41 + 1.65
15.35 = 1.45
15.36 + 1.46
3.84 = 0.26
5.60 + 0.48
5.59 0.48
2.19 1 0.12
0.61 0.08
0.61 + 0.08
3.41 0.08
3.34 1 0.08
3.30 0.09
4.94
5.02 - 0.24
5.02 + 0.24
TRUST-EXPERIS
TRUST-EXPERTS
TRUST-EXPERTS
ADENOMA
ADENOMA
ADENOMA
DATASET
SCHEETZ
SCHEETZ
SCHEETZ
TUMOUR
TUMOUR
TUMOUR
CELIAC
CELIAC
CELIAC
BRCA1
BRCAI
BRCA1
METHOD
DFR-ASGL
DFR-SGL
SPARSEGL
DFR-ASGL
DFR-SGL
SPARSEGL
DFR-ASGL
DFR-SGL
SPARSEGL
DFR-ASGL
DFR-SGL
SPARSEGL
DFR-ASGL
DFR-SGL
SPARSEGL
DFR-ASGL
DFR-SGL
SPARSEGL
30

---

Dual Peature Reduction for SGL
Table A18: Variable screening metrics corresponding to the real data studies (Figures 8, 9, and 10) averaged over all path points, shown with standard errons.
O. /
2 10 2 x 10
0.0041 + x 10 -4
0.0166 - 0.0011
0.0047 = 4 10
0.0042 1 5 x 10 -4
0.0096 = 4 10
0..174 X 10-
0.3904 0.0108
0.0337 + 0.0011
0.0041 + 0.0053
0.1597 + 0.0203
0.0491 + 0.0021
0.0531 + 0.0029
0.2121 + 0.0065
6 10-4 1 5 10-6
0.0011 = X 10
0.0133 = x -4
4
INPUT PROPORTION
O, O A.
1.1828 + 0.0259
8.9655 1 0.7776
26.0099 + 1.6361
0.0442
16187 1 0.0455
111176
0.001
1 2,6945
95.8171 66.017
1.3678 1.00050
16688 1 00.055
6.1978 0.3477
+ 0.0066
00.055
4.9628 0.14448
0.0394
2.3432 1 00.1889
1 0.9193
1.6545
27.6753
1.1998
3,938
1.0231
1.0008
1.3363
25.9945
KC.
0.0404 = 0.0199
0
0.1010 0.0337
0T0
0.0505 = 0.0221
0
0.0142
0
0.0199
0
0.0142
0.0142
0
0.0202
0.0202
0.0404
O.
4.15 0.30 0.30
75.51 + 3.85
20.81
68 = 6.51
61.91 1 6.77
1019.13 106.78 78
165.83 = 6.19 6.19
301.8 8
6762.31 + 186.59
1019.11 20.19
836 836.23 23 + 101.02
3030.24 + 385.60 60
4.96 - 0.21
5.36 = 0.29
21.42 = 0.65
10.57 - 0.84
20.34 = 1.09
246.8 15.39
308.
CARDINALITY
C.
0.81 0.11
61.47 11 2.90
308 308.64 63.64422 + 20.81
29. 63 2.79
26.11 1 3.05
1019.13 + 106.78 78
31.93 1: 2.77
63.42 42.42 4.26
186.59
281 911.911 + 13.50
344.92 1 50.24
3030.24 : 385.60 60
0.17 0.04
0.35 + 0.07
0.27 0.14
3.31 0.31
9.87 + 0.50
246.8 1 39
6762.31
A.
3.38 0.21
14.38 1.11
14.41 = 1.12
40.13 33.85
37.13 3.992
37.26 3.99
135 75 4.00
241. 69 7.34
241 59 7.34
743 43 13.50
501.78 + 60.68
501. 49 + 60.66
4.83 0.22
5.10 0:28
5.07 0.28
7. 7.40 0.58
10.70 0.72
10.70 0.72
TRUST-EXPERTS
TRUST-EXPERIS
TRUST-EXPERTS
DATASET
ADENOMA
ADENOMA
ADENOMA
SCHEETZ
SCHEETZ
SCHEETZ
TUMOUR
TUMOUR
TUMOUR
CELIAC
CELIAC
CELIAC
BRCAI
BRCA1
BRCAI
DER DFR-ASGL ASGL
DFR ASGL
DFR DFR-ASC ASGL
DFR DFR-ASCO ASGL
DFR ASSL
DFR -SGGL
METHOD
DFR-SGL S
SPARSEGL
DFR DFR-SGL SGL
SPARSEGL
DFR SGL
SPARSEGL
DER DFR-SGL SGL
SPARSEGL
DER DFR-SGI SGL
SPARSEGI.
DER DFR-SGI SGL
SPARSEGL
31

---

Dual Feature Reduction for SGL
Table A19: Model fitting metrics corresponding the real data studies (Figures 8, 9. and 10) averaged over all path points, shown with standard enrons. There are standard errors for the timing results as the time to calculate the whole path was evaluated.
SCREEN
U
0
0.01.00
0
0
0
0+
0
0
0
0
0.0390
0
D 0=
0
0 0
0
0
FAILED CONVERCENCE
0.0404
0.1818
No SCREEN
0.0372
0.8687 +: 0.0341
0.8687 0.0341
0
0 0
0
0+
0
0
0.5657 + 0.0501
0.1818 + 0.0390
0.0390
0
0 0
0
0.6708 4: 0.0472
0.8386 +: 0.0352
0,8586 + 0.0852
0.8384
0.1818
DISTANCE
TO NO SCREEN
16 X 10-6
+ x 10
+2 x 10 6
10
2 10 1 X 10
8 10 #3 x 10
2 9 10 10
10-12 13 x 10-18 2 10
11x10-1
10-10
6 #3 x -0
+ 10-10
#2 10-22
#3 x 10
#9 x 10
2 10 12 x 10
x 10
10
10
12x
x
x 10
12
22 #2
19
2
x 10 2
x 10 10
10
3x 5
1x 10
1x 10
3 10-9
3 10
x 10
4 10
5 10
1 10
x 10
SCREEN
110118 1101811440 14.60
4374.61 +: 280.62
5140.16 16 390.91
120.96 1 14.51
75.37 1 9.60
93. 08.29 8.86
243.27 11.70
334.16 1: 13:44
1580.73 1580.73144.22 44.23
90.27
642 38 + 136. 12
1890 386
85. 16.11 3.96
92.53 + 5.98
10.24 5.15
82.75 5.61
186 = 7.64
197 40 - 9.38
1030.48
ITERATIONS
51
SCREEN
270.62 64
90 203.30 .333
9272 90 205.31
1042. 07 1 90.79
1195.34 + 98.13
1195.31 + 98.13
33.00
1674.07 + 19.77
1674.07 = 19.77
457.60
1899 1.1 + 386 40
76.74 :: 1.80
98.36 + 4.17
98.36 + 4.17
8783.1 1. 263 36
9272 02 1: 224 60
9272.02 d: 224.60
1891 211.21 386
No
76
1653
60-1.811
800%.
9272
L.F.
1072 10
60.48 48
45.46 46.46
74.89
1341.95
84.40
183 833.00
214.17
5:38
685 08
30.31
10.21
1.92
2.28
1.61
81.48
83.40
82.93
SCREEN (8)
7.91
149. 10
15.86
10.31
16. 19
103.78
4132.01
132. 20
2246. 13
65
3.10
20
4.52
89. 43
90.03
198.
1
86
TIMINGS
No SCREEN (S)
8476 04
9017 .70
9917-700 7.70
1188. 14
1391 788
1391 .78
21889-33
22227 01
22227 001
68084.17 77
68084.77
5. 9.06
7.29
7. 7.29
7027 .969
74668 83
7468 83
90569
TRUST-EXPERTS
TRUST-EXPERTS
TRUST TRUST-EXPPER EXPERTS
ADENOMA
ADENOMA
ADENOMA
DATASET
SCHEETZ
SCHEETZ
SCHEETZ
TUMOUR
TUMOUR
TUMOUR
CELIAC
CELIAC
CELIAC
BRCA
BRCA1
BRCA1
METROD
DFR-ASGL
DI.
SPARSEGI.
DFR-ASGL
DFR-SGL
SPARSEGI.
DFR-ASGI.
DFR-SGL
SPARSEGL
DFR-ASGL
DFR-SGI.
SPARSEGL
DFR-ASGL
DFR-SGL
SPARSEGI.
ASLL
DER-SGL
SPARSEGL
32


Reviews:
[{"review_id": "a91ZIuNIlt", "reviewer": "Reviewer_fLQH", "summary": "This paper introduces a new feature reduction method in order to improve the computational complexity in solving Sparse-Group Lasso (SGL) problems.\nThe Dual Feature Reduction (DFR) method that is presented relies on two screening stages (one for inactive groups and another for inactive variables within a group) and the authors provide both theoretical groundings and experimental results to support their claims.\n\n### Update after rebuttal.\nI have acknowledged the response of the authors and will maintain my recommendation. Although for the completeness of the experimental protocol, I would still recommend including GAP Safe in all experiments.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for their time in reviewing our work and for their helpful and positive feedback. As suggested, we will increase the font sizes in the figures in the camera-ready version. With regards to your question:\n\n>Why is the GAP Safe approach not included in the evaluations of the impact of dimensionality, sparsity proportion or data correlation (Figs 4 to 6), or in the real data analysis (Figs 8 and 9)?\n\nWe found that the GAP safe rules did not work for most simulation settings we tried. We encountered issues with convergence and solution optimality. We used an implementation provided by the authors of the GAP safe rules (see Link 1), so that we are confident the issue was not with the implementation but with the approach itself. We understand that a comparison to the safe rules is important, so we included the comparison when it was possible (Figures 1-3). The poor performance of GAP safe in these settings convinced us that further investigation of the safe rules would not be fruitful.   \n\nLink 1: https://github.com/EugeneNdiaye/Gap\\_Safe\\_Rules\n\n**Please don't hesitate to ask us any additional questions about our work.**", "claims_and_evidence": "The authors provide evidence through coherent theoretical bases building upon well-established literature, and convincing empirical results.\nRefer to §Methods and evaluation criteria for more details.", "methods_and_evaluation_criteria": "The problem at hand is concerned with reducing the computational costs in solving SGL problems.\nIt is well established in the screening literature that reducing the input feature size by identifying those that will be inactive at the optimal solution induces significant computational gains to the subsequent optimization scheme.\nThe proposed DFR method does therefore make sense for the problem at hand. \n\nIn their experimental evaluations, the authors clearly distinguish the gain in performance through what they call the _improvement factor_ (the ratio of computation time in methods without and with screening approach) and the _input proportion_ (quantifying how much the feature space was reduced). Their proposed DFR method is compared against the sparsegl and GAP Safe approaches, which are two other competing rules of screening for SGL problems.", "theoretical_claims": "I have not checked the proofs of the theoretical claims.", "experimental_designs_or_analyses": "I have checked the soundness of the experimental design (both synthetic and real data). Please refer to §Questions.", "relation_to_broader_scientific_literature": "The key contributions of this paper are regarding screening rules for Sparse-Group Lasso problems, which are commonly found in various machine learning fields where relevant groups of variables need to be identified while ensuring limited spurious variables within a given group.", "essential_references_not_discussed": "N.A.", "other_strengths_and_weaknesses": "To the best of my knowledge, the DFR method proposed by the authors is novel.\nThe presentation of their approach is clear and gradual, building upon well-established existing literature.\nTheir experimental evaluation is thorough (though please also refer to §Questions).\nI believe this is a valid contribution to the broader scientific community.", "other_comments_or_suggestions": "For improved readability, I would suggest increasing the font sizes of axis labels in all the figures.", "questions_for_authors": "Why is the GAP Safe approach not included in the evaluations of the impact of dimensionality, sparsity proportion or data correlation (Figs 4 to 6), or in the real data analysis (Figs 8 and 9) ?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fwS1bxrIAf", "reviewer": "Reviewer_yM6R", "summary": "This paper introduces Dual Feature Reduction (DFR), a novel screening method to enhance the computational efficiency of Sparse-Group Lasso (SGL) and its adaptive variant (aSGL).\n\nDFR applies two-layer screening:\n- Group Reduction eliminates inactive groups using a strong screening rule based on dual norms and KKT conditions.\n- Variable Reduction further removes inactive features within active groups.\nDFR is the first bi-level strong screening method for SGL and the first screening rule for aSGL, producing the same optimal solution with significantly lower computational cost than GAP Safe and sparsegl.\n\nExperiments on synthetic and real-world datasets show that DFR achieves significant speedup while maintaining selection accuracy. It enables expanded hyperparameter tuning and makes SGL more scalable for high-dimensional learning tasks, particularly in ML modeling in genetics.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We want to thank the reviewer for taking the time to review our work and for their helpful comments. In the camera-ready version, the experimental section will be restructured to improve readability, reducing the need for frequent cross-referencing. In response to specific points raised:\n\n>Claims Requiring More Evidence: The “Improvement Factor” (IF) is a fair and reliable efficiency metric. \n\nWe use IF as it provides  **direct insight into how screening impacts the user**. For instance, an IF of $2$ means DFR halves fitting time. The metric does not inherently favour any method. We also report Input Proportion as an alternative screening impact measure, independent of computational considerations. Both metrics show DFR is effective. Additionally, raw runtimes are available in the appendix, enabling direct method comparison.\n\n>If the no-screening solver is suboptimal, IF might overstate DFR’s efficiency gain.\n\nWe considered this when using ATOS with DFR. To ensure IF does not unfairly benefit DFR over GAP safe due to the solver, we also implemented DFR with BCD (the solver used for GAP) and found similar results (Figure 1).\n\n>Claims Requiring More Evidence: DFR’s two-layer screening does not significantly increase false exclusions.\n\nWe detailed our findings on KKT violations in the 'KKT violations' section (Lines 363-377). The higher violations in aSGL likely stem from the Lipschitz assumptions' dependence on additional hyperparameters. However, across all results (synthetic and real), KKT violations for DFR-SGL and DFR-aSGL were minimal. Specifically, DFR-SGL had only one violation overall, and DFR-aSGL had a violation every $108$ fits (Figures 1-3, Table A4).\n\n>Would it be possible to offer a more detailed runtime breakdown—including screening, KKT checks, and reduced optimization—alongside the improvement factor?\n\nWe did not provide a detailed timing breakdown as our focus was on end-to-end efficiency for model users, which is most relevant in practice. We agree that a detailed runtime breakdown would be insightful. We have added an analysis below, and we will add a more comprehensive analysis with figures in the camera-ready version.\n\n*Breakdown comparison*: We ran a breakdown analysis for Figure 1 and found that on average across all values of $\\alpha$ the following breakdown occurred for fitting a full path (as a percentage of the total runtime):\n\n* **Fitting algorithm.** DFR-SGL: 88% ($133$s), DFR-aSGL: 86% ($134$s).\n* **$\\epsilon$-norm evaluation.** DFR-SGL: 3.9% ($3$s), DFR-aSGL: 3.6% ($3$s).\n* **Group screening.** DFR-SGL: 3.9% ($3$s), DFR-aSGL: 3.6% ($3$s).\n* **Variable screening.** DFR-SGL: 0.01% ($0.01$s), DFR-aSGL: 0.01% ($0.01$s).\n* **KKT checks.** DFR-SGL: 0.6% ($0.4$s), DFR-aSGL: 0.6% ($0.4$s).\n\n Additionally, we ran a breakdown analysis on the real dataset *scheetz* for fitting a full path:\n* **Fitting algorithm.** DFR-SGL: 77% ($275$s), DFR-aSGL: 65% ($244$s).\n* **$\\epsilon$-norm evaluation.** DFR-SGL: 0.46% ($1.64$s), DFR-aSGL: 0.57% ($2$s).\n* **Group screening.** DFR-SGL: 0.46% ($1.66$s), DFR-aSGL: 0.58% ($2.18$s).\n* **Variable screening.** DFR-SGL: 0.01% ($0.01$s), DFR-aSGL: 0.02% ($0.08$s).\n* **KKT checks.** DFR-SGL: 0.2% ($0.68$s), DFR-aSGL: 0.3% ($1.14$s).\n\nThe analysis shows that screening adds minimal overhead while significantly improving fitting efficiency. In Figure 1, without screening, the average fitting time was $544$s for SGL and $603$s for aSGL.\n\n>Need to clarify solver settings across all methods. Potential solver bias if different methods are not optimized consistently.\n \nSee lines 260-266 and the 'Comparison to BCD' section, where we state that DFR uses ATOS while GAP safe uses BCD. To check the solver does not bias results, we did implement DFR with BCD and found similar outcomes (Figure 1).\n\n>The paper does not compare GAP Safe in certain synthetic scenarios (e.g., increased dimensionality, uneven group sizes, logistic models) and omits it for real data.\n\nThe GAP safe rules failed in most simulations we tested, with issues in convergence and solution optimality. Using the authors' implementation (see Link 1), we are confident the problem lies with the approach, not the code. We appreciate that a comparison is important, so we included it where feasible (Figures 1-3). The poor performance suggested further investigation would not be fruitful.\n\nLink 1: https://github.com/EugeneNdiaye/Gap\\_Safe\\_Rules\n\n>Could you elaborate on how the synthetic datasets were generated for the increasing-dimensionality experiments (e.g., correlation structures, signal placement, group definitions)?\n\nFor the increasing dimension case, the setting is the same as for the other cases (described in Section 3.1), aside from the grouping structure. As explained in the 'Increasing dimensionality' section, the variables were grouped into groups of sizes $20$, so that there were $p/20$ groups for each case of $p$.\n\n**Please don't hesitate to ask us any additional questions about our work.**", "claims_and_evidence": "**Well-Supported Claims**\n1. DFR reduces computational cost while preserving solution optimality.  \n   - Evidence: The theoretical analysis shows that DFR maintains the same optimal solution by leveraging dual norms and KKT-based strong screening rules.  \n   - Experimental Support: DFR substantially lowers the number of variables to be optimized, resulting in significant computational savings.\n2. DFR outperforms existing screening methods.  \n   - Evidence: Experiments indicate that DFR reduces the input dimensionality more effectively than both GAP and sparsegl, yielding a lower optimization cost.\n3. DFR enables expanded hyperparameter tuning.  \n   - Evidence: Because DFR significantly cuts computation time, it becomes feasible to jointly tune \\(\\lambda\\) and \\(\\alpha\\) in SGL-based models.\n\n**Claims Requiring More Evidence**\n1. The “Improvement Factor” (IF) is a fair and reliable efficiency metric.  \n   - Issue: While IF appears to be based on overall runtime, the paper does not explicitly confirm this. Its fairness is uncertain for several reasons:  \n     - Baseline Dependence: If the no-screening solver is suboptimal, IF might overstate DFR’s efficiency gain.  \n     - KKT Overhead: Without a runtime breakdown for screening, KKT checks, and optimization, it’s unclear whether KKT overhead negates the benefit of reduced dimensionality.  \n     - Comparison with GAP Safe: IF could favor heuristic methods like DFR over exact approaches like GAP Safe, which might be more conservative in screening.\n2. DFR outperforms GAP Safe.  \n   - Issue: The paper does not compare GAP Safe in certain synthetic scenarios (e.g., high dimensionality, uneven group sizes, logistic models) and omits real-data comparisons. Therefore, the practical advantage of DFR over GAP Safe remains uncertain.  \n3. DFR’s two-layer screening does not significantly increase false exclusions.  \n   - Issue: There were some KKT violations, especially in the adaptive SGL setting. The frequency and consequences of these violations are not thoroughly discussed, leaving open questions about screening accuracy.", "methods_and_evaluation_criteria": "**Strengths**  \n1. DFR directly addresses SGL’s computational bottleneck, leveraging dual norms and KKT-based screening for feature reduction.  \n2. Comprehensive evaluation includes synthetic data (controlled sparsity, correlation, dimensionality) and real datasets (genetics, machine learning).  \n3. Baseline comparisons with GAP Safe (exact screening) and sparsegl (heuristic screening) contextualize DFR’s performance.  \n\n**Potential Issues**  \n1. GAP Safe is omitted from several synthetic data and real data experiments, leaving uncertainty about its practical efficiency.  \n2. Computational breakdown is missing, making it unclear how much speedup comes from screening vs. solving the reduced problem. Please consider provide absolute runtime comparisons and a screening vs. KKT vs. optimization time breakdown.  \n3. Need to clarify solver settings across all methods. Potential solver bias if different methods are not optimized consistently.", "theoretical_claims": "Generally checked and no major issues found in the submission.", "experimental_designs_or_analyses": "**Strengths**\n1. Broad Experimental Scope: The paper evaluates DFR on synthetic datasets (varying sparsity, correlation, dimensionality) and real datasets (genetics, classification), capturing both controlled and practical scenarios.  \n2. Comparison with Established Baselines: In parts of the synthetic experiments, GAP Safe and sparsegl are used as benchmarks, offering a relevant performance context for DFR.  \n3. Robustness Checks: The analyses investigate factors like signal strength, data correlation, and logistic vs. linear models, highlighting how DFR fares under diverse conditions.\n\n**Potential Issues**\n1. Incomplete Details for Increasing Dimensionality: While DFR is tested in high dimensions, the paper lacks a clear description of how the synthetic data are generated in the increasing-dimensionality scenarios. This hampers reproducibility and leaves open questions about the underlying correlation structures and signal placement.  \n2. Omission of GAP Safe in Later Synthetic Tests: In the high dimensionality, robustness and logistic experiments, GAP Safe is not included despite being considered earlier. This limits insight into how GAP Safe compares to DFR under these more diverse conditions.  \n3. Reliance on Improvement Factor: Though informative, the improvement factor might overlook the runtime overhead for KKT checks and solver differences. Absolute runtime or time-breakdown analyses could provide a fuller picture.  \n4. Solver Consistency: The paper does not clearly state if all methods (DFR, GAP Safe, sparsegl) use identical solver settings, raising potential fairness concerns in runtime comparisons.", "relation_to_broader_scientific_literature": "This paper extends strong screening frameworks (Tibshirani et al., 2010) to the Sparse-Group Lasso (SGL) and adaptive SGL, building on prior safe/strong rules for the group lasso (e.g., Ndiaye et al., 2016). By employing dual norms and KKT-based subgradients, the method follows the dual polytope projection logic found in other screening approaches (Wang et al., 2013). However, unlike GAP Safe (Ndiaye et al., 2016) which iterates for exact screening, DFR uses single-pass strong rules coupled with KKT checks. The bi-level screening structure aligns with earlier group-sparse penalties but adds a second screening stage to reduce dimensionality within each group—an innovation that refines prior one-layer methods like sparsegl (Liang et al., 2022). By extending the framework to adaptive SGL, the paper addresses the oracle property aspect (Poignard, 2020), contributing to literature on sparsity-inducing regularization with adaptive weights.", "essential_references_not_discussed": "NA", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "- Reorganize Experimental Section: The paper’s experiment write-up often references both the main text and multiple Appendix sections, causing fragmented reading. Consider consolidating key experimental details or providing clearer cross-references so readers don’t have to jump back and forth.\n- Clarify Improvement Factor Usage: Although the paper uses the improvement factor metric, it might overlook runtime overhead (e.g., KKT checks). Presenting absolute runtimes or a time breakdown alongside the improvement factor would help address fairness concerns.", "questions_for_authors": "1. The paper does not compare GAP Safe in certain synthetic scenarios (e.g., increased dimensionality, uneven group sizes, logistic models) and omits it for real data. Could you provide results—or at least partial findings—on those settings to clarify whether DFR consistently outperforms GAP Safe across diverse conditions?\n2. Would it be possible to offer a more detailed runtime breakdown—including screening, KKT checks, and reduced optimization—alongside the improvement factor?\n3. Could you elaborate on how the synthetic datasets were generated for the increasing-dimensionality experiments (e.g., correlation structures, signal placement, group definitions)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "NA"}}, {"review_id": "eVYpWZyYgk", "reviewer": "Reviewer_mWUc", "summary": "The #6131 presents dual feature reduction framework, a novel bilevel screening method specifically for the sparse-group Lasso (SGL) and its adaptive variant (aSGL). SGL works by applying $\\ell\\_1$ (variable-level) and $\\ell_2$ (group-level) shrinkage, and the paper's problem setting minimizes a convex differentiable loss function $f(\\beta)$ with a sparse-group penalty $\\\\|\\beta\\\\|\\_{\\mathrm{sgl}}=\\alpha\\\\|\\beta\\\\|\\_1+(1-\\alpha) \\sum\\_{g=1}^m \\sqrt{p\\_g}\\left\\\\|\\beta^{(g)}\\right\\\\|\\_2$. The proposed dual reduction  technique reduces computational complexity by pre-screening and eliminating inactive groups and variables before it runs into optimization. It achieves this using strong screening rules derived from dual norms and Lipschitz assumptions on the gradients, specifically leveraging the subdifferential characterizations of the SGL norm. DFR performs bi-level screening, i.e., initially at the group-level (discarding groups satisfying $\\left\\\\|\\nabla\\_g f\\left(\\hat{\\beta}\\left(\\lambda\\_k\\right)\\right)\\right\\\\|\\_{\\epsilon\\_g} \\leq \\tau\\_g\\left(2 \\lambda\\_{k+1}-\\lambda\\_k\\right)$ ) and then variable-level within active groups (discarding variables when $\\left|\\nabla\\_i f\\left(\\hat{\\beta}\\left(\\lambda\\_k\\right)\\right)\\right| \\leq$ $\\alpha\\left(2 \\lambda\\_{k+1}-\\lambda\\_k\\right)$ ). KKT conditions are checked to correct any screening violations, ensuring optimality. \n\nLastly, their numerical experiments (synthetic and real datasets) show the DFR significantly reduces computational cost while maintaining robustness and achieving identical optimal solutions to standard SGL methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and thought they invested in our manuscript and in providing helpful feedback. In the camera-ready version, we will add the pseudocode from the appendix into the main text to improve readability. In response to specific points raised:\n\n>Although KKT checks prevent incorrect feature elimination, they introduce additional computational costs that are not explicitly benchmarked in isolation. The paper notes increased KKT violations for DFR-aSGL compared to DFR-SGL, suggesting the Lipschitz assumptions are less robust when adaptive penalties are introduced.\n\nWe did not provide a detailed timing breakdown as **our focus was on end-to-end efficiency for model users**, which is most relevant in practice. We agree that a detailed runtime breakdown would be insightful. We have added an analysis below, and we will add a more comprehensive analysis with figures in the camera-ready version.\n\n*Breakdown comparison*: We ran a breakdown analysis for Figure 1 and found that on average across all values of $\\alpha$ the following breakdown occurred for fitting a full path (as a percentage of the total runtime):\n\n* **Fitting algorithm.** DFR-SGL: 88% ($133$s), DFR-aSGL: 86% ($134$s).\n* **$\\epsilon$-norm evaluation.** DFR-SGL: 3.9% ($3$s), DFR-aSGL: 3.6% ($3$s).\n* **Group screening.** DFR-SGL: 3.9% ($3$s), DFR-aSGL: 3.6% ($3$s).\n* **Variable screening.** DFR-SGL: 0.01% ($0.01$s), DFR-aSGL: 0.01% ($0.01$s).\n* **KKT checks.** DFR-SGL: 0.6% ($0.4$s), DFR-aSGL: 0.6% ($0.4$s).\n\n Additionally, we ran a breakdown analysis on the real dataset *scheetz* for fitting a full path:\n* **Fitting algorithm.** DFR-SGL: 77% ($275$s), DFR-aSGL: 65% ($244$s).\n* **$\\epsilon$-norm evaluation.** DFR-SGL: 0.46% ($1.64$s), DFR-aSGL: 0.57% ($2$s).\n* **Group screening.** DFR-SGL: 0.46% ($1.66$s), DFR-aSGL: 0.58% ($2.18$s).\n* **Variable screening.** DFR-SGL: 0.01% ($0.01$s), DFR-aSGL: 0.02% ($0.08$s).\n* **KKT checks.** DFR-SGL: 0.2% ($0.68$s), DFR-aSGL: 0.3% ($1.14$s).\n\nThe analysis shows that screening adds minimal overhead while significantly improving fitting efficiency. In Figure 1, without screening, the average fitting time was $544$s for SGL and $603$s for aSGL.\n\n> Can your DFR be extended to handle nonlinear models like kernel-based methods or neural networks, or other losses beyond simple square and logistic loss? If not, what are the fundamental barriers to applying this method beyond convex optimization problems?\n\nThe core assumptions are that the **loss function is convex and differentiable**, so DFR can be applied to any loss function satisfying these. Convexity prevents multiple optimal solutions that could complicate screening, and we require differentiability to derive the screening rules and KKT checks (we need access to $\\nabla f$). \n\nIn the manuscript, we have focused on linear and logistic regression as the solver used, ATOS, has the additional assumption that the loss must also have a Lipschitz gradient (also known as L$-smooth). However, we have also showcased that DFR can be efficiently implemented using BCD, which does not have these limitations. Therefore, a potential future direction can be the exploration of applying DFR to other loss functions. \n\n>The approach is developed specifically for convex SGL penalties, and it's unclear if it would extend to non-convex sparse-group penalties like sparse-group SCAD.\n \nDFR, like any screening rule, is derived using the subdifferentials of SGL. Extending it to other penalties requires deriving their subdifferentials. Strong rules work best with uniqueness properties on subgradients, well-behaved KKT conditions, and strong duality—properties that are often absent in non-convex cases. Thus, DFR is not inherently limited to SGL, aside from the usual challenges of screening non-convex penalties, as discussed.\n\nThe two-layer screening framework used for DFR applies to any sparse-group model, but literature on screening rules applied to non-convex penalties is relatively light (see (a)).\n\n**Please don't hesitate to ask us any additional questions about our work.**\n\n*References*\n\n(a) Alain Rakotomamonjy, et al. “Screening rules for Lasso with non-convex Sparse Regularizers”. PMLR, 2019.", "claims_and_evidence": "The evidence is generally convincing.", "methods_and_evaluation_criteria": "Yes the evaluation criteria are appropriate. The work uses metrics like the improvement factor (the ratio of computational time with and without screening) and input proportion (the fraction of variables retained after screening), which directly measure the efficiency gains from the proposed method.", "theoretical_claims": "I did not find any major mathematical mistakes.\n\nThe proof (as with many strong rules) relies on a Lipschitz assumption for the gradients. In practice, if the loss function $f$ does not have a Lipschitz‐continuous gradient (or, if the constant is underestimated), the assumption might fail. The authors are aware of this issue and use KKT checks to guard against potential violations. The derivation of the KKT check conditions (expressed in terms of a soft-thresholding operator) as well as many tools is standard in literature series.", "experimental_designs_or_analyses": "They are reasonable from my perspective.", "relation_to_broader_scientific_literature": "The paper’s key contributions build on a rich literature in sparse estimation and feature screening rule.  Prior work on screening rules for the lasso (most notably the strong rules by Tibshirani et al. 2010) and safe screening techniques (like those by El Ghaoui et al. 2010) provided the conceptual and mathematical framework for discarding inactive features prior to optimization.", "essential_references_not_discussed": "There do not appear to be any significant missing references.", "other_strengths_and_weaknesses": "- Although KKT checks prevent incorrect feature elimination, they introduce additional computational costs that are not explicitly benchmarked in isolation. The paper notes increased KKT violations for DFR-aSGL compared to DFR-SGL, suggesting the Lipschitz assumptions are less robust when adaptive penalties are introduced.\n\n- While DFR is empirically faster, the computation of the $\\epsilon$-norm has a worst-case complexity of $O(p_g log p_g)$, which could become a bottleneck for very large group sizes.", "other_comments_or_suggestions": "Integrating the pseudocode from the appendix into the main body could improve readability and help readers grasp the key ideas more effectively.", "questions_for_authors": "1. Can your DFR be extended to handle nonlinear models like kernel-based methods or neural networks, or other losses beyond simple square and logistic loss? If not, what are the fundamental barriers to applying this method beyond convex optimization problems?\n\n2. The approach is developed specifically for convex SGL penalties, and it's unclear if it would extend to non-convex sparse-group penalties like sparse-group SCAD.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper proposes a new screening rule for the Sparse Group Lasso. As many screening rules, it relies on duality and KKT conditions, at both group leavel and the feature level. 
Though this is already a fairly studied area of research, in some settings, the proposed approach seems to improve upon the bsaelines, Gap Safe Screening rules of Ndiaye et al (2016) and the more recent sparsegl.
Decision: Accept (poster)