A Half-Space Stochastic Projected Gradient Method for Group Sparsity Regularization
Authors: Tianyi Chen
Guanyi Wang
Tianyu DING
Bo Ji
Sheng Yi
Zhihui Zhu
Venue:  (2021)
Keywords: Group Sparsity
Stochastic Learning
Half-Space Projection
Group-Sparsity Identification

Abstract: Optimizing with group sparsity is significant in enhancing model interpretability in machining learning applications, e.g., feature selection, compressed sensing and model compression.  However, for large-scale stochastic training problems, effective group-sparsity exploration are typically hard to achieve. Particularly, the state-of-the-art stochastic optimization algorithms usually generate merely dense solutions. To overcome this shortage, we propose a stochastic method—Half-space Stochastic Projected Gradient method (HSPG) to search solutions of high group sparsity while maintain the convergence. Initialized by a simple Prox-SG Step, the HSPG method relies on a novel Half-Space Step to substantially boosts the sparsity level.  Numerically, HSPG demonstrates its superiority in deep neural networks, e.g., VGG16, ResNet18 and MobileNetV1, by computing solutions of higher group sparsity, competitive objective values and generalization accuracy.

Under review as a conference paper at ICLR 2021
A HALF-SPACE STOCHASTIC PROJECTED GRADIENT METHOD FOR GROUP SPARSITY REGULARIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
Optimizing with group sparsity is significant in enhancing model interpretability in machining learning applications, e.g., feature selection, compressed sensing and model compression. However, for large-scale stochastic training problems, effective group sparsity exploration typically hard to achieve. Particularly, the state-of-the-art stochastic optimization algorithms usually generate merely dense solutions. To overcome this shortage, we propose a stochastic method Bastood-Hall---sssp Half-space Stochastic Projected Gradient (HSPG) method to search solutions of high group sparsity while maintain the convergence. Initialized by a simple Prox-SG Step, the HSPG method relies on a novel Half-Space Step to substantially boost the sparsity level. Numerically, HSPG demonstrates its superiority in deep neural networks, e.g. VGG16, ResNer18 and MobileNetV1, I, by computing solutions of higher group sparsity, competitive objective values and generalization accuracy.
INTRODUCTION
In many recent machine learning optimization tasks, researchers not only focus on finding solutions with small prediction/generrlizatioo model by filtering out redundant parameters and achieving slimmer model architectures. One technique to achieve the above goal is by augmenting the sparsity-inducing regularization terms to the raw objective functions to generate sparse solutions (including numerous zero elements). The popular l (-regularriztion promotes the sparsity of solutions by element-wise penalizing the optimization variables. However, in many practical applications, there exist additional constraints on variables such that the zero coefficients are often not randomly distributed but tend to be clustered into varying more sophisticated sparsity structures, e.g.. disjoint and overlapping groups and hierarchy (Yuan & Lin, 2006; Huang et al., 2010; 2009). As the most important and natural form structured sparsity, the disjoint group-sparsity regularization, which assumes the pre-specified disjoint blocks of variables are selected (non-zero variables) or ignored (zero variables) simultaneously (Bach et al., 2012), serves momentous role in general structured sparsity learning tasks since other instances such as overlapping group and hierarchical sparsity are typically solved by converting into the equivalent disjoint group versions via introducing latent variables (Bach et al., 2012), and has found numerous applications in computer vision (Elhamifar et al., 2012), signal processing (Chen & Selesnick, 2014), medical imaging (Liu et al., 2018), and deep learning (Scardapane et al., 2017), especially on the model compression of deep neural networks, where the group sparsity is leveraged to remove redundant entire hidden structures directly.
error but also concentrate on improving the interpretation of
Problem Setting. We study the disjoint group sparsity regularization problem which can be typically formulated as the mixed f f./p-rregularizatio problem, and pay special attention to the most popular and widely used instance p as 2 (Bach et al., 2012; Halabi et al., 2018),
minimize 1E
(Z(Z) def + +c))=A((x)) N
Σ fl(z) 1=1
A ΣΣ |لواا||
(1)
9EG
where À > 0 is weighting factor, ||||| denotes fn-norm, f(x) is the average of numerous N continuously differentiable instance functions fo : IR" + IR, such as the loss functions measuring the deviation from the observations in various data fitting problems, Q(x) is the so-called mixed f /
Group sparsity defined of zero groups where zero group means all its variables are exact zeros.

---

Under review as a conference paper at ICLR 2021
norm, G is prescribed fixed partition of index set I {1, .n), wherein each component g E 9 indexes a group of variables upon the perspective of applications. Theoretically. a larger À typically results in higher group sparsity while sacrifices more on the bias of model estimation, hence À needs to be carefully fine-tuned to achieve both low f and high group-sparse solutions.
Literature Review. Problem (1) has been well studied in deterministic optimization with various algorithms that are capable of returning solutions with both low objective value and high group sparsity under proper (Yuan & Lin, 2006; Roth & Fischer, 2008; Huang et al., 2011; I; Ndiaye et al., 2017). Proximal methods are classical approaches to solve the structured non-smooth optimization (1), including the popular proximal gradient method (Prox-FG) which only uses the first-order derivative information. When N is huge, stochastic methods become ubiquitous to operate on a small subset to avoid the costly evaluation over all instances in deterministic methods for large-scale problems. Proximal stochastic gradient method (Prox-SG) (Duchi & Singer, 2009) is the natural stochastic extension of Prox-FG. Regularized dual-averaging method (RDA) (Xiao, 2010; Yang et al., 2010) is proposed by extending the dual averaging scheme in (Nesterov, 2009). To improve the convergence rate, there exists set of incremental gradient methods inspired by SAG (Roux et al., 2012) to utilizes the average of accumulated past gradients. For example, proximal stochastic variance-reduced gradient method (Prox-SVRGe (Xiao & Zhang, 2014) and proximal spider (Prox-Spider) (Zhang & Xiao, 2019) are developed to adopt multi-stage schemes based on the well-known variance reduction technique SVRG proposed in (Johnson & Zhang, 2013) and Spider developed in (Fang et al., 2018) respectively. SAGA (Defazio et al., 2014) stands as the midpoint between SAG and Prox-SVRG.
Compared to deterministic methods, the studies of mixed L //la-reegularizationn (1) in stochastic field become somewhat rare and limited. Prox-SG, RDA, Prox-SVRG, Prox-Spider and SAGA are valuable state-of-the-art stochastic algorithms for solving problem (1) but with apparent weakness. Particularly, these existing stochastic algorithms typically meet difficulties to achieve both decent convergence and effective group sparsity identification simultancously (c.g., small function values but merely dense solutions), because of the randomness and the limited In depth, Prox-SG, RDA, Prox-SVRG, Prox-Spider and SAGA derive from proximal gradient method to utilize the proximal operator to produce group of zero variables. Such operator is generic to extensive non-smooth problems, consequently perhaps not sufficiently insightful if the target problems possess certain properties, e.g., the group sparsity structure as problem (1). In fact, in convex setting, the proximal operator suffers from variance of gradient estimate; and in non-convex setting, especially deep learning, the discreet step size (learning rate) further deteriorates its effectiveness on the group sparsity promotion, as will show in Section 2 that the projection region vanishes rapidly except RDA. RDA has superiority on finding manifold structure to others (Lee & Wright, 2012) but inferiority on the objective convergence. Besides, the variance reduction techniques are typically required to measure over a huge mini-batch data points in both theory and practice which is probably prohibitive for large-scale problems, and have been observed as sometimes noneffective for deep learning applications (Defazio & Bottou, 2019). On the other hand, to introduce sparsity, there exist heuristic weight pruning methods (Li et al., 2016; Luo et al., 2017), whereas they commonly do not equip with theoretical guarantee, so that easily diverge and hurt generalization accuracy.
mechanisms.
Our Contributions. Half-Space Stochastic Projected Gradient (HSPG) method overcomes the limitations of the existing stochastic algorithms on the group sparsity identification, while maintains comparable convergence characteristics. While the main-stream works on (group) sparsity have focused on using proximal operators of regularization, our method is unique and fresh in enforcing group sparsity more effectively by leveraging half-space structure and is well supported by the theoretical analysis and empirical evaluations. We now summarize our contributions as follows.
Algorithmic Design: We propose the HSPG to solve the disjoint group sparsity regularized problem (1). Initialized with a Prox-SG Step for seeking close-enough but perhaps dense solution estimate, the algorithmic framework relies on a novel Half-Space Step to exploit group sparse patterns. We delicately design the Half-Space Step with the following main features: (i) it utilizes previous iterate as the normal direction to construct a reduced space consisting of a set of half- spaces and the origin; (ii) a new group projection operator maps groups of variables onto zero if they fall out of the constructed reduced space to identify group sparsity considerably more effectively than the proximal operator; and (iil) with proper step size, the Half-Space Step enjoys the sufficient decrease property, and achieves progress to optimum in both theory and practice.
Theoretical Guarantee: We provide the convergence guarantees of HSPG. Moreover, we prove HSPG has looser requirements identify the sparsity pattern than Prox-SG, revealing its superiority
2

---

Under review as a conference paper at ICLR 2021
the group sparsity exploration. Particularly, for the sparsity pattern identification, the required distance to the optimal solution x* of HSPG better than the distance required by Prox-SG.
Numerical Experiments: Experimentally, HSPG outperforms the state-of-the of-thh--rtt-t-rt methods in the aspect of the group sparsity exploration, and achieves competitive objective value convergence and runtime both convex and non-convex problems. In the popular deep learning tasks, HSPG usually computes the solutions with multiple times higher group sparsity and similar generalization performance on unseen testing data than those generated by the competitors, which may further used to construct smaller and more efficient network architectures.
2 THE HSPG METHOD
We state the Half-Space Stochastic Projected Gradient (HSPG) method in Algorithm 1. In general, contains two stages: Initialization Stage and Group-Sparsity Stage. The first Initialization Stage employs Prox-SG Step (Algorithm 2) to search for close-enough but usually non-sparse solution estimate. Then the second and fundamental stage proceeds Half-Space Step (Algorithm 3) started with the non-sparse solution estimate to effectively exploit the group sparsity within sequence of reduced spaces, and converges to the group-sparse solutions with theoretical convergence property.
Algorithm 1 Outline of HSPG for solving (1).
1: Input: 2n E R", ag E c E [0, 1), and Np E Zt.
2: for k 0, 1..... do
3:
if l: < Np then
4:
Compute #1+1 Prox-SG a.k) by Algorithm 2.
5: else
6:
Compute #x+1 Half-Space(22, an e) by Algorithm 3.
7:
Update aş+1-
Algorithm 2 Prox-SG Step.
Input: Current iterate æt, and step size og--
2: Compute the stochastic gradient of on mini-batch B3
VBB (xx)
V)x()))
1&
BEB
3: Return 26+11 (06 (xk ag (xx))
Initialization Stage. The Initialization Stage performs the vanilla proximal stochastic gradient method (Prox-SG, Algorithm 2) to approach the solution of (1). At ktth iteration, mini-batch B, sampled to generate unbiased estimator of the full gradient of f (line 2, Algorithm 2) to compute a trial iterate . == 3g ar Vf8. (æx), where og is the step size, and fB, is the average of the instance functions fr cross Bg. The next iterate 2x+1 is then updated based on the proximal mapping
3c++1 Prox,,, (#x+1) arg min
|
t 11 Xl(()),
mER"
20y
where the regularization term ((x) is defined in (1). Notice that the above subproblem (3) has closed-form solution, where for each G, we have
[ = max {0, 1 and/ [| [
In HSPG, the Initialization Stage proceeds Prox-SG Step Np times as localization mechanism to seek an estimation which close enough to a solution of problem (1), where Np : min k k E =+ || x=1| < 12/2] associated with positive constant It related to the optima, see (23) in Appendix C. In practice, although the close-enough requirement is perhaps hard to be verified, we empirically suggest to keep running the Prox-SG Step until observing testing on the stationarity of objective values, norm of (sub) gradient or validation accuracy similarly to (Zhang al., 2020). However, the Initialization Stage alone is insufficient exploit the group
stage-switch signal by
3

---

Under review as a conference paper at ICLR 2021
[0
S
Pros SG
[
HSPG
PVo
Proe-Spidee
HSPG e (0.1)
SAGA
RDA
#141
A
[a]]
O
Fr++1
[2]]
(a) Half-Space Projection
(b) Projection Region
Figure 1: Illustration of Half-Space Step with projection in (9), where G = [[1,1)1
sparsity structure, i.e., the computed solution estimate is typically dense, due to the randomness and the moderate truncation mechanism of proximal operator constrained in its projection region, iee. the trial iterate [#x+1]g و is projected to zero only if it falls into an f2-ball centered at the origin with radius ar by (4). Our remedy is to incorporate it with the following Half-Space Step, which exhibits an effective sparsity promotion mechanism while still remains the convergent property.
Algorithm 3 Half-Space Step
1: Input: Current iterate #he step size Okk and c.
2: Compute the stochastic gradient of y I (ax) by mini-batch Bg
WV (ox) (Ix)lI+o(
(5)
BR
FEB
3: Compute [#x+1]I=0 f x ag V (2x (2=)) and ( + 0.
4: for each group in (xx)) do
5: if [#x+1]] [x
||| then
6:
[ 19 f 0.
7: Return +++11 <- $1+1
Group-Sparsity Stage. The Group-Sparsity Stage is designed to effectively determine the groups of zero variables and capitalize convergence characteristic, which is in sharp contrast other heuristic aggressive weight pruning methods but typically lacking thcoretical guarantee (Li et al., 2016; Luo al., 2017). The underlying intuition of its atomic Half-Space Step (Algorithm 3) to project [ to zero only if [ serves as descent step to V (*+), i.e., - [axlg [VV( < 0, hence updating [x*+1lg + [xx]g [æx]g = 0 still results in some progress to the optimality. Before introducing that, we first define the following index sets for any æ R":
(x) = {g : E G, [x], 0} and (x) = {g g E G. [=ly + #),
(6)
where (x) represents the indices of groups of zero variables at æ, and [=) indexes the groups of nonzero variables at æ. To proceed, we further define an artificial set that æ lies in:
S(x) R [z], = 0 if g E (xt), and [=]] [al, 2 [|]]||] if e (x) U(O),
which consists of half-spaces and the origin. Here the parameter >> 0 controls the grey region presented in Figure 1b, and the exact way to will be discussed in Section and Appendix. Hence, x inhabits S(æx), i.e., xr E S(xx), only if: (i) [x]] lies in the upper half-space for all E (xx) for some prescribed c E [0, 1) as shown in Figure la; and (ii) [z), equals to zero for all E (xx). The fundamental assumption for Half-Space Step to success is that: the Initialization Stage has produced (possibly non-sparse) solution estimate 2; nearby group sparse solution x of problem (I), i.e,, the optimal distance | x²I] is sufficiently small. As seen in Appendix, further indicates that the group sparse optimal solution x² inhabits S₁ = S(xrg). which implies that
4

---

Under review as a conference paper at ICLR 2021
S₃ has already covered the group- of x', i.e., (z") C (xx)) Our goal now becomes
minimizing V (x) over Sk to identify the remaining groups of zero variables, ie., (x'')))(x)))))))))))) (x* 1
which is formulated as the following smooth optimization problem:
1k++1
arg min (x) f(x) + Al(()).
(8)
mES,
By the definition of Sg. (ax) = 0 are constrained as fixed during Algorithm 3 proceeding, and only the entries in ( are allowed to move. Hence (ax) is smooth on S,, and (8) is a reduced space optimization problem. A standard way to solve problem (8) would be the stochastic gradient descent equipped with Euclidean projection (Nocedal & Wright, 2006). However, such a projected method rarely produces zero (group) variables as the dense F illustrated in Figure la. To address we introduce a novel projection operator to effectively conduct group projection as follows.
As stated in Algorithm 3, we first approximate the gradient of V on the free variables in (*xs by [W B. (#*)) (line 2, Algorithm 3), then employ SGD to compute trial point Tt-1 (line 3, Algorithm 3) which is passed into new projection operator Prois, (-) defined as
[z|g [Prois, (x)l 5 0
if [2]]] a 2 < <<1[2>]gg1" otherwise.
The above projector of form (9) is not the standard Euclidean projection operator in most cases?, but still satisfies the following two advantages: (i) the actual search direction di == (Projs, (#x+1) zx)/aa performs as a descent direction to VB, (2x) := fs. (ax)) + (2(x)) i... [dx]g [Ws, (ax)))l < 0 as 0 < 90 in Figure 1a, then the progress to the optimum is made via the sufficient decrease property drawn in Lemma I; and (ii) effectively project groups of variables to zero simultaneously if the inner product of corresponding entries is sufficienlly small. In contrast, the Euclidean projection operator is far away effective to promote group sparsity, as the Euclidean projected point &e F O versus x++1 = Prois, (i++1) = 0 shown in Figure la.
Lemma 1. Algorithm 3 yields the next iterate +x+1 1 as Projs. (2 - V.V B, (ax)),, then the search di- rection dx= := - au, is a descent direction for WB, (xx), i.e., d W. (xx) < O. Moreover. letting L be the Lipschitz constant for B B, on the feasible domain, and 9x == I (axx n (xx+1) be the sets of groups which projects or not onto zero we have
and 9k= == - (arx I I 1)
Vs, (I SVB, (axx)
(an
LEL 2
Σ (IW))l RE9R
(1 (1---
(1---1
L.
Σ || (10) DE
akk
We then intuitively illustrate the strength of HSPG on group sparsity exploration. In fact, the half- space projection (9) is more effective sparsity promotion mechanism compared to the existing methods. Particularly, it benefits from a much larger projection region to map a reference point tx+1 := 3g as Wfs, (z+) or its variants to zero. As the 2D case described in Figure 1b, the projection regions of Prox-SG, Prox-SVRG, Prox-Spider and SAGA are fa-balls with radius ag.. In stochastic learning, especially deep learning tasks, the step size ag usually selected around 10 to 10-4 or even smaller for convergence. Together with the common setting of À < 1, their projection regions would vanish rapidly, resulting in the difficulties to produce group sparsity. As a sharp contrast, even though og À is near zero, the projection region of HSPG {2 az. x < (oy + || || (seen in Appendix) is still an open half-space which contains those Eg balls as well as RDA's e is large enough. Moreover, the positive control parameter e adjusts the level of aggressiveness of group sparsity promotion (9), i.e., the larger the more aggressive, and meanwhile maintains the progress to the optimality by Lemma 1. In practice, proper fine tuning is sometimes required to achieve both group sparsity enhancement and sufficient decrease on objective value as will see in Section 4.
To end this section, we discuss the advantage of designing such two stage schema rather than an adaptive switch back and forth between the Prox-SG Step and Half-Space Step based on some evaluation switching criteria, as many multi-step deterministic op- timization algorithms (Chen et al., 2017). In fact, we numerically observed that switching back to the Prox-SG Step consistently deteriorate the progress of group sparsity exploration by Salf-Space Step while without obvious gain on convergence. Such regression on group sparsity by the Prox- SG Step
Intuition of Two-Stage Method:
Unless ((x) is || where each g E 9 is singleton, then Sş. becomes orthant face (Chen et al., 2020).
5

---

Under review as a conference paper at ICLR 2021
is less attractive in realistic applications, e.g., model compression, where people usually possess heavy models of high generalization accuracy ahead and want to filter out the redundancy effectively. Therefore, in term of the case of application, we end at organizing Prox-SG Step and Half-Space Step as such a two-stage schema, controlled by switching hypermeter Np. In theory, we require Np sufficiently large let the initial iterate of Half-Space Step be close enough to the local minimizer as shown in Section 3. In practice, HSPG is sensitive to the choice of Np at early iterations, i.e., switching to Half-Space Step too early may result accuracy loss. But such sensitivity vanishes rapidly if switching to Half-Space Step after some acceptable evaluation switching criteria.
3 CONVERGENCE ANALYSIS
In this section, we give the convergence guarantee of our HSPG. Towards that end, make the following widely used assumption in optimization literature (Xiao & Zhang, 2014; Yang et al., 2019) and active set identification analysis of regularization problem (Nutini et al., 2019; Chen et al., 2018).
Assumption 1. Each ft :E + R. for = 1, .N. is differentiable and bounded below. Their gradients (z) are Lipschitz continuous, and let L. be the shared Lipschitz constant.
Assumption 2. The least and the largest l2-norm of non-zero groups in x' are lower and up- per bounded by some constants, i.e., 0 < 26 == mingeI+*(e")) ||[""]|| and O < 28 == maXgeT=" [|æ="|a|t Moreover, we request a common strict complementarity on any B, i.e. 0 < 28g == (ing (2 [[[[s((=*))) for regularization optimization.
Notations:
Let x be a local minimizer of problem (1) with group sparsity property, 4 be the local minimum value corresponding to x²,, and {= be the iterates generated from Algo- rithm 1. Denote the gradient mapping of (xx) and its estimator on mini-batch B as Ex(x)
[x Prox,)n()) (z - (=))) and (z) ==
" -x Prox, (x - T)Vs(()))) respectively.
We say ā a stationary point of (x) if (x) = O. To be simple, let X² be neighbor of x as
X = fo || x='l II S R} with R a positive constant related 61, 82 and E (see (23) in Appendix C), and M be the supremum of (z))] on the compact set X².
Remark: Assumption I implies that (z) measured on mini- batch B is Lipschitz continuous on R" with the same Lipschitz constant L. while ( is not as shown in Appendix. However, the Lipschitz continuity of V (x) still holds on X' = {x ||(=|||| 2 61 for each g E 9) by excluding a fz-ball centered at the origin with radius 81 from R". For simplicity, let W share the same
Lipschitz constant L on X' with Wfa (x), since we can always select the bigger value as their shared Lipschitz constant. Now, we state the first main theorem of HSPG.
Theorem 1. Suppose f is convex on X, E [0,min min { 62 25, 2824R]) | x² 11 S R for K 2 No.
Ser ls = K + t, (t E z+). Then for any T E (0, 1). there exist step size ag 0 vNT) E
(0, min { 1 2=: II'
(21-0) 1 26--R-1(6..R))
M }). and mini-batch size |Bal = O(1) < N - 25 such
N
that { converges to some stationary point in expectation with probability at least T. i.e. Pllmm-oo E Ill.aa,.B (x))))l 0) 22)21 T.
Remark: Theorem only requires local convexity of f on a neighborhood X of x while itself can be
non-convex in general. This local convexity assumption appears in many non-convex analysis, such as: tensor decomposition (Ge et al., 2015) and shallow neural networks (Zhong et al., 2017). Theorem I implies that if the Kthh iterate locates close enough to x², the step size ag and mini-batch size [By] is set as above, (it further indicates x: inhabits the {S&) k of all subsequent iterates updated by Half-Space Step with high probability in Appendix), then the Half-Space Step in Algorithm guarantees the convergence to the stationary point. The (t)) mini-batch size is commonly used in the analysis of stochastic algorithms, e.g., Adam and Yogi (Zaheer et al., 2018). Later based on numerical results in Section 4, we observe that much weaker increasing even constant mini-batch size is sufficient. In fact, experiments show that practically, a reasonably large mini-batch size can work well if the variance is not large. Although the assumption || x= || < R/2 is hard to be verified in practice, setting Np large enough usually performs quite well.
We then reveal the sparsity identification guarantee of HSPG as stated in Theorem 2.

---

Under review as a conference paper at ICLR 2021
Theorem 2. If 2 Np and | -="l=
20 I-cta,
then HSPG yields (x²) S I (2
Remark: Theorem 2 shows that when 2y is in the ba-ball centered at xx with radius 2ay8a
I-c+ou HSPG
identifies the optimal sparsity pattern, i.e., [x*x)) C Id (zx++)) ). In contrast, to identify the sparsity pattern, Prox-SG requires the iterates to fall into the fz-ball centered at xx with radius dggz3 (Nutini
et al., 2019). Since ag < 1/L and E [0, 1), then 1-cto, I. 2 وggz implies that the ba-ball of HSPG contains the lz-ball of Prox-SG, i.e.. HSPG has stronger performance in sparsity pattern identification. Therefore, Theorem 2 reveals a better sparsity identification property of HSPG than Prox-SG, and no similar results exist for other methods to our knowledge.
The Initialization Stage Selection: To satisfy the pre-requirement of convergence of Half- Half-Space Step as Theorem 1, i.e., initial iterate close enough to x", there exists several proper candidates e.g., Prox- Prox-SG, Prox-SVRG and SAGA to form as the Initialization Stage. Considering the tradeoff between computational efficiency and theoretical convergence, our default setting is to select Prox-SG. Although
may have better theoretical convergence property than Prox-SG, they require higher time and space complexity to compute or estimate full gradient on huge mini-batch or store previous gradient, which may be prohibitive for large-scale training especially when the memory is often limited. Besides, it is well noticed that SVRG does not work as desired on the popular non-convex deep learning applications (Defazio & Bottou, 2019; Chen et al., 2020). In contrast, Prox-SG is efficient and can also achieves the good initialization assumption in Theorem 1, 2, in the manner of high probability via performing sufficiently many times,
ie.. N1 - x= || S R/2, as revealed Appendix C.4 by leveraging related literature (Rosasco al., 2019) associated with an additional strongly convex ussumption. However, one should notice that Prox-SG does not guarantee any group sparsity property of Ny due to the limited projection region and randomness.
We emphasize that this paper focuses on improving the group sparsity identification, which rarely explored and also a key indicator of success for structured sparsity regularization problem. Meanwhile, we would like to point out improving the convergence rate has been very well explored in series of literatures (Reddi et al., 2016; Li & Li, 2018), but out of our main consideration.
Remark:
4 NUMERICAL EXPERIMENTS
In this section, we present results of several benchmark numerical experiments in deep neural networks to illustrate the superiority of HSPG than other related algorithms on group sparsity exploration and the comparable convergence. Besides, two extensible convex experiments are conducted in Appendix empirically demonstrate the validness and superiority of the group sparsity identification by HSPG.
We now consider the popular Deep Convolutional Neural Networks (DC- NNs) for image classification tasks. Specifically, we select several popular and benchmark DCNN architectures, i.e., VGG16 (Simonyan & Zisserman, 2014), ResNet18 (He et al., 2016) and MobileNetV1 (Howard et al., 2017) two benchmark datasets CIFAR1O (Krizhevsky & Hinton, 2009) and Fashion-MNIST (Xiao et al., 2017). We conduct all experiments for 300 epochs with a mini-batch size of 128 and À as 10-3, since it returns competitive testing accuracy to the models trained without regularization, (see more in Appendix D.3). The step size ag is initialized as 1, and decayed by factor 0.1 periodically. We set each kernel in the convolution layers as a group variable.
Image Classification:
In these experiments, we proceed a test on the objective value stationarity similarly to (Zhang et al., 2020, Section 2.1) and switch Half-Space Step roughly on 150 epochs with No as 150N/ The control parameter e in the half-space projection (9) controls the aggressiveness level of group sparsity promotion, which is first set as 0, then fined tuned to be around 0.02 to favor the sparsity level whereas does not hurt the target objective V the detailed procedure is in Appendix D.3. We exclude RDA because of no acceptable results attained during our tests with the step size parameter y setting throughout all powers of 10 from 10 to 103, and skip Prox-Spider and SAGA since Prox-SVRG has been a superb representative to the proximal incremental gradient methods.
Table I demonstrates the effectiveness and superiority of HSPG, where we mark the best values as bold, and the group sparsity ratio is defined as the percentage of zero groups. In particular, (i) HSPG computes remarkably higher group sparsity than other methods on all tests under both e 0 and fine tuned e, of which the solutions are typically multiple times sparser in the manner of group than those of Prox-SG, while ProX-SVRG performs not comparably since the variance reduction techniques may
7

---

Under review as a conference paper at ICLR 2021
Table
Final V/group sparsity ratio/testing accuracy for tested algorithms on non-convex problems.
Backhone
Dataset
Prox-SG
Prox-VVRG
HSPG
line tuned
VGG16
Fashion-MNIST
CIFARIO
0.39753.99% 7 90.57%
0.54 15.63% / 92.99%
0.827 14.7% 789422%
2.66/0.45%
/92.69%
0.597 74.00% 7 191.10%
0.54/22.18%
/92.98%
0.59 7 775.6% & 790922%
0.53 60.77% / /9.877%
CIFARIO
0..5/9 .9%
0.36/2.79%
194.17%
0.31741.88%
794,39%
0.31/62.97% 794.5%%
RosNet18
Fachion MNIST
0.14/0.00% / 94482%%
0.19
194.64%
0.13/6.60%
99.93%%
0.13 63.93% 94.866
ChART
0.66/31.818 80%
0.667
32.22
7 7
79136%
0.37/7666% 391.345
MohileNetV1
Fashion-MNIST
0.22/ 65.80% /94.36%
0.48 38.76%
/93.95%
07,3/74.528
0.23 74.52%
/94.43%
0.24/ 83.71% 94.44%
Group Rusio (SQ)
Teslig Aocanto Cliy
100
b. 5
Tesling Areunay
60-g
325
1.9 Epoch
Epock
Epuch
(a) Objective V/
(b) Group Sparsity Ratio
(c) Testing Accuracy
(d) HSPG VS Truncation
Figure 2: On ResNer18 with CIFARIO, (a)-(c): Evolution of V, group sparsity ratio and testing accuracy,
(d): HSPG versus Prox-SG* and Prox-SVRG* (Prox-SG and Prox-SVRG with simple truncation mechanism).
not work as desired for deep learning applications (Defazio & Bottou, 2019); (ii) HSPG performs competitively with respect the final objective values V/ and I (see f in Appendix). In addition, all the methods reach a comparable generalization performance on unseen test data. On the other hand, sparse regularization methods may yield solutions with entries that are not exactly zero but are very small. Sometimes all entries below certain threshold (T) are set to zero (Jenatton et al., 2010; Halabi et al., 2018). However, such simple truncation mechanism is heuristic-rule based, hence may hurt convergence and accuracy. To illustrate this, we set the groups of the solutions of Prox-SG and Prox-SVRG to zero if the magnitudes the group variables are less than some T, and denote the corresponding solutions as Prox-SG* and Prox-SVRG As shown in Figure 2d()), under the T with no accuracy regression, Prox-SG* and Prox-SVRG* reach higher group sparsity ratio as 60% and 32% compared to Table 1, but still significantly lower than the 70% of HSPG under 0.05 without simple truncation. Under the T to reach the same group sparsity ratio as HSPG, the testing accuracy of Prox-SG* * and Prox-SVRG* regresses drastically to 28% and 17% in Figure 2d(ii) respectively. Remark here that although further refitting the models from Prox-SG* and Prox-SVRCE on active (non-zero) groups of weights may recover the accuracy regression, it requires additional engincering efforts and training cost, which is less attractive and convenient than HSPG (with no need to refit).
Finally, we investigate the group sparsity evolution under different e's. As shown in Figure 2b, HSPG produces the highest group-sparse solutions compared with other methods. Notably, at the early No iterations, HSPG performs merely the same as Prox-SG, However, after switching to Half-Space Step at the 150th epoch, HSPG outperforms all the other methods dramatically, and larger results in higher sparsity level. It strong evidence that our half-space based technique is much more successful than the proximal mechanism and its variants in terms of the group sparsity identification. Besides, the evolutions of V/ and testing accuracy confirm the comparability on convergence among the tested algorithms. Particularly, the objective V generally monotonically decreases for small e 0 to 0.02, and experiences a mild pulse after switch to Half-Space Step for larger c, e.g., 0.05, which matches Lemma 1. As result, with the similar generalization accuracy, HSPG allows dropping entire hidden units of networks, which may further achieve automatic dimension reduction and construct smaller model architectures for efficient inference.
5 CONCLUSIONS AND FUTURE WORK
We proposed a new Half Space Stochastic Projected Gradient (HSPG) method for disjoint group- sparsity induced regularized problem, which can be applied to various structured sparsity stochastic learning problem. HSPG makes use of proximal stochastic gradient method to seek near-optimal solution estimate, followed by a novel half-space group projection to effectively exploit the group sparsity structure. In theory, provided the convergence guarantee, and showed its better sparsity identification performance. Experiments on both convex and non-convex problems demonstrated that HSPG usually achieves solutions with competitive objective values and significantly higher group sparsity compared with state-of-the--e stochastic solvers. Further study is needed to investigate the proper leverage of group sparsity into diverse deep learning applications, e.g. help people design and understand optimal network architecture by removing redundant hidden structures.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Francis Bach, Rodolphe Jenatton, Julien Mairal, Guillaume Obozinski, et al. Structured sparsity through convex optimization. Statistical Science, 27(4))450-468, 2012.
Chih- Bhih-Chung Chang and Chih-Jen Lin. Libsvm: Data repository. 2011. URL httph:/ttp//t csie. ntu. edu. tw/ - jecii/inn///..n http//ibssae wibssiinn vmt /awaaaaaaass////////////..
Po- Yu Chen and Ivan W Selesnick. Group-sparse signal denoising: non-convex regularization, convex
optimization. IEEE Transactions on Signal Processing, 62(13):3464 3478, 2014.
Tianyi Chen. Fast Reduced- Spacce Algorithmic Framework for Sparse Optimization. PhD thesis, Johns Hopkins University, 2018.
Tianyi Chen, Frank E Curtis, and Daniel P Robinson. A reduced-space algorithm for minimizing l-rreulaaried convex functions. SIAM Journal on Optimization, 2763):1583-1610 2017.
Tianyi Chen, Frank E Curtis, and Daniel Robinson. Farsa for f1-regularized convex optimization: local convergence and numerical experience. Optimization Methods and Software, 2018.
Tianyi Chen, Tianyu Ding, Bo Ji, Guanyi Wang, Yixin Shi, Sheng Yi, Xiao Tu, and Zhihui Zhu. Orthant based proximal stochastic gradient method for &-regularized optimization. arXiv preprint NXXv:2004.03639, 2020.
Aaron Defazio and Léon Bottou. On the ineffectiveness of variance reduced optimization for deep learning. In Advances in Neural Information Processing Systems, 2019.
Aaron Defazio, Francis Bach, and Simon Lacoste-Julien. Saga: A fast incremental gradient method with support for non-strongly convex composite objectives. In Advances in neural information processing systems, pp. 1646-1654, 2014.
Dmitriy Drusvyatskiy and Adrian S Lewis. Error bounds, quadratic growth, and linear convergence of proximal methods. Mathematics of Operations Research, 43(99:119 1991999994222 2018.
John Duchi and Yoram Singer. Efficient online and batch learning using forward backward splitting. Journal of Machine Learning Research, 10(Dec):299-29334 2009.
Ehsan Elhamifar, Guillermo Sapiro, and Rene Vidal. See all by looking at few: Sparse modeling for finding representative objects. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1600-1607.... IEEE, 2012.
Cong Fang. Chris Junchi Li, Zhouchen Lin, and Tong Zhang. Spider: Near-optimal non-convex opti- mization via stochastic path-integrated differential estimator. In Advances in Neural Information Processing Systems, pp. 689-6999 19996999220 2018
Rong Ge, Furong Huang, Chi Jin, and Yang Yuan. Escaping from saddle points-onling stochastic gradient for tensor decomposition. In Conference on Learning Theory, pp. 797-842, 2015.
Robert M. Gower. Convergence theorems for gradient descent. University of Illinois, Urbana Champaign, 2018
Marwa El Halabi, Francis Bach, and Volkan Cevher. Combinatorial penalties: Which structures are preserved by convex relaxations? In International Conference on Artificial Intelligence and Staristics, pp. 1551-1560. PMLR, 2018
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, 2016.
Andrew Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. arXiv preprint arXiv: / 704.04461, 2017.
Junzhou Huang, Xiaolei Huang, and Dimitris Metaxas. Learning with dynamic group sparsity. In 2009 IEEE 12th International Conference on Computer Vision, pp. 64-71. IEFF, 2009.

---

Under review as a conference paper at ICLR 2021
Junzhou Huang, Tong Zhang, al. The benefit of group sparsity. The Annals of Staristics, 38(4): 1978-2004, 2010.
Junzhou Huang, Tong Zhang, and Dimitris Metaxas. Learning with structured sparsity. Journal of Machine Learning Research, 12(Nov):3311-3412, 2011.
Rodolphe Jenatton, Guillaume Obozinski, and Francis Bach. Structured sparse principal component analysis. In Proceedings of the Thirteenth International Conference on Artificial Intelligence and Statistics, pp. 366-375 2010.
Rie Johnson and Tong Zhang. Accelerating stochastic gradient descent using predictive variance reduction. In Advances in neural information processing systems, pp. 315-323, 2013.
Hamed Karimi, Julie Nutini, and Mark Schmidt. Linear convergence of gradient and proximal- gradient methods under the polyak-lojasiewicz condition. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer. 2016.
A. Krizhevsky and G. Hinton, Learning multiple layers of features from tiny images. Master's 's thesis, Department of Computer Science, University of Toronto, 2009.
Sangkyun Lee and Stephen J Wright. Manifold identification in dual averaging for regularized stochastic online learning. The Journal of Machine Learning Research, ((1):1705-1744 2012.
Hao Li, Asim Kadav, Igor Durdanovic, Hanan Samet, and Hans Peter Graf. Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710, 2016.
Zhize Li and Jian Li. A simple proximal stochastic gradient method for nonsmooth nonconvex optimization. In Advances in neural information processing systems, pp. 3564-5574 2018.
Shujun Liu, Jianxin Cao, Hongqing Liu, Xichuan Zhou, Kui Zhang, and Zhengzhou Li. Mri reconstruction via enhanced group sparsity and nonconvex regularization. Neurocomputing. 272: 108-121, 121, 2018.
Jian- Hao Luo, Jianxin Wu, and Weiyao Lin. Thinet: A filter level pruning method for deep neural network compression. In Proceedings of the IEEE international conference computer vision, pp. 5058-5066, 2017.
Michael Mitzenmacher. Probability and compuring-ranndomized algorithms and probabilistic analysis. JOURNAL-OPERRXIONAL RESEARCH SOCIETY, 56(12):1454, 2005.
Eugene Ndiaye, Olivier Fercoq, Alexandre Gramfort, and Joseph Salmon. Gap safe screening rules for sparsity enforcing penalties. The Journal of Machine Learning Research, 18(1):4671-4703, 2017.
Yurii Nesterov. Primal-dual subgradient methods for convex problems. Mathematical programmingg 2009.
Jorge Nocedal and Stephen Wright. Numerical optimization. Springer Science & Business Media, 2006.
Julie Nutini, Mark Schmidt, and Warren Hare. "active-set complexity" of proximal gradient: How long does it take to find the sparsity pattern? Optimization Letters, 13(4):645 655, 2019.
Sashank Reddi, Suvrit Sra, Barnabas Poczos, and Alexander Smola. Proximal stochastic methods for nonsmooth nonconvex finite-sum optimization. Advances in Neural Information Processing Systems, pp- 1145-1153, 2016.
Lorenzo Rosasco, Silvia Villa, and Bng Công Vü. Convergence of stochastic proximal gradient algorithm. Applied Mathematics & Optimization, pp. 1-27, 2019.
Volker Roth and Bernd Fischer. The group-lasso for generalized linear models: uniqueness of solutions and efficient algorithms. In Proceedings of the 25th international conference on Machine learning, pp. 848 855, 2008
10

---

Under review as a conference paper at ICLR 2021
Nicolas L Roux, Mark Schmidt, and Francis R Bach. A stochastic gradient method with exponential convergence rrate for finite training sets. In Advances in neural information processing systems, pp. 012226662277 2012.
Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regular- ization for deep neural networks. Neurocomputing, 241:81-88 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1555 2014.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
Lin Xiao. Dual averaging methods for regularized stochastic learning and online optimization. Journal of Machine Learning Research, 11(0ct):25433-555) 2596, 2010.
Lin Xiao and Tong Zhang. proximal stochastic gradient method with progressive variance reduction.
SIAM Journal on Optimization, (4(4):2057-2075 2014.
Haiqin Yang, Zenglin Xu, Irwin King, and Michael R Lyu. Online learning for group lasso. In Proceedings of the 27th International Conference on Machine Learning (ICML-10), pp. 1191-1198, 2010.
Minghan Yang, Andre Milzarek, Zaiwen Wen, and Tong Zhang. A stochastic extra-step quasi-newton method for nonsmooth nonconvex optimization. arXiv preprint arXiv: 191:::90000000 0.09973, 2019.
Ming Yuan and Yi Lin. Model selection and estimation in regression with grouped variables. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 68(1):49-67, 2006.
Manzil Zaheer, Sashank Reddi, Devendra Sachan, Satyen Kale, and Sanjiv Kumar. Adaptive methods for nonconvex optimization. In Advances in neural information processing systems, pp. 9793-9803, 2018.
Junyu Zhang and Lin Xiao. Multi-level composite stochastic optimization via nested variance reduction. arXiv preprint arXiv: 1468, 2019.
Pengchuan Zhang, Hunter Lang, Qiang Liu, and Lin Xiao. Statistical adaptive stochastic gradient methods. arXiv preprint arXiv:2002. 10597, 2020.
Kai Zhong, Zhao Song, Prateek Jain, Peter L Bartlet, and Inderjit S Dhillon. Recovery guarantees for one-hidden-layer neural networks. In International Conference on Machine Learning, 2017.
A PROJECTION REGION
In this Appendix, derive the projection region of HSPG, and reveal that is superset of those of Prox-SG, Pvox-SVRG and Spider under the same ag and X.
Proposition The Half-Space Step of HSPG yields next iterate 06+1 based on the trial iterate Et+1 Or VfB. (Ix) as follows for each g E I1/(xx)
[ [ 1gg -aa
( 11 [xn]e > (arx +
||[x]]ll| (
(11)
otherwise.
Consequently, if 11(@x++1gg11 S agλ, then [x4+1] for any e
Proof. For g E (xx) 0 (xx+1), by Algorithm 3, it is equivalent to
[
Fk a (x6. - a+d
[xx]y > e [|[
||
(12)
[ _ [ack]
a4 |
[1[
[ [xxl > (axd + (||
[#x]|l|
11

---

Under review as a conference paper at ICLR 2021
Similarly, 9 (x)( (zt) (xx-1) is equivalent to
2k ax Vf6. (xx) - and [
[2n]g - <1|[21101*
|
(13)
[ T [2llg
a+ |1 - <11[<>>g11*
[ 1g [xllo S (ax) + 1 [[ [2w]ell
If ||[#++1]g|| I agA, then
[ [xl]a I [| 1l ||[ S ond ||
(14)
Hence [0x+1lg o holds for any 2 0 by (13), which implies that the projection region of Prox-SG and its variance reduction variants, e.g., Prox-SVRG, Prox-Spider and SAGA are the subsets of HSPG's.
B NON-LIPSCHITZ CONTINUITY OF W ON R"
The first-derivative of ((x) at æ of can be written as
[=]
w Vf(x) +
(15)
|<>||||
9EG
We next show [al
is not Lipschitz continuous on I if [g| 2 2. Take example for æ E IR? and || ||
select Φ = (t, a,t) x2
22 = (t, (z)) ai * d2 and R. Then suppose there exists a positive
constant L co such that Lipschitz continuity holds follows
22
< L |
|
(16)
(1, +
(1,
< L|a a2l |t]
V1+
holds for any & E IR, and note the left hand side is a positive constant. However, letting -> 0, we [e]a
is not Lipschitz continuous on have that L, - oo which contradicts the L < co. Therefore, Te].T|
RR? specifically the region surrounding the origin point.
Although (z))>>>>< is not Lipsethiz continuous R", the Lipschitz continuity still holds on by excluding fixed size fz-ball centered at the origin for the group of non-zero variables I+ from R". For our paper, we define the region where Lipsethiz continuity of [VV still holds as
X = (x || 2 51 for each E I= (x), and [z) for each g E I (z)).
(17)
C CONVERGENCE ANALYSIS PROOF
Denote the sets of groups which projected not onto zero
G. = I (mx) n I (mx+1), and
(18)
G. = I (xx)
(x++1). (
(19)
Denote X' : (x ||[ 2 6, for each g E G} where the Lipschitz continuity of (x) still holds by excluding ba-ball centered at the origin with radius ô, from R". Let M denote one upper bound of |0 and ||E||.
Additionally, establishing some convergence results require the below constants measure the least and largest magnitude of non-zer zero group variables in x".
0 81
min (E
[|[<") " and
(20)
2
1
0 < 8g
||[ (E
max
(21)
2
12

---

Under review as a conference paper at ICLR 2021
and subsequent results of strict complementary assumption on any B uniformly,
min (A 0 < 63 := 2 (xT (x*)
[IVff(>")))) 1l,1l)
(22)
And denote the following frequently used constant R describing the size of neighbor around x".
- + 26δ2) + (0i + 25dg)² - 4c-02 + 400 81 >0 0.
R = min
(23)
L 82
Remark: (23) is well defined
and degenerated as c 0.
C.I SUFFICIENT DECREASE OF PROX- SG STEP AND HALF-SPACE STEP
Our convergence analysis relies on the following sufficient decrease properties of Half-Space Step and Prox-SG Step.
Sufficient Decrease of Half-Space Step: We prove the Lemma the below.
Proof of Lemma 1:
It follows Algorithm and the definition of Gr and &R as (19) and (18) that #x+1 25k + azdz where dę
B, (ar))la [owkgg/ ook
i E 9r = I (xx) ) (ax+1),
[d
if Gr = (xx) (I
(24)
otherwise.
also notice that for any g E Gx. the following holds
[x a B. (xx)) [xwlg < ||
(25)
(1 <) || < on s, (axx }b [mx]g
For simplicity, let It = I (x*). Since [ds], O for any g E I (xx), then by (24) and (25), we
have
df LVV, (*N) = [d [ 10V (11))1240
Σ
|| BR (2t))
Σ
[owl] 10 (xx)la
(26)
9E9x
nEci
S
Σ |1 (ax) gEd.
(1 <) ||
gEd.
holds for any e [0, 1). which implies that d₃ is descent direction for MB (xx))
Now, we start prove the suffeient decrease of Half-Space Step. By the descent lemma, 3y. X and the Lipschitz continuity of [8 B. 1zę00 on X, we have that
L a | V. ( + ag dx) - v (xx) + ar ( B. (Ix (Ix) IFpo ddllzo +
(27)
2
Then it follows (24) that (27) can be rewritten follows V (xx + axdx))
< <V (xx) + ax (Ix ( TT (delzço +
2 a
[d
11,20
L
-V (xx) - Σ
110V8 (ax))al1? ass
Σ
Ja (2x) 13 loxls
||
ax
2
2
gE
gEóe
(28)
13

---

Under review as a conference paper at ICLR 2021
Consequently, combining with E 10, and (25), (28) can be further shown as
af I.
(1 (1-e
1.
B, (xx+1) - (xx))
Σ
I1 (xx))=l1²
Σ
||
|[
2
2
nEce
(29)
which completes the proof.
Sufficient Decrease of Prox-SG Step:
The second lemma is well known for proximal operator under our notations, include this proof for completeness. Lemma 2. Line 3 of Algorithm 2 yields that #x+1
C, (EL), where
Bar, (xx) E VfB. (xck) + (00((x)+1).
(30)
And the objective value Bs satisfies
agL. 2
v (x++1) & (xx)) -
15, (xx))1
(31)
Proof. It follows from the line (3) in Algorithm and the definitions of proximal operator that
#h++1 = arg min
|| (ar - On V/B, (x=)))] + XQ(x))
#ER
20c
(32)
1
arg min
VIB, (ark) ( (xc .xk) 1l((z) + 20c
| xx
=ER"
By the optimal condition, we have
1 ag
0 (Ix+1 3&)) + VfB, (xx) x05((x*+1)
(33)
Since #1+11 2t 04k So B (2 we have
0 E Ba (xx+) + (TB((xx)) + A051(xx+1).
(34)
which implies that
Eor Ba (ax) E Vfs. (axx) + (0 And thus there exists some v E 02(x++1) such that
(35)
La. (mx) = Vf8, (æx)) + Xv.
(36)
By Lipschitz continuity of V/8, and convexity of (2.)), we have fB, (0x+1) S8, (2t E. (2++)
(37)
S fB, (xx) - ag VfB. (Ix) Ear (2k) +
11 (x+))1
2
and
AΩ(x)+1) = AΩ(Ix axE,,, (x
S <A((m))) X (Ir ar 18, (xx) - xxk)
(38)
= >l(())) agdv Ea,, (xx).
Hence, by (36), (37) and (38), the objective V. (xx++1 satisfies (2x41) fu. (xx+1) ) + A[(x++1)
S/8, (ark) ar VSs. (mx) 1,,, (#x) + a 1, (xxx)l1 + Xl2(ax)) T So,, (#r)
-VB, (ark) ou. (V/B. (ax)) + Xx) Eon,B, (2x) + aRL
15
2
azL L
-V (2x))
1E (xxx)11
which completes the proof.
14

---

Under review as a conference paper at ICLR 2021
According to Lemma 1 and Lemma 2, the objective value mini-batch tends to achieve sufficient decrease in both Prox-SG SG Step and Half-Space Step given ag is small enough. By taking the expectation both sides, obtain the following result characterizing the sufficient decrease from Vi (mx) to E [ (2++
Corollary 1. For iteration k we have
(i) if kth iteration conducts Prox-SG Step, then
E (v+x+++) & ((xx)) (aR
ajL
E
[mmm (22x)
(39)
2
(ū) kit iteration conducts Half-Space Step, 2ik E X, then
aLL 2
E (++++1) & ((x)))
Σ BEAR
E |lo (xW))
Σ gECx
[
Ork
2
(40)
Corollary 1 shows that the bound of V depends on step size ag and norm of search direction. It further indicates that both Half-Space Step and Prox-SG Step can make some progress to optimality with proper selection of ogk.
C.2 PROOF OF THEOREM 1
Toward that end, we first show that the optimal distance from 3y the local minimizer xt is sufficiently small, then HSPG already covers the supports of x", i... (x=" C IT*(
Lemma 3. If 2 x S R, then (x*) C (xx)
Proof. For any g E T T=0'")) by the assumption of this lemma and the definition of R as (23) and 51 as (20), we have that
[|(==)a] Iall - || |l S [a xx") =="gll s [ x01 = R
(41)
| 2 ||[="]=|| 51 20, 61 01
Hence || # 0, Le., g I= (xxx) Therefore, I (x*) I= (2+)).
The next lemma shows that if the distance between current iterate æy and x², i.e., |x x² is
sufficiently small, then x" inhabits the reduced space S₃ := S(xx).
Lemma 4. Under Assumption 1, if0 S
82
| x01l S R, then for each g E I#0
[xx], [x'1, 2 e ||[
(42)
Consequently, it implies x² E Sy by the definition as (7).
Proof. It follows the assumption of this lemma and the definition of R in (23), 8n and 8, in (23), (20)
and (21) that for any E I (*')))) (x*),
[|[ nellgll= S [1[ + R < 26, + R,
(43)
and the [-(6, 2c8g) V(6, 2c62)² 2c + 4c57 /c in (23) is actually the solution of cz
(4ed2 + 26 + 4edž 40² = 0 regarding E R Then we have that
[ [x="], = |a0k x=" + x xx]] [x=)] ","
= | x²1 [x"], 1g + ||[ "1 l
2 ||[=="]]|| |1[ - x111 [[ 111l
(44)
||[a="]a [al") [1I[ "lll -I[ xx "²"11 l))
26, (281 R) 2 c(25 + R)
2e |
15

---

Under review as a conference paper at ICLR 2021
holds for any 9 (x",, where the second last inequality holds because that 261 (281
R)
c(22 + R)² as R = - + 2cδ2) + (ō1 + 2c82)² 4 + 4c0f /c. Now combing with the definition of Sy as (7), we have x" inhabits Sg, which completes the proof.
Furthermore, if ||ag a=1 is small enough and the step size is selected properly, every recovery of group sparsity by Half-Space Step can be guaranteed as successful as stated in the following lemma.
Lemma 5. Suppose & 2 Np. || x1'l S R. then for any g 9k- = I (mx) (Ix++)) 9 must be in T Le., 9 E (xx")
26 and 0 04k 25 R
26 R- (2+
Proof. To prove it by contradiction, suppose there exists some g E 9. such that g E (x*) Since g E 9x = =I=00xx I (x<++1). 1), then the group projection (9) trigerred at g such that
[ [axlg = [>> a V (I ax
(45)
= =|[ar]]0|1 ou W. (2x))l a < E [
On the other hand, it follows the assumption of this lemma and y E I (x"(x") that
|| [ x=], 1|||== S ||
| x'1l S R
(46)
Combining the definition of 6, as (20) and 82 as (21), we have that
| |all 2 ||| [xx= 1g
R 2 26: R
(47)
|
< ||| 1oll + R < 262 + R
then follows 0 < axk 5
26: R (126++
where note 261 R - c(20g + R) > 0 as
< 8 and
M8
26, RR that 262+-R R
[ g
T [xkg = ||[=*|||| a W B. (Ix))l [
[[ax]]ll' 11 or ||| [V,, (a (ar) [
[ [1[ - on [|[VV (2))
IV I W w A
[all
(I) [[ - Mx
||| [[
[(28] R) a&M
(48)
(26) R) 28, R - -R-(2202 + R)
||
M
M
|| 1
[(25] R) -2, + R c(22 + R)]
|||==||||| (28g + R)
<||[0x|o11
which contradicts with (45). Hence, we conclude that any g of variables projected to zero, i.e.
g E 9r- = (xxx 0 I (x*+1) are exactly also the zeros on the optimal solution x", i.e., 9 E
T (x)"
L
We next present that if the iterate of Half-Space Step is close enough to the optimal solution x".
then x² inhabits all reduced spaces constructed by the subsequent iterates of Half-Space Step with high probability. To establish this results, we require the below two lemmas. The first bounds the accumulated error because of random sampling. Here we introduce the error gradient estimator on I+ (x) for V mini-batch B as
eg(z) = [ V (z))
(49)
where by the definition of Ω in problem (1), we have es(Φ) also equals the error of estimation for V..
eσ(x) (B)() (=))))za)))) = [Vf((x) vf(x))
(50)
16

---

Under review as a conference paper at ICLR 2021
Lemma 6. Given any 0 1, K 2 Np, := K tt E Uf0), then there exists an. O1/tt) and |Bx| = O(t), such that for any y E R"
00
3R²
max (wr F Ed
ex (2:)l 8(4R + 1)
t=0
holds with probability at least
1 6/.
Proof. Define random variable Y, == uo++olless, 22) for all / 2 0. Since (y:)i-, are arbitrarily chosen, then the random variables {Yi] are independent. Let = Ei= Y. Using Chebshev's inequality, we obtain
Y M 0V 0 VarlV - P (V - [[[] 2 0 1 Var[Y] <
(51)
82
And based on the Assumption 1, there exists upper bound x² > 0 for the variance of random noise e(x) generated from the one-point mini-batch, i.e., == {i}, i = ,,...,.. ... Consequently, for each
t t2 0, we have E|Yi] <
ag410 and Var[Yi] - 4 [x+ then combining with (51), we have
VBx+eel
Y S +
+ + Var[Y]
(52)
06
2
02
ak+10
K
(53)
Betel
K
1=0
aK++00
+0.
O.K40
(1 + 0)
Σ t= BBK+
(54)
Butel
VBKK+
1=0
holds with probability at least 5z. Here, for the second inequality, we use the property that the equality TE E YjJ = E [E[Y.] holds whenever Σi= [[]]]]]] convergences, see Section 2.1 in Mitzenmacher (2005); and for the third inequality, we use / VBBx+e|
aa 10 I 1 without loss of generality as
the common setting of large mini-batch size and small step size.
Given any > 1, there exists some ag O(1(t) and |B&] Ot),, the above series converges and satisfies that
3R²
aK+00
01+0)
(55)
BK++1
8(4R+1
holds. Notice that the above proof holds for any given sequence ] E 2 thus
3R²
Σ
Cxlles (3))
max {er
8(4R +
t=0
holds with probability at least
82-
The second lemma draws if previous iterate of Half-Space Step falls into the neighbor of x", then under appropriate step size and mini-batch setting, the current iterate also inhabits the neighbor with high probability.
Lemma 7. Under the assumptions of Lemma 6, suppose
|æ - xx1| S R/2; for any satisfying
K < l < K + +t I mmnn 26, R then
}. Br 2 N 2M
N and ||
x11l & R holds,
M
| x011 00115 R.
(56)
holds with probability at least 1-1 gz.
Proof. It follows the assumptions of this lemma, Lemma 5, (18) and (19) that for any & satisfying
K < f K +t
[ gg = 0, for any g Gr.
(57)
17

---

Under review as a conference paper at ICLR 2021
Hence we have that for K < < +t,
| - =x=11
-
1100 xx= - -o. - ages, (ze)]a]) + Σ ||| x== - -zelg1
3E9e
Σ
|1[20e - x "gll -2a22ee - 2 (++)
(V( + eB, (xe))l + al [I( + (x)))o + Σ
l| [| 1اl6
gEde
DEse
Σ
[Ilara x:" a="g"1²= 11f-22xleer (aze- =="] (WV()))) 2cz ax n²), les, (mr)) + a [( + es, es, (wee)lol)
SEgr
|| x") ,l1 [1 (2 až) - -2ax [ x xelgles,
les, (xe)l, + a |1[es, (x)))a1
NE
+ 20 Wi (Z (es, (a:;)la
|| | - 1 (x)e( (2 - -a²) 200e 11 - I'1sll [[le, (xe)lll + aZ llies, (2t))ss)
9E9r
+ 20 (Z) |es,, (xr))1g)
||| x ,01 ||\ (2)))) 11? (2 - -at) + +2ar + +aa2L) [ - x"1, llle, (2;)1,g] + o [1le,,(x)) (2t)la
3E9
Σ
{IlEer x= ""|=11 - 11 (1)))l) (2 ae ot) + (2ar + 2) || lell (=xi) as lles, (Ix)
.EA
(58)
On the other hand, by the definition of eg(x) as (49), we have that
(g(x) -IVV (a)
VVV(
(f(x))
N
(z (Z)))
B
N
jEB
N
[+)
[V (2) (20
(59)
JEB
N
B
N
B
N
JEB
Thus taking the norm on both side of (59) and using triangle inequality results in the following:
N
B
1 N
[ea(x))l
5
(
+
[[
N
B
N
jES
(60)
N
B
1
2(N
188 M
B&
M +
(N - -BIM
<
N
B
N
N
18

---

Under review as a conference paper at ICLR 2021
Since ag I 1, and [By] 2 N can be further simplified as
N hence oz lleg, (2t) & 1. Then combining with de $ 1/L, (58)
[8++1 - x01
||| || - -x=0=-1 -1[ (2% - aa}} + (2ar + 2a(L) || - x'1l lls, (x+)) ll + a l|e,, (x+)
Σ
[| - x "0]1l1" 1 W (2)))1 112 + 4ar |lare - x01l | (2x) + a² ees (2+1)
g69
< [.g
112 + 40g ||0g-
=01l
|| (xx) + ore ||eg, (aze))
(61)
Following from the assumption that | x:1| S R, then (61) can be further simplified as
|
x=1 < [2 x" w²'l + 4o, Rt ees (ate)) + on (en,(a))
(62)
xe x=' || (4/2 + 1) Dx les (are)]
Summing the the both side of (62) from f = K to f = K + results in
K+t-1
| 2²11²= < o 2²1 + (4R+1)
ae (xx))
(63)
t=
follows Lemma 6 that the followng holds with probability at least Az.
3R² 4(4R + 1)
anlles,
(x,))
(64)
t=
Thus we have that
K+-1
|0 x xxx - x²112+ (4R+1
Σ
at (xx,
t=K
$ IKK - x²1| + (4R + 1)
Σ
0rl selle,, (xx))
(65)
t=
R²
3R2
R²
3R
< 4 + (4R 11)
<
+
<
R²
4(4R+11 +
4
holds with probability at least 1 - 1 Ø3, which completes the proof.
Based on the above lemmas, the Lemma 8 below shows if initial iterate of Half-Space Step locates closely enough to x", step size a polynomially decreases, and mini-batch size By polynomially increases, then x" inhabits all subsequent reduced space {S constructed in Half-Space Step with high probability.
Lemma 8. Suppose | - x <
R K > Np, k = K +L,EE 0 = O(I(vNtt) S b.
min 2(1-c) R 1. T.
M } and |Bx O(t) 2 N
Then for any constant T E (0, 1), K.
N
2M-
[ x= < R with probability at least 1 for any k >
Proof. It follows Lemma 4 and the assumption of this lemma that x' E Sg. Morcover, it follows the assumptions of this lemma, Lemma 6 and the definition of finite-sum f(x) in (1), and the bound of error as (60) that
O(N-K)
P((x))E E (x: || xx 1 R3
11-
(66)
(2
where the last two inequalities comes from that the error vanishing to zero as [Bg] reaches the upper bound N, and 0 is sufficiently large depending on 7 and O(N K).
19

---

Under review as a conference paper at ICLR 2021
Corollary 2. Lemma 8 further implies x" inhabits all subsequent Sk, i.e., x² for any k 2 K.
Next, we establish that after finitely number of iterations, HSPG generates sequences that inhabits in the feasible domain X' where Lipschitz continuity of y holds.
Lemma 9. Suppose the assumptions of Lemma 8 hold, then after finite number of iterations, all subsequent iterates 2k E X with high probability.
Proof. It follows Lemma 8 that all subsequent Dy satisfying |æ x²1l S R with high probability. Combining with Lemma 3, we have that I (x*) C (xx for all k 2 with high probability. Then for any 9 E I (mzg). there
two possbilities, either g I=*(""" first case E I (x') I (xx).. it follows the definitions of R as (23) and ô1 as (20) that
(ar". For the
||a x 1=ll=| < 2 w'1l
R 81
[
| I 61
(67)
[[ s| 2 [1 1g|| 61 > 26, 51 81
For any g (x=" 0 I (mx).. by Algorithm its norm is bounded below by
81 2 2 "lal = || 2 ||
(68)
where by the Theorem will shown in Appendix C.3, if [[ S -Ic+aa then [2x+ 1l, equals to
zero and will be fixed as zero since Algorithm 3 operates on Ss. as (7). Note 0r = (1/t), follow-
ing (Karimi et al., 2016, Theorem 4) and (Drusvyatkkiy & Lewis, 2018, Theorem 3.2). 1
(L/tt. If E > 0, then after finite number of iterations ((1/E²), g E (x=" ) NI (xx)) becomes zero. If
note B₃ = (tt) and Theorem 2.3, Theorem 3.2) (f needs further strongly convexity on X). Hence with high probability, after finite number of iterations, denoted by T, all subsequent æg. k > K + T inhabits X. Regarding [xx]ge (m)) for K I k S K + T, note e' ||[xx|g is also bounded below by constant e" [ > 0 given æk, for simplicity, denote the Lipschitz constant of [Vi(<x)]] }], as L as well.
finite-sum, then similar result holds by (Gower, 2018,
We now prove the first main theorem of HSPG, ie., Theorem 1.
Proof of Theorem 1:
We know that Algorithm performs an infinite sequence of iterations. It follows Corollary that for any l E Z+,
[[((xx) - E E(((((222))
[[x))
[[+++++)
K
(69)
2
ak
L
[[ IP|
1-E
Σ
Ock
Kgkge
KSASÉ
S
Combining the assumption that V is bounded below and letting E + CO, we obtain
a.L
(1- 1 L
([ - Σ
Σ
||
< 00
(70)
2
2
K2K
#2K
DEsR
By Algorithm 3, variables on I (Ix) are fixed during kth Half-Space Step and n is finite, then the group projection appears finitely many times, consequently,
-
L
Σ gEse
||<*|||||
< co.
(71)
2
KSK
Thus (70) implies that
aLL
Σ EE [
(72)
2
$ kZK
K
DEGx
Σ 04 Σ E
Σ
Σ E
<0
(73)
L
K
G
k2
gEg,
20

---

Under review as a conference paper at ICLR 2021
Since ag = (11/(VNt)) then ΣNSK an 0o and
RRnK o cc. Combining with (72) and the
boundness of (V, it implies
Σ
E
(2
< <0.
(74)
K2K
SE9n
By Σx>K as and (74), we have that
EE [1l =
lim inf
(75)
REK
DEGe
then there exists a subsequence such that
Σ R. [I =0 GELx
lim KEA
(76)
follows from the assumptions of this theorem and Lemma
and Corollay 2 that with high for each k 2 K, x² inhabits Sg. Note as |B& = O(t) linearly increases,
probability at least the error of gradient estimate vanishes. Hence, (76) naturally implies that the sequence { converges to some stationary point with high probability. And we can extend K. to {k k 2 K} due to the non-decreasing distance to optimal solution as shown in the Lemma 8. By the above, we conclude that
P( lim (Il = 0) 2 1-
(77)
k-+00
C.3 PROOF OF THEOREM 2
In this Appendix, we compare the group sparsity identification property HSPG and Prox-SG. We first show the generic sparsity identification property of Prox-SG for any mixed li regularization for parp& 2 1.
Lemma 10. If ||
x S min I, og where 1/p + 1/p' (p' = x ifp = 1), then
the Prox-SG yields that for each (x*). [Ix+1lg holds, i.e., (x*) S (*
Proof. It follows from the reverse triangle inequality, basic norm inequalities, Lipschitz continuity of (x) and the assumption of this lemma that for any E G,
[[V/,, (al)) [I[f,, (al))
S [[V5,, (Ixx) Vf8, (o (aa))alppr
S VfB, (xx)) - Vf8. (x (e')]l=:
(78)
- ax x x211p S L. 53 = 8g.
T.
By (78), we have that for any g (x))..
V[ (2x)lgl, S |[[ (x" (x')1g 1 + 8z
(79)
A 25, + 63 A 53
Combining (79) and the assumption of this lemma, the following holds for any > 0 that
| ag Vf8. (1w))ll,,
I 11| ||[ + |I Vfs. (Ix))
(80)
< 5 + ax 63) asd
which further implies that the Ecludiean projection yields that (rl [ ag Vfs. (mx))g) = [a - O VfB, (x*))gg-
(81)
Combining with (81), the fact that proximal operator is the residual of identity operator subtracted by Euclidean project operator onto the dual norm ball and [æ = for any g E [x(") (Chen, 2018), we have that
[ Prox cr All (I= az Vf8, (mx))g)
= I - Proj Projf((-),
[ as Vf8, (Iw))l
(82)
[ok as Vfs, (Ix))1
[i0 - as VT8, (I*)]g = =0,
consequently (xx" C (x which completes the proof.
21

---

Under review as a conference paper at ICLR 2021
Now establish the group- spoussspasstyy identification of HSPG as Theorem 2.
Proof of Theorem 2:
Suppose | - x01l _ 20, There is nothing to prove if g
For g E
T-
(x) II compute that
[x Ork V 1s (2rx)]g [x ||(2x]]11
||< axl V B3 (xx)]] [x [1
[2*k]
T
= =(1 e) [|[ 112 - on [...,, (3rx)ls 1lg - >
[ace]a
[[ 9
(83)
(1 ||[ am [VB.. (3xx 31 [20x] - <x | |
51 <) ||[=*|]]|| ork |1 VB. (xx) [gg ||[=*|11| a+d [
||< {[1 -) | ag [[V,, (#*))]gl ax1}
By the Lipschitz continuity of Vf. have that for each I (x)" (xx)
BB (mx)
Vfs, (z 1llll - L || - x"1sl1 |||l= = L ||
(84)
[[ (mx))all S [[ || + [[[ (a: ")s]lll
Combining with the definition of 8g. which implies that [|| (xx) (="))
26g that
[s, (mx)] < L | + X - 263
(85)
(83) can be further written as Hence combining with || < Ttogh
20g8
a ag W. (xx) 11 [x - - |0
$ |x {(1 - <) ||[=>|g|| + 04 Vf (Ix )l1l axd}
[ {(1 [|[
L || + and 200ds a&d}
|| {(1 - c + a4) ||[ 22
(86)
20
S |<*>]g||| (1-c+ + + c +12L 2000s
| g (20403 20kdg) = 0.
which shows that [z) W. (xx) }1g [x*]g I e <1[[=*]g]² Hence the group projection operator trigerred on map the variables to zero, then & E I (=x=1)), i.e. [Zs+1]g 1g 0. Therefore, the group sparsity of x can be successfully identified by Half-Spacc Step, i.e., (xx") C (x
C.4 UPPER BOUND OF Np UNDER STRONGLY CONVEXITY
Proposition Suppose the following conditions hold:
(AI) E S8, (w)) V.(x).
(A2) there exists σ 0 such that LE [B[[// -V(=))) o² for any mini-batch B.
1-8 (A3) there exists B E (0. such that 0 < as -
(A4) is p-sttrongly convex.
Set the step-size O%k = Znp Ko = max (1, 2nd }1. For any 7 E (0, 1), there exists a Np E Z+ such
Bk 22y = 111
807 log(No No }1. such that performing Prox-SG Np times
that Np 2 max yields
RºT
a2,92 R²
I - x S R/2
(87)
with probability at least
22

---

Under review as a conference paper at ICLR 2021
Proof. By the conditions (A1, A2, A3), Assumption 3.1 and Theorem 3.2 in Rosasco al. (2019), we have for any 2 2,
kp k
o²
log(k: - 1)
B 2t. - x "11 & El x x=1""
+
(88)
1
g2
k
Let BB
Sko- For any (0, 1), there exists a Np zz satisfying
ShuSko R
80 802
log
}
No 2
max
(89)
12-2RRT
we have
RP2
BlllaN, x= x=1
(90)
Therefore, by Markov inequality, we have that
R²
R
| - x 112 2
+
lNND - x
x011
(91)
holds with probability at least T.
D ADDITIONAL NUMERICAL EXPERIMENTS
In this section, provide additional numerical experiments to (i) demonstrate the validness of group sparsity identification of HSPG; (üi provide comprehensive comparison to Prox-SG, RDA and Prox-SVRG on benchmark convex problems; and (iii) describe more details regarding our non-convex deep learning experiments shown in the main body.
D.1 LINEAR REGRESSION ON SYNTHETIC DATA
We first numerically validate the proposed HSPG on group sparsity identification by linear regression problems with f regularizations using synthetic data. Consider data matrix Nx xn consisting of N instances and the target variable E R.N, we are interested in the following problem:
minimize DER
2N ||Ax
311
Σ |x|,|||-
(92)
SEG
Our goal is to empirically show that HSPG is able to identify the ground truth zero groups with synthetic data. We conduct the experiments as follows: (i) generate the data matrix whose elements uniformly distributed among [-1,1]; (ūi generate vector x= working as the ground truth solution, where the elements are uniformly distributed among [-1, 1] and the coordinates are equally number of groups of x to be 0 according to
divided into 10 groups (6 10); (iii) randomly
a pre-specified group sparsity ratio; (iv) compute the target variable Ax": (v) solve the above problem (92) for with A and only, and then evaluate the Intersection over Union (IoU) with respect to the identities of the zero groups between the computed solution estimate E by HSPG and the ground truth "..
We test HSPG on (92) under different problem settings. For slim matrix A where N 2 n. we test with various group sparsity ratios among {0.1, 0.3, 0.5, 0.7, 0.9), and for a fat matrix where N < we only test with a certain group sparsity value since recovery of x" requires that the number of non-zero elements in x" bounded by N. Throughout the experiments, we set À to be 100/N, the mini-batch size [B] to be 64, step size a% to be 0.1 (constant), and fine-tune per problem. Based on a similar statistical test on objective function stationarity (Zhang et al., 2020), we switch to Half-Space Step roughly after 30 epoches. Table 2 shows that under each setting, the proposed HSPG correctly identifies the groups of zeros as indicated by ( x = 1.0, which is a strong evidence to show the correctness of group sparsity identification of HSPG.
23

---

Under review as a conference paper at ICLR 2021
Table 2: Linear regression problem settings and IoU of the recovered solutions by HSPG.
N
Group sparsity ratio of x²
loU x")
10000
1000
{0.1. 0.3, 0.5, 0.7, 0.9)
1.0
10000
2000
{0.1, 0.3, 0.5, 0.7, 0.9
1.0
Slim A
10000
3000
{0.1, 0.3, 0.5, 0.7.
1.0
10000
4000
{0.1, 0.3, 0.5, 0.7, 0.9)
1.0
200
TO00
0.9
1.0
300
1000
0.8
1.0
Fat
400
1000
0.7
1.0
500
1000
0.6
1.0
D.2 LOGISTIC REGRESSION
We then focus on the benchmark convex logistic regression problem with the mixed l regularization given N examples (d1,11) (x) where d, IR and 4 E {-1,1] with the form
minimize (m:))ER==1
log(1 + - ((t)))
Σ ||]|||| DEG
(93)
for binary classification with bias E R. We set the regularization parameter À as 100/N throughout the experiments since yields high sparse solutions and low object value f's, equally decompose the variables into 10 groups to form G. and test problem (93) on 8 standard publicly available large-scale datasets from LIBSVM repository (Chang & Lin, 2011) as summarized in Table 3. All convex experiments are conducted on 64-bit operating system with an Intel(R) Core(TM) i7-7700K CPU a 4.20 GHz and 32 GB random-access memory.
We run the solvers with maximum number of epochs as 60. The mini-batch size [B] is set to be min [0.01N]} similarly to (Yang et al., 2019). The step size og setting follows [Section 4](Xiao & Zhang, 2014). Particularly, we first compute a Lipschitz constant L as max; ||d |1² /4, then fine tune and select constant ag results. For RDA, the step size parameter y is fined tuned as the one with the best performance among all powers of 10. For HSPG, we ar as the same as Prox-SG and Prox-SVRG in practice. We set Np as 30N/IB] such that Half-Space Step is triggered after employing Prox-SG Step 30 epochs similarly Appendix D.I, and the control parameter The final objective value & and f, and group sparsity in the solutions are reported in Table 4-6, where we mark the best values as bold to facilitate the comparison. Furthermore, Figure plots the relative runtime of these solvers for each dataset, scaled by the runtime of the most time- time-consuming solver.
1/L Prox-SG and Prox-SVRG since it exhibits the best
(9) as 0.05. We select two e's
and 0.05.
Table shows that our HSPG is definitely the best solver on exploring the group sparsity of the solutions. In fact, HSPG under second best solver on group sparsity exploration, which demonstrates that the variance reduction techniques works well in convex setting to promote sparsity. but not in non-convex settings. HSPG under e = 0 performs much better than Prox-SG which matches the better sparsity recovery property of HSPG as stated in Theorem 2 even under 0. Moreover, observe that all solvers perform quite competitively in terms of final objective values (round up to decimals) except RDA, which demonstrates that HSPG reaches comparable convergence as Prox-SG and Prox-SVRG in practice. Finally, Figure 3 indicates that Prox-SG, RDA and HSPG have similar computational cost to proceed, except ProX-SVRG due its periodical full gradient computation.
0.05 performs all the best except ijcnnl. Prox-SVRG is the
shown in Table 4 and 5,
Table 3: Summary of datasets.
Datasct
N
n
Attribute
Datasct
n
Attribute
a9a
32561
123
binary {0.
news20
19996
1355191
unit-length
higgs
11000000
28
real [-3, 41
real-sim
72309
20958
real [0.
ijennl
49990
real [-1, 1]
url. url_combined
2396130
3231961
real | 9]
kdda
8407752
20216830
real 1,
w8a
49749
300
binary {0, 1}
24

---

Under review as a conference paper at ICLR 2021
Table Final objective values v for tested algorithms on convex problems.
HSPG
Dataset
Prox-SC
RDA
Prex-SVRG
0,05
a9n
0.355
0.359
0.355
0.355
0.355
biggs
0.357
0,360
0.365
0.358
0.358
ijenn]
0.248
0.278
0.248
0.248
0.248
kdda
0.103
0.103
news20
0.538
real-sim
0.242
0,666
0.244
0.242
0.242
url _combined
0.397
0.579
0.391
0.405
0.405
wia
0.110
0.111
0.112
0.110
0.110
Table 5: Final objective values f for tested algorithms on convex problems.
HSSG
Dataset
Prox-SG
RDA
Prox SVRG
0.329
higgs
0.357
0,360
0.365
0.358
0.358
ijennl
0.213
0,270
0.213
0.213
0,214
kdda
0.103
0.124
0.103
0.103
0.103
nows20
0.373
0.693
0.381
0.372
0.372
real-sim
0.159
combined
0.391
w&a
0.089
0,098
0.091
0.089
0.089
Table 6: Group sparsity for tested algorithms convex problems.
HSPG
Prox-SVRG
8s 0.05
a9a
20%
30%
30%
30%
30%
biggs
0%
10%
0%%
0%
30%
ijennl
60%
kdda
0%
0%
80%
ncws20
20%
80%
90%
80%
90%
real-sim
0%
0%
80%
0%
80%
url ceommbieddd
0%
0%
0%
0%
90%
w8a
0%
0%
0%
1.0
Prox-SG RDA Prox-SVRG
Relative Runtime
HSProx-SG
0.5
0.0
a9aa
higgs
ijcnnl kdda news20 real-sim url
w8a
Figure 3: Relative runtime.
D.3
DEEP LEARNING EXPERIMENTS
We conduct all deep learning experiments on one GeForee GTX 1080 GPU, and describe how to fine-tune the control parameter in (9) in details.
The balances the sparsity level of the exact optimal solution and the bias of model estimation. Larger À encourages higher sparsity but may hurt the objective function. Therefore. in order to obtain solution of both high group sparsity and low objective value, we need to fine-tune carefully. In experiments, iterate À through all powers of 10 from 10-5 to 10-1 1 by proceeding
Fine-tune A:
25

---

Under review as a conference paper at ICLR 2021
1.5
Prox-SC
Prox-SVRG
1.2
(6+0.00)
value
0.9
(6-0.01)
HSPG 0.00))
HSPG (c-0.05)
0.6
0.3
0.0
75
150
225
300
Epoch
Figure 4: Evolution of value on ResNet18 with CIFAR1O.
Prox-SG, and pick up the largest À which has the same level of test accuracy to the model trained without any regularization.
Fine-tune According Theorem 2, larger results in faster group sparsity identification, while by Lemma on the other hand too large may cause a significant regression on the target objective value, iee, the value increases lot. Hence, in our experiments, from the point of view of optimization,
search proper in the following ways: start from = 0.0 and the models trained by employing Np Prox-SG Steps, incrementally increase by 0.01 and check if the on the first Half- Half-Space Step has obvious increase, then accept the largest without regression V as fine tuned e shown the main body of the paper. Particularly, the fine tuned é's equal to 0.03, 0.05, 0.02 and 0.02 for VGG16 with CIFARIO. VGG16 with Rashion-MNISS ResNet18 with CIFARIO and ResNet18 with Pashion-MNIS, respectively. Note from the perspective of different applications, there different criterions to fine tune i..., for model compression, may accept based the validation accuracy regression to reach higher group sparsity.
Additionally, we also report the final f comparison in Table and its evolution on ResNet18 with CIFAR 10 in Figure 4, where we can see that all tested algorithms can achieve competitive f values as they do convex settings. And the evolution of is similar to that of V iee. the raw objective f generally monotonically decreases for small 0 to 0.02, and experiences mild pulse after switch to Half-Space Step for larger e, e.g., 0.05, which matches Lemma 1.
Final f comparison:
Table 7: Final objective values f for tested algorithms on non-convex problems.
HSPG
Backhune
Dutaset
Prux SG
ProX-SVRG
tuned
CIFARIO
0.010
0.036
0.010
0.009
VGG16
Fashico-MNNSST
0.181
0.165
0.181
0.182
CIFAR10
0.001
0.002
0.001
0.004
ResNet18
Fashion MNIST
FNIST
0.006
0.008
0.005
0.010
CIFARIO
0.021
0.031
0.021
0.031
MobileNetV
Fashion-MNIIS
0.074
0.057
0.074
0.088
26


Reviews:
[{"review_id": "ONutkfltc7", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In summary, this paper proposes a post-processing algorithm on the estimator obtained by the usual proximal Stochastic gradient method. This leads to an estimator with enhanced group sparsity without the sacrifice of accuracy.\n\nMy major concern is how to use such a group sparsity result? We end up with a more group sparse estimator, which is good. But I feel like that is not the end of the story. In a deep neural network, the group sparsity seems not the major point people care about. My hope is that the enhanced group sparsity can be used to guide maybe the design of the neural network structure, or at least provide some better understandings of the model. For example, if we always see that some groups of filters are inactive, then we may modify the neural network structure accordingly, etc. In all, my understanding is that the group sparsity could be an intermediate result that can be further analyzed and used to improve the design of the model, instead of being the final goal itself.\n\nIf we start with different initialization (probably just slightly different), then will we end up with the same final estimator or at least the same group sparsity results?\n\nAfter obtaining the final estimator with group sparsity, can we refit the model on the active group index only? Will this improve the performance? ", "extra_scores": {}, "extra_text": {}}, {"review_id": "IQMqSSD2-Pr", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a new algorithm for the group sparsity regularization problem. They claim most existing algorithms, though return solutions with low objective function value, only give dense solutions and cannot effectively ensure the desired structured sparsity. The new technique requires an initialization that is closed to some truly sparse local minimum, which is achieved by running proximal gradient descent first. Then, they proposed a new half-space iterative step to force elements in specific groups exactly to zero. The authors also provide convergence analysis and numerical evidence for the newly proposed algorithm.\n\nComments:\n1. It is not clear to me, in Theorem 1, how are the parameters depend on the confidence \\tau? I am confused as it seems no parameter is explicitly dependent on \\tau, so the convergence in Theorem is almost surely one? I skim the proof and find that dependence vanished on the page (appendix) 10, proof of Lemma 6. I don’t understand why (1+\\theta) can be omitted. Please clarify.\n2. Where is N_P defined? N_P is used almost everywhere, for example, in statement of Theorem 1 and Algorithm 1. I didn’t find the definition of it. I guess N_P := min{k: ||x_k - x^*|| <= R/2} and R is further constrained by 2\\delta_1 - R > 0.\n3. In Theorem 1 and Proposition 1, only asymptotic and polynomial bounds are given. No rate of convergence for either the initialization phase or the half-space projection phase.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fe6PQ-Wk5Ty", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies how to solve a class of group sparsity regularized minimization problems. In particular, a half-space stochastic projected gradient (HSPG) method is proposed, which is based on the Prox-SG and a new half-space step that promotes group sparsity. This step is to decompose the feasible space and then perform group projection. Convergence analysis is provided, together with the theoretical discussion that HSPG has looser requirements to identify the sparsity patter than Prox-SG. Numerical experiments on the DCNNs based image classification shows the proposed method achieves the state-of-the-art performance in terms of accuracy. The work looks interesting with wide applications, especially in deep neural networks. However, the novelty is incremental and limited. \n\n1. There are some places where the notation is confusing. Vectors and scalars are constantly not distinguished.\n2. Practical guidance on the selection of the parameters $\\lambda$ and $\\varepsilon$ could be provided. \n3. In the numerical experiments, comparison of computational complexity and running time for the listed methods is not provided. Discussions on the group sparsity level and noise robustness could be included. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "3On7Zkjmubz", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "[Summary]\nThis paper proposes a new method called Half-space Stochastic Projected Gradient (HSPG) to find a group sparse solution of regularized finite-sum problems. Theoretical analysis tries to show the sparsity identification guarantees. In experiments, the effectiveness of HSPG was verified on the classification tasks.\n\n[Strength]\nThe idea behind the proposed method seems to be reasonable and interesting.\n\n[Weakness]\nA major concern is the correctness of the statements. In the equation (97) in the proof, the equation $E_B[e(x)] = 0$ is used essentially, and it is also stated in page 6. However, I think it does not hold because the proximal operator associated with sparse regularization is nonlinear. It may be probably difficult to fix this issue.\n\n[Minor comment]\nThere is a missing reference. It is known that RDA has the superior ability to find a manifold structure of solutions as shown in the following paper. \n\nS. Lee and S. J. Wright. Manifold identification in dual averaging for regularized stochastic online learning. JMLR, 2012.\n\n[Improvement]\nIf there is a misunderstanding in my review, I'd appreciate it if you could mention them.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper received four borderline reviews.  Overall, the manuscript has improved after the rebuttal (in particular, an issue in the convergence proof has been fixed), and a reviewer has increased his score to borderline accept. Yet, the paper did not convince the reviewers that the contribution was significant enough and none of the reviewer got enthusiastic about the paper. The main issue with the paper seems to be the unclear positioning between the optimization literature for stochastic composite optimization, the literature on support identification (e.g., Nutini, 2019), and the (more empirical) deep learning literature.

The paper postulates that the group-sparsity regularization is crucial for deep neural networks, which seems to be the main motivation of the paper. Yet, the experiments do not demonstrate any concrete consequence of better group sparsity, wether it is in terms of accuracy or interpretability.  If positioned in this literature, a comparison should be made with classical pruning approaches, where pruning occurs as an iterative procedure that is distinct from optimization. If positioned instead in the stochastic optimization literature, better analysis of the convergence rates should be provided; if positioned in the support identification literature, the paper should explain how the results compare to those of the literature (e.g., Nutini, 2019 and others). In other words, any point of view requires clarifications and additional discussions. 

Besides, 
   - the theoretical assumptions need to be discussed:  does the Lipschitz assumption holds for multilayer neural networks ? Certainly not for ReLu networks, but what can we say something useful, even with smooth activation functions?
   - the experimental setup needs more details. Reproducing the experiments with the current paper seems difficult; in particular, the choice of hyper-parameters is not crystal clear.

For these reasons, the area chair recommends to reject the paper, but encourages the authors to resubmit to a future venue while taking into account the previous comments.
Decision: Reject