Title: Implicit Regularization in Over-Parameterized Support Vector Machine
Venue: NeurIPS 2023
Accepted: yes
Keywords: Over-parameterization

SVM

Sparsity

Lasso
Abstract: In this paper, we design a regularization-free algorithm for high-dimensional support vector machines (SVMs) by integrating over-parameterization with Nesterov's smoothing method, and provide theoretical guarantees for the induced implicit regularization phenomenon. In particular, we construct an over-parameterized hinge loss function and estimate the true parameters by leveraging regularization-free gradient descent on this loss function. The utilization of Nesterov's method enhances the computational efficiency of our algorithm, especially in terms of determining the stopping criterion and reducing computational complexity. With appropriate choices of initialization, step size, and smoothness parameter, we demonstrate that unregularized gradient descent achieves a near-oracle statistical convergence rate. Additionally, we verify our theoretical findings through a variety of numerical experiments and compare the proposed method with explicit regularization. Our results illustrate the advantages of employing implicit regularization via gradient descent in conjunction with over-parameterization in sparse SVMs.

Implicit Regularization in Over-Parameterized Support Vector Machine

Abstract

In this paper, we design a regularization-free algorithm for high-dimensional support vector machines (SVMs) by integrating over-parameterization with Nesterov's smoothing method, and provide theoretical guarantees for the induced implicit regularization phenomenon. In particular, we construct an over-parameterized hinge loss function and estimate the true parameters by leveraging regularization-free gradient descent on this loss function. The utilization of Nesterov's method enhances the computational efficiency of our algorithm, especially in terms of determining the stopping criterion and reducing computational complexity. With appropriate choices of initialization, step size, and smoothness parameter, we demonstrate that unregularized gradient descent achieves a near-oracle statistical convergence rate. Additionally, we verify our theoretical findings through a variety of numerical experiments and compare the proposed method with explicit regularization. Our results illustrate the advantages of employing implicit regularization via gradient descent in conjunction with over-parameterization in sparse SVMs.

1 Introduction

In machine learning, over-parameterized models, such as deep learning models, are commonly used for regression and classification [22, 36, 31, 10]. The corresponding optimization tasks are primarily tackled using gradient-based methods. Despite challenges posed by the nonconvexity of the objective function and over-parameterization [40], empirical observations show that even simple algorithms, such as gradient descent, tend to converge to a global minimum. This phenomenon is known as the implicit regularization of variants of gradient descent, which acts as a form of regularization without an explicit regularization term [30, 29].

Implicit regularization has been extensively studied in many classical statistical problems, such as linear regression [35, 23, 44] and matrix factorization [14, 24, 3]. These studies have shown that unregularized gradient descent can yield optimal estimators under certain conditions. However, a deeper understanding of implicit regularization in classification problems, particularly in support vector machines (SVMs), remains limited. Existing studies have focused on specific cases and alternative regularization approaches [33, 27, 2]. A comprehensive analysis of implicit regularization via direct gradient descent in SVMs is still lacking. We need further investigation to explore the implications and performance of implicit regularization in SVMs.

The practical significance of such exploration becomes evident when considering today's complex data landscapes and the challenges they present. In modern applications, we often face classification

challenges due to redundant features. Data sparsity is notably evident in classification fields like finance, document classification, and gene expression analysis. For instance, in genomics, only a few genes out of thousands are used for disease diagnosis and drug discovery [17]. Similarly, spam classifiers rely on a small selection of words from extensive dictionaries [1]. These scenarios highlight limitations in standard SVMs and those with explicit regularization, like the ℓ 2 norm. From an applied perspective, incorporating sparsity into SVMs is an intriguing research direction. While sparsity in regression has been deeply explored recently, sparse SVMs have received less attention. Discussions typically focus on generalization error and risk analysis, with limited mention of variable selection and error bounds [32]. Our study delves into the implicit regularization in classification, complementing the ongoing research in sparse SVMs.

In this paper, we focus on the implicit regularization of gradient descent applied to high-dimensional sparse SVMs. Contrary to existing studies on implicit regularization in first-order iterative methods for nonconvex optimization that primarily address regression, we investigate this intriguing phenomenon of gradient descent applied to SVMs with hinge loss. By re-parameterizing the parameters using the Hadamard product, we introduce a novel approach to nonconvex optimization problems. With proper initialization and parameter tuning, our proposed method achieves the desired statistical convergence rate. Extensive simulation results reveal that our method outperforms the estimator under explicit regularization in terms of estimation error, prediction accuracy, and variable selection. Moreover, it rivals the performance of the gold standard oracle solution, assuming the knowledge of true support.

1.1 Our Contributions

First, we reformulate the parameter β as w ⊙ w -v ⊙ v , where ⊙ denotes the Hadamard product operator, resulting in a non-convex optimization problem. This re-parameterization technique has not been previously employed for classification problems. Although it introduces some theoretical complexities, it is not computationally demanding. Importantly, it allows for a detailed theoretical analysis of how signals change throughout iterations, offering a novel perspective on the dynamics of gradient descent not covered in prior works [13, 33]. With the help of re-parameterization, we provide a theoretical analysis showing that with appropriate choices of initialization size α , step size η , and smoothness parameter γ , our method achieves a near-oracle rate of √ s log p/n , where s is the number of the signals, p is the dimension of β , and n is the sample size. Notice that the near-oracle rate is achievable via explicit regularization [32, 46, 20]. To the best of our knowledge, this is the first study that investigates implicit regularization via gradient descent and establishes the near-oracle rate specifically in classification.

Second, we employ a simple yet effective smoothing technique [28, 45] for the re-parameterized hinge loss, addressing the non-differentiability of the hinge loss. Additionally, our method introduces a convenient stopping criterion post-smoothing, which we discuss in detail in Section 2.2. Notably, the smoothed gradient descent algorithm is not computationally demanding, primarily involving vector multiplication. The variables introduced by the smoothing technique mostly take values of 0 or 1 as smoothness parameter γ decreases, further streamlining the computation. Incorporating Nesterov's smoothing is instrumental in our theoretical derivations. Directly analyzing the gradient algorithm with re-parameterization for the non-smooth hinge loss, while computationally feasible, introduces complexities in theoretical deductions. In essence, Nesterov's smoothing proves vital both computationally and theoretically.

Third, to support our theoretical results, we present finite sample performances of our method through extensive simulations, comparing it with both the ℓ 1 -regularized estimator and the gold standard oracle solution. We demonstrate that the number of iterations t in gradient descent parallels the role of the ℓ 1 regularization parameter λ . When chosen appropriately, both can achieve the near-oracle statistical convergence rate. Further insights from our simulations illustrate that, firstly, in terms of estimation error, our method generalizes better than the ℓ 1 -regularized estimator. Secondly, for variable selection, our method significantly reduces false positive rates. Lastly, due to the efficient transferability of gradient information among machines, our method is easier to be paralleled and generalized to largescale applications. Notably, while our theory is primarily based on the Sub-Gaussian distribution assumption, our method is actually applicable to a much wider range. Additional simulations under heavy-tailed distributions still yield remarkably desired results. Extensive experimental results indicate that our method's performance, employing implicitly regularized gradient descent in SVMs,

rivals that of algorithms using explicit regularization. In certain simple scenarios, it even matches the performance of the oracle solution.

1.2 Related Work

Frequent empirical evidence shows that simple algorithms such as (stochastic) gradient descent tend to find the global minimum of the loss function despite nonconvexity. To understand this phenomenon, studies by [30, 29, 19, 38, 43] proposed that generalization arises from the implicit regularization of the optimization algorithm. Specifically, these studies observe that in over-parameterized statistical models, although optimization problems may contain bad local errors, optimization algorithms, typically variants of gradient descent, exhibit a tendency to avoid these bad local minima and converge towards better solutions. Without adding any regularization term in the optimization objective, the implicit preference of the optimization algorithm itself plays the role of regularization.

Implicit regularization has attracted significant attention in well-established statistical problems, including linear regression [13, 35, 39, 23, 44] and matrix factorization [14, 24, 3, 25, 26]. In highdimensional sparse linear regression problems, [35, 44] introduced a re-parameterization technique and demonstrated that unregularized gradient descent yields an estimator with optimal statistical accuracy under the Restricted Isometric Property (RIP) assumption [6]. [11] obtained similar results for the single index model without the RIP assumption. In low-rank matrix sensing, [14, 24] revealed that gradient descent biases towards the minimum nuclear norm solution when initiated close to the origin. Additionally, [3] demonstrated the same implicit bias towards the nuclear norm using a depthN linear network. Nevertheless, research on the implicit regularization of gradient descent in classification problems remains limited. [33] found that the gradient descent estimator converges to the direction of the max-margin solution on unregularized logistic regression problems. In terms of the hinge loss in SVMs, [2] provided a diagonal descent approach and established its regularization properties. However, these investigations rely on the diagonal regularization process [4], and their algorithms' convergence rates depend on the number of iterations, and are not directly compared with those of explicitly regularized algorithms. Besides, Frank-Wolfe method and its variants have been used for classification [21, 34]. However, sub-linear convergence to the optimum requires the strict assumption that both the direction-finding step and line search step are performed exactly [18].

2 Model and Algorithms

2.1 Notations

Throughout this work, we denote vectors with boldface letters and real numbers with normal font. Thus, w denotes a vector and w i denotes the i -th coordinate of w . We use [ n ] to denote the set { 1 , 2 . . . , n } . For any subset S in [ n ] and a vector w , we use w S to denote the vector whose i -th entry is w i if i ∈ S and 0 otherwise. For any given vector w , let ∥ w ∥ 1 , ∥ w ∥ and ∥ w ∥ ∞ denote its ℓ 1 , ℓ 2 and ℓ ∞ norms. Moreover, for any two vectors w , v ∈ R p , we define w ⊙ v ∈ R p as the Hadamard product of vectors w and v , whose components are w i v i for i ∈ [ p ] . For any given matrix X ∈ R p 1 × p 2 , we use ∥ X ∥ F and ∥ X ∥ S to represent the Frobenius norm and spectral norm of matrix X , respectively. In addition, we let { a n , b n } n ≥ 1 be any two positive sequences. We write a n ≲ b n if there exists a universal constant C such that a n ≤ C · b n and we write a n ≍ b n if we have a n ≲ b n and b n ≲ a n . Moreover, a n = O ( b n ) shares the same meaning with a n ≲ b n .

2.2 Over-parameterization for ℓ 1 -regularized SVM

Given a random sample Z n = { ( x i , y i ) } n i =1 with x i ∈ R p denoting the covariates and y i ∈ { 0 , 1 } denoting the corresponding label, we consider the following ℓ 1 -regularized SVM:

where ( · ) + = max {· , 0 } denotes the hinge loss and λ denotes the ℓ 1 regularization parameter. Let L Z n ( β ) denote the first term of the right-hand side of (1). Previous works have shown that the ℓ 1 -regularized estimator of the optimization problem (1) and its extensions achieve a near-oracle rate of convergence to the true parameter β ∗ [32, 37, 43]. In contrast, rather than imposing the ℓ 1

regularization term in (1), we minimize the hinge loss function L Z n ( β ) directly to obtain a sparse estimator. Specifically, we re-parameterize β as β = w ⊙ w -v ⊙ v , using two vectors w and v in R p . Consequently, L Z n ( β ) can be reformulated as E Z n ( w , v ) :

Note that the dimensionality of β in (1) is p , but a 2 p -dimensional parameter is involved in (2). This indicates that we over-parameterize β via β = w ⊙ w -v ⊙ v in (2). We briefly describe our motivation on over-parameterizing β this way. Following [16], ∥ β ∥ 1 = argmin β = a ⊙ c ( ∥ a ∥ 2 + ∥ c ∥ 2 ) / 2 . Thus, the optimization problem (1) translates to min a , c E Z n ( a , c ) + λ ( ∥ a ∥ 2 + ∥ c ∥ 2 ) / 2 . For a better understanding of implicit regularization by over-parameterization, we set w = a + c 2 and v = a -c 2 , leading to β = w ⊙ w -v ⊙ v . This incorporation of new parameters w and v effectively overparameterizes the problem. Finally, we drop the explicit ℓ 2 regularization term λ ( ∥ a ∥ 2 + ∥ c ∥ 2 ) / 2 and perform gradient descent to minimize the empirical loss E Z n ( w , v ) in (2), in line with techniques seen in neural network training, high-dimensional regression [35, 23, 44], and high-dimensional single index models [11].

2.3 Nesterov's smoothing

It is well-known that the hinge loss function is not differentiable. As a result, traditional first-order optimization methods, such as the sub-gradient and stochastic gradient methods, converge slowly and are not suitable for large-scale problems [45]. Second-order methods, like the Newton and QuasiNewton methods, can address this by replacing the hinge loss with a differentiable approximation [8]. Although these second-order methods might achieve better convergence rates, the computational cost associated with computing the Hessian matrix in each iteration is prohibitively high. Clearly, optimizing (2) using gradient-based methods may not be the best choice.

To address the trade-off between convergence rate and computational cost, we incorporate Nesterov's method [28] to smooth the hinge loss and then update the parameters via gradient descent. By employing Nesterov's method, (2) can be reformulated as the following saddle point function:

where P 1 = { µ ∈ R n : 0 ≤ µ i ≤ 1 } . According to [28], the above saddle point function can be smoothed by subtracting a prox-function d γ ( µ ) , where d γ ( µ ) is a strongly convex function of µ with a smoothness parameter γ > 0 . Throughout this paper, we select the prox-function as d γ ( µ ) = γ 2 ∥ µ ∥ 2 . Consequently, E Z n ( w , v ) can be approximated by

Since d γ ( µ ) is strongly convex, µ i can be obtained by setting the gradient of the objective function in (3) to zero and has the explicit form:

For each sample point Z i = { ( x i , y i ) } , i ∈ [ n ] , we use E Z i ( w , v ) and E ∗ Z i ,γ ( w , v ) to denote its hinge loss and the corresponding smoothed approximation, respectively. With different choices of µ i for any i ∈ [ n ] in (4), the explicit form of E ∗ Z i ,γ ( w , v ) can be written as

Note that the explicit solution (5) indicates that a larger γ yields a smoother E ∗ Z i ,γ ( w , v ) with larger approximation error, and can be considered as a parameter that controls the trade-off between smoothness and approximation accuracy. The following theorem provides the theoretical guarantee of the approximation error. The proof can be directly derived from (5), and thus is omitted here.

Theorem 1. For any random sample Z i = { ( x i , y i ) } , i ∈ [ n ] , the corresponding hinge loss E Z i ( w , v ) is bounded by its smooth approximation E ∗ Z i ,γ ( w , v ) , and the approximation error is completely controlled by the smooth parameter γ . For any ( w , v ) , we have

2.4 Implicit regularization via gradient descent

In this section, we apply gradient descent algorithm to E Z n ,γ ( w , v ) in (3) by updating w and v to obtain the estimator of β ∗ . Specifically, the gradients of (3) with respect to w and v can be directly obtained, with the form of -2 /n ∑ n i =1 y i µ i x i ⊙ w and 2 /n ∑ n i =1 y i µ i x i ⊙ v , respectively. Thus, the updates for w and v are given as

where η denotes the step size. Once ( w t +1 , v t +1 ) is obtained, we can update β as β t +1 = w t +1 ⊙ w t +1 -v t +1 ⊙ v t +1 via the over-parameterization of β . Note that we cannot initialize the values of w 0 and v 0 as zero vectors because these vectors are stationary points of the algorithm. Given the sparsity of the true parameter β ∗ with the support S , ideally, w and v should be initialized with the same sparsity pattern as β ∗ , with w S and v S being non-zero and the values outside the support S being zero. However, such initialization is infeasible as S is unknown. As an alternative, we initialize w 0 and v 0 as w 0 = v 0 = α 1 p × 1 , where α > 0 is a small constant. This initialization approach strikes a balance: it aligns with the sparsity assumption by keeping the zero component close to zero, while ensuring that the non-zero component begins with a non-zero value [11].

We summarize the details of the proposed gradient descent method for high-dimensional sparse SVM in the following Algorithm 1.

Algorithm 1 : Gradient Descent Algorithm for High-Dimensional Sparse SVM.

Given : Training set Z n , initial value α , step size η , smoothness parameter γ , maximum iteration number T 1 , validation set ˜ Z n .

Initialize : w 0 = α 1 p × 1 , v 0 = α 1 p × 1 , and set iteration index t = 0 .

While t < T 1 , do

End if t > T 1 or µ t = 0 . Return Set ̂ β as β t .

We highlight three key advantages of Algorithm 1. First, the stopping condition for Algorithm 1 can be determined based on the value of µ in addition to the preset maximum iteration number T 1 . Specifically, when the values of µ i are 0 across all samples, the algorithm naturally stops as no further updates are made. Thus, µ serves as an intrinsic indicator for convergence, providing a more efficient stopping condition. Second, Algorithm 1 avoids heavy computational cost like the computation of the Hessian matrix. The main computational load comes from the vector multiplication in (6). Since a considerable portion of the elements in µ are either 0 or 1 , and the proportion of these elements increases substantially as γ decreases, the computation in (6) can be further simplified. Lastly, the utilization of Nesterov's smoothing not only optimizes our approach but also aids in our theoretical derivations, as detailed in Appendix E.

3 Theoretical Analysis

In this section, we analyze the theoretical properties of Algorithm 1. The main result is the error bound ∥ β t -β ∗ ∥ , where β ∗ is the minimizer of the population hinge loss function for β without the ℓ 1 norm: β ∗ = arg min β ∈ R p E (1 -y x T β ) + . We start by defining the δ -incoherence, a key assumption for our analysis.

Definition 1. Let X ∈ R n × p be a matrix with ℓ 2 -normalized columns x 1 , . . . , x p , i.e., ∥ x i ∥ = 1 for all i ∈ [ n ] . The coherence δ = δ ( X ) of the matrix X is defined as

̸

where 1 K denotes the n -dimensional vector whose i -th entry is 1 if i ∈ K and 0 otherwise. Then, the matrix X is said to be satisfying δ -incoherence.

Coherence measures the suitability of measurement matrices in compressed sensing [12]. Several techniques exist for constructing matrices with low coherence. One such approach involves utilizing sub-Gaussian matrices that satisfy the low-incoherence property with high probability [9, 7]. The Restricted Isometry Property (RIP) is another key measure for ensuring reliable sparse recovery in various applications [35, 44], but verifying RIP for a designed matrix is NP-hard, making it computationally challenging [5]. In contrast, coherence offers a computationally feasible metric for sparse regression [9, 23]. Hence, the assumptions required in our main theorems can be verified within polynomial time, distinguishing them from the RIP assumption.

Recall that β ∗ ∈ R p is the s -sparse signal to be recovered. Let S ⊂ { 1 , . . . , p } denote the index set that corresponds to the nonzero components of β ∗ , and the size | S | of S is given by s . Among the s nonzero signal components of β ∗ , we define the index set of strong signals as S 1 = { i ∈ S : | β ∗ i | ≥ C s log p √ log p/n } and of weak signals as S 2 = { i ∈ S : | β ∗ i | ≤ C w √ log p/n } for some constants C s , C w > 0 . We denote s 1 and s 2 as the cardinalities of S 1 and S 2 , respectively. Furthermore, we use m = min i ∈ S 1 | β ∗ i | to represent the minimal strength for strong signals and κ to represent the condition number-the ratio of the largest absolute value of strong signals to the smallest. In this paper, we focus on the case that each nonzero signal in β ∗ is either strong or weak, which means that s = s 1 + s 2 . Regarding the input data and parameters in Algorithm 1, we introduce the following two structural assumptions.

Assumption 1. The design matrix X / √ n satisfies δ -incoherence with 0 < δ ≲ 1 / ( κs log p ) . In addition, every entry x of X is i.i.d. zero-mean sub-Gaussian random variable with bounded subGaussian norm σ .

Assumption 2. The initialization for gradient descent are w 0 = α 1 p × 1 , v 0 = α 1 p × 1 where the initialization size α satisfies 0 < α ≲ 1 /p , the parameter of prox-function γ satisfies 0 < γ ≤ 1 /n , and the step size η satisfies 0 < η ≲ 1 / ( κ log p ) .

Assumption 1 characterizes the distribution of the input data, which can be easily satisfied across a wide range of distributions. Interestingly, although our proof relies on Assumption 1, numerical results provide compelling evidence that it isn't essential for the success of our method. This indicates that the constraints set by Assumption 1 can be relaxed in practical applications, as discussed in Section 4. The assumptions about the initialization size α , the smoothness parameter γ , and the step size η primarily stem from the theoretical induction of Algorithm 1. For instance, α controls the strength of the estimated weak signals and error components, γ manages the approximation error in smoothing, and η affects the accuracy of the estimation of strong signals. Our numerical simulations indicate that extremely small initialization size α , step size η , and smoothness parameter γ are not required to achieve the desired convergence results, highlighting the low computational burden of our method, with details found in Section 4. The primary theoretical result is summarized in the subsequent theorem.

Theorem 2. Suppose that Assumptions 1 and 2 hold, then there exist positive constants c 1 , c 2 , c 3 and c 4 such that there holds with probability at least 1 -c 1 n -1 -c 2 p -1 that, for every time t with c 3 log( m/α 2 ) / ( ηm ) ≤ t ≤ c 4 log(1 /α ) / ( η log n ) , the solution of the t -th iteration in Algorithm 1 , β t = w t ⊙ w t -v t ⊙ v t , satisfies

Theorem 2 demonstrates that if β ∗ contains s nonzero signals, then with high probability, for any t ∈ [ c 3 log( m/α 2 ) / ( ηm ) , c 4 log(1 /α ) / ( η log n )] , the convergence rate of O ( √ s log p/n ) in terms of the ℓ 2 norm can be achieved. Such a convergence rate matches the near-oracle rate of sparse SVMs and can be attained through explicit regularization using the ℓ 1 norm penalty [32, 46], as well as through concave penalties [20]. Therefore, Theorem 2 indicates that with over-parameterization, the implicit regularization of gradient descent achieves the same effect as imposing an explicit regularization into the objective function in (1).

Proof Sketch. The ideas behind the proof of Theorem 2 are as follows. First, we can control the estimated strengths associated with the non-signal and weak signal components, denoted as ∥ w t ⊙ 1 S c 1 ∥ ∞ and ∥ v t ⊙ 1 S c 1 ∥ ∞ , to the order of the square root of the initialization size α for up to O (log(1 /α ) / ( η log n )) steps. This provides an upper boundary on the stopping time. Also, the magnitude of α determines the size of coordinates outside the signal support S 1 at the stopping time. The importance of choosing small initialization sizes and their role in achieving the desired statistical performance are further discussed in Section 4. On the other hand, each entry of the strong signal part, denoted as β t ⊙ 1 S 1 , increases exponentially with an accuracy of around O (log p/n ) near the true parameter β ∗ ⊙ 1 S 1 within roughly O (log( m/α 2 ) / ( ηm )) steps. This establishes the left boundary of the stopping time. The following two Propositions summarize these results.

Proposition 1. (Analyzing Weak Signals and Errors) Under Assumptions 1 -2 , with probability at least 1 -cn -1 , we have

for all t ≤ T ∗ = O (log(1 /α ) / ( η log n )) , where c is some positive constant.

Proposition 2. (Analyzing Strong Signals) Under Assumptions 1 -2 , with probability at least 1 -c 1 n -1 -c 2 p -1 , we have

holds for all O (log( m/α 2 ) / ( ηm )) ≤ t ≤ O (log(1 /α ) / ( η log n )) where c 1 , c 2 are two constants.

Consequently, by appropriately selecting the stopping time t within the interval specified in Theorem 2, we can ensure convergence of the signal components and effectively control the error components. The final convergence rate can be obtained by combining the results from Proposition 1 and Proposition 2.

4 Numerical Study

In our simulations, unless otherwise specified, we follow a default setup. We generate 3 n independent observations, divided equally for training, validation, and testing. The true parameters β ∗ is set to m 1 S with a constant m . Each entry of x is sampled as i.i.d. zero-mean Gaussian random variable, and the labels y are determined by a binomial distribution with probability p = 1 / (1 + exp( x T β ∗ )) . Default parameters are: true signal strength m = 10 , number of signals s = 4 , sample size n = 200 , dimension p = 400 , step size η = 0 . 5 , smoothness parameter γ = 10 -4 , and initialization size α = 10 -8 . For evaluation, we measure the estimation error using ∥ β t / ∥ β t ∥ -β ∗ / ∥ β ∗ ∥∥ (for comparison with oracle estimator) and the prediction accuracy on the testing set with P (ˆ y = y test ) . Additionally, "False positive" and "True negative" metrics represent variable selection errors. Specifically, "False positive" means the true value is zero but detected as a signal, while "True negative" signifies a non-zero true value that isn't detected. Results are primarily visualized employing shaded plots and boxplots, where the solid line depicts the median of 30 runs and the shaded area marks the 25 -th and 75 -th percentiles over these runs.

Effects of Small Initialization Size. We investigate the power of small initialization size α on the performance of our algorithm. We set the initialization size α = { 10 -4 , 10 -6 , 10 -8 , 10 -10 } , and other parameters are set by default. Figure 1 shows the importance of small initialization size in inducing exponential paths for the coordinates. Our simulations reveal that small initialization size leads to lower estimation errors and more precise signal recovery, while effectively constraining the error term to a negligible magnitude. Remarkably, although small initialization size might slow the convergence rate slightly, this trade-off is acceptable given the enhanced estimation accuracy.

0.4

0.6

0.8

1.0

0

200

400

600

Number of Iterations

Estimation Error

Initialization

1e-10

1e-4

1e-6

1e-8

(a) Effects of Initialization

0

2

4

6

8

0

140

280

420

560

700

Number of Iterations

Strong Signals

Initialization

1e-10

1e-4

1e-6

1e-8

(b) Paths of Signals

0e+00

1e-05

2e-05

3e-05

0

140

280

420

560

700

Number of Iterations

Error Components

Initialization

1e-10

1e-4

1e-6

1e-8

(c) Paths of Errors

Effects of Signal Strength and Sample Size. We examine the influence of signal strength on the estimation accuracy of our algorithm. We set the true signal strength m = 0 . 5 ∗ k, k = 1 , . . . , 20 and keep other parameters at their default values. As depicted in Figure 2, we compare our method (denoted by GD ) with ℓ 1 -regularized SVM (denoted by Lasso method), and obtain the oracle solution (denoted by Oracle ) using the true support information. We assess the advantages of our algorithm from three aspects. Firstly, in terms of estimation error, our method consistently outperforms the Lasso method across different signal strengths, approaching near-oracle performance. Secondly, all three methods achieve high prediction accuracy on the testing set. Lastly, when comparing variable selection performance, our method significantly surpasses the Lasso method in terms of false positive error. Since the true negative error of both methods is basically 0 , we only present results for false positive error in Figure 2.

0.00

0.25

0.50

0.75

1.00

1.25

2.5

5.0

7.5

10.0

Strength of Signal

Estimation Error

Method

Oracle

(a) Estimation Result

0.5

0.6

0.7

0.8

0.9

2.5

5.0

7.5

10.0

Strength of Signal

Prediction Accuracy

Method

Oracle

(b) Prediction Result

0

50

100

2.5

5.0

7.5

10.0

Strength of Signal

False Positive

Method

(c) Variable Selection Result

We further analyze the impact of sample size n on our proposed algorithm. Keeping the true signal strength fixed at m = 5 , we vary the sample size as n = 50 ∗ k for k = 1 , . . . , 8 . Other parameters remain at their default values. Consistently, our method outperforms the Lasso method in estimation, prediction, and variable selection, see Figure 3 for a summary of the results.

Performance on Complex Signal Structure. To examine the performance of our method under more complex signal structures, we select five signal structures: A -(5 , 6 , 7 , 8) , B -(4 , 6 , 8 , 9) , C -(3 , 6 , 9 , 10) , D -(2 , 6 , 10 , 11) , and E -(1 , 6 , 11 , 12) . Other parameters are set by default. The

0.00

0.25

0.50

0.75

100

200

300

400

Number of Samples

Estimation Error

Method

Oracle

(a) Estimation Result

0.6

0.7

0.8

0.9

100

200

300

400

Number of Samples

Prediction Accuracy

Method

Oracle

(b) Prediction Result

0

20

40

60

80

100

200

300

400

Number of Samples

False Positive

Method

(c) Variable Selection Result

results, summarized in Figure 4, highlight the consistent superiority of our method over the Lasso method in terms of prediction and variable selection performance, even approaching an oracle-like performance for complex signal structures. High prediction accuracy is achieved by the both methods.

0.0

0.1

0.2

0.3

A

B

C

D

E

Different Signals

Estimation Error

Method

Oracle

(a) Estimation Result

0.80

0.85

0.90

0.95

A

B

C

D

E

Different Signals

Prediction Accuracy

Method

Oracle

(b) Prediction Result

0

20

40

60

80

A

B

C

D

E

Different Signals

False Positive

Method

(c) Variable Selection Result

Performance on Heavy-tailed Distribution. Although the sub-Gaussian distribution of input data is assumed in Assumption 1, we demonstrate that our method can be extended to a wider range of distributions. We conduct simulations under both uniform and heavy-tailed distributions. The simulation setup mirrors that of Figure 4, with the exception that we sample x from a [ -1 , 1] uniform distribution and a t (3) distribution, respectively. Results corresponding to the t (3) distribution are presented in Figure 5, and we can see that our method maintains strong performance, suggesting that the constraints of Assumption 1 can be substantially relaxed. Additional simulation results can be found in the Appendix A.

Sensitivity Analysis with respect to smoothness parameter γ . Weanalyze the impact of smoothness parameter γ on our proposed algorithm. Specifically, the detailed experimental setup follows the default configuration, and γ is set within the range [2 . 5 × 10 -5 , 1 × 10 -3 ] . The simulations are replicated 30 times, and the numerical results of estimation error and four estimated signal strengths are presented in Figure 6. From Figure 6, it's evident that the choice of γ is relatively insensitive in the sense that the estimation errors and the estimated strengths of the signals under different γ s are very close. Furthermore, as γ increases, the estimation accuracy experiences a minor decline, but it remains within an acceptable range. See Appendix A for simulation results of Signal 3 and Signal 4 .

0.0

0.2

0.4

0.6

A

B

C

D

E

Different Signals

Estimation Error

Method

Oracle

(a) Estimation Result

0.80

0.85

0.90

0.95

A

B

C

D

E

Different Signals

Prediction Accuracy

Method

Oracle

(b) Prediction Result

0

25

50

75

100

A

B

C

D

E

Different Signals

False Positive

Method

(c) Variable Selection Result

0.4

0.5

0.6

2.5e-05

5.0e-05

7.5e-05

1.0e-04

2.5e-04

5.0e-04

0.001

Gamma Values

Estimation Error

(a) Estimation Result

4

5

6

7

2.5e-05

5.0e-05

7.5e-05

1.0e-04

2.5e-04

5.0e-04

0.001

Gamma Values

Signal 1

(b) Signal

1

4

5

6

2.5e-05

5.0e-05

7.5e-05

1.0e-04

2.5e-04

5.0e-04

0.001

Gamma Values

Signal 2

(c) Signal

2

5 Conclusion

In this paper, we leverage over-parameterization to design an unregularized gradient-based algorithm for SVM and provide theoretical guarantees for implicit regularization. We employ Nesterov's method to smooth the re-parameterized hinge loss function, which solves the difficulty of non-differentiability and improves computational efficiency. Note that our theory relies on the incoherence of the design matrix. It would be interesting to explore to what extent these assumptions can be relaxed, which is a topic of future work mentioned in other studies on implicit regularization. It is also promising to consider extending the current study to nonlinear SVMs, potentially incorporating the kernel technique to delve into the realm of implicit regularization in nonlinear classification. In summary, this paper not only provides novel theoretical results for over-parameterized SVMs but also enriches the literature on high-dimensional classification with implicit regularization.

References

Tiago A Almeida, José María G Hidalgo, and Akebo Yamakami. Contributions to the study of sms spam filtering: new collection and results. In Proceedings of the 11th ACM symposium on Document engineering , pages 259-262, 2011.

Vassilis Apidopoulos, Tomaso Poggio, Lorenzo Rosasco, and Silvia Villa. Iterative regularization in classification via hinge loss diagonal descent. arXiv preprint arXiv:2212.12675 , 2022.

Sanjeev Arora, Nadav Cohen, Wei Hu, and Yuping Luo. Implicit regularization in deep matrix factorization. Advances in Neural Information Processing Systems , 32, 2019.

MA Bahraoui and B Lemaire. Convergence of diagonally stationary sequences in convex optimization. Set-Valued Analysis , 2:49-61, 1994.

Afonso S Bandeira, Edgar Dobriban, Dustin G Mixon, and William F Sawin. Certifying the restricted isometry property is hard. IEEE transactions on information theory , 59(6):3448-3450, 2013.

Emmanuel J Candes and Terence Tao. Decoding by linear programming. IEEE transactions on information theory , 51(12):4203-4215, 2005.

Lawrence Carin, Dehong Liu, and Bin Guo. Coherence, compressive sensing, and random sensor arrays. IEEE Antennas and Propagation Magazine , 53(4):28-39, 2011.

Olivier Chapelle. Training a support vector machine in the primal. Neural computation , 19(5):1155-1178, 2007.

David L Donoho, Michael Elad, and Vladimir N Temlyakov. Stable recovery of sparse overcomplete representations in the presence of noise. IEEE Transactions on information theory , 52(1):6-18, 2005.

Jianqing Fan, Cong Ma, and Yiqiao Zhong. A selective overview of deep learning. Statistical science: a review journal of the Institute of Mathematical Statistics , 36(2):264, 2021.

Jianqing Fan, Zhuoran Yang, and Mengxin Yu. Understanding implicit regularization in overparameterized single index model. Journal of the American Statistical Association , pages 1-14, 2022.

Simon Foucart, Holger Rauhut, Simon Foucart, and Holger Rauhut. An invitation to compressive sensing . Springer, 2013.

Suriya Gunasekar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit bias of gradient descent on linear convolutional networks. Advances in neural information processing systems , 31, 2018.

Suriya Gunasekar, Blake E Woodworth, Srinadh Bhojanapalli, Behnam Neyshabur, and Nati Srebro. Implicit regularization in matrix factorization. Advances in Neural Information Processing Systems , 30, 2017.

Xin He, Shaogao Lv, and Junhui Wang. Variable selection for classification with derivativeinduced regularization. Statistica Sinica , 30(4):2075-2103, 2020.

P. Hoff. Lasso, fractional norm and structured sparse estimation using a hadamard product parametrization. Computational Statistics & Data Analysis , 115 :186-198, 2017.

Yuan Huang, Jian Huang, Ben-Chang Shia, and Shuangge Ma. Identification of cancer genomic markers via integrative sparse boosting. Biostatistics , 13(3):509-522, 2012.

Martin Jaggi. Revisiting frank-wolfe: Projection-free sparse convex optimization. In International conference on machine learning , pages 427-435. PMLR, 2013.

Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. In International Conference on Learning Representations .

Rachid Kharoubi, Abdallah Mkhadri, and Karim Oualkacha. High-dimensional penalized bernstein support vector machines. arXiv preprint arXiv:2303.09066 , 2023.

Simon Lacoste-Julien, Martin Jaggi, Mark Schmidt, and Patrick Pletscher. Block-coordinate frank-wolfe optimization for structural svms. In International Conference on Machine Learning , pages 53-61. PMLR, 2013.

Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. Nature , 521(7553):436-444, 2015.

Jiangyuan Li, Thanh Nguyen, Chinmay Hegde, and Ka Wai Wong. Implicit sparse regularization: The impact of depth and early stopping. Advances in Neural Information Processing Systems , 34:28298-28309, 2021.

Yuanzhi Li, Tengyu Ma, and Hongyang Zhang. Algorithmic regularization in over-parameterized matrix sensing and neural networks with quadratic activations. In Conference On Learning Theory , pages 2-47. PMLR, 2018.

Jianhao Ma and Salar Fattahi. Sign-rip: A robust restricted isometry property for low-rank matrix recovery. arXiv preprint arXiv:2102.02969 , 2021.

Jianhao Ma and Salar Fattahi. Global convergence of sub-gradient method for robust matrix recovery: Small initialization, noisy measurements, and over-parameterization. arXiv preprint arXiv:2202.08788 , 2022.

Cesare Molinari, Mathurin Massias, Lorenzo Rosasco, and Silvia Villa. Iterative regularization for convex regularizers. In International conference on artificial intelligence and statistics , pages 1684-1692. PMLR, 2021.

Y. Nesterov. Smooth minimization of non-smooth functions. Mathematical Programming , 103 :127-152, 2005.

Behnam Neyshabur, Ryota Tomioka, Ruslan Salakhutdinov, and Nathan Srebro. Geometry of optimization and implicit regularization in deep learning. arXiv preprint arXiv:1705.03071 , 2017.

Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. In search of the real inductive bias: On the role of implicit regularization in deep learning. International Conference on Learning Representations , 2015.

Daniel W Otter, Julian R Medina, and Jugal K Kalita. A survey of the usages of deep learning for natural language processing. IEEE transactions on neural networks and learning systems , 32(2):604-624, 2020.

Bo Peng, Lan Wang, and Yichao Wu. An error bound for l1-norm support vector machine coefficients in ultra-high dimension. The Journal of Machine Learning Research , 17(1):82798304, 2016.

Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research , 19(1):2822-2878, 2018.

Kenya Tajima, Yoshihiro Hirohashi, Esmeraldo Ronnie Rey Zara, and Tsuyoshi Kato. Frankwolfe algorithm for learning svm-type multi-category classifiers. In Proceedings of the 2021 SIAM International Conference on Data Mining (SDM) , pages 432-440. SIAM, 2021.

Tomas Vaskevicius, Varun Kanade, and Patrick Rebeschini. Implicit regularization for optimal sparse recovery. Advances in Neural Information Processing Systems , 32, 2019.

Athanasios Voulodimos, Nikolaos Doulamis, Anastasios Doulamis, Eftychios Protopapadakis, et al. Deep learning for computer vision: A brief review. Computational intelligence and neuroscience , 2018, 2018.

Junhui Wang, Jiankun Liu, Yong Liu, et al. Improved learning rates of a functional lasso-type svm with sparse multi-kernel representation. Advances in Neural Information Processing Systems , 34:21467-21479, 2021.

Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. Advances in neural information processing systems , 30, 2017.

Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory , pages 3635-3673. PMLR, 2020.

Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks. In International Conference on Learning Representations .

Hao Helen Zhang. Variable selection for support vector machines via smoothing spline anova. Statistica Sinica , pages 659-674, 2006.

Xiang Zhang, Yichao Wu, Lan Wang, and Runze Li. Variable selection for support vector machines in moderately high dimensions. Journal of the Royal Statistical Society Series B: Statistical Methodology , 78(1):53-76, 2016.

Yingying Zhang, Yan-Yong Zhao, and Heng Lian. Statistical rates of convergence for functional partially linear support vector machines for classification. Journal of Machine Learning Research , 23(156):1-24, 2022.

Peng Zhao, Yun Yang, and Qiao-Chu He. High-dimensional linear regression via implicit regularization. Biometrika , 109(4):1033-1046, 2022.

Tianyi Zhou, Dacheng Tao, and Xindong Wu. Nesvm: A fast gradient method for support vector machines. In 2010 IEEE International Conference on Data Mining , pages 679-688. IEEE, 2010.

Xingcai Zhou and Hao Shen. Communication-efficient distributed learning for high-dimensional support vector machines. Mathematics , 10(7):1029, 2022.

Appendix

The appendix is organized as follows.

In Appendix A, we provide supplementary explanations and additional experimental results for the numerical study.

In Appendix B, we discuss the performance of our method using other data generating schemes.

In Appendix C, we characterize the dynamics of the gradient descent algorithm for minimizing the Hadamard product re-parametrized smoothed hinge loss E ∗ Z n ,γ .

In Appendix D, we prove the main results stated in the paper.

In Appendix E, we provide the proof of propositions and technical lemmas in Section 3.

A Additional Experimental Results

A.1 Effects of Small Initialization on Error Components

We present a comprehensive analysis of the impact of initialization on the error components, as depicted in Figure 7. Our results demonstrate that, while the initialization size α does not alter the error trajectory, it significantly affects the error bounds. Numerically, when the initialization α is 10 -4 , the maximum absolute value of the error components is around 3 × 10 -5 , which is bounded by the initialization size 10 -4 . Similarly, when the initialization is 10 -10 , the maximum absolute value of the error components is around 7 . 5 × 10 -14 , which is bounded by the initialization size 10 -10 . The same result is obtained for the other two initialization sizes. The specific numerical results we obtained validate the conclusions drawn in Proposition 1, as the error term satisfies

A.2 True Negative Results

In the main text, we could not include the True Negative error figures in 2, 3, 4, and 5 due to space constraints. However, these figures are provided in Figure 8. We observe that both the gradient descent estimator and the Lasso estimator effectively detect the true signals across different settings.

A.3 Additional Results under Uniform Distribution

As previously described, we sample X from a [ -1 , 1] uniform distribution and set other parameters as follows: true signal structures are A = (5 , 6 , 7 , 8) , B = (4 , 6 , 8 , 9) , C = (3 , 6 , 9 , 10) , D = (2 , 6 , 10 , 11) , and E = (1 , 6 , 11 , 12) . The number of signals is s = 4 , the sample size is n = 200 , dimension is p = 400 , step size is η = 0 . 5 , smoothness parameter is γ = 10 -4 , and the initialization size is α = 10 -8 . The experimental results are summarized in Figure 9. As depicted in Figure 9, the gradient descent (GD) estimator consistently outperforms the Lasso estimator in terms of estimation accuracy and variable selection error. Both methods slightly surpass the oracle estimator in terms of prediction. However, this advantage mainly stems from the inevitable overestimation of the number of signals in the estimates by both the GD and Lasso methods.

A.4 Additional Results of Sensitivity Analysis

Within our sensitivity analysis, the simulation results for Signal 3 and Signal 4 are presented in Figure 10. We observe that the estimated strengths of the two signals exhibit similar distributions across different values of the smoothness parameter γ .

B Other Data Generating Schemes

In section 4, the scheme for generating y in our original submission is based on a treatment similar to those in [41, 42], where the task of variable selection for the support vector machine is considered.

0e+00

1e-05

2e-05

3e-05

0

140

280

420

560

700

Number of Iterations

Error Components

Initialization

1e-4

-1e-08

0e+00

1e-08

2e-08

3e-08

0

140

280

420

560

700

Number of Iterations

Error Components

Initialization

1e-6

-2.5e-11

0.0e+00

2.5e-11

5.0e-11

0

140

280

420

560

700

Number of Iterations

Error Components

Initialization

1e-8

-2.5e-14

0.0e+00

2.5e-14

5.0e-14

7.5e-14

0

140

280

420

560

700

Number of Iterations

Error Components

Initialization

1e-10

In this section, we also introduce other generating schemes as adopted in [32, 15]. Specifically, we generate random data based on the following two models.

Model 1 : x ∼ MN ( 0 p , Σ ) , Σ = ( σ ij ) with nonzero elements σ ij = 0 . 4 | i -j | for 1 ≤ i, j ≤ p , P ( y = 1 | x ) = Φ( x T β ∗ ) , where Φ( · ) is the cumulative density function of the standard normal distribution, β ∗ = (1 . 1 , 1 . 1 , 1 . 1 , 1 . 1 , 0 , . . . , 0) T and s = 4 .

̸

Model 2 : P ( y = 1) = P ( y = -1) = 0 . 5 , x | ( y = 1) ∼ MN ( µ , Σ ) , x | ( y = -1) ∼ MN ( -µ , Σ ) , s = 5 , µ = (0 . 1 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 5 , 0 , . . . , 0) T , Σ = ( σ ij ) with diagonal entries equal to 1, nonzero entries σ ij = -0 . 2 for 1 ≤ i = j ≤ s and other entries equal to 0 . The bayes rule is sign(1 . 39 x 1 +1 . 47 x 2 +1 . 56 x 3 +1 . 65 x 4 +1 . 74 x 5 ) with bayes error 6 . 3% .

We follow the default parameter setup from Section 4, and the experiments are repeated 30 times. The averaged estimation and prediction results are presented in Figure 11. From Figure 11, it's evident that the GD estimator closely approximates the oracle estimator in terms of estimation error and prediction accuracy.

C Gradient Descent Dynamics

First, let's recall the gradient descent updates. We over-parameterize β by writing it as w ⊙ w -v ⊙ v , where w and v are p × 1 vectors. We then apply gradient descent to the following optimization problem:

0.0

0.5

1.0

1.5

2.0

2.5

5.0

7.5

10.0

Strength of Signal

True Negative

Method

(a) Results in Figure 2

0.0

0.5

1.0

1.5

2.0

100

200

300

400

Number of Samples

True Negative

Method

(b) Results in Figure 3

0.00

0.25

0.50

0.75

1.00

A

B

C

D

E

Different Signals

True Negative

Method

(c) Results in Figure 4

0.00

0.25

0.50

0.75

1.00

A

B

C

D

E

Different Signals

True Negative

Method

(d) Results in Figure 5

where d γ ( µ ) = γ 2 ∥ µ ∥ 2 . The gradient descent updates with respect to w and v are given by

For the sake of convenience, let G t ∈ R p represent the gradients in the form of G t = n -1 X T Y µ t , where Y is a diagonal matrix composed of the elements of y . Consequently, the i -th element of G t can be expressed as G t,i = n -1 ∑ n k =1 µ t,k y k x ki , where µ t,k = median (0 , (1 -y k x T k β t ) /γn, 1) . Subsequently, the updating rule can be rephrased as follows:

Furthermore, the error parts of w t and v t are denoted by w t ⊙ 1 S c 1 and v t ⊙ 1 S c 1 , which include both weak signal parts and pure error parts. In addition, strong signal parts of w t and v t are denoted by w t ⊙ 1 S 1 and v t ⊙ 1 S 1 , respectively.

We examine the dynamic changes of error components and strong signal components in two stages. Without loss of generality, we focus on analyzing entries i ∈ S 1 where β ∗ i > 0 . The analysis for the case where β ∗ i < 0 and i ∈ S 1 is similar and therefore not presented here. Specifically, within Stage One , we can ensure that with a high probability, for t ≥ T 0 = 2log( m/α 2 ) / ( ηm ) , the following

0.0

0.2

0.4

A

B

C

D

E

Different Signals

Estimation Error

Method

Oracle

(a) Estimation Result

0.70

0.75

0.80

0.85

0.90

A

B

C

D

E

Different Signals

Prediction Accuracy

Method

Oracle

(b) Prediction Result

0

25

50

75

A

B

C

D

E

Different Signals

False Positive

Method

(c) False Positive Result

0.00

0.25

0.50

0.75

1.00

A

B

C

D

E

Different Signals

True Negative

Method

(d) True Negative Result

results hold under Assumptions 1 and 2:

From (8), we can observe that for t ≥ T 0 , the iterate ( w t , v t ) reaches the desired performance level. Specifically, each component β t,i of the positive strong signal part β t ⊙ 1 S 1 increases exponentially in t until it reaches β ∗ i / 2 . Meanwhile, the weak signal and error part β t ⊙ 1 S c 1 remains bounded by O ( α ) . This observation highlights the significance of small initialization size α for the gradient descent algorithm, as it restricts the error term. A smaller initialization leads to better estimation but with the trade-off of a slower convergence rate.

After each component β t,i of the strong signal reaches β ∗ i / 2 , Stage Two starts. In this stage, β t,i continues to grow at a slower rate and converges to the true parameter β ∗ i . After this time, after 3 log( m/α 2 ) / ( ηm ) iterations, β t,i of the strong signal enters a desired interval, which is within a distance of C ϵ √ log p/n from the true parameter β ∗ i . Simultaneously, the error term β t ⊙ 1 S c 1 remains bounded by O ( α ) until O (log(1 /α ) / ( η log n )) iterations.

We summarize the dynamic analysis results described above in Proposition 1 and Proposition 2. By combining the results from these two propositions, we can obtain the proof of Theorem 2 in the subsequent subsection.

it holds that

Since p is much larger than n , the last term, 2 C 4 α /p , is negligible. Considering the constants C w , C α and C ϵ , we finally obtain the error bound of gradient descent estimator in terms of ℓ 2 norm. Therefore, we conclude the proof of Theorem 2.

E Proof of Propositions 1 and 2

In this section, we provide the proof for the two propositions mentioned in Section 3. First, we introduce some useful lemmas, which are about the coherence of the design matrix X and the upper bound of sub-Gaussian random variables.

4

5

6

7

2.5e-05

5.0e-05

7.5e-05

1.0e-04

2.5e-04

5.0e-04

0.001

Gamma Values

Signal 3

(a) Signal

3

4

5

6

7

2.5e-05

5.0e-05

7.5e-05

1.0e-04

2.5e-04

5.0e-04

0.001

Gamma Values

Signal 4

(b) Signal

4

D Proof of Theorem 2

Proof. We first utilize Proposition 1 to upper bound the error components β t ⊙ 1 S c 1 . By Proposition 1, we are able to control the error parts of the same order as the initialization size α within the time interval 0 ≤ t ≤ T ∗ = log(1 /α ) / (4 ση log n ) . Thus, by direct computation, we have

where ( i ) is based on the relationship between ℓ 2 norm and ℓ ∞ norm and ( ii ) follows form the results in Proposition 1. As for the strong signal parts, by Proposition 2, that with probability at least 1 -c 1 n -1 -c 2 p -1 , we obtain

Finally, combining (9) and (10), with probability at least 1 -c 1 n -1 -c 2 p -1 , for any t that belongs to the interval

0.1

0.2

0.3

0.4

Oracle

method

Estimation Error

method

Oracle

(a) Model 1 : Estimation Error

0.2

0.4

0.6

Oracle

method

Estimation Error (Non-normalized)

method

Oracle

(b) Model 1 : Non-normalized Error

0.800

0.825

0.850

0.875

0.900

Oracle

method

Prediction Accuracy

method

Oracle

(c) Model 1 : Prediction Result

0.1

0.2

0.3

0.4

0.5

Oracle

method

Estimation Error

method

Oracle

0.2

0.4

0.6

Oracle

method

Estimation Error (Non-normalized)

method

Oracle

0.84

0.86

0.88

0.90

0.92

0.94

Oracle

method

Prediction Accuracy

method

Oracle

(d) Model

2

: Estimation Error

(e) Model 2 : Non-normalized Error

(f) Model 2 : Prediction Result

E.1 Technical Lemmas

Lemma 1. (General Hoeffding's inequality) Suppose that θ 1 , · · · , θ m are independent mean- zero sub-Gaussian random variables and µ = ( µ 1 , · · · , µ m ) ∈ R m . Then, for every t > 0 , we have

where d = max i ∥ θ i ∥ ψ 2 and c is an absolute constant.

Proof. General Hoeffding's inequality can be proofed directly by the independence of θ 1 , · · · , θ m and the sub-Gaussian property. The detailed proof is omitted here.

̸

Lemma 2. Suppose that X / √ n is a n × p ℓ 2 -normalized matrix satisfying δ -incoherence; that is 1 /n |⟨ x i ⊙ 1 K , x i ⊙ 1 K ⟩| ≤ δ, i = j for any K ⊆ [ n ] . Then, for s -sparse vector z ∈ R p , we have

where X K denotes the n × p matrix whose i -th column is x i if i ∈ K and 0 p × 1 otherwise.

Proof. The proof of Lemma 2 is similar to Lemma 2 in [23]. According to the condition

̸

we can verify that for any i ∈ [ p ] ,

Therefore,

E.2 Proof of Proposition 1

Proof. Here we prove Proposition 1 by induction hypothesis. It holds that our initializations w 0 = α 1 p × 1 and v 0 = α 1 p × 1 satisfy our conclusion given in Proposition 1. As we initialize w 0 ,i and v 0 ,i for any fixed i ∈ S c 1 with

Proposition 1 holds when t = 0 . In the following, we show that for any t ∗ with 0 ≤ t ∗ ≤ T ∗ = log(1 /α ) / (4 ση log n ) , if the conclusion of Proposition 1 stands for all t with 0 ≤ t < t ∗ , then it also stands at the ( t ∗ +1) -th step. From the gradient descent updating rule given in (7), the updates of the weak signals and errors w t,i , v t,i for any fixed i ∈ S c 1 are obtained as follows,

Recall that | µ t,k | ≤ 1 for k = 1 , . . . , n , for any fixed i ∈ S c 1 , then we have

For ease of notation, we denote M i = n -1 ∑ n k =1 | x ki | and M = max i ∈ S c 1 M i . Then we can easily bound the weak signals and errors as follows,

By General Hoeffding's inequality of sub-Gaussian random variables, with probability at least 1 -cn -n , we obtain M i ≤ σ √ log n, i ∈ S c 1 , where c is a constant. Since p is much larger than n , the inequality (1 -cn -n ) p > 1 -cn -1 holds. Then, with probability at least 1 -cn -1 , where c is a constant, we have M ≤ σ √ log n .

Combined (11) and the bound of M , we then have

where ( i ) and ( ii ) follow from t ∗ +1 ≤ T ∗ and log(1 + x ) < x when x > 0 , respectively. Moreover, ( iii ) is based on the definition of T ∗ . Similarly, we can prove that ∥ v t ∗ ⊙ 1 S c 1 ∥ ∞ ≤ √ α . Thus, the induction hypothesis also holds for t ∗ +1 . Since t ∗ < T ∗ is arbitrarily chosen, we complete the proof of Proposition 1.

E.3 Proof of Proposition 2

Proof. In order to analyze strong signals: β t,i = w 2 t,i -v 2 t,i for any fixed i ∈ S 1 , we focus on the dynamics w 2 t,i and v 2 t,i . Following the updating rule (7), we have

Without loss of generality, here we just analyze entries i ∈ S 1 with β ∗ i > 0 . Analysis for the negative case with β ∗ i < 0 , i ∈ S 1 is almost the same and is thus omitted. We divide our analysis into two stages, as discussed in Appendix C.

Specifically, in Stage One , since we choose the initialization as w 0 = v 0 = α 1 p × 1 , we obtain β 0 ,i = 0 for any fixed i ∈ S 1 , we show after 2 log( β ∗ i /α 2 ) /ηβ ∗ i iterations, the gradient descent coordinate β t,i will exceed β ∗ i / 2 . Therefore, all components of strong signal part will exceed half of the true parameters after 2 log( m/α 2 ) / ( ηm ) iterations. Furthermore, we calculate the number of iterations required to achieve β t,i ≥ β ∗ i -ϵ , where ϵ = C ϵ √ log p/n , in Stage Two . Thus, we conclude the proof of Proposition 2.

Stage One. First, we introduce some new notations. In t -th iteration, according to the values of µ t,i , we divide the set [ n ] into three parts, K t, 1 = { i ∈ [ n ] : µ t,i = 1 } , K t, 2 = { i ∈ [ n ] : 0 < µ t,i < 1 } and K t, 3 = { i ∈ [ n ] : µ t,i = 0 } , with cardinalities of k t, 1 , k t, 2 and k t, 3 , respectively. From [28, 45], most elements of µ t are 0 or 1 and the proportion of these elements will rapidly increase with the decreasing of γ , which means that k t, 2 ≪ ( k t, 1 + k t, 3 ) controlling γ in a small level.

According to the updating formula of w 2 t +1 ,i in (12), we define the following form of element-wise bound ξ t,i for any fixed i ∈ S 1 as

Rewriting (13), we can easily get that

From (14), it is obvious that if the magnitude of ξ t,i is sufficiently small, we can easily conclude that w 2 t +1 ,i ≥ w 2 t,i . Therefore, our goal is to evaluate the magnitude of ξ t,i in (13) for any fixed i ∈ S 1 . First, we focus on the term G t,i +( β t,i -β ∗ i ) / ( γn ) in (13). In particular, recall the definition of G t,i and we expand G t,i +( β t,i -β ∗ i ) / ( γn ) in the following form

For the term ( I 1 ) , by the condition k t, 2 ≪ k t, 1 and General Hoeffding's inequality, we have

holds with probability at least 1 -cp -1 , where c is a constant.

For the term ( I 2 ) , let X t, 2 ∈ R n × p denote the matrix whose k -th column is x k if k ∈ K t, 2 . Then, we approximate ( I 2 ) based on the assumption on the design matrix X via Lemma 2,

By (15), (16), condition of the minimal strength m ≥ σ log p √ log p/n and condition δ ≲ 1 / ( κs log p ) , we have

Then, we can bound G t,i through

Note that for any fixed i ∈ S 1 , G t,i ≤ n -1 ∑ n k =1 | x ki | . By General Hoeffding's inequality, we have with probability at least 1 -c 1 n -1 , | G t,i | ≲ log n and then | ηG t,i | < 1 based on the assumption of η . Therefore, w 2 t,i /w 2 t +1 ,i is always bounded by some constant greater than 1 .

Recalling the definition of element-wise bound ξ t,i , and combining (17) and (18), we have that

.

where we use the bound of w 2 t,i /w 2 t +1 ,i for any fixed i ∈ S 1 in step ( i ) and step ( ii ) is based on ηκm ≤ m/ log p and κm ≲ 1 . Thus, we conclude that ξ t,i is sufficiently small for any fixed i ∈ S 1 Now, for any fixed i ∈ S 1 , when 0 ≤ β t,i ≤ β ∗ i / 2 , we can get the increment of w 2 t,i for any fixed i ∈ S 1 according to (19),

where ( i ) follows from 0 ≤ β t,i ≤ β ∗ i / 2 and ( ii ) holds since m/ log p ≲ β ∗ i . Similarly, we can analyze v 2 t,i to get that when 0 ≤ β t,i ≤ β ∗ i / 2 ,

Therefore, w 2 t,i increases at an exponential rate while v 2 t,i decreases at an exponential rate. Stage One ends when β t,i exceeds β ∗ i / 2 , and our goal is to estimate t i, 0 that satisfies

Note that { v 2 t,i } t ≥ 0 forms a decreasing sequence and thus is bounded by α 2 . Hence, it suffices to solve the following inequality for t i, 0 ,

which is equivalent to obtain t i, 0 satisfying

For T i, 0 , by direct calculation, we have

where ( i ) follows from x log x -x +1 ≥ 0 when x ≥ 0 and ( ii ) holds due to log( x/ 2 + 1) ≤ log x when x ≥ 2 as well as the assumption on γ and ηβ ∗ i ≤ 1 . Thus, we set t i, 0 = 2log( β ∗ i /α 2 ) / ( ηβ ∗ i ) such that for all t ≥ t i, 0 , β t,i ≥ β ∗ i / 2 for all i ∈ S 1 .

Stage Two. Define ϵ = C ϵ √ log p/n and a i, 1 = ⌈ log 2 ( β ∗ i /ϵ ) ⌉ . Next, we refine the element-wise bound ξ t,i according to (13). For any fixed i ∈ S 1 , if there exists some a such that 1 ≤ a ≤ a i, 1 and (1 -1 / 2 a ) β ∗ i ≤ βt, i ≤ (1 -1 / 2 a +1 ) β ∗ i , then, based on the analytical thinking in Stage One , we can easily deduce that

Using this element-wise bound (20), we get the increment of w 2 t +1 ,i and decrement of v 2 t +1 ,i ,

We define t i,a as the smallest t such that β t + t i,a ,i ≥ (1 -1 / 2 a +1 ) β ∗ i . Intuitively, suppose that the current estimate β t,i is between (1 -1 / 2 a ) β ∗ i and (1 -1 / 2 a +1 ) β ∗ i , we aim to find the number of iterations required for the sequence { β t,i } t ≥ 0 to exceed (1 -1 / 2 a +1 ) β ∗ i . To obtain a sufficient condition for t i,a , we construct the following inequality,

Similar with Stage One , the sequence { v 2 t,i } t ≥ 0 is bounded by α 2 . Then, it is sufficient to solve the following inequality for t i,a ,

To obtain a more precise upper bound of T i,a , we have

where ( i ) is based on the condition w 2 t,i ≥ (1 -1 / 2 a ) β ∗ i , ( ii ) stands since x log x -x +1 ≥ 0 when x ≥ 0 . By direct calculation, we obtain

where we use log( x +1) ≤ x when x ≥ 0 in step ( iii ) and 1 -1 / 2 a ≥ 1 / 2 in step ( iv ) , respectively. Recall the assumption a ≤ a i, 1 = ⌈ log 2 ( β ∗ i /ϵ ) ⌉ with ϵ = C ϵ √ log p/n , then we get 2 a +2 ≤

4 β ∗ i /ϵ ≤ 4 √ n/ log pβ ∗ i /C ϵ . Moreover, by the assumption on the initialization size α , we have α 2 ≤ C α /p 2 . Thus, we can bound 2 a +2 α 2 /ηβ ∗ 2 i in (21) as

where ( i ) holds when p 2 log p ≥ 4 C α √ n/C ϵ . Combined (21) and (22), we calculate the final bound of T i,a as

for any a ≤ a i, 1 . If there exists an 1 ≤ a ≤ a i, 1 such that β t,i ∈ [(1 -1 / 2 a ) β ∗ i , (1 -1 / 2 a +1 ) β ∗ i ] , after t i,a ≥ 3 /ηβ ∗ i iterations, we can guarantee that β t,i ≥ (1 -1 / 2 a +1 ) β ∗ i . Now recalling the definition of a i, 1 , we have β ∗ i / 2 a i, 1 ≤ ϵ = C ϵ √ log p/n . Therefore, with at most ∑ a i, 1 a =0 T i,a iterations, we have β t,i ≥ β ∗ i -ϵ . By the assumption of α , we obtain

Since the function log( x/α 2 ) / ( ηx ) is decreasing with respect to x , we have after 5 log( m/α 2 ) / ( ηm ) iterations that | β t,i -β ∗ i | ≲ √ log p/n for any fixed i ∈ S 1 . Thus, we complete the proof of Proposition 2.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA regularization-free algorithm for high-dimensional support vector machines (SVMs) is designed by integrating over-parameterization with Nesterov's smoothing method, which induces implicit regularization. An over-parameterized hinge loss function is constructed and true parameters are estimated by leveraging regularization-free gradient descent on it. Nesterov's method enhances the computational efficiency of the algorithm, particularly in terms of determining the stopping criterion and reducing computational complexity. With appropriate choices of initialization, stepsize, and prox-parameter, unregularized gradient descent achieves a near-oracle statistical convergence rate. The theoretical findings are verified through a variety of numerical experiments and the proposed method is compared with explicit  $ l\\_1 $ regularization. The advantages of employing implicit regularization via gradient descent subsequent to over-parameterization in sparse SVM are illustrated by the results.\n\nSTRENGTHS:\nStrenth 1: This manuscript presents an interesting finding about the over-parameterized SVM by showing that if we re-parameterize the sparse unknown parameter $\\beta$ by $w\\odot w-v\\odot v$, then Nesterov's smoothing based alternative update of variables leads to the effect of implicit regularization. That is to say, even we did not explicitly impose $l\\_1$-norm onto the underlying parameter, the algorithm behaves as if the $l\\_1$-norm is implicitly added.\n\nStrength 2: The theoretical analysis is interesting and easy to follow. \n\nStrength 3: The theoretical findings are validated with numerical simulations.\n\nWEAKNESSES:\nWeakness 1: Although it is an interesting result for over-parameterized SVM, the proposed algorithm is not directly designed for the original model (2), but its modifed version in Sec. 2.3. It seems that the proposed Algorithm 1 might have some explicit regulrization term in its single iteration. Specifically, the update of $\\mu$ is due to Eq. (4) which introduces an extra smoothing term $d\\_\\gamma(\\mu)=\\gamma/2\\|\\mu\\|_2^2$ and this $\\mu$-update surely affects $w$ and $v$, which may result an regulurization effect like $l\\_1$.  \n\nWeakness 2: The Assumption 1 requires the design matrix to satisfy the $\\delta$-incoherence and the sub-Gaussianality. It seems this assumption is relatively strict for real datasets. \n\nWeakness 3: The empirical effectiveness of the proposed algorithm is insufficiently discussed.\n\nQUESTIONS:\nQuestion 1: Since the update of $\\mu$ in Algorithm 1 is due to Eq. (4) which introduces an extra smoothing term $d\\_\\gamma(\\mu)=\\gamma/2\\|\\mu\\|_2^2$ and this $\\mu$-update surely affects $w$ and $v$.  Is it possible that this additional smoothing term intruduces some explicit regularization effects?\n\nQuestion/Suggestion 2: As  Assumption 1 requires the design matrix to satisfy the $\\delta$-incoherence and the sub-Gaussianality, it is suggested to discuss whether this assumption holds on real data. \n\nQuestion 3: Is it possible to discuss the empirical effectiveness of the proposed algorithm?\n\nLIMITATIONS:\nIt seems the authors did not sufficiently address the limitation due to the lack of explicit discussion in the main text.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes an iterative/implicit regularization algorithm for sparse SVM, using an Hadamard product overparametrization of the iterate $\\beta = u \\odot u - v \\odot v$. In addition, a Nesterov smoothing of the Hinge loss (replaced by its Moreau envelope, a term that is lacking in the paper) is performed.\n\nThe main result is Theorem 2, stating that results equivalent to that of explicit penalization approach can be obtained, under similar assumptions (subgaussianity of the samples, low incoherence of the design matrix), in terms of oracle error with high probability.\n\nSTRENGTHS:\nImplicit regularization is a very active area of research; implicitly regularized algorithms usually require a lower computational budget. Extending known results from regression and sparse matrix factorization to classification is interesting.\n\nWEAKNESSES:\n- no code is included in the supplementary material, and no code release is mentioned in the paper.\n- the writing of the paper can be strongly improved; it would benefit from being proofread by a proficient English reader.\n    - E.g. in the first two lines, do not use \"the\" in \"based on the gradient-based methods\", and use plural in \"models, such as the deep learning model\" (there are more than one DL model...)\n    - L20, \"tend to converge to the global minimum.\" : a global minimum\n    - \"$s$ is number of signals\"\n    -  near-oracle rate is achievable via explicit regularization using explicit regularization\n    -  show that gradient descent estimator: missing \"the\"\n    - L110 why \"that\"?\n    - L120 repeats the same idea twice\n    - proxy parameter instead of prox parameter, and it would be clearer to call it \"smoothing paramter\" or Moreau envelope paramter.\n    - etc etc.\n- the paper also lacks rigor and clarity in its writing in many places, e.g. \"the zero component is initialized close to zero\". What the authors mean is that the coordinates which are zero **in $\\beta^*$** are initialized closed to 0 **in the optimization variable**. $\\beta^*$ is not even defined.\n- the algorithm's stopping criterion is based on $\\mu_i$, and the algorithm stops\nwhen all $\\mu_i$'s are negative. This means that all training points are classified in their observed classes. In the case of label noise, doesn't this lead to overfitting? How is this stopping criterion related to the bound on number of iterations $t$ in Theorem 1 and Proposition 2?\n- why is the performance measure not $\\Vert \\beta - \\beta^* \\Vert$ in the experiments? Why are the vectors normalized? This does not match the metric used in the theoretical results.\n- the approach simply drops the two L2 regularization terms, $\\Vert w \\Vert^2 + \\Vert v \\Vert^2$, that come from the original L1 regularization term $\\Vert \\beta \\Vert_1$. Why?\n- the authors claim that there is no parameter in their approach, but the smoothing of the hinge loss does require one.\n- why is $d_\\gamma$ called a prox-function, when it's just a strongly convex function? why introducing a general notation when only the squared L2 norm is used (l142)\n- why are there 3 orange curves in Fig 1c? shouldn't those be shaded plots? Where is the rud curve? can you sort the legend by increasing value instead of having 1e-10, 1e-4, 1e-6 in that order?\n- How is Lasso's (L1 regularized SVM) regularization parameter tuned?\n- Unless I'm mistaken, thee iterates of the proposed method are never exactly zero. Some thresholding needs to be performed in order to avoid dense vectors and many false positive (Fig 2c for example). How is this done?\n\n\n- \"we present finite sample performances of our method in comparisons with the Lasso estimator\": SVM is for classification, Lasso is for regression. How can you compare them? Do you mean L1 regularized SVM?\n- under the assumption of Sub-Gaussian distribution: mention that this is about the noise\n- L114 the fact that there is a generative model with \"true parameters\" is completely missing\n- L139 it's a saddle point problem not a saddle point function\n- In Thm 1 there is no need to mention randomness, as the generative model does not come into play. Making this a theorem is a bold move.\n- the legends in the plots are too small to be read.\n\n### References\n- The proposed Hadamard parametrization is, up to my knowledge, proposed by Vaskevicius. This could be explicitely stated L43 (though it is currently mentioned, it comes later in the paper)\n- A generic algorithm for any type of sparse regularization is proposed in \"Iterative regularization for low complexity regularizers\" (Molinari et al, 2022). In particular, handling of sparse classification is proposed in example 1. How does the proposed approach compare?\n- the authors should cite \"Smoothing and first order methods: a unified framework\" by Beck and Teboulle (2012). Their thm 1 is a known result of this paper, which holds for \"inf-convolution\"/Nesterov smoothing regardless of the function being smoothed.\n\nQUESTIONS:\nsee weaknesses above\n\nLIMITATIONS:\nno societal impact", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper design a regularization-free algorithm for high-dimensional support vector machines (SVMs) by integrating over-parameterization with Nesterov's smoothing method, and provide theoretical guarantees for the induced implicit regularization phenomenon.

STRENGTHS:
This paper provides a regularization-free gradient method that has proven to be effective in practice and is supported by some theoretical guarantees. The theoretical results are novel, and the experiments are comprehensive.

WEAKNESSES:
The theoretical results are relatively weak.

1. The constants $c1\sim c{4}$ in Theorem 2 are not clearly specified, so we cannot see the trend of these constants changing with the problem size (are they independent of $p$?), which weakens the results. 

2. The author did not compare the strength of the $\delta$-incoherence assumption and the RIP (Restricted Isometry Property) assumption (only a comparison for verifying difficulty was made). This comparison should be included.

3.The restrictions on the initial values in Assumption 2 may seem somewhat stringent. The author may provide an explanation for why they are designed this way.

QUESTIONS:
Please refer to the Weakness.

LIMITATIONS:
No
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-07-06

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles the problem of implicit regularization for classification in the context of over-parameterization. Starting from a $L^1$ regularized SVM, they voluntarily over-parameterize the feature vector $\\beta = w \\odot w - v \\odot v $ using two vectors $w, v \\in \\mathbb{R}^p$.\n\nThe optimization process then consists in (i) dropping the explicit regularization, (ii) smoothing the hinge loss function, (iii) applying gradient descent with early stopping to get the solution.\n\nA theoretical analysis of the estimated parameter is provided, showing under mild conditions that the proposed scheme behaves as well as the explicit regularization scheme if the algorithm is stopped at a certain time.\n\nNumerical experiments highlight the benefits of the approach on synthetic data.\n\nSTRENGTHS:\n- The topic is of interest for the machine learning community.\n- The paper is well written and mathematically sound.\n- The developed method is novel.\n- The experimental study is convincing.\n\nWEAKNESSES:\n- No real weakness\n\nQUESTIONS:\n- The problem considered here is a linear SVM with $L^1$ penalization. Would the results presented here still hold for a non-linear SVM based on a finite-dimensional feature map ?\n\n- At one point, you drop the explicit regularization to only minimize an empirical risk. What is the ingredient in the algorithm that allows to recover properties similar to the explicit regularization ? Once the regularization is dropped, we could say that the original problem considered another one - e.g. Tychonov regularization - but no the solution presented here retains specific properties usually associated with the $L^1$ penalty. Could you comment on that ?\n\nTypos/remarks:\n\n- Line 18: on gradient based methods\n- I believe it would be of interest to cite [A] alongside [1] and [27] in the introduction, given the link they explore with the Lasso problem.\n- Line 44: Why $u_0$ and not $w_0$ ? Please be consistent with the notation from line 42\n- Line 47: \"near-oracle rate is achievable via explicit regularization using explicit regularization\"\n- Line 145: \"and has an explicit form that\" writes ?\n- Line 146: given that you already use a superscript in equation (4) for $\\mathcal{Z}^n$, you should use a subscript here to avoid overcharge.\n- Line 229: \"that scales as $\\mathcal{O}$\"\n- Line 236: the subscripts are not correct (two times the complement)\n\n[A]: Iterative regularization for convex regularizers; Molinari, Cesare and Massias, Mathurin and Rosasco, Lorenzo and Villa, Silvia; AISTATS 2021\n\nLIMITATIONS:\nLimitations have overall been addressed.", "TITLE": "Official Review", "RECOMMENDATION": 8, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the implicit regularization in over-parameterized (sparse) support vector machine. The paper by nature is an extension of Vaskevicius et al [28], applying the quadratic reparametrization on SVM (hinge loss). Due to the non-differentiability of hinge loss, Nesterov’s smoothing is applied to enhance the computational efficiency. The authors then propose a regularization free gradient descent algorithm and show that the sparse solution is obtained with early stopping. The convergence result is demonstrated via numerical simulations. This is an active area of research, and the contribution is relevant. My concern is more about the setting/significance of this work. The authors claim that “this is the first study that investigates implicit regularization via gradient descent and establishes the near-oracle rate specifically in classification” (line 49). First of all, the implicit regularization via gradient descent in classification has been studied in [12, 27] for logistic/exponential losses. Therefore, the contribution to hinge-loss is appreciated (with the remedy of Nesterov's smoothing). Secondly, when it comes to classification problems, sparsity is relatively minor and the implicit regularization effect is shown via max-margin/separability. More discussion would be helpful about the sparse SVM. \n\nThese two papers [W]&[Z] below may be closely related to the implicit regularization effect studied in this paper but missed in the related works.\n\n[W] B. Woodworth et al, Kernel and Rich Regimes in Overparameterized Models, COLT, 2020. \n\n[Z] P. Zhao et al, High-Dimensional Linear Regression via Implicit Regularization, Biometrika, 2022.\n\nSTRENGTHS:\n1. The implicit regularization under hinge-loss is novel.\n2. Some theoretical results in this paper are interesting.\n\nWEAKNESSES:\n1. The importance of sparsity in SVM and how that affects the classification performance is not discussed.\n2. Some metrics used in simulations are not defined.\n\nQUESTIONS:\n1. I didn’t find the definition of either $\\beta^\\star$ or the data generating mechanism. This may cause a big concern if the authors implicitly assume $y=X\\beta^\\star$ somewhere, which would be very restrictive. In line 258, the simulation follows a logistic model. Is that also the case for theorem 2? If $\\beta^\\star$ is a minimizer of the population version of the hinge loss, is that for Eq. (1) or Eq. (3)?\n2. Formatting issue in [11]\n3. Is there a reason that the oracle method performs the worst in Figure 2 (b)?\n4. Is Nesterov’s smoothing necessary here? How about plain sub-gradient descent with quadratic parameterization?\n5. For Proposition 2, isn’t the upper bound of t in Proposition 1 still needed? I didn’t check the proof details, but intuitively the convergence on the support is only guaranteed while the error accumulation outside the support remains small. Correct me if I’m wrong.\n\nLIMITATIONS:\nThe significance/importance/potential application of this work is not well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-25"}