SlowGuess/NeurIPS_2023_OCR huggingface dataset:
We highlight three key advantages of Algorithm 1. First, the stopping condition for Algorithm 1 can be determined based on the value of  $\mu$  in addition to the preset maximum iteration number  $T_{1}$ . Specifically, when the values of  $\mu_{i}$  are 0 across all samples, the algorithm naturally stops as no further updates are made. Thus,  $\mu$  serves as an intrinsic indicator for convergence, providing a more efficient stopping condition. Second, Algorithm 1 avoids heavy computational cost like the computation of the Hessian matrix. The main computational load comes from the vector multiplication in (6). Since a considerable portion of the elements in  $\mu$  are either 0 or 1, and the proportion of these elements increases substantially as  $\gamma$  decreases, the computation in (6) can be further simplified. Lastly, the utilization of Nesterov's smoothing not only optimizes our approach but also aids in our theoretical derivations, as detailed in Appendix E.
In this section, we analyze the theoretical properties of Algorithm 1. The main result is the error bound  $\| \beta_t - \beta^* \|$ , where  $\beta^*$  is the minimizer of the population hinge loss function for  $\beta$  without the  $\ell_1$  norm:  $\beta^* = \arg \min_{\boldsymbol{\beta} \in \mathbb{R}^p} \mathbb{E}(1 - y\mathbf{x}^T\boldsymbol{\beta})_+$ . We start by defining the  $\delta$ -incoherence, a key assumption for our analysis.
Definition 1. Let  $\mathbf{X} \in \mathbb{R}^{n \times p}$  be a matrix with  $\ell_2$ -normalized columns  $\mathbf{x}_1, \ldots, \mathbf{x}_p$ , i.e.,  $\|\mathbf{x}_i\| = 1$  for all  $i \in [n]$ . The coherence  $\delta = \delta(\mathbf{X})$  of the matrix  $\mathbf{X}$  is defined as
$$
\delta := \max _ {K \subseteq [ n ], 1 \leq i \neq j \leq p} | \langle \mathbf {x} _ {i} \odot \mathbf {1} _ {K}, \mathbf {x} _ {j} \odot \mathbf {1} _ {K} \rangle |,
$$
where  $\mathbf{1}_K$  denotes the  $n$ -dimensional vector whose  $i$ -th entry is 1 if  $i \in K$  and 0 otherwise. Then, the matrix  $\mathbf{X}$  is said to be satisfying  $\delta$ -incoherence.
Coherence measures the suitability of measurement matrices in compressed sensing [12]. Several techniques exist for constructing matrices with low coherence. One such approach involves utilizing sub-Gaussian matrices that satisfy the low-incoherence property with high probability [9, 7]. The Restricted Isometry Property (RIP) is another key measure for ensuring reliable sparse recovery in various applications [35, 44], but verifying RIP for a designed matrix is NP-hard, making it computationally challenging [5]. In contrast, coherence offers a computationally feasible metric for sparse regression [9, 23]. Hence, the assumptions required in our main theorems can be verified within polynomial time, distinguishing them from the RIP assumption.
Recall that  $\beta^{*} \in \mathbb{R}^{p}$  is the  $s$ -sparse signal to be recovered. Let  $S \subset \{1, \dots, p\}$  denote the index set that corresponds to the nonzero components of  $\beta^{*}$ , and the size  $|S|$  of  $S$  is given by  $s$ . Among the  $s$  nonzero signal components of  $\beta^{*}$ , we define the index set of strong signals as  $S_{1} = \{i \in S : |\beta_{i}^{*}| \geq C_{s} \log p \sqrt{\log p / n}\}$  and of weak signals as  $S_{2} = \{i \in S : |\beta_{i}^{*}| \leq C_{w} \sqrt{\log p / n}\}$  for some constants  $C_{s}, C_{w} > 0$ . We denote  $s_{1}$  and  $s_{2}$  as the cardinalities of  $S_{1}$  and  $S_{2}$ , respectively. Furthermore, we use  $m = \min_{i \in S_{1}} |\beta_{i}^{*}|$  to represent the minimal strength for strong signals and  $\kappa$  to represent the condition number-the ratio of the largest absolute value of strong signals to the smallest. In this paper, we focus on the case that each nonzero signal in  $\beta^{*}$  is either strong or weak, which means that  $s = s_{1} + s_{2}$ . Regarding the input data and parameters in Algorithm 1, we introduce the following two structural assumptions.
Assumption 1. The design matrix  $\mathbf{X} / \sqrt{n}$  satisfies  $\delta$ -incoherence with  $0 < \delta \lesssim 1 / (\kappa s \log p)$ . In addition, every entry  $x$  of  $\mathbf{X}$  is i.i.d. zero-mean sub-Gaussian random variable with bounded sub-Gaussian norm  $\sigma$ .
Assumption 2. The initialization for gradient descent are  $\mathbf{w}_0 = \alpha \mathbf{1}_{p\times 1}$ ,  $\mathbf{v}_0 = \alpha \mathbf{1}_{p\times 1}$  where the initialization size  $\alpha$  satisfies  $0 < \alpha \lesssim 1/p$ , the parameter of prox-function  $\gamma$  satisfies  $0 < \gamma \leq 1/n$ , and the step size  $\eta$  satisfies  $0 < \eta \lesssim 1 / (\kappa \log p)$ .
Assumption 1 characterizes the distribution of the input data, which can be easily satisfied across a wide range of distributions. Interestingly, although our proof relies on Assumption 1, numerical results provide compelling evidence that it isn't essential for the success of our method. This indicates that the constraints set by Assumption 1 can be relaxed in practical applications, as discussed in Section 4. The assumptions about the initialization size  $\alpha$ , the smoothness parameter  $\gamma$ , and the step size  $\eta$  primarily stem from the theoretical induction of Algorithm 1. For instance,  $\alpha$  controls the strength of the estimated weak signals and error components,  $\gamma$  manages the approximation error in smoothing, and  $\eta$  affects the accuracy of the estimation of strong signals. Our numerical simulations indicate that extremely small initialization size  $\alpha$ , step size  $\eta$ , and smoothness parameter  $\gamma$  are not required to achieve the desired convergence results, highlighting the low computational burden of our method, with details found in Section 4. The primary theoretical result is summarized in the subsequent theorem.
Theorem 2. Suppose that Assumptions 1 and 2 hold, then there exist positive constants  $c_{1}, c_{2}, c_{3}$  and  $c_{4}$  such that there holds with probability at least  $1 - c_{1}n^{-1} - c_{2}p^{-1}$  that, for every time  $t$  with  $c_{3}\log (m / \alpha^{2}) / (\eta m)\leq t\leq c_{4}\log (1 / \alpha) / (\eta \log n)$ , the solution of the  $t$ -th iteration in Algorithm 1,  $\beta_{t} = \mathbf{w}_{t}\odot \mathbf{w}_{t} - \mathbf{v}_{t}\odot \mathbf{v}_{t}$ , satisfies
$$
\left\| \boldsymbol {\beta} _ {t} - \boldsymbol {\beta} ^ {*} \right\| ^ {2} \lesssim \frac {s \log p}{n}.
$$
Theorem 2 demonstrates that if  $\beta^{*}$  contains  $s$  nonzero signals, then with high probability, for any  $t\in [c_3\log (m / \alpha^2) / (\eta m),c_4\log (1 / \alpha) / (\eta \log n)]$ , the convergence rate of  $\mathcal{O}(\sqrt{s\log p / n})$  in terms of the  $\ell_{2}$  norm can be achieved. Such a convergence rate matches the near-oracle rate of sparse SVMs and can be attained through explicit regularization using the  $\ell_{1}$  norm penalty [32, 46], as well as through concave penalties [20]. Therefore, Theorem 2 indicates that with over-parameterization, the implicit regularization of gradient descent achieves the same effect as imposing an explicit regularization into the objective function in (1).
Proof Sketch. The ideas behind the proof of Theorem 2 are as follows. First, we can control the estimated strengths associated with the non-signal and weak signal components, denoted as  $\| \mathbf{w}_t\odot \mathbf{1}_{S_1^c}\|_\infty$  and  $\| \mathbf{v}_t\odot \mathbf{1}_{S_1^c}\|_\infty$ , to the order of the square root of the initialization size  $\alpha$  for up to  $\mathcal{O}(\log (1 / \alpha) / (\eta \log n))$  steps. This provides an upper boundary on the stopping time. Also, the magnitude of  $\alpha$  determines the size of coordinates outside the signal support  $S_{1}$  at the stopping time. The importance of choosing small initialization sizes and their role in achieving the desired statistical performance are further discussed in Section 4. On the other hand, each entry of the strong signal part, denoted as  $\beta_{t}\odot \mathbf{1}_{S_{1}}$ , increases exponentially with an accuracy of around  $\mathcal{O}(\log p / n)$  near the true parameter  $\beta^{*}\odot \mathbf{1}_{S_{1}}$  within roughly  $\mathcal{O}(\log (m / \alpha^2) / (\eta m))$  steps. This establishes the left boundary of the stopping time. The following two Propositions summarize these results.
Proposition 1. (Analyzing Weak Signals and Errors) Under Assumptions 1-2, with probability at least  $1 - cn^{-1}$ , we have
$$
\left\| \mathbf {w} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} \right\| _ {\infty} \leq \sqrt {\alpha} \lesssim \frac {1}{\sqrt {p}} \quad a n d \quad \left\| \mathbf {v} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} \right\| _ {\infty} \leq \sqrt {\alpha} \lesssim \frac {1}{\sqrt {p}},
$$
for all  $t \leq T^{*} = \mathcal{O}(\log(1 / \alpha) / (\eta \log n))$ , where  $c$  is some positive constant.
Proposition 2. (Analyzing Strong Signals) Under Assumptions 1-2, with probability at least  $1 - c_{1}n^{-1} - c_{2}p^{-1}$ , we have
$$
\left\| \boldsymbol {\beta} _ {t} \odot \mathbf {1} _ {S _ {1}} - \boldsymbol {\beta} ^ {*} \odot \mathbf {1} _ {S _ {1}} \right\| _ {\infty} \lesssim \sqrt {\frac {\log p}{n}},
$$
holds for all  $\mathcal{O}(\log (m / \alpha^2) / (\eta m))\leq t\leq \mathcal{O}(\log (1 / \alpha) / (\eta \log n))$  where  $c_{1},c_{2}$  are two constants.
Consequently, by appropriately selecting the stopping time  $t$  within the interval specified in Theorem 2, we can ensure convergence of the signal components and effectively control the error components. The final convergence rate can be obtained by combining the results from Proposition 1 and Proposition 2.
4 Numerical Study
In our simulations, unless otherwise specified, we follow a default setup. We generate  $3n$  independent observations, divided equally for training, validation, and testing. The true parameters  $\beta^{*}$  is set to  $m\mathbf{1}_S$  with a constant  $m$ . Each entry of  $\mathbf{x}$  is sampled as i.i.d. zero-mean Gaussian random variable, and the labels  $y$  are determined by a binomial distribution with probability  $p = 1 / (1 + \exp (\mathbf{x}^T\boldsymbol{\beta}^*))$ . Default parameters are: true signal strength  $m = 10$ , number of signals  $s = 4$ , sample size  $n = 200$ , dimension  $p = 400$ , step size  $\eta = 0.5$ , smoothness parameter  $\gamma = 10^{-4}$ , and initialization size  $\alpha = 10^{-8}$ . For evaluation, we measure the estimation error using  $\| \beta_t / \| \beta_t\| -\beta^* /\| \beta^*\| \|$  (for comparison with oracle estimator) and the prediction accuracy on the testing set with  $P(\hat{y} = y_{test})$ . Additionally, "False positive" and "True negative" metrics represent variable selection errors. Specifically, "False positive" means the true value is zero but detected as a signal, while "True negative" signifies a non-zero true value that isn't detected. Results are primarily visualized employing shaded plots and boxplots, where the solid line depicts the median of 30 runs and the shaded area marks the 25-th and 75-th percentiles over these runs.
Effects of Small Initialization Size. We investigate the power of small initialization size  $\alpha$  on the performance of our algorithm. We set the initialization size  $\alpha = \{10^{-4}, 10^{-6}, 10^{-8}, 10^{-10}\}$ , and other parameters are set by default. Figure 1 shows the importance of small initialization size in inducing exponential paths for the coordinates. Our simulations reveal that small initialization size leads to lower estimation errors and more precise signal recovery, while effectively constraining the error term to a negligible magnitude. Remarkably, although small initialization size might slow the convergence rate slightly, this trade-off is acceptable given the enhanced estimation accuracy.
(a) Effects of Initialization
(b) Paths of Signals
(c) Paths of Errors
Figure 1: Effects of small initialization size  $\alpha$ . In Figure 1(a), the estimation error is calculated by  $\| \beta_t - \beta^* \| / \| \beta^* \|$ .
Effects of Signal Strength and Sample Size. We examine the influence of signal strength on the estimation accuracy of our algorithm. We set the true signal strength  $m = 0.5 * k, k = 1, \ldots, 20$  and keep other parameters at their default values. As depicted in Figure 2, we compare our method (denoted by GD) with  $\ell_1$ -regularized SVM (denoted by Lasso method), and obtain the oracle solution (denoted by Oracle) using the true support information. We assess the advantages of our algorithm from three aspects. Firstly, in terms of estimation error, our method consistently outperforms the Lasso method across different signal strengths, approaching near-oracle performance. Secondly, all three methods achieve high prediction accuracy on the testing set. Lastly, when comparing variable selection performance, our method significantly surpasses the Lasso method in terms of false positive error. Since the true negative error of both methods is basically 0, we only present results for false positive error in Figure 2.
(a) Estimation Result
Figure 2: Effects of signal strength  $m$ . The orange vertical line in Figure 2(a) show the threshold of strong signal  $\log p\sqrt{\log p / n}$ .
(b) Prediction Result
(c) Variable Selection Result
We further analyze the impact of sample size  $n$  on our proposed algorithm. Keeping the true signal strength fixed at  $m = 5$ , we vary the sample size as  $n = 50 * k$  for  $k = 1, \ldots, 8$ . Other parameters remain at their default values. Consistently, our method outperforms the Lasso method in estimation, prediction, and variable selection, see Figure 3 for a summary of the results.
Performance on Complex Signal Structure. To examine the performance of our method under more complex signal structures, we select five signal structures:  $\mathbf{A} - (5,6,7,8)$ ,  $\mathbf{B} - (4,6,8,9)$ ,  $\mathbf{C} - (3,6,9,10)$ ,  $\mathbf{D} - (2,6,10,11)$ , and  $\mathbf{E} - (1,6,11,12)$ . Other parameters are set by default. The
(a) Estimation Result
(c) Variable Selection Result
Figure 3: Effects of sample size  $n$ .
results, summarized in Figure 4, highlight the consistent superiority of our method over the Lasso method in terms of prediction and variable selection performance, even approaching an oracle-like performance for complex signal structures. High prediction accuracy is achieved by the both methods.
(a) Estimation Result
Figure 4: Performance on complex signal structure. The boxplots are depicted based on 30 runs.
Performance on Heavy-tailed Distribution. Although the sub-Gaussian distribution of input data is assumed in Assumption 1, we demonstrate that our method can be extended to a wider range of distributions. We conduct simulations under both uniform and heavy-tailed distributions. The simulation setup mirrors that of Figure 4, with the exception that we sample  $\mathbf{x}$  from a  $[-1,1]$  uniform distribution and a  $t(3)$  distribution, respectively. Results corresponding to the  $t(3)$  distribution are presented in Figure 5, and we can see that our method maintains strong performance, suggesting that the constraints of Assumption 1 can be substantially relaxed. Additional simulation results can be found in the Appendix A.
Sensitivity Analysis with respect to smoothness parameter  $\gamma$ . We analyze the impact of smoothness parameter  $\gamma$  on our proposed algorithm. Specifically, the detailed experimental setup follows the default configuration, and  $\gamma$  is set within the range  $[2.5 \times 10^{-5}, 1 \times 10^{-3}]$ . The simulations are replicated 30 times, and the numerical results of estimation error and four estimated signal strengths are presented in Figure 6. From Figure 6, it's evident that the choice of  $\gamma$  is relatively insensitive in the sense that the estimation errors and the estimated strengths of the signals under different  $\gamma$ s are very close. Furthermore, as  $\gamma$  increases, the estimation accuracy experiences a minor decline, but it remains within an acceptable range. See Appendix A for simulation results of Signal 3 and Signal 4.
Figure 5: Performance on complex signal structure under  $t(3)$  distribution. The boxplots are depicted based on 30 runs.
(a) Estimation Result
Figure 6: Sensitivity analysis of smoothness parameter  $\gamma$ . The boxplots are depicted based on 30 runs. The estimation error is calculated by  $\| \beta_t - \beta^*\| / \| \beta^*\|$ .
(b) Signal 1
(c) Signal 2
In this paper, we leverage over-parameterization to design an unregularized gradient-based algorithm for SVM and provide theoretical guarantees for implicit regularization. We employ Nesterov's method to smooth the re-parameterized hinge loss function, which solves the difficulty of non-differentiability and improves computational efficiency. Note that our theory relies on the incoherence of the design matrix. It would be interesting to explore to what extent these assumptions can be relaxed, which is a topic of future work mentioned in other studies on implicit regularization. It is also promising to consider extending the current study to nonlinear SVMs, potentially incorporating the kernel technique to delve into the realm of implicit regularization in nonlinear classification. In summary, this paper not only provides novel theoretical results for over-parameterized SVMs but also enriches the literature on high-dimensional classification with implicit regularization.
The authors sincerely thank the anonymous reviewers, AC, and PCs for their valuable suggestions that have greatly improved the quality of our work. The authors also thank Professor Shaogao Lv for the fruitful and valuable discussions at the very beginning of this work. Dr. Xin He's and Dr. Yang Bai's work is supported by the Program for Innovative Research Team of Shanghai University of Finance and Economics and the Shanghai Research Center for Data Science and Decision Technology.
[37] Junhui Wang, Jiankun Liu, Yong Liu, et al. Improved learning rates of a functional lasso-typesvm with sparse multi-kernel representation. Advances in Neural Information Processing Systems, 34:21467-21479, 2021.
[38] Ashia C Wilson, Rebecca Roelofs, Mitchell Stern, Nati Srebro, and Benjamin Recht. The marginal value of adaptive gradient methods in machine learning. Advances in neural information processing systems, 30, 2017.
[39] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, pages 3635-3673. PMLR, 2020.
[40] Chulhee Yun, Suvrit Sra, and Ali Jadbabaie. Small nonlinearities in activation functions create bad local minima in neural networks. In International Conference on Learning Representations.
[41] Hao Helen Zhang. Variable selection for support vector machines via smoothing spline anova. Statistica Sinica, pages 659-674, 2006.
[42] Xiang Zhang, Yichao Wu, Lan Wang, and Runze Li. Variable selection for support vector machines in moderately high dimensions. Journal of the Royal Statistical Society Series B: Statistical Methodology, 78(1):53-76, 2016.
[43] Yingying Zhang, Yan-Yong Zhao, and Heng Lian. Statistical rates of convergence for functional partially linear support vector machines for classification. Journal of Machine Learning Research, 23(156):1-24, 2022.
[44] Peng Zhao, Yun Yang, and Qiao-Chu He. High-dimensional linear regression via implicit regularization. Biometrika, 109(4):1033-1046, 2022.
[45] Tianyi Zhou, Dacheng Tao, and Xindong Wu. Nesvm: A fast gradient method for support vector machines. In 2010 IEEE International Conference on Data Mining, pages 679-688. IEEE, 2010.
[46] Xingcai Zhou and Hao Shen. Communication-efficient distributed learning for high-dimensional support vector machines. Mathematics, 10(7):1029, 2022.
In Appendix A, we provide supplementary explanations and additional experimental results for the numerical study.
In Appendix B, we discuss the performance of our method using other data generating schemes.
In Appendix C, we characterize the dynamics of the gradient descent algorithm for minimizing the Hadamard product re-parametrized smoothed hinge loss  $\mathcal{E}_{\mathcal{Z}^n,\gamma}^*$ .
In Appendix D, we prove the main results stated in the paper.
In Appendix E, we provide the proof of propositions and technical lemmas in Section 3.
A.1 Effects of Small Initialization on Error Components
We present a comprehensive analysis of the impact of initialization on the error components, as depicted in Figure 7. Our results demonstrate that, while the initialization size  $\alpha$  does not alter the error trajectory, it significantly affects the error bounds. Numerically, when the initialization  $\alpha$  is  $10^{-4}$ , the maximum absolute value of the error components is around  $3 \times 10^{-5}$ , which is bounded by the initialization size  $10^{-4}$ . Similarly, when the initialization is  $10^{-10}$ , the maximum absolute value of the error components is around  $7.5 \times 10^{-14}$ , which is bounded by the initialization size  $10^{-10}$ . The same result is obtained for the other two initialization sizes. The specific numerical results we obtained validate the conclusions drawn in Proposition 1, as the error term satisfies
$$
\| \boldsymbol {\beta} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} \| _ {\infty} = \| \mathbf {w} _ {t} \odot \mathbf {w} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} - \mathbf {v} _ {t} \odot \mathbf {v} _ {t} \odot \mathbf {1} _ {S _ {1} ^ {c}} \| _ {\infty} \lesssim (\sqrt {\alpha}) ^ {2} = \alpha .
$$
A.2 True Negative Results
In the main text, we could not include the True Negative error figures in 2, 3, 4, and 5 due to space constraints. However, these figures are provided in Figure 8. We observe that both the gradient descent estimator and the Lasso estimator effectively detect the true signals across different settings.
A.3 Additional Results under Uniform Distribution
As previously described, we sample  $\mathbf{X}$  from a  $[-1, 1]$  uniform distribution and set other parameters as follows: true signal structures are  $\mathbf{A} = (5, 6, 7, 8)$ ,  $\mathbf{B} = (4, 6, 8, 9)$ ,  $\mathbf{C} = (3, 6, 9, 10)$ ,  $\mathbf{D} = (2, 6, 10, 11)$ , and  $\mathbf{E} = (1, 6, 11, 12)$ . The number of signals is  $s = 4$ , the sample size is  $n = 200$ , dimension is  $p = 400$ , step size is  $\eta = 0.5$ , smoothness parameter is  $\gamma = 10^{-4}$ , and the initialization size is  $\alpha = 10^{-8}$ . The experimental results are summarized in Figure 9. As depicted in Figure 9, the gradient descent (GD) estimator consistently outperforms the Lasso estimator in terms of estimation accuracy and variable selection error. Both methods slightly surpass the oracle estimator in terms of prediction. However, this advantage mainly stems from the inevitable overestimation of the number of signals in the estimates by both the GD and Lasso methods.
A.4 Additional Results of Sensitivity Analysis
Within our sensitivity analysis, the simulation results for Signal 3 and Signal 4 are presented in Figure 10. We observe that the estimated strengths of the two signals exhibit similar distributions across different values of the smoothness parameter  $\gamma$ .
B Other Data Generating Schemes
In section 4, the scheme for generating  $y$  in our original submission is based on a treatment similar to those in [41, 42], where the task of variable selection for the support vector machine is considered.
Figure 7: Effects of small initialization  $\alpha$  on error components.
In this section, we also introduce other generating schemes as adopted in [32, 15]. Specifically, we generate random data based on the following two models.