SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Yang Sui* Xin He* Yang Bai†
School of Statistics and Management
Shanghai University of Finance and Economics
suiyang1027@stu.sufe.edu.cn;he.xin17@mail.shufe.edu.cn;
statbyang@mail.shufe.edu.cn
In this paper, we design a regularization-free algorithm for high-dimensional support vector machines (SVMs) by integrating over-parameterization with Nesterov's smoothing method, and provide theoretical guarantees for the induced implicit regularization phenomenon. In particular, we construct an over-parameterized hinge loss function and estimate the true parameters by leveraging regularization-free gradient descent on this loss function. The utilization of Nesterov's method enhances the computational efficiency of our algorithm, especially in terms of determining the stopping criterion and reducing computational complexity. With appropriate choices of initialization, step size, and smoothness parameter, we demonstrate that unregularized gradient descent achieves a near-oracle statistical convergence rate. Additionally, we verify our theoretical findings through a variety of numerical experiments and compare the proposed method with explicit regularization. Our results illustrate the advantages of employing implicit regularization via gradient descent in conjunction with over-parameterization in sparse SVMs.
In machine learning, over-parameterized models, such as deep learning models, are commonly used for regression and classification [22, 36, 31, 10]. The corresponding optimization tasks are primarily tackled using gradient-based methods. Despite challenges posed by the nonconvexity of the objective function and over-parameterization [40], empirical observations show that even simple algorithms, such as gradient descent, tend to converge to a global minimum. This phenomenon is known as the implicit regularization of variants of gradient descent, which acts as a form of regularization without an explicit regularization term [30, 29].
Implicit regularization has been extensively studied in many classical statistical problems, such as linear regression [35, 23, 44] and matrix factorization [14, 24, 3]. These studies have shown that unregularized gradient descent can yield optimal estimators under certain conditions. However, a deeper understanding of implicit regularization in classification problems, particularly in support vector machines (SVMs), remains limited. Existing studies have focused on specific cases and alternative regularization approaches [33, 27, 2]. A comprehensive analysis of implicit regularization via direct gradient descent in SVMs is still lacking. We need further investigation to explore the implications and performance of implicit regularization in SVMs.
The practical significance of such exploration becomes evident when considering today's complex data landscapes and the challenges they present. In modern applications, we often face classification
challenges due to redundant features. Data sparsity is notably evident in classification fields like finance, document classification, and gene expression analysis. For instance, in genomics, only a few genes out of thousands are used for disease diagnosis and drug discovery [17]. Similarly, spam classifiers rely on a small selection of words from extensive dictionaries [1]. These scenarios highlight limitations in standard SVMs and those with explicit regularization, like the  $\ell_2$  norm. From an applied perspective, incorporating sparsity into SVMs is an intriguing research direction. While sparsity in regression has been deeply explored recently, sparse SVMs have received less attention. Discussions typically focus on generalization error and risk analysis, with limited mention of variable selection and error bounds [32]. Our study delves into the implicit regularization in classification, complementing the ongoing research in sparse SVMs.
In this paper, we focus on the implicit regularization of gradient descent applied to high-dimensional sparse SVMs. Contrary to existing studies on implicit regularization in first-order iterative methods for nonconvex optimization that primarily address regression, we investigate this intriguing phenomenon of gradient descent applied to SVMs with hinge loss. By re-parameterizing the parameters using the Hadamard product, we introduce a novel approach to nonconvex optimization problems. With proper initialization and parameter tuning, our proposed method achieves the desired statistical convergence rate. Extensive simulation results reveal that our method outperforms the estimator under explicit regularization in terms of estimation error, prediction accuracy, and variable selection. Moreover, it rivals the performance of the gold standard oracle solution, assuming the knowledge of true support.
First, we reformulate the parameter  $\beta$  as  $\mathbf{w} \odot \mathbf{w} - \mathbf{v} \odot \mathbf{v}$ , where  $\odot$  denotes the Hadamard product operator, resulting in a non-convex optimization problem. This re-parameterization technique has not been previously employed for classification problems. Although it introduces some theoretical complexities, it is not computationally demanding. Importantly, it allows for a detailed theoretical analysis of how signals change throughout iterations, offering a novel perspective on the dynamics of gradient descent not covered in prior works [13, 33]. With the help of re-parameterization, we provide a theoretical analysis showing that with appropriate choices of initialization size  $\alpha$ , step size  $\eta$ , and smoothness parameter  $\gamma$ , our method achieves a near-oracle rate of  $\sqrt{s \log p / n}$ , where  $s$  is the number of the signals,  $p$  is the dimension of  $\beta$ , and  $n$  is the sample size. Notice that the near-oracle rate is achievable via explicit regularization [32, 46, 20]. To the best of our knowledge, this is the first study that investigates implicit regularization via gradient descent and establishes the near-oracle rate specifically in classification.
Second, we employ a simple yet effective smoothing technique [28, 45] for the re-parameterized hinge loss, addressing the non-differentiability of the hinge loss. Additionally, our method introduces a convenient stopping criterion post-smoothing, which we discuss in detail in Section 2.2. Notably, the smoothed gradient descent algorithm is not computationally demanding, primarily involving vector multiplication. The variables introduced by the smoothing technique mostly take values of 0 or 1 as smoothness parameter  $\gamma$  decreases, further streamlining the computation. Incorporating Nesterov's smoothing is instrumental in our theoretical derivations. Directly analyzing the gradient algorithm with re-parameterization for the non-smooth hinge loss, while computationally feasible, introduces complexities in theoretical deductions. In essence, Nesterov's smoothing proves vital both computationally and theoretically.
Third, to support our theoretical results, we present finite sample performances of our method through extensive simulations, comparing it with both the  $\ell_1$ -regularized estimator and the gold standard oracle solution. We demonstrate that the number of iterations  $t$  in gradient descent parallels the role of the  $\ell_1$  regularization parameter  $\lambda$ . When chosen appropriately, both can achieve the near-oracle statistical convergence rate. Further insights from our simulations illustrate that, firstly, in terms of estimation error, our method generalizes better than the  $\ell_1$ -regularized estimator. Secondly, for variable selection, our method significantly reduces false positive rates. Lastly, due to the efficient transferability of gradient information among machines, our method is easier to be paralleled and generalized to large-scale applications. Notably, while our theory is primarily based on the Sub-Gaussian distribution assumption, our method is actually applicable to a much wider range. Additional simulations under heavy-tailed distributions still yield remarkably desired results. Extensive experimental results indicate that our method's performance, employing implicitly regularized gradient descent in SVMs,
rivals that of algorithms using explicit regularization. In certain simple scenarios, it even matches the performance of the oracle solution.
Frequent empirical evidence shows that simple algorithms such as (stochastic) gradient descent tend to find the global minimum of the loss function despite nonconvexity. To understand this phenomenon, studies by [30, 29, 19, 38, 43] proposed that generalization arises from the implicit regularization of the optimization algorithm. Specifically, these studies observe that in over-parameterized statistical models, although optimization problems may contain bad local errors, optimization algorithms, typically variants of gradient descent, exhibit a tendency to avoid these bad local minima and converge towards better solutions. Without adding any regularization term in the optimization objective, the implicit preference of the optimization algorithm itself plays the role of regularization.
Implicit regularization has attracted significant attention in well-established statistical problems, including linear regression [13, 35, 39, 23, 44] and matrix factorization [14, 24, 3, 25, 26]. In high-dimensional sparse linear regression problems, [35, 44] introduced a re-parameterization technique and demonstrated that unregularized gradient descent yields an estimator with optimal statistical accuracy under the Restricted Isometric Property (RIP) assumption [6]. [11] obtained similar results for the single index model without the RIP assumption. In low-rank matrix sensing, [14, 24] revealed that gradient descent biases towards the minimum nuclear norm solution when initiated close to the origin. Additionally, [3] demonstrated the same implicit bias towards the nuclear norm using a depth- $N$  linear network. Nevertheless, research on the implicit regularization of gradient descent in classification problems remains limited. [33] found that the gradient descent estimator converges to the direction of the max-margin solution on unregularized logistic regression problems. In terms of the hinge loss in SVMs, [2] provided a diagonal descent approach and established its regularization properties. However, these investigations rely on the diagonal regularization process [4], and their algorithms' convergence rates depend on the number of iterations, and are not directly compared with those of explicitly regularized algorithms. Besides, Frank-Wolfe method and its variants have been used for classification [21, 34]. However, sub-linear convergence to the optimum requires the strict assumption that both the direction-finding step and line search step are performed exactly [18].
2 Model and Algorithms
Throughout this work, we denote vectors with boldface letters and real numbers with normal font. Thus,  $\mathbf{w}$  denotes a vector and  $w_{i}$  denotes the  $i$ -th coordinate of  $\mathbf{w}$ . We use  $[n]$  to denote the set  $\{1,2,\ldots,n\}$ . For any subset  $S$  in  $[n]$  and a vector  $\mathbf{w}$ , we use  $\mathbf{w}_{S}$  to denote the vector whose  $i$ -th entry is  $w_{i}$  if  $i \in S$  and 0 otherwise. For any given vector  $\mathbf{w}$ , let  $\|\mathbf{w}\|_1$ ,  $\|\mathbf{w}\|$  and  $\|\mathbf{w}\|_{\infty}$  denote its  $\ell_1$ ,  $\ell_2$  and  $\ell_{\infty}$  norms. Moreover, for any two vectors  $\mathbf{w}, \mathbf{v} \in \mathbb{R}^p$ , we define  $\mathbf{w} \odot \mathbf{v} \in \mathbb{R}^p$  as the Hadamard product of vectors  $\mathbf{w}$  and  $\mathbf{v}$ , whose components are  $w_{i} v_{i}$  for  $i \in [p]$ . For any given matrix  $\mathbf{X} \in \mathbb{R}^{p_1 \times p_2}$ , we use  $\|\mathbf{X}\|_F$  and  $\|\mathbf{X}\|_S$  to represent the Frobenius norm and spectral norm of matrix  $\mathbf{X}$ , respectively. In addition, we let  $\{a_n, b_n\}_{n \geq 1}$  be any two positive sequences. We write  $a_n \lesssim b_n$  if there exists a universal constant  $C$  such that  $a_n \leq C \cdot b_n$  and we write  $a_n \asymp b_n$  if we have  $a_n \lesssim b_n$  and  $b_n \lesssim a_n$ . Moreover,  $a_n = \mathcal{O}(b_n)$  shares the same meaning with  $a_n \lesssim b_n$ .
2.2 Over-parameterization for  $\ell_1$ -regularized SVM
Given a random sample  $\mathcal{Z}^n = \{(\mathbf{x}_i,y_i)\}_{i = 1}^n$  with  $\mathbf{x}_i\in \mathbb{R}^p$  denoting the covariates and  $y_{i}\in \{0,1\}$  denoting the corresponding label, we consider the following  $\ell_1$ -regularized SVM:
$$
\min  _ {\boldsymbol {\beta} \in \mathbb {R} ^ {p}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - y _ {i} \mathbf {x} _ {i} ^ {T} \boldsymbol {\beta}\right) _ {+} + \lambda \| \boldsymbol {\beta} \| _ {1}, \tag {1}
$$
where  $(\cdot)_{+} = \max \{\cdot, 0\}$  denotes the hinge loss and  $\lambda$  denotes the  $\ell_1$  regularization parameter. Let  $L_{\mathcal{Z}^n}(\beta)$  denote the first term of the right-hand side of (1). Previous works have shown that the  $\ell_1$ -regularized estimator of the optimization problem (1) and its extensions achieve a near-oracle rate of convergence to the true parameter  $\beta^{*}$  [32, 37, 43]. In contrast, rather than imposing the  $\ell_1$
regularization term in (1), we minimize the hinge loss function  $L_{\mathcal{Z}^n}(\beta)$  directly to obtain a sparse estimator. Specifically, we re-parameterize  $\beta$  as  $\beta = \mathbf{w}\odot \mathbf{w} - \mathbf{v}\odot \mathbf{v}$ , using two vectors  $\mathbf{w}$  and  $\mathbf{v}$  in  $\mathbb{R}^p$ . Consequently,  $L_{\mathcal{Z}^n}(\beta)$  can be reformulated as  $\mathcal{E}_{\mathcal{Z}^n}(\mathbf{w},\mathbf{v})$ :
$$
\mathcal {E} _ {\mathcal {Z} ^ {n}} (\mathbf {w}, \mathbf {v}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - y _ {i} \mathbf {x} _ {i} ^ {T} (\mathbf {w} \odot \mathbf {w} - \mathbf {v} \odot \mathbf {v})\right) _ {+}. \tag {2}
$$
Note that the dimensionality of  $\beta$  in (1) is  $p$ , but a  $2p$ -dimensional parameter is involved in (2). This indicates that we over-parameterize  $\beta$  via  $\beta = \mathbf{w} \odot \mathbf{w} - \mathbf{v} \odot \mathbf{v}$  in (2). We briefly describe our motivation on over-parameterizing  $\beta$  this way. Following [16],  $\| \beta \|_1 = \operatorname*{argmin}_{\beta = \mathbf{a} \odot \mathbf{c}} (\|\mathbf{a}\|^2 + \|\mathbf{c}\|^2)/2$ . Thus, the optimization problem (1) translates to  $\min_{\mathbf{a}, \mathbf{c}} \mathcal{E}_{\mathcal{Z}^n}(\mathbf{a}, \mathbf{c}) + \lambda (\|\mathbf{a}\|^2 + \|\mathbf{c}\|^2)/2$ . For a better understanding of implicit regularization by over-parameterization, we set  $\mathbf{w} = \frac{\mathbf{a} + \mathbf{c}}{2}$  and  $\mathbf{v} = \frac{\mathbf{a} - \mathbf{c}}{2}$  leading to  $\beta = \mathbf{w} \odot \mathbf{w} - \mathbf{v} \odot \mathbf{v}$ . This incorporation of new parameters  $\mathbf{w}$  and  $\mathbf{v}$  effectively overparameterizes the problem. Finally, we drop the explicit  $\ell_2$  regularization term  $\lambda (\|\mathbf{a}\|^2 + \|\mathbf{c}\|^2)/2$  and perform gradient descent to minimize the empirical loss  $\mathcal{E}_{\mathcal{Z}^n}(\mathbf{w}, \mathbf{v})$  in (2), in line with techniques seen in neural network training, high-dimensional regression [35, 23, 44], and high-dimensional single index models [11].
2.3 Nesterov's smoothing
It is well-known that the hinge loss function is not differentiable. As a result, traditional first-order optimization methods, such as the sub-gradient and stochastic gradient methods, converge slowly and are not suitable for large-scale problems [45]. Second-order methods, like the Newton and Quasi-Newton methods, can address this by replacing the hinge loss with a differentiable approximation [8]. Although these second-order methods might achieve better convergence rates, the computational cost associated with computing the Hessian matrix in each iteration is prohibitively high. Clearly, optimizing (2) using gradient-based methods may not be the best choice.
To address the trade-off between convergence rate and computational cost, we incorporate Nesterov's method [28] to smooth the hinge loss and then update the parameters via gradient descent. By employing Nesterov's method, (2) can be reformulated as the following saddle point function:
$$
\mathcal {E} _ {\mathcal {Z} ^ {n}} (\mathbf {w}, \mathbf {v}) \equiv \max  _ {\boldsymbol {\mu} \in \mathcal {P} _ {1}} \frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - y _ {i} \mathbf {x} _ {i} ^ {T} (\mathbf {w} \odot \mathbf {w} - \mathbf {v} \odot \mathbf {v})\right) \mu_ {i},
$$
where  $\mathcal{P}_1 = \{\pmb {\mu}\in \mathbb{R}^n:0\leq \mu_i\leq 1\}$ . According to [28], the above saddle point function can be smoothed by subtracting a prox-function  $d_{\gamma}(\pmb {\mu})$  , where  $d_{\gamma}(\pmb {\mu})$  is a strongly convex function of  $\pmb{\mu}$  with a smoothness parameter  $\gamma >0$ . Throughout this paper, we select the prox-function as  $d_{\gamma}(\pmb {\mu}) = \frac{\gamma}{2}\| \pmb {\mu}\| ^2$ . Consequently,  $\mathcal{E}_{\mathcal{Z}^n}(\mathbf{w},\mathbf{v})$  can be approximated by
$$
\mathcal {E} _ {\mathcal {Z} ^ {n}, \gamma} ^ {*} (\mathbf {w}, \mathbf {v}) \equiv \max  _ {\boldsymbol {\mu} \in \mathcal {P} _ {1}} \left\{\frac {1}{n} \sum_ {i = 1} ^ {n} \left(1 - y _ {i} \mathbf {x} _ {i} ^ {T} (\mathbf {w} \odot \mathbf {w} - \mathbf {v} \odot \mathbf {v})\right) \mu_ {i} - d _ {\gamma} (\boldsymbol {\mu}) \right\}. \tag {3}
$$
Since  $d_{\gamma}(\mu)$  is strongly convex,  $\mu_{i}$  can be obtained by setting the gradient of the objective function in (3) to zero and has the explicit form:
$$
\mu_ {i} = \operatorname {m e d i a n} \left(0, \frac {1 - y _ {i} \mathbf {x} _ {i} ^ {T} (\mathbf {w} \odot \mathbf {w} - \mathbf {v} \odot \mathbf {v})}{\gamma n}, 1\right). \tag {4}
$$
For each sample point  $\mathcal{Z}^i = \{(\mathbf{x}_i,y_i)\}$ ,  $i\in [n]$ , we use  $\mathcal{E}_{\mathcal{Z}^i}(\mathbf{w},\mathbf{v})$  and  $\mathcal{E}_{\mathcal{Z}^i,\gamma}^* (\mathbf{w},\mathbf{v})$  to denote its hinge loss and the corresponding smoothed approximation, respectively. With different choices of  $\mu_{i}$  for any  $i\in [n]$  in (4), the explicit form of  $\mathcal{E}_{\mathcal{Z}^i,\gamma}^* (\mathbf{w},\mathbf{v})$  can be written as
$$
\mathcal {E} _ {\mathcal {Z} ^ {i}, \gamma} ^ {*} (\mathbf {w}, \mathbf {v}) = \left\{ \begin{array}{l l} 0, & \text {i f} y _ {i} \mathbf {x} _ {i} ^ {T} (\mathbf {w} \odot \mathbf {w} - \mathbf {v} \odot \mathbf {v}) > 1, \\ (1 - y _ {i} \mathbf {x} _ {i} ^ {T} (\mathbf {w} \odot \mathbf {w} - \mathbf {v} \odot \mathbf {v})) / n - \gamma / 2, & \text {i f} y _ {i} \mathbf {x} _ {i} ^ {T} (\mathbf {w} \odot \mathbf {w} - \mathbf {v} \odot \mathbf {v}) <   1 - \gamma n, \\ (1 - y _ {i} \mathbf {x} _ {i} ^ {T} (\mathbf {w} \odot \mathbf {w} - \mathbf {v} \odot \mathbf {v})) ^ {2} / (2 \gamma n ^ {2}), & \text {o t h e r w i s e .} \end{array} \right. \tag {5}
$$
Note that the explicit solution (5) indicates that a larger  $\gamma$  yields a smoother  $\mathcal{E}_{\mathcal{Z}^i,\gamma}^* (\mathbf{w},\mathbf{v})$  with larger approximation error, and can be considered as a parameter that controls the trade-off between smoothness and approximation accuracy. The following theorem provides the theoretical guarantee of the approximation error. The proof can be directly derived from (5), and thus is omitted here.
Theorem 1. For any random sample  $\mathcal{Z}^i = \{(\mathbf{x}_i, y_i)\}$ ,  $i \in [n]$ , the corresponding hinge loss  $\mathcal{E}_{\mathcal{Z}^i}(\mathbf{w}, \mathbf{v})$  is bounded by its smooth approximation  $\mathcal{E}_{\mathcal{Z}^i, \gamma}^*(\mathbf{w}, \mathbf{v})$ , and the approximation error is completely controlled by the smooth parameter  $\gamma$ . For any  $(\mathbf{w}, \mathbf{v})$ , we have
$$
\mathcal {E} _ {\mathcal {Z} ^ {i}, \gamma} ^ {*} (\mathbf {w}, \mathbf {v}) \leq \mathcal {E} _ {\mathcal {Z} ^ {i}} (\mathbf {w}, \mathbf {v}) \leq \mathcal {E} _ {\mathcal {Z} ^ {i}, \gamma} ^ {*} (\mathbf {w}, \mathbf {v}) + \frac {\gamma}{2}.
$$
2.4 Implicit regularization via gradient descent
In this section, we apply gradient descent algorithm to  $\mathcal{E}_{\mathcal{Z}^n,\gamma}(\mathbf{w},\mathbf{v})$  in (3) by updating  $\mathbf{w}$  and  $\mathbf{v}$  to obtain the estimator of  $\beta^{*}$ . Specifically, the gradients of (3) with respect to  $\mathbf{w}$  and  $\mathbf{v}$  can be directly obtained, with the form of  $-2 / n\sum_{i = 1}^{n}y_{i}\mu_{i}\mathbf{x}_{i}\odot \mathbf{w}$  and  $2 / n\sum_{i = 1}^{n}y_{i}\mu_{i}\mathbf{x}_{i}\odot \mathbf{v}$ , respectively. Thus, the updates for  $\mathbf{w}$  and  $\mathbf{v}$  are given as
$$
\mathbf {w} _ {t + 1} = \mathbf {w} _ {t} + 2 \eta \frac {1}{n} \sum_ {i = 1} ^ {n} y _ {i} \mu_ {t, i} \mathbf {x} _ {i} \odot \mathbf {w} _ {t} \quad a n d \quad \mathbf {v} _ {t + 1} = \mathbf {v} _ {t} - 2 \eta \frac {1}{n} \sum_ {i = 1} ^ {n} y _ {i} \mu_ {t, i} \mathbf {x} _ {i} \odot \mathbf {v} _ {t}, \tag {6}
$$
where  $\eta$  denotes the step size. Once  $(\mathbf{w}_{t + 1},\mathbf{v}_{t + 1})$  is obtained, we can update  $\beta$  as  $\beta_{t + 1} = \mathbf{w}_{t + 1}\odot$ $\mathbf{w}_{t + 1} - \mathbf{v}_{t + 1}\odot \mathbf{v}_{t + 1}$  via the over-parameterization of  $\beta$ . Note that we cannot initialize the values of  $\mathbf{w}_0$  and  $\mathbf{v}_0$  as zero vectors because these vectors are stationary points of the algorithm. Given the sparsity of the true parameter  $\beta^{*}$  with the support  $S$ , ideally,  $\mathbf{w}$  and  $\mathbf{v}$  should be initialized with the same sparsity pattern as  $\beta^{*}$ , with  $\mathbf{w}_S$  and  $\mathbf{v}_S$  being non-zero and the values outside the support  $S$  being zero. However, such initialization is infeasible as  $S$  is unknown. As an alternative, we initialize  $\mathbf{w}_0$  and  $\mathbf{v}_0$  as  $\mathbf{w}_0 = \mathbf{v}_0 = \alpha \mathbf{1}_{p\times 1}$ , where  $\alpha >0$  is a small constant. This initialization approach strikes a balance: it aligns with the sparsity assumption by keeping the zero component close to zero, while ensuring that the non-zero component begins with a non-zero value [11].
We summarize the details of the proposed gradient descent method for high-dimensional sparse SVM in the following Algorithm 1.
Algorithm 1: Gradient Descent Algorithm for High-Dimensional Sparse SVM.
Given: Training set  $\mathcal{Z}^n$ , initial value  $\alpha$ , step size  $\eta$ , smoothness parameter  $\gamma$ , maximum iteration number  $T_1$ , validation set  $\widetilde{\mathcal{Z}}^n$ .
Initialize:  $\mathbf{w}_0 = \alpha \mathbf{1}_{p\times 1}$ $\mathbf{v}_0 = \alpha_1p\times 1$  , and set iteration index  $t = 0$
While  $t < T_{1}$ , do
$$
\mathbf {w} _ {t + 1} \quad = \mathbf {w} _ {t} + 2 \eta \frac {1}{n} \sum_ {i = 1} ^ {n} y _ {i} \mu_ {t, i} \mathbf {x} _ {i} \odot \mathbf {w} _ {t};
$$
$$
\mathbf {v} _ {t + 1} \quad = \mathbf {v} _ {t} - 2 \eta \frac {1}{n} \sum_ {i = 1} ^ {n} y _ {i} \mu_ {t, i} \mathbf {x} _ {i} \odot \mathbf {v} _ {t};
$$
$$
\boldsymbol {\beta} _ {t + 1} \quad = \mathbf {w} _ {t + 1} \odot \mathbf {w} _ {t + 1} - \mathbf {v} _ {t + 1} \odot \mathbf {v} _ {t + 1};
$$
$$
\mu_ {t + 1, i} \quad = \operatorname {m e d i a n} \left(0, \frac {1 - y _ {i} \mathbf {x} _ {i} ^ {T} \boldsymbol {\beta} _ {t + 1}}{n \gamma}, 1\right);
$$
$$
t \quad = t + 1.
$$
End if  $t > T_{1}$  or  $\pmb{\mu}_{t} = \mathbf{0}$ .
Return Set  $\hat{\beta}$  as  $\beta_{t}$ .