SlowGuess/NeurIPS_2023_OCR huggingface dataset:
regularized loss are suitable for greedy pruning, while at the same time achieving a small but non-zero generalization error. Note that the  $\ell_2$  norm is a non-smooth function at the origin, and therefore, the overall regularized loss is non-smooth and non-convex. While there are several notions of approximate stationary points for non-differentiable and non-convex functions, for technical convenience, we replace  $\mathcal{R}$  by a smooth proxy. In particular, for a smoothing parameter  $\beta > 0$ , define a smooth version of the  $\ell_2$  norm, and the corresponding smooth regularizer  $\mathcal{R}_{\beta}$  as,
$$
\mathcal {R} _ {\beta} (U) = \sum_ {i = 1} ^ {k} \ell_ {2} ^ {\beta} \left(U e _ {i}\right), \quad \text {w h e r e} \quad \ell_ {2} ^ {\beta} (v) = \frac {\| v \| _ {2} ^ {2}}{\sqrt {\| v \| _ {2} ^ {2} + \beta}}. \tag {8}
$$
Note that the smaller the value of  $\beta$  is, the closer is  $\ell_2^\beta (v)$  to  $\| v\| _2$ . Considering this definition, the overall regularized loss we study in the population setting is
$$
f _ {\text {p o p}} (U) = \mathcal {L} _ {\text {p o p}} (U) + \lambda \mathcal {R} _ {\beta} (U), \tag {9}
$$
where  $\mathcal{L}_{\mathrm{pop}} = \| UU^T - U_\star U_\star^T\|_F^2$  as defined in eq. (2). The above optimization problem is nonconvex due to the structure of  $\mathcal{L}_{\mathrm{pop}}(U)$ , and finding its global minimizer can be computationally prohibitive. Fortunately, for our theoretical results, we do not require achieving global optimality and we only require an approximate second-order stationary point of the loss in eq. (9), which is defined below.
Definition 2. We say that  $U$  is an  $(\epsilon, \gamma)$ -approximate second-order stationary point of  $f$  if,
1. The gradient norm is bounded above by  $\epsilon$ , i.e.,  $\| \nabla f(U) \|_2 \leq \epsilon$ .
2. The eigenvalues of the Hessian are larger than  $-\gamma$ , i.e.,  $\lambda_{\min}(\nabla^2 f(U)) \geq -\gamma$ .
The full algorithmic procedure that we analyze in this section is summarized in Algorithm 1. Once we find an  $(\epsilon, \gamma)$ -approximate second-order stationary point (SOSP) of eq. (9) for some proper choices of  $\epsilon$  and  $\gamma$ , we apply greedy pruning on the obtained model  $U$  and by eliminating all of its columns with  $\ell_2$  norm below a specific threshold. As a result, the pruned model  $U_{\mathrm{prune}}$  has fewer columns than the trained model  $U$ . In fact, in our following theoretical results, we show that if the parameters are properly selected,  $U_{\mathrm{prune}}$  has exactly  $r$  columns, which is the same as  $U_{\star}$ . Finally, we fine-tune the pruned solution by running gradient descent on the unregularized loss  $\mathcal{L}_{\mathrm{pop}}$  (resp.  $\mathcal{L}_{\mathrm{emp}}$ ). Next, we state the properties of the pruned model generated by Algorithm 1 in the infinite sample setting.
Theorem 3. Consider the population loss with regularization in eq. (9), where  $U_{\star}$  has rank  $r$  and its smallest singular value is denoted by  $\sigma_r^\star$ . Moreover, consider  $U_{\text{prune}}$  as the output of the pruning phase in Algorithm 1 with parameters  $\beta, \lambda, \epsilon, \gamma$  satisfying the conditions<sup>2</sup>,
$$
\beta = c _ {\beta} \frac {\left(\sigma_ {r} ^ {\star}\right) ^ {2}}{r}, \quad \lambda = c _ {\lambda} \frac {\left(\sigma_ {r} ^ {\star}\right) ^ {3}}{\sqrt {k r}}, \quad \gamma \leq c _ {\gamma} \frac {\left(\sigma_ {r} ^ {\star}\right) ^ {3}}{\sqrt {k r} ^ {5 / 2}}, \quad \epsilon \leq c _ {\epsilon} \frac {\left(\sigma_ {r} ^ {\star}\right) ^ {7 / 2}}{\sqrt {k r} ^ {5 / 2}}, \tag {10}
$$
for some absolute constants  $c_{\beta}, c_{\lambda}, c_{\epsilon}, c_{\gamma} > 0$ . Then, we have
1.  $U_{\text{prune}}$  has exactly  $r$  columns.
2.  $\| U_{prune}U_{prune}^T -U_\star U_\star^T\| _F\leq \frac{1}{2} (\sigma_r^\star)^2.$
2This style of result exists for LASSO as well (see [44, Theorem 1]), where the optimal choice of the regularization parameter,  $\lambda^{\star}$ , depends on the true sparsity  $r$ , but a general guarantee can be established as well, which degrades as  $\lambda$  deviates from  $\lambda^{\star}$ . In practice  $\lambda$  is chosen using cross-validation. For simplicity of presentation, we state the result when  $r$  and  $\sigma_r^\star$  are known up to constants.
This result relies on showing that all bounded SOSPs of the regularized loss in eq. (9) are suitable for greedy pruning: removing the columns of  $U$  below a certain  $\ell_2$ -norm threshold results in a solution  $U_{\mathrm{prune}}$  having exactly  $r$  columns, while at the same time having a small generalization error. Hence, it can serve as a proper warm-start for the fine-tuning phase.
Proof sketch of Theorem 3. The key idea is to identify that if we have a matrix  $U$  such that  $U U^T = U_\star U_\star^T$ , and the columns of  $U$  are orthogonal to one another, then  $U$  has exactly  $r$  non-zero columns, where  $r$  is the rank of  $U_\star$ . This statement can be shown to hold even when  $U U^T \approx U_\star U_\star^T$  and the columns of  $U$  are only approximately orthogonal. The main observation we prove is that a bounded  $(\epsilon, \gamma)$ -approximate SOSPs of eq. (9) denoted by  $U$  satisfies the following condition:
$$
\forall i, j: \| U e _ {i} \| _ {2}, \| U e _ {j} \| _ {2} \geq 2 \sqrt {\beta}, \quad \frac {\langle U e _ {i} , U e _ {j} \rangle}{\| U e _ {i} \| _ {2} \| U e _ {j} \| _ {2}} \approx 0. \tag {11}
$$
In other words, all the large columns of  $U$  have their pairwise angle approximately  $90^{\circ}$ . Thus, by pruning away the columns of  $U$  that have an  $\ell_{2}$  norm less than  $2\sqrt{\beta}$ , the remaining columns of  $U$ , i.e., the columns of  $U_{\mathrm{prune}}$ , are now approximately at  $90^{\circ}$  angles to one another. If  $\beta$  is chosen to be sufficiently small, after deleting the low-norm columns, the approximation  $U_{\mathrm{prune}}U_{\mathrm{prune}}^{T} \approx UU^{T}$  holds. By the second order stationarity of  $U$ , we also have that  $UU^{T} \approx U_{\star}U_{\star}^{T}$ . Together, this implies that  $U_{\mathrm{prune}}U_{\mathrm{prune}}^{T} \approx U_{\star}U_{\star}^{T}$  and  $U_{\mathrm{prune}}U_{\mathrm{prune}}^{T}$  is close to a rank  $r$  matrix. Since  $U_{\mathrm{prune}}$  has orthogonal columns, this also means that it has exactly  $r$  columns. Finally, to establish a bound on the approximation error, we simply use the triangle inequality that  $\| U_{\mathrm{prune}}U_{\mathrm{prune}}^{T} - U_{\star}U_{\star}^{T}\|_{F} \leq \| UU^{T} - U_{\star}U_{\star}^{T}\|_{F} + \| UU^{T} - U_{\mathrm{prune}}U_{\mathrm{prune}}^{T}\|_{F}$ . The former is small by virtue of the fact that  $U$  is an approximate second order stationary point of  $f_{\mathrm{pop}} \approx \mathcal{L}_{\mathrm{pop}}$  when  $\lambda$  is small; the latter term is small by the fact that only the small norm columns of  $U$  were pruned away.
Now the only missing part that remains to justify is why the  $\mathcal{R}_{\beta}$  regularizer promotes orthogonality in the columns of approximate second order stationary points and the expression in eq. (11) holds. This is best understood by looking at the regularized loss for the case  $\beta = 0$ , which is equivalent to  $\| U U^T - U_\star U_\star^T \|_F^2 + \lambda \sum_{i=1}^{k} \| U e_i \|_2$  and consider any candidate first-order stationary point  $U$  of this objective. Let  $Z \in \mathbb{R}^{d \times k}$  be a variable constrained to satisfy  $ZZ^T = U U^T$ . Stationarity of  $U$  implies that the choice  $Z = U$  must also be a first-order stationary point of the constrained optimization problem,
$$
\text {M i n i m i z e :} \| Z Z ^ {T} - U _ {\star} U _ {\star} ^ {T} \| _ {F} ^ {2} + \lambda \sum_ {i = 1} ^ {k} \| Z e _ {i} \| _ {2}, \quad \text {S u b j e c t t o :} Z Z ^ {T} = U U ^ {T}. \tag {12}
$$
The first term in the objective is a constant under the constraint and we may remove it altogether. When  $U$  is a full-rank stationary point, constraint qualification holds, and it is possible to write down the necessary KKT first-order optimality conditions, which reduce to,
$$
\forall i \in [ k ], \quad - \lambda \frac {Z e _ {i}}{\| Z e _ {i} \| _ {2}} + (\Lambda + \Lambda^ {T}) Z e _ {i} = 0 \tag {13}
$$
where  $\Lambda \in \mathbb{R}^{d\times d}$  is the set of optimal dual variables. Since  $Z = U$  is a first-order stationary point of the problem in eq. (12) and it satisfies eq. (13), the above condition means that the columns  $Ue_{i}$  are the eigenvectors of the symmetric matrix  $\Lambda +\Lambda^T$ . If all the eigenvalues of  $\Lambda +\Lambda^T$  were distinct, then this implies that the eigenvectors are orthogonal and  $Ue_{i}\perp Ue_{j}$  for all  $i\neq j$ .
While this analysis conveys an intuitive picture, there are several challenges in extending this further. It is unclear how to establish that the eigenvalues of  $\Lambda + \Lambda^T$  are distinct. Moreover, this analysis only applies for full-rank stationary points and does not say anything about rank deficient stationary points  $U$ , where constraint qualification does not hold. Furthermore, it is even more unclear how to extend this analysis to approximate stationary points. Our proof will circumvents each of these challenges by (a) showing that at approximate SOSPs, even if the eigenvalues of  $\Lambda + \Lambda^T$  are not distinct, the columns of  $U$  are orthogonal, and (b) directly bounding the gradient and Hessian of the regularized loss, rather than studying the KKT conditions to establish guarantees even for approximate stationary points which may be rank deficient. Having established guarantees for the pruning phase of the algorithm in the population setting, we next prove a result in the finite sample setting.
5 Finite sample analysis
Next, we extend the results of the previous section to the finite sample setting. Here, we also focus on the smooth version of the regularizer and study the following problem
$$
f _ {\mathrm {e m p}} (U) = \mathcal {L} _ {\mathrm {e m p}} (U) + \lambda \mathcal {R} _ {\beta} (U), \tag {14}
$$
where the empirical loss  $\mathcal{L}_{\mathrm{emp}}$  is defined in eq. (1) and the smooth version of the group Lasso regularizer  $\mathcal{R}_{\beta}(U)$  is defined in eq. (8). In the finite sample setting, we assume that the measurement matrices satisfy the restricted isometry property (RIP) [45], defined below.
Assumption 1. Assume that the measurement matrices  $\{A_1, \dots, A_n\}$  are  $(2k, \delta)$ -RIP. In other words, for any  $d \times d$  matrix with rank  $\leq 2k$ ,
$$
(1 - \delta) \| X \| _ {F} ^ {2} \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle A _ {i}, X \right\rangle^ {2} \leq (1 + \delta) \| X \| _ {F} ^ {2}. \tag {15}
$$
This condition is satisfied, for instance, if the entries of the  $A_{i}$ 's were sampled  $\mathcal{N}(0,1 / d)$  (i.e. Gaussian measurements), as long as  $n\gtrsim dk / \delta^2$  [45].
Theorem 4. Consider the empirical loss with smooth regularization in eq. (14), where  $U_{\star}$  has rank  $r$  and unit spectral norm, with its smallest singular value denoted  $\sigma_r^\star$ , and noise variance  $\sigma^2$ . Consider  $U_{\text{prune}}$  as the output of the pruning phase in Algorithm 1 with parameters  $\beta, \lambda, \epsilon, \gamma$  chosen as per eq. (10). If Assumption 1 is satisfied with  $\delta \leq c_{\delta} \frac{(\sigma_r^\star)^{3/2}}{\sqrt{kr^{5/2}}}$  and the number of samples is at least  $n \geq C_4 \frac{\sigma^2}{(\sigma_r^\star)^4} dk^2 r^5 \log(d/\eta)$ , where  $C_4 > 0$  is a sufficiently large constant, then with probability at least  $1 - \eta$ ,
1.  $U_{\text{prune}}$  has exactly  $r$  columns.
2.  $U_{\text{prune}}$  satisfies the spectral initialization condition:  $\| U_{\text{prune}} U_{\text{prune}}^T - U_\star U_\star^T \|_F \leq \frac{1}{2} (\sigma_r^\star)^2$ .
With Gaussian measurements, the overall sample requirement (including that from the RIP condition) in Theorem 4 is satisfied when  $n \geq \widetilde{\Omega}\left(\frac{1 + \sigma^2}{(\sigma_r^*)^4} dk^2 r^5\right)$ . The high level analysis of this result largely follows that of Theorem 3 in the population setting - we approximate the finite-sample gradient and Hessian by their population counterparts and show that the approximation error decays with the number of samples as  $O(1 / \sqrt{n})$ .
5.1 Fine-tuning phase: Realizing the benefits of pruning
In the fine-tuning phase, the learner runs a few iterations of gradient descent on  $\mathcal{L}_{\mathrm{emp}}$  initialized at the pruned solution  $U_{\mathrm{prune}}$ . Since the model is no longer overparameterized after pruning (by Theorem 4), there are several works analyzing the generalization performance and iteration complexity of gradient descent. Here we borrow the local convergence result of [22] which requires the initial condition that  $\| U_0U_0^T - U_\star U_\star^T\|_F \leq c(\sigma_r^\star)^2$ , where  $c$  is any constant less than 1. As shown in part (b) of Theorem 4, this initial condition is satisfied by  $U_{\mathrm{prune}}$ .
Theorem 5. [22, Corollary 2] Suppose  $\| U_{\star} \|_{op} = 1$ . If we use the output of the greedy pruning in Algorithm 1 denoted by  $U_{prune} \in \mathbb{R}^{d \times r}$  as the initial iterate for the fine-tuning phase, then after  $t \geq m_{\text{fine-tune}} = C_5 (\sigma_1^* / \sigma_r^*)^{10} \cdot \log (\sigma_r^* / \sigma_1^* \cdot n / d)$  iterations, for some sufficiently large absolute  $C_5 > 0$ , the iterates  $\{U_t\}_{t \geq 1}$  of factored gradient descent on  $\mathcal{L}_{\text{emp}}(\cdot)$  satisfy,
$$
\left\| U _ {t} U _ {t} ^ {T} - U _ {\star} U _ {\star} ^ {T} \right\| _ {F} \lesssim \frac {\sigma}{\left(\sigma_ {r} ^ {\star}\right) ^ {2}} \sqrt {\frac {r d}{n}}. \tag {16}
$$
Theorem 5 shows that in the fine-tuning phase, the iterates of gradient descent converges at a linear rate to the generalization error floor of  $\sigma / (\sigma_r^\star)^2 \cdot \sqrt{rd / n}$ , which is also known to be statistically (minimax) optimal [46, 47]. This is possible because the learner is able to correctly identify the rank of the model in the pruning phase and operate in the exactly specified setting in the fine-tuning phase. In contrast, one may ask how this guarantee compares with running vanilla factored gradient descent in the overparameterized setting. This corresponds to the case where no pruning is carried out to reduce
the size of the model. [23] presented a guarantee for the convergence of factored gradient descent from a warm start in the overparameterized setting. In comparison with the exactly specified case, the generalization error floor  $\lim_{t\to \infty}\| U_tU_t^T -U_\star U_\star^T\| _F$  is shown to scale as  $O(\sigma /(\sigma_r^\star)^2\cdot \sqrt{kd / n})^3$  which now depends on  $k$ . Furthermore, linear convergence can no longer be established for vanilla gradient descent because of the ill-conditioning of the objective. This is not just an artifact of the proof - experimentally too, the convergence slowdown was noticed in [23, Figure 1].
This discussion shows that greedy pruning the model first, prior to running gradient descent, in fact generates solutions which generalize better and also converge much faster.
Remark 6. Based on Theorem 4 and Theorem 5 (eq. (16)), under Gaussian measurements, given
$$
n \geq n _ {\varepsilon , \eta} = O \left(\frac {\sigma^ {2}}{\left(\sigma_ {r} ^ {\star}\right) ^ {4}} \frac {r d}{\varepsilon^ {2}} + \frac {1 + \sigma^ {2}}{\left(\sigma_ {r} ^ {\star}\right) ^ {4}} d k ^ {2} r ^ {5} \log (d / \eta)\right) \tag {17}
$$
samples, with probability  $\geq 1 - \eta$ . Algorithm 1 produces  $U_{out}$  that has exactly  $r$  columns and satisfies  $\| U_{out}U_{out}^T - U_\star U_\star^T\|_F \leq \varepsilon$ . Note that the sample complexity depends on the amount of overparameterization,  $k$ , only in the lower order (independent of  $\varepsilon$ ) term.
Note that this result uses the fact that under Gaussian measurements, the RIP condition required in Theorem 4 is satisfied if  $n \geq n_{\varepsilon}$ .
6 Implementing Algorithm 1: Smoothness and optimization oracles
In this section we instantiate the optimization oracle in Algorithm 1, which outputs an approximate SOSP with bounded operator norm. First, we establish that the loss  $f_{\mathrm{emp}}$  is well behaved on the domain  $\{U: \| U \|_{\mathrm{op}} \leq 3\}$ , in that its gradient and Hessian are Lipschitz continuous. These conditions are required by many generic optimization algorithms which return approximate second order stationary points [48, 49]. We establish these properties for the population loss for simplicity and leave extensions to the empirical loss for future work.
Theorem 7. Consider the population loss  $f_{pop}$  in eq. (9). Assume  $\lambda \leq \min \{\beta, \sqrt{\beta}\}$  and  $\| U_{\star} \|_{op} = 1$ . The objective  $f_{pop}(\cdot)$  defined in eq. (9) satisfies for any  $U, V \in \mathbb{R}^{d \times k}$  such that  $\| U \|_{op}, \| V \|_{op} \leq 3$ ,
(a) Lipschitz gradients:  $\| \nabla f_{pop}(U) - \nabla f_{pop}(V)\| _F\lesssim \| U - V\| _F,$
(b) Lipschitz Hessian:  $\| \nabla^2 f_{pop}(U) - \nabla^2 f_{pop}(V)\|_{op}\lesssim \| U - V\| _F$
Under the Lipschitz gradients and Lipschitz Hessian condition, a large number of algorithms in the literature show convergence to an approximate SOSP. A common approach for finding such a point is using the noisy gradient descent algorithm [50]. However, note that we establish these Lipschitzness properties on the bounded domain  $\{U: \| U\|_{\mathrm{op}}\leq 3\}$ , and it remains to verify whether these algorithms indeed approach such points. A similar concern is present with Algorithm 1, which requires access to an optimization oracle which finds an  $(\epsilon ,\delta)$ -approximate SOSP of  $f_{\mathrm{emp}}$  which are also bounded, in that  $\| U\|_{\mathrm{op}}\leq 3$ . The final step is to identify conditions under which these algorithms indeed output stationary points which are bounded, satisfying  $\| U\|_{\mathrm{op}}\leq 3$ . We establish this behavior for a wide family of perturbed gradient based methods.
Perturbed gradient descent: We consider gradient descent with the following update rule: starting from the initialization  $U_0$ , for all  $t \geq 0$
$$
U _ {t + 1} \leftarrow U _ {t} - \alpha \left(\nabla f _ {\text {p o p}} (U _ {t}) + P _ {t}\right) \tag {18}
$$
where  $P_{t}$  is a perturbation term, which for example, could be the explicit noise  $(P_{t} \sim \mathrm{Unif}(\mathbb{B}(r))$  for appropriate  $r$ ) added to escape strict saddle points in [50]. Over the course of running the update rule eq. (18), we show that  $\| U_{t}\|_{\mathrm{op}}$  remains bounded under mild conditions if the algorithm is initialized within this ball. In combination with Theorem 7, this shows that the noisy gradient descent approach of [50] can be used to find the SOSPs required for Algorithm 1.
Theorem 8. Consider optimization of the regularized population loss using the update rule in eq. (18). Assume that  $\| U_{\star}\|_{op} = 1$  and suppose the parameters are selected as  $\alpha \leq 1 / 8$  and  $\lambda \leq \sqrt{\beta}$ , and we have  $\| P_t\|_{op} \leq 1$  almost surely for each  $t \geq 0$ . Then, assuming that the condition  $\| U_0\|_{op} \leq 3$  is satisfied at initialization, for every  $t \geq 1$ , we have  $\| U_t\|_{op} \leq 3$ .
3The result is often stated in terms of the smallest non-zero eigenvalue of  $X_{\star} = U_{\star}U_{\star}^{T}$  , which equals  $(\sigma_r^\star)^2$
Thus, when perturbed gradient descent is carried out to optimize the regularized loss  $f_{\mathrm{pop}}$  eq. (9), the algorithm always returns bounded iterates. In conjunction with Theorem 7, this implies, for example, that noisy gradient descent [50] converges to an approximate second-order stationary point of the regularized population objective. Moreover, the authors of the same paper show the number of gradient calls made by noisy gradient descent to find an  $(\epsilon, \gamma)$ -SOSP is upper bounded by  $\widetilde{O}\left(1 / \min \{\epsilon^2, \gamma^2\}\right)$  for a non-convex objective with  $O(1)$ -smooth gradients and Hessians. Combining with the choices of  $\epsilon$  and  $\gamma$  in eq. (10) results in the following theorem.
Theorem 9. Assume Gaussian measurements and rescale so that  $\| U_{\star}\|_{op} = 1$ . Given  $n\geq n_{\varepsilon ,\eta}$  samples (eq. (17)), consider the algorithm which uses the output of greedy pruning (Algorithm 1),  $U_{prune}\in \mathbb{R}^{d\times r}$ , as the initial iterate for the fine-tuning phase, and runs noisy gradient descent [50] on  $\mathcal{L}_{emp}$  in the fine-tuning phase. Overall, the algorithm makes,
$$
T _ {\epsilon} = \widetilde {O} \left(\frac {1}{\left(\sigma_ {r} ^ {\star}\right) ^ {1 0}} \log \left(\frac {n}{d \sigma_ {r} ^ {\star}}\right) + \frac {k r ^ {5}}{\left(\sigma_ {r} ^ {\star}\right) ^ {7}}\right) \tag {19}
$$
gradient calls and with probability  $\geq 1 - \eta$ , returns a  $U_{out}$  satisfying  $\| U_{out}U_{out}^{T} - U_{\star}U_{\star}^{T}\|_{F} \leq \varepsilon$ .
In this paper, we studied the efficacy of the greedy pruning + fine-tuning pipeline in learning low-complexity solutions for the matrix sensing problem, as well as for learning shallow neural networks with quadratic activation functions. We showed that training on the mean squared error augmented by a natural group Lasso regularizer results in models which are suitable for greedy pruning. Given sufficiently many samples, after pruning away the columns below a certain  $\ell_2$ -norm threshold, we arrived at a solution with the correct column sparsity of  $r$ . Running a few iterations of gradient descent to fine-tune the resulting model, the population loss was shown to converge at a linear rate to an error floor of  $O(\sqrt{rd / n})$ , which is also statistically optimal. We also presented a negative result showing the importance of regularization while training the model. To the best of our knowledge, our results provide the first theoretical guarantee on the generalization error of the model obtained via the greedy pruning + fine-tuning framework.
The research of A. Mokhtari is supported in part by NSF Grants 2007668 and 2127697, ARO Grant W911NF2110226, the National AI Institute for Foundations of Machine Learning (IFML), the Machine Learning Lab (MLL), and the Wireless Networking and Communications Group (WNCG) industrial affiliates program at UT Austin. Kannan Ramchandran would like to acknowledge support from NSF CIF-2002821 and ARO fund 051242-00.
[1] Raghuraman Krishnamoorthi. Quantizing deep convolutional networks for efficient inference: A whitepaper, 2018.
[2] Yunhui Guo. A survey on methods and theories of quantized neural networks, 2018.
[3] Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
[4] Yang He, Guoliang Kang, Xuanyi Dong, Yanwei Fu, and Yi Yang. Soft filter pruning for accelerating deep convolutional neural networks, 2018.
[5] Yawei Li, Shuhang Gu, Luc Van Gool, and Radu Timofte. Learning filter basis for convolutional neural network compression. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5623-5632, 2019.
[6] Aditya Kusupati, Vivek Ramanujan, Raghav Somani, Mitchell Wortsman, Prateek Jain, Sham Kakade, and Ali Farhadi. Soft threshold weight reparameterization for learnable sparsity. In International Conference on Machine Learning, pages 5544-5555. PMLR, 2020.