SlowGuess/NeurIPS_2023_OCR huggingface dataset:
${}^{1}$  Constants may depend on  ${\sigma }_{x}$  ,and we omit this dependency thereafter for clarity.
5 Proof Sketch and Techniques
5.1 A Time Variant Bound
We further introduce an additional type of effective dimension besides  $k_{0}, k_{1}$ , which is time variant and is utilized to track the optimization dynamics.
Definition 5.1 (Effective Dimensions). Given a feature covariance matrix  $\Sigma$ , define the effective dimension  $k_{2}$  as
$$
k _ {2} \triangleq \min  \left\{l \geq 0: \sum_ {i > l} \lambda_ {i} + n \lambda_ {l + 1} \leq c _ {2} c (t, n) \sum_ {i > 0} \lambda_ {i} \right\}, \tag {8}
$$
where  $c_2$  is a constant independent of the dimension  $p$ , sample size  $n$ , and time  $t$ . The term  $c(t, n)$  is a function to be discussed later. When the context is clear, we omit its dependencies on  $c_2, c(t, n), n, \Sigma$  and only denote it by  $k_2$ .
Based on the effective rank and effective dimensions defined above, we provide a time-variant bound in Theorem 5.1 for overparameterized linear regression, which further leads to the argument in Theorem 4.1. Compared to the existing bound [8], Theorem 5.1 focuses on investigating the role of training epoch  $t$  in the excess risk, to give a refined bias-variance decomposition.
Theorem 5.1 (Time Variant Bound). Suppose Assumption 1, 2 and 3 hold. Fix a function  $c(t,n)$ . Given  $\delta \in (0,1)$ , assume that  $k_0 \leq \frac{n}{c}$ ,  $\log \frac{1}{\delta} \leq \frac{n}{c}$ ,  $0 < \lambda \leq \frac{1}{c\sum_{i > 0}\lambda_i}$  for a large enough constant  $c$ . Then with probability at least  $1 - \delta$ , we have for any  $t \in \mathbb{N}$
$$
R \left(\boldsymbol {\theta} _ {t}\right) \lesssim B \left(\boldsymbol {\theta} _ {t}\right) + V \left(\boldsymbol {\theta} _ {t}\right), \tag {9}
$$
$$
B (\boldsymbol {\theta} _ {t}) = \| \boldsymbol {\theta} ^ {*} \| ^ {2} \left(\frac {1}{\lambda t} + \| \boldsymbol {\Sigma} \| \max  \left\{\sqrt {\frac {r (\boldsymbol {\Sigma})}{n}}, \frac {r (\boldsymbol {\Sigma})}{n}, \sqrt {\frac {\log \left(\frac {1}{\delta}\right)}{n}} \right\}\right),
$$
$$
V (\boldsymbol {\theta} _ {t}) = \sigma_ {y} ^ {2} \log \left(\frac {1}{\delta}\right) \left(\frac {k _ {1}}{n} + \frac {k _ {2}}{c (t , n) n} + c (t, n) \left(\frac {\lambda t}{n} \sum_ {i > 0} \lambda_ {i}\right) ^ {2}\right).
$$
We provide a high-level intuition behind Theorem 5.1. We decompose  $R(\pmb{\theta}_t)$  into the bias term and the variance term. The variance term is then split into the leading part and tailing part based on the spectrum of the feature covariance  $\pmb{\Sigma}$ . The eigenvalues in the tailing part will cause the variance term in the excess risk of the min-norm interpolating solution to be  $\Omega(1)$  for fast decaying spectrum, as is the case in [8]. However, since the convergence in the tailing eigenspace is slower compared with the leading eigenspace, a proper early stopping strategy will prevent the overfitting in the tailing eigenspace and meanwhile avoid underfitting in the leading eigenspace.
The  $c(t, n)$  Principle. It is worth emphasizing that our bound holds for arbitrary positive function  $c(t, n)$ . Therefore, one can fine-tune the generalization bound by choosing a proper  $c(t, n)$ . In the subsequent sections, we show how to derive consistent risk bounds for different time  $t$ , based on different choices of  $c(t, n)$ . We present the case of choosing a constant  $c(t, n)$  in the next section. We leave the case of choosing a varying  $c(t, n)$  to Appendix 5.3.
5.2 Varying  $t$ , Constant  $c(t,n)$
Theorem 5.1 provides an excess risk upper bound uniformly for  $t \in \mathbb{N}$ . However, it is still non-trivial to derive Theorem 4.1, where the remaining question is to decide the term  $c(t,n)$ . The following corollary shows the generalization bound when setting  $c(t,n)$  to a constant.
Corollary 5.1. Let Assumption 1, 2 and 3 hold. Fix a constant  $c(t,n)$ . Suppose  $k_{0} = O(n)$ ,  $k_{1} = o(n)$ ,  $r(\Sigma) = o(n)$ ,  $\lambda = O\left(\frac{1}{\sum_{i > 0}\lambda_i}\right)$ . Then there exists a sequence of positive constants  $\{\delta_n\}_{n\geq 0}$  which converge to 0, such that with probability at least  $1 - \delta_{n}$ , the excess risk is consistent for  $t\in \left(\omega \left(\frac{1}{\lambda}\right),o\left(\frac{n}{\lambda}\right)\right)$ , i.e.
$$
R \left(\boldsymbol {\theta} _ {t}\right) = o (1). \tag {10}
$$
Table 1: Comparisons of excess risk bound with Bartlett et al. [8] and Zou et al. [76]. We provide four types of feature covariance with eigenvalues  $\lambda_{k}$ , including Inverse Polynomial ( $\lambda_{k} = \frac{1}{k^{\alpha}}$ ,  $\alpha > 1$ ), Inverse Log Polynomial ( $\lambda_{k} = \frac{1}{k\log^{\beta}(k + 1)}$ ,  $\beta > 1$ ), Constant ( $\lambda_{k} = \frac{1}{n^{1 + \varepsilon}}$ ,  $1 \leq k \leq n^{1 + \varepsilon}$ ,  $\varepsilon > 0$ ), and Piecewise Constant ( $\lambda_{k} = \frac{1}{s}$  if  $1 \leq k \leq s$  and  $\lambda_{k} = \frac{1}{d - s}$  if  $s + 1 \leq k \leq d$ , where  $s = n^{r}$ ,  $d = n^{q}$ ,  $0 < r \leq 1$ ,  $q > 1$ ). In light of these bounds, ours outperforms Bartlett et al. [8] in all the cases, and outperforms Zou et al. [76] in Constant / Piecewise Constant cases if  $\varepsilon < \frac{1}{2}$  and  $q < \min\{2 - r, \frac{3}{2}\}$ . We refer to Appendix B for more details.
DISTRIBUTIONS	OURS	BARTLETT ET AL. [8]	ZOU ET AL. [76]
INVERSE POLYNOMIAL	O(n- min{α-1/α, 1/2})	O(1)	O(n- α-1/α)
INVERSE LOG POLYNOMIAL	O(1/logβn)	o(1)	O(1/logβn)
CONSTANT	O(n- 1/2)	O(n- min{ε, 1/2})	O(n- min{ε,1})
PIECEWISE CONSTANT	O(n- min{1-r, 1/2})	O(n- min{1-r,q-1, 1/2})	O(n- min{1-r,q-1})
Furthermore, for any positive constant  $\delta$ , with probability at least  $1 - \delta$ , the minimal excess risk on the training trajectory can be bounded as
$$
\min  _ {t} R \left(\boldsymbol {\theta} _ {t}\right) \lesssim \frac {\max  \left\{\sqrt {r (\boldsymbol {\Sigma})} , 1 \right\}}{\sqrt {n}} + \frac {\max  \left\{k _ {1} , 1 \right\}}{n}. \tag {11}
$$
Lemma 5.1 below shows that  $k_{1} = o(n)$  always holds for fixed distribution. Therefore, combining Corollary 5.1 and the following Lemma 5.1 completes the proof of Theorem 4.1.
Lemma 5.1. For any fixed (i.e. independent of sample size  $n$ ) feature covariance  $\Sigma$  satisfying assumption 1, we have  $k_{1}(n) = o(n)$ .
Next we apply the bound in Corollary 5.1 to several data distributions. These distributions are widely discussed in [8, 76]. We also derive the existing excess risk bounds, which focus on the min-norm interpolator [8] and one-pass SGD iterate [76], of these distributions and compare them with our theorem. The results are summarized in Table 1, which shows that the bound in Corollary 5.1 outperforms previous results for a general class of distributions.
Example 5.1. Under the same conditions as Theorem 5.1, let  $\Sigma$  denote the feature covariance matrix. We show the following examples:
1. (Inverse Polynomial). If the spectrum of  $\pmb{\Sigma}$  satisfies  $\lambda_{k} = \frac{1}{k^{\alpha}}$  for some  $\alpha > 1$ , we derive that  $k_{0} = \Theta(n)$ ,  $k_{1} = \Theta\left(n^{\frac{1}{\alpha}}\right)$ . Therefore,  $\min_{t} V(\pmb{\theta}_{t}) = O\left(n^{\frac{1 - \alpha}{\alpha}}\right)$  and  $\min_{t} R(\pmb{\theta}_{t}) = O\left(n^{-\min\left\{\frac{\alpha - 1}{\alpha}, \frac{1}{2}\right\}}\right)$ .
2. (Inverse Log-Polynomial). If the spectrum of  $\pmb{\Sigma}$  satisfies  $\lambda_{k} = \frac{1}{k\log^{\beta}(k + 1)}$  for some  $\beta > 1$ , we derive that  $k_{0} = \Theta\left(\frac{n}{\log n}\right)$ ,  $k_{1} = \Theta\left(\frac{n}{\log^{\beta}n}\right)$ . Therefore,  $\min_t V(\pmb{\theta}_t) = O\left(\frac{1}{\log^{\beta}n}\right)$  and  $\min_t R(\pmb{\theta}_t) = O\left(\frac{1}{\log^{\beta}n}\right)$ .
3. (Constant). If the spectrum of  $\pmb{\Sigma}$  satisfies  $\lambda_{k} = \frac{1}{n^{1 + \varepsilon}}, 1 \leq k \leq n^{1 + \varepsilon}$ , for some  $\varepsilon > 0$ , we derive that  $k_{0} = 0$ ,  $k_{1} = 0$ . Therefore,  $\min_{t} V(\pmb{\theta}_{t}) = O\left(\frac{1}{n}\right)$  and  $\min_{t} R(\pmb{\theta}_{t}) = O\left(\frac{1}{\sqrt{n}}\right)$ .
4. (Piecewise Constant). If the spectrum of  $\pmb{\Sigma}$  satisfies  $\lambda_{k} = \left\{ \begin{array}{ll} \frac{1}{s} & 1 \leq k \leq s, \\ \frac{1}{d - s} & s + 1 \leq k \leq d, \end{array} \right.$  where  $s = n^r, d = n^q, 0 < r \leq 1, q \geq 1$ . We derive that  $k_0 = n^r$ ,  $k_1 = n^r$ . Therefore,  $\min_t V(\pmb{\theta}_t) = O(n^{r-1})$  and  $\min_t R(\pmb{\theta}_t) = O\left(n^{-\min\left\{1-r,\frac{1}{2}\right\}}\right)$ .
Table 2: The effective dimension  $k_{1}$ , the optimal early stopping excess risk, and the min-norm excess risk for different feature distributions, with sample size  $n = 100$ ,  $p = 1000$ . The table shows that early stopping solutions generalize significantly better than min-norm interpolators, and reveals a positive correlation between the effective dimension  $k_{1}$  and excess risk of early stopping solution. We calculate the 95% confidence interval for each excess risk.
DISTRIBUTIONS	k1	OPTIMAL EXCESS RISK	MIN-NORM EXCESS RISK
λi=1/i	Θ (n)	2.399 ± 0.0061	24.071 ± 0.2447
λi=1/ i2	Θ (n1/2)	0.214 ± 0.0050	43.472 ± 0.6463
λi=1/ i3	Θ (n1/3)	0.077 ± 0.0005	10.401 ± 0.2973
λi=1/i log(i+1)	Θ (n/log n)	0.697 ± 0.0053	89.922 ± 0.9591
λi=1/i log2(i+1)	Θ (n/log2n)	0.298 ± 0.0054	82.413 ± 0.9270
λi=1/i log3(i+1)	Θ (n/log3n)	0.187 ± 0.0047	38.145 ± 0.5862
5.3 Varying  $t$ , Varying  $c(t, n)$
Although setting  $c(t, n)$  to a constant as in Corollary 5.1 suffices to prove Theorem 4.1, in this section we show that the choice of  $c(t, n)$  can be much more flexible. Specifically, we provide a concrete example and demonstrate that by setting  $c(t, n)$  to a non-constant, Theorem 5.1 can indeed produce larger compatibility regions.
Example 5.2. Under the same conditions as Theorem 5.1, let  $\pmb{\Sigma}$  denote the feature covariance matrix. If the spectrum of  $\pmb{\Sigma}$  satisfies  $\lambda_{k} = \frac{1}{k^{\alpha}}$  for some  $\alpha > 1$ , we set  $c(t,n) = \Theta \left(n^{\frac{\alpha + 1 - 2\alpha\tau}{2\alpha + 1}}\right)$  for a given  $\frac{\alpha + 1}{2\alpha} \leq \tau \leq \frac{3\alpha + 1}{2\alpha + 2}$ . Then for  $t = \Theta (n^{\tau})$ , we derive that  $V(\pmb{\theta}_t) = O\left(n^{\frac{2\alpha\tau - 3\alpha + 2\tau - 1}{2\alpha + 1}}\right)$ .
Example 5.2 shows that by choosing  $c(t, n)$  as a non-constant, we exploit the full power of Theorem 5.1, and extend the region to  $t = \Theta \left(n^{\frac{3\alpha + 1}{2\alpha + 2}}\right) = \omega (n)$ . In this example, Theorem 5.1 outperforms all  $O\left(\frac{t}{n}\right)$ -type bounds, which become vacuous when  $t = \omega (n)$ .
In this section, we provide numerical studies of overparameterized linear regression problems. We consider overparameterized linear regression instances with input dimension  $p = 1000$ , sample size  $n = 100$ . The features are sampled from Gaussian distribution with different covariances. The empirical results (a.) demonstrate the benefits of trajectory analysis underlying the definition of compatibility, since the optimal excess risk along the algorithm trajectory is significantly lower than that of the min-norm interpolator (b.) validate the statements in Corollary 5.1, since the optimal excess risk is lower when the eigenvalues of feature covariance decay faster. We refer to Appendix C for detailed setups, additional results and discussions.
Observation One: Early stopping solution along the training trajectory generalizes significantly better than the min-norm interpolator. We calculate the excess risk of optimal early stopping solutions and min-norm interpolators from 1000 independent trials and list the results in Table 2. The results illustrate that the early stopping solution on the algorithm trajectory enjoys much better generalization properties. This observation corroborates the importance of data-dependent training trajectory in generalization analysis.
Observation Two: The faster covariance spectrum decays, the lower optimal excess risk is. Table 2 also illustrates a positive correlation between the decaying rate of  $\lambda_{i}$  and the generalization performance of the early stopping solution. This accords with Theorem 5.1, showing that the excess risk is better for a smaller effective dimension  $k_{1}$ , where small  $k_{1}$  indicates a faster-decaying eigenvalue  $\lambda_{i}$ . We additionally note that such a phenomenon also illustrates the difference between min-norm and early stopping solutions in linear regression, since Bartlett et al. [8] demonstrate that the min-norm solution is not consistent when the eigenvalues decay too fast. By comparison, early stopping solutions do not suffer from this restriction.
In this paper, we investigate how to characterize and analyze generalization in a data-dependent and algorithm-dependent manner. We formalize the notion of data-algorithm compatibility and study it under the regime of overparameterized linear regression with gradient descent. Our theoretical and empirical results demonstrate that one can ease the assumptions and broaden the scope of generalization by fully exploiting the data information and the algorithm information. Despite linear cases in this paper, compatibility can be a much more general concept. Therefore, we believe this paper will motivate more work on data-dependent trajectory analysis.
The authors would like to acknowledge the support from the 2030 Innovation Megaprojects of China (Programme on New Generation Artificial Intelligence) under Grant No. 2021AAA0150000.
[1] Madhu S. Advani, Andrew M. Saxe, and Haim Sompolinsky. High-dimensional dynamics of generalization error in neural networks. Neural Networks, 132:428-446, 2020. doi: 10.1016/j.neunet.2020.08.022. URL https://doi.org/10.1016/j.neunet.2020.08.022.
[2] Alnur Ali, J. Zico Kolter, and Ryan J. Tibshirani. A continuous-time view of early stopping for least squares regression. In Kamalika Chaudhuri and Masashi Sugiyama, editors, The 22nd International Conference on Artificial Intelligence and Statistics, AISTATS 2019, 16-18 April 2019, Naha, Okinawa, Japan, volume 89 of Proceedings of Machine Learning Research, pages 1370-1378. PMLR, 2019. URL http://proceedings.mlr.press/v89/ali19a.html.
[3] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via a compression approach. In Jennifer G. Dy and Andreas Krause, editors, Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 254-263. PMLR, 2018. URL http://proceedings.mlr.press/v80/arora18b.html.
[4] Sanjeev Arora, Simon S. Du, Wei Hu, Zhiyuan Li, and Ruosong Wang. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 322-332. PMLR, 2019. URL http://proceedings.mlr.press/v97/arora19a.html.
[5] Yingbin Bai, Erkun Yang, Bo Han, Yanhua Yang, Jiatong Li, Yinian Mao, Gang Niu, and Tongliang Liu. Understanding and improving early stopping for learning with noisy labels. CoRR, abs/2106.15853, 2021. URL https://arxiv.org/abs/2106.15853.
[6] Pradeep Kr. Banerjee and Guido Montúfar. Information complexity and generalization bounds. CoRR, abs/2105.01747, 2021. URL https://arxiv.org/abs/2105.01747.
[7] Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normalized margin bounds for neural networks. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 6240-6249, 2017. URL https://proceedings.neurips.cc/paper/2017/bit/ b22b257ad0519d4500539da3c8bcf4dd-Abstract.html.
[8] Peter L. Bartlett, Philip M. Long, Gábor Lugosi, and Alexander Tsigler. Benign overfitting in linear regression. CoRR, abs/1906.11300, 2019. URL http://arxiv.org/abs/1906.11300.
[74] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Commun. ACM, 64(3):107-115, 2021. doi: 10.1145/3446776. URL https://doi.org/10.1145/3446776.
[75] Lijia Zhou, Danica J. Sutherland, and Nati Srebro. On uniform convergence and low-norm interpolation learning. In Hugo Larochelle, Marc'Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https://proceedings.neurips.cc/paper/2020/bit/4cc5400e63624c44fadeda99f57588a6-Abtract.html.
[76] Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M. Kakade. Benign overfitting of constant-stepsize SGD for linear regression. In Mikhail Belkin and Samory Kpotufe, editors, Conference on Learning Theory, COLT 2021, 15-19 August 2021, Boulder, Colorado, USA, volume 134 of Proceedings of Machine Learning Research, pages 4633-4635. PMLR, 2021. URL http://proceedings.mlr.press/v134/zou21a.html.
[77] Difan Zou, Jingfeng Wu, Vladimir Braverman, Quanquan Gu, and Sham M. Kakade. Risk bounds of multi-pass SGD for least squares in the interpolation regime. In NeurIPS, 2022. URL http://papers.nips.cc/paper_files/paper/2022/bit/543924fdf260ba990f2ef84f940f3db2-Abstract-Conference.html.
A Proofs for the Main Results
We first sketch the proof in section A.1 and give some preliminary lemmas A.2. The following sections A.3, A.4 and A.5 are devoted to the proof of Theorem 5.1. The proof of Theorem 4.1 is given in A.6.
We start with a standard bias-variance decomposition following Bartlett et al. [8], which derives that the time-variant excess risk  $R(\pmb{\theta}_t)$  can be bounded by a bias term and a variance term. We refer to Appendix A.3 for more details.
For the bias part, we first decompose it into an optimization error and an approximation error. For the optimization error, we use the spectrum analysis to bound it with  $O(1 / t)$  where  $t$  denotes the time. For the approximation error, we bound it with  $O(1 / \sqrt{n})$  where  $n$  denotes the sample size, inspired by Bartlett et al. [8]. We refer to Appendix A.4 for more details.
For the variance part, a key step is to bound the term  $(I - \frac{\lambda}{n} XX^{\top})^{t}$ , where  $X$  is the feature matrix. The difficulty arises from the different scales of the eigenvalues of  $XX^{\top}$ , where the largest eigenvalue has order  $\Theta(n)$  while the smallest eigenvalue has order  $O(1)$ , according to Lemma 10 in Bartlett et al. [8]. To overcome this issue, we divide the matrix  $XX^{\top}$  based on whether its eigenvalues is larger than  $c(t,n)$ , which is a flexible term dependent on time  $t$  and sample size  $n$ . Therefore, we split the variance term based on eigenvalues of covariance matrix  $\Sigma$  (leading to the  $k_{1}$ -related term) and based on the eigenvalues of  $XX^{\top}$  (leading to the  $k_{2}$ -related term). We refer to Appendix A.5 for more details.
The following result comes from Bartlett et al. [8], which bounds the eigenvalues of  $XX^{\top}$ .
Lemma A.1. (Lemma 10 in Bartlett et al. [8]) For any  $\sigma_x$ , there exists a constant  $c$ , such that for any  $0 \leq k < n$ , with probability at least  $1 - e^{-\frac{n}{c}}$ ,
$$
\mu_ {k + 1} \leq c \left(\sum_ {i > k} \lambda_ {i} + \lambda_ {k + 1} n\right). \tag {12}
$$
This implies that as long as the step size  $\lambda$  is small than a threshold independent of sample size  $n$ , gradient descent is stable.
Corollary A.1. There exists a constant  $c$ , such that with probability at least  $1 - e^{-\frac{n}{c}}$ , for any  $0 \leq \lambda \leq \frac{1}{c \sum_{i > 0} \lambda_i}$  we have
$$
\boldsymbol {O} \preceq \boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X} \preceq \boldsymbol {I}. \tag {13}
$$
Proof. The right hand side of the inequality is obvious since  $\lambda > 0$ . For the left hand side, we have to show that the eigenvalues of  $I - \frac{\lambda}{n} X^{\top} X$  is non-negative. Since  $X^{\top} X$  and  $X X^{\top}$  have the same non-zero eigenvalues, we know that with probability at least  $1 - e^{-\frac{n}{c}}$ , the smallest eigenvalue of  $I - \frac{\lambda}{n} X^{\top} X$  can be lower bounded by
$$
1 - \frac {\lambda}{n} \mu_ {1} \geq 1 - c \lambda \left(\frac {\sum_ {i > 0} \lambda_ {i}}{n} + \lambda_ {k + 1}\right) \geq 1 - 2 c \lambda \sum_ {i > 0} \lambda_ {i} \geq 0. \tag {14}
$$
where the second inequality uses lemma A.1, and the last inequality holds if  $\lambda \leq \frac{1}{2c\sum_{i > 0}\lambda_i}$ .
A.3 Proof for the Bias-Variance Decomposition
Let  $\pmb{X}^{\dagger}$  denote the Moore-Penrose pseudoinverse of matrix  $\pmb{X}$ . The following lemma gives a closed form expression for  $\theta_t$ .
Lemma A.2. The dynamics of  $\{\pmb {\theta}_t\}_{t\geq 0}$  satisfies
$$
\boldsymbol {\theta} _ {t} = \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \left(\boldsymbol {\theta} _ {0} - \boldsymbol {X} ^ {\dagger} \boldsymbol {Y}\right) + \boldsymbol {X} ^ {\dagger} \boldsymbol {Y}. \tag {15}
$$
Proof. We prove the lemma using induction. The equality holds at  $t = 0$  as both sides are  $\theta_0$ . Recall that  $\theta_t$  is updated as
$$
\boldsymbol {\theta} _ {t + 1} = \boldsymbol {\theta} _ {t} + \frac {\lambda}{n} \boldsymbol {X} ^ {\top} (\boldsymbol {Y} - \boldsymbol {X} \boldsymbol {\theta} _ {t}). \tag {16}
$$
Suppose that the dynamic holds up to the  $t$ -th step. Plug the expression for  $\pmb{\theta}_t$  into the above recursion and note that  $\pmb{X}^{\top}\pmb{X}\pmb{X}^{\dagger} = \pmb{X}^{\top}$ , we get
$$
\begin{array}{l} \boldsymbol {\theta} _ {t + 1} = \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \boldsymbol {\theta} _ {t} + \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {Y} \\ = \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t + 1} \left(\boldsymbol {\theta} _ {0} - \boldsymbol {X} ^ {\dagger} \boldsymbol {Y}\right) + \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) \boldsymbol {X} ^ {\dagger} \boldsymbol {Y} + \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {Y} \tag {17} \\ = \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t + 1} \left(\boldsymbol {\theta} _ {0} - \boldsymbol {X} ^ {\dagger} \boldsymbol {Y}\right) + \boldsymbol {X} ^ {\dagger} \boldsymbol {Y}. \\ \end{array}
$$
Next we prove two identities which will be used in further proof.
Lemma A.3. The following two identities hold for any matrix  $X$  and non-negative integer  $t$ :
$$
\boldsymbol {I} - \boldsymbol {X} ^ {\dagger} \boldsymbol {X} + \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \boldsymbol {X} ^ {\dagger} \boldsymbol {X} = \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t}, \tag {18}
$$
$$
\left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} ^ {\top} \boldsymbol {X}\right) ^ {t} \right] \boldsymbol {X} ^ {\dagger} \boldsymbol {X} \boldsymbol {X} ^ {\top} = \boldsymbol {X} ^ {\top} \left[ \boldsymbol {I} - \left(\boldsymbol {I} - \frac {\lambda}{n} \boldsymbol {X} \boldsymbol {X} ^ {\top}\right) ^ {t} \right]. \tag {19}
$$
Proof. Note that  $\pmb{X}^{\top}\pmb{X}\pmb{X}^{\dagger} = \pmb{X}^{\top}$ , we can expand the left hand side of the first identity above using binomial theorem and eliminate the pseudo-inverse  $\pmb{X}^{\dagger}$ :