Understanding the Gains from Repeated Self-Distillation

Authors: divyansh  pareek
simon s du
sewoong  oh
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 02:36:24

Abstract: Self-Distillation is a special type of knowledge distillation where the student model has the same architecture as the teacher model. Despite using the same architecture and the same training data, self-distillation has been empirically observed to improve performance, especially when applied repeatedly. For such a process, there is a fundamental question of interest: How much gain is possible by applying multiple steps of self-distillation? To investigate this relative gain, we propose using the simple but canonical task of linear regression. Our analysis shows that the excess risk achieved by multi-step self-distillation can significantly improve upon a single step of self-distillation, reducing the excess risk by a factor of $d$, where $d$ is the input dimension. Empirical results on regression tasks from the UCI repository show a reduction in the learnt model's risk (MSE) by up to $47$%.

Main paper: 
 introduction: 
 Knowledge distillation [12] was initially proposed as a way to transfer the knowledge learnt by a larger teacher model to a smaller student model, which can then be deployed in limited resource settings. The process is as follows: Train a teacher (T ) model using ground-truth labels, then use its predictions to supervise the training of a student (S) model via a combined per-sample loss,EQUATIONwhere ℓ denotes the loss function, y is the ground-truth label, ŷT denotes the teacher's prediction, and y S (θ) denotes the student's prediction, parameterized by the learnable θ. The extra hyperparameter ξ is called the imitation parameter [25] , generally restricted to ξ ∈ [0, 1]. It gives additional freedom to the student to balance importance between labels and teacher's predictions. The student trained via this distillation objective (i.e., utilizing the teacher's predictions through ξ ̸ = 0) has been widely observed to generalize better than when trained only on the labels (i.e., ξ = 0). This gain has been attributed to 'dark knowledge' that is (i) impossible to be directly extracted from the training data by the small model, but (ii) easily learnt by the large model and transferred to the small model.Challenging this interpretation, Li et al. [20] and Furlanello et al. [10] empirically observed performance gains through distillation even when the teacher and student are same-sized models. One can set T and S to have the same architecture, and S trained with the objective in Eq. (1) outperforms T . This is referred to as Born-Again Networks (BANs) or Self-Distillation (SD). Furthermore, repeatedly applying self-distillation on the same training data with a student model having the same architecture provides additional gains on benchmark datasets and architectures [10, 36, 44] . At each step, the student from the previous step acts as the teacher used to train a new student model under the self-distillation loss of Eq. ( 1). For such multi-step self-distillation, there is a fundamental question of interest: How much more gain can we get by repeatedly applying self-distillation?and show a different conclusion. In particular, we observe that more steps of SD always provide an improvement, if the ξ parameters are chosen optimally. Allen-Zhu and Li [2] analyzed a stylized setting, where a different view of the data is learned by different models, and show how ensemble methods can combine the views, achieving improved test accuracy. This framework is used to show how self-distillation can also improve model accuracy by implicitly performing ensembling. Menon et al. [26] theoretically studied distillation in the classification setting, and also observed a bias-variance tradeoff underlying teacher supervision. Das and Sanghavi [9] theoretically studied one-step self-distillation for fixed design linear regression and binary classification, and showed that the student can provably achieve a lower test error than the teacher. We take inspiration from them and study the multi-step SD to characterize this performance gain, showing that the multi-step SD can outperform one-step SD by a large factor. Borup and Andersen [4] also studied multi-step SD and obtained an analytical form for the k-step SD similar to ours [4, Theorem 4.1] . The crucial difference is the freedom of the ξ parameters being different at each step of self-distillation. Whereas [4, Lemma 4.2 and Theorem 4.3] assume the ξ values at each step are equal (similar to [27] ), concluding that after a point, more steps of SD will result in a poorer performing model (similar to [27] ); our main result (Theorem 1) is different as it says that subsequent steps of SD strictly provide more freedom, and that the best multi-step SD can outperform the best 1-step SD by an Ω(r) factor. Similar to us, Jeong and Chung [16] also take inspiration from [9] , [16] however aim to provide understanding of multi-step self-distillation in the multi-class classification setting.3 Problem formulation and background on self-distillationFocusing on the simple but canonical task of linear regression, we investigate the performance gain from applying repeated self-distillation. 
limitations: 
 Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: Wherever applicable, limitations have been discussed (e.g. Remark 4.2).Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.• The full details can be provided either with the code, in appendix, or as supplemental material. 
linear regression: 
 For the observed response Y ∈ R and the covariate X ∈ R d , the following assumption is standard in linear regression, e.g., [9] . The training set of size n is denoted by X ∈ R d×n , the collection of covariates, and Y ∈ R n , the responses; Y = X ⊤ θ ⋆ + η, with η satisfying E[η] = 0, E[ηη ⊤ ] = γ 2 I n . The problem instance is defined by its parameters (X, θ ⋆ , γ 2 ), treating X = [X 1 , X 2 , • • • X n ] as fixed but Y as random. The training set (X, Y) is one occurrence of the random noise η ∈ R n . In this fixed design setup, the excess risk of an estimator θ is defined using the standard Σn -norm, ∥v∥ Σn = ∥ Σ1/2 n v∥ 2 , asEQUATIONwhere Σn := ( 1 /n)XX ⊤ is the covariance matrix, and the expectation is over the randomness in η.Measuring the error in the Σn -norm ensures that the signal-to-noise ratio is uniform in all directions.The popular ridge estimator serves as a baseline, using a single hyperparameter λ > 0: θ(λ) := arg minEQUATIONWe use Ω λ := XX ⊤ + λI d throughout. We consider only λ > 0, but surprisingly, Kobak et al. [19] showed that the optimal penalty λ ⋆ (one with the lowest risk) can indeed be negative. However we will largely work in the non-overparameterized case (n > d), where λ ⋆ > 0 holds.Applying the self-distillation loss in Eq. ( 1) to linear regression with hyperparameters λ and ξ, θ(λ, ξ) := arg minEQUATION= XX ⊤ + λI d -1 X ξ • X ⊤ θ(λ) + (1 -ξ) • Y New label (5) = (1 -ξ) • I d + ξ • Ω -1 λ XX ⊤Pre-conditioner: function of (λ,ξ)•EQUATIONwhere ξ ∈ R is not restricted to the conventional [0, 1] interval. This additional freedom is meaningful since it can result in a strictly better solution, as noted by Das and Sanghavi [9] both theoretically (Remark 3.6) and empirically (Table 1 ). It is worth noting that the optimization problem in eq. ( 4) remains convex for any ξ ∈ R, since its hessian evaluates to ξ•2XX ⊤ +(1-ξ)•2XX ⊤ = 2XX ⊤ ⪰ 0.On the other hand, the teacher and student use the same ridge penalty λ for simplicity.We call this estimator 1-step self-distillation. This can be interpreted as (i) assigning new labels that combine the ground-truth labels with the teacher's predictions, or (ii) pre-multiplying the usual ridge estimator with a pre-conditioner. Note that ξ = 0 recovers ridge. Das and Sanghavi [9, Theorem 3.8] show that under a certain condition, 1-step self-distillation strictly dominates ridge, i.e., min λ≥0,ξ∈REQUATIONwhere the risk is measured in the non-standard Euclidean norm. The same strict inequality can be shown under the standard Σn -norm under a slightly modified condition stated in Proposition B.1. This naturally leads to a fundamental question: How much more gain can we get by repeatedly applying self-distillation?T S 1-step self-distillation ξ S 0 S 1 S 2 • • • S k k-step self-distillation ξ (k) 1 ξ (k) 2 ξ (k) k Figure 1:The standard 1-step self-distillation defined in Eq. ( 1) with parameter ξ and k-step selfdistillation that repeatedly applies Eq. ( 1) with parameter ξ(k) = [ξ (k) 1 , ξ (k) 2 , . . . , ξ (k) k ] ∈ R k .The standard repeated application of self-distillation starts with the teacher model, T (which we also refer to as the zeroth model, S 0 ), and applies self-distillation sequentially for k steps. At each step i, Eq. ( 1) is applied with the (i -1) th model, S i-1 as the teacher, and the i th model, S i as the student, with an imitation parameter ξ(k) i , i.e., θ ∈ arg min θ ξ (k) i ℓ ŷSi-1 , y Si (θ) + (1 -ξ (k) i )ℓ y, y Si (θ) for i ∈ [k]. The collection of parameters is denoted by ξ (k) = [ξ (k) 1 , ξ (k) 2 , . . . , ξ (k) k ] ∈ R k .This repeated self-distillation has been studied, for example, theoretically in [27] and empirically in [10] . We aim to understand its gain under linear regression, where we prove thatEQUATIONi := (1 -ξ (k) k-i ) k l=k-i+1 ξ (k) l for each i ∈ [k],and we let ξ(k) 0 = 0.The proof that repeated SD with ξ (k) ∈ R k results in Eq. ( 8) is provided in Appendix C.2. Here ξ (k) ∈ R k denote the imitation parameters, λ denotes the ridge coefficient for all the models, andξ(k) ∈ R k is a reparametrization of ξ (k) ∈ R k (details in Appendix C.2). We call this k-step self-distillation.Note the increasing flexibility in the pre-conditioner matrix. The increasing powers of Ω -1 λ XX ⊤ in the above expression are still numerically stable, since, for λ > 0, Ω -1 λ XX ⊤ is PSD with all eigenvalues in [0, 1]. As an aside, one can also consider a version of SD where the i th model receives supervision from all S <i instead of just S i-1 . Appendix C.1 shows that this version provides no extra representational capacity over the repeated version presented above, when all k entries of ξ (k) are optimized as free parameters. Hence, the procedure in Figure 1 suffices for analysis. 
main results for linear regression: 
 The main object of our study is to theoretically demonstrate the gains from repeated self-distillation. Concretely, we aim to show that there can be a significant multiplicative separation between the excess risk achieved by r-step SD (Self-Distillation), where r is the rank of the input X; compared to the ridge estimator, as well as the 1-step SD (Section 4.1). The necessity of the two main assumptions is shown in Section 4.2. The sufficiency of r steps of SD is shown in Section 4.3. In Section 4.4, we provide an exact characterization of the excess risk achieved by k-step SD (for any k).We show the desired separation under the following assumption (and two more mild technical assumptions specified fully in Appendix E).Assumption 2. Assume the following two conditions hold on the problem instance X, θ ⋆ , γ 2 :1. No two non-zero singular values of X collide, i.e. s 1 > s 2 > • • • > s r > 0, where {s j } r j=1 denote the non-zero singular values of the input data matrix X whose rank is denoted by r.2. For a β ∈ [0, 1), there exists an index j ∈ [r] such that ⟨θ ⋆ , u j ⟩ 2 ≥ (1 -β) • ∥θ ⋆ ∥ 2 ; where {u j } d j=1 denote the eigenvectors of XX ⊤ , u 1 being the leading one. Assumption 2 is needed to show that r-step SD achieves a small excess risk in Eq. ( 9). The quantity β captures the similarity of the (unknown) θ ⋆ to the eigenbasis directions. The case of β = 0 entails perfect alignment of θ ⋆ with one of u j , j ∈ [r]. As we will see in the result in Theorem 1, the gains provided by multi-step SD are large when β is small. In general, both these conditions are somewhat necessary for the separation, as we show in Theorems 2 and 3. We now state our main result. We show that under the above assumption, there exists a family of problem instances, (X, θ ⋆ , γ 2 ), such that the excess risk achieved by r-step SD is a factor of r := rank(X) smaller than that of the ridge estimator and the 1-step SD.Theorem 1. Under the fixed design linear regression in Assumption 1, there exists a family of problem instances satisfying Assumption 2 such that for any instance (X, θ ⋆ , γ 2 ) in the family, it holds thatEQUATION)EQUATIONEQUATIONwhere r := rank(X), n is the number of samples, θ(λ, ξ (r) ) and θ(λ, ξ) are the r-step and 1-step SD estimators defined in Eqs. (8) and (4) respectively, and θ(λ) is the ridge estimator defined in Eq. (3).This theorem captures a general result for the gains of multi-step SD. In particular, the special case of β = 0 (i.e. θ ⋆ being completely aligned with one of the eigenvectors u j , j ∈ [r]) presents an Ω(r) multiplicative separation between the excess risk of r-step SD and {1, 0}-step SD. We provide precise conditions and a proof in Appendix E. Since each k-step SD includes (k -1)-step SD as a special case with the proper choice of ξ (k) , the hyperparameter-tuned excess risk of repeated SD is monotonically non-increasing. However, it is perhaps unexpected that the multiplicative separation between r-step SD and 1-step SD can be as large as Ω(r), demonstrating the gains of repeated SD. Figure 2 illustrates this Ω(r) multiplicative separation on a synthetic family of problems. Note that Ω(d) separation can be achieved by choosing the problem instance to have rank r = d, at the cost of requiring many more steps of SD. This Ω(d) factor is the largest multiplicative separation possible with self-distillation, as shown by the fundamental lower bound in Theorem 4 for any pre-conditioning based approach.In general, ifβ = O γ 2 ∥θ ⋆ ∥ 2 s 2 1(akin to the inverse signal-to-noise ratio), then there exists an Ω(r) multiplicative separation between r-step SD and {1, 0}-step SD. In our particular construction of the problem instance for Theorem 1, the additional technical condition (i.e. Condition #2 in the detailed theorem statement in Appendix E) effectively translates this to β = O( 1 /r). Remark 4.1. SD significantly outperforms ridge by primarily reducing the variance. For the lower bound on ridge's excess risk, i.e., Eq. (11), we ignored the bias term and only used the variance term. The repeated SD (Eq. ( 9)) primarily reduces the variance to improve the excess risk over Eq. (11) . 2); we set the singular values of X with a power law from s 1 = 1 to s r = {0.8, 0.5} (left and right panels) and vary r = rank(X). Both plots show a linear increase of the relative gain of r-step self-distillation in excess risk, i.e. the ratio A /B where A := min λ>0 ExcessRisk θ(λ) and B := min λ>0,ξ (r) ∈R r ExcessRisk θ(λ, ξ (r) ) ; demonstrating that r-step SD outperforms ridge by a factor of Ω(r), with the constant inside the Ω (i.e. slope of the line) changing with the effective condition number, s1 /sr.In Figure 3 , we empirically show on synthetic tasks how violating Assumption 2.1 or 2.2 leads to higher excess risks, even for the r-step SD (r = 4 in the example). This supports the necessity of both assumptions, which we analytically investigate in the following.Necessity of Assumption 2.1 on X. We assume that the non-zero singular values of X are unique. This allows us to tightly upper bound the excess risk achieved by r-step SD in Eq. ( 9) via Theorem 4. We show in the following that some version of Assumption 2.1 is also necessary. For a more detailed explanation of why we need Assumption 2.1, we refer the reader to Remark 4.2.Theorem 2. Under the hypotheses of Theorem 1 except for Assumption 2.1, if the singular values of X satisfy s 1 = . . . = s r = 1, where r = rank(X), for all k ≥ 1, λ > 0, and ξ (k) ∈ R k , we haveEQUATIONFurthermore, there exists λ > 0 such that the ridge, θ(λ), achieves this lower bound with equality.We provide a proof in Appendix F. This implies that when there is no gap in the singular values of the input X, there is no separation between ridge and SD estimators (repeated or not). Intuitively, if the s j are all equal, the pre-conditioner for ridge (i.e., Ω -1 λ ) and the pre-conditioner for the repeated SD, both are restricted to have all eigenvalues to be equal. (Repeated) SD has no degrees of freedom to deviate from this. However, s j 's being unequal provides the freedom for the ξ (k) to control the SD's pre-conditioner matrix, and reduce the excess risk. This is also why in Remark 4.2, we hypothesize that numerically, the optimal ξ (k) ⋆ depends inversely on the min-gap of the singular values. Figure 4 demonstrates this increasing relationship of the magnitude of the optimal ξ parameters w.r.to the decreasing singular gap.Necessity of Assumption 2.2 on θ ⋆ . Theorem 1 shows that there is a large separation in the performance of repeated SD over ridge when Assumption 2.2 holds with a small β. In general, this translates to θ ⋆ being highly aligned with any one of the eigenvectors of XX ⊤ (not necessarily the leading eigenvector). We show next that if θ ⋆ is equally (mis)aligned with all the eigenvectors {u j } r j=1 of XX ⊤ , then again there is no separation between ridge and repeated SD. ). Each additional step of SD with optimal choice of ξ (k) reduces ExcessRisk( θ(λ, (ξ (k) ) ⋆ )) for any choice of λ on the x-axis. Panel (a) satisfies Asmp. 2 and hence 4-step SD is necessary to achieve the optimal excess risk. This is no longer true when Asmp. 2.1 is violated (b) or Asmp. 2.2 is violated (c). Excess risk achieved by 4-step SD (i.e. the green lines) in panels (a) and (c) exactly match the numerical value given by RHS of eq. ( 14), i.e. the fundamental lower bound for any SD estimator. But this is not the case in panel (b) [which has the same lower bound from eq. ( 14) as panel (a)], because Asmp. 2.1 is violated. Theorem 3. Under the hypotheses of Theorem 1 except for Assumption 2.2, if the true parameter θ ⋆ satisfies ⟨θ ⋆ , u j ⟩ 2 = z for all j ∈ [r], it holds that for all z > 0, k ≥ 1, λ > 0, andEQUATION)Furthermore, there exists λ > 0 such that the ridge, θ(λ), achieves this lower bound with equality.We provide a proof in Appendix G. Similar conditions are needed when analyzing 1-step SD in [9] as well; [9, Eq. ( 9) in Theorem 3.8] is required for the 1-step SD to strictly outperform ridge. Observe that Eq. ( 9) is violated when either (i) s j 's are all equal or (ii) ⟨θ ⋆ , u j ⟩ 2 's are all equal.For a given problem instance (X, θ ⋆ , γ 2 ), the excess risk achieved by the k-step SD with parameter ξ (k) can be exactly characterized (Theorem 5), but it is complicated and can only be evaluated numerically in general. On the other hand, we show that there exists a fundamental lower bound that holds for a linear family of estimators including all repeated SD, and this lower bound has a simple characterization (Lemma 4.1). Furthermore, we show that under a mild assumption on the eigenvalues of XX ⊤ in Assumption 2.1, the r-step SD achieves the lower bound (Theorem 4). This allows a precise characterization of the performance of r-step SD.Theorem 4. Under the fixed design linear regression in Assumption 1, the excess risk of any k-step SD estimator on an instance (X, θ ⋆ , γ 2 ), is lower bounded for all k ≥ 1, λ > 0, andEQUATIONwhere (s j , u j ) is the j th eigenvalue and eigenvector of X and r := rank(X). Furthermore, if Assumption 2.1 holds then there exists λ > 0 and ξ (r) ∈ R r such that the equality is achieved by the r-step SD estimator θ(λ, ξ (r) ).Proof sketch. (Proof in Appendix H). The lower bound in Eq. ( 14) is an instantiation of Lemma 4.1, since θ(λ, ξ (k) ) is a specific linear family estimator with P = P λ, ξ (k) Ω -1 λ defined in Eq. ( 8). To show achievability, we need to show that P λ, ξ (k) Ω -1 λ = P ⋆ for some value of k, λ, and ξ (k) . This holds when the below system of r linear equations admits a solution for the k parameters (i.e. ξ (k) ), with an extra free parameter λ > 0. We show that with k = r and under Assumption 2.1, there exists λ > 0 that will ensure the existence of a solution for this system of equations.EQUATIONRemark 4.2 (Necessity of Assumption 2.1). This assumption is required for (15) . Otherwise, the LHS would be the same for indices j and j + 1 if s j = s j+1 , but the RHS could still be different as ⟨θ ⋆ , u j ⟩ ̸ = ⟨θ ⋆ , u j+1 ⟩ generally. If Assumption 2.1 does not hold, there might not be any ξ (k) satisfying the set of equations for a general θ ⋆ ∈ R d . Further, the system of linear equations in Eq. ( 15) becomes more ill-conditioned as the singular values s j , j ∈ [r] get closer to each other. Capturing this dependence explicitly is outside the scope of this paper.Lower bound for a linear family. Consider a linear family of estimators of the form θ(P) := P•XY,for P := U d SU ⊤ d , whose eigenspace coincides with that of XX ⊤ (i.e., U d = [u 1 , . . . , u d ])and has d degrees of freedom represented by the eigenvalues S = diag[s 1 , • • • , sd ]. This is a generic form of any linear estimator, albeit with the restriction of the eigenvectors matching the underlying U d . In particular, k-step SD is an instantiation of this with P = P(λ, ξ (k) ) Ω -1 λ (refer to Eq (8)). Lemma 4.1. The Excess Risk for θ(P) = P • XY whereEQUATIONwith equality achieved atP = P ⋆ = U d S⋆ U ⊤ d , given by s⋆ j = ⟨θ ⋆ ,uj ⟩ 2 (⟨θ ⋆ ,uj ⟩ 2 s 2 j +γ 2 ), j ≤ r (i.e., s j > 0)EQUATION)Proof sketch. (Proof in Appendix H.1). One can expand the excess risk for θ(P), which is a quadratic expression in sj , j ∈ [d]. Completing the squares gives the result immediately.k-step SD estimator is quadratic in ξ(k) ∈ R kWe give an explicit formula for the excess risk achieved by for the k-step SD estimator from Eq. ( 8).Sinceθ(λ, ξ (k) ) is linear in each of ξ(k) i , i ∈ [k] (recall that ξ(k) is a reparametrization of ξ (k)), the overall excess risk is quadratic in ξ(k) as shown below. Appendix I provides a proof and the expressions for M (k) , m (k) , and c. This quadratic form will be especially useful in experiments.EQUATIONFrom the detailed expressions given in Appendix I, we note that M (k) is a sum of r symmetric rank-1 matrices, which means it can have a maximum rank of r. This implies that M (k) ∈ R k×k for k > r is rank-deficient (causing no additional decrease in the excess risk if the ξ(r) ∈ R r were chosen optimally to minimize the excess risk). This indicates that r steps of SD might be sufficient to achieve the optimal excess risk, which is indeed what we observe in Theorem 4. 
experiments: 
 In this section, we empirically show that multi-step SD can outperform the ridge and single-step SD.We first present a synthetic setting (section 5.1) to validate our theory. In section 5.2, we discuss a strategy to select ξ parameters based on the theoretical insight from section 4.4. In section 5.3, we implement that strategy on real-world regression tasks and show that it can indeed select performant ξ values that provide multi-step SD estimators that achieve a smaller test risk.We validate our theoretical results with a fixed design synthetic experiment. We consider a problem with d = r = 4, and set problem parameters (X, θ ⋆ , γ 2 ). Namely, X's singular values are set as s j := 1 /j for j ∈ [4] , and θ ⋆ := u 1 as in Theorem 1. Figure 3 shows the result for γ = 0.125, along with two more settings that validate the necessity of our assumptions (validating Theorems 2 and 3). Figure 3a confirms that repeated steps of SD do provide a reduction in the excess risk, since the lowest point of the curve for each k reduces as k increases. Also note that the optimal λ for each k (one that produces lowest excess risk estimator) is different. Appendix J presents some more settings, includingθ ⋆ := 1 / √ 2(u 1 + u 2 )for comparison with [9] . An interesting phenomenon in Figure 3 is that local maxima in k-step SD's curve coincide with local minima in (k -1)-step SD's curve, which was proven for k = 1 in [9] , and we observe empirically for all values of k. 3, 5 . For the fixed design synthetic experiment in Figure 3 and the random design real-world experiment in Figure 5 (section 5.3), the curves plotted are with the optimal (ξ (k) ) ⋆ for each λ. Hence, the curve of k-step SD will point-wise be lower/equal to the curve of (k -1)-step SD, since more steps of SD only provide more freedom. We say k-step SD strictly dominates (k -1)-step SD when the minimum value of k-step SD's excess risk is strictly lower than that of (k -1)-step SD. For Figure 3 , the optimal (ξ (k) ) ⋆ is found analytically from the problem parameters. For real-world datasets in Figure 5 , we use the strategy in section 5.2 to find (ξ (k) ) ⋆ .We have shown that at the cost of introducing additional hyperparameters and setting them to their optimal values, one can extract a large (upto Ω(d)) performance gain. However, how does one select these ξ's for real-world datasets? The standard practice is to use a validation set, and perform a grid search. But this becomes infeasible for k-step SD for larger values of k, since performing a search over parameters in R k+1 (i.e k values of ξ (k) and 1 value of λ) quickly becomes impractical. However, our theoretical analysis provides an insight that can be used to directly compute the optimal ξ (k) ∈ R k (for a chosen λ) given a few evaluations on the validation set with certain chosen ξ (k) values. Namely, Theorem 5 tells us that the ExcessRisk is quadratic in ξ(k) (the reparameterized version). Now the coefficients of the quadratic depend on unknown quantities (like θ ⋆ , γ 2 ), however just knowing the quadratic nature of the functional, we can use the validation set to estimate these coefficients. To estimate the coefficients for k-step SD, we need k(k+3) /2 + 1 evaluations on the validation set. Appendix K provides more discussion, and a detailed illustration of the above process for k = 1, 2. Note that this is feasible when the cost/time needed for a single training run of a k-step SD is small (since we need to perform it O(k 2 ) times), which holds true for linear regression.We implement multi-step SD for real-world regression tasks from the UCI repository [18] , and demonstrate that 2-step SD can outperform ridge and 1-step SD. Note that for this section, the test set will contain fresh samples of X ∈ R d , i.e. random design linear regression instead of fixed design. Our metric for an estimator's performance will now be mean squared error (MSE) on a test set of unseen examples, which is the empirical version of total risk (i.e. excess risk translated by the unknown noise variance γ 2 ). Using the training and validation splits, we compute (i) Optimal ridge:θ(λ ⋆ 0 ), (ii) Optimal 1-step SD: θ(λ ⋆ 1 , ξ ⋆ ), and (iii) Optimal 2-step SD: θ λ ⋆ 2 , (ξ ⋆ 1 , ξ ⋆ 2 ). The procedure is to plot the MSE on the validation set for a grid of λ values for all three estimators, and choose the λ that achieves the lowest error for each one (for any given λ, the optimal ξ ⋆ (λ) is chosen by the strategy described in Section 5.2). Finally, we evaluate the MSE of all three selected estimators (i.e. with the chosen optimal hyperparameters) on the test set, which serves as our performance metric (refer to Table 1 ). Appendix L explains the overall methodology in greater detail. We apply this methodology on three datasets (dataset descriptions in Appendix L.1).Table 1 describes the results we observe. For two of the three datasets, 2-step SD can outperform both ridge and 1-step SD. For the Air Quality dataset, 2-step SD significantly outperforms both ridge and 1-step SD, reducing the MSE by 47.2% compared to the optimal ridge. In contrast, for the AEP dataset, we observe that the SD process cannot improve upon the ridge at all. The MSE curves in Figure 5 also shed light on these observations. Notice how Figures 5a, 5b show a gap in the ridge and 2-step SD (similar to Figure 3a ), whereas Figure 5c shows no such gap (similar to Figure 3c ).In Appendix L.2, we explain the lack of gains from self-distillation on the AEP dataset from the lens of Assumption 2.2. Recall that Theorem 1 says that the gains of repeated SD are most prominent with a small β in Assumption 2.2. Figure 10 shows that the θ ⋆ for the AEP dataset is at most ∼ 35% aligned with any of the eigenbasis directions, corresponding to a β ≈ 0.65. The same alignment for the other two datasets is ∼ 80%, corresponding to a much smaller β ≈ 0.2. In Appendix L.3, we also verify that the strategy described in section 5.2 indeed selects performant ξ values. λ ⋆ 0 = 10 2 λ ⋆ 1 , ξ ⋆ = 10 3 , -4.1 λ ⋆ 2 , (ξ ⋆ 1 , ξ ⋆ 2 ) = 10λ ⋆ 0 = 10 2 λ ⋆ 1 , ξ ⋆ = 10 0 , 66.5 λ ⋆ 2 , (ξ ⋆ 1 , ξ ⋆ 2 ) = 10λ ⋆ 0 = 10 2.5 λ ⋆ 1 , ξ ⋆ = 10 2.5 , 0.1 λ ⋆ 2 , (ξ ⋆ 1 , ξ ⋆ 2 ) = 10 2.5 , (-2. 
conclusion and broader impacts: 
 In this paper, we theoretically studied the multi-step self-distillation for fixed design linear regression, with the goal of characterizing its performance compared to the single-step SD. Perhaps surprisingly, we demonstrated that the optimal multi-step SD can outperform the optimal single-step SD by a factor as large as d in the estimator's excess risk, where d is the input dimension of the regression. Our analysis is limited by the fixed design assumption, and it would be useful to study the case of random design linear regression as well. We empirically demonstrated the gains from using 2-step SD on simple linear regression tasks. Larger scale empirical studies of multi-step SD, especially leveraging the insights of Section 5.2 on hyperparameter search, remain as a direction of future work.Our contributions are largely on the theoretical understanding of multi-step self-distillation, and its potential performance gains. At a high-level, self-distillation can use data more effectively, since it allows us to extract more knowledge from the same training dataset. In today's age with data being one of the most important resources, this has positive potential impacts through more judicious use of data. On the other hand, we propose to use multiple steps of self-distillation, requiring more compute and potentially contributing to higher environmental costs. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [NA] Justification: Since our experiments don't have randomized components (estimators have a closed-form solution), we do not report error bars.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: Our experiments are smaller scale and all experiments run on a single CPU within a minute. We mention this in the Appendix.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code Of Ethics 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?Answer: [NA] Justification: Not relevant.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.• According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector. 
institutional review board (irb) approvals or equivalent for research with human subjects: 
 Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?Answer: [NA] Justification: Not relevant.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: In this short section, we collect some notation used throughout the proofs.Decomposition of X. Let rank(X) = r and the SVD of X be X = r j=1 s j u j v T j where s 1 ≥ s 2 ≥ • • • ≥ s r > 0, and each u j ∈ R d and v j ∈ R n . Further, let {u 1 , u 2 , • • • , u d } be the full set of left singular vectors of X (even those corresponding to zero singular values), forming an orthonormal basis of R d . Let U d ∈ R d×d and U r ∈ R d×r denote the left singular matrix of X for the full and truncated set of left singular vectors respectively. Similarly, let V d ∈ R n×d and V r ∈ R n×r denote the full and truncated right singular matrix. Let S d ∈ R d×d and S r ∈ R r×r denote the collection of the singular values (with and without zeros respectively). Then, it holds thatEQUATION) Indices. Throughout the text, the indices i lie in [k], i.e. they denote subsequent steps of selfdistillation. The indices j lie in [r] or [d], ie they denote dimensions of the d-dimensional space (aligned with vectors of U r or U d ). v j , V i,j will denote indexing into a vector v, matrix V . There is one exception to v j denoting a vector's j th element, which is the below. Components of θ ⋆ on U d . We will denote θ ⋆ j := ⟨θ ⋆ , u j ⟩ 2 , j ∈ [d] as the components of θ ⋆ onto X's left singular space. Note thatd j=1 θ ⋆ j = ∥θ ⋆ ∥ 2 2 .B Discussion on norm used in excess risk metricWe point out that Das and Sanghavi [9] used the ∥.∥ 2 norm instead of the more natural ∥.∥ Σn norm to measure their fixed design excess risk (in eq (2) ). Almost all our results have an equivalent version in the ∥.∥ 2 norm setting also, since the only difference is in the relative weighing of the underlying dimensions of variation, i.e. with the ∥.∥ Σn norm, ∀j ∈ [d], direction u j is weighed by s 2 j /n instead of a constant 1 weight (independent of j). In particular, we also present the version of [9, Eq. ( 9) from Theorem 3.8] that will result in a strict dominance like Eq. ( 7) under the Σn norm, i.e. min λ≥0,ξ∈REQUATION)Proposition B.1. Let λ ⋆ := argmin λ>0 ExcessRisk( θ(λ)). Then Eq. (20) holds on a problem instance (X, θ ⋆ , γ 2 ) whenEQUATIONNote that this differs from [9, Eq. ( 9)] in just one respect: it has s 4 j s 4 k instead of s 2 j s 2 k .C Details on ξ parameters for general k-step SD C.1 Full k-step SD is representationally no larger than Repeated k-step SD We first illustrate the Full k-step SD in Figure 6 . The repeated version introduces k extra hyperparameters in the form of ξ (k) ∈ R k parameters, whereas the full version introduces k(k+1) /2. Consider the case of k = 2, since that is the lowest value of k for which the full version and the repeated version differ. Figure 7 illustrates this difference explicitly. We will show that the freedom of ξ ∈ R 3 is no more than the freedom of ξ ∈ R 2 . This shows the equivalence of the full 2-step and the repeated 2-step versions, when ξ ∈ R 3 , ξ ∈ R 2 are free parameters optimized over the entire respective spaces. Such equivalence for the general k-step case is then easy to see.S 0 S 1 S 2 • • • S k Repeated k-step self-distillation S 0 S 1 S 2 • • • S k Full k-step self-distillationNodes S 0 , S0 are both solving the ridge regression problem (3). Let θ 0 = θ0 = Ω -1 λ XY denote the estimator for both these nodes. Similarly S 1 and S1 are solving the same problem, although with different parameters. We haveθ 1 (ξ 1 ) = Ω -1 λ X ξ 1 • X ⊤ θ0 + (1 -ξ 1 )• Y (and similarly, one can write θ1 with ξ1 instead of ξ 1 ). Now the node S 2 is also solving a 1 parameter supervised SD problem, soEQUATION)But the optimization problem for S2 is a 2 parameter supervised SD. It evaluates toargmin θ∈R d ξ2a 2 ∥X ⊤ θ0 -X ⊤ θ∥ 2 + ξ2b 2 ∥X ⊤ θ1 -X ⊤ θ∥ 2 + (1 -ξ2a -ξ2b ) 2 ∥Y -X ⊤ θ∥ 2 + λ 2 ∥θ∥ 2 .Following through a similar calculation, we observe that θ2 for node S2 is given byEQUATIONEQUATIONFigure 7 : Repeated vs Full 2-step SD. We show that the extra freedom of the parameter ξ2a does not provide any additional freedom, when the other two ξ1 , ξ2b are optimized as free parameters.Equations ( 22) vs (23) show that the full 2-step offers the same freedom as the repeated 2-step.However the repeated version has one shortcoming. To generate the optimal 2-step SD estimator, ξ 1 needs to be different than the value needed for generating the optimal 1-step SD estimator. That is, the repeated k-step version, with its k free parameters, allows only to generate the final k th -step estimator as the optimal one (i.e. if we choose the (ξ 1 , ξ 2 ) values so that the 2 nd estimator is the optimal 2-step SD estimator, then the 1 st estimator with the chosen ξ 1 won't be the optimal 1-step SD estimator). Whereas the full k-step version, with all its k(k + 1)/2 free parameters, allows us to generate a sequence of all k optimal estimators.C.2 Proof that k-step SD estimator with ξ (k) ∈ R k will have the form given in eq (8)Consider Figure 1 . ξ (k) ∈ R k is the set of actual imitation parameters used for running k-step (repeated) SD. Let θ(λ, ξ (k) ) denote the k-step estimator generated by using ξ (k) parameters. In what follows, we will prove that θ(λ, ξ (k) ) will have the form given in eq (8) , with ξ(k) ∈ R k as the described reparametrization of ξ (k) ∈ R k . Since we're in the repeated version, the k th step objective will be a combination of losses w.r.to ground-truth labels and predictions from the (k -1) th step. So, the objective for the k th step isθ(λ, ξ (k) ) := argmin θ∈R d ξ (k) k 2 ∥X ⊤ θ(λ, ξ (k-1) ) -X ⊤ θ∥ 2 + (1 -ξ (k) k ) 2 ∥Y -X ⊤ θ∥ 2 + λ 2 ∥θ∥ 2 .(24) Note that θ(λ, ξ (k) ) recursively depends on predictions from the previous θ(λ, ξ (k-1) ). This objective is of the form in eq (4), so similar to eq (5), we have the following expressionEQUATIONUsing this, we can inductively prove that the general form of this estimator is captured by Eq. ( 8).Claim. With the described reparametrization, i.e., ξ(k) i := (1 -ξ (k) k-i ) k l=k-i+1 ξ (k) l for each i ∈ [k](where we let ξ (k) 0 = 0), Eq. ( 8) is the solution to the recursive form Eq. ( 26). Proof. We will prove this by induction. Base Case. From eqs ( 8) and ( 26), the case for k = 1 is true (with ξ(1) = ξ (1) ). Inductive Step. Assuming Eq. ( 8) captures the solution of Eq. ( 26) for k -1, we getθ(λ, ξ (k) ) = 1 -ξ (k) k I d + ξ (k) k Ω -1 λ XX ⊤ 1 - k-1 i=1 ξ(k-1) i I d + k-1 i=1 ξ(k-1) i Ω -1 λ XX ⊤ i • Ω -1 λ XY .This again satisfies the form in equation ( 8), when the following coefficients matchξ (k) k = k i=1 ξ(k) i , ξ (k) k • (1 - k-1 i=1 ξ(k-1) i ) = ξ(k) 1 , ξ (k) k • ξ(k-1) i-1 = ξ(k) i ∀i ∈ {2, 3, • • • , k} .One can then see that the described reparametrization makes the above hold true.Remark. Since θ(λ, ξ (1) ) is simply the 1-step SD estimator with the form θ(λ, ξ (1) ) = P • Ω -1 λ XY for some preconditioner P, plugging this in the equation eq (26) , we realize that we can factor out Ω -1 λ XY (i.e. the ridge solution) from the expression on the right side. That is, θ(λ, ξ (2) ) = P ′ • Ω -1λ XY for some different pre-conditioner P ′ . This is why inductively we get that θ(λ, ξ (k) ), k ≥ 1 all produce a pre-conditioning on the ridge solution, as shown in eq (8) . Further, eq (26) also dictates why we get increasing powers of the term Ω -1 λ XX ⊤ in the final expression.We explicitly demonstrate the reparametrization for k = 2, 3. As noted in Section 5.2 (and Appendix K), owing to the quadratic form of excess risk in ξ(k) , one can find the optimal ξ(k) analytically. It can then be translated back to the original ξ (k) as follows.For k = 2, the form is (dropping the . (2) for ease)EQUATIONFor k = 3, the form is (dropping the . (3) for ease)EQUATIOND Algebraic expansion of θ(λ, ξ (k) ) from eq (8)Lemma D.1. The estimator θ(λ, ξ (k) ) given in eq (8) can be expanded asθ(λ, ξ (k) ) = d j=1   1 - k i=1 ξ(k) i    1 - s 2 j λ + s 2 j i      • s 2 j λ + s 2 j • ⟨θ ⋆ , u j ⟩ • u j + d j=1   1 - k i=1 ξ(k) i    1 - s 2 j λ + s 2 j i      • s j λ + s 2 j • ⟨η, v j ⟩ • u j Proof. The expansion relies on XX ⊤ = U d S 2 d U ⊤ d and XX ⊤ + λI d = U d (S 2 d + λI d )U ⊤ d . U d is orthonormal (U d U ⊤ d = I d = U ⊤ d U d ), which neatly cancels all occurences of U d in the middle, and allows us to directly combine the matrices in the eigenvalue space.We first write a detailed version of the theorem that exactly characterizes the family of problem instances that admits the separation. We then present a proof. We point the reader to Appendix A for notations used throughout the proof.Theorem 6 (Detailed version). Consider the following conditions on θ ⋆ , X that characterize a family of problem instances {(X, θ ⋆ , γ 2 )}: 1. θ ⋆ satisfies ∃j ∈ [r], ⟨θ ⋆ , u j ⟩ 2 ≥ (1 -β) • ∥θ ⋆ ∥ 2 [Assumption 2.2] 2. ∥θ ⋆ ∥ 2 ≥ 198 • rγ 2 /s 2 r • ( s1 /∃λ > 0, ∃ξ (r) ∈ R r , ExcessRisk θ(λ, ξ (r) ) ≤ γ 2 n 1 + β ∥θ ⋆ ∥ 2 s 2 1 γ 2 (Rewriting eq (9))Under Condition #1 + Condition #2 + Condition #4, it holds that ∀λ > 0, ∀ξ ∈ R, ExcessRisk θ(λ, ξ) ≥ 0.99 2 11 (1 -β)rγ 2 n (Rewriting eq (10) )Under Condition #1 + Condition #2, it holds that ∀λ > 0, ExcessRisk θ(λ) ≥ 0.98 1 -β 1 -0.99β 2 rγ 2 n(Rewriting eq (11) )Proof. We will analyze all three: k-step SD, ridge, and 1-step SD.Let j ′ ∈ [r] be the index that corresponds to the maximum (unsigned) alignment between θ ⋆ and u j , among all j ∈ [r]. If it is not unique, pick any of the corresponding indices. That is, j ′ satisfies ⟨θ ⋆ , u j ′ ⟩ 2 ≥ ⟨θ ⋆ , u j ⟩ 2 for all j ∈ [r]. Then, Condition #1 translates to the below two inequalities,EQUATIONEQUATIONk-step SD: Since Assumption 2.1 holds, from Theorem 4 we directly haveEQUATIONUsing eq (30) in the above, we haveEQUATION)This completes the proof for eq (9) .Ridge: We will now show (11) , by characterizing the Excess Risk for the ridge estimator in this regime, showing that it can be upto r times worse. From Lemma I.1, we realize that the Excess Risk expression for the simple ridge estimator θ(λ) is given byEQUATION≥ γ 2 n r j=1 s 4 j (λ + s 2 j ) 2Just the Variance term .Inequality (38) above comes from ignoring the bias term (since it is non-negative). Note that the variance term is a decreasing function of λ. And again from Lemma I.1, we get the following expression for λ ⋆ > 0 that minimizes the ExcessRisk.EQUATIONEQUATION(Ignoring positive terms in the denominator above)EQUATIONNow since the variance term in ( 38) is a decreasing function of λ, we can use the upper bound of λ ⋆ from (40) to lower bound the ExcessRisk of optimal ridge asExcessRisk θ(λ ⋆ ) ≥ γ 2 n • r j=1 1 1 + λ ⋆ s 2 j 2(Rewriting eq (38) )≥ rγ 2 n • 1 1 + λ ⋆ s 2 r 2 (Since 0 < s r ≤ s j for all j) ≥ rγ 2 n • 1 1 + 1 1-β • 2rγ 2 ∥θ ⋆ ∥ 2 s 2 r • s1 sr 4 2(Using eq (40) )EQUATION)This completes the proof of eq (11) .To evaluate 1-step SD's ExcessRisk, we make use of Theorem 5. Observe that since k = 1, the ExcessRisk is a simple quadratic in one variable. We will call ξ (1) ∈ R as just ξ (similar to eq And similarly, we will call M (1) , m (1) as just M, m, both real-valued. We then haveEQUATION)Note that M, m, c are all functions of λ. Now we evaluate their expressions. c is simply ExcessRisk of ridge, which we will fetch from Lemma I.1. Evaluate M from Theorem 5:M = 1 n r j=1 λθ ⋆ j ρj + γ 2 (1 + ρ j ) 2 • C j (1) • C j (1) = 1 n r j=1 λθ ⋆ j ρj + γ 2 (1 + ρ j ) 2 • ρ 2 j (1 + ρ j ) 2 (Using C j (1) = 1 -1 1+ρj ) = 1 n r j=1 λθ ⋆ j ρ j + γ 2 ρ 2 j (1 + ρ j ) 4 = 1 n   r j=1 λ 2 θ ⋆ j s 6 j + γ 2 λ 2 s 4 j (λ + s 2 j ) 4   . (Using ρ j = λ s 2 j )Evaluate m from Theorem 5:m = 1 n r j=1 λθ ⋆ j -γ 2 (1 + ρ j ) 2 • C j (1) = 1 n r j=1 λθ ⋆ j -γ 2 (1 + ρ j ) 2 • ρ j 1 + ρ j (Using C j (1) = 1 -1 1+ρj ) = 1 n   r j=1 λ 2 θ ⋆ j s 4 j -γ 2 λs 4 j (λ + s 2 j ) 3   . (Using ρ j = λ s 2 j )We now write the expressions together for comparison, before we use them. That is,n • M = λ 2 r j=1 θ ⋆ j s 6 j (λ + s 2 j ) 4 + λ 2 γ 2 r j=1 s 4 j (λ + s 2 j ) 4 , n • m = λ 2 r j=1 θ ⋆ j s 4 j (λ + s 2 j ) 3 -λγ 2 r j=1 s 4 j (λ + s 2 j ) 3 , n • c = λ 2 r j=1 θ ⋆ j s 2 j (λ + s 2 j ) 2 + γ 2 r j=1 s 4 j (λ + s 2 j ) 2 .(Directly from Lemma I.1)With all these pieces, for the numerator in eq (43) , we haven 2 • M c -m 2 = T 1 + T 2 + T 3, where we use T 1 , T 2 , T 3 to capture terms of different forms. T 1 will capture the θ ⋆ j product terms, T 2 will capture the γ 2 product terms, and T 3 will capture the cross terms. Namely,T 1 = λ 4 r j=1 r l=1 θ ⋆ j θ ⋆ l s 2 j s 2 l (λ + s 2 j ) 2 (λ + s 2 l ) 2 • s 4 j (λ + s 2 j ) 2 + s 4 l (λ + s 2 l ) 2 - 2s 2 j s 2 l (λ + s 2 j )(λ + s 2 l ) = λ 4 r j=1 r l=1 θ ⋆ j θ ⋆ l s 2 j s 2 l (λ + s 2 j ) 2 (λ + s 2 l ) 2 • s 2 j (λ + s 2 j ) - s 2 l (λ + s 2 l ) 2 ≥ 0 . ( †) T 2 = λ 2 γ 4 r j=1 r l=1 s 4 j s 4 l (λ + s 2 j ) 2 (λ + s 2 l ) 2 • 1 (λ + s 2 j ) 2 + 1 (λ + s 2 l ) 2 - 2 (λ + s 2 j )(λ + s 2 l ) = λ 2 γ 4 r j=1 r l=1 s 4 j s 4 l (λ + s 2 j ) 2 (λ + s 2 l ) 2 • 1 (λ + s 2 j ) - 1 (λ + s 2 l ) 2 2 ≥ 0 . ( † †) T 3 = λ 2 γ 2   r j=1 θ ⋆ j s 6 j (λ + s 2 j ) 4     r j=1 s 4 j (λ + s 2 j ) 2   + λ 4 γ 2   r j=1 θ ⋆ j s 2 j (λ + s 2 j ) 2     r j=1 s 4 j (λ + s 2 j ) 4   + 2λ 3 γ 2   r j=1 θ ⋆ j s 4 j (λ + s 2 j ) 3     r j=1 s 4 j (λ + s 2 j ) 3   ≥ λ 2 γ 2   r j=1 θ ⋆ j s 6 j (λ + s 2 j ) 4   Z1   r j=1 s 4 j (λ + s 2 j ) 2   Y1 +λ 4 γ 2   r j=1 θ ⋆ j s 2 j (λ + s 2 j ) 2   Z2   r j=1 s 4 j (λ + s 2 j ) 4   Y2 .(Ignoring the 3 rd term)We lower bound the terms Z 1 , Y 1 and Z 2 , Y 2 in the above lower bound on T 3 using simple functional analysis. Consider the fact that the function f p (z) := z 2p (λ+z 2 ) p over z > 0 is strictly monotonically increasing for any p ∈ N, since f ′ p (z) = 2pλz 2p-1 (λ+z 2 ) p+1 > 0 for all z > 0. We will use the instantiation f 4 for Z 1 and Y 2 , and f 2 for Z 2 and Y 1 . For Z 1 , we getZ 1 = r j=1 θ ⋆ j • 1 s 2 j • s 8 j (λ + s 2 j ) 4 ≥ s 8 r (λ + s 2 r ) 4 r j=1 θ ⋆ j • 1 s 2 j (Using monotone increasingness of f 4 ) ≥ s 8 r (λ + s 2 r ) 4 • 1 s 2 1 r j=1 θ ⋆ j ≥θ ⋆ j ′ (Using monotone decreasingness of f (z) := 1 /z 2 ) ≥ s 8 r (λ + s 2 r ) 4 • (1 -β)∥θ ⋆ ∥ 2 s 2 1(Using eq (29) )EQUATION)Following a similar analysis scheme, we getEQUATIONEQUATIONEQUATION)Using eqs ( 44), ( 45), ( 46), (47), we get an overall lower bound on T 3 . For the overall lower bound on the numerator of eq (43) , we combine this with eqs ( †), ( † †) for the terms T 1 and T 2 . We getEQUATION)And for the denominator in eq (43) , we haveEQUATION)Upper Bound ∝ λ 2 from eq (49):EQUATION)Upper Bound ∝ 1 /λ 2 from eq (49):EQUATION)Note that we usedr j=1 θ ⋆ j ≤ d j=1 θ ⋆ j = ∥θ ⋆ ∥ 2 in eqs ( ‡, ‡ ‡). Now we put together the numerator lower bound from eq (48), with the denominator upper bound from eq (50) for the first term, and from eq (51) for the second term. We then getEQUATION) Under Condition #2, it holds that rγ 2 ∥θ ⋆ ∥ 2 s 2 1 ≤ 1 198 • s 6 r s 6 1 ≤ 1 198 (since s r ≤ s 1). This givesEQUATIONAnd now we analyze Q 2 using simple calculus. Note thatQ 2 = t 1 + t 2 λ 6 (λ + s 2 r ) 6 ,whereEQUATION.Simple calculus shows that this function is minimized at λ = ( t1 /(t2s 2 r ))0.2 = s 8 1 s 2 r 0.2 .And the min value of the function (evaluated at λ) can be used as a lower bound for this, givingEQUATIONCombining eqs ( 43), ( 52), ( 53), (54), we get∀λ > 0, ∀ξ ∈ R ExcessRisk θ(λ, ξ) ≥ (1 -β) • rγ 2 n • 0.99 2 5 • s r s 1 12 .Using Condition #4 in the above gives the desired eq (10) .We point the reader to Appendix A for notations used throughout the proof.Proof. Under the condition of ∀j ∈ [r], s j = 1, we need to analyze the ExcessRisk for both k-step SD and ridge. Denote Q := r j=1 θ ⋆ j for simplicity. k-step SD: Since assumption 2.1 is violated, we cannot use Theorem 4 for the k-step SD. Instead, we will work with the quadratic expansion of ExcessRisk from Theorem 5. We will rewrite the expressions from Appendix I for the quadratic coefs. Note that ρ j = λ/s 2 j = λ for all j ∈ [r] becomes independent of j in this case. And soC j (i) = 1 -1 (1+λ) i for all j ∈ [r], i ∈ [k] also becomes independent of j. Let C(i) := 1 -1 (1+λ) i denote the coefs C j (i), since they're now independent of j. Let ω := [C(1), C(2), • • • , C(k)] ∈ R k . Then, we get ∀(i 1 , i 2 ) ∈ [k] × [k] M (k) i1,i2 = 1 n r j=1 θ ⋆ j + γ 2 (1 + λ) 2 • C(i 1 ) • C(i 2 ) = 1 n • Q + rγ 2 (1 + λ) 2 • C(i 1 ) • C(i 2 ) =⇒ M (k) = 1 n • Q + rγ 2 (1 + λ) 2 • ωω ⊤ ∈ R k×k becomes a rank-1 matrixSimilarly, for m (k) we get∀i 1 ∈ [k] m (k) i1 = 1 n r j=1 λθ ⋆ j -γ 2 (1 + λ) 2 • C(i 1 ) =⇒ m (k) = 1 n • λQ -rγ 2 (1 + λ) 2 • ω ∈ R kAnd similarly, for c we getc = 1 n • λ 2 Q + rγ 2 (1 + λ) 2 ∈ RUsing the above expressions, we rewrite the overall ExcessRisk for any k-step SD (k ≥ 1) asExcessRisk θ(λ, ξ (k) ) = 1 n Q + rγ 2 (1 + λ) 2 • ⟨ω, ξ(k) ⟩ 2 + 2 • λQ -rγ 2 (1 + λ) 2 • ⟨ω, ξ(k) ⟩ + λ 2 Q + rγ 2 (1 + λ) 2We're aiming for a lower bound on the above quantity, so we can first minimize with respect to ξ (k) , and then analyze the remaining as a function of λ. Note that this is a quadratic in the scalar ⟨ω, ξ(k) ⟩.Since q(x):= ax 2 + 2bx + c = c -b 2 a + a x + b a 2 ≥ c -b 2 a , we have ∀k ≥ 1, ∀ξ (k) ∈ R k ExcessRisk θ(λ, ξ (k) ) ≥ 1 n • 1 (1 + λ) 2 λ 2 Q + rγ 2 - λQ -rγ 2 2 Q + rγ 2 = 1 n • 1 (1 + λ) 2 λ 2 Qrγ 2 + Qrγ 2 + 2λQrγ 2 Q + rγ 2 = 1 n • Qrγ 2 Q + rγ 2 = rγ 2 n • 1 1 + rγ 2 QNote this expression is independent of λ. Hence the above lower bound holds ∀k ≥ 1, ∀ξ (k) ∈ R k , ∀λ > 0. This concludes the proof of eq (12) .Ridge: For ridge, we can simply borrow the expression of c from above for its ExcessRisk. That isExcessRisk θ(λ) = 1 n • λ 2 Q + rγ 2 (1 + λ) 2Let λ ⋆ denote its minimizer over λ > 0. Simple calculus gives λ ⋆ = rγ 2 Q . Plugging this in, we get the same expressionExcessRisk θ(λ ⋆ ) = rγ 2 n • 1 1 + rγ 2 QThis completes the proof of ridge achieving the lower bound in eq (12) .We point the reader to Appendix A for notations used throughout the proof.Proof. Under the condition of ∀j ∈ [r], θ ⋆ j = z > 0, we need to analyze the ExcessRisk for both k-step SD and ridge. k-step SD: Again from Theorem 4, any k-step SD estimator's ExcessRisk is∀k ≥ 1, ∀λ > 0, ∀ξ (k) ∈ R k , ExcessRisk θ(λ, ξ (k) ) ≥ γ 2 n r j=1 θ ⋆ j θ ⋆ j + γ 2 s 2 j ≥ γ 2 n r j=1 1 1 + γ 2 z • 1 s 2 j (Since ∀j ∈ [r], θ ⋆ j = z)This completes the proof of eq (13) .Ridge: Now for the ridge estimator, we will use Lemma I.1. From eq (61), we get an exact expression for λ ⋆ > 0 that minimizes the ExcessRisk. Namely λ ⋆ = γ 2 z Substituting this in eq (60), we getExcessRisk θ(λ ⋆ ) = 1 n r j=1 (λ ⋆ ) 2 θ ⋆ j + γ 2 s 2 j • s 2 j (λ ⋆ + s 2 j ) 2 = γ 2 n r j=1 λ ⋆ + s 2 j • s 2 j (λ ⋆ + s 2 j ) 2 (Since λ ⋆ θ ⋆ j = γ 2 z z = γ 2 ) = γ 2 n r j=1 1 (1 + λ ⋆ s 2 j ) = γ 2 n r j=1 1 (1 + γ 2 z • 1 s 2 j ) (Substituting λ ⋆ = γ 2 z )This completes the proof of ridge achieving the lower bound in eq (13) .We point the reader to Appendix A for notations used throughout the proof.Proof. The proof of ( 14) is a simple instantiation of Lemma 4.1. Since the SD estimator is a particular instance of the general θ(P), i.e.θ λ, ξ (k) = θ P ← P λ, ξ (k) • Ω -1 λEq (14) follows from the lower bound in Lemma 4.1.For proving the equality being achieved, we will work with the general k-step SD estimator, and show that: Under assumption 2.1, k = r steps are sufficient to provide enough freedom to ξ (k) so that the k-step SD estimator achieves the lowest possible ExcessRisk.Using lemma 4.1, note that the condition P λ, ξ (k) • Ω -1 λ = P ⋆ is sufficient to ensure that θ λ, ξ (k) = θ(P ⋆ ), which would mean that the k-step SD estimator admits the lowest possible ExcessRisk. Since the eigenspaces for both sides of the equation are the same (U d ), we only need to ensure that the eigenvalues match on both sides. That is, we need the following condition (for indices j ≥ r + 1, there's no condition since lemma 4.1 tells us that any real value of sj suffices).EQUATION)Let a j (λ) :=s 2 j λ+s 2 j . Since λ > 0, a j ∈ (0, 1), ∀j ∈ [r].The above condition can then be written succinctly in matrix form asEQUATIONWith the following describing the elements ofA (k) (λ), α(λ) as [α(λ)] j := 1 -λ + s 2 j s⋆ j j ∈ [r] A (k) (λ) j,i := 1 -(a j (λ)) i j ∈ [r], i ∈ [k]The notation A (k) (λ), α(λ) explicitly denotes the dependence on λ. Now, A (k) (λ) being invertible would ensure that ∃ ξ(k) ∈ R k that makes equation (57) hold true (i.e. the system of equations admits a solution). For that, we need (1) k = r and (2) A (r) (λ) being full-rank. The first condition is stated in the lemma. In what follows, we will prove that ∃λ > 0 such that the second condition is satisfied, i.e. A (r) (λ) being full-rank. Decompose A (r) (λ) asA (r) (λ) =     1 1 • • • 1 1 1 • • • 1 . . . . . . . . . . . . 1 1 • • • 1     r×r W -     a 1 (λ) a 1 (λ) 2 • • • a 1 (λ) r a 2 (λ) a 2 (λ) 2 • • • a 2 (λ) r . . . . . . . . . . . . a r (λ) a r (λ) 2 • • • a r (λ) r     r×r V (λ)Where W = 11 ⊤ is the matrix of all ones, and V (λ) is akin to the square Vandermonde matrix (only difference being that the standard definition of Vandermonde also has a column of ones).Using the Matrix Determinant Lemma, we can writedet(A (r) (λ)) = 1 -1 ⊤ V (λ) -1 1 • det(-V (λ))First note that, with the determinant expansion rule based on the first rowdet V (λ) = det 1 0 ⊤ 1 V (λ) (r+1)×(r+1)The matrix on the right is exactly the Vandermonde matrix with a 0 = 0, a 1 (λ), a 2 (λ), • • • a r (λ).Using the formula for the det of a standard (with a row of ones) Vandermonde matrix, we getdet V (λ) = 0≤i<j≤r (a j (λ) -a i (λ)) = 1≤i≤r a i (λ) • 1≤i<j≤r (a j (λ) -a i (λ))Since a j (λ) ∈ (0, 1) for all j ∈ [r], and a j (λ)'s are all distinct, we conclude that det V (λ) ̸ = 0, i.e. V (λ) is full-rank. What remains to show is that the scalar 1 -1 ⊤ V (λ) -1 1 is non-zero for some λ > 0. With the SVD of V (λ) = CDE -1 where C, E are orthonormal and D is diagonal with positive entries, one can expand this term as:1 ⊤ V (λ) -1 1 = (E -1 1) ⊤ D -1 (C -1 1) =⇒ |1 ⊤ V -1 1| ≥ 1 d max • |⟨C -1 1, E -1 1⟩|Now observe that as λ → ∞, a j (λ) → 0 for all j ∈ [r], which means that V (λ) → 0 r×r , which means that (1) d max → 0 + , and (2) C ↔ E (since V (λ) becomes closer to a symmetric matrix, allowing its left and right singular matrices to approach equality to each other). That is, 1 dmax → ∞ and ⟨C -1 1, E -1 1⟩ → ∥1∥ 2 = r. This would ensure that |1 ⊤ V (λ) -1 1| > 1, meaning that the scalar (1 -1 ⊤ V (λ) -1 1) would be non-zero. Hence ∃λ > 0, such that A (r) (λ) is full-rank.H.1 Proof of Lemma 4.1 Proof. The estimator θ(P) expands asθ(P) = U d SS 2 d U ⊤ d θ ⋆ + U d SS d V ⊤ d η . Expanding the Excess Risk shows ExcessRisk θ(P) = d j=1 sj • s 2 j -1 2 • θ ⋆ j • w j Bias + γ 2 d j=1 s2 j • s 2 j • w j V ariance, where w j = I Quadratic ExcessRisk: detailed version of Theorem 5 and a proofWe point the reader to Appendix A for notations used throughout the proof.Theorem 7 (Formal version of Theorem 5). The Excess Risk is quadratic in ξEQUATION)where the below holds, for indices (i1 , i 2 ) in [k] × [k], M (k) i1,i2 = λ n r j=1 θ ⋆ j ρ j (1 + ρ j ) 2 • C j (i 1 ) • C j (i 2 ) B (k) i 1 ,i 2 + γ 2 n r j=1 1 (1 + ρ j ) 2 • C j (i 1 ) • C j (i 2 ) V (k) i 1 ,i 2 = 1 n r j=1 λθ ⋆ j ρj + γ 2 (1 + ρ j ) 2 • C j (i 1 ) • C j (i 2 ) , m (k) i1 = λ n r j=1 θ ⋆ j (1 + ρ j ) 2 • C j (i 1 ) b (k) i 1 + γ 2 n r j=1 (-1) (1 + ρ j ) 2 • C j (i 1 ) v (k) i 1 = 1 n r j=1 λθ ⋆ j -γ 2 (1 + ρ j ) 2 • C j (i 1 ) , c = λ n r j=1 θ ⋆ j ρ j (1 + ρ j ) 2 From Bias + γ 2 n r j=1 1 (1 + ρ j ) 2 From Variance = 1 n r j=1 λθ ⋆ j ρ j + γ 2 (1 + ρ j ) 2 .The B, b and V, v notation is used to indicate the (squared) Bias and Variance terms respectively. Here ρ j , j ∈ [r] is used to simplify the notation, and is defined asρ j := λ s 2 j , j ∈ [r].And the coefs C j , j ∈ [r] with i ∈ [k] have the formEQUATIONProof. We first use the expansion from Lemma D.1. Secondly, expand θ ⋆ = d j=1 ⟨θ ⋆ , u j ⟩u j . Using these, we can expand Excess Risk asExcessRisk θ(λ, ξ (k) ) = E η θ(λ, ξ (k) ) -θ ⋆ 2 Σn = d j=1 s 2 j n Σn weighing •θ ⋆ j • s 2 j λ + s 2 j 2 •   λ s 2 j + k i=1 ξ(k) i •    1 - s 2 j λ + s 2 j i      2 + d j=1 s 2 j n Σn weighing •γ 2 • s 2 j (λ + s 2 j ) 2 •   -1 + k i=1 ξ(k) i •    1 - s 2 j λ + s 2 j i      2Writing down the above expansion and collecting the corresponding quadratic, linear and constant terms' coefficients in the ξ(k) , give the desired expressions.Since we will compare the ridge estimator's ExcessRisk to the k-step SD estimator, we state the ExcessRisk expression for the ridge estimator (eq (3)) formally here.Lemma I.1. The ridge estimator θ(λ) satisfiesEQUATION)And the optimal penalty λ ⋆ > 0 that minimizes this ExcessRisk satisfiesEQUATIONProof. Eq (60) is a simple instantiation of Theorem 5 in the vacuous case of k = 0. Specifically, the quantity c captures exactly what we need. Borrowing its expression from the detailed theorem statement (Appendix I) gives us eq (60). Now, taking the derivative of the expression in eq (60) and setting it to zero, we get the stated expression for λ ⋆ > 0.This section follows up Section 5.1 with more details and examples about the synthetic problem.Figure 8 shows four more settings, with γ ∈ {0.125, 0.25} and θ ⋆ ∈ {u 1 , 1 / √ 2(u 1 + u 2 )}.Figure 8a validates that repeated steps of SD do provide a reduction in the excess risk, since the lowest point of the curve for each k is reducing as k increases. Observe that at k = r = 4 steps of SD, the curves in Figure 8 become flat. This is because we stated Theorem 4 with a "∃λ > 0" such that r-step SD can achieve the lower bound, but in practice it can happen "∀λ > 0". For the k-step SD estimator (eq (8)), for any chosen λ, Theorem 5 tells us that the ExcessRisk is quadratic in ξ(k) ∈ R k . To estimate the coefficients of this quadratic from certain chosen evaluations of ξ (k) , we need a total of k(k+3) /2 + 1 evaluations. This is because the number of unknown coefficients are (i) k for square terms, (ii) k(k-1) /2 for cross-square terms, (iii) k for linear terms, and (iv) 1 for the constant term. After estimating the coefficients of the quadratic, we only need to perform a grid search over one parameter, which is λ. We now illustrate this in detail for k = 1, 2.For 1-step SD, we know that the ExcessRisk( θ(λ, ξ)) = Aξ 2 + 2Bξ + C for unknown A, B, C (that depend on λ). Training 3 specific 1-step SD estimators, with ξ = {-1, 0, 1}, and measuring each of those 3 estimators' Risk on the validation set, lets us estimate A, B, C. We then know that ξ ⋆ = -B /A, for the chosen value of λ. We explicitly mention the field names (all real-numbered non-negative).k = 2 S 0 S 1 S 2 ξ 1 ξ 2• X covariates' names in the dataset: [PT08.S1(CO), PT08.S2(NMHC), PT08.S3(NOx), PT08.S4(NO2), PT08.S5(O3), T, RH, AH]• Y target's name in the dataset: NO2(GT) UCI Airfoil Self-Noise. The UCI Airfoil Self-Noise dataset [5] contains data obtained from NASA's aerodynamic and acoustic tests of airfoil blade sections. It contains 5 real-valued covariates, which are relevant physical quantities. The target is also real-valued, which is the sound pressure created (in dB). There are no missing entries. This dataset has n = 1503 rows and d = 5 covariates. We explicitly mention the field names:• X covariates' names in the dataset: [frequency, attack-angle, chord-length, free-stream-velocity, suction-side-displacement-thickness]• Y target's name in the dataset: scaled-sound-pressure UCI Appliances Energy Prediction. The UCI AEP dataset [6] contains energy appliances' data collected at 10 min frequency for about 4.5 months. This dataset has no missing entries, and a total of 19735 instances with 28 covariates. We downsample the dataset to hourly frequency, giving n = 3290 rows with d = 24 relevant covariates (removing degenerate covariates: 'date', 'lights', 'windspeed', 'visiblity'). The full list of covariates is 24 long, and so we do not list it out here (we already mentioned the 4 we removed from the total of 28).The goal here is to compute the alignment between the θ ⋆ and the eigenbasis directions {u j } d j=1 for the three real-world datasets. The eigenspace U d is known from the design matrix X, but θ ⋆ is unknown for real-world tasks. As a proxy, we use the ridge solution θλ (with a small λ).Methodology of choosing λ: We considered using the OLS solution θOLS := (XX ⊤ ) -1 XY as the proxy for θ ⋆ , but (XX ⊤ ) -1 was numerically unstable for these datasets, so we instead used the ridge solution θλ with a small λ. We calculated this λ methodically for all datasets as a constant fraction of the sum of squared singular values. Explicitly, we (i) computed the SVD of the design matrix X, and (ii) set λ := C • d j=1 s 2 j using the obtained singular values. The value C := 10 -5 was chosen arbitrarily (and the above trend is stable across other reasonably small values of C).Figure 10 shows the result. The sum of all bars in a single plot is one, since {u j } d j=1 are unit-norm vectors that form an orthogonal basis of R d . We infer two things. Firstly, for multi-step SD to outperform ridge (as is the case for the Air Quality and Airfoil datasets), θ ⋆ can be well-aligned with any of the u j , j ∈ [d]; not necessarily u 1 . Secondly, this gives insight into why multi-step SD could not outperform ridge on the AEP task (Table 1 ). Unlike the other two datasets, the θ ⋆ for the AEP dataset is not strongly aligned with any of the u j , j ∈ [d]. The top component in AEP only explains ∼ 35% of the total θ ⋆ norm, whereas that number is close to ∼ 80% for the Air Quality and Airfoil datasets. This indicates a large β value for the AEP dataset in Assumption 2.2.In this section, we show that the optimal ξ values given in Table 1 , selected via the strategy in section 5.2, are indeed the ones that minimize the validation MSE. By inspecting the minima of each of the curves in Figure 11 , we observe that the values of ξ ⋆ for 1-step SD in Table 1 (found through the strategy described in section 5.2) indeed coincide with the minima producing values in the below curves. Further, Figure 11 empirically demonstrates the quadratic nature of risk (MSE) vs ξ described in Theorem 5. Justification: We present a transparent top-level summary of all our results in the abstract and introduction.Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 


Decision: The authors provide a theoretical analysis of repeated (multi-step) self-distillation. This is a variation of the standard distillation setting where labels from a teacher model are used to augment the standard training objective; but this distillation is iterated for several rounds (i.e., distilled students become teachers for the next round of distillation.) For the simplified setting of ridge regression (along with additional assumptions), the authors show that if r is the rank of the data matrix, repeated self-distillation outperforms standard ridge regression by a factor of r.

The overall result is nice and rests on somewhat interesting connections between repeated self-distillation and matrix pre-conditioning in ridge regression. The authors also do a good job in justifying their additional assumptions. The major limitation (of course) is that distillation is usually done for much more complicated models and for typically classification using the cross-entropy loss, so it is as yet an open question whether any of the theoretical insights carry over to practice.

Reviews: review: paper_summary: paper_summary: This paper explores self-distillation from a theoretical perspective in the context of linear regression. Distillation is when a model is trained simultaneously to predict training labels and the predictions of another model that has already been trained on the data. Self-distillation is when the trained model has the same architecture as the model being trained. There has been recent work empirically show that self-distillation can result in better models and this paper gives a theoretical understanding of multi-step self-distillation in the context of linear regression.

The starting insight of the paper is that repeated self-distillation can be thought of as a pre-conditioner matrix multiplied by the optimal solution to ridge regression. Using this insight, they prove that there is a linear regression problem where self-distillation repeated r times (the rank of the data matrix) can outperform one-step self-distillation and ridge regression by a factor of r. They measure performance in terms of "excess risk" which is the distance to the optimal parameters under a norm weighted by the covariance matrix.

Their example with the factor of r gap requires two assumptions: all the singular values of the data matrix are distinct and the optimal parameters are orthogonal to all but one eigenvector. They show theoretically that both assumptions are necessary for such a gap to exist. They also derive an expression for the excess risk and conjecture that using more than r steps doesn't help the self-distillation process (as long as the self-distillation parameters are chosen optimally?).

They have some experiments where they show that 2-step self-distillation is robust to the ridge regression hyperparameter lambda. Because finding optimal parameters for self-distillation is difficult on real data, they do not show the performance of 3 or more -step self-distillation.
strength_weakness: strength_weakness: Larger weaknesses:

* The paper only shows results in terms of excess risk. I'm not familiar enough with the fixed design setup to determine how useful excess risk is.

* The paper shows that r-step self-distillation can be very powerful but they give no way to practically run it i.e., choose the parameters on real data.

Smaller points to improve:

* I'm confused by the statement of Assumption 2.1. Shouldn't this say that the optimal parameter is orthogonal to every eigenvector *except* the first one?

* More generally, I was confused about the way you stated the assumptions. You referred to two necessary assumptions which I assumed would be Assumption 1 and Assumption 2 but they were actually sub-assumptions in Assumption 2. I would ask that you rework what you call these assumptions and how you refer to them for clarity.

* In your appendix, you referred back to equations in the main body without reproducing them. This made it annoying to check your work e.g., you compared equation 8 and 26 but I kept having to flip between them (until I eventually copied equation 8). Since there's no page limit in the appendix, I would appreciate if you reproduced all equations when you refer to them.

* Your figures are difficult to read when printed out in black and white (and, I assume, for color blind people). Could you please change the marker for each line so they're more distinguishable?
questions: questions: * You show a gap between 1- and r-step self-distillation. Is there a similar gap between 2-step and r-step self-distillation? Based on your experiments, it looks like 2-step is already very good. Maybe this is a setting where 2-steps basically gives you all the power of r-steps like e.g., load balancing and the "power-of-2 choices".

* You justify using excess risk because it "ensures that the signal-to-noise ratio is uniform in all directions." I don't understand this and I don't see a direct relationship between excess risk and MSE. Could you please tell me why I should care about excess risk? And show me where it comes from theoretically?
limitations: limitations: Yes
paper_summary: paper_summary: The paper tries to provide a theoretical analysis of gains from applying self-distiliation repeatedly, in particular by trying to show that it is important to optimally set the imitation paramter for each step rathen than having a fixed Value. The study is conducted on using ridge estimator for linear regressin. The authors provide a theorm on the excess risk under two assumption when multi-step SD is applied. Experimental results on synthetic data and thre real world datasets. The paper also provides insights on how to set the imitation pramter real datasets.
strength_weakness: strength_weakness: The paper does not provide neither discussion nor experiments on how the proposed analysis can be used in more complex setting (e.g. more expressive models). On the other hand, for the experimental results the author do not provide any insight into why on AEP dataset using multi-step SD degrades the results. Could it be due to violation of the assumptions? 
In general, the paper provide theoretical analysis for a known technique without providing much insight into how one can use it to improve experimental work, for example it could have been more interesting if the authors could provide more principled techniques for hyperparameter setting for multi-step SD.  It would be more helpful if more experimental analysis on either more datasets or in depth were conducted.
questions: questions: - Can the authors provide any thoughts on how/challenges of expanding the analysis into more complex family of models?
- How does the Assumption 2.2 can be interpreted as constraints on more complex models, for example for MLPs?
- What could be the reason for drop in performance for AEP?
limitations: limitations: The theoretical analysis is limited in the scope of models and it is not immediately clear how it can be extended into more expressive models. Furthermore, it is not straightforward to incorporate them on large problems by optimally setting hyperparameters.
paper_summary: paper_summary: The paper analyzes the gains a model can achieve from multi-step self-distillation and presents a solid theory showing that the excess risk does not increase with more self-distillation steps. The synthetic task in the paper effectively proves this analysis.
strength_weakness: strength_weakness: 1) The study of self-distillation has already been explored in [1,2]. [1] shows that any distillation step can be calculated directly from the initial distillation step. Can the authors compare to [1]? In the experiments of [1], it is shown that accuracy does not always increase with more self-distillation steps. In this paper, the experiments only show results for 2-step self-distillation. How about more steps, like 5 or $\frac{r}{2}$? The datasets used in the experiments do not seem to be widely used. How about results on widely used datasets like CIFAR-10?

2) Assumption 2 is not guaranteed in real tasks. As shown in Figure 2(c), the proposed analysis heavily relies on Assumption 2.2. However, Assumption 2.2 does not seem to be satisfied in real tasks, which weakens the proposed analysis.

3) The experiments are not sufficient to support the proposed theory.

[1] Kenneth Borup and Lars N.Andersen. Even your Teacher Needs Guidance: Ground-Truth Targets Dampen Regularization Imposed by Self-Distillation. In Advances in neural information processing systems, 2020

[2] Rudrajit Das, Sujay Sanghavi. Understanding self-distillation in the presence of label noise. In Proceedings of the 40th International Conference on Machine Learning, 2023.
questions: questions: 1) Please address the weaknesses mentioned.

2) What networks are used in the experiments?
limitations: limitations: Yes, imitations have been discussed.
paper_summary: paper_summary: This paper theoretically investigates the effect of multiple rounds of self-distillation (SD) in linear regression. Under some conditions on the ground truth model $\theta^{\ast}$ and the data matrix $X$, it is shown that multi-step SD can improve the risk bound by a factor of $r = \text{rank}(X)$. Specifically, this happens when the non-zero singular values of $X$ are all distinct and when $\theta^{\ast}$ is *perfectly* parallel to $u_1$, where $u_1$ is the leading singular vector of $X$. The authors show the necessity of such conditions. The improvement yielded by multi-step SD is demonstrated empirically on some simple regression tasks.
strength_weakness: strength_weakness: **1.** The condition of $\theta^{\ast}$ being *perfectly* parallel to $u_1$ is too strong and unrealistic in my opinion. This makes the results of this paper less interesting. A better result would be quantifying the gains of multi-step SD assuming the angle between $\theta^{\ast}$ and $u_1$ is bounded by some quantity, say $\beta$ -- this is a more realistic setting. Then, focus on the special case of $\beta$ being small (and so $\beta = 0$ would be a very special case of these general results). Is it possible to do something like this?

**2.** There are no practically relevant insights for multi-step SD in classification problems which are more interesting now.

**3.** Overall, I'm not sure about the relevance of the results in this paper due to the above two points.
questions: questions: Please see Weakness #1.
limitations: limitations: Not discussed in detail.

Comments: title: Thank you
Thank you
Thank you
Thank you
Official Comment by Reviewer 2wSn
Addressing the smaller points to improve
comment: Thank you for your review, and for considering our rebuttal to your concerns. Please let us know if you have any more questions!
Thank you for your review, especially for the suggestion of analyzing the characteristics of the AEP dataset. This provides valuable insights, and we will include this in the revised manuscript of this work.
Thank you for your in-depth review, as well as for considering our rebuttal points.
Thank you for your review and the valuable suggestion of $\beta$-controlled analysis. We appreciate your reconsideration of this work in light of the rebuttal!
Thank you for the reply. Most of my concerns have been addressed.
Thank you for the response; I would retain my rating and recommend the clarification on the characteristics of AEP dataset to be include in the main body of the paper.
Thank you for your thorough response! I will retain my score.
Thanks for the rebuttal! The revised version of Theorem 1 is what I was looking for. I agree that this is a nice result. So I have updated my score accordingly.
**Smaller points to improve**

Thank you for these comments. We address them below.

1. **Assumption 2.2 as stated actually *implies* that $\theta^\star$ is orthogonal to every eigenvector except the first one.** We state Assumption 2.2 as $\measuredangle (\theta^\star, \mathbf{u}_1) = 0$, which means that $\theta^\star$ is perfectly parallel to $\mathbf{u}_1$. Since $\mathbf{u}_j, j \in [d]$ are all eigenvectors of $\mathbf{X} \mathbf{X}^T$, they form an *orthonormal basis* of $\mathbb{R}^d$ (also stated in Appendix A), i.e. $\langle \mathbf{u}_i, \mathbf{u}_j \rangle = 0$ for $i, j \in [d], i \neq j$. So indeed, the assumption as stated implies that $\langle \theta^\star, \mathbf{u}_j \rangle = 0$ for $j \in \{2, 3, \cdots, d\}$. There is a separate discussion on whether this condition is too strong, which we address in the shared (global) response window.
2. Thank you for catching this! We agree that a clearer way of stating this would explicitly call out the sub-assumptions 2.1 and 2.2. We will work this out in the final draft.
3. Totally agreed, we will modify the appendix to have relevant equations handy.
4. Also a valid point. We will rework this to a friendlier plotting style (for example, plotting with different marker styles as opposed to different colors).