SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
x ^ {t + 1} = x ^ {t} - \gamma (\nabla f _ {\xi} (x ^ {t}) - \lambda \nabla f _ {\xi} (\theta)), \tag {7}
$$
with initialization  $x^0 \in \mathbb{R}^d$ , step size or learning rate  $\gamma > 0$ , teacher model's parameters  $\theta \in \mathbb{R}^d$  and distillation weight  $\lambda$ . To analyze the convergence behavior of iterates (7), we need to impose some assumptions in order to derive reasonable convergence guarantees. First, we assume that the problem (6) has a non-empty solution set  $\mathcal{X} \neq \emptyset$  and  $f^{*} \coloneqq f(x^{*})$  for some minimizer  $x^{*} \in \mathcal{X}$ .
Assumption 1 (Strong quasi-convexity). The function  $f\colon \mathbb{R}^d\to \mathbb{R}$  is differentiable and  $\mu$ -strongly quasi-convex for some constant  $\mu >0$ , i.e., for any  $x\in \mathbb{R}^d$  it holds
$$
f \left(x ^ {*}\right) \geq f (x) + \left\langle \nabla f (x), x ^ {*} - x \right\rangle + \frac {\mu}{2} \| x ^ {*} - x \| ^ {2}. \tag {8}
$$
Strong quasi-convexity [9] is a weaker version of strong convexity [35], which assumes that the quadratic lower bound above holds for at every point  $y \in \mathbb{R}^d$  instead of  $x^* \in \mathcal{X}$ . Notice that strong quasi-convexity implies that the minimizer  $x^*$  is unique<sup>1</sup>. A more relaxed version of this assumption is the Polyak-Łojasiewicz (PL) condition [40].
Assumption 2 (Polyak-Łojasiewicz condition). Function  $f\colon \mathbb{R}^d\to \mathbb{R}$  is differentiable and satisfies PL condition with parameter  $\mu >0$ , if for any  $x\in \mathbb{R}^d$  it holds
$$
\left\| \nabla f (x) \right\| ^ {2} \geq 2 \mu \left(f (x) - f ^ {*}\right). \tag {9}
$$
Note that the requirement imposed by the PL condition above is weaker than by strong convexity. Functions satisfying PL condition do not have to be convex and can have multiple minimizers [17]. We make use of the following form of smoothness assumption on the stochastic gradient commonly referred to as expected smoothness in the optimization literature [11, 10, 19].
Assumption 3 (Expected Smoothness). Functions  $f_{\xi}(x)$  are differentiable and  $\mathcal{L}$ -smooth in expectation with respect to subsampling  $\xi \sim \mathcal{D}$ , i.e., for any  $x \in \mathbb{R}^d$  it holds
$$
\mathbb {E} _ {\xi \sim \mathcal {D}} \left[ \| \nabla f _ {\xi} (x) - \nabla f _ {\xi} \left(x ^ {*}\right) \| ^ {2} \right] \leq 2 \mathcal {L} \left(f (x) - f ^ {*}\right) \tag {10}
$$
for some constant  $\mathcal{L} = \mathcal{L}(f,\mathcal{D})$
The expected smoothness condition above is a joint property of loss function  $f$  and data subsampling strategy from the distribution  $\mathcal{D}$ . In particular, it subsumes the smoothness condition for  $f(x)$  since (10) also implies  $\| \nabla f(x) - \nabla f(x^{*})\|^{2}\leq 2\mathcal{L}(f(x) - f^{*})$  for any  $x\in \mathbb{R}^d$ . We denote by  $L$  the smoothness constant of  $f(x)$  and notice that  $L\leq \mathcal{L}$ .
4.2. Convergence Theory and Partial Variance Reduction. Equipped with the assumptions described in the previous part, we now present our convergence guarantees for the iterates (7) for both strong quasi-convex and PL loss functions.
Theorem 1 (See Appendix C.2). Let Assumptions 1 and 3 hold. For any  $\gamma \leq \frac{1}{8\mathcal{L}}$  and properly chosen distillation weight  $\lambda$ , the iterates (7) of SGD with self-distillation using teacher's parameters  $\theta$  converge as
$$
\mathbb {E} \left[ \| x ^ {t} - x ^ {*} \| ^ {2} \right] \leq (1 - \gamma \mu) ^ {t} \| x ^ {0} - x ^ {*} \| ^ {2} + \frac {2 \sigma_ {*} ^ {2}}{\mu} \min  (\gamma , \mathcal {O} (f (\theta) - f ^ {*})), \tag {11}
$$
where  $\sigma_{*}^{2}\coloneqq \mathbb{E}[\| \nabla f_{\xi}(x^{*})\|^{2}]$  is the stochastic noise at the optimum.
Theorem 2 (See Appendix C.3). Let Assumptions 2 and 3 hold. For any  $\gamma \leq \frac{1}{4\mathcal{L}}\frac{\mu}{L}$  and properly chosen distillation weight  $\lambda$ , the iterates (7) of SGD with self-distillation using teacher's parameters  $\theta$  converge as
$$
\mathbb {E} \left[ f \left(x ^ {t}\right) - f ^ {*} \right] \leq (1 - \gamma \mu) ^ {t} \left(f \left(x ^ {0}\right) - f ^ {*}\right) + \frac {L \sigma_ {*} ^ {2}}{\mu} \min  (\gamma , \mathcal {O} (f (\theta) - f ^ {*})), \tag {12}
$$
If  $f(x^{*}) = f(x^{**})$ , then  $\frac{\mu}{2} \| x^{*} - x^{**} \| \leq f(x^{*}) - f(x^{**}) = 0$ . Hence,  $x^{*} = x^{**}$ .
Proof overview. Both proofs follow similar steps and can be divided into three logical parts.
Part 1 (Descent inequality). Generally, an integral part of essentially any convergence theory for optimization methods is a descent inequality quantifying the progress of an algorithm in one iteration. Our theory is not an exception: we first define our "potential"  $e^{t} = \| x^{t} - x^{*}\|^{2}$  for the strongly quasi-convex setup, and  $e^{t} = f(x^{t}) - f^{*}$  for the PL setup. Then, we start our derivations by bounding  $\mathbb{E}_t[e^{t + 1}] - (1 - \gamma \mu)e^t$ . Here,  $\mathbb{E}_t$  is the conditional expectation with respect to the randomness of previous iterate  $x^{t}$ . Specifically, up to constants, both setups allow the following bound:
$$
\mathbb {E} _ {t} \left[ e ^ {t + 1} \right] \leq (1 - \gamma \mu) e ^ {t} - \mathcal {O} (\gamma) (1 - \mathcal {O} (\gamma)) \left(f \left(x ^ {t}\right) - f ^ {*}\right) + \mathcal {O} (\gamma) N (\lambda), \tag {13}
$$
where  $N(\lambda) = \lambda^2\| \nabla f(\theta)\| ^2 +\gamma \mathbb{E}\left[\| \nabla f_{\xi}(x^{*}) - \lambda \nabla f_{\xi}(\theta)\| ^2\right]$ . Choosing the learning rate  $\gamma$  to be small enough, we ensure that the second term is non-positive and hence negligible in the upper bound.
Part 2 (Optimal distillation weight). Next, we focus our attention on the third term in (13) involving the iteration-independent neighborhood term  $N(\lambda)$ . Note that the  $\mathcal{O}(\gamma)$  factor next to  $N(\lambda)$  will be absorbed once we unfold the recursion (13) up to initial iterate. Hence, the convergence neighborhood is proportional to  $N(\lambda)$ . Now the question is how small this term can get if we properly tune the parameter  $\lambda$ . Notice that  $N(0) = \gamma \sigma_{*}^{2}$  corresponds to the neighborhood size for plain SGD without any distillation involved. Luckily, due to the quadratic dependence, we can minimize  $N(\lambda)$  analytically with respect to  $\lambda$  and find the optimal value
$$
\lambda_ {*} = \frac {\mathbb {E} [ \langle \nabla f _ {\xi} (x ^ {*}), \nabla f _ {\xi} (\theta) \rangle ]}{\mathbb {E} [ \| \nabla f _ {\xi} (\theta) \| ^ {2} ] + \frac {1}{\gamma} \| \nabla f (\theta) \| ^ {2}}. \tag {14}
$$
Consequently, the analysis puts further emphasis on the need for careful parametrization with respect to the weighting  $\lambda$  of the distillation loss as there exists a particularly privileged value  $\lambda_{*}$ .
Part 3 (Impact of the teacher). In the final step, we quantify the impact of the teacher on the reduction in the neighborhood term  $N(\lambda_{*})$  compared to the plain SGD neighborhood  $N(0)$ . Via algebraic transformations, we show that
$$
\frac {N \left(\lambda_ {*}\right)}{N (0)} = 1 - \rho^ {2} \left(x ^ {*}, \theta\right) \frac {1 - \beta (\theta)}{1 + \frac {1}{\gamma} \beta (\theta)}, \tag {15}
$$
where  $\beta (\theta) = \| \nabla f(\theta)\| ^2 /\mathbb{E}[\| \nabla f_{\xi}(\theta)\| ^2 ]\in [0,1]$  is the signal-to-noise ratio, and  $\rho (x^{*},\theta)\in [-1,1]$  is the correlation coefficient between stochastic gradients  $\nabla f_{\xi}(x^{*})$  and  $\nabla f_{\xi}(\theta)$ . This representation gives us analytical means to measure the impact of the teacher. For instance, the optimal teacher  $\theta = x^{*}$  satisfies  $\rho (x^{*},\theta) = 1$  and  $\beta (\theta) = 0$ , and thus  $N(\lambda_{*}) = 0$ . In general, if the teacher is not optimal, then the reduction (15) is of order  $\frac{1}{\gamma}\mathcal{O}(f(\theta) - f^{*})$  and  $N(\lambda_{*}) = \sigma_{*}^{2}\min (\gamma ,\mathcal{O}(f(\theta) - f^{*}))$ .
We discuss these results highlighting the key aspects and significance.
- Structure of the rates. The structure of these two convergence rates is typical in gradient-based stochastic optimization literature: linear convergence up to some neighborhood controlled by the stochastic noise term  $\sigma_{*}^{2}$  and learning rate  $\gamma$ . In fact, these results (including learning rate restrictions) are identical to ones for SGD [8] except the non-vanishing terms in (11) and (12) include an additional  $\mathcal{O}(f(\theta) - f^{*})$  factor due to distillation and proper selection of weight parameter  $\lambda$ . For both setups, the rate of SGD is the same (11) or (12) with only one difference:  $\min (\gamma ,\mathcal{O}(f(\theta) - f^{*}))$  term is replaced with  $\gamma$ . So,  $\mathcal{O}(f(\theta) - f^{*})$  is the factor that makes our results better compared to SGD in terms of optimization performance.
- Importance of the results. First, observe that in the worst case when the teacher's parameters are trained inadequately (or not trained at all), that is  $\mathcal{O}(f(\theta) - f^{*}) \geq \gamma$ , then the obtained rates recover the known results for plain SGD. However, the crucial benefit of these results is to show that a sufficiently well-trained teacher, i.e.  $\mathcal{O}(f(\theta) - f^{*}) < \gamma$ , provably reduces the neighborhood size of SGD without slowing down the speed of convergence. In the best case scenario, when the teacher's model is perfectly trained, namely  $f(\theta) = f^{*}$ , then the neighborhood term vanishes, and the method converges to the exact solution (see SGD-star Algorithm 4 in [9]). Thus, self-distillation in the form of iterates (7) acts as a form of partial variance reduction, which can reduce the stochastic gradient noise, but may not eliminate it completely, depending on the properties of the teacher model.
- Choice of distillation weight  $\lambda$ . As we discussed in the proof sketch above, our analysis reveals that the performance of distillation is optimized for a specific value (14) of distillation weight  $\lambda_{*}$  depending on the teacher model. One way to interpret the expression (14) for weight parameter
(a) Self-distillation on MNIST.
(b) Self-distillation on CIFAR-10.
Figure 2: The minimum training loss, and average over the last 10 epochs, for models trained with SGD and with self-distillation, using different values of the distillation parameter  $\lambda$ , on MNIST and CIFAR-10. SGD is equivalent to  $\lambda = 0$ . The curves for the SGD-based teachers (which do not have zero loss) reflect our analysis, corroborating the existence of the "optimal" distillation weight. By contrast, in the L-BFGS teacher, higher distillation weight always leads to lower loss.
intuitively is that the better the teacher's model  $\theta$  is, the bigger  $\lambda_{*} \leq 1$  gets. In other words,  $\lambda_{*}$  quantifies the quality of the teacher:  $\lambda_{*} \approx 0$  indicates a poor teacher model ( $f(\theta) \gg f^{*}$ ) and  $\lambda_{*} = 1$  is for the optimal teacher ( $f(\theta) = f^{*}$ ).
4.3. Experimental Validation. In this section we illustrate that our theoretical analysis in the convex case also holds empirically. Specifically, we consider classification problems using linear models in two different setups: training a linear model on the MNIST dataset [24] and linear probing on the CIFAR-10 dataset [23], using a ResNet50 model [12], pre-trained on the ImageNet dataset [42]. For the second setup we train a linear classifier on top of the features extracted from a ResNet50 model pre-trained on ImageNet. This is a standard setting, commonly used in the transfer learning literature, see e.g. [21, 45, 15]. In both cases we train using SGD without momentum and regularization, with a fixed learning rate and mini-batch of size 10, for a total of 100 epochs. The models trained with SGD are compared against self-distillation (Equation 7), using the same training hyper-parameters, where the teacher is the model trained with SGD. In the case of CIFAR-10 features, we also consider the "optimal" teacher, which is a model trained with L-BFGS [27]. We perform all experiments using multiple values of the distillation parameter  $\lambda$  and measure the cross entropy loss between student and true labels. At each training epoch we computing the running average over all mini-batch losses seen during that epoch.
The results presented in Figure 2 show the minimum cross entropy train loss obtained over 100 epochs, as well as the average over the last 10 epochs, for models trained with SGD, as well as with self-distillation, with  $\lambda \in [0.01, 1]$ . We observe that when the teacher is the model trained with SGD ( $\lambda = 0$ ), there exists a  $\lambda > 0$  which achieves a lower training loss than SGD, which is in line with our statement from Theorem 1. Furthermore, when the teacher is very close to the optimum,  $\lambda$  closer to 1 reduces the training loss the most compared to SGD, which is also in line with the theory (see Theorem 1). This behavior is illustrated in Figure 2b, when using an L-BFGS teacher.
5 Removing Bias and Improving Variance Reduction
In this section, we investigate the cause of having variance reduction only partially and suggest a possible workaround to obtain complete variance reduction. In brief, the potential source of partial variance reduction is the biased nature of distillation. Essentially, distillation bias is reflected in the iterates (7) since the expected update direction  $\mathbb{E}\left[\nabla f_{\xi}(x^{t}) - \lambda \nabla f_{\xi}(\theta)\mid x^{t}\right] = \nabla f(x^{t}) - \lambda \nabla f(\theta)$  can be different from  $\nabla f(x^t)$ . This comes from the fact that distillation loss (4) modifies the initial loss (1) composed of true outputs. To make our argument compelling, next we correct the bias by adding  $\lambda \nabla f(\theta)$  to iterates (7) and analyze the following dynamics:
$$
x ^ {t + 1} = x ^ {t} - \gamma (\nabla f _ {\xi} (x ^ {t}) - \lambda \nabla f _ {\xi} (\theta) + \lambda \nabla f (\theta)). \tag {16}
$$
Besides making the estimate unbiased, the advantage of this adjustment is that no tuning is required for the distillation weight  $\lambda$ ; we may simply set  $\lambda = 1$ . The obvious disadvantage is that  $\nabla f(\theta)$  is the batch gradient over the whole train data that can be very costly to compute. However, we could compute it once and reuse it for all further iterates. The resulting iteration is similar to the popular and well-studied SVRG [16, 57, 41, 20, 25, 22, 29] method, and therefore iterates (16) will enjoy full variance reduction.
Figure 3: (Left plot) The train loss of self-distillation, unbiased self-distillation and vanilla SGD training. (Right plot) The progress of gradient variances (averaged over the iterations within each epoch) for the same setup
Theorem 3 (See Appendix D). Let Assumptions 2 and 3 hold. Then for any  $\gamma \leq \frac{\mu}{3L\mathcal{L}}$  the iterates (16) with  $\lambda = 1$  converge as
$$
\mathbb {E} _ {t} \left[ f \left(x ^ {t}\right) - f ^ {*} \right] \leq (1 - \gamma \mu) ^ {t} \left(f \left(x ^ {0}\right) - f ^ {*}\right) + \frac {3 L (L + \mathcal {L})}{\mu} \cdot \gamma (f (\theta) - f ^ {*}). \tag {17}
$$
The key improvement that bias correction brings in (17) is the convergence up to a neighborhood  $\mathcal{O}(\gamma (f(\theta) - f^{*}))$  in contrast to  $\min (\gamma ,\mathcal{O}(f(\theta) - f^{*}))$  as in (11) and (12). The multiplicative dependence of learning rate and the quality of the teacher leads the method (16) to full variance reduction. Indeed, if we choose the teacher model as  $\theta = x^0$ , then the rate (17) becomes
$$
\mathbb {E} \left[ f \left(x ^ {t}\right) - f ^ {*} \right] \leq \left[ (1 - \gamma \mu) ^ {t} + \gamma \cdot 3 L (L + \mathcal {L}) / \mu \right] \left(f \left(x ^ {0}\right) - f ^ {*}\right) \leq \frac {1}{2} \left(f \left(x ^ {0}\right) - f ^ {*}\right),
$$
provided sufficiently small step-size  $\gamma \leq \frac{\mu}{12L\mathcal{L}}$  and enough training iterations  $t = \mathcal{O}(1 / \gamma \mu)$ . Hence, following SVRG and updating the teacher model in every  $\tau = \mathcal{O}(1 / \gamma \mu)$  training iterations, that is choosing  $\theta^m = x^{m\tau}$  as the teacher at the  $m^{th}$  distillation iteration (see line 3 of Algorithm 1), we have
$$
\mathbb {E} \left[ f (x ^ {m \tau}) - f ^ {*} \right] \leq \frac {1}{2} \left(f (x ^ {(m - 1) \tau}) - f ^ {*}\right) \leq \frac {1}{2 ^ {m}} \left(f (x ^ {0}) - f ^ {*}\right).
$$
Thus, we need  $\mathcal{O}(\log \frac{1}{\epsilon})$  bias-corrected distillation iteration phases, each with  $\tau = \mathcal{O}(1 / \gamma \mu)$  training iterates, to get  $\epsilon$  accuracy in function value. Overall, this amounts to  $\mathcal{O}\left(\frac{1}{\gamma\mu}\log \frac{1}{\epsilon}\right)$  iterations of (16).
Experimental Validation. Similarly to the previous section, we further validate empirically the result from Theorem 3. Specifically, we consider the convex setup described before, where we train linear models on features extracted on the CIFAR-10 dataset. Based on Figure 2b, we select  $\lambda = 0.4$  achieving the largest reduction in train loss, compared to SGD, and we additionally perform unbiased self-distillation (Equation 16), using the same training hyperparameters. Similar to the setup from Figure 2, we measure the cross entropy train loss of the student and with the true labels, which is computed at each epoch by averaging the mini-batch losses. The results are averaged over three runs and presented in Figure 3. The first plot on the left shows that, indeed, the unbiased self-distillation update further reduces the training loss, compared to the update from Equation 7. The second plot explicitly tracks gradient variance (averaged over the iterations within each epoch) for the same setup. As expected, both variants of KD (biased and unbiased) have reduced gradient variance compared to plain SGD. The plot also highlights that both variants of KD have similar variance reduction properties, while the unbiasedness of unbiased KD amplifies the reduction of train loss.
6 Convergence for Distillation of Compressed Models
So far, the theory we have presented is for self-distillation, i.e., the teacher's and student's architectures are identical. To understand the impact of knowledge distillation, we relax this requirement and allow the student's model to be a sub-network of the larger and more powerful teacher's model. Our approach to model this relationship between the student and the teacher is to view the student as a masked or, in general, compressed version of the teacher. Hence, as an extension to (7) we analyze the following dynamics of distillation with compressed iterates:
$$
x ^ {t + 1} = \mathcal {C} \left(x ^ {t} - \gamma \left(\nabla f _ {\xi} \left(x ^ {t}\right) - \lambda \nabla f _ {\xi} (\theta)\right)\right), \tag {18}
$$
where student's parameters are additionally compressed in each iteration using an unbiased compression operator defined below.
Assumption 4. The compression operator  $\mathcal{C}:\mathbb{R}^d\to \mathbb{R}^d$  is unbiased and there exists finite  $\omega \geq 0$  bounding the compression variance variance, i.e., for all  $x\in \mathbb{R}^d$  we have
$$
\mathbb {E} [ \mathcal {C} (x) ] = x, \quad \mathbb {E} [ \| \mathcal {C} (x) - x \| ^ {2} ] \leq \omega \| x \| ^ {2}. \tag {19}
$$
Typical examples of compression operators satisfying conditions (19) are sparsification [55, 47] and quantization [1, 56], which are heavily used in the context of communication efficient distributed optimization and federated learning [30, 44, 37, 53]. In this context, we obtain the following:
Theorem 4 (See Appendix E). Let smoothness Assumption 3 hold and  $f$  be  $\mu$ -strongly convex. Choose any  $\gamma \leq \frac{1}{16\mathcal{L}}$  and compression operator with variance parameter  $\omega = \mathcal{O}(\mu/\mathcal{L})$ . Then, properly selecting distillation weight  $\lambda$ , the iterates (18) satisfy
$$
\mathbb {E} \left[ \| x ^ {t} - x ^ {*} \| ^ {2} \right] \leq \mathcal {O} (\omega + 1) \left[ (1 - \gamma \mu) ^ {t} \| x ^ {0} - x ^ {*} \| ^ {2} + \frac {\omega \mathcal {L}}{\mu} \| x ^ {*} \| ^ {2} + \frac {\sigma_ {*} ^ {2}}{\mu} \min (\gamma , \mathcal {O} (f (\theta) - f ^ {*})) \right].
$$
Clearly, there are several factors influencing the speed of the rate and the neighborhood of the convergence that require some discussion. First of all, choosing the identity map as a compression operator  $(\mathcal{C}(x) = x$  for all  $x\in \mathbb{R}^d)$ , we recover the same rate (11) as before  $(\omega = 0$  in this case). Next, consider the case when the stochastic noise at the optimum vanishes  $(\sigma_{*}^{2} = 0)$  and distillation is switched off  $(\lambda = 0)$  in (18). In this case, the convergence is still up to some neighborhood proportional to  $\| x^{*}\|^{2}$  since compression is applied to the iterates. Intuitively, the neighborhood term  $\mathcal{O}(\| x^{*}\|^{2})$  corresponds to the compression noise at the optimum  $x^{*}$  ((19) when  $x = x^{*}$ ). Also note that the presence of this non-vanishing term  $\mathcal{O}(\| x^{*}\|^{2})$  and the variance restriction  $\omega = \mathcal{O}(\mu /\mathcal{L})$  is consistent with the prior work [18].
So, the convergence neighborhood of iterates (18) has two terms, one from each source of randomness: compression noise/variance  $\mathcal{O}\left(\| x^{*}\|^{2}\right)$  at the optimum and stochastic noise/variance  $\mathcal{O}(\sigma_{*}^{2})$  at the optimum. Therefore, in this case as well, distillation with a properly chosen weight parameter (partially) reduces the stochastic variance of sub-sampling.
Our work has provided a new interpretation of knowledge distillation, examining this mechanism for the first time from the point of view of optimization. Specifically, we have shown that knowledge distillation acts as a form of partial variance reduction, whose strength depends on the characteristics of the teacher model. This finding holds across several variants of distillation, such as self-distillation and distillation of compressed models, as well as across various families of objective functions.
Prior observations showed that significant capacity gap between the student and the teacher may in fact lead to poorer distillation performance [31]. To reconcile the issue of large capacity gap our results, notice that, in our case "better teacher" means better parameter (i.e., weights and biases) values, evaluated in terms of training loss. In particular, in the case of self-distillation, covered in Sections 4 and 5, the teacher and student architectures are identical, and hence they have the same capacity. In our second regime, distillation for compressed models (Section 6), we actually consider the case when the student network is a subnetwork of the teacher; we consider a sparsification compression operator that selects  $k$  parameters for the student out of  $d$  parameters of the teacher. Then, clearly, the teacher has a larger capacity with a capacity ratio  $d / k \geq 1$ . However, our result in this direction (Theorem 4) does not allow the capacity ratio to be arbitrarily large. Indeed, the constraint  $\omega = \mathcal{O}(\mathcal{L} / \mu)$  on compression variance implies a constraint on capacity ratio since  $\omega = d / k - 1$  for the sparsification operator. Thus, our result holds when the teacher's size is not significantly larger than the student's size, which is in line with the prior observations on large capacity gap.