## 3 Methods
### Theoretical Analysis  
#### 3.4.1 Theoretical Analysis of Generalization Uncertainty (GU)  
In this section, the authors of this paper demonstrate that the generalized uncertainty index between two student models can be used to estimate the model test error on image-text sample pairs. Thus, generalized uncertainty is used in the training process to help students learn unbiased knowledge. Following the research of [PERSON] and [PERSON] [57], [PERSON] et al. [58] and others [59; 60; 61], the authors of this paper use class-segregated calibration (or class-wise calibration) [58; 62; 63; 64; 65; 66] to prove the proportional relationship between the generalized uncertainty and the test error.  
**Notation 1**.: _The authors in this paper define two neural networks trained from different random seeds as \(n\), \(n^{\prime}\). The data of this model include \(K\) categories with input \(I_{i}\), \(Q_{i}\) and label \(a_{i}\) (\(Y\)). The model is parameterized by stochastic learning. The probability expression of the predicted output of the model is \(\mathbf{p}(\hat{a}\mid I,Q)\). \(D_{vqa}\) is the distribution map from \(I_{i},Q_{i}\times[K]\), and \(\mathbf{p}(\omega)\) is the sample estimate for the different parameter distribution of models. The parameters of the model can be defined as \(\Omega\). The \(1[\ldots]\) function is the indicator function, which means the prediction is true or otherwise._  
**Definition 1**.: _The model \(N\) (\(\mathbf{p}(\hat{a},\omega\mid I,Q)\)) satisfies the generalization uncertainty proportional (GLUP) on the distribution \(D_{vqa}\) if:_  
\[\begin{split} TestErr_{D_{vqa}}(n)E_{D_{vqa}}[1|n(I,Q)\
eq a] \propto\\  GUErr_{D_{vqa}}(n,n^{\prime})(TopK)E_{D_{vqa}}[1|n(I,Q)\
eq n^{ \prime}(I,Q)]\end{split} \tag{13}\]  
**Definition 2**.: _The self-knowledge distillation model \(N\) (\(n,n^{\prime}\)) satisfies class-wise calibration (or class-segregated calibration) on \(D_{vqa}\) if any kind of confidence value \(q\) falls in \([0,1]\) and for any class \(k\) falls in \([K]\),_  
\[p(a=k\mid\tilde{n}_{k}(I,Q)=q)=q \tag{14}\] \[\frac{\sum_{k=0}^{K-1}p(a=k,\tilde{n}_{k}(I,Q)=q)}{\sum_{k=0}^{K- 1}p(\tilde{n}_{k}(I,Q)=q)}=q \tag{15}\]  
**Theorem 1**.: _If the self-knowledge distillation model \(N\) (\(n,n^{\prime}\)) satisfies class-wise calibration (or class-segregated calibration) on \(D_{vqa}\), then \(N\) satisfies the generalization uncertainty proportional (GLUP) on \(D_{vqa}\)._  
\[\mathbb{E}_{\Omega,\Omega^{\prime}}\Big{[}\mathrm{GUErr}_{D_{vqa}}\big{(}n, n^{\prime}\big{)}\Big{]}\propto\mathbb{E}_{\Omega}\big{[}\mathit{TestErr}_{D_{vqa}}(n) \big{]} \tag{16}\]  
Proof.: The authors in this paper define the Expected Test error as TE (Test error). The TOP-K error of the two-student model with generalized uncertainty is fixed at GUE. By simplifying the two errors, the following results can be obtained, and the proportional relationship (GUP) between them can be obtained. Since \(K\) previously represented the number of categories, for this reason, the authors of this paper represent \(J\) as the \(K\) term in TOP-K, \(i\) as the corresponding prediction at the sample of \(J\)-th value.  
\[\mathrm{TE}=\int_{q\in[0,1]}q(1-q)\sum_{k=0}^{K-1}p(\tilde{n}_{k}(I,Q)=q)dq \tag{17}\]\[\begin{split}\text{GUE}&=\sum_{j=0}^{J}1/p_{i}\underbrace{ \sum_{k=0}^{K-1}\int_{q\in[0,1]}}_{\text{swap}}p(\bar{n}_{k}(I,Q)=q)q(1-q)dq\\ &=\sum_{j=0}^{J}1/p_{i}\int_{q\in[0,1]}q(1-q)\sum_{k=0}^{K-1}p( \bar{n}_{k}(I,Q)=q)dq\propto TE.\end{split} \tag{18}\]  
The detailed proof of generalization uncertainty can be found in Appendix A.