## 3 Methods
### Theoretical Analysis
#### 3.4.2 Theoretical Analysis of Debiased Self-Distillation  
In this section, the authors of this paper demonstrate that self-knowledge distillation and generalized uncertainty can enable models to learn more complete multi-view feature sets and reduce language bias in VQA. The authors of this paper followed the research of [PERSON] and [PERSON] [12,19].  
**Notation 2**.: _Let us set up a model whose dataset contains \(K\) categories, \(p\)-input patch \(s\) and the ReLU function. The model input is \(I_{i},Q_{i}\) and the label is \(a_{i}\). To simplify the problem, the authors of this paper assume that each category contains related features that are orthogonal to each other. The authors of this paper define these features as vectors of \(vqa_{j,1},vqa_{j,2}\)._  
Following the settings of [PERSON] et al.'s research. The authors of this paper get the definitions as follows. The set of all features:  
\[X_{vqa}\ \stackrel{{\text{def}}}{{=}}\ \big{\{}vqa_{j,1},vqa_{j,2} \big{\}}_{j\in[k]} \tag{19}\]  
\[vqa_{j,\ell}\perp vqa_{j,\ell^{\prime}}vwhen(j,\ell)\
eq(j^{\prime},\ell^{ \prime}) \tag{20}\]  
**Definition 3**.: _(Data distribution) The authors of this paper define the multi-view and single-view distribution \(D_{vqa}\) and \(D_{vqa}\), \(D\in D_{vqa/s}\), and \((I_{i},Q_{i},a_{i})\sim D\). Sample features with probability \(s/k\), \(s\in\big{[}1,k^{0.2}\big{]}\). The coefficients \(z_{p}\), \(n_{p,vqa^{\prime}}\) is the feature noise, \(\xi_{p}\) is the random Gaussian noise. For each \(p\in[P]\backslash\mathcal{P}(I_{i},Q_{i})\), the authors of this paper set:_  
\[x_{p}=z_{p}vqa+\sum_{vqa^{\prime}\in X_{vqa}}n_{p,vqa^{\prime}}vqa^{\prime}+ \xi_{p} \tag{21}\]  
**Definition 4**.: _(The final data distribution \(D\) and the training dataset \(S_{d}\). Suppose \(D\) contains \(1-\mu\)\(D_{vqa}\) and \(\mu\)\(D_{vqa}\). For \(N\) samples in \(D\), the training dataset \(S_{d}=S_{dm}\cup S_{ds}\). \((I_{i},Q_{i},a_{i})\) random sampling from the set \(S_{d}\). \(\mu=\frac{1}{\text{poly}(k)}\), and \(N=k^{1.2}/\mu\)._  
**Definition 5**.: _The authors of this paper define a network \(VQA(I_{i},Q_{i})\) with a cross-entropy loss function using a stochastic learning algorithm as follows:_  
\[L(VQA)=\mathbb{E}_{(I_{i},Q_{i},a_{i})\sim S_{d}}[L(VQA;I_{i},Q_{i},a_{i})] \tag{22}\]  
_The logits function of the single model (\(\eta\leq\frac{1}{\text{poly}(k)},T=\frac{\text{poly}(k)}{\eta}\)) can be defined as_  
\[\text{logit}_{i}(VQA,I,Q)\ \stackrel{{\text{def}}}{{=}}\ \frac{e^{VQA_{i}(I,Q)}}{\sum_{j\in[k]}e^{VQA_{j}(I,Q)}} \tag{23}\]  
_The logits function of the model using knowledge distillation can be defined as_  
\[\text{logit}_{i}^{\tau}(VQA,I,Q)=\frac{e^{\min\big{\{}\tau^{2}VQA_{i}(I,Q),I \big{\}}/\tau}}{\sum_{j\in[k]}e^{\min\big{\{}\tau^{2}VQA_{j}(I,Q),I\big{\}}/\tau}} \tag{24}\]  
**Theorem 2**.: _For the single model, the authors of this paper use the prediction error as follows, \(\exists i\in[k]\backslash\{a\}\):_  
\[\underset{(I,Q,a)\sim D}{\mathbf{Pr}}\Big{[}VQA_{a}^{(T)}(I,Q)<VQA_{i}^{(T)}(I,Q )\Big{]}\in(0.5\pm 0.01)\mu \tag{25}\]  
**Theorem 3**.: _For self-knowledge distillation with the generalization uncertainty model, \(\lambda\) (\(\lambda>1\)) is the gain from generalized uncertainty. The authors of this paper use the prediction error as follows, \(\exists i\in[k]\backslash\{y\}\):_  
\[\underset{(I,Q,a)\sim D}{\mathbf{Pr}}\Big{[}VQA_{a}^{(T+T^{\prime})}(I,Q)<VQA _{i}^{(T+T^{\prime})}(I,Q)\Big{]}\leq 0.26\mu/\lambda \tag{26}\]  
_The authors of this paper find that when comparing Theorems 2 and 3, the prediction error of the model decreased. That means the LBSD method can reduce language bias. The detailed proof can be found in Appendix A._