## 3 Methods
### Language Bias-Driven Self-Distillation  
The method aims to learn unbiased visual knowledge via the mutual learning of two debiased models so as to reduce VQA language bias. The training strategy, which can be integrated with the current debiased methods, consists of the mutual learning of two debiased models. A dataset is given \(\mathcal{D}=\left\{I_{i},Q_{i},a_{i}\right\}^{N}\) containing N triplets of images \(I_{i}\in\mathcal{I}\), questions \(Q_{i}\in\mathcal{Q}\) and answers \(a_{i}\in\mathcal{A}\), it can be input into two identical models with different random initializations, \(N1\) and \(N2\), and two probability vectors \(p\) can be predicted by the model, \(z\) means Softmax output.  
\[p_{1}^{k}(I_{i},Q_{i})=\frac{\exp\left(z_{1}^{k}\right)}{\sum_{k=1}^{K}\exp \left(z_{1}^{k}\right)} \tag{2}\]  
where \(k\) represents the number of outputs or classes of the neural network.  
At the same time, the VQA model is generally defined as multi-type. Therefore, for multiple types, the objective function of the training network \(N1\) is defined as the cross entropy error between the prediction and the correct label, as shown as follows, \(K\) means samples, \(M\) means classes and \(L_{\text{C}}\) means the cross entropy error:  
\[L_{\text{C}}=-\sum_{i=1}^{K}\sum_{m=1}^{M}I(a_{i},m)\log(p_{1}^{m}(I_{i},Q_{i})) \tag{3}\]  
In order to allow the two student models to learn unbiased visual features from each other (similar to self-knowledge distillation), the authors of this paper use KL divergence to constrain all the predictions, thus distilling the unbiased knowledge of the two models. The formula of KL divergence between \(N1\) and \(N2\) is shown as follows:  
\[D_{KL}(\mathbf{p}_{2}\|\mathbf{p}_{1})=\sum_{i=1}^{K}\sum_{m=1}^{M}p_{2}^{m}(I_{i},Q_{ i})\log\frac{p_{1}^{m}(I_{i},Q_{i})}{p_{1}^{m}(I_{i},Q_{i})} \tag{4}\]  
The two student models simultaneously start parameter optimization, and the optimization loss is shown as follows. The consistency constraint of the predictions of the two models can realize the mutual learning of unbiased knowledge between the two models.  
\[L_{N_{1}} =L_{\text{C}_{1}}+D_{KL}(\mathbf{p}_{2}\|\mathbf{p}_{1}) \tag{5}\] \[L_{N_{2}} =L_{\text{C}_{2}}+D_{KL}(\mathbf{p}_{1}\|\mathbf{p}_{2})\]  
Since KL divergence is asymmetric, it can be replaced by Jensen-Shannon (JS) divergence (a variation of KL divergence) to ensure the consistency constraint between the two student models. Such replacement will not affect the final precision of the model.  
\[L_{JS(\mathbf{p}_{1}\|\mathbf{p}_{2})}=\frac{1}{2}KL\bigg{(}\mathbf{p}_{1}\|\frac{\mathbf{p}_ {1}+\mathbf{p}_{2}}{2}\bigg{)}+\frac{1}{2}KL\bigg{(}\mathbf{p}_{2}\|\frac{\mathbf{p}_{1}+ \mathbf{p}_{2}}{2}\bigg{)} \tag{6}\]  
Moreover, all the current self-knowledge distillation models use student models with different random initializations. The strategy is effective because the model learns more complete sets of multi-view features. The authors of this paper also explore the case where two heterogeneous student networks serve as the student models. The heterogeneous networks have the same feature extraction structure, but they have different loss functions and network branches.