# Dual-Training-Based Semi-Supervised Learning with Few Labels
## 3 Our Approach
### Dual Training Strategy  
[PERSON] [16] found that training the model with strongly augmented data whose labels are generated by the model prediction of weakly augmented data can achieve great success for image classification. However, it set a threshold to select samples with high confidence and ignore other samples. In addition, the model converges slowly because of the high randomness of data augmentation. A large number of iterations is required for FixMatch to achieve satisfactory results. Therefore, we proposed a novel dual training strategy to fully utilize all data and help the model learn more meaningful information.  
Given a labeled dataset \(D_{l}=\left\{(x_{i}^{l},y_{i}^{l})\right\}_{i=1}^{M}\) and an unlabeled dataset \(D_{u}=\left\{x_{i}^{u}\right\}_{i=1}^{N}\), where \(M\) and \(N\) respectively represent the number of labeled and unlabeled data, \(x_{i}^{l}\) and \(y_{i}^{l}\) are the \(i^{th}\) labeled datum and its label. \(x_{i}^{u}\) is the \(i^{th}\) unlabeled datum. For the unlabeled dataset, we first augment the data with weak data augmentation and fit them into the model to obtain the predictions, which are used as the soft labels.  
\[pl_{i}^{soft}=h(g(f(x_{i}^{u,w}))) \tag{1}\]  
Inspired by the effectiveness of distribution alignment [21; 36], we also apply the distribution alignment to soft labels to improve their accuracy:  
\[pl_{i}^{soft}=pl_{i}^{soft}\times h(g(f(x_{i}^{l,w})))/pl_{i}^{\sim soft} \tag{2}\]  
\[pl_{i}^{soft}=\frac{pl_{i}^{soft}}{\sum_{j=1}^{N}pl_{j}^{soft}} \tag{3}\]  
where \(f(\cdot)\), \(g(\cdot)\) and \(h(\cdot)\), respectively, mean the encoder, the projection layer and the classifier. \(pl_{i}^{\sim soft}\) is computed by the moving average of the prediction of unlabeled data.  
Next, we can generate hard labels from soft labels:  
\[pl_{i}^{hard}=argmax(pl_{i}^{soft}) \tag{4}\]where the \(argmax(\cdot)\) function is used to yield one-hot labels from soft labels. Then, the threshold \(\tau\) is set to select the data with high confidence:  
\[mask=\mathbb{1}(max(pl_{i}^{soft})\geq\tau) \tag{5}\]  
where \(\mathbb{1}(\cdot)\) is the mask function to choose the samples whose maximum prediction is higher than threshold \(\tau\). After computing the soft and hard labels, we can train the model with unlabeled data by standard cross-entropy loss:  
\[L_{soft}=\frac{1}{N}{\sum_{i=1}^{N}\omega L_{cls}(x_{i,1}^{u,s},pl_{i}^{soft})} \tag{6}\]  
\[L_{hard}=\frac{1}{N}{\sum_{i=1}^{N}mask_{cls}(x_{i,2}^{u,s},pl_{i}^{hard})} \tag{7}\]  
where \(\omega\) is the weight of each sample, which is described in Section 3.2.  
In summary, the total loss of dual training can be written as  
\[L_{dual}=\alpha L_{soft}+\beta L_{hard} \tag{8}\]  
where \(\alpha\) and \(\beta\) are the weights of \(L_{dual}\). With dual training strategy, we train the model with all unlabeled data to fully utilize the existing data. Furthermore, we train the model with samples augmented by different strong augmentation at the same time and the labels of used samples are different (i.e., soft labels and hard labels). Therefore, the model can learn more meaningful information at once, which significantly improves the performance of the model.