# Text Classification Model Enhanced by Unlabeled Data for LaTeX Formula
## 3 Unlabeled Data Enhanced Text Classification  
### Double-Prediction Pseudo-Labeling Method  
Besides the role in pretraining, a new pseudo-label dataset can be produced based on unlabeled data by self-training, which is a helpful and effective in-model fine-tuning. The self-training method has been proposed for image recognition [2], where it assigns pseudo-labels for unlabeled images by way of a noisy and inaccurate model which is only trained on a few labeled data. This noisy model is improved and retrained on both labeled and unlabeled images.  
A revised self-training method on unlabeled text data is proposed to generate a high confidence pseudo-label from an input text and its variant. Considering the immaturity of the initial model trained with few labeled data, we improve the robustness of pseudo-label prediction, where the Double-Prediction Pseudo-labeling (DPP) method is proposed to obtain high confidence Pseudo-labels predicted by the last trained model.  
The original training data are defined as \(x_{i}\) with label \(y_{i}\), unlabeled data \(\widetilde{x}_{i}\). The variants, the augmented unlabeled data \(\widetilde{x}^{\prime}_{i}\), are obtained by adding auxiliary sentences to \(\widetilde{x}_{n}\). Auxiliary sentences are meaningless texts that do not affect classification, such as 'Without using a calculator,...' to \(\widetilde{x}_{n}\). The DPP method in self-training is as follows:  
Figure 1: Two-stages Pretraining BERT model (TP-BERT).  
Figure 2: Word vectors spaces of BERT and TP-BERT. Red: word vectors of LaTeX mathematical formulas; Purple: other word vectors.  
The UL-BERT model is fine-tuned based on TP-BERT by the original training dataset \((x_{i},y_{i})\). The cross-entropy loss function \(L_{t}\) is defined by: \[L_{t}=-\frac{1}{m}\sum_{i}^{m}\sum_{j}^{l}y_{i}^{j}logp_{i}^{j},\] (1) where \(m\) is the batch size and \(l\) is the number of and classes, \(y\) is the ground truth, and \(p\) is the predicted label. 2. Labels \(\widetilde{y}_{i}\) and \(\widetilde{y}_{i}^{\prime}\) are predicted based on an unlabeled dataset \(\widetilde{x}_{i}\) and \(\widetilde{x}_{i}^{\prime}\) by UL-BERT. \[\widetilde{y}_{i}=f(\widetilde{x}_{i},\theta^{t})\] (2) \[\widetilde{y}_{i}^{\prime}=f(\widetilde{x}_{i}^{\prime},\theta^{t}).\] (3)
3. For data with the same predicted labels whose prediction probabilities are both above a specific threshold \(t\), label \(\widetilde{y}_{i}^{p}\) is taken as its pseudo-label. The higher prediction probability has better class confidence. \[\widetilde{y}_{i}^{p}=\left\{\begin{array}{l}\widetilde{y}_{i}^{\prime}, \qquad\widetilde{y}_{i}=\widetilde{y}_{i}^{\prime}and\widetilde{p}_{i}>=tand \widetilde{p}_{i}^{\prime}>=t\\ null,\quad other,\end{array}\right.\] (4) where \(\widetilde{p}_{i}\) is the predicted probability, and \(t\) is the threshold. 4. The new dataset \(\left(\widetilde{x}_{i}^{p},\widetilde{y}_{i}^{p}\right)\) with pseudo-labels is obtained and combined with the original label dataset to train the classification model. In general, the problem of few labeled datasets can be solved by retraining UL-BERT with the new pseudo-labeled dataset from unlabeled data self-training, which improves the robustness, generalization and accuracy of the classification model.