# A Joint Domain-Specific Pre-Training Method Based on Data Enhancement
## 3 Approach
### Joint Pre-Training Method Based on Data Enhancement
#### 3.2.2 Model Optimization
Therefore, increasing the \(top-k\) parameter beyond a certain point does not proportionally increase the amount of information in the reconstructed domain-specific text corpora, while a larger \(top-k\) may increase the size of the reconstructed domain-specific text corpora, it does not necessarily lead to a significant increase in information. On the other hand, setting the parameter \(top-k\) too large can significantly increase the probability of introducing noisy and irrelevant corpora, which may lead to a decline in model performance. If \(top-k\) is 0, \"Task2 Domain\" similarity dictionary is empty; this indicates that data augmentation is not performed. This is equivalent to the traditional two-stage pre-training method, and the performance is relatively poor (see BERT in Tables 2 and 3). In summary, we recommend selecting a moderate value of \(top-k\) to achieve optimal performance. This value should be carefully chosen to balance the trade-off between incorporating new knowledge and avoiding the introduction of noise and irrelevant data. Figure 3: Parameter and time complex analysis of varying the \(top-k\) for BERT-JDE. (**a**) NER performance on Scie. (**b**) NER performance on NCBI-disease. (**c**) CLS performance of F1-score on Intent. (**d**) CLS performance of Accuracy on Intent. #### 4.4.2 Time Complex Analysis

This subsection presents an analysis of the time complexity of BERT-JDE with the \(top-k\) parameter, which specifies the number of top items to be returned. During the data augmentation process, the size of the reconstructed pre-training corpus is directly related to the \(top-k\). A larger \(top-k\) means a larger 'Task2 Domain' dictionary, which leads to higher time complexity for each round of pre-training corpus reconstruction and model training. From Figure 3, it can be observed that as \(top-k\) increases, the time complexity of the pre-training process exponentially grows, while the model performance only slightly fluctuates. This result confirms that \(top-k\) significantly affects the training speed of the model. Particularly, when \(top-k=0\), the model degenerates to the traditional method. Although the training time is reduced, the pre-training model without data augmentation strategy experiences a significant drop in downstream task performance (decreased by 2.2% on the Scie NER experiment). This indicates that it is essential to incorporate a data augmentation strategy based on the \(top-k\) in pre-training, which can significantly enhance the performance of the pre-training model on downstream tasks. ## 5 Conclusions

In order to make up for the lack of research on corpus sample selection in the current pre-training field, this paper proposes a novel joint pre-training method based on data enhancement.The method reconstructs the pre-trained corpus by selecting the approximate text from the pre-trained corpus for the downstream task of judging the incorrect task text. The method can correct the pre-training BERT's understanding of features such as judgment error text expressions and linguistic description logic, so as to improve the accuracy of downstream tasks. The experiments show that without any data analysis or data processing, our method can achieve the improvement of the basic BERT model based on any raw domain corpus. From the practical perspectives, since the corpus construction in this study mainly relied on academic research papers in the field of computer science, this limited the performance of the model in the biology domain. To improve the performance of the joint training model proposed in this study, a more extensive and higher quality pre-training corpus needs to be developed in the future. Therefore, how to build and expand a more universal pre-training corpus is one of the important directions that we need to study in the future. From the academic perspectives, this study only used the BERT representations as features to construct a dictionary and selected similar samples from the pre-training corpus.