# A Joint Domain-Specific Pre-Training Method Based on Data Enhancement
## 3 Approach
### Joint Pre-Training Method Based on Data Enhancement
#### 3.2.2 Model Optimization
Based on above experiments, it can be observed that the joint pre-training method based on data augmentation is effective for most BERT-based pre-training models. Without any pre-processing or analysis of the pre-training corpus, the proposed joint pre-training strategy and data augmentation method can further improve the performance of the pre-trained BERT-based pre-training model on downstream tasks. This approach achieves data augmentation through joint training with feedback from downstream tasks, which not only alleviates the catastrophic forgetting problem caused by downstream fine-tuning tasks, but also achieves the best results in many indicators. It has strong practical value as it achieves promising results without significantly increasing the time complexity and hardware requirements of the model training. ### Ablation Experiments

In order to verify whether the data enhancement strategy proposed in this paper is effective, we removed the data enhancement module and kept only the joint training method, named BERT-J. Because the size of the domain corpus in the pre-training phase of the joint training process is very large (more than 1.19 GB), considerable computational resources are required if multiple pre-training rounds are performed. Therefore, only two datasets (Sci-cite and Intent) were selected for the ablation experiments, and the text classification experimental results are shown in Table 4.

Compared with the BERT-J method without data augmentation, the BERT-JDE method enhances the understanding of the BERT model for domain-specific, model-unrecognizable expressions by reconstructing the pre-trained model using unrecognizable task text. In the text classification task, the domain corpus-enhanced BERT model has a more forward feature extraction capability to extract more discriminating text features under the domain restriction, and a stronger understanding of domain expression features. This leads to more significant extraction for the downstream text classification task. It indicates that the corpus reconstruction strategy with downstream feedback can greatly improve the model's effectiveness for the downstream task. ### Parameter and Time Complex Analysis

In this section, we present a detailed analysis of the impact of the hyperparameter \(top-k\) on the performance and time complexity of our proposed model. The range of \(top-k\) was set from 0 to 500, and experiments were conducted on two popular natural language processing tasks: text classification(CLS) using Intent dataset and named entity recognition (NER) using the NCBI-disease and Scie datasets. Furthermore, other hyperparameters set \begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{2}{c}{**BERT-J**} & \multicolumn{2}{c}{**BERT-JDE**} \\ \cline{2-5}  & **ACC** & **F1** & **ACC** & **F1** \\ \hline Sci-cite & 0.812 & 0.787 & 0.847 & 0.828 \\ Intent & 0.905 & 0.851 & 0.918 & 0.893 \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Comparison among data-enhancement strategies. as \(M=2000\) and \(epoch=20\). Particularly, when \(top-k\) is set to 0, the model degenerates to the traditional method. The primary goal of this analysis is to investigate how \(top-k\) affects the model's accuracy and efficiency, providing insights into the optimal setting of this hyperparameter. Furthermore, this study aims to provide a comprehensive evaluation of the proposed model's practical applicability for real-world NLP tasks. The results of the experiments are shown in Figure 3.

#### 4.4.1 Parameter Analysis

The results shown in Figure 3 indicate that the \(top-k\) has little effect on the model performance. This is because the selected \(top-k\) domain-specific text corpora for a hard-to-classify task sample have a consistent theme and characteristics, which limits the knowledge that can be learned from the reconstructed domain-specific text corpora.