# A Joint Domain-Specific Pre-Training Method Based on Data Enhancement
## 3 Approach
### Joint Pre-Training Method Based on Data Enhancement
#### 3.2.2 Model Optimization
It indicates that by learning the domain pre-trained samples selected by this method, it can enhance the representation of the BERT model under the domain and provide more discriminating features for downstream tasks. Furthermore, compared with the SciBERT method, our BERT-JDE performs better in most cases. Because the corpus built in this paper covers a wide range of scientific and technological papers, BERT-JDE performs better on the Sci-cite and Intent datasets. However, although it performs more poorly than SciBERT on ChemProt dataset, it still achieves very competitive performance in BioText classification tasks. #### 4.2.2 Named Entity Recognition Experiment

In addition to above benchmarks, a data enhancement joint pre-training method with random sampling is specifically incorporated in named entity recognition, named BERT-RDE. This method randomly selects equal amounts of corpus data to constitute a pre-training corpus during each training round. The experimental results of macro-F1 scores are shown in Table 3. The Corpora-BERT method is less effective than the original BERT model in BioText named entity recognition tasks. The reason is that the corpora constructed in this paper lacks special biological expressions and proper nouns. Therefore, some expressions of CS are noise for this task, which leads to poor performance of Corpora-BERT. Compared with the basic BERT model, the BERT-RDE method is a joint pre-training method which randomly selects 2000 texts at each round as the pre-training corpus. Although the BERT-RDE method reduces the corpus size, it has produced positive benefits (improvements of around 2.2%) compared with BERT model. This is because the randomly selected corpora of BERT-RDE can reduce the impact noise in pre-training corpus and enhance the generalization ability of the joint pre-training model, thereby avoiding over-fitting. \begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{2}{c}{**BERT [8]**} & \multicolumn{2}{c}{**Corpora-BERT**} & \multicolumn{2}{c}{**SciBERT [27]**} & \multicolumn{2}{c}{**BERT-JDE**} \\ \cline{2-7}  & **ACC** & **F1** & **ACC** & **F1** & **ACC** & **F1** & **ACC** & **F1** \\ \hline Sci-cite & 0.836 & 0.815 & 0.809 & 0.788 & 0.853 & 0.836 & **0.857** & **0.850** \\ Intent & 0.909 & 0.870 & 0.902 & 0.85 & **0.920** & 0.892 & 0.918 & **0.893** \\ ChemProt & 0.867 & 0.745 & 0.847 & 0.701 & **0.902** & 0.742 & 0.898 & **0.801** \\ \hline \hline \end{tabular}
\end{table} 
Table 2: Text Classification Experimental Results. \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Dataset** & **Sciie** & **NCBI-Disease** & **BC5 DR** & **Average** \\ \hline BERT [8] & 0.652 & 0.886 & 0.862 & 0.800 \\ SciBERT [27] & 0.729 & **0.925** & **0.919** & 0.858 \\ Corpora-BERT & 0.671 & 0.879 & 0.863 & 0.804 \\ BERT-RDE & 0.694 & 0.893 & 0.88 & 0.822 \\ BERT-JDE & 0.713 & 0.895 & 0.918 & 0.842 \\ SciBERT-JDE & **0.749** & **0.925** & **0.919** & **0.864** \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Named Entity Recognition Experimental Results. Compared with the basic BERT model, the BERT-JDE model proposed in this paper achieved a significant improvement of 5% on average, which shows the effectiveness of our data enhancement strategy. However, compared with the SciBERT model, the BERT-JDE method is slightly inferior. The reason is that the SciBERT model has a stronger feature extraction ability in the biological text NER task than the BERT model. Therefore, this experiment conducted a joint training of data enhancement on SciBERT and obtained SciBERT-JDE. The experiment shows that SciBERT-JDE achieves the best effects on the task of NER in multi-disciplinary papers (Scie). On the BioText datasets (NCBI-disease and BC5 DR), the SciBERT-JDE method is not worse even though the corpus lacks biological knowledge. Overall, the data enhancement strategy proposed in this paper still works for SciBERT, and the average entity recognition accuracy of SciBERT-JDE has improved.