# A Joint Domain-Specific Pre-Training Method Based on Data Enhancement
## 3 Approach
### Joint Pre-Training Method Based on Data Enhancement
#### 3.2.2 Model Optimization
Among more than 400 papers, the directions are extremely diverse. The largest proportion of deep learning articles accounted for 11% of all accepted papers, followed by convex optimization accounting for 5% and statistical learning theory accounting for 3%. Furthermore, ElsevierOA [30] is a corpus of 40k (40,001) open access (OA) CC-BY articles from across Elsevier's journals representing the cross-discipline research of data at this scale to support NLP and ML research. This dataset was released to support the development of ML and NLP models targeting science articles from across all research domains. For these two corpus of papers, only the body text and abstract of each paper are selected as corpus sources to build this dataset. ### Evaluation Tasks

In order to verify the effectiveness of the proposed method, two downstream tasks are selected: (1) named entity recognition (NER), and (2) text classification (CLS). NER is a span-level text classification task, which mainly refers to extracting entities with specific meanings or strong denotations from unstructured text. CLS is a sentence-level text classification task, which mainly refers to predicting text sentences into a specified category or categories. To verify the effectiveness of the proposed method, text classification experiments on Sci-cite [31], Intent [32], and Chemprot [25] datasets were conducted first. In addition to the sentence-level classification task, this paper also performs the NER task on NCBI-disease [33], BC5 CDR [34], and Scie [35] datasets. The detail information about above datasets can be found in Table 1. For brevity, we do not explain the details of older datasets and readers can refer to the corresponding references. #### 4.2.1 Text Classification Experiments

In addition to the basic BERT [8], two BERT-based models are compared in text classification: (1) Corpora-BERT pre-trained on above corpora, and (2) SciBERT [27] trained on the scientific literature text. Furthermore, the data enhancement joint pre-training method proposed in this paper based on the basic BERT model is called BERT-IDE. For each pre-trained model, we chose the base setting with 12 transformer layers and 768 hidden embedding sizes. As shown in Table 2, the Corpora-BERT performs poorly. The reason is that Corpora-BERT learns from the raw corpora corpus during the pre-training process, which contains a large amount of noise and out-of-distribution domain corpus data. Without any pre-processing, the Corpora-BERT model finds it difficult to extract discriminating text features. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline
**Task** & **Dataset** & **Domain** & **\#Label** & **\#Entity** & **\#Sents** \\ \hline \multirow{3}{*}{CLS} & Sci-cite & Multi & 3 & - & 10,104 \\  & Intent & CS & 6 & - & 1941 \\  & ChemProt & Bio & 13 & - & 10,065 \\ \hline \multirow{3}{*}{NER} & NCBI-diseas & Bio & - & 2 & 7287 \\  & BC5 CDR & Bio & - & 2 & 15,030 \\ \cline{1-1}  & Scie & CS & - & 7 & 3187 \\ \hline \hline \end{tabular} Note: \#Label and \#Entity mean the number of types of Labels and Entities, \#Sents means the number of sentences.

\end{table} 
Table 1: Statistics of the datasets. Therefore, even if fine-tuning is performed based on Corpora-BERT, the model still cannot achieve good downstream results. On the contrary, the BERT model trained by Google is based on a corpus with less noise and higher quality, which greatly enhances its feature extraction ability. After a small amount of fine-tuning, the pre-trained model can achieve better downstream results. Furthermore, the BERT-JDE method proposed in this paper resamples a part of pre-trained samples through the feedback of the CLS task. It can enhance the classification ability of the model itself and improve the classification accuracy of the original BERT that was fine-tuned without any data analysis or processing of the pre-training sample.