# A Joint Domain-Specific Pre-Training Method Based on Data Enhancement
## 3 Approach
### Joint Pre-Training Method Based on Data Enhancement
#### 3.2.2 Model Optimization  
Thus, the final loss function of the domain pre-training method based on data enhancement can be rewritten as follows:  
\[L=\sum_{s\in D_{domain}}L_{MLM}(s)+\sum_{s\in D_{task}}L_{task}(s), \tag{3}\]  
where the domain corpus \(D_{domain}\)\({}^{\prime}\) is dynamically changing in each training episode and the composition of \(D_{domain}\)\({}^{\prime}\) strongly depends on the downstream task \(L_{task}\). To obtain the best pre-training model, the model typically undergoes multiple iterations of pre-training and fine-tuning training joint training. After the first round of downstream fine-tuning tasks, a set of hard-to-classify samples \(D_{Err}^{t}=\{s_{e}|s_{e}\in D_{task},f^{t}(s_{e})==Error\}\) is obtained by identifying the samples that the current epoch(\(t\))'s model \(f^{t}(. )\) cannot classify correctly in downstream training set. Subsequently, using a pre-established \"Task2 Domain\" similarity dictionary, we can obtain the \(=\)\(top-k\) domain samples \(Sim(s_{e})=\{s|s\in T2D[S_{e}],s\in d_{domain}\}\) that have similar expressions with each hard-to-classify sample \(s_{e}\). By aggregating the \(top-k\) domain samples of all the hard-to-classify samples, we can obtain a smaller and higher quality pre-training corpus \(D_{domain}^{t}=\{s|s\in sim(s_{e}),s_{e}\in D_{Err}\}\) at epoch \(t\). This approach makes the joint training of pre-training and fine-tuning feasible on the one hand, and reduces the influence of irrelevant samples and noise in pre-training on the model on the other hand. The reconstruction of the in-domain corpus (\(D_{domain}^{t}\)\({}^{\prime}\)) for round \(t\) can provide an extra corpus for the next iteration of MLM pre-training stage. The main purpose of this round of pre-training is to supplement and strengthen the model's understanding of the knowledge that was lacking in the previous round of downstream tasks. Afterward, the model continues with the downstream fine-tuning tasks for round \(t+1\). After multiple iterations, the model can retain as much knowledge as possible from the pre-training corpus while improving its accuracy on downstream tasks. The pseudocode for the entire training process using the joint pre-training method based on data enhancement can be found in Algorithm 1. ``` Input: Domain Corpus (\(D_{domain}\)), Downstream Task Data (\(D_{task}\)),Training times (\(T\)) and Initial BERT. Output:
1 Initialize BERT and downstream\(f(. )\) parameter. 2 Establishing \"Task2 Domain\" similarity dictionary (\(T2D\)) according to BERT model based on \(D_{domain}\) and \(D_{task}\). 3 fort from 0 to Tdo
4 ift!=0 then
5 Updating the BERT parameter according to \(L_{MLM}\) based on \(D_{domain}^{t-1}\)\({}^{\prime}\). 6 fors in \(D_{task}\)do
7 Updating the downstream parameter\(f^{t}(. )\) according to \(L_{task}\) based on \(D_{task}\)\({}^{\prime}\) and marking the hard-to-classify sample \(s_{e}\). 8 Composing the hard-to-classify set as \(D_{Err}^{t}\). 9 Restructuring the pre-train corpus \(D_{domain}^{t}\)\({}^{\prime}\) based on the \(D_{Err}^{t}\) and \(T2D\). ## 4 Experiments and Discussion

### Corpora Datasets

This paper focuses on the construction and pre-training corpus of scientific and technical papers, and the pre-training corpus comes from two parts of all articles published in 2015 by ElsevierOA ([[https://elsevier.digitalcommonsdata.com/datasets/zm33](https://elsevier.digitalcommonsdata.com/datasets/zm33) cdndxs/2]([https://elsevier.digitalcommonsdata.com/datasets/zm33](https://elsevier.digitalcommonsdata.com/datasets/zm33) cdndxs/2), accessed on 10 December 2022) and NIPS2015 ([[https://nips.cc/Conferences/2015](https://nips.cc/Conferences/2015)]([https://nips.cc/Conferences/2015](https://nips.cc/Conferences/2015)), accessed on 10 December 2022) (Advances in Neural Information Processing Systems, a conference on advances in neural information processing systems). Among them, NIPS received a total of 1838 submissions and accepted 403 papers in 2015, with an acceptance rate of 21.9%.