# Survey of BERT-Base Models for Scientific Text Classification: COVID-19 Case Study
## 4 Pre-Training BERT-Base on COV-Dat-20
### Evaluation Metrics  
In the main subsection, we present several indicators measuring the quality of the model. To measure the performance of this classifier, we introduce four types of elements classified for the desired class, namely, _TP_, _FP_, _TN_ and _FN_:  
* _TP_: the positive class correctly predicted by the models;
* _FP_: the positive class incorrectly predicted by the models;
* _TN_: the false class correctly predicted by the models;
* _FN_: the false class incorrectly predicted by the models. In what follows, we present the evaluation metrics adopted to measure the performance of the different DL models used. Indeed, our assessment is based on four different measures, including: _Accuracy_, _Precision_, _Recall_, _F1-Score_. The evaluation metrics are defined as follows:  
\[Accuracy=\frac{TP+TN}{TP+TN+FP+FN} \tag{3}\]  
\[Precision=\frac{TP}{TP+FP} \tag{4}\]  
\[Recall=\frac{TP}{TP+FN} \tag{5}\]  
\[F1 Score=2*\frac{Precision*Recall}{Precision+Recall} \tag{6}\]  
We tackled the text-classification task based on the BERT-base model after exploring DL and experimenting with its limitations. Let us begin by describing our experiments. We started with the automatic collection of scientific papers related to COVID-19 from PubMed. We then created our dataset made up of 4304 scientific papers distributed in an equitable way on four different categories: COVID-19, Virology, Public Health and Mental Health. In order to validate the data classification process in our dataset, we contacted epidemiologists to carry out this task. Based on the manual verification, we concluded that some papers were misclassified. Based on our previous experiments and considering some related works, we opted for performing the scientific paper classification with the PubMed engine. We converged towards a DL solution to tackle the pandemic of COVID-19. We named our model CovBERT. To adapt the existing pre-trained BERT model to our needs, we applied some modifications, and we trained it on our dataset Cov-Dat-20. We then explored the modified BERT-base model with the graphics processing unit (GPU) to obtain a better performance in terms of learning cost. In addition, we fixed the training hyper-parameters such as the max. length, the batch size, the learning rate, the epoch's number, and the seed val. We monitored the validation loss and kept the best model on the validation set. Figure 3 presents the CovBERT accuracy with four epochs. We notice, with CovBERT, an accuracy of 94% at epoch four. Beyond four epochs, we risk falling into overfitting. The related confusion matrix of the proposed model is presented in Figure 4. We noticed that the high precision was maintained for the Public Health category with 94%. It  
Figure 3: CovBERT model accuracy. was also maintained for the Mental Health and the Virology categories with 79% being the least precision value. The error related to the training set (training loss) of the proposed dataset is shown in Figure 5. The training loss value starts with 0.8. Then, it gradually falls until 0.2 in the third epoch. Figure 6 shows the accuracy evolution over time. We noticed that the accuracy increases over time from 84% in epoch 1 to 94% in epoch 4. We concluded that, from one epoch to another, the model acquired more knowledge and proved its effectiveness. Figure 4: Model confusion matrix of our proposed CovBERT model. Figure 5: Training loss of our proposed CovBERT model. Figure 6: CovBERT model accuracy evolution over time with 4 epochs. In summary, we proved the high performance of the modified BERT model. We noticed that this training model is better suited for the specific NLP task compared to other DL training models such as CNN and BiLSTM. Among the advantages of the modified BERT model, we noticed that it is easy to implement. The pre-trained modified BERT model weights already encode a large quantity of information on the English language. The fine-tuning step is based on a much smaller dataset for a specific task.