# Automated Confirmation of Protein Annotation Using NLP and the UniProtKB Database
## 3 Results and Discussion
Ensemble learning models work well when the error for the individual models is uncorrelated. Given that logistic regression, SVM, and RCNN models differ substantially, it is not surprising that combining them provides considerable improvement. For comparison, we fine-tuned BioBERT on the same training set for the same classification task. Based on the performance of the validation set, we stopped at six epochs. We found that BioBERT performs better in terms of accuracy and precision mainly because of its deep bidirectionality, which masks random input words and conditions each word bidirectionally to predict the masked words. However, its recall value is much lower than that of our ensemble learning model, and its F1 score is lower as well. In addition, fine-tuning and inference were very slow due to BERT's huge parameter size. Table 2 presents results obtained using our ensemble learning model for a number of candidate protein annotations. The entry identifiers are unique and stable accession numbers in the UniProtKB database, which can be easily cross-referenced with accession numbers in the NCBI database. With the first two examples, we demonstrate that our \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Accuracy (\%)** & **Precision (\%)** & **Recall (\%)** & **F1 (\%)** \\ \hline Logistic Regression (th = 0.25) & 59.46 & 55.55 & 94.64 & 70.01 \\ \hline SVM with Linear Kernel & 71.53 & 71.66 & 71.21 & 71.43 \\ \hline Recurrent Convolutional & & & & \\ Neural Network (RCNN) & 70.14 & 65.79 & 83.91 & 73.76 \\ \hline Logistic Regression (th = 0.5) & 71.06 & 71.00 & 71.21 & 71.10 \\ \hline Ensemble Learning & 71.26 & 65.19 & 91.25 & 76.05 \\ \hline BioBERT & 75.12 & 73.70 & 78.12 & 75.85 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance of logistic regression (threshold = 0.5), logistic regression (threshold = 0.25), support vector machine (SVM), recurrent convolutional neural network (RCNN), ensemble learning, and BioBERT models using test data (9398 peer-reviewed journal publications). ensemble learning model can identify peer-reviewed biomedical publications indicating experimental evidence of functional annotation for a target protein. We used our smart program with the UniProtKB/Swiss-Prot publications associated with glucarate dehydratase and methyltransferase ammonia-lysae, which are among a \"gold standard\" set of known enzyme superfamilies rigorously studied by domain experts [32]. Our model successfully identified glucarate dehydratase (entry identifier P0 AES2) and methylaspartate ammonia-lysae (entry identifier Q05514) as correct annotations based on the available publications. More specifically, of the six publications associated with glucarate dehydratase, our model identified \"Evolution of enzymatic activities in the enolase superfamily: characterization of the (D)-glucarate/galactarate catabolic pathway in _Escherichia coli_\" and \"Evolution of enzymatic activities in the enolase superfamily: crystallographic and mutagenesis studies of the reaction catalyzed by D-glucarate dehydratase from _Escherichia coli_\" as positive. Our classification results match exactly with UniProtKB/Swiss-Prot, which classifies only these two publications in the Function category. Of the eight publications associated with methylaspartate ammonia-lysae in UniProtKB/Swiss-Prot, four of them, \"The purification and properties of beta-methylaspartate\", \"Alteration of the diastereoselectivity of 3-methylaspartate ammonia lyase by using structure-based mutagenesis\", \"The structure of 3-methylaspartate from _Clostridium tetanomorphum_ functions via the common enolase chemical step\", and \"Engineering methylaspartate ammonia lyase for the asymmetric synthesis of unnatural amino acids\", were correctly classified as positive by our program. Our model only misclassified \"Cloning, sequencing, and expression in _Escherichia coli_ of the _Clostridium tetanomorphum_ gene encoding beta-methylaspartate and characterization of the recombinant protein\" as negative.