<figure><img src="image_1.png" /></figure>

HAL
open science

# Evaluating deep learning models for plant protein function prediction

Minh Ngoc Vu, Ha Hoang, Antoine Toffano, Pierre Larmande

► To cite this version:

Minh Ngoc Vu, Ha Hoang, Antoine Toffano, Pierre Larmande. Evaluating deep learning models for plant protein function prediction. JOBIM 2025 - 25es Journées Ouvertes en Biologie, Informatique et Mathématiques, Jul 2025, Bordeaux, France. ird-05194043

HAL Id: ird-05194043

https://ird.hal.science/ird-05194043v1

Submitted on 31 Jul 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Evaluating deep learning models for plant protein function prediction

Minh Ngoc VU<fnref n="1" />, Hoang Ha NGUYEN<fnref n="2" />, Antoine TOFFANO<fnref n="3" /> and Pierre LARMANDE<fnref n="3" />,<fnref n="4" />

Corresponding author: pierre.larmande@ird.fr

**Keywords** Protein function prediction, Gene Ontology, deep learning

**Abstract** Predicting the functions of proteins remains a critical yet challenging task in computational biology. Advances in high-throughput sequencing, the expansion of protein databases, and the continuous development of artificial intelligence have led to the emergence of many computational methods dedicated to protein function prediction. In this study, we evaluated the performance of four state-of-the-art models - DeepGOPlus, DeepGraphGO, DeepGOZero, and DeepGOSE - using experimentally annotated proteins from the UniProt-KB/Swiss-Prot database. We also trained and tested these models on species-specific datasets from Arabidopsis thaliana and Oryza sativa to investigate their potential and applicability in plant protein studies. Our results showed that DeepGOPlus consistently achieved the best evaluation scores across all datasets. DeepGOSE and DeepGOZero performed comparably and only marginally outperformed DeepGraphGO in certain training attempts. Further analysis revealed that dataset stratification into training, validation, and testing sets introduced variations in Gene Ontology annotation specificity, which may have influenced model performance.

## Introduction

Proteins are fundamental constituents of all cellular processes, metabolism, and structures in living organisms. Their building blocks are amino acid sequences that fold into secondary, tertiary, and quaternary structures that dictate the biochemical functions of the protein and its interactions with other molecules, such as those within protein-protein interaction (PPI) networks [1]. Determining protein function often involves multiple steps and complex techniques to resolve protein structures from amino acid sequences and study their bioactivities [1], making the process time-consuming, costly, and labor-intensive. Computational methods have been proposed as a powerful solution to these experimental limitations. The Critical Assessment of Function Annotations (CAFA) Challenge, launched in 2010, has served as a framework for benchmarking function prediction models using large multi-species protein datasets and evaluation metrics [2]. Over the years, numerous methods have been introduced, incorporating sequence data, structures, and protein-protein interaction (PPI) networks [3,4]. These methods excel in predicting different classes of the Gene Ontology (GO) [5,6], such as molecular functions (MF), cellular components (CC), or biological processes (BP) and are suited for specific usage scenarios.

Despite these advances, several challenges remain. Firstly, most models are built with proteins of well-studied species and may be less effective for those with limited annotations [3]. Secondly, pre-

<footnote n="1">¹ Université Évry Paris-Saclay, 91000, Évry-Courcouronnes, France</footnote>

<footnote n="2">² ICT Department, University of Science and Technology of Hanoi, 100000, Hanoi, Vietnam</footnote>

<footnote n="3">³ LIRMM, Univ. Montpellier, 34000, Montpellier, France</footnote>

<footnote n="4">⁴ DIADE, IRD, CIRAD, Univ. Montpellier, 34000, Montpellier, France</footnote>
dicting specific and informative functions is considerably challenging [4]. These limitations are relevant when the models are applied in plant biology, which has distinct genomic and proteomic complexities due to their specialized metabolism and biosynthetic pathways [7]. Furthermore, recent reviews have shown that excluding a target species from the training data reduces the accuracy of models' predictions on its proteins. [4,8]. This is especially problematic for plant species, where experimental annotations are sparse, even in model species like *Arabidopsis thaliana*, potentially leading to plant proteins being underrepresented in multi-species training sets. Moreover, species-specific evaluations have primarily focused on human and mouse, leaving the performance of prediction models on plant-specific data largely unexplored [4,8].

In this study, we compared four leading methods, DeepGOPlus [9], DeepGraphGO [8], DeepGOZero [10], and DeepGOSE [11] in predicting GO annotations of proteins in the plant domain. To train and validate these models, we used a multi-species dataset of experimentally annotated proteins from the UniProt-KB/Swiss-Prot database [12], and proteins of *A. thaliana* and *Oryza sativa* plant species. This allowed us to evaluate their effectiveness in species-specific contexts and explore their potential for plant protein function prediction.

# Methods

## Dataset preparation

All datasets in this study were prepared from the UniProt-KB/Swiss-Prot knowledgebase (version 2024-07-22) [13] and the Gene Ontology (GO) database (version 2024-04-24) [5,6]. Following the DeepGO series methodology [10,11], a benchmark dataset of 80,815 proteins annotated from experimental and high-throughput experiments (i.e., having evidence codes among EXP, IDA, IPI, IMP, IGI, IEP, TAS, IC, HTP, HDA, HMP, HGI, and HEP [14]) from all species was compiled. We also extracted subsets for *O. sativa* and *A. thaliana* species, containing respectively 1,093 and 10,679 proteins. All models were trained and evaluated separately on three GO sub-ontologies. The number of proteins and annotations in each dataset were summarized in Table 1. PPI information was obtained from the STRING database (v11) [15]. We followed the method of Kulmanov et al. (2024) [11] to split proteins into training, validation, and testing sets. Briefly, sequences were grouped based on their pairwise alignment scores calculated using DIAMOND (v2.1.8) [16], and each group was then split into three sets to minimize sequence similarity within the same set.

<table><caption>**Tab. 1.** Number of proteins and GO terms in each dataset</caption><thead><tr><th></th><th></th><th>Biological process</th><th>Molecular function</th><th>Cellular function</th></tr></thead><tbody><tr><td rowspan="2">Benchmark</td><td>Protein</td><td>61910</td><td>45958</td><td>60446</td></tr><tr><td>Term</td><td>20805</td><td>7025</td><td>2924</td></tr><tr><td rowspan="2">A.thaliana+O.sativa</td><td>Protein</td><td>8687</td><td>5799</td><td>8464</td></tr><tr><td>Term</td><td>4944</td><td>2294</td><td>597</td></tr><tr><td rowspan="2">A.thaliana</td><td>Protein</td><td>7895</td><td>5355</td><td>7866</td></tr><tr><td>Term</td><td>4866</td><td>2242</td><td>576</td></tr><tr><td rowspan="2">O.sativa</td><td>Protein</td><td>792</td><td>444</td><td>598</td></tr><tr><td>Term</td><td>1686</td><td>571</td><td>175</td></tr></tbody></table>
## Model implementation

The datasets were then used to train DeepGOPlus, DeepGraphGO, DeepGOZero, and DeepGOSE, each representing a distinct learning strategy. While DeepGOPlus, DeepGraphGO, and DeepGOZero have been benchmarked in recent comparative studies [4,3], the application of DeepGOSE has not been reviewed. DeepGOPlus is a sequence-based function prediction method that employs one-dimensional convolutional neural networks to learn sequence patterns and motifs, then combines these features with annotations of similar proteins to classify GO terms [9]. DeepGraphGO is a hybrid method that uses InterPro feature vectors from protein sequences and implements graph convolutional neural networks that leverage features from protein sequence annotations alongside multi-species PPI networks [8]. DeepGOZero, a "zero-shot" prediction model, utilizes amino acid sequences and employs GO axioms embedded through the EL Embedding method, where the embeddings are generated by multi-layer perceptron blocks [10]. As an improvement of DeepGOZero, DeepGOSE was developed, leveraging protein sequences represented as features from the ESM2 protein language model [17], PPIs, GO background knowledge, and semantic entailment approximation [11].

The number of epochs used for each model varied from 10 to 20 until convergence. To avoid overfitting, only the best weights were retained for evaluation. All models were trained on a Linux system with an NVIDIA GeForce RTX 3080 Ti, CUDA 11.7, and Python 3.9.

## Evaluation metrics

The metrics used to evaluate the model performance included $F_{max}$ and $S_{min}$, defined as follows:

$$F_{max} = \max_{t} \left( \frac{2 \cdot P(t) \cdot R(t)}{P(t) + R(t)} \right)$$

where $P(t)$ and $R(t)$ denote precision and recall at threshold $t$, respectively.

$$S_{min} = \min_{t} \sqrt{\left(\frac{1}{n} \sum_{i,c \in FN_{i}(t)} IC(c)\right)^{2} + \left(\frac{1}{n} \sum_{i,c \in FP_{i}(t)} IC(c)\right)^{2}}$$

where $IC(c)$ is the information content of the GO term $c$, and $FN_i(t)$ and $FP_i(t)$ represent the false negatives and false positives for protein $i$ at threshold $t$.

Additionally, the area under the precision-recall curve (AUPR) and the area under the receiver operating characteristic curve (AUC) were computed.

## Results

Better model performance is indicated by higher $F_{max}$, AUPR, and AUC values, and lower $S_{min}$ values. The results showed that when trained and tested on the benchmark dataset, DeepGraphGO achieved the lowest $S_{min}$ in all three sub-ontologies (Figure 1), thus outperforming other methods in terms of the semantic accuracy of the GO predictions. For CC annotations, it also obtained the highest $F_{max}$ and AUPR, indicating both strong overall performance and a lower tendency for false positives, followed by DeepGOPlus, which only marginally surpassed DeepGOZero and DeepGOSE. However,
for BP and MF terms, DeepGOPlus achieved the highest $F_{max}$ and AUPR. Interestingly, when the AUC metric is considered, DeepGOPlus consistently outperformed all other models.

<figure><img src="image_3.png" /><figcaption>**Fig. 1.** Prediction results on the benchmark dataset (BP: biological process, MF: molecular function, CC: cellular component)</figcaption></figure>

When trained on *A. thaliana* and *O. sativa* proteins, DeepGOPlus also achieved top evaluation scores in all sub-ontologies (Figure 2). In BP predictions for *A. thaliana*, it attained an $F_{max}$ of 0.42 compared to 0.36 by DeepGraphGO and 0.31 by DeepGOSE. For MF annotations, DeepGOPlus reached an AUPR of 0.48, whereas all other methods scored below 0.24. Similarly, for *O. sativa*, DeepGOPlus achieved an $F_{max}$ score of 0.70 for BP annotations, and 0.67 for MF (Figure 2). These scores were two to three times higher than those of other methods, while its $S_{min}$ values were approximately half as large. Meanwhile, DeepGOZero demonstrated the poorest performance in BP and MF prediction of *A. thaliana*. DeepGOSE did not surpass DeepGOZero and only slightly outperformed DeepGraphGO. For the combined dataset of *A. thaliana* and *O. sativa* proteins, the performance gap between different models was less pronounced, yet DeepGOPlus still stood out as the best method (Figure 2). Another interesting result was that all models achieved significantly lower $S_{min}$ when trained and tested on *O. sativa* than on *A. thaliana* or the combined dataset. For instance, when predicting MF annotations for *O. sativa* proteins, DeepGOSE obtained 7.45 in $S_{min}$ value, and 9.73 for *A. thaliana* proteins. However, for $F_{max}$ and AUPR metrics, this trend was observed only in DeepGOPlus and not in the other models.

Previous research has shown that using datasets with varying depths of GO terms can influence model performance [4]. Therefore, we compared the specificity of GO annotations in our datasets using the one-factor analysis of variance and Tukey's Honest Significant Difference post-hoc test. Figure 3 shows that there was a significant difference in the annotation specificity between all protein datasets ($p < 0.001$). In particular, the benchmark data displayed a higher overall specificity, likely due to its inclusion of multiple species and potentially more complex proteins. *O. sativa* proteins exhibited lower GO specificity than *A. thaliana* in BP and MF ontologies ($p < 0.001$), and no statistical difference in CC ontologies. We also found that for BP and MF annotations of all datasets, the training sets showed significantly higher specificity than the validation and testing sets ($p < 0.001$), while little to no differences were found between the validation and testing sets (Figure 4). All the sets prepared for CC annotations displayed no statistical difference in term specificity, except for *O. sativa* in which the training set has higher specificity than the validation set ($p < 0.05$).
<figure><img src="image_4.png" /><figcaption>**Fig. 2.** Prediction results on *A. thaliana* and *O. sativa* datasets</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>**Fig. 3.** The specificity of GO annotations in each dataset, represented as the depths from the root node in the GO hierarchy (BP: biological process, MF: molecular function, CC: cellular component, *: p<0.05, **: p<0.01, ***: p<0.001)</figcaption></figure>
<figure><img src="image_6.png" /><figcaption>**Fig. 4.** The specificity of GO annotations in training, validation, and testing sets (BP: biological process, MF: molecular function, CC: cellular component, *: p<0.05, **: p<0.01, ***: p<0.001)</figcaption></figure>

Discussion

Our study showed that among all the tested models, DeepGOPlus demonstrated superior perfor-
mance in all datasets and GO classes. These findings differ from those of Wang et al. (2024) [4],
who reported that DeepGraphGO achieved better scores than DeepGOPlus in most training scenar-
ios. While both studies identified these models as top performers, our results indicated that Deep-
GOPlus excelled in predicting BP and MF annotations, whereas DeepGraphGO performed best for
CC. The discrepancy may be attributed to differences in dataset composition and the versions of
UniProt-KB/Swiss-Prot used. Consistent with previous studies [9,11], BP prediction remains the most
challenging task, whereas CC prediction yielded the highest and most consistent evaluation scores.
This may be attributed to the substantial difference in the number of annotations among the three
sub-ontologies (Figure 1). Nevertheless, our results suggest that DeepGOPlus and DeepGraphGO
could be useful models for studying plant proteins with limited annotations or novel proteins. How-
ever, given the small dataset sizes for both model plants (A. thaliana and O. sativa) and non-model
species, overfitting may be an issue and comprehensive hyperparameter tuning is crucial.

Moreover, we observed that *O. sativa* and *A. thaliana* exhibit lower GO annotation specificity com-
pared to multi-species datasets, with *O. sativa* showing even lower specificity than *A. thaliana*. This
may be advantageous, indicated by the improved *S*<sub>min</sub> scores for *O. sativa* as more general anno-
tations are easier for models to predict [4]. However, the variability in annotation specificity across
training, validation, and testing sets may raise concerns about potential biases in model evaluation.
Given the limited data available for most plant species, the conventional data partitioning approach
that relies only on sequence similarity may be insufficient. Therefore, adopting refined strategies that
address differences in annotation specificity is essential to accurately assess model performance.
Furthermore, considering the models' tendency to favor general annotations, it is important to in-
corporate evaluation metrics that account for GO specificity. While our study explored both species-
specific training for plant proteins and a combined multi-species approach, additional analyses are required to determine the most suitable model for each plant dataset.

Conclusion

Our evaluation showed that when implemented on a multi-species dataset, DeepGOPlus demonstrated the best overall performance in predicting BP and MF annotations, while DeepGraphGO performed best for CC annotations. For *O. sativa* and *A. thaliana* proteins, DeepGOPlus was the most suitable model. However, these small single-species plant datasets exhibited lower GO specificity. The data partitioning process also led to training sets having higher annotation specificity than validation and testing sets. Our findings highlighted the necessity for comprehensive optimization of models for specific target species to apply them in plant protein research.

References

[1] Alberts B, Johnson A, Lewis J, Raff M, Roberts K, Walter P. Analyzing Protein Structure and Function. In: Molecular Biology of the Cell. 4th edition. Garland Science; 2002. Available from: https://www.ncbi.nlm.nih.gov/books/NBK26820/.

[2] Zhou N, Jiang Y, Bergquist TR, Lee AJ, Kacsoh BZ, Crocker AW, et al. The CAFA challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens. Genome Biology. 2019 Nov;20(1):244. Available from: https://doi.org/10.1186/s13059-019-1835-8.

[3] Lin B, Luo X, Liu Y, Jin X. A comprehensive review and comparison of existing computational methods for protein function prediction. Briefings in Bioinformatics. 2024 Jul;25(4):bbae289. Available from: https://doi.org/10.1093/bib/bbae289.

[4] Wang W, Shuai Y, Yang Q, Zhang F, Zeng M, Li M. A comprehensive computational benchmark for evaluating deep learning-based protein function prediction approaches. Briefings in Bioinformatics. 2024 Mar;25(2):bbae050. Available from: https://doi.org/10.1093/bib/bbae050.

[5] Ashburner M, Ball CA, Blake JA, Botstein D, Butler H, Cherry JM, et al. Gene Ontology: tool for the unification of biology. Nature Genetics. 2000 May;25(1):25-9. Publisher: Nature Publishing Group. Available from: https://www.nature.com/articles/ng0500_25.

[6] The Gene Ontology Consortium, Aleksander SA, Balhoff J, Carbon S, Cherry JM, Drabkin HJ, et al. The Gene Ontology knowledgebase in 2023. Genetics. 2023 May;224(1):iyad031. Available from: https://doi.org/10.1093/genetics/iyad031.

[7] Ellens KW, Christian N, Singh C, Satagopam VP, May P, Linster CL. Confronting the catalytic dark matter encoded by sequenced genomes. Nucleic Acids Research. 2017 Nov;45(20):11495-514. Available from: https://doi.org/10.1093/nar/gkx937.

[8] You R, Yao S, Mamitsuka H, Zhu S. DeepGraphGO: graph neural network for large-scale, multi-species protein function prediction. Bioinformatics. 2021 Jul;37(Supplement_1):i262-71. Available from: https://doi.org/10.1093/bioinformatics/btab270.

[9] Kulmanov M, Hoehndorf R. DeepGOPlus: improved protein function prediction from sequence. Bioinformatics. 2020 Jan;36(2):422-9. Available from: https://doi.org/10.1093/bioinformatics/btz595.

[10] Kulmanov M, Hoehndorf R. DeepGOZero: improving protein function prediction from sequence and zero-shot learning based on ontology axioms. Bioinformatics. 2022 Jun;38(Supplement_1):i238-45. Available from: https://doi.org/10.1093/bioinformatics/btac256.

[11] Kulmanov M, Guzmán-Vega FJ, Duek Roggli P, Lane L, Arold ST, Hoehndorf R. Protein function prediction as approximate semantic entailment. Nature Machine Intelligence. 2024 Feb;6(2):220-8. Publisher: Nature Publishing Group. Available from: https://www.nature.com/articles/s42256-024-00795-w.
[12] Boutet E, Lieberherr D, Tognolli M, Schneider M, Bansal P, Bridge AJ, et al. UniProtKB/Swiss-Prot, the Manually Annotated Section of the UniProt KnowledgeBase: How to Use the Entry View. In: Plant Bioinformatics. Humana Press, New York, NY; 2016. p. 23-54. ISSN: 1940-6029. Available from: https://link.springer.com/protocol/10.1007/978-1-4939-3167-5_2.

[13] The UniProt Consortium. UniProt: the Universal Protein Knowledgebase in 2025. Nucleic Acids Research. 2025 Jan;53(D1):D609-17. Available from: https://doi.org/10.1093/nar/gkae1010.

[14] Guide to GO Evidence Codes - GO Wiki;. Available from: https://wiki.geneontology.org/ Guide_to_GO_Evidence_Codes.

[15] Szklarczyk D, Kirsch R, Koutrouli M, Nastou K, Mehryary F, Hachilif R, et al. The STRING database in 2023: protein-protein association networks and functional enrichment analyses for any sequenced genome of interest. Nucleic Acids Research. 2023 Jan;51(D1):D638-46. Available from: https://doi.org/10.1093/nar/gkac1000.

[16] Buchfink B, Reuter K, Drost HG. Sensitive protein alignments at tree-of-life scale using DIA-MOND. Nature Methods. 2021 Apr;18(4):366-8. Publisher: Nature Publishing Group. Available from: https://www.nature.com/articles/s41592-021-01101-x.

[17] Lin Z, Akin H, Rao R, Hie B, Zhu Z, Lu W, et al. Evolutionary-scale prediction of atomic-level protein structure with a language model. Science. 2023 Mar;379(6637):1123-30. Publisher: American Association for the Advancement of Science. Available from: https://www.science.org/ doi/10.1126/science.ade2574.