FusionProt: Fusing Sequence and Structural Information for Unified Protein Representation Learning
Authors: Dan Kalifa
Uriel Singer
Kira Radinsky
Venue: Accepted by TMLR ()

Abstract: Accurate protein representations that integrate sequence and three-dimensional (3D) structure are critical to many biological and biomedical tasks. Most existing models either ignore structure or combine it with sequence through a single, static fusion step. Here we present FusionProt, a unified model that learns representations via iterative, bidirectional fusion between a protein language model and a structure encoder. A single learnable token serves as a carrier, alternating between sequence attention and spatial message passing across layers. FusionProt is evaluated on Enzyme Commission (EC), Gene Ontology (GO), and mutation stability prediction tasks. It improves F\textsubscript{max} by a median of $+1.3$ points (up to $+2.0$) across EC and GO benchmarks, and boosts AUROC by $+3.6$ points over the strongest baseline on mutation stability. Inference cost remains practical, with only $\sim2\text{--}5\%$ runtime overhead.
Beyond state-of-the-art performance, we further demonstrate FusionProt’s practical relevance through representative biological case studies, suggesting that the model captures biologically relevant features.

Published in Transactions on Machine Learning Research (11/2025)
FusionProt: Fusing Sequence and Structural Information for Unified Protein Representation Learning
Dan Kalifa
balifadan&ccsfechniio.cc
Deportment of Computer Science Technion Isruel Institute of Technology
Uriel Singer Meta AI
wrielsinger@mmeta.ccm com
Kira Radinsky
Department of Computer Science
Technion Israel Institute of Technology
Reviewed on OpenReview: //tp:/e//peereeteeee net/ forum? id- emconaaHod
Abstract
Accurate protein representations that integrate sequence and ftree-dimensional (3D) struc- ture are critical to many biological and biomedical tasks. Most existing models either ignore structure or combine it with sequence through a single, static fusion step. Here we present FusionProt, unified model that learns representations via ilerative, bidirectional fusion be- tween a protein language model and a structure encoder. A single learnable token serves as carrier, alternating between sequence attention and spatial message passing across layers. FusionProt is evaluated on Enzyme Commission (EC), Gene Ontology (GO), and mutation stability prediction tasks. It improves F Fmax by median of +1.3 points (up to +2.0) across EC and GO benchmarks, and boosts AUROC by +3.6 points over the strongest baseline on mutation stability. Inference cost remains practical, with only 2-5% runtime over- head. Beyond state-of-the-art performance, we further demonstrate FusionProt's practical relevance through representative biological case studies, suggesting that the model captures biologically relevant features.
1 Introduction
Protcins essential for biological processes and for understanding complex mechanisms in living organisms. They comprise linear chains of amino acids that fold into a specific three-dimennionall (3D) structure, which underscores their functional diversity and dynamic behaviors (Zhang et al. 2023b). An effective understand- ing of proteins is essential for understanding disease mechanisms and synthetic biology and for advancing drug development (Liu et al. 2024).
Current methodologics for protein representation primarily emphasize the exploration of proteins' one- dimensional (1D) structures, specifically the relationships between amino acids. These approaches, such ProteinBERT (Brandes et al., 2021) and ESM (Rives et al.) 2019 Lin et al. 2023). often utilize text- based techniques and transformer architectures (Vaswani et al. 2017), which are trained on extensive protein sequence datasets. These models take an amino acid sequence as input and typically produce protein repre- sentations by averaging the representation ations of individual amino acids. However, this narrow focus on amino acid sequences neglects crucial protein structure details, thereby limiting the effectiveness of these methods.
The intricate 3D structure of proteins is crucial, as the conformation plays a pivotal role determining their activities (Zhang et al. 2023a) Proteins possess specific act sites for interactions with other molecules.

---

Published in Transactions on Machine Learning Research (11/2025)
which are defined by the 3D arrangement of amino acids. This 3D structure determines the specificity and affinity of binding interactions, aspects that 1D representation cannot adequately capture (Kastritis & Bonvin 2013 Yan et al. 2013). Furthermore, drug design relies on an understanding of 3D structures to identify binding sites and find molecules that modulate protein function (Luo 2022 Liu ct al. 2022). One of the state-of-Uhe-arl (SOTA) techniques loday for 3D profein represenlation is GearNel (Zhang el al. 2023b)
This approach converts the 3D structure of protein into a graph that captures its biological characteristics.
Subsequently, graph neural network techniques (Kipf & Welling 2017 Schlichtkrull et al. 2017 are applied to this graph, facilitating the creation of comprehensive protein representations.
Recent rescarch emphasizes the importance of comprchensive protein representation that includes both ID and 3D structures to capture the protein's functional and interactional properties accurately. ESM- GearNet (Zhang et al. 2023a) was one of the first approaches to integrate these modalities. Although the study explored various fusion strategies, empirical results showed that the most effective method is using a large protein language model (PLM) such as ESM (Lin et al. 2023) to generate representations, which were then used as conlext for a graph encoder like GearNet (Zhang al.] 2023b). Other approaches, such LiphaFold-based model (van Kempen al. 2022) to reduce the 3D
SaProt (Su et al. 2024), leverage structure to tokens and train them along with amino acid tokens using a traditional PLM. However, these approaches are limited as they reduce one modality into context for another model, which potentially leads to the loss of critical structure information.
In this study, we introduce FusionProt (see Figure I a novel approach designed to learn a unified represen- tation of the 1D and 3D structures of proteins simultaneously. Despite the vast number of proteins in nature, the number of known 3D structures remains limited (Váradi et al., 2023 To address this, we leverage an AlphaFold model (Jumper et al. 2021) for accurate protein structure predictions. We introduce an innova- tive learnable fusion token that serves as an adaptive bridge, enabling an iterative exchange of information between a PLM and the protein's 3D structure graph. This token is integrated into the training process of both modalitics, enabling scamless propagation of information and facilitating comprehensive represen- fation through iterative learning cycles. In practice, this token is concalenated to the sequence, allowing attention mechanisms to query the unique fusion token alongside the amino acids. This process extracts and integrates valuable information, enhancing the learning of amino acid representations. Then, the fusion token is incorporated as an additional node in the graph representing the protein's 3D structure, connected to all nodes. A graph encoder (i.e., structure model) processes this graph, generating a new representation for the fusion token, which is subsequently used in the PLM training over the amino acids. Through this iterative process, the model representations are combined to form a refined protein representation.
Unlike prior models such as FSM-GeeaNeet (Zhang et al. 2023a), which perform static or one-shot fusion, such as simple concatenation after independent encoding, FusionProt introduces a dynamic mechanism where the fusion token continuousl evolves through repeated interaction across layers. This design allows sequence and structure modalities to co-adapt throughout training, resulting in richer, functionally informed reprosentations that better capture the complexity of protein behavior.
We perform an empirical evaluation over several protein lasks, spanning a broad spectrum of biological domains, comparing numerous methods of protein representations and achieving SOTA performance across various benchmarks, with statistical significance improvements. We present ablation tests to better study the performance of the algorithm.
The contributions of this study are threefold: (1) We introduce FusionProt, a novel approach that learns unified representation for both the 1D and 3D structures of proteins simultancously. Our main focus is on the fusion of 1D and 3D models in an effective manner. Our method enhances the accuracy of capturing functional and interactional properties of proteins, addressing limitations of previous methods that treated these structures separately; (2) We propose a novel fusion architecture that utilizes a specialized learnable fusion token, enabling an iterative exchange of information between a PLM and the protein's 's 3D structure graph. This token is integrated into the training process of both modalities, enabling seamless propagation of information and facilitating comprchensive representation through iterative learning cycles. The itcrative process facilitates the exchange of contextually relevant structure and sequential features, improving the model's ability to capture both 1D and 3D protein characteristies; (3) We conduct an empirical evaluation
2

---

Published in Transactions on Machine Learning Research (11/2025)
of our work over several protein tasks establishing SOTA results and presenting biological case studies that further demonstrate the model's strengths. Also, we contribute our code to the community H
2
Related Work
2.1
Sequence-based Representation Learning
Proteins are comprised of sequences of amino acids, that establish natural analogy to tokens in natural language processing. Recently, the adoption of unsupervised deep learning techniques has become prevalent in modeling protein sequence data.
The advent Transformers Vaswani et al. 2017) has led to the development of numerous PLMs such as MSA Transformer (Rao et al. 2021]. ProteinBERT (Brandes al., 2021), ProteinLM (Xiao al., 2021) ProtBERT-BFD Elnaggar et al. 2021). Prott (Flnaggar et al. 2022 ESM-1b (Rives et al. 2019 and ESM-2 Lin et al. 2023) which is considered as the SOTA PLM. These models were trained on data from UniRef (Suzek el al. 2007 which contains hundreds of billions of protein sequences, via masked language
modeling (Devlin et al. 2019).
A protein's biological function hinges on its 3D native structure Dill & MacCallum 2012). However, many PL.Ss do not explicitly encode protein 3D structures, which are pivotal understanding protein functions.
In this work, we aim overcome this limitation by enhancing a PLM through novel fusion algorithms that integrate protein 3D structure models into the protein embedding. This approach seeks to capture both sequential and structure attributes of proteins, thereby advancing the capabilities of existing PLMSs in biological research and applications.
2.2 Structure-based Representation Learning
The rising success of AlphaFold (Jumper et al. 2021 Senior et al. 2020 in predicting the 3D structure of proteins has led to deeper insights into their functional roles. Moreover, the release of more than 200 million protein structures in AlphaFoldDB (Váradi et al. 2023 2021 has significantly advanced the development of large-scale protein structure models (van der Weg et al. 2025
Protein structures are commonly represented as graphs, where amino acids serve as the nodes. Therefore, utilizing protein structure models on these graphs a common practice. Models such as GVP (Jing et al. 2021), CDConv [Fan et al.) 2023 and GearNet (Zhang et al. 2023b have shown promising results, with different learning techniques. GearNet, which incorporates a Multiview Contrastive pre-training algorithm, is considered SOTA (Zhang al. 2023b) and has outperformed the IEConv model (Hermosilla et al. 2022)) which proposed to apply Iearnable kernel function edge features. Furthermore, CDConv (Fan et al.
2023 has outperformed HoloProt (Somaah et al. 2022 and ProNet /Wang et al. 2022a in recent study Liu et al. 2023a).
Foldseck fyan Kempen et al. 2022) suggested a different approach, optimized for protein structural search, which utilizes a VQ- VAE (van den Oord al. 2017) to encode protein structures into informative tokens. Then, SaProt (Su al. 2024) integrates residue and structure tokens during training, derived from encoding 3D protein structures using Foldseek.
In this study, enhance the structure-based representation learning approach by integrating sequential information, thus capturing both sequential and structure attributes of proteins.
2.3 Joint Representation Learning
The integration of protein sequence- based models with protein structure models has gained popularity in
recent years QQan et al. 2024 Ko al. 2024 Wu et al. 2022
al.) 2024).
http:://ggtaub.com//aaafaaaaaaat
3

---

Published in Transactions on Machine Learning Research (11/2025)
Sequential Layer
Structure Layer
Sequential Layer
Structure Layer
Linen Transformation
Sequential Loyer
Structure Layer
Embedding Layer
Sequence
ALGKE
L
Structure
Protein
Figure The FusionProt pre-training architecture. The model is trained on amino acid sequences and their corresponding 3D structures, utilizing the SOTA AlphaFold2 (Jumper et al., 2021) for accurate protein structure predictions. We introduce an innovative learnable fusion token that serves as an adaptive bridge, enabling iterative exchange of information between PLM and the protein's 3D structure graph. This token is concatenated to the protein sequence, allowing attention mechanisms to query the unique fusion token alongside the amino acids. Then, the fusion token is incorporated as an additional node in the graph representing the protein's 's 3D structure, connected to all nodes. A graph encoder (i.e., a structure model) processes this graph, generating a new representation for the fusson token. This representation is subsequently usod the PLM's sequential layers. learnable linear transformation applied between each sequential and structure layer pair aligu and adapt their distinct modality spaces. Through this iterative process, the model representations combined to form refined protein representation.
Early efforts, such as LM-GVP (Wng al. 2022b). SSEmb (Blaabjerg et al., 2024 MIF (Yang et al. 2022 and MIF-ST YYngg al. 2022). aimed combine PLMs with Graph Neural Networks (GNNs) (Scarseli et al. 2009). Related contrastive alignment methods, such as RLA (Birnbaum et al. 2024 and S-
PLM Wang et al. 2025 learn to align sequence and structure embeddings but typically fuse modalities only at the representation level rather than within the token-level encoder.
More recently, ESM-GearNet (Zhang et al. 2023a proposed to incorporate sequential information into distinct residue-level models such as GearNel, GVP, and CDConv (Zhang el al. 2023a). SaProt-GearNet (Su et al. 2024) also integrates GearNet and achieves similar results. However, these approaches are limited by reducing one modality into context for another model, potentially losing critical structure information.
Unlike earlier approaches, representation for both the 1D and 3D structures of proteins. To the best of our knowledge, no existing PLMs are based on an iterative fusion of structure and sequential models simultaneously.
introduce novel fusion architecture that simultaneously learns unified
3 Methods
We introduce the FusionProt model (see Figure ! Given a protein p (S, R), which combines its amino acid sequence S and its 3D structure R, FusionProt simultancously learns a unified representation for both the 1D and 3D structures of the protein. We propose a specialized learnable fusion token designed enable iterative exchange of information between PLM and the protein's 's 3D structure graph.
4

---

Published in Transactions on Machine Learning Research (11/2025)
In this section, formally define sequential (ID) and structure (3D) protein layers, which are used in the model as part of the protein representation learning, and present the FusionProt algorithm. Information from both layers is fused simultaneously during learning throughout the proposed specialized fusion token.
3.1 Protein Sequential Layer
3.1.1
Protein Sequence
The simplest level of protein structure, known the primary structure, structure, consisting of a linear sequence of amino acids, referred to as residues. The protein sequence exhibits similarities with natural language sequences, making the application of language models common practice in this domain (Xiao et al.
2021 Brandes et al. 2021
Given a protein sequence S = [$1, .., sn]: where denotes the number of residues, the sequential model aims to capture the essential features of the protein sequence and outputs a protein representation denoted [1, 22, 2n] E _=*D, where D the embedding dimension.
by z
3.1.2 Sequential Layer Definition
We utilize ESM-2 [Ln et al. 2023) as our sequential model, leveraging its superior performance as a PLM. Alternatively, other advanced sequential models (i.e., PL.Ss can be used.
Within the context of given sequential layer denoted by I, we denote z(4) as the output representation of this layer, initialized with z! = Embeddingg(s) E RD,, where D denotes the embedding representation dimension. The layer performs the following update:
20) Attention (211-1) (1-1) 2-,,),2111 2n 11-1))
where n denotes the number of residues in the protein, and Attention is a multi-head self-attention layer Vaaswani el. al. 2017).
3.2 Protein Structure Layer
3.2.1
Protein 3D Structure
A proteiin 3D structure is uniquely determined by its primary structure (amino acid sequences) (Dill & Mac-
Callum 2012). There are only 20 standard residue (amino acid) types, each containing multiple components connected to central carbon atom known alpha carbon. Following GearNet (Zhang al. 2023b we use only alpha carbons to represent the main backbone structure of each protein. Therefore, we define a protein 3D structure as R = T2..Trl E Rox3 where r; represents the Cartesian coordinates of the i - th alpha carbon atom of cach amino acid, and n denotes the number of residues, which the sequence length.
3.2.2 Protein Structure Graph
We represent proteins using multi-relational residue graph G = (V, E, E,T), where V is the set of residues, E is the set of edges, and T is the edge types. The set of edges E consists of three directed edge types, namely sequential edges, radius edges, and KNN edges:
Eseg = {(i,5) i,j V. - dseg}
Eradius {(i.]) i.j E V. -ral < dradius
ERNN [(i,]) i,j E V,j E KNN(O)
E Eseq U Eradius U ERNN
where doog defines the sequential distance threshold, drodius 10A 1 [nm] defines the spatial distance threshold, and KNN(I) indicates the K-nearest neighbors (Peterson) 2009) of node with k = 10 (all param-
5

---

Published in Transactions on Machine Learning Research (11/2025)
eters were set reported in GearNet (Zhang et al. 2023b)) )) In Section 5.5 present ablation study
quantifying the impact of varying these values on predictive performance across the tasks.
3.2.3 Structure Layer Definition
We utilize GearNet (Zhang ct al., 2023b/ the structure model, leveraging its contextual understanding of protein structures. Alternalively, other advanced structure models can be used in the algorithı (see Section 5.2).
Given a protein's 3D structure, R, we construct its corresponding protein structure graph G = (V, V, (see Section B.2.2.. Then, the structure layer employs a relational message passing procedure, based on a relational graph convolutional neural network (Schlichtkrull et al. 2017).
Within the context of given structure layer denoted by we denote u() the output representations of this layer, initialized with un = Embedding(v)) E IR, where v, E V, and denotes the embedding representation dimension. The layer performs the following update:
4 = o W, ( -
where Mi(1 is the set of neighbors of with edge type 1, (.) is a ReLU activation function, and the weight matrix W, learned per edge type t. This approach allows the model to incorporate various types of relational information, thereby enhancing its ability to learn comprehensive protein representations.
3.3 PusionProt
3.3.1 Fusion Design
We introduce a specialized learnable fusion token, designed to enable an iterative exchange of information between a PLM and the protein's 3D structure graph. This fusion token serves as a dynamic bridge, enabling iterative information exchange between the two modalities. During training (see Figure [ the specialized fusion token holds information from both models, the sequential and the structure. In practice, the amino acids query the associated unique fusion token extract and integrate valuable information for learning amino acid representations via attention mechanisms. Additionally, the fusion token is incorporated as additional node in the graph representing the protcin's 3D structure. We connect it to all nodes, with new edge lype, thus enabling the structure model to learn important features from the sequential model. Hence, the token enables a novel fusion between both models, leading to an enhanced protein representation.
Our choice to connect the fusion token to all residues follows GNN principles (Scarselli et al. 2009), enabling information exchange that captures both global and local dependencies. This universal reach provides single- hop access to any site, preventing bottlenecks in elongated or multi-domain proteins that arise with limited neighborhood expansion. The design is analogous to virtual nodes that aggregate and broadcast global context (Hwang et al. 2022 Xu et al. 2019 Zhang al. 2025). but here the token mediates cross-modal communication between the PT.M and the 3D graph rather than graph-only messaging. A complementary line, graph prompt learning, prepends learned prompt tokens to steer a single-modality GNN or a pre-trained graph model toward the task (Liu et al. 2023b Zi et al.) 2024]. In contrast, our fusion token participates at every block both PLM attention and GNN message passing, instance-conditioned and updated layer-by- layer, and thus provides bidircctional routing while prescrving modality-spociffc inductive biases (sequence order; 3D geometry).
The fusion token is not merely an addition to the sequence structure; rather, it mediates the dynamic interaction between the two modalities, ensuring itcrative refinement of the protcin representation. This approach contrasts with previous methods that simply concalenate or independently process the sequence and structure information. This novel fusion mechanism leverages multi-modi learning and graph-based information propagation. By iteratively combining the 1D protein sequence and 3D structure features, FusionProt captures both local and global dependencies, enabling the model to learn richer and more holistic representations of proteins.

---

Published in Transactions on Machine Learning Research (11/2025)
To provide transparency into this process, visualize (see Supplementary Section A.3 insion-toke atten- tion across layers and around each structure injection.
Algorithm FusionProt Algorithm
Input: Protein - (S, Rt) with sequence S = [8,, sn], and 3D structure R = [ri, Tal E R Sequen- tial encoder with L1 layers, Structure encoder Lg layers.
Output: Unified protein representation
Create learnable fusion token 8n+1 $n+1=f
Augment. sequence: S [,,,-.. 8n, Sn+1
4: Construct graph = (V, from Embedding(ss) Vi [I,..., 1] Add node 1x+1 to G, connect eh+1 <> v with edge type b E [
3: Initialize sequential embeddings: (0)
Set (0) Embedding((:)) for all Us V
7: for = 1 to L2 do
8:
for 1 to L1 + L2 do
9:
1, = j + (1- (1-1) (L1 + L2)
204) Allention(2)
11,-1) am Am
(1:-1) (11-1)
10:
11:
end for
(1-1) Linearr 2441)
12:
"n+1
Update node un+1 in graph G with (1-1)
13:
u u; (1-1) +o EiET W, jje uj 1)1 Aug V
14:
(4) = Linear/ (M
15:
2++1
16: end for
(((a)) (12) 17: Concatenate final embeddings: (Lx), Uim
18: return
3.3.2 Fusion Algorithm
We provide formal description of the FusionProt algorithm in Algorithm
Given a protein p (S, R), which combines its amino acid sequence S and its 3D structure R, FusionProt concatenates the learnable fusion token to the sequence, resulting in sequence S [81, $2,.... 8a, with a length of n + 1. Then, it is used to initialize the sequential embedding layer:
Vs: S' 2 Embedding(ss)
Within the context of a given sequential layer denoted by I, the algorithm yields a sequence representation:
20) Attention 2) ((-1), (1-11
2n (1-1)
=x+11
where (1-1) 2x+1 is the intermediate representation of the fusion token. Then, the fusion token representation is passed into the corresponding structure layer represented node 1, while connecting to all nodes in the 3D structure graph, yielding a structure representation:
u (1) (1(-) + (Eier W, Eyenio 1
where u (111 Linear (= T = TU No, (n +1) = and E [I,...,n] : Ni, (1) (n + 1}, as
we connected the fusion token to all residues in the protein structure graph with a new type of edge ty (in both directions). To bridge the gap between the sequential and structure representation space, utilize

---

Published in Transactions on Machine Learning Research (11/2025)
affine transformation Lineary (a linear layer) per each structure layer to project these representations into the same space, Subsequently, we set the fusion token representation w as the input for the following sequential layer 1+ 1, resulting in:
(1) 2++1 = Lineary (1u+1)
where Lineary the affine transformation in the opposite direction (i.e., structure to sequential space vice versa).
The protein representation undergoes transformations across the structure and sequential layers, where L1 and L2 are the number of sequential and structure layers, respectively, and L 2 L2 (note that L need not be equal to I2). Our aim is to uniformly integrate 3D structure information across the I sequential and Le structure layers. Therefore, the fusion token passed after L + L2 sequential layers into the structure model. Then, the fusion token is passed back into the sequential model after every structure layer.
Lastly, the final hidden states of both the sequential and structure layers concatenated to form the final protein output representation, which is:
h (((1), (bb
3.4 Pre-training Objective
Following GearNet (Zhang et al. 2023b and ESM-GearNet (Zhang et al. 2023a Multiview Con- trastive learning
pre-training objective, for a fair comparison. Other pre-training algorithms could also be applied. We compare different pre-training algorithms in Section 5.3
For each protein structure graph G (see Figure 2 for an illustration), we construct two augmented views, Is and Oy. We first perform subsequence cropping by sampling, for each view, a contiguous residue window I [s, stw] along the protein sequence and taking the induced subgraph G[Z]. This captures putative recuuring contiguous subsequences that often signify function (Zhang al. 2023a). We then follow standard self-supervised learning practice by injecting stochastic noise in structure space: for each view, we perform random edge masking, independently dropping each remaining edge with probability = 0.15. This generates structurally perturbed but still semantically consistent views of the same protein. Encoding frr and gy with FusionProt to get representations, projecting and fg-normalizing them, we optimize an InE loss (van den Oord et al. 2018 where (gn) form the positive pair and negatives g' are views from other proteins in the batch.
FusionProt
LVCGERA...
Contrative Learning
Subeequance cropping
Subsequence cropping
FusionProt
LVCGERA...
Figure 2: Illustaaioon of the Multiview Contrastive pre-training objective Zhang et al. 2023a.). Two con- tiguous residue windows are cropped to form induced subgraphs (ga,gy), 15% of edges are randomly masked, and our FusionProl's 's representations are aligned with InfoNCE against negatives g from other proteins the batch.
8

---

Published in Transactions on Machine Learning Research (11/2025)
4 Empirical Results
In this section, we present our empirical results. For each lask, we report the mean and standard deviation of FusionProt'ss performance over five independent fine-tuning runs (with different random seeds) to ensure reproducibility. Additionally, we validate the statistical significance of performance differences, using a two- lailed paired t-lest. with 95% confidence level (p 0.05), comparing observations from the tested models. The normality of the paired differences confirmed using the Shapiro- Wilk test (Shapiro & Wilk 1965 Significant results are marked with asterisk (*) in the tables.
Finally, we compute Cohen's 's effect sizes (Cohen 1969) to quantify the magnitude of Fusion Prol's improve- ments over baselines on each task. We observe large effects (d > 0.8) across all tasks, indicating that the improvements are not only statistically significant but also practically meaningful.
In Supplementary Section we outline our empirical setting, including the pre-training dataset, task details, baselines, implementation details, and computational complexity analysis.
Table 1: Evaluation results on EC and GO prediction under various pre-trained baseline models. "PLM" and "Structure Info." indicate the usage of protein language models and structure information in the model, respectively. The Fmax score is the evaluation metric. Statistically significant. results (p 0.05) using paired t-test across proteins in the test set marked with asterisk (*). For FusionProt, we report the mcan and standard deviation over 5 independent finc-tuning runs (different seeds; EC: n 1604, GO:
= 3350). The best result is highlighted in bold.
Structure Info.
EC
GO-BP
GO-MF
GO-CC
Method
PLM
Faax
Fmax
Fmax
Frax
ProtBERT BBED
Elnaggar
2021
0.838
0.279
0,456
0.408
DeepFRI
Gligorijevéé et al. 2021
0.631
0.399
0.465
0.460
ESM-IB 1b
Rives
2019
X
0.859
0.320
0.661
0.392
ESM-2
Lin et al.
2023
0.877
0.345
0.668
0.411
GVP
Jing et al.
2021
0.886
0.495
0.672
0.420
CDConv
Fan
2023
0.820
0.453
0.634
0.479
GearNet
Zhang et al.
2023b
0.871
0.481
0.650
0.476
MIF-ST
Yang et al.
2022
0.803
0.239
0.627
0.322
S-PLM
Wang
2025
0.885
0.470
0.674
0.460
ESM-Gear.ett
(Zhang et al.
2023a
0.886
0.512
0.670
0.495
SaProt
Su
2024
0.884
0.486
0.678
0.479
SaProt-GearNet
Su et al.
2024
0.886
0.512
0.672
0.504
FusionProt
0.904"+4.000
0.524"+0.004
0.689"+0.0002
0.518"10.00
4.1 Task 1: EC Number Prediction
In Table 1 we compared the performance of FusionProl with eleven baseline methods on the EC number prediction task. In particular, FusionProt significantly outperformed all baseline methods, achieving the highest. Fmax score.
PLMs such ProLBERT-BED (Elnaggar al. 2021 or ESM-2 (Lin al. 2023 which rely solely on sequence data, produced substantially lower Fmax scores, compared to FusionProt. This highlights the benefit of incorporating 3D structure information into representation learning.
Similarly, FusionProt, which leverages both structure and sequential information, performed better than SOTA structure models such as GearNet (Zhang et al.) 2023b or CDConv (Fan al. 2023), which trained only on 3D structure data. This indicates that while structure models are strong, they are not sufficient alonc, as they require PLMs for optimal performance, probably due their effective self-attention layers (Vaswani lt al.
2017
9

---

Published in Transactions on Machine Learning Research (11/2025)
While models such MIF-ST Yang et al.
2022 S-PLM (Wang et al. 2025
ESM-GearNer (Zhang
et al. 2023a), and SaProt-GearNet (Su et al. 2024) also attempt to utilize both types of information, they reduce one modality into a context for another model, leading to a loss of critical structure information. In addition, the importance of the fusion technique emphasized by the significant underperformanee MIF- ST compared to other fusion models. Similarly to ESM-GearNet, MIF-ST uses outputs from pre-trained soquence-only PLM as input to a structure model.
By integrating 1D and 3D protein structure information synergistically, FusionProt attempts capture subtle structure features that influence enzyme specificity and activity, which are crucial for EC prediction. This performance improvement is essential for real-world applications, such as the diagnosis of enzyme deficiency-related diseases Li et al. 2017
4.2
Task 2: GO Term Prediction
Table 1 presents the results of the GO term prediction tasks compared to eleven baseline methods, including PLMs, structure models, and ensemble models. FusionProt showed strong performance across the board, achioving the highest Fmax scores in all tasks, with a statistical significance.
The results demonstrate that sequential or structure information alone insufficient, similarly to the FCC prediction task (see Section 4.1). This indicates that the novel fusion of FusionProt is critical for achieving superior performance.
In the GO-BP task, which involves predicting protein's 's role in biological processes, FusionProt achieves an Fmax score of 0.524, significantly outperforming the next-best method. Biological processes often rely on long-range structural interactions and cooperative protein functions, which makes this task particularly sensitive to accurate 3D structure representations. The substantial performance gap between SaProt and SaProt-GearNet further emphasizes the critical role of structure this setting, a factor that FusionProt leverages more elfectively through early integration.
In the GO-CC task, which predicts the cellular component of proteins, FusionProt achieves an Fmax of 0.518, outperforming all basclines including ESM-GearNet (Zhang et al.) 2023a), probably due to the more direct connection between 3D structure characteristics and subcellular localization (Gillani & Pollastri 2024). The prediction of cellular components strongly based on the spatial organization of proteins within the cell, which is well represented by detailed 3D structures (Song et al., 2022).
This reliance structural information benefits models such CDConv (an et al. 2023 although it exhibits weaker performance in EC prediction. In contrast, predicting molecular functions involves complex interactions and dynamic changes that are less directly captured by 3D structures (Saraç et al. 2010 resulting in comparatively smaller performance gap than in the other tasks.
4.3 Task 3: Mutation Stability Prediction
Table 2 reports the results for the Mutation stability prediction (MSP) task, which evaluates a model's ability to assess the effect of amino acid substitutions on protein stability, a critical problem in protein engineering, variant effect prediction, and drug design.
Following the evaluation protocol of ESM-GearNet (Zhang et al. 2023a 1, we compare FusionProi on the MSP task against the SOTA structuro-based method for this task, GVP (Jing et al. 2021) and the joint sequence structure baseline ESM-GearNet. (Zhang et al., 2023a).. Sequence-only PLMs do not take structural inputs and therefore cannot address grructur-depennden tasks such MSP (Zhang al. 2023ah.
FusionProt achieves the highest AUROC among all evaluated methods, reaching 0.745 with statistical signif- icance (p < 0.05), out performmngngg both struclure-aware (GVP) and sequence-structure fused (ESM-GearNet) baselines. Notably, FusionProt improves upon
by approximately 24.37%, and surpasses the current SOTA GVP by 5.1%. This highlights the offectiveness of our iterative fusion mechanism in pre- serving and integrating long-range sequence and structural dependencies, which are especially important for modeling stability changes that may arise from distal context-dependent mutations.
10

---

Published in Transactions on Machine Learning Research (11/2025)
Compared to GVP, which is limited to local spatial interactions, and ESM-GearNett, which performs shallow one-shot fusion, Fusion allows multi-layer cross-modal refinement via learnable token and bidirectional interaction path. This enables more expressive and biologically grounded representation, particularly for capturing distributed compensatory effects allosteric regions, commonly overlooked in residuc-centric models.
Table 2: Evaluation results on MSP prediction across different models. "PLM" and "Structure" indicate the usage of protein language models and structure information in the model, respectively. The AUROC score is used as the evaluation metric. Statistically significant results (p < 0.05) using a paired t-test across proteins in the test set are marked with an asterisk (*). For FusionProt, we report the mean and standard deviation over independent fine-tuning runs (different seeds; MSP: 347). The best result. is highlighted in bold.
Method
PLM
Structure
MSP (AUROC)
ESM-GearNet Zhang et al. 2023a GVP JJing et al. 2021 (SOTA)
0.599
X
0.709
V
FusionProt
0. 0.745"10.000
5 Ablation and Analysis
We conduct series of ablation and diagnostic studies to evaluate the key design decisions behind FusionProt These include varying the fusion-injection frequency, comparing different structure encoders, and testing alternative pre-training objectives. We also assess the model's robustness to noise in predicted 3D structures. Finally, we present biological case studies that highlight FusionProt 's practical utility in real-world tasks such as drug discovery and discase research.
5.1 Ablation on Fusion-Injection Frequency
We aimed determine the optimal number of fusion injections our FusionProt model, which fused information between the structure and sequential models using a specialized learnable fusion token. In our standard method, given sequential model with L₃ layers and structure model with Lg layers, where I 2 L2 the fusion token is passed after T, + I2 sequential layers into the structure model (see Section 3.3.2 Then, the fusion token is passed back into the sequential model after every structure layer. In this ablation, we experimented with different injection frequencies: half the regular number (every 2 (L1 + L2) sequential layers and two structure layers) and one-third the regular number (every 3 (L1 + L2) sequential layers and three structure layers). The results in Table 3 indicate that the standard fusion injection frequency consistently provides the best. performance across tasks. Redueing fusion injections negatively impacted performance, highlighting the importance of the fusion token. By decreasing the frequency of injections, the model's ability integrate these two types of information is compromised, leading to suboptimal predictions.
Table 3: Ablation test for the number of fusion injections in FusionProt. Statistically significant results with < 0.05 using a t-test are marked with an asterisk (*). The best result is highlighted in bold.
EC
GO-BP GO-MF GO-CE
Number of Fusion
Fmax
Fmax
Fmax
Fmax
Few (One-Third)
0.876
0.511
0.659
0.498
Medium (Half)
0.882
0.514
0.669
0.504
Full (Standard)
0.904
0.524
0.689"
0.518*
11

---

Published in Transactions on Machine Learning Research (11/2025)
5.2 3D Structure Models Comparison
FusionProt is designed to be flexible and modular, allowing the structare encoder component to be replaced with alternative architectures. The ablation study for various SOTA 3D structure models is detailed in Table 4 We evaluated several structure models, including GcarNet (Zhang ct al., 2023b), GVP (Jing et al. 2021h, and CDConv (Fan et al. 2023). For each structure model, we applied our fusion technique and compared its performance with FusionProt, which utilizes the GearNet (Zhang et al. 2023b) model (see Section 3.2.3). The GVP model replaces standard MLPs (Murtagh) 1991] in GNN (Scarselli et al. 2009) layers with generalized vector perceptrons, which handle scalar and geometric features as vectors that adapt to spatial rotations. In contrast, CDConv utilizes GearNet's multi-type message passing to capture sequential and spatial interactions among residues. For consistency, follow the model selection protocol of ESM-GearNet (Zhang et al.
2023a). Our results demonstrate that the FusionProt model, utilizing the GearNet structure, consistently achieves superior performance compared to other models. This aligns with previous studies (Zhang et al. 2023a which have also established GearNet as a SOTA approach for joint representation learning.
Table 4: Ablation test for various structure models used by FusionProt. For all models, the PLM is the ESM-2 (Lin et al. 2023). Statistically significant results with p 0.05 using t-test are marked with an asterisk (*). The best result is highlighted in bold.
EC
GO-BP
GO-MF
GO-CC
Method
Fmax
Fmax
Fmax
Fmax
FusionProt (GVP)
0.889
0.507
0.677
0.462
FusionProt (CDConv)
0.837
0.478
0.665
0.497
FusionProt (GearNet)
0.904"
0.524*
0.689*
0.518*
5.3 Pre-training Algorithm Ablation
Table compares FusionProt, trained with Multiview Contrastive learning (Section 3.4 training algorithms. First, tested self-prediction methods (Zhang et al. 2023b), which aim to predict one part of the protein given the remaining conlext. Specifically, utilized the Residue Type Prediction method, self-supervised task that performs masked prediction on individual residues. Next, we evaluated diffusion- based methods (Zhang et al., 2023c), inspired by the success of diffusion models in capturing the relationship between sequences and structures. During training, noise levels are added to structures and sequences, with higher noise levels indicating more distortion. We tested the Siam objective, which refines structures via torsional adjustments and noise reduction. The results show that the Multiview Contrastive approach outperformed both Residue Type Prediction (using only sequence data) and Sifm (which integrates sequence and structure separately). Unlike methods that treat sequence and structure separately, Multiview Contrast integrates both, aligning subsequence representations from the same protein to capture interrelated sequences and structure patterns.
with other pre-
Table 5: Ablation test for the FusionProt model with different pre-training algorithms. Statistically signif- icant results with p 0.05 using t-test marked with asterisk (*). The best result is highlighted in bold.
EC
GO-BP GO-MF GO-CC
Method
Fmax
Fmax
Fmax
Fmax
Residue Type
0.890
0.524
0.673
0.506
SiamDiff
0.874
0.509
0.645
0.512
Multiview Contrast
0.904*
0.524*
0.689*
0.518*
12

---

Published in Transactions on Machine Learning Research (11/2025)
FusionProt performance on EC across K (KNN)
FusionProt performance dradus
0.9050
0.9025
0.9025
0.9000
Fmax
0.8975
0.8975
0.8950
0.8925
0,8900
0.8900
30 (KNN)
12
12
discks (A)
(a) Fmax EC vs. k (KNN). Points at l E {8, 10, 12}; k=10 in the original settings.
(b) Fmax on EC vs. dradins- Points at
dradius E {8, 10, A dradius=10A in the original settings.
Figure 3: Ablation of graph-conntruution parameters the EC prediction task. Fmax function of neighborhood size: k (KNN) and dradius (spatial threshold). All other settings are fixed.
5.4 Robustness to Cow-Confidence Regions in 3D Structures
Robustness in protein modeling often refers a model's ability tolerate minor perturbations (Cho et al. 2024) an important property given that predicted structures (e.g., from AlphaFold2) may contain low- confidence regions, reflected as segments with low pLDDT scores. These regions are less reliable for practical To assess FusionProt's robustness in such scenarios, simulate low-confidence regions by injecting Gaussian perturbations into the alpha-carbon backbone coordinates of AlphaFoldl-preciiteed structures. The noise was zero-mean, with standard deviations ranging from 0.1 Å to 1.5 Å, simulating local structure deviations commonly observed in predicted or low-confidence regions. We then fine-tune the model on the EC and GO-BP downstream tasks and evaluate performance across these noise levels.
FusionProt maintained stable performance up noise level of 0.9 A, with only a modest 3 points absolute drop in Fma EC and GO-BP tasks beyond this threshold. These results indicate that FusionProt is resilient to realistic structural noise and remains effective even when relying on imperfect 3D structures, commonly encountered in high-throughput or large-scale prediction settings.
5.5 Graph Construction Parameters Ablation
We ablate graph-construction choices the EC prediction task (see Figure 3) by varying k (of the KNN) and the spatial threshold dradius used to build the protein structure graph (see Section 3.2.2 Across k E (8, 10, 12) and dradius E {8, 10, 12} Å, performance is stable with a broad optimum near 1=10 and 11 Å, indicating that FusionProt is robust to reasonable graph choices and that our defaults are optimal and recommended..
Across EC and GO tasks, FusionProt is largely robust to protein length, which we attribute chiefly to requence-truncation protocol (see Section A.1.4p. When stratifying input proteins by sequence length, longer proteins (2 600 residues) benefit slightly from denser connectivity (A=12 or dradius=12 Å), which better preserves long-range contacts, resulting in higher Fmax in downstream tasks (EC: +0.5 percentage points in Fmax with < 0.05 using a t-tcst).
5.6 Biological Case Studies
FusionProt consistently outperforms SOTA baselines across all downstream tasks (Table I Table 2 When such performance gains are driven by biologically meaningful inputs, such as detailed 3D structural features, they can indicate that the model is capturing relationships of functional relevance.
To better understand the nature of these improved learned protein representations, we evaluated model's ability to predict F.C. numbers, outlined in Section A.1.2 using pre-trained representations without, any
13

---

Published in Transactions on Machine Learning Research (11/2025)
fine-tuning. This approach enables assessment of the inherent quality of the learned embeddings, inde- pendent of task-specific training. We then compared FusionProt's predictions with those of ESM-GearNet, focusing on the cases with the largest discrepancies.
In Supplementary Table we present these proteins into distinct biological groups based on shared mecha- nistic characteristics. We focus on two common mechanisms and provide a representative example for each group (see Supplementary Section A.2
6 Conclusions
In this paper, we introduced FusionProt, a novel architecture designed to learn unified representation of the 1D and 3D structures of proteins simultaneously. FusionProt incorporates specialized learnable fusion token that enables iterative exchange of information between PLM and the protein's 3D structure graph, facilitating more comprchensive representation through iterative learning cycles between PLM and a structure model.
We propose a novel fusion algorithm that enables effective propagation of information between a PLM and a structure model. This fusion technique outperforms previous methods, which often convert one modality into context for another model, potentially leading to the loss of crucial structural information.
We evaluated our proposed method several protein-level tasks to assess its effectiveness in protein rep- resentation learning. These tasks include protein annotations, which are essential for real-world medical applications. FusionnProt significantly outperformed all baseline methods, including SOTA models, across all tasks, with statistically significant improvements.
We perform ablation tests to examine the contribution of our novel approach of learning unified represen- tations of both ID and 3D structures of proteins simultaneously. The tests confirm that our approach significantly enhances FusionProt's ability to capture the intricate relationships between protein sequences and their 3D structures, leading to improved performance across various real-world tasks.
Although the proposed fusion token mechanism is tailored for bidirectional interaction between sequence and structure modalitios, it is conceptually extensible to more than two information channols. For example, additional modalities such as ligand descriptors, protein dynamics, or expression context could, in principle, be integrated into the framework by assigning separate fusion tokens or adopting a shared token passed through each encoder block in sequence. We leave the systematic evaluation of such multi-modal extensions to future work. Additionally, although we evaluate MSP for residue-level perturbations, MSP does not replace standardized DMS benchmarks; it is closely related and indicates benefits from structural cues. In future work, we will expand evaluation to DMS-based datasets (c.g., ProtcinGym).
References
Foster Birnbaum, Saachi Jain, Aleksander Madry, and Amy E. Keating. Jointly embedding protein structures and sequences through residue level alignment. PRX life, 2, 2024. URL https://doi.orge: 1103/
PRXLife 2. 043013
Lasse M. Blaabjerg, Nicolas Jonsson, Wouter Boomsma, Amelie Stein, and Kresten Lindorff-Larsen. Ssemb: A joint embedding of protein sequence and structure enables robust variant effect predictions. Nature Communications, 15, 2024. URL hhtt:::tt///oo 100..//0000000100//1/7700000--00
Nadav Brandes, Dan Ofer, Yam Peleg, and et al. Proteinbert: universal deep-learning model of protein sequence and function. Bioinformatics, 38:2102 2110, 2021.
Yehlin Cho, Sergey Ovchinnikov, and Christopher Frank. Enhancing protein design robustness through noise- informed sequence design. In ICML 2024 AI for Science Workshop, 2024. URL https://openreevier.
het//forumiid--ttTPheaaaa
Jacob Cohen. Statistical power analysis for the behavioral sciences. The SAGE Encyclopedia of Research Design, 1969. URT. httpp://app www.nticscholart..t....ee..c o wtt.://.........oooooopo//aaaa2.
14

---

Published in Transactions on Machine Learning Research (11/2025)
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In North American Chapler of the Association for Compulational Linguistics, 2019. URL htapss//app.ssenaniicccchl.all http:///wwaaaaaaaooooooooooooaaa
Ken A. Dill and Justin L. MacCallum. The protein-folding problem, 50 years on. Science, 338:1042 1046, 2012. URL http://app...emnntccccchoo.ooosa wtt.://www...iiooooooooooooooooa org/CorppsID:5556665
Ahmed Elnaggar, Michael Heinzinger, and al. Prottrans: Towards cracking the language of lifes code through self-supervised deep learning and high performance computing, IEEE Transactions Pattern Analysis and Machine Intelligence, pp. 1-1, 2021. doi: 041109//PAMM.2001.330000
Ahmed Elnaggar, Michael Heinzinger, and et al. Prottrans: Toward understanding the language of life through self-supervised learning. IEEE Transactions Pattern Analysis and Machine Intelligence, 44: 7112-7127, 2022.
Hehe Fan, Zhangyang Wang, Yi Yang, and Mohan Kankanhalli. Continuous-disereee convolution for seqmetry-sequence modeling in proteins. In The Eleventh International Conference on Learning Rep- resentations, 2023.
Maryam Gillani and Gianluca Pollastri. Protein subcellular localization prediction tools. Computational and Structural Biotechnology Journal, 23:1796 1807, 2024. ISSN 2001-0370. doi: httpe://doi.org//1..006///
2sb..2024.04.033. CRL https://www. sciencedirect. com http://ww.....cccc.c.c....aaa//a
Vladimir Gligorijević, P. Douglas Renfrew, and et al. Structure-based protein function prediction using graph convolutional networks. Nature Communications, 12, 2021. URL httpss//app..smmmaaiicccco....a
org/Coon Botg/CorpusID::00 235218243
Midori A. Harris, Jennifer I. Clark, Amelia Treland, Jane Lomax, Michael Ashburner, and et al. Gene ontology consortium: The gene ontology (go) database and informatics resource. Nucleic acids research, 32 Database issuc:D258-6i0 61, 2004. URL https://api www.nticschollatte......e.c.eaae hwt.://ar...arrrrorrr0.a0222a5..
Pedro Hermosilla, Marco Schäfer, Matej Lang, and et al. Intrinsic-cxtrinsic convolution and pooling for learning on 3d protein structures. International Conference on Learning Representations, 2021.
EunJeong Hwang, Veronika Thost, Shib Sankar Dasgupta, and Tengfei Ma. An analysis of virtual nodes in graph neural networks for link prediction (extended abstract). In The First Learning on Graphs Conference, 2022. URL http::///////o wttpa//wwwww......eo.oooooooaaaa
Jay J. Jiang and David W. Conrath. Semantic similarity based on corpus statistics and lexical taxonomy. wtt.://aaatttcccooo...ooooo.aa.p
Tn ROCLING//JCI.CI.P 1997. URL
Bowen Jing, Stephan Eismann, Pratham N. Soni, and al. Learning from protein structure with geometric vector perceptrons. In International Conference on Learning Representations, 2021. URL https://
John M. Jumper, Richard Evans, Alexander Pritzel, Tim Green, and et al. Highly accurate protein structure prediction with alphafold. Nature, 596:583 589, 2021. URL pttps::/apt..smmaiicsss org/
2
Panagiotis L. Kastritis and Alexandre M.J.J. Bonvin. On the binding affinity of macromolecular interactions: daring to ask why proteins interact. Journal of the Royal Society Interface, 10, 2013. URL https:
s/aptsemmattccccoolar htt.://ccccccooa.ooooooorrrppooo org/CooppsID:146555 146598814
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. Tn International Conference on Learning Representations, 2017. URL sttp::///pparrrrie....tffffooo SJU4ayYg1
Eunji Ko, Seul Lee, Minscon Kim, Dongki Kim, and Sung Ju Hwang. Protein representation learning by capturing protein sequeenc-strructur---eeee relationship. In ICLR 2024 Workshop on Machine Learning for Genomics Faplorations, 2024. URL http:://ppaaaeeeeer.aaa/r//rr/aa
15

---

Published in Transactions on Machine Learning Research (11/2025)
Juha Kurkela, Julia Fredman, Tina A Salminen, and Thina Tyystjärvi. Revealing secrets of the enigmatic omega subunit of bacterial rma polymerase. Molecular Microbiology, 115(1):1-11, 2021. doi: https: /do..
arg/10.111//mml 14603. URL https://bonlnnellrrray
htw.leyy.co//////a///////// 1ttp///0..111/mm..... 146003
Mingehen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Pan Tan, and Liang Hong. ProSST: Protein language modeling with quantized structure and disentangled attention. In The Thirty-eightt Annual Conference on Neural Information Processing Systems, 2024. URL
httts://openrevier. htwprriewweeefforreeee weet//forum11d=272222223
Yu Li, Sheng Wang, Ramzan Umarov, and et al. Deepre: sequence-based enzyme ec number prediction by deep learning. Bioinformatics, 2017. URL htap:s///.p...mmmnicccccoooooo Dtt.aa/iii.o.oooooooooooooooooaa S3/// 3790226.
Zeming Lin, Halil Akin, Roshan Rao, and et al. Language models of protein sequences at the scale of
evolution enable accurate structure prediction. bioRriv, 2022. doi: 10.1101/2022,0 /2002220.0.02.00000000 URL.
https://www biwx:/xxwwwbbioxvvvvvo http:///ww....ooooooooo/ooo/tott wttp://///.////////////////////0
Zeming Lin, Halil Akin, Roshan Rao, and ct al. Evolutionary-scall prediction of atomic-level protein structure with a language model. Science, (79 1130, 2023. doi: 10.1126/sciennce.dde27777 URL https:
//wwwsscience www.ssc....eee.ooi.i......o. brrg/doi/bbs//a
Jiajia Liu, Mengyuan Yang, Yankai Yu, Haixia Xu, Kang Li, and Xiaobo Zhou. Large language models in bioinformatics: applications and perspectives. ArXiv, 2024. URL https://api. semaniicccamt org/
CorpusIDD266899778
Meng Liu, Youzhi Luo, Kanji Uchino, Koji Maruhashi, and Shuiwang Ji. Generating 3d molecules for target protein binding, In International Conference on Machine Learning, 2022.
Shengchao Liu, weitao Du, Yanjing Li, and et al. Symmetry-informed geometric representation for molecules, proteins, and crystalline matorials. In Thirtg-seventh Conference on Newral Information Processing Sys- lems Dalasels and Benchmarks Track, 2023a. URL hnttps://openreriia http:///ppppuueeieeeeeeeeeraaddr
Zemin Liu, Xingtong Yu, Yuan Fang, and Xinming Zhang. Graphprompt: Unifying pre-training and down- stream tasks for graph neural networks. In Proceedings of the ACM Web Conference 2023, WWW '23, pp. 417 428, New York, NY, USA, 2023b. Association for Computing Machinery. ISBN 9781450394161. doi: 011145/5543307..36
URL
3583386
Shitong Luo. A 3d generative model for structure-based drug design. In Neural Information Processing Systems, 2022. URL
2449066772
Renjith Mathew and Dipankar Chatterji. The evolving story of the omega subunit of bacterial rna poly- merase. Trends in microbiology, 14 10:450 5, 2006. URL https:///pppii. semanticscholar org/CorpusII: 27701631
Fiomn Murtagh. Multilayer perceptrons for classification and regression. Neurocomputing, 2:183-197, 1991. URL https://apl. Sat.mattt.cccoola..oooo///ollllp 799497799
Catia Pesquita, Daniel Faria, Hugo Bastos, André Falcao, and Francisco Couto. Evaluating based semantic
similarity measures. In Proc. 10th Annual Bio-Ontologies Meeting, volume 37, pp. 38, 2007.
Leif E. Peterson. K-ncarest neighbor. Scholarpedia, 4:1883, 2009. URL stttp:://pi..smantiiccc ptg//oorpusIDD:2611122
Ruijie Quan, Wenguan Wang, Fan Ma, Hehe Fan, and Yi Yang. Clustering for protein representation learning, 2024. URL httpps://pperrevier htt.:/ooruummmdd--/gaprtttt
Roshan M Rao, Jason Liu, Robert Verkuil, and et al. Msa transformer. In Marina Meila and Tong Zhang (eds.). Proceedings of the 38th International Conference on Machine Learning, 2021. URL https:// proceedings.co mlr. www.mmlr..p.... #rccaddigg.nt-pprree///////aaaa wttp:ees///////rlla.aaaattt
16

---

Published in Transactions on Machine Learning Research (11/2025)
Alexander Rives, Siddharth Goyal, Joshua Meier, and et al. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences of the United States of America, 118, 2019.
Jae Yong Ryu, Hyun Uk Kim, and Sang Yup Lee. Deep learning enables high-quality and high-throughput prediction of enzyme commission numbers. Proceedings of the National Academy of Sciences, 116:13996 14001, 2019. URL
otg/CorpusID: ottpaaaai.iiiiiaaooooooooooaaaaa 1995rpp221::191122222 195192023
Hira Saleem, Usman Ali Ashfaq, Habibullah Nadeem, Muhammad Zubair, Muhammad Hussmain Siddique, and Ijaz Rasul. Subtractive genomics and molccular docking approach identify drug targets against stenotrophomonas maltophilia. PLOS ONE, 16(12):1-17, 12 2021. doi: 10.1371 /oo177//oorraal.pon.0005
URL httph:: :/ttpa wttp.oog///....//.. 13rr//ouaaa... pone pone. 0261111
Ömer Sinan Saraç, Volkan Atalay, and Rengul Cotin-Atalay. Gopred: Go molccular function prediction by combined classifiers. PLoS ONE, 5, 2010. URL https://api caammaaticsccolaa.ooo//ooooooo :ttp://///orpu.:1110070.... 1258725
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neurol Networks, 20:61-80 1061180, 2009. URL https://api.
semanticscholar. www.s/oooo..r.ro...ooooooo :ngg/CorpusI::20G6 206756462
M. Schlichtkrull, Thomas Kipf, Peter Bloem, Rianne van den Berg, Ivan Tilov, and Max Welling. Modeling relational data with graph convolutional networks. In Extended Semantic Web Conference, 2017. URL
attts://app. www.nticscoollatteeeoeoeeeieeeee wwt.:/orro////eee.e
5458500 5445500
Andrew W. Senior, Richard Evans, John M. Jumper, and al. Improved protein structure prediction using potentials from deep learning. Nature, 577:706- 710, 2020.
S. S. Shapiro and M. B. Wilk, An analysis of variance test for normality (complete samples). Biometrika, 52 (3-4):591-611, dee 1965. doi: 10.1093/biomett/ /52.3 /ttp://0oomtt2.334555500 3-4.591
URL https://ddl..rrg///
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-scale representation learning on proteins. In Neural Information Processing Systems, 2022. URL https://ppi..emnnttcccchoorrr org/
1o333S1D:236317311
Bosheng Song, Xiaoyan Luo, Xiaoli Luo, Yuansheng Liu, Zhangming Niu, and Xiangxiang Zeng. Learning spatial structures proteins improves protein-protein interaction prediction. Briefings in Bioinformatics, 23 2022. ISSN 1477-4054. doi: 10.1093/bbl bib/bbab558
URL lattps:///ot. hatts:://ad..or//1O11000 org//0.1093/
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. Saprot: Protein lan- guage modeling with structure-aware vocabulary. In The Twelfth International Conference on Learning Representations, 2024. URL https://opennreview net/ wttp://wwww.eetotuumuttaa http//orummii-6//a////AA..
Baris E. Suzek, Hongzhan Huang, Peter B. McGarvey, Raja Mazunder, and Cathy H. Wu. Uniref: compre- hensive and non-redundant uniprot reference clusters. Bioinformatics, 23 10:1282-8. 8, 2007.
Aaron van den Oord, Oriol Vinyals, and Koray Kavukeuoglu. Neural discrete representation learning. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, pp. 6309-6318, Red Hook, NY, USA, 2017. Curran Associates Inc. ISBN 9781510860961.
Aäron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. ArXiv, abs/1807.03744 2018. URL https://api www.:/eewaaa.ccco...or.ooo/////e htt.://cccoaaaoooor///ooppaaaooa
Karel van der Weg, Erinc Merdivan, Maric Piraud, and Holger Gohlke. Topec: prediction of enzyme com- mission classes by 3d graph neural networks and localized 3d protein descriptor. Nature Communications,
16, 2025. URL https://app. semanticscholar. wtt.://rr...rrooouuu///22711111.
17

---

Published in Transactions on Machine Learning Research (11/2025)
Michel van Kempen, Stephanie Kim, Charlotte Tumescheit, and et al. Fast and accurate protein structure search with foldseek. Nature Biotecknology, 42:243 246, 2022. URL
Drg/CorpusID:25707 257837735
Mihály Váradi, Stephen Anyango, Mandar S. Deshpande, and et al. Alphafold protein structure database: massively expanding the structural coverage of protein-sequence space with high-accuracy models. Nucleic Acids Research, 50:D439 D444, 2021. URT. httph::/ppp..ppp.. wtt..ttpee///pp...eemmtii http://ppp..emmnticcchhho..... http:iissccchaa..oooorrrreeeeeee 2457701129
Mihály Váradi, Damian Bertoni, Paulyna Magaña, and et al. Alphafold protein structure database in 2024:
providing structure coverage for over 214 million protein sequences. Mucleie Acids Research, 52:D368
D375, 2023. URL httpp://appp www.sssmmtnnichhlaaaaem.em wwwp:lar..rr//roous...::::. 265042036
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Neural Information Processing Systems, 2017.
URL atctas//aap..eemaniicccccoaaa
semantccccoolaar :rtpaaarrrr/rooor:::::300 137566899
Duolin Wang, Mahdi Pourmirzaei, Usman L. Abbas, Shuai Zeng, Negin Manshour, Farzaneh Esmaili, Bi- plab Poudel, Yuexu Jiang, Qing Shao, Jin Chen, and Dong Xu. S-plm: Structure-aware protein lan- guage model via contrastive Icarning between sequence and structure. Advanced Science, 12(6):2404212, 2025. doi: https:/ btops::/ddi.orgg.. 1twpg///...0000//11111 /tw.000///v////// 2024442... URL lnttps///advanced www.ammerrr......nnnnrnaarraaaar wiley. com/àoi/abs 1002/ advs. hd 202404212
https///bb..rrg//1.000///
Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning hierarchical protein representa- tions via complete 3d graph networks. In International Conference Leurning Representations, 2022a.
URL attpppoo/ap...smmm.ccccccolaaaaa wrrg/orrpsssl... http:/CopuSD::5536444130 257364758
Zichen Wang, Steven A. Combs, Ryan Brand, and et al. Lm-gvp: an extensible sequence and structure informed deep learning framework for protein property prediction. Scientific Reports, 12, 2022b. URL
248415326
Edwin C. Webb. Enzyme nomenclature 1992. recommendations of the nomenclature committee of the inter- national union of biochemistry and molecular biology on the nomenclature and classification of enzymes. 1992. URL CtLpppppce/pppg..eeaartttccctcre wtt.p//...........oooooooooaaaa.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xie, Bonnie Berger, Jianzhu Ma, and Jian Peng. High-resolution de novo structure prediction from primary sequence. bioRriv, 2022. doi: 10.1101/2022.0.222 20090999 URL hwtp::://ww..boxiii /rg//ontent/ early/2022/07/22222222 07. 21. 500999
Yijia Xiao, Jiczhong Qiu, Ziang Li, Chang- Yu Hsich, and Jie Tang. Modeling protcin using largo-scale pretrain language model. ArXiu, abs/2108.07433, 2021.
Keyulu Xu, Weihua Hu, Jure Teskovec, and Stefanie Jegelka. How powerful are graph neural networks? Tn International Conference on Learning Representations, 2019. URL http:://oppnrreviee.nnefffrooooo
ryGa6iA5Km
Zhiqiang Yan, Liyong Guo, Liang Hu, and Jin Wang. Specificity and affinity quantification of protein-protein interactions. Bioinformaties, 29 9:1 22112733, 2013. URL
org/CorpusID: o
1339889
Kevin K Yang, Niccolò Zanichelli, and Hugh Yeh. Masked inverse folding with sequence transfer for protein representation ion learning. Protein Engineering, Design and Selection, 36:gzad015, 10 2022. ISSN 1741-0126.
doi: www.1093/protei/geee /gad0115 URL https://doi.org: http:://oo...rrr.00001199/eeeeer 1ww.///rroeen/gzaa0000 gzad0155
Yuclin Zhang, Jiacheng Cen, Jiaqi Han, and Wenbing Huang. Fast and distributed equivariant graph neural networks by virtual node learning. ArXiv, abs/2506.19982, 2025. URL https://arxxi: wttprvv..rr//bb//56666
19482
18

---

Published in Transactions on Machine Learning Research (11/2025)
Zuobai Zhang, Chuanrui Wang, Minghao Xu, and et al. A systematic study of joint representation learn- ing on protein sequences and structures. 2023a. URL. https://api http://pp...emmatiicccccooollll wtt. | 257496831
Zuobai Zhang, Minghao Xu, Arian Rokkum Jamasb, and et al. Protein representation learning by geometric structure pretraining, In The Eleventh International Conference on Learning Representations, 2023b. URL
Zuobai Zhang, Minghao Xu, Aurélie Lozano, and et al. Pro-training protein encoder via siamese sequence- structure diffusion trajectory prediction. In Advances in Newral Information Processing Systems, 2023c.
Chenyi Zi, Haihong Zhao, Xiangguo Sun, Yiqing Lin, Hong Cheng, and Jia Li. Prog: A graph prompt learning benchmark. In The Thirty-eight Conference on Neural Information Processing Systems Datasels and Benchmarks Track, 2024. URL https: htap:://ppenrae http///v/pereeeeeee///eeee net/forummid=wwooxzze
A Appendix
A.1 Empirical Evaluation
In this section, we outline our empirical sctting. We performed analysis on several downstream tasks, demonstrating the value of the representation we provide in real-world bioinformatics tasks.
A.1.1 Protein Structure Dataset
To reduce reliance the availability and quality of external structural data, leverage AlphaFold2 (Jumper ct al.] 2021), well-known SOTA model, which enables us to generate 3D structures and eliminate the need for external sources. Furthermore, many recent high-performing methods, such as SaProt (Su et al. 2024) and ESM-GeerNel. (Zhang et al. 2023a also used it for structure generation, ensuring consistency in comparisons. Therefore, we use the AlphaFold protein structure database (Váradi et al.) 2023 2021 for pre-training, which contains 805K protcin structures prodicted by AlphaFold2 (Jumper et al., 2021 Alternatively, other models, such as ESMFold (Lin al. 2022), can be applied. Since AlphaFold predictions may include local inaccuracies, we assess Fusion Prot's robustness to structure noise in Section 5.4
A.1.2 Tasks
We evaluated our proposed method on several common downstream tasks, selected based on SOTA works of GearNet (Zhang al. 2023a] and SaProt (Su et al.] (024)) to assoss its effectiveness for protein representation learning. These tasks include protein annotations such as Enzyme Commission (EC) prediction, Gene Ontology (GO) prediction, and Mutation Stability Prediction. The GO prediction task includes three sub- tasks: predicting protein's biological processes (BP), molecular functions (MF), and cellular components (CC).
To ensure the validity of our results, all downstream task splits are constructed so that sequences in one set share no more than 30% Needleman Wunsch sequence identity with any sequence in the other sets. addition, the hold-out sets for each task share no more than 30% identity with any protein in the AlphaFold2 pre-training corpus (Jumper et al. 2021).
Enzyme Commission Number Prediction Annotation of enzyme function has a wide range of real- world applications, including melagenomics, diagnosis of enzyme-deefccieecy-rlaeee diseases [Li et al. 2017). and cellular metabolism (Ryu et al.) 2019). This task focuses on determining enzyme function by predicting EC numbers, which characterize a protein's catalytic activity in biochemical reactions. It involves 538 binary classification problems derived from the third and fourth levels of the EC classification tree Webb 1992).
We used dataset splits from DeepFRT (Gligorijevié et al. 2021 and the evaluation metric is the Fmax score.
19

---

Published in Transactions on Machine Learning Research (11/2025)
Gene Outology Term Prediction The GO knowledgebase (Harris et al. 2004 provides set of struc- tured and controlled terms describing gene products and their molecular properties. Many real-world bi- ological applications, such as predictions of protein-protein interactions, rely on GO term-protein annota- tions (Pesquita et al., 2007 Jiang & Conrath 1997). This benchmark includes three tasks: predicting a protein's biological processes, molecular funct ions, and cellular components. Each task involves multiple binary classification problems based on GO term annotations. We used dataset splits from DeepFRI (Glig- orijević al., 2021; and the evaluation metric is the Fax score.
Mutation Stability Prediction Mutation stability prediction (MSP) is crucial in bioinformatics for understanding how genetic mutations affect protein stability, which plays key role in disease mechanisms and drug development. This task aims to predict whether a mutation enhances a protein complex's stability. We utilize the datasots and hyperparameters from ESM-GearNott (Zhang al. 2023a). Evaluation is basod AUROC.
A.1.3 Baselines
We compare with numerous baseline models, including PLMs and structure models. We used ProtBERT-
BFD (Elnaggar et al.] 2021), DeepFRI (Gligorijević et al. 2021). ESM-1b (Rives et al., 2019 and ESM-
Lin et all 2023) as SOTA PLM sequential models. GearNet (Zhang al. 2023b)
GVP (Jing et al., 2023) are used SOTA structure models. Furt we include MIF-
)0 and CDConv Fan et al.,
ST Yang et all 2022 S-PLM [Wang et al. 2025) ESM-GearNet (Zhang et al. 2023ab, SaProt (Su et al, 2024) and GearNet (Su et al. 2024) as SOTA in joint learning of structure models with sequential models. For consistency, when using ESM-2 we used the ESM-2-65000 variant (Lin et al., 2023). In addition, GearNet (Zhang et al. (023b), ESM-GearNeet (Zhang et al. 2023a). ) and FusionProt are pre-trained with the same objective, which is the Multiview Contrast (Zhang et al. 2023b objective, as prior work showed its superior performance.
A.1.4 Implementation Details
Pre- Pre-Training Phase We follow ESM-GearNet (Zhang et al., 2023a in adopting the same training ob- jective and model selection strategy. Our model uses a pre-trained ESM-2-650M (Lin et al. 2023) as the base PLM, with 33 layers (L, = =3),, and GearNel (Zhang et al. 2023b) with 6 layers (L2 = 6) and 512 hidden dimensions as the structure encoder. The embedding dimension is set to 1280. Multiview Con- trast [Zhang et al. 2023b) used as the pre-training objective. Hyperparameters were tuned using the same search procedure and ranges reported in ESM-2 (Lin et al., 2023 and ESM-GearNet (Zhang et al. 2023a with the best, configuration selected based on validation performance. For FusionProt, this configuration corresponded to training for 50 epochs with a learning rate of 2e-4 and global batch size of 256 proteins. To accommodate long sequences, inputs are truncated to a maximum of 1,024 tokens. All implementations use the TorchDrug library.
Fine-Tuning Phase During inference, we incorporate task-specific classification heads to generate predic- tions for each downstream task. Following the protocol of the recent SOTA model SaProt (Su et al. 2024 we evaluated our model and baselines under a consistent hyperparameter tuning procedure to ensure fair comparison. For all methods, we performed tuning within the hyperparameter ranges reported in SaProt (Su iet al. 2024h, with the best configuration for each model selected based on validation performance. We use AdamW with 31 = 0.9 and 3g- 0.98, treating learning rate, weight decay, and batch size as tunable; default initial values and exact ranges are provided in our Gut repository. All models were trained to convergence, and the final checkpoint was chosen by the highest validation score (task-specific primary metric).
A.1.5 Computational Complexity
FusionProt was trained on 4x NVIDIA A100 80 GB GPUs for 48 hours (192 GPU-hours). It retains the underlying architecture of the base sequential and structure encoders (e.g., ESM-2 and GearNet), applying each layer once in prior baselines (e.g., FSM--Ges -GaeNee...).
20

---

Published in Transactions on Machine Learning Research (11/2025)
Let L denote the protein sequence length (in our settings, we limit the imput to L 1,024). FusionProt adds one token on the sequential encoder and one universally connected node on the structure encoder. On the sequential side, per layer the self-attention size changes from L to (L+1)² a relative increase of (++11==
2 = L + _=; at I, = 1,024 this is Ri 0.195% per layer. Because the baseline and FusionProt use the same depth, this fractional increase does not amplify with depth; in practice, feed-forward blocks dominate compute, so the impact is small.
On the structure side, the fusion node contributes L additional edges of new type. If the original graph has on average M edges per residue, the bascline has z ML edges; adding L fusion edges increases message passing by about 1/M. With an approximate bound of 1=100 edges per residue in our settings, this is a ~ 1% overhead per structure layer. Since GearNet has 6 layers and message passing is a minority of end-to-end cost, the overall runtime and memory impact remain modest.
Empirically, we observe a consistent 2-5% increase in inference latency and throughput across sequence lengths and model scales; for example, latency rises from 12.0 ms to 12.6 ms per 1,000 residues at L=1,024 (~5%). We also observe a small meao-m increase (= 2.6%) from the extra token activations and fusion edges, and about 3% increase in training step time with the same number of epochs to convergence, Together, the scaling analysis and these measurements support that FusionProt introduces a small, well- bounded overhead.
A.2 Biological Insights
When such performance gains driven by biologically meaningful imputs, such as detailed 3D structural features, they can indicate that the model is capturing relationships of functional relevance. However, we note that one must avoid conflating improved prediction with causation: model's ability to predict biological outcomes with more precision does not in itself support causal inferences. Nevertheless, significant increases in specific classes of predictions can provide valuable hints that frame hypotheses, guide experimental questions, and motivate follow-up studics.
FusionProt consistently outperforms SOTA baselines across all downstream tasks (Table 1 Table 2
For FusionProt, the largest improvements are often observed in proteins whose function, stability, or in- teractions depend strongly complex structural organization or long-range spatial effects. In contrast to methods such as ESM-GearNet (Zhang et al. 2023a) SaProt (Su et al. 2024), which integrate sequence and structure in single pass, FusionProt employs an iterative fusion token mechanism that enables multi- ple rounds of bidirectional information exchange between sequence and structure encoders. This repeated refinement allows the model to amplify subtle, spatially localized features that might otherwise be diluted in one-shot fusion.
To better understand the nature of these improved learned protein representations, we evaluated model's ability to predict EC numbers, as outlined in Section A.1.2 using pre-trained representations without any fine-tuning. This approach enables an assessment of the inherent quality of the learned embeddings, inde- pendent of task-specific training. We then compared FusionProt's predictions with those of ESM- Net focusing on the cases with the largest discropancies.
We then organized these proteins into distinct biological groups based on shared mechanistic characteristics. We focus on two common mechanisms (Table 6) and provide representative example for each group.
Structural subunits for macromolecular assembly. The bacterial RNA polymerase w subunit (Fig- ure is small and poorly conserved, yet plays key structural role in holoenzyme assembly by recruiting and stabilizing the 8 subunit (Mathew & Chatterji 2006). Its short length and weak sequence signal make it difficult for sequence-only models to classify correctly. FusionProt's iterative sequenco-structure fusion appears to preserve features on the 3' contact surface that are tied to assembly and stability, while one-pass fusion may reduce such localized signals. Consistent with this interpretation, FusionProt assigned the cor- rect complex-level EC labels with high confidence (mean of 0.90), while the bascline ESM-GcarNet (Zhang) et al.) (2023a) did not (mean of 0.21). Because w noncatalytic and the EC number reflects holoenzyme activity, we hypothesize that FusionProt. recognizes the context of the quaternary structure, specifically an
21

---

Published in Transactions on Machine Learning Research (11/2025)
Table 6: Mechanisms highlighted by our case studies, with generalizable structural signals, representative example, and EC-probe confidence.
Biological mechanism
Insight / structural signal
Example protein
Confidence improvement
Assembly interface
Interface fingerprint subunit contacts RNAP B3' interface
0.90 0.21
halngp//eeccrrelllaoooyyyaaoppp gains when function depends qua- ternary context rather than active-site chemistry.
Loop-gated pocket
Dynamic loops that gate ligand/catalytic sites; loop residues occupy broader allowed (ø, v) basins (flexibility) supporting access and specificity.
D-Ala-D- D-All ligase (ATP-grasp)
0.88 vs 0.62
a2
B
al
Maanddnoon SURODRR
B
Figure 4: DN A-direcce RNA polymerase highlighting the w 8' assembly interface (w in magenta, B in light green) (urreela et al. 2021 Although w is small and poorly conserved, it is essential for recruiting and stabilizing 3' during holoenzyme assembly, FusionProt captured this interface-specifie structural fingerprint and classified the correct complex EC labels with high confidence, while ESM-GcarNet failed.
interface-centric fingerprint of local shape, charge, and hydrophobic patterning the w-3' surface, rather than active-site chemistry. Figure highlights this interface (magenta: w; light green: 8'), where the model appcars to localize signal more strongly than the bascline. This implics that weakening these interface features would reduce assembly propensity and lower model confidence, providing concrete avenue for validation.
highhittts
Conserved ATP-dependent peptide ligases. D-alanine-Daalamine ligase (Figure is a well- characterized member of the ATPgrasp enzyme superfamily, which catalyzes the formation of the D-Ala- D-Ala dipeptide, an essential step in bacterial peptidoglycan biosynthesis. Its conserved 3D fold encloses ATP-binding site and supports relatively simple, well-understood catalytic chemistry. Both FusionProt and NeM-GearNet correctly predicted its EC classes, but FusionProt did with higher confidence (mean of 0.88 vs. 0.62). We hypothesize that the iterative fusion mechanism amplifics local structural cues from the mobile loops surrounding the ligand-binding pocket, which mediate substrate recognition and catalysis, beyond what the global fold alone conveys. Consistent with this, the Ramachandran plot (Saleem et al. 2021) for the same structure (Figure 5 shows broader distribution of the dihodral angles (o and & angles) of the backbone for the loop residues, within allowed regions: hallmarks of conformational flexibility such pocket gating and substrate specifieity. This suggests that repeated sequence structure refinement
22

---

Published in Transactions on Machine Learning Research (11/2025)
-13P
31P
Figure 5: D-alanine-D-alaniee ligase. Left: 3D structure showing the ATP-binding cleft and the surrounding pocket-adjacent loops. Right: Ramachandran plot (Saleem et al. 2021] of the dihedral angles (ó and & angles) of backbone for the same protein; shaded contours denote sterically allowed conformations. The broader spread within allowed basins is characteristic of flexible loop regions that gate the pocket, supporting our interpretation that FusionProt captures this loop-centric fingerprint and yields higher EC prediction confidence than ESM-GearNet.
sharpen flexible, function-critical motifs even in enzymes with highly conserved catalytic cores, implying that perturbations to these loops could influence both enzymatic activity and model confidence.
Across categories, FusionProt's unified embeddings capture structural determinants often missed in less iterative fusion frameworks, such allosteric loops, oligomerization interfaces, and convergent active-site geometries. These enriched representations may facilitate advances in function prediction, mutagenesis design, drug discovery, and the annotation of orphan proteins from metagenomic datasets.
A.3 Interpretability Analyses
We examine how the fusion token aggregates structural during ECC prediction. FusionProt augments the sequence encoder with single fusion token and periodically injects structure-derived information into the sequence stack (every few Transformer layers). To visualize what the token allends to, at each layer we record the attention distribution from the fusion token to all sequence positions (averaged over heads). For plotting, we remove padding and special tokens, and we normalize cach layer's fusion-to-residue row over residues so patterns remain visible for long sequences. We also track the fusion voken's self-attention (fusion to fusion) on its original softmax scale. Horizontal guidelines mark structure-injection depths (every layers).
Figure G shows three representative proteins of different lengths. In each panel, the heatmap (left) depicts fusion token to residue attention across depth, and the thin strip (right) shows the token's self-attention per layer. Across cases we observe: (i) attention progressively concentrates on a small subset of residues in later layers, indicating targeted querying rather than diffuse patterns; (ii) immediately after each structure injection, the attention map refocuses, consistent. with incorporation of new 3D evidence; and (iii) self- attention varies with depth and not persistently high, suggesting the token predominantly routes attention outward to residues. Together, these observations indicate that FusionProt. integrates structural cues into the sequence stream and contributes an informative signal rather than noise.
23

---

Published in Transactions on Machine Learning Research (11/2025)
(A) Short protein
(B) Medium protein
(C) Long protein
Figure 6: Attention patterns of the fusion token the EC prediction task. In each panel, the heatmap the left shows fusion-token residue attention across transformer layers (rows) and residue indices (columns). Padding and special tokens are removed; each row is normalized over residues to aid visibility. The thin strip on the right shows fusion-token self-attention per layer on the original softmax scale. Horizontal guidelines mark the structure-injection depths. Across short (A), medium (B), and long (C) proteins, attention sharpens in deeper layers and refocuses after injections, while self-attention remains non-saturated, indicating that the fusion token queries specific residue subsets rather than self-attending.
24


Reviews:
[{"review_id": "gpXMjB8E7w", "reviewer": "Reviewer_p7um", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents FusionProt, a simple yet general framework for integrating protein sequence and structure information through a single learnable “fusion token” that shuttles between a sequence language model (ESM-2-650M) and a structure encoder (default: GearNet). The authors evaluate FusionProt on enzyme commission (EC) prediction, Gene Ontology (GO: BP/MF/CC), and mutation stability prediction (MSP). Across these tasks, FusionProt outperforms strong sequence-only, structure-only, and prior joint models, with reported statistical significance. \n\n## Strengths:\n- **Clear Model Desgin**: A single fusion token provides a lightweight, model-agnostic mechanism to couple existing sequence and structure encoders.\n- **Comprehensive Experiments**: Improvements are demonstrated across multiple benchmarks (EC, GO, MSP) with replicated runs and significance testing. Meanwhile, the ablation study is also extensive. \n\n## Weakness:\n- **Insufficient literature review: ** Several recent multimodal protein models are not cited / discussed.\n- **Novelty issue of \"fusion token\": ** The idea of the \"fusion token\" is not new. Some related studies should be discussed.", "claims_and_evidence_yn": "Yes", "claims_explanation": "This paper provides multi-benchmark gains with repeated runs and significance tests, which aligns with TMLR’s primary criterion that claims be backed by accurate, convincing, and clear evidence.", "audience_yn": "Yes", "audience_explanation": "Protein ML is a fast-moving, high-impact application area—energized by AlphaFold/AlphaFold 3 and the rapid rise of protein language models—so a method that improves sequence–structure fusion is likely to interest part of the TMLR audience.", "requested_changes": "Add discussion and, where doable, numbers vs.  MIF / MIF-ST (seq+struct pretraining/fusion), and SSEmb / Residue-Level Alignment (joint seq-struct embedding). If benchmarking isn’t feasible, include an explicit discussion in related works.\n\nRegarding the novelty of the fusion token, add deeper discussion on recent graph virtual node methods and graph prompt learning approaches.\n\nAblate radius/k-NN/edge types and summarize trends vs. protein length/domain count (curves + recommended ranges).\n\n\n\n\n\n\n- [MIF / MIF-ST] Masked inverse folding with sequence transfer for protein representation learning\n- [SSEmb] SSEmb: A joint embedding of protein sequence and structure enables robust variant effect predictions", "broader_impact_concerns": "N/A", "additional_comments": "N/A", "extra_scores": {}, "extra_text": {}}, {"review_id": "U2PxXZMiip", "reviewer": "Reviewer_stWx", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Summary of Contributions\n\nThis paper presents FusionProt, a unified model that learns representations via iterative, bidirectional fusion between a protein language model and a structure encoder.\n\nStrengths:\n1. The paper introduces a novel fusion mechanism that leverages a learnable fusion token, which iteratively and bidirectionally integrates 1D sequence and 3D structural information.\n2. Experiments are broad, covering EC, GO, and mutation stability tasks, with performance exceeding baselines. Ablation studies further analyze fusion frequency and the impact of different structure encoders.\n3. The structure of the paper is clear, and the methodology is well-explained.\n\nWeaknesses:\n1. The design choices of the fusion token, particularly its universal connection to all nodes, are not deeply discussed and may raise concerns of redundancy.\n2. The hierarchical interaction with the fusion token may incur additional computational and memory costs. Although the authors claim inference overhead is only 2–5%, detailed data is missing.\n3. While improvements over baselines are reported, the performance gains appear modest in some cases, and statistical significance of these comparisons is not consistently provided.\n4. Since the structural data may come from predicted structures (e.g., AlphaFold2), the model should be tested more rigorously under low-confidence or erroneous structural inputs to validate robustness.", "claims_and_evidence_yn": "Yes", "claims_explanation": "Most major claims, such as performance improvements over baselines and the effectiveness of the proposed fusion mechanism, are supported by experimental evidence, including ablations and comparisons with alternative structure encoders. However, some claims extend beyond the evidence: assertions of capturing “biologically relevant features,” “strong generalization,” and “minimal overhead” are not fully quantified. Robustness to extreme cases (e.g., long proteins, low-quality structures, sparse labels) and scalability analysis are insufficiently demonstrated.", "audience_yn": "Yes", "audience_explanation": "The integration of sequence and structural information is an active topic in protein machine learning, and the unified design with iterative cross-modal fusion is of clear interest to researchers in representation learning and bioinformatics. Even beyond protein-specific tasks, the token-based fusion strategy offers insights applicable to other multimodal learning settings.", "requested_changes": "1. Provide a more detailed analysis of computational and resource costs, including inference time, memory use, and training efficiency, across different protein lengths and model scales.\n2. Evaluate robustness under low-quality or uncertain structural inputs (e.g., low-confidence AlphaFold2 predictions), possibly through controlled ablation or threshold-based experiments.\n3. Offer deeper comparative analysis with alternative design choices, such as static fusion models or unidirectional fusion, to clarify the specific advantages of the proposed mechanism.\n4. Please consider extend evaluation to more challenging downstream tasks, such as extremely long proteins or low-quality structural datasets.\n5. Add interpretability analyses, for example visualizing attention weights of the fusion token across layers, to provide more transparency.", "broader_impact_concerns": "No.", "additional_comments": "No.", "extra_scores": {}, "extra_text": {}}, {"review_id": "XIPCQfXMqn", "reviewer": "Reviewer_JhLE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "## Summary\n\nThis paper presents FusionProt, a unified model to learn a protein representation while incorporating both the protein’s amino acid sequence and its 3D structure. The sequence is processed using a protein language model (ESM-2 [1]) and the 3D structure is parsed into a structure graph that is processed using a graph neural network (GearNet [2]). Crucially, to share information between these two models, the authors use a “fusion token” that is shuttled between the models after every processing block (attention block in the language model and relational message passing block in the graph neural network). This is in contrast to previous approaches such as ESM-GearNet [3] that do not bidirectionally fuse information. The authors pretrain FusionProt using a multi-view contrastive loss and evaluate downstream fine-tuning performance on a set of established benchmarks. FusionProt consistently outperforms existing approaches by significant margins.\n\n## Strengths\n\n- The paper presents a relatively simple and intuitive method to fuse the two types of protein representations. The method is well-motivated and the authors clearly describe how it builds on existing work.\n- The authors demonstrate that FusionProt is very effective using many relevant benchmarks, often substantially improving on existing methods.\n- The paper is generally very well-written. I was able to understand most of the methods and results without referring to external resources since adequate background is provided by the authors wherever necessary. This also makes it accessible to a larger audience.\n\n## Weaknesses\n\n- The multi-view contrastive loss used for pretraining should be explained in a bit more detail. For example, it is unclear to me what it means to “select consecutive subsequences and apply random edge masking to hide 15% of edges in the protein graph”. Are these consecutive residues in the 1D amino acid sequence? And how are these related to the edges being masked?\n- The benchmarks chosen by the authors are based on those used by GearNet [2] and SaProt [4]. However, the authors do not evaluate performance on tasks from popular sequence-based benchmarks such as ProteinGym [5] that are based on deep mutational scanning (DMS) data. These benchmarks are often used to evaluate protein language models and it would be useful to understand if using protein structures helps for these tasks.\n\nReferences:\n1. Lin, Zeming, et al. \"Evolutionary-scale prediction of atomic-level protein structure with a language model.\" Science 379.6637 (2023): 1123-1130.\n2. Zhang, Zuobai, et al. \"Protein Representation Learning by Geometric Structure Pretraining.\" The Eleventh International Conference on Learning Representations.\n3. Zhang, Zuobai, et al. \"A systematic study of joint representation learning on protein sequences and structures.\" arXiv preprint arXiv:2303.06275 (2023).\n4. Su, Jin, et al. \"SaProt: Protein Language Modeling with Structure-aware Vocabulary.\" The Twelfth International Conference on Learning Representations.\n5. Notin, Pascal, et al. \"Proteingym: Large-scale benchmarks for protein fitness prediction and design.\" Advances in Neural Information Processing Systems 36 (2023): 64331-64379.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In my opinion, all claims made by the authors are adequately supported. The authors clearly explain how FusionProt builds on existing work and thoroughly benchmark FusionProt against existing methods to show convincing performance improvements.", "audience_yn": "Yes", "audience_explanation": "Protein representation learning is an important application of machine learning in computational biology. Since this paper presents a robust method to integrate protein sequence and structural information, it would be of interest to many researchers who work on similar problems. It could also be of interest to those who work on model-guided protein design.", "requested_changes": "In my opinion, this work already passes the acceptance threshold. However, addressing the weaknesses mentioned above could further strengthen this submission. These are:\n- Providing a more detailed explanation of the pretraining objective\n- Evaluating FusionProt on DMS-based benchmarks such as ProteinGym.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)