Greater than the Sum of Its Parts:  Building Substructure into Protein Encoding Models
Authors: Robert Calef
Arthur Liang
Manolis Kellis
Marinka Zitnik
Venue: ICLR 2026 Poster (2026)
Keywords: Protein
biology
representation learning
benchmark
multiscale protein models
structure representation learning
substructures
motifs
domain
protein function
protein structure
protein representation learning
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: Prevailing protein models ignore the fact that proteins composed of recurrent, modular substructures that are functional and evolutionarily-conserved. We think that should change.

Abstract: Protein representation learning has achieved major advances using large sequence and structure datasets, yet current models primarily operate at the level of individual residues or entire proteins. This overlooks a critical aspect of protein biology: proteins are composed of recurrent, evolutionarily conserved substructures that mediate core molecular functions. Despite decades of curated biological knowledge, these substructures remain largely unexploited in modern protein models. We introduce Magneton, an integrated environment for developing substructure-aware protein models. Magneton provides (1) a large-scale dataset of 530,601 proteins annotated with over 1.7 million substructures spanning 13,075 types, (2) a training framework for incorporating substructures into existing models, and (3) a benchmark suite of 13 tasks probing residue-, substructure-, and protein-level representations. Using Magneton, we develop substructure-tuning, a supervised fine-tuning method that distills substructural knowledge into pretrained protein models. Across state-of-the-art sequence- and structure-based models, substructure-tuning improves function-related tasks while revealing that substructural signals are complementary to global structural information. 
The Magneton environment, datasets, and substructure-tuned models are all openly available.

Published as a conference paper at ICLR 2026
GREATER THAN THE SUM OF ITS PARTS: BUILDING SUBSTRUCTURE INTO PROTEIN ENCODING MODELS
Robert Calef1.2.,, Arthur Liang*.*, Manolis Kellis', Marinka Zitnik? MIT, Harvard University frcalef, artliang, manoli 1 /mmit marinka@hms harvard. edu
edu edu
ABSTRACT
Protein representation learning has advanced rapidly with the scale-up of se- quence and structure supervision, but most models still encode proteins either as per-residue token sequences or as single global embeddings. This overlooks defining property of protein organization: proteins built from recurrent, evolutionarily conserved substructures that concentrate biochemical activity and mediate core molecular functions. Although substructures such domains and functional sites are systematically cataloged, they are rarely used as training sig- nals representation units in protein models. We introduce Magneton, an envi- ronment for developing substructure-aware protein models. Magneton provides (1) dataset of 530,601 proteins annotated with over 1.7 million substructures spanning 13,075 types, (2) training framework for incorporating substructures into existing protein models, and (3) benchmark suite of 13 tasks probing representations at the residue, substructural, and protein levels. Using Magne- ton, we develop substructure-tuning tills substructural knowledge into pretrained protein models. Across state-of- the-art sequence- and structure-based models, substructure-tuning improves func- tion prediction, yields more consistent representations of substructure types never observed during tuning, and shows that substructural supervision provides in- formation that is complementary global structure inputs. The Magneton vironment, datasets, and substructure-tuned models are all openly available at http:///gitubb.oommrrrcc////ggmp
supervised fine-tuning method that dis-
INTRODUCTION
Protein representation learning has progressed from models trained on large sequence databases (Rives al. 2021 Elnaggar al., 2022) models incorporating experimenaally determined or pre- dicted structures (Glioorjević et al. 2021 Zhang et al.) (2022b), enabling advances in folding (Lin cet al. 2023). function prediction (Rao et al. 2019), and variant effect prediction (Meier et al. 2021 Brandes et al. 2023). However, these models have largely ignored the recurrent and modular com- position of proteins, which introduces substantial challenges. Protein substructures occur at multiple spatial and functional scales, from local motifs spanning only handful of residues to domains that cover large fractions of proteins (Durairaj et al.] 2023). They are often non-contiguous sequence space, making them difficult to encode with standard sequence-based protein models (Ovchinnikov & Huang 2021). A single residue can belong to several overlapping substructures, inducing hier- archical and context-dependent relationships that are not naturally handled by flat representations. Finally, annotated substructures are distributed in a long-tailed fashion, with abundant secondary structure elements but scarce examples of specialized motifs, complicating the design of training objectives and evaluation protocols (Durairaj et al. (023).
"Equal contribution

---

Published as a conference paper at ICLR 2026
These challenges arise because proteins not uniform chains but organized into recurrent, modular substructures that provide natural multiscale vocabulary for their representation (Sun et al. 2025). At the finest level amino acids, which assemble into secondary structure elements such as alpha helices and heta sheets; these in turn combine into higher-order motifs and domains such beta barrels and zinc fingers (Figure[TA) These substructures are responsible for core molec- ular functions of proteins, such as coordinating metal ions for reaction catalysis or binding to other proteins as parts of cellular signaling networks, and their importance is underscored by their occur- rence in proteins across the tree of life. Decades of biological research has led to the categorization of these recurrent substructures, resulting large databases that exhaustively annotate these ele- ments across proteins (Sonnhammer et al. 1997 Paysan-Lafosse et al. 2025: Blum et al. 2025a). However, prevailing protein representation learning methods still rely on self-supervised objectives that operate at the scale of single amino acids, such as masked language modeling or structural de- noising, or occasionally operate on full proteins (Yu et al.) 2023). This is despite abundant evidence that evolutionarily conserved substructures are key components of protein function (Rossman & Lil- jas 1974). In this work, we ask, how should we systematically incorporate decades of biological knowledge about protein substructures into protein encoding models?
A
Reec
Primary sequence
Secondary structure
Folds and domains
Full tertiary structure
B
Magneton
Curated substructure annotations
Downstream tasks
Substructure tuning
Secondary structure
Functional sites
Domains
prediction
Protein model
Substructure classification
Sequence
Full tertiery
Variant effect prediction
structure
Existing Methods
Figure Overview protein structure the Magneton environment. (A) Proteins are built from mod- ular substructures that assemble into full structures. (B) Magneton leverages decades of substructure research provide environment for developing and evaluating substructureawwars models.
While there exists a growing body of work exploring how to best integrate protein sequence and structure into a single model, either via direct incorporation of structural tokens (Su al. 2023] Li
al. 2024 HHyes al., 2025 Lu et al., 2025 Yuan al., (225) or finetuning of sequence models to better align with structural representations (Zhang et al. 2024b Ouyang-Zhhe Zhang al. 2025), there few examples of incorporating substructure information into protein encoding models. GearNet (Zhang et al. 2022b) uses multi-view contrastive objective and cite recurrent substructures as motivation, but multiple views of subsets of the same protein rather than considering recurrent substructures across proteins. The Functional Community Invariance approach (Wang et al. 2025b) employs secondary structure annotations
guide graph augmentations but ignores higher-order substructures. Other threads of work seek to construct hierarchical representations of proteins, either by connecting residues their exposed surface areas (Somnath al. 2022] Zhang al. 2024c
Mallet et al., 2025). or by progressing from all-atom graphs to residue graphs (Wang et al. 2023 these works pass over protein substructure as a valuable part of the structural hierarchy.
Present work. To close this gap, we create a new environment for developing substructure-awann protein models, which we call Magneton. Magneton has three main components: (1) dataset of proteins with curated substructures; (2) framework for using these substructures to train or finetune protein encoding models; and (3) a benchmark of evaluation tasks that probe the learned representations at the residue, substructure, and protein levels (Figure IB). By curating data from Pfam (Paysan- (Paysan-Lafosse Lafosse et al.) 2025), InterPro (Blum et al. 2025b), and DSSP (Hekkelman et al, create dataset of 530,601 proteins with 1.7 million substructural annotations (37
2025).
2

---

Published as a conference paper at ICLR 2026
million when including secondary structure) across six substructure classes with 13,075 distinct substructure types.
Using Magneton, explore substructtur-ttunigg supervised fine-tuning strategy that distills sub- structural information into protein encoders. Concretely, we formulate substructure-tuning as clas- substructures, where residue-level embeddings produced by
sification of volltiionaillyccosseee base encoder are pooled to construct substructure representations, which are then used to tune the full model with a cross-entropy loss. This objective is model-agnostic, requiring only residue-level embeddings, and naturally extends to multiple structural scales through a multi-task formulation in which each substructure class is assigned its own prediction head and the total loss is the sum across scales.
Our key contributions are:
We present Magneton, an environment for exploring substructure-awarc protein models, consist- ing of large-scale curated dataset of substructural annotations, an associated Python library for training protein encoding models, and suite of 13 evaluation tasks spanning residue, substruc- ture, protein, and interaction levels.
supervised fine-tuning method for distilling substructural in- formation into pretrained models. We exhaustively evaluate its design space across highly local substructures (e.g.. active sites spanning 10 residues) to larger domains, and apply it to six state- of the-art sequence-only and sequence-structtue encoders.
We introduce substructure-tuning
We show that substructure-tunm tuntunggeetnnggggggge improves models' ability to represent protein function and yields consistent gains on function-relatee prediction tasks.
We find that substructure-tuningg produces more consistent representations of substructures of the same type, including types never seen during training, demonstrating that substructure-tuning encourages models learn general features of functional substructures.
The above results hold for both sequence-only models and sequence-structure models, showing that substructural information is complementary to global structure.
We envision that Magneton will catalyze the integration of protein substructures into protein models and motivate the development of new approaches and inductive biases that incorporate decades of knowledge about protein structure across scales.
RELATED WORK
Integrating structure- and function-basse based inductive biases into protein sequence models. A large body of work has explored distilling auxiliary modalities into protein sequence models. Some methods incorporate free-text descriptions. such Gene Ontology terms (Zhang et al. 2022a) or SwissProt annotations (Xu et al., (023). The majority. however, focus on structural information. Explicit approaches integrate structure directly, either through structure graphs (Zhang et al. 2023) or structural tokenization (Su al., 2023 Li et al. 2024). Structural distillation methods instead use structure only at training time, preserving sequence-only inference. For example, Implicit Structure Model (ISM) (Ouyang-Zhim -Zhan et al., 2025) trains residue-level predictors on tokens from a struc- tural autoencoder, while ESM-S (Zhang et al. 2024b) distills global structural information via fold classification. S-PLM (Wang et al. (022a) employs contrastive learning to align representations of an ESM encoder with those of a contact-map encoder. Magneton differs by focusing protein substructures rather than only residue-level or global structural signals. It provides large-scale cu- rated annotations of conserved substructures and framework for supervised finetuning on these elements to encode modular, recurrent units of protein organization. This is orthogonal to existing sequence-structure integration and structural distillation approaches.
Substructure-aware training and hierarchical models. Protein substructure admits a hicrarchical view, but most hierarchical modeling approaches focus on geometric relations rather than functional substructures. Some methods connect residues to exposed surface areas (Somnath et al.) 2022 Zhang et al. 2024c] Mallet et al. 2025), while others connect residues to constituent atoms (Wang et al. 2023). Few approaches incorporate substructural information directly. GearNet (Zhang al. 2022b) uses a multiview contrastive objective that samples local regions within protein, but su- pervision is restricted to intra- intra-protein partitions rather than conserved substructures across proteins. SES- Adaater (Tan et al., (024) augments sequence models with cross-ates atteentton to DSSP-derived secondary structure tokens, but does not extend beyond this single level of annotation. Protein lan-
3

---

Published as a conference paper at ICLR 2026
span- masking, but the masked spans are random residue segments rather than biologicall--dffned substructures. ESM3 (Hayes et al. 2025) introduces multi-track tokenization, including secondary structure and function tracks, where the function track is derived from ontology terms often correlated with substructural annotations. However, the learning remains self-supervised and intra-protein, without supervision on conserved substructures across proteins. Magneton differs by providing annotations of conserved substructures across proteins and by defining supervised training objectives that operate directly on these anno- tations. This design moves beyond local partitions, random spans, or ontology proxies, enabling systematic study of substructure-ave -ware modeling across residue-, motif-, domain-, and protein-level representations.
guage models such XTrimoPGLM (Chen al. 2025)
Geometric protein models. Geometric deep learning has been widely applied to proteins, with
models developed for folding (Jumper et al. [2221] Abramson et al. 2024), structure design (Passaro)
et al. 2025; Watson et al.) 2023 Huang et al. (024)) and representation learning (Jing et al. 2020
Fang et al. 2025). These approaches operate the atom scale (Qu et al., 2025: Widatalla et al, 2025 and encode spatial coordinates of all atoms to model global protein geometry. Magneton addresses complementary problem: representing recurrent substructures that span residues, motifs, and domains, and recur across proteins. Rather than optimizing directly on atomic coordinates or global geometry, Magneton introduces supervised objectives on conserved substructures, providing functional supervision across structural scales. Substructural objectives can also be integrated with atom-scale geometric encoders to yield models that capture fine- -raigga geometry and functional modularity.
METHODS
Preliminaries. Two possible views a protein P are the residue-level, P (0.,....00 az) where a, is the i'th residue in the primary sequence, and the substructure-level,, P = (s (....... 8n) where each 8; represents a substructure contained within a protein. Other views are possible (e.g. atom-level), but these two views are the most relevant for work. In the substructure view, each substructure is subset of k residues, 8; [ where the residues aj may or may not be contiguous in the primary sequence. Since substructures exist at multiple scales, a given residue may be a member of multiple, possibly overlapping, substructures, e.g. residue may be part of secondary structure element, such as a beta strand, that is itself part of larger fold, such as beta barrel. It is also possible for a given residue not be included in any annotated substructure. While the substructure view of a protein is common in the biological community (Rose 1979; Vogel et al.] (2044] Alberts al. 2002), there is lack of curated datasets for exploring in the context of protein modeling.
3.1 MAGNETON ENVIRONMENT
Magneton is environment for developing substructure-aware protein models, and consists of three main parts: (1) a curated dataset of proteins with annotated substructures, (2) framework for using this dataset for substructure- aware training, and (3) an integrated benchmark of evaluation tasks that probe a model's learned representations at multiple structural scales.
Dataset. We use the 2024_0 __0 release of UniProtKB/THEMME
(The UniProt Consortium 2025
as our core protein dataset, containing roughly 254M proteins. We obtain annotations of 8-class secondary structure from DSSP (Kabsch & Sander 1983 Hekkelman al. 2025) and annotations of higher-order structures (Homologous superfamilies, domains, conserved sites, active sites, binding sites) from the 103.0 release of InterPro (Blum al. 2025a We process these raw releases into Magneton's 's core datatypes representing protein and its associated substructures. Due to the scale of the dataset at this stage and the size of protein structure data, we focus our further exploration on the manually curated SwissProt subset of UniProtKB, but make the processed version of the full UniProtKB/TTEMBL dataset available to the community. We obtain amino acid sequences from UniProtKB and predicted structures from AlphaFold DB (Varadi al.) 2022 For consistency across sequence-based and structure- based models, we subset the SwissProt dataset to only proteins with calculated structures in the current (Nov 2022) release of AlphaFold DB, leaving 530,601 proteins. Additional details on the dataset, processing, and example substructures can be found in Appendix A.1.1
4

---

Published as a conference paper at ICLR 2026
Substructure class
Unique types (pre-filter)
Total occurrences (pre-filter)
Unique types (post-filter)
Total occurrences (post-filter)
Median protein span
Homologous
2978
1.09M
1133
1.05M
50% (137 AA)
superfamily
Domain
9133
389K
917
301K
34.8% (127 AA)
Conserved site Binding site Active site Secondary structure
739
175K
356
162K
5.18% (16 AA)
20.1K
19.0K
4.28% (16 AA)
132
31.1K
82
29.2K
3.47% (12 AA)
8
35.2M
8
35.2M
0.94% (3.4 AA)
Total
13075
1.71M
2542
1.56M
w/o secondary structure
Table 1: Summary of Magneton substructure dataset (SwissProt subset). Before and after refer to filtering substructures. Median protein span is the median length of type of substructure, expressed percentage of the protein and as an absolute amino acid count.
To focus learning efforts on substructures where sufficient data is present, create a restricted label set of substructures that occur at least times in the SwissProt dataset, corresponding to retaining only the top 10% most frequently occurring domains. While this may seem stringent, this retains the vast majority of actual substructure occurrences across types, since many substructures have very few occurrences. We additionally generate versions of our dataset using more permissive cutoffs (minimum counts of 25 or 10) (Appendix (A.1.3). Our published datasets retain all substructure annotations enable future research by the community. Table I summarizes the different classes of substructures, their counts, number of types, and typical span on the protein. As expected for substructural elements, the majority of the substructures span less than 10% of the annotated protein, with the scale varying by the class of substructure. We then split this dataset into training, validation, and test sets using the AFDB50 sequence- based clusters Gearii-HHrnannee et al., 2023), ensuring that sequences sharing more than 50% identity and 90% overlap are assigned to the same split.
protein modeling within Magneton, integrate numerous evaluation tasks from the community. These tasks probe a model's learned representations at multiple scales: individual residues, substructures, proteins, and protein interactions (Table 2). At the residue-level, we include contact prediction (Rao al., 2019]. zero-shot prediction of variant effects (Notin et al. 2023). and multiple types of functional residue prediction tasks (Dallago et 2021 Yuan et 2025); at the substructure-level. we in- clude multiclass substructure classification problems derived from the Magneton dataset itself; at the protein-level, we include function prediction (GO and EC terms) (Gligorijevi et al.] (20TT, sub- cellular localization (Almagro Armenteros et al. 2017). and fitness prediction (Rao et al. 2019).
Evaluation benchmark. provide a holistie evaluation of substructure-
Finally, we include a human PPI prediction task (Pan et al. 2010 Xu al. 2022). Full details of evaluation datasets can be found in Appendix A.1.5.
3.2 SUBSTRUCTURE REPRESENTATION AND TUNING
Given the dataset in Magneton, we now have large collection of proteins P, where each pro- tein has curated substructural annotations, P = (s.....,;;); E P. We first use this dataset to assess whether existing protein models can generate meaningful representations of substructures. Specifically, for a protein model f. we construct a representation of each substructure calculating residue-level embeddings, f(P) = (f)...., u2), 22 E Rd where v; the embedding of residue a,, We then perform substructure pooling operation over the constituent residues of f(s) = pool(fe; : a; E s). E Rd, where pool can be any arbitrary pooling operation. These substructure- labels for the final substructure classification task. Since a substructure 's constituent residues are given to the model, this is diagnostic lask meant probe each model's ability to represent sub-
E P by
representations are then input to a classifier over the possible substructure
5

---

Published as a conference paper at ICLR 2026
Scale
Task
Task type
Metric
Data
Interaction
Human PPI prediction
Binary
Accuracy
Pan et al.
Gene Ontology prediction
Multilabel
Fiaa
Gligorijević et al.
Enzyme Commission prediction
Multilabel
Finxx
Protein
Subcellular localization
Multiclass
Accurucy
Almagro Armenteros al.
Binary localization
Binary
Accuracy
Therostabilitty prediction
Regression
Spearman's 's
Rao et al.
Substructure
Substructure classification
Multiclass
Macro accuracy
Ours
Contact prediction
Binary
Precision @
Rao el al
Variant effect prediction
Regression
Spearman's
Notin et al.
Residue
Binding residue categorization
Multilabel
Fmax AUROC
Dallago et al
Functional site prediction
Binary
Yuan et al.
Table 2: Evaluation tasks contained within Magneton. Grouped by the seale of structural representation they interrogate.
structures, task meant measure the ability identify previously unannotated substructures. For this diagnostic assessment, we freeze the parameters of the underlying protein model and train only the substructure classification head.
Substructure-tuning
substructure pools Information throughout protein
Substructure pooling
Secondary structure clossriler
Sequence
Active site classifier
Substructure clessification loss
Protein model
Domain clessif
Curated substructure annotations
Su. Subbstrcchuee embbdddi go
Secondary
Functional
Domain
ambedddigs
Figure 2: Overview of using Magneton for substructure-tuning Given a pre-trained protein model, substructure-tumingg first pools residuc-lovel embeddings create substructure representations, which are then used for supervised finetuming via substructure type-specific classifier heads.
We next explore imbuing existing protein models with substructural information. In a process we refer substructure-tuue -uninng again perform the substructure classification task outlined above, but with finetuning of the original protein model's parameters (Figure 2) to encourage the model to distinguish between the many different types of biologically-relewe Although we use supervised finetuning, other losses, such as a contrastive objective (van den Oord al. 2019), could also be used. The substracture-tuning process is compatible with any finetun- ing method, including parameter-efficien methods such as LoRA (Hu et al.) (021] for larger base models. We perform substructure- tuning using the Magneton training set and explore tuning with different substructure types as well as their combinations. When finetuning with multiple substruc- ture classes, each class uses own predictor module with the cross-entropy loss across all types summed to form the final substructure classification loss.
substructures in dataset.
3.3 IMPLEMENTATION DETAILS
For our experiments, we select base protein models that represent state-of-the-art models across range of model sizes and modality inputs. For sequence-based models, we use ESM2-150M and -650M (Lin et al. 2023) and ESM-C 300M and 600M (ESM Team 2024). For models that incorporate protein structure,
SaProt (Su al. 2023 and ProSST-2048 (Li et al. 2024), both which use both protein sequence and structure. We opt to exclude purely structural models their performance is generally below that of the sequence-structure models we've included.

---

Published as a conference paper at ICLR 2026
For substructure classification and tuning, mension size matches that of the base model, as our prediction modules, and mean pooling for the substructure pooling operation (sce Appendix A.4.I] for exploration of alternat pooling methods). For substructure-tuningg we perform full finetuning of the base model. We use elastic weight consol- idation (EWC) (Kirkpatrick et al. (017) to avoid catastrophic forgetting of the base model's original objective. Detailed training methodology is available in Appendix [..2.
single hidden layer MLPs, where the hidden di-
For supervised downstream evaluations, train head models on top of either the original base model or the substructure-tuncl base model. For these evaluations, we freeze the base model to focus on evaluating the representations learned during substructure-tuningg Results across all tasks and models were generated within the Magneton environment and use identical datasets and splits. We unfortunately exclude ProSST from the functional site prediction and contact prediction tasks due to its incompatibility with experimental structures from PDB. Full training details for all models and evaluation tasks are available Appendix A.2.2
EXPERIMENTS
4.1 SUBSTRUCTURE REPRESENTATION ASSESSMENT
Table shows that base models are readily able to produce effective representations of substructures across scales, with structure- based models generally outperformig sequence-only models. We also find that models can correctly classify substructures within proteins that contain multiple substruc- tures (e.g., accurately classifying all domains within single protein containing multiple domains), indicating that classification relies on local structural cues rather than global structural similarity (Figure 3A). While performance degrades for some rarer substructures, we generally see high accu- racy even for substructures (Figure 3B).
Homologous superfamily
Conserved site
Binding site
Active site
Secondary structure
Model
Domain
ESM2-150M
0.899
0.969
0.988
1.000
0.995
0.827
+ST
0.925
0.983
0.991
0.999
0.994
0.916
ESM2-650M
0.926
0.982
0.986
1.000
0.995
0.892
+ST
0.902
0.967
0.986
1.000
0.996
0.938
ESM-C 300M
0.913
0.962
0.990
0.998
0.994
0.863
+ST
0.946
0.982
0.983
0.999
0.996
0.757
ESM-C 600M
0.919
0.975
0.992
0.977
0.994
0.891
+ST
0.907
0.966
0.993
0.997
0.996
0.927
SaProt (650M)
0.916
0.967
0.992
0.999
0.996
0.955
+ST
0.925
0.980
0.993
0.999
0.996
0.972
ProSST-2048
0.888
0.945
0.995
0.996
0.993
0.927
+ST
0.879
0.976
0.991
0.991
0.995
0.961
Table Comparison of substructure classification performance. Performance the diagnostic task of classifying substructures given their annotated residues, for base and substructure-tumm (+ST) models, values are macro- avero-averaged accuracy.
4.2 SUBSTRUCCURE--TUNIGG
Substructure-tuning configurations. Table shows the results of substructure-tuning with a range of different substructure classes, both individually and their combinations, measured by down- stream evaluation tasks. Due to the large number of possible configurations, we restricted this initial exploration single model (ESM-C 300M), subset of evaluation tasks, and selection of the 20 possible substructure class combinations aimed at exploring combinations of substructure classes across scales.
Our exploration of substructure configurations revealed the following: 1) The effects of substructure- tuning largely consistent across the selected substructure types used, with performance boosts in

---

Published as a conference paper at ICLR 2026
B
1.0
1-
N-12038
2-
Number of domains per protein
Per domain test set accuracy
3-
N-1843
4-
N-234
0.4
5-
N-42
0.2
6-
N-8
1.0
100
200
Average accuracy per protein
Domain occurrences train set
Figure 3: (A) Domain classification uses local cues. Even within proteins containing multiple domains, classification accuracy remains high for all contained domains, Labels within bars show the number of test set proteins containing that number of domains. (B) Domain classification accuracy function of training set representation. Results shown for ESM-C 300M.
Substructures used
EC
GO:BP
GO:CC
GO:MF
Localization (Accuracy)
Thermostability (Spearman's p)
Zero-shot DMS
F
Binary
Subcellular
(Spearman's 's p)
None
0.688
0.307
0.416
0.429
0.871
0.703
0.648
0.432
H
0,805
0.312
0.395
0.518
0.851
0.632
0.662
0.308
D
0.776
0.307
0.403
0.501
0.811
0.640
0.666
0,340
0.749
0.318
0.398
0.491
0.706
0.661
B
0.745
0.315
0.415
0.478
0.852
0.686
0.663
0.423
0.794
0.318
0.403
0,518
0.851
0.639
0.663
0.340
S 0.618
0.297
0.379
0.381
0.823
0.587
0.612
0.264
HD
0.774
0.316
0.388
0,500
0.847
0.606
0.639
0,302
0.765
0.297
0.395
0.466
0.883
0.651
0.644
0.346
HD
0.754
0.318
0.413
0.473
0.868
0.633
0.658
H.CBA.
0.800
0.322
0.389
0.515
0.857
0.611
0.663
0.340
D_.SS
0.751
0.308
0.384
0.462
0.872
0.646
0.643
0.369
0.815
0.329
0.395
0,525
0.851
0.662
0.659
0.369
-CBA_
0.761
0.325
0.403
0.488
0.879
0.681
0.660
0.410
BA..
0.740
0.319
0.406
0.467
0.841
0.677
0.656
0.418
CBAS
0.719
0.313
0.393
0.453
0.839
0.666
0.636
0.379
HDCBAS
0.760
0.315
0.383
0.457
0.832
0.624
0.640
0.359
Table 4: Comparison of substructure--tuingg configurations. Performance across tasks for ESM-C 300M with range of substructure- tuning configurations. For each configuration, the substructures used are indi- cated by the presence of that substructure type's single-letter code: H--Homlogoous superfamily, D=Domain, C=Conserved site, B-Binding site, A=Active site, 5=Secondary structure; an underscore (-) means that sub- structure type was not used.
tasks related to protein function (GO:ME, GO:BP, EC, Thermostability) and neutral to negative ef- fects on localization tasks (GO:CC, Binary localization, Subcellular localization) and residue-level variant-effect prediction. 2) These effects are present even when tuning with very small substruc- tures, such as active sites, which typically consist of only 12 amino acids (median protein span of 3.47%). Based these results, we selected the combination of active site, binding site, and con- served site as the substructure substructure-tuning configuration for use in the full set of models and benchmarks, as this configuration represented a balance of positive gains in function-relate tasks and neutral effects in localization and variant-effect tasks at the residue level.
Substructure-tuning across models. Tables 5 and show how the selected substructure-tuning configuration affects the downstream performance of the full set of base protein models across protein-level and residue-level tasks, respectively (see Appendix Tables A.9 A. uncertainty estimates). The full evaluation across models and benchmarks led to the following conclusions: 1) Results across models are consistent with the initial exploration: performance boosts in function- related tasks and neutral to negative effects on localization and residue-level tasks. 2) Importantly, these results hold for models that already incorporate protein structure as an input (ProSST-2048 and SaProt), suggesting complementarity between structural and substructural information.
8

---

Published as a conference paper at ICLR 2026
BC
GO:BP
GO:CC
GO:MF
Localization (Accuracy)
Thermostability (Spearman's '
Human PPI (AUROC)
Model
Binary
Suhcellular
ESM2-150M
0.727
0.316
0.416
0.441
0,869
0.694
0.627
0.933
+ST
0.742
0.324
0.415
0.473
0.866
0.679
0.582
0.919
ESM2-650M
0.755
0.319
0,431
0.486
0,876
0.710
0.643
0.939
+ST
0.745
0.321
0.440
0.534
0.895
0.749
0.655
0.935
ESM-C 300M
0.688
0.307
0.416
0.429
0.871
0.703
0.648
0.917
+ST
0.761
0.325
0.403
0.488
0,879
0.681
0,933
ESM-C 600M
0.701
0,312
0)403
0,436
0.863
0.713
0.668
0.927
+ST
0.780
0.319
0,385
0.527
0.872
0.635
0.667
0.902
SaProt (650M)
0.778
0.326
0,453
0.538
0.887
0.784
0.692
0,952
+ST
0.839
0.339
0.446
0.584
0.896
0.741
0.697
0.932
ProSST-2048
0.778
0.317
0.426
0.522
0.878
0.693
0.686
0.925
+ST
0.791
0.314
0.420
0.567
0.853
0.683
0.648
0.883
Table 5: Protein-lever task performance for base models and models with substructure-tuningg (+ST),
Binding residue
Functional site prediction
Contact Prediction
Variant Effect (Spearman's 's p)
Model
Binding
Catalytic (AUROC)
Short
Medium (Precision @ L)
Long
(Fmax)
ESM2-150M
0.379
0.871
0.910
0.487
0.452
0.289
0.342
+ST
0.327
0.852
0.890
0.460
0.445
0.285
0.262
ESM2-650M
0.366
0.849
0.912
0.551
0,528
0.372
0.359
+ST
0.362
0.851
0.927
0.532
0.518
0.367
0.317
300M
0.367
0.851
0.923
0.339
0.364
0.174
0.432
+ST
0.411
0.866
0,910
0.350
0.374
0,180
0,410
ESM-C 600M
0.357
0.850
0,921
0.329
0.362
0.161
0.434
+ST
0.368
0.852
0.906
0,313
0.315
0.141
0.381
SaProt (650M)
0.423
0.891
0.923
0.788
0.747
0.697
0.457
+ST
0.400
0.871
0.924
0.765
0.726
0.647
0,405
ProSST-2048
0.375
N/A
N/A
N/A
N/A
N/A
0,507
+ST
0.342
N/A
N/A
N/A
N/A
N/A
0.356
Table 6: Residue-level task performance for base models and models with substructure-tuningg (+ST).
Due to the close relationship between substructures and protein function, we additionally verify that performance increases from cubstructure-tuning are not trivially attributed to leakage between the Magneton substructure training set and the test sets of the evaluation tasks (Appendix (A.1.4). An ablation of EWC finds that it moderates the performance improvements of substructure--tuiing while reducing the amount of degradation in tasks where substructure- (Appendix II. Additionally, we found that substructure-tunie compares favorably to existing methods that distill global structural information into sequence-only models (Appendix A.4.2)).
has negative effects
We explored how substructure-tuning interacts with task-specific finetuning by repeating the evalu- ations above for subset of models and tasks with full finetuning of the protein model for each task (Appendix (.22e) A.2.3). We found that task-specific finetuning results in similar performance across mod- els trained with and without substructure- tuning, indicating that aggressive task speciffe finetuning may dominate the substructural information imbued during the substructure-tuning process.
Mechanistic exploration of substructure-tuning. We next investigated the effects of substructure- tuning on the learned embeddings of the underlying protein models. We found that substructure- tuning greatly increased a model's ability to group substructures of the same type, as measured by silhouette score (Table 7 Appendix Figure A.13). Furthermore, by restricting our analysis to the rare substructure types that were excluded from the Magneton training set entirely, we find that cubstructure-tuning tuning results in more consistent representations of even substructure types that were never seen during training. This indicates that cebistruccure-tuning encourages models to leam general features of functional substructures, rather than just signatures of specific substruc-

---

Published as a conference paper at ICLR 2026
Homologous superfamily
Domain
Conserved site
Binding site
Active site
Model
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
Seen
Unseen
ESM-C 300M
-0.183
0.180
-0.184
0.201
0.279
0.466
0.378
0.641
0.490
0,476
+ST
0.339
0.584
0.486
0.652
0.830
0.747
0.882
0.894
0.933
0.816
SaProt (650M)
0.079
0.301
0.122
0.412
0.534
0.623
0.613
0.796
0.714
0.701
+ST
0.478
0.684
0.554
0.717
0.796
0.764
0.843
0.938
0.912
0.866
Table 7: Silhouette scores for substructure types included ("seen"" and excluded ("unseen") from train- ing. Higher silhouette scores indicate tighter clustering of substructures within a type. "Seen" scores are generated using the Magneton test set proteins (i.e. "seen" refers to substructure types, not individual proteins).
ture types. These experiments focused on ESM-C 300M and SaProt as representative sequence-only and sequence-structure models.
To understand the task-specific effects of substructure-tuning we performed a gradient conflict anal- ysis, in which compared the gradient updates for FSM-C for the substructure classification task and for a set of evaluation tasks, including protein-level function prediction and residue-level classification tasks (Appendix A.3.2). Looking across batches within task, we found that gra- dients for the evaluation tasks were highly consistent and gradients for substructure classification had lower, although still positive, similarity. However, substructure classification gradients were close to orthogonal to gradients for the evaluation tasks. While these results do not fully explain the task-specific effects of substructure-tuning, they suggest that the behavior is not due a simple misalignment between the substructure objective and certain downstream tasks. Instead, we hy- pothesize that our current instantiation of substructure-tuning biases the model against fine-grained residue-level distinctions, because it explicitly encourages residues within the same substructure to share similar representations.
Finally, performed explainability analysis to understand if substructure-tuning increases model's 's utilization of substructural information. For the subset of GO:MF terms that can be mapped to domain annotations, found that substructure-tunin dictions to residues within domains by an average of 17% over the untuned base model (Appendix A.3.3
resulted in increased attribution of pre-
5 CONCLUSION AND FUTURE WORK
Our study has several limitations and directions for future work. We focused on a model-agnostic gubstructure-tuning objective applied to existing protein encoders. While this approach consistently improves function-centric tasks, it yields mixed effects across the full benchmark and can be atten- uated by aggressive task-specific finetuning. These findings motivate exploring alternative ways to incorporate substructural information, including architectures and objectives that explicitly integrate substructures. Modifications at the architectural level, such as hierarchical or graph-based encoders, or training objectives that operate across multiple scales simultaneously, may provide a more stable integration strategy. Our current exploration of substructure-tuning focused on which substructure types to use for tuning, but their representation varies greatly across the dataset (c.g., millions of sec- ondary structures, tens of thousands of active sites). Exploring how to best balance or weight these different types is another avenue of future exploration. Finally, our experiments were restricted to SwissProt proteins. Extending to the full UniProtKB and incorporating the long tail of infrequent substructures could enable deeper insights into poorly characterized aspects of protein modularity.
In this work, we have presented the open problem: how to best incorporate decades of research on protein substructures into protein models? To this end, we introduced Magneton, an integrated environment for developing and evaluating substructure-aware protein models. Using Magneton, we explored both how well existing models can represent protein substructures and whether a su- pervised fine-tuning paradigm can be used to effectively imbue those models with substructural information. We found that while this direct, intuitive bobstructure-tuningg approach has both pos- itive and negative effects downstream tasks, it also encourages models to learn general features of functional substructures and suggests that substructural information is complementary to global structure. Our work lays the foundation for the development of substructure-awant protein models.
10

---

Published as a conference paper at ICLR 2026
6 ETHICS STATEMENT
This work involves the analysis of publicly available protein sequence and structure data from es- tablished databases (UniProtKB/SwissPro. AlphaFold DB, InterPro, and Pfam). All data used in this study are derived from previously published sources and do not involve human subjects, ani- mal experiments, or the generation of new biological data that require ethical oversight. Our work improves computational methods for understanding protein function, which could contribute to ad- vances drug discovery and biotechnology. We encourage responsible use of our methods and datasets, which are publicly available, to promote scientífic reproducibility and advancement.
7 REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our work, we provide the following:
1. All code for Magneton, including data processing pipelines, model training scripts, and evaluation benchmarks, available at
The processed datasets are publicly available at htttps://huggihgaacc...
Bataasetts/raallef///mnntttttttt
2. We provide comprehensive implementation details, including model architectures and hy- perparameters, training procedures, optimization details, and data splitting procedures (Methods B.3).
3. We specify all experimental details, including dataset statisties and preprocessing steps such as substructure filtering criteria and thresholds (Table [ Appendix A.IIII. as well as evaluation metrics and protocols for all benchmark tasks (Table
4. All experiments can be reproduced using 1-4 NVIDIA A100 GPUs.
The modular design of Magneton facilitates easy plug-and-play usability of our benchmark suite, supporting not only reproducibility but also future research this area.
8 ACKNOWLEDGEMENNTS
We gratefully acknowledge the support of NIH ROI-HDI08794, NSF CAREER 2339524, U.S. DoD FA8702-15-D-0001, ARPA-H Biomedical Data Fabric (BDF) Toolbox Program, Harvard Data Science Initiative, Amazon Faculty Research, Google Research Scholar Program, AstraZeneca Research, Roche Alliance with Distinguished Scientists (ROADS) Program, Sanofi iDEA-iTECH Award, GlaxoSmith Award, Boehringer Ingelheim Award, Merck Award, Optum AI Research Collaboration Award, Pfizer Research, Gates Foundation (INV-079038), Aligning Science Across Parkinson's 's Initiative (ASAP), Chan Zuckerberg Initiative, John and Virginia Kaneb Fellowship at Harvard Medical School, Biswas Computational Biology Initiative in partnership with the Milken Institute, Harvard Medical School Dean's 's Innovation Fund for the Use of Artificial Intelligence, and the Kempner Institute for the Study of Natural and Artificial Intelligence at Harvard University. RC was supported by the National Science Foundation Graduate Research Fellowship under Grant No. 214 2141064. Any opinions, findings, conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the funders.
REFERENCES
Josh Abramson, Jonas Adler, Jack Dunger, Richard Evans, Tim Green, Alexander Pritzel, Olaf Ronneberger, Lindsay Willmore, Andrew J. Ballard, Joshua Bambrick, Sebastian W. Bodenstein, David A. Evans, Chia-Chun Hung, Michael O' O'Neill, David Reiman, Kathryn Tunyasuvunakool, Zachary Wu, Akvile Zemgulyté, Eirini Arvaniti, Charles Beattie, Ottavia Bertolli, Alex Bridg- land, Alexey Cherepanov, Miles Congreve, Alexander I. Cowen-Rivers, Andrew Cowie, Michael Figurnov, Fabian B. Fuchs, Hannah Gladman, Rishub Jain, Yousuf A. Khan, Caroline M. R. Low, Kuba Perlin, Anna Potapenko, Pascal Savy, Sukhdeep Singh, Adrian Stecula, Ashok Thillaisun- daram, Catherine Tong. Sergei Yakneen, Ellen D. Zhong, Michal Zielinski, Augustin Zidek, Vic- Bapst, Pushmeet Kohli, Max Jaderberg, Demis Hassabis, and John M. Jumper. Accurate
11

---

Published as a conference paper at ICLR 2026
structure prediction of biomolecular interactions with AlphaFold 3. Nature, 630(8016):493-50)) 2024. ISSN 1476-4687. doi: 00008//41586-024-0774445
Bruce Alberts, Alexander Johnson, Julian Lewis, Martin Raff, Keith Roberts, and Peter Walter. The Shape and Structure of Proteins. In Molecular Biology of the Cell. 4th edition. Garland Science, 2002.
José Juan Almagro Armenteros, Casper Kaae Sønderby, Søren Kaae Sønderby, Henrik Nielsen, and Ole Winther. DeepLoe: prediction of protein subcellular localization using deep learning. Bioinformaticx, 33(21):3387-33955 2017. ISSN 1367-4803. doi: 10..09//hionffommatic/oxxx33
Inigo Barrio-Hernandez, Jingi Yeo, Jürgen Jänes, Milot Mirdita, Cameron L. M. Gilchrist, Tanita Wein, Mihaly Varadi, Sameer Velankar, Pedro Beltrao, and Martin Steinegger. Clustering pre- dicted structures at the scale of the known protein universe. Nature, 622(7983))637-664 2023.
ISSN 1476-4687. doi: 10.1038//41556-02--05555
Matthias Blum, Antonina Andreeva, Laise Cavalcanti Florentino, Sara Rocio Chuguransky, Tiago Grego, Emma Hobbs, Beatriz Lazaro Pinto, Ailsa Orr, Typhaine Paysan-Lafosse, Irina Pona- mareva, Gustavo A Salazar, Nicola Bordin, Peer Bork, Alan Bridge, Lucy Colwell, Julian Gough, Daniel H Haft, Ivica Letunic, Felipe Hliares-Lóóezz Aron Marchler-Bauer. Lactitia Meng- Papaxamthos, Huaiyu Mi, Darren Natale, Christine Orengo, Arun Pandurangan, Damiano Piovesan, Catherine Rivoire, Christian J A Sigrist, Narmada Thanki, Françoise Thibaud-Nissen, Paul D Thomas, Silvio E Tosatto, Cathy H Wu, and Alex Bateman. InterPro: the protein sequence classification resource 2025. Nucleic Acids Research, 53(D1):D414-4556 2025a. ISSN 1362-4962. doi: 10. 10.1093/nar/kkae11000
Matthias Blum, Antonina Andreeva, Laise Cavalcanti Florentino, Sara Rocio Chuguransky, Tiago Grego, Emma Hobbs, Beatriz Lazaro Pinto, Ailsa Orr, Typhaine Paysan-Lafose Irina Pona- mareva, al. InterPro: the protein sequence classification resource in 2025. Nucleic Acids Research, 53(DI) D456, 2025b.
Nadav Brandes, Grant Goldman, Charlotte H. Wang, Chun Jimmie Ye, and Vasilis Ntranos. Genome-wide prediction of disease variant effects with deep protein language model. Nature Genetics, 55(9):1512-1522 2023. ISSN 1546-1718.
doi: 10. 10.10388441588-00330044666
Bo Chen, Xingyi Cheng, Pan Li, Yangli-ao Geng, Jing Gong, Shen Li, Zhilei Bei, Xu Tan, Boyan Wang, Xin Zeng, Chiming Liu, Aohan Zeng, Yuxiao Dong, Jie Tang, and Le Song. XTrimoPGLM: unified 100-billion- parameter billeion-parameter pretrained transformer for deciphering the language of proteins. Nature Methods, 1028 1039, 2025. ISSN 1548-7105. doi: 10. 10.1038/s41550
Christian Dallago, Jody Mou, Kadina E. Johnston, Bruce Wittmann, Nick Bhattacharya, Samuel Goldman, Ali Madani, and Kevin K. Yang. FLIP: Benchmark tasks in fitness landscape inference for proteins. In Thirty-fifth Conference Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
Janani Durairaj, Andrew M Waterhouse, Toomas Mets, Tetiana Brodiazhenko, Minhal Abdullah, Gabriel Studer, Gerardo Tauriello, Mehmet Akdel, Antonina Andreeva, Alex Bateman, et al. Uncovering new families and folds in the natural protein universe. Nature, 622(7983):666-653. 2023.
Ahmed Elnaggar, Michael Heinzinger, Christian Dallago, Ghalia Rehawi, Yu Wang. Llion Jones, Tom Gibbs, Tamas Feher, Christoph Angerer, Martin Steinegger, Debsindhu Bhowmik, and Burkhard Rost. ProfTrans: Toward Understanding the Language of Life Through Self-SSuper Learning. IEEE transactions on pattern analysis and machine intelligence, 44(10):112-71277, 2022. ISSN 1939-3539, doi: 10. 00.1109/TAAMM12211339333
ESM Team. ESM Cambrian: Revealing the mysteries of proteins with unsupervised learning. http::/wwwwwooluuuoooaaccc..l/ee
2024.
Ada Fang, Michael Desgagné, Zaixi Zhang, Andrew Zhou, Joseph Loscalzo, Bradley L. Pente- lute, and Marinka Zitnik. Learning Universal Representations of Intermolecular Interactions with ATOMICA, 2025. ISSN 2692-8205.
12

---

Published as a conference paper at ICLR 2026
Vladimir Gligorijević, P. Douglas Renfrew, Tomasz Kosciolek, Julia Koehler Leman, Daniel Beren- berg, Tommi Vatanen, Chris Chandler, Bryn C. Taylor, lan M. Fisk, Hera Vlamakis, Rammik J. Xavier, Rob Knight, Kyunghyun Cho, and Richard Bonneau. Structure- based protein function prediction using graph convolutional networks. Nature Communications, 2(1)::1688 2021. ISSN 2041-1723. doi: 10.10388/41147--22-22333
Thomas Hayes, Roshan Rao, Halil Akin, Nicholas J. Sofroniew, Deniz Oktay, Zeming Lin, Robert Verkuil, Vincent Q. Tran, Jonathan Deaton, Marius Wiggert, Rohil Badkundri, Irhum Shafkat, Jun Gong, Alexander Derry, Raul S. Molina, Neil Thomas, Yousuf A. Khan, Chetan Mishra, Carolyn Kim, Liam J. Bartie, Matthew Nemeth, Patrick D. Hsu, Tom Sercu, Salvatore Candido, and Alexander Rives. Simulating 500 million years of evolution with a language model. Science, 387(6736))850-888 2025, doi: 10. 126/science.ads0015
Maarten L. Hekkelman, Daniel Álvarez Salmoral, Anastassis Perrakis, and Robbie Joosten. DSSP 4: FAIR annotation of protein secondary structure. Protein Science, 34(8):e70208, 2025. ISSN 1469-896X. doi: 10. 10.1100//oo.7000 70208.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models, 2021.
Yufei Huang, Yunshu Liu, Lirong Wu, Haitao Lin, Cheng Tan, Odin Zhang, Zhangyang Gao, Siyuan Li, Zicheng Liu, Yunfan Liu, Tailin Wu, and Stan Z. Li. EVA: Geometric Inverse Design for Fast Protein Motif-Scaffolding with Coupled Flow. In The Thirteenth International Conference on Learning Representations, 2024.
Bowen Jing, Stephan Eismann, Patricia Suriana, Raphael John Lamarre Townshend, and Ron Dror. Learning from Protein Structure with Geometric Vector Perceptrons. In International Conference on Learning Representations, 2020.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, Alex Bridgland, Clemens Meyer, Simon A. A. Kohl, Andrew J. Ballard, Andrew Cowie, Bernardino Romera- Paredes, Stanislav Nikolov, Rishub Jain, Jonas Adler, Trevor Back, Stig Petersen, David Reiman, Ellen Clancy, Michal Zielinski, Martin Steinegger, Michalina Pacholska, Tamas Berghammer, Se- bastian Bodenstein, David Silver, Oriol Vinyals, Andrew W. Senior, Koray Kavukcuoglu, Push- meet Kohli, and Demis Hassabis. Highly accurate protein structure prediction with AlphaFold. doi: 1100388/41586-021-0003383
Nature, 596(7873) 2021. ISSN 1476-4687.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: Pattern recogni- tion of hydrogen-bonded and geometrical features. Biopolymers, 22((2):2577-2637, 1983. ISSN 1097-0282. doi: (0.00///p..3022221.......
Felix Kallenborn, Alejandro Chacon, Christian Hundt, Hassan Sirelkhatim, Kieran Didi, Sooyoung Cha, Christian Dallago, Milot Mirdita, Bertil Schmidt, and Martin Steinegger. GPU-accelerated homology search with MMseqs2. Nature Methods, 22(10):2024-200 2027, 2025. ISSN 1548-7105.
doi: 10.1038/441592-02--222 028-9-8.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Barwinkk, Demis Hass- abis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences, 11<(1):3521-3526 2017. doi: 10. 1073333/p 161 14.
Mingehen Li, Yang Tan, Xinzhu Ma, Bozitao Zhong, Huiqun Yu, Ziyi Zhou, Wanli Ouyang, Bingxin Zhou, Pan Tan, and Liang Hong. ProSST: Protein Language Modeling with Quantized Structure and Disentangled Attention. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, Allan dos Santos Costa, Maryam Zarandi, Tom Sercu, Salvatore Candido, and Alexander Rives. Evolutionary-scale prediction of ateelc-level protein structure with a language model. Science, 379(6637::1123 1130, 2023, doi: 10.1 126/ ac1554
13

---

Published as a conference paper at ICLR 2026
Iya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization, 2019.
Jiarui Lu, Xiaoyin Chen, Stephen Zhewen Lu, Chence Shi, Hongyu Guo, Yoshua Bengio, and Jian Tang. Structure Language Models for Protein Conformation Generation. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Vincent Mallet, Yangyang Miao, Souhaib Attaiki, Bruno Correia, and Maks Ovsjanikov. Atom- Surf: Surface Representation for Learning on Protein Structures. The Thirteenth International Conference on Learning Representutions, 2025.
Joshua Meier, Roshan Rao, Robert Verkuil, Jason Liu, Tom Sercu, and Alexander Rives. Language models enable zero-shot prediction of the effects of mutations on protein function. In Advances in Neural Information Processing Systems, 2021.
Pascal Notin, Aaron Kollasch, Daniel Ritter, Lood van Niekerk, Steffanie Paul, Han Spinner, Nathan Rollins, Ada Shaw, Rose Orenbuch, Ruben Weitzman, Jonathan Frazer, Mafalda Dias, Dinko Franceschi, Yarin Gal, and Debora Marks. ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design. Advances Neural Information Processing Systems, 36:64331- 64379, 2023.
Jeffrey Joyang-Zhange Chengyue Gong, Yue Zhao, Philipp Krachenbuchl, Adam Klivans, and Daniel Jesus Diaz. Distilling Structural Representations into Protein Sequence Models. In The Thirteenth International Conference on Learning Representations, 2025.
Sergey Ovchinnikov and Po-Ssu Huang. Structure- protein design with deep learning. Current Opinion in Chemical Biology, 65: 65:136-144, 2021.
Xiao- Yong Pan, Ya-Nan Zhang, and Hong-Bin Shen. Large-scale prediction of human protein- protein interactions from amino acid sequence based on latent topic features. Journal of Proteome Research, 9100):992-5001, 2010. ISSN 1535-3907. doi: 10.1021/pr1006118
Saro Passaro, Gabriele Corso, Jeremy Wohlwend, Mateo Reveiz, Stephan Thaler, Vignesh Ram Somnath, Noah Getz, Tally Portnoi, Julien Roy, Hannes Stark, David Kwabi-Addee Dominique Beaini, Tommi Jaakkola, and Regina Barzilay. Boltz-2: Towards Accurate and Efficient Binding Affinity Prediction, 2025.
Typhaine Paysan-Lafossse Antonina Andreeva, Matthias Blum, Sara Rocio Chuguransky, Tiago Grego, Beatriz Lazaro Pinto, Gustavo A Salazar, Maxwell L Bileschi, Felipe Llinares-López, Laetitia Pangg-Papaxatthos, Lucy Colwell, Nick V Grishin, Dustin Schaeffer, Damiano Clementel, Silvio C E Tosatto, Erik Sonnhammer, Valerie Wood, and Alex Bateman. The Pfam protein families database: embracing AI/ML. Nucleic Acids Research, 53(D1):D523 2D00--5222---5.,..-- 2025. ISSN 1362-4962. doi: 10. 10193/nar/kaae999,
Typhaine Paysan-Lafosse, Antonina Andreeva, Matthias Blum, Sara Rocio Chuguransky, Tiago Grego, Beatriz Lazaro Pinto, Gustavo A Salazar, Maxwell L Bileschi, Felipe Llinares-López, Laetitia Meng-Papaxanther et al. The Pfam protein families database: embracing AI/ML. Nu- cleic Acids Research, 53(D1))D5233D534 2025.
Wei Qu, Jiawei Guan, Rui Ma, kezhai, Weikun. Wu, and Haobo Wang. P(all-atom) Unlocking New Path For Protein Design. In Forty-second International Conference on Machine Learning, 2025.
Roshan Rao, Nicholas Bhattacharya, Neil Thomas, Yan Duan, Xi Chen, John Canny, Pieter Abbeel, and Yun S. Song. Evaluating Protein Transfer Learning with TAPE, 2019.
Alexander Rives, Joshua Meier, Tom Sercu, Siddharth Goyal, Zeming Lin, Jason Liu, Demi Guo, Myle Ott, C. Lawrence Zitnick, Jerry Ma, and Rob Fergus. Biological structure and function emerge from scaling unsupervised learning to 250 million protein sequences. Proceedings of the National Academy of Sciences, 118(15):c21163911
2021. doi: 10. 20.1073/pnas.2016239111
George D. Rose. Hierarchie organization of domains in globular proteins. Journal of Molecular Biology, 134(3):447-470, 1979. ISSN 0022-2836. doi:
14

---

Published as a conference paper at ICLR 2026
Michael G. Rossman and Anders Liljas. Recognition of structural domains in globular pro- teins. Journal of Molecular Biology, 85(1):177-81,, 1974. ISSN 0022-2836. doi: 10.1016/ 0022-2836(79990133-1
Vignesh Ram Somnath, Charlotte Bunne, and Andreas Krause. Multi-Scale Representation Learning on Proteins, 2022.
Erik L.L. Sonnhammer, Sean R. Eddy, and Richard Durbin. Pfam: A comprehensive database of protein domain families based on seed alignments. Proteins: Structure, Function, and Bioin- formatics, 28(3):405-420, 1997. ISSN 1097-0134. doi: 10. 101102//(CC11999:013440077777719
Martin Steinegger and Johannes Söding. MMseqs2 enables sensitive protein sequence searching for the analysis of massive data sets. Nature Biotechnology, 35(11):1026-1022 2017. ISSN
1546-1696. doi: 10.1038/nbt.398.
Jin Su, Chenchen Han, Yuyang Zhou, Junjie Shan, Xibin Zhou, and Fajie Yuan. SaProt: Protein Language Modeling with Structure-aware Vocabulary, 2023.
Michael Sun, Weize Yuan, Gang Liu, Wojeiech Matusik, and Marinka Zitnik. Protein structure tokenization via geometric byte pair encoding, arXiv:2511.11758 2025.
Yang Tan, Mingchen Li, Bingxin Zhou, Bozitao Zhong, Lirong Zheng. Pan Tan, Ziyi Zhou, Huiqun Yu, Guisheng Fan, and Liang Hong. Simple, Efficient, and Scalable Structure-ANi Awwree Adapter Boosts Protein Language Models. Journal of Chemical Information and Modeling, 64(16):6338- doi: 10. (a002//ac..ccm.Ac006....
6349, 2024. ISSN 1549-9596.
The UniProt Consortium. UniProt: the Universal Protein Knowledgebase in 2025. Nucleic Acids Research. (3(DI):D609-D617 2025. ISSN 1362-4962. doi: 10. 10.10933mar/gkae1000
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation Learning with Contrastive Pre- dictive Coding, 2019.
Mihaly Varadi, Stephen Anyango, Mandar Deshpande, Sreenath Nair, Cindy Natassia, Galabina Yordanova, David Yuan, Oana Stroe, Gemma Wood, Agata Laydon, Augustin Zidek, Tim Green, Kathryn Tunyasuvunakool, Stig Petersen, John Jumper, Ellen Clancy, Richard Green, Ankur Vora, Mira Lutfi, Michael Figurnov, Andrew Cowie, Nicole Hobbs. Pushmeet Kohli, Gerard Kleywegt, Ewan Birney, Demis Hassabis, and Sameer Velankar. AlphaFold Protein Structure Database: massively expanding the structural coverage of protein-sqquence space with high- accuracy models. Nucleic Acids Research, 50DDI):D439-D444 2022. ISSN 0305-1048. doi: 10.1093/nar/gkab1001.
Christine Vogel, Matthew Bashton, Nicola D Kerrison, Cyrus Chothia, and Sarah A Teichmann. Structure, function and evolution of multidomain proteins. Current Opinion in Structural Biology, 14(2):208-215 2004. ISSN 0959-440X. doi: 10. s016//).bi.2004.00..
Duolin Wang, Mahdi Pourmirzaei, Usman L. Abbas, Shuai Zeng, Negin Manshour, Farzaneh Es- maili, Biplab Poudel, Yuexu Jiang. Qing Shao, Jin Chen, and Dong Xu. S-PLM: StructureAAware Protein Language Model via Contrastive Learning Between Sequence and Structure. Advanced Science, 12(5):244421,, 2025a. ISSN 2198-3844. doi: (0.1002/advvs.22244222
Limei Wang, Haoran Liu, Yi Liu, Jerry Kurtin, and Shuiwang Ji. Learning Hierarchical Protein Representations via Complete 3D Graph Networks, 2023.
Yusong Wang, Shiyin Tan, Jialun Shen, Yicheng Xu, Haobo Song, Qi Xu, Prayag Tiwari, and Mingkun Xu. Enhancing Graph Contrastive Learning for Protein Graphs from Perspective of Invariance. In Forty-second International Conference on Machine Learning, 2025b.
Joseph L. Watson, David Juergens, Nathaniel R. Bennett, Brian L. Trippe, Jason Yim, Helen E. Eisenach, Woody Ahern, Andrew J. Borst, Robert J. Ragotte, Lukas F. Milles, Basile I. M. Wicky, Nikita Hanikel, Samuel J. Pellock, Alexis Courbet, William Sheffler, Jue Wang. Preetham Venkatesh, Isaac Sappington, Susana Vázquez Torres, Anna Lauko, Valentin De Bortoli, Emile Mathieu, Sergey Ovchinnikov, Regina Barzilay, Tommi S. Jaakkola, Frank DiMaio, Minkyung
15

---

Published as a conference paper at ICLR 2026
Back, and David Baker. De design of protein structure and function with RFdiffusion. Nature, 620(7976;: 2023. ISSN 1476-4687. doi: 10.1038864158--0220666555
Talal Widatalla, Richard W. Shuai, Brian Hie, and Possu Huang. Sidechain conditioning and mod- eling for full-atom protein sequence design with FAMPNN. In Forty-second International Con- ference on Machine Learning, 2025.
Minghao Xu, Zuobai Zhang, Jiarui Lu, Zhaocheng Zhu, Yangtian Zhang, Chang Ma, Runcheng Liu, and Jian Tang. PEER: Comprehensive and Multi-Task Benchmark for Protein Sequence Understanding. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Minghao Xu, Xinyu Yuan, Santiago Miret, and Jian Tang. ProtST: Multi-Modality Learning of Protein Sequences and Biomedical Texts. https://arxiv.or//bb//331... 12040v2, 2023.
Tianhao Yu, Haiyang Cui, Jianan Canal Li, Yunan Luo, Guangde Jiang, and Huimin Zhao. Enzyme function prediction using contrastive learning, Science, 379(6639))1358-1366 2023. doi: 10.
httpscieece.adf2465.
Xinyu Yuan, Zichen Wang, Marcus Collins, and Huzefa Rangwala. Protein Structure Tokeniza- tion: Benchmarking and New Recipe. In Forty-second International Conference Machine Learning. 2025.
Chengxin Zhang. Xi Zhang, Lydia Freddolino, and Yang Zhang. BioLiP2: an updated structure database for biologically relevant ligand-protein interactions. Nucleic Acids Research, 52(D1): D404-D412, 2024a. ISSN 1362-4962. doi: 10. 1090///////aa
Ningyu Zhang, Zhen Bi, Xiaozhuan Liang, Siyuan Cheng. Haosen Hong, Shumin Deng, Jiazhang Lian, Qiang Zhang, and Huajun Chen. OntoProtein: Protein Pretraining With Gene Ontology Embedding, 2022a.
Zuobai Zhang, Minghao Xu, Arian Rokkum Jamasb, Vijil Chenthamarakshan, Aurelie Lozano, Payel Das, and Jian Tang. Protein Representation Learning by Geometric Structure Pretraining. In The Kleventh International Conference Learning Representations, 2022b.
Zuobai Zhang, Chuanrui Wang, Minghao Xu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang. A Systematic Study of Joint Representation Learning on Protein Sequences and Structures. In ICLR 2023 Machine Learning for Drug Discovery Workshop. arXiv. 2023.
Zuobai Zhang, Jiarui Lu, Vijil Chenthamarakshan, Aurélie Lozano, Payel Das, and Jian Tang. Stracture-Informed Protein Language Model, 2024b.
Zuobai Zhang, Pascal Notin, Yining Huang, Aurelie Lozano, Vijil Chenthamarakshan, Debora Su- san Marks, Payel Das, and Jian Tang. Multi-Scale Representation Learning for Protein Fitness Prediction. In The Thirth-eighth Annual Conference on Neural Information Processing Systems, 2024c.
16

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DATASETS
A.1.1 MAGNETON SUBSTRUCTURE DATASET
Here we provide additional details on the curated protein substructure dataset that makes up a core part of the Magneton environment.
Dataset processing. Below we outline the steps to create the protein substructure dataset:
We start from the full XML file of all InterPro annotations (mat chwccclleteeeeee.cc xml g2) for the 103.0 release, downloaded from the InterPro FTP server
We parse each XML entry, which each correspond to a single protein, by extracting all <natch> elements that contain at least one <ipr> element. The presence of the <ipr> element indicates that the annotation integrated into InterPro and has been assigned a unique InterPro accession ID. No additional filtering is performed at this stage.
At this point, the dataset covers all of the approximately 254 million proteins in the 2024_06 release of BniProtKBTTrEMBB the set of all SwissProt proteins (UniProt IDs and amino acid sequences) from the uniprot sprot sprot-only22 -on1y0024 tar tar. gz file on the UniProt FTP server
To subset the SwissProt set, we obtain
To ensure consistency between models trained using only sequence data and models trained using sequence and structure data, we further subset the SwissProt set to only proteins contained within the v4 release of AlphaFold DB (swissprot. cif.v4. taa
We use the CIF files from AlphaFold DB to source secondary structure annotations as calculated using DSSP.
To select for non-redundant set of substructural annotations, select only annotations marked as "representative" within InterPro, when such annotations are available (as of the 103.0 release, these were only available for the "Repeat", "Family", and "Domain" types)
The full of scripts used for the above steps alongside a detailed README file are available in our GitHub repository in the scripts / proccssssttpocsssssieee ing directory.
Substructure type descriptions. Here we provide additional information about the various types d of substructures contained within the Magneton dataset.
Homologous superfamily - a group of protcins that share a common evolutionary origin, reflected by similarity in their structure, even if sequence similarity is low. Examples in- clude the alpha-helical portion of some viral capsid proteins (IPR008935) and group of single-stranded DNA-binding transcriptional regulator proteins (IPR009044).
Domain distinct functional, structural, or sequence units that may exist in variety of biological contexts. Examples include zinc finger binding domains which serve as binding sites for various types of ligands (one example type is [PRO00058) and various types of phosphatase domains which enable regulation of protein phosphorylation (such as IPR000242).
Conserved site a short sequence that contains one or more conserved residues. Examples include the helix-tur--ellix motif found in all known DNA binding proteins that regulate gene expression (IPR000047) and the helix- haiix-hairpin-helii motif found in proteins that ex- hibit non-specific DNA binding activity (IPR000445).
Binding site a short sequence that contains one more conserved residues, which form a protein interaction site. Examples include sites for binding copper in various enzymes (IPR001505) and sites for binding proteins with other well-characterized motifs (for exam- ple, the IQ motif which binds the EFFhand domain, [PR000048).
https://w..iiiioooooo/////aaabbs http:///wwsssssso///oo/o///eeeee relenses/nelense- 2024 06/knowledgebasee/
http://ieee..ooooooooooooaaeeeee dom.html
17

---

Published as a conference paper at ICLR 2026
Active site short sequence that contains the protein to bind ligand. Examples include active sites for catalyzing hydrolysis of DNA and RNA (IPR002071 and active sites for catalyzing the breakdown of lipids (IPR008265).
conserved residues, which allow
Secondary structure conserved local spatial arrangements of span of amino acids in protein. Canonical examples are alpha helices and beta sheets. For our work, we use 8-class secondary structure definitions from DSSP
Example dataset entry. For illustrative purposes, here we provide an abbreviated example of single entry the Magneton dataset in JSONL format:
; "ADACOSIHMU",
"name": "ABTER ACT8"*
"I elenghh";
"entriis"::
"id": "IPR035897*, "eiement__type type";
*match. id"t 40..5A 101400
"element_nanc". "toll/Ancesicuuda11 receptor homology (TIR) domain supertanily",
reppooentativve"" positions":
BO,
266
"IPR300157",
taiement_type"" *Domain",
mmatch_id"; "SF13676",
"element_nane": "Iel/iaterieukde-1 honology (TIR) demain", "oppessntaaive": talse, positions":
138,
*id": "IPR300157*,
"element__ypee" "Demain",
mnacch_id"; "P850104",
"eiemont_nane": "fol1/incecicukdn-1 receptor homology (IIR) demain", iep Lzve, positions":
133,
296
"scondayy_etructt":
"dsss_typa"; "Alphabelix", *start": 3, *end":
Lype": "Pan""
*stast": 21,
Rend":
"ypep_yyp"" *Pum""
Patact": 24,
tend": 26
3
18

---

Published as a conference paper at ICLR 2026
A.1.2 SUBSTRUCTURE COMPOSITION ANALYSIS
Here we provide exploratory plots to give a sense of the overall size and composition of the different types of substructures contained within the Magneton dataset. These plots show data collected from the SwissProt dataset prior to filtering
As shown Table I we find that the typical length of substructure varies widely, with domains spanning hundreds of residues and various functional sites spanning tens of residues (Appendix Fig- ures A.T A.2). We also find some binding site outliers in terms of length, which may be indicative of annotation artifacts. Despite the wide variance in length for substructures, of the protein they cover is relatively consistent for domains and functional sites (Appendix Figure A.4). We additionally inspect the amino acid composition of the various substructure types, both at the individual amino acid level (Appendix Figures AA. A.5), and with amino acids grouped by chemical characteristics of their sidechains (Appendix Figure A.7
find that the amount
Homologous superfamily (n-1.089.2211
Domain (0-309550))
Conserved site (n-175.185)
10a
10a
10*
100
10a
10"
1.7
107
10
107
100
102
101
10r
10a
109
10
500
1000
1500
2000
500
1000
1500
100
200
300
Length duues)
Lengtth (residues)
Lengtth (residues)
Sinding (n-20,
Active site (n-31.059)
10
10P
107
Count
3.0
102
10t
107
0
250
500
750
1000
20
30
Length fresi dues)
Length (residues)
Figure A.1: Distribution of substructure lengths by type. Here, length is defined as the total number of residues contained within the substructure, regardless of whether they are contiguous within the sequence.
19

---

Published as a conference paper at ICLR 2026
Homologous superfamily (m=1.078.744)
Domain (m= 385.652)
Conserved (n 173.433
10
10
100
200
300
400
500
200
400
600
10
20
30
40
50
Length (residues)
Length (residues)
Lengthh (residues)
Binding (n-19,933)
Active site 30,744
10"
10
10'
100
10a
10
10
20
30
40
50
30
1.5
20
25
Length (residues)
Length (rasidues)
Figure A.2: Distribution of substructure lengths by type. Same as Figure A.1 but filtered to remove outliers (greater than 99th percentile of length within that type).
Homolagous superfamily (n-1,089,211
Domain (n-389,558)
Conserved site (n-175,185)
10'
10'
10a
10a
Count
100
10a
102
10a
10a
10
10
Number Segments
Number Segments
Segments
Binding site (n-20.1330)
Active site In-31.059)
3.0"
10a
100
1.0
302
107
103
10a
10
20
JU
50
Number Segments
Number Segments
Figure A.3: Distribution of number of contiguous segments per substructure, by type.
20

---

Published as a conference paper at ICLR 2026
Nm11,00,704
THt 175755185
no26,136
ne 30,000
1.0
Coverage of protein
0.8
0.6
0.4
0.2
0.0
Homalogous superfamily
Domain
Conserved
Binding
Active
Substructure Type
Figure A.4: Distribution of protein coverage substructure. Protein coverage is defined as the total number of residues contained within the substructure divided by the length of the protein.
Homologous superfemily
0:09
3:37
6.66
(U06)
Subssructtre Type
Domain
0:00
3.37
3.64
8.09
0.09
6.02
0.64
0:04
0.09
0.01
Conserved site
0000
3:22
3.05
8.35
3.83
8:14
423
827
6.65
6:4
663
0.03
6.04
0.00
0.01
0U0
0006
0.00
0.01
0.32
Freques
Binding site
9.20
3.00
0.00
8.25
3:55
8:3
8:33
8.09
0.09
6:67
6:22
0.03
0.00
0.03
0.09
0.00
0000
0.0
0.01
0.03
Active site
8.31
3.2
8:7
0.99
Secondary structure
0.30
Figure A.5: Amino acid composition of substructure types. Each row shows the amino composition substructure type. Rows sum to 1.
0.25
Alpha helix
8.18
0.02
0.07
0.06
0.13
0.03
0.00
0.01
0:05
0.08
0.00
0.55
0.07
0.02
Beta strand
0128
0.04
0.80
0.00
0.86
MB5 D15
044
010
0.00
0.20
Secondary Structure
Helix 9:07
0.02
0.08
0.08
0.04
0.06
0.1.
0.02
0.05
0.62
0.04
0.05
0.09
0.58
0.4
0.02
0.15
Frequency
Helix 5
9.08
9.0L
0.08
0.07
0.05
6.14
0.01
0.06
0.60
D/2
0:0
0.05
0.07
0.13
940
Helix PPII
8.07
0.01
0.04
0.66
682
8.04
0.0
0.09
0.55
8.07
0.01
0.02
0.25
803
0.05
0.08
0.06
0.85
0.0
0.02
0.10
Turn
0.06
0.01
0.08
0.65
6/94
9188
0.0 0.64 0.55 8.05 0.01 0.07 0:1
6/04
0.05
0.09
0.55
0.94
0.0 0.0
0.05
Bend
9.06
0.02
0.08
0.66
694
9.88
0.02
0.04
0:55 9.05 0.01
0.05
0.34
9.05
0.10
0.06
688
9.0 0.03
A
C
D
G
M
N P
Q
R
S
T
V
W Y
Amino Acid
Figure A.6: Amino acid composition of granular secondary structure types. Each shows the amino acid composition that secondary substructure type. Rows sum
21

---

Published as a conference paper at ICLR 2026
Homolegous superfamily
Substructure type
Domein
0.25
Conserved site
0.33
0.00
5.00
0.14
0.15
411
6.4
0.20
Binding site
0.15
0.10
Secondary structure
0.44
0.10
5,22
0.31
0.3
0.7
0:03
0.05
Hydrophabic (AILMI))
Hydrophobic (1WY)
Glycinc
(NCOS 1)
Group
Figure A.7: Amino acid composition of substructure types, with amino acids grouped by side chain chemical properties. Each row shows the amino acid composition of substructure type. Rows sum
22

---

Published as a conference paper at ICLR 2026
A.1.3
SUBSTRUCTURE DATASET FILTERING
Distinct types remaining by threshold
Homologous superfamily
Domain
Conserved site
1.0
3000
1.0
1.0
8000
600
fracton
0.6
2000
Fraction
0.6
6000
Fraction
0.6
400
Taaal Count
4000
1000
200
0.2
0.2
0.2
10
10a 10"
10a
10a
threshold
threshold
threshold
Binding site
Active site
1.0
1.0
125
Total Count
fraction
0.6
Fection
0.6
0.4
50
20
0.0
0.0
10a
10
102
10
102
threshold
threshold
Figure A.8: Inverse CDF of unique types retained given count threshold. The r--xis specifies the mini- count for substructure type be retained, the left y-axis shows the fraction of all unique types retained the given threshold, and the right y-axis shows the absolute count of unique types retained. Facets show differ- ent classes substructural elements. The vertical dashed red lines show the threshold selected for downstream substructure-tung ttuiig.
Total element occurences by threshold
Homologous superfamily
1e6
Domain
Conserved site
400000
Fraction remaring
0.8
1.0
Numbe remaining
Fraction remairing
0.8
-300000
Number remaining
0.8
150000
Number remaining
0.6
0.6
200000
100000
0.4
0.4
Fraction
100000
50000
0.2
0.2
0.2
10a
10a
Min threshold
Min threshold
Min threshold
Binding site
Active site
20000
1.0
30000
Frection remaining
0.8
15000
Number remaining
Fraction remaining
0.8
0.6
20000
10000
0.4
0.4
10000
0.2
5000
0.2
Min thresholdd
Min threshold
Figure A.9: Inverse CDF of total occurrences relained at given count cutoff. This is analogous to Figure A.8 above, but showing total occurrences of substructures rather than unique types. demonstrating that for classes like domains, the majority of annotations come from small number of domain types.
23

---

Published as a conference paper at ICLR 2026
EC
GO:BP
GO:CC
GO:MF
Localization (Accuracy)
Thermostability (Spearman's 's p)
Human PPT
Model
(AUROC)
Finxx
Binary
Subcellular
ESM-C 300M
0.688
0.307
0.416
0.429
0.871
0,703
0.648 0.660
0.917
+ST (original, 2 75)
0.761
0.325
0.403
0.488
0.879
0.681
0.933
+ST (2 25)
0.802
0.32
0.399
0.526
0.851
0.693
0.64
0.917
+ST (2 10)
0.792
0.327
0.401
0,514
0.89
0.728
0.64
0.852
Table A.1: Effect of substructure frequency cutoff on protein-level task performance. The performance of substructure-tuning is robust to the presence of rare substructures. Values in parentheses indicate the minimum number of times substructure type must occur in the overall SwissProt dataset be included for substructure- tuning.
Substructure type
Full SwissProt set
Count 2 75
Count 2 25
Count 2 10
Homologous superfamily
3511
1133
1685
2159
Domain
15868
917
1964
3506
Conserved site
748
356
573
691
Binding site
76
48
72
74
Active site
133
82
114
127
Table A.2: Number of substructure types included in the dataset at different count cutoffs. Reducing the cutoff down to 10 greatly expands the number of substructure types included in the dataset.
24

---

Published as a conference paper at ICLR 2026
Model
Frnac
Localization (Accuracy)
Rinary
Subeellulor
Thermostahility (Spearman's
()
PP1 (AUROC)
FLiPhind (Fms)
(AUROC)
bunding
Biolip
(AUROC)
cotalyric
Raslip
ESM C 300M
0,688
00.307
0.116
0.429
0.871
0.703
0.648
0.917
0.367
0.851
0.925
+ST (original)
0.761
0.325
0.403
0.488
0.879
0.681
0.660
0,933
0:111
0.866
0:910
0.789
0.879
+ST (similar sog)
0.777
0.317
0.385
0.507
0.549
0.912
Table A.3: Effect of stringent data split substructtre--tunnng performance.
A. 1.4 DATASET LEAKAGE ANALYSES
It's generally standard practice when training protein models to consider the dataset used for self- supervised learning as distinct from the downstream evaluation sets, presumably this because the sequence or structure data is considered sufficiently far removed from the labels used in downstream evaluations (e.g. GO terms, experimental fitness values, etc.). However, substructural annotations different from sequence global structure in that they can be more directly tied to protein function. For this reason, we performed a series of analyses to understand whether any of the effeets of cettrructuuettunnnn could be attributed data leakage between the substructure-tune tuning training set and the test sets of any of the evaluation benchmarks.
First, performed a direct experiment by constructing two versions of more stringent dataset splits:
"Exact match": To construct this split, we identified all proteins in the Magneton train set that were also contained in the test split of any evaluation benchmark, based on UniProt ID. We then removed all such proteins and all of their corresponding AFDB50 clusters from the Magneton train set. This removed 31,191 of the 423,88 proteins in the Magneton training split.
"Similar seq": To construct an even more stringent split, we collected the amino acid se- quences of all proteins in the test split of any evaluation benchmark, and aligned these sequences to the proteins in the Magneton train set. We then removed from the Magne- ton train set any protein with greater than 30% sequence similarity and 80% overlap with any evaluation test set protein. Alignments were performed using MMseqs2 (Steinegger & Söding 2017 Kallenborn et al. 2025) with sequence similarity defined using the fident output (command: mmsegs easy-search eval.seqs fa train_seqsii output .tguuuttsss tsv --cov-mode ing the dataset split in this manner, we ensure that protein contained in the Magneton train set has more than 30% sequence similarity to any protein in any of the evaluation benchmark sets (Figure A.10). This process removed 131,440 of the 423,885 proteins in the Magneton training split.
tmp 3). By construct-
-c 0.8 --ali i gowent-mmooeo
We then performed substructure-tuning of ESM-C 300M using active, binding, and conserved site annotations with each of these stringent training splits and evaluated the resulting models evaluation suite (Table A.3). We found that benchmark performance was largely similar to that achieved with the original training, providing evidence that data leakage does not drive any of the effects of substructure-tuning.
As further step, we leveraged mappings from Gene Ontology terms to InterPro entrie to check whether substructure-tuningg improvement on GO MF prediction could be attributed to this map- ping. In particular, we checked if the improvement predicting GO MF term after substructure- tuning correlated with the representation of its corresponding domains in the Magneton train set. We found that the performance improvement for predicting a given GO MF does not depend on its corresponding domain's 's representation in the Magneton train set, again supporting the lack of data leakage (Figure A.11 Table A.4).
25

---

Published as a conference paper at ICLR 2026
Number of test set proteins
Number of test set proteins
500
J00
400
300
200
100
0.2
0.3
0.4
0.5
0.6
0.7
0.3 similarity
0.9
1.0
0.18
0.20
0.22
0.24
0.26
0.28
0.30
0.32
0.34
Maximum
Maximum
similarity
Sequence similarittis before filtering,
Sequence similarities after filtering
Figure A.10: Distribution sequence similarities (fnn from MMseqs2) between proteins the of any evaluation benchmark and proteins in the Magneton training set. Plotted values are the maximum similarity to any Magneton training set protein.
GO:MF prediction Improvement not trivially exple ned by InterPro2GG maoping
(O:MF prediction imorovement not trivilly exple ned by InterPro2Go mapping
Foe pecoee I00 intrrmmentt eboy Borain
+300%
+209%
-.80
100%
1000
Hebe
62
103
Whether
Performance number of Magneton train damann zocura
(a) Performance improvement split inclusion only.
Figure A.l1: GO:MF prediction improvement by representation of associated domain in the Magneton train set.
26

---

Published as a conference paper at ICLR 2026
F max improvement after substructure-tuning
Mean
25th percentile
Median
75th percentile
Domains in train set Domains not in train set +16.84%
+ +17.82%
-2.66%
+5.1 11%
+18.66% +23.35%
-3.40%
+4.88%
Table A.4: GO:MF prediction improvement by domain presence training set. Improvement pre- dicting a GO:MF term after substructure-tuning does not correlate with the representation of its corresponding domains in the train set.
27

---

Published as a conference paper at ICLR 2026
A.1.5 MAGNETON EVALUATION DATASETS
Magneton contains eleven different benchmarking datasets, comprising 14 evaluation tasks. These evaluation tasks represent years of work from the scientific community, both in their original gener- ation and later processing that we build upon, and we acknowledge and thank all of those involved. The included tasks are:
Human PPI prediction. The goal of this task is to predict whether not two proteins form an interacting pair. This is binary classification task where the input is two proteins and the output is binary label indicating interaction metric is accuracy. The original dataset is sourced from Pan et al. (2010). We build off of processed data files from Su et al. (2023).
interaction. The evaluation
Gene Ontology and Enzyme Commission prediction. The goal of these tasks are to predict the Gene Ontology (GO) or Enzyme Commission (EC) annotations for protein. There are three categories of GO annotations: Molecular Function (MF), Cellular Compo- nent (CC), and Biological Process (BP), each of which captures a different aspect of protein biology and is treated as a separate benchmarking task, giving a total of four tasks. These are multilabel classification tasks where protein have multiple annotations. The eval- uation metric is Fmax. the maximum Fi score over possible thresholds. We source original data from Gligorijević al. (021))
Subcellular and binary localization. The goal of these tasks is to predict a protein's 's local- ization either within multiple cellular compartments (subcellular localization) the protein is membrane-boum or soluble (binary localization). The input is a single pro- tein and this is either a multiclass or binary classification task. The evaluation metric is accuracy. The original dataset is sourced from Almagro Armenteros et al. (2017). We build off of processed data files from Su et al. (202)).
whether
Thermostability. The goal of this task is to predict the stability of a protein under extreme temperatures. The output continuous value indicating the thermostability, and the goal is to rank-order proteins according to their experimental values. The evaluation metric is Spearman rank correlation (Spearman's p) calculated against the experimental values. The original dataset is sourced from Rao et al. (2019).
Binding residue categorization. The goal of this task is to predict whether a given residue binds three different types of ligands: metal ions, small molecules, or nucleic acids. This is residue-level multilabel classification task. The evaluation metric is Fmaxx The original dataset is sourced from Dallago et al. (02T).
Binding and catalytic site prediction. The goal of these tasks are to predict whether a given residue is part of an annotated binding or catalytic site. This is residue-level binary classification task. The evaluation metric is AUROC. The original datasets are sourced from experimentally determined structures curated by Zhhnn et al. (2024a). We build off of processed data files from Yuan et al. (0225).
Contact prediction. The goal of this task is to predict whether two residues within the same protein are "in contact" with each other, which is defined as having alpha-carbon atoms within 8 angstroms of each other in the tertiary structure. The input is a single protein of length L and the output is L x L contact map, where element i, of the contact map is the predicted probability that residues and j are in contact. The evaluation metric Precision @ L. which calculates precision over the top L most confident contact predictions where L is the protein length. This metric is further stratified into short, medium, and long- range contacts in which the possible residue pairs considered are those whose pairwise separation along the primary sequence is either in [6, 10), 112, 22]. or [24, L.J. respectively. The underlying data experimentally determined structures from PDB, originally curated by Rao et al. (2019).
Zero-shot DMS variant effect prediction. The goal of this task is to predict the effect of a single or multiple amino acid mutations on a protein's 's function. The input is the mutated sequence and the output is continuous value representing fitness. The evaluation metric is Spearman's 's against the experimentally-detemmi scans (DMS) experiments. These tasks are zero-shot in that supervised training for
fitness values from deep mutational
28

---

Published as a conference paper at ICLR 2026
variant effect prediction is performed. For each model, use the author's recommended methods for VEP. The data is sourced from Notin et al. (2023).
For all datasets, protein structures are sourced from AlphaFold DB (Varadi et al. 2022) unless otherwise specified. Accounting of samples per split in each dataset are available in Table A.5
Task
Train
Validation
Test
Number of classes
Task type
EC
14,466
1,599
1,715
538
Multilabel
GO:BP
21,470
2,393
3,394
1,943
Multilabel
GO:CC
9,793
1,118
3,394
320
Multilabel
GO:MF
22,621
2,495
3,394
489
Multilabel
Subcellular localization
8,741
2,190
2,744
10
Multiclass
Binary localization
5,473
1,335
1,728
2
Binary
Thermostability
5,020
636
1,329
N/A
Regression
Binding residue categorization
890
102
286
3
Multilabel
Binding site prediction
8,231
2,389
5,182
2
Binary
Catalytic site prediction
2,856
603
1,165
2
Binary
Contact prediction
20,653
209
40
2
Binary
Variant effect prediction
N/A
N/A
217
N/A
Regression
Human PPI prediction?
26,313
234
180
2
Binary
Table A.5: Dataset sizes (in proteins) and number of classes for cach benchmarking task.
This is a zero-shot task, hence the lack training and validation data. Samples correspond to assays, covering 2.3M mutations.
Samples correspond to pairs of proteins rather than individual proteins.
A.2 TRAINING DETAILS
A.2.1
SUBSTRUCTURE CLASSIFICATION AND TUNING
For substructure prediction modules, of the hidden layer matches that the underlying base model. We extract residue-level embeddings from the final hidden layer of the base model, and use mean pooling across a substructure's 's con- stituent residues to construct a single embedding per substructure. When training with multiple categories of substructures, we use separate prediction module for each category. The training loss is the sum of the classification losses across all categories.
single hidden-layer MIPs where the dimensionality
We train using AdamW (Loshchilov & Hutter 2019) (81 0.9, B2 0.999). learning rates of 10-3 and dropout rate of 0.1 for the prediction heads, learning rate of 10 for the base model, and EWC weight of 400 (as used by original authors). Training proceeded until convergence of validation loss. All runs used batches of 32 proteins with a variable number of substructures per protein. All training was performed using bfloat16 on one to four NVIDIA A100 GPUs
Elastic weight consolidation. Bricfly, EWC uses the diagonal of the Fisher information matrix F as weights on loss that regularizes towards the original model parameters, 0p:
L = (o) + Σ > F; (0, -
where Le is the substructure classification loss. can be estimated the beginning of training as the squared gradients of the original loss with respect to the model parameters using the training set. In our case, the original loss corresponds the training objective of the underlying model (e.g. masked amino acid prediction for ESM models, masked amino acid prediction in presence of structure tokens for ProSST or SaProt). In practice, estimated by making a single pass over the training set, running backward passes using the original loss, and averaging the squared gradients over minibatches.
While similar to Le loss, EWC has two advantages ization: (1) weights are decayed towards the original weights of the base model, (2) per par-maraetter weights are applied which correspond the importance of that parameter for the original task. We
simple L2 weight decay regular-
29

---

Published as a conference paper at ICLR 2026
Frozen base model
Full finetuning
EWC finetuning
Validation MLM loss
1.24
2.30
1.75
Table A.6: Masked language modeling loss under different finetuning strategies.
Zero-shot DMS (pearrman's p)
Binary Localization (Accuracy)
Subcellular
Substructures used for tuning
EC
GO:BP
GO:CC
GO:MF
Localization
Model
(Accuracy)
Fraee
ESM-C (300M)
N/A
0.688
0.307
0,416
0.429
0.432
0.871
0.703
EWC
Domain
0.776
0.307
0.403
0,501
0,340
0.811
0.640
No EWC
Domain 0.831
0.311
0,386
0.531
TBD
0.802
0.643
Table A.7: Effect EWC on substructure-tuning performance. Training with EWC offers good balance of maintaining improvement on function-related tasks while mitigating performance decreases on other tasks.
selected EWC due its simplicity and ease of use, the estimate of F be calculated single time and used for the remainder of training or for other training runs using the same base model, as opposed alternate methods like replay buffers. For more details, please refer Kirkpatrick et al. 2017).
Empirically, use of EWC is motivated by an increase in the masked language modeling loss and validation perplexity when performing substructure-tumm -tenttnrrettingggggg as shown in Table|A.6 Figure A2 We additionally observe that EWC offers a good balance of maintaining improvement on function- related tasks while mitigating performance decreases on other tasks (Table A.7)
7.35
2.30
235
3.75
frature
2.30
1.25
1.00
Figure A.12: Validation perplexity for finetuning ESM-C 300M on domain annotations with and without EWC.
A.2.2 DOWNSTREAM TASK TRAINING DETAILS
We use different head models for different scales of supervised downstream tasks:
Protein-level. For protein-level tasks such as GO term prediction, construct protein- level representation for each protein following author's recommendations. For ESM-2, we use the final embedding of the CLS token the protein-level embedding. For all other models, we mean pool the final hidden layer representations of all residue tokens (i.e. excluding CLS, EOS and PAD tokens). Prediction heads are then single hidden-layer MLPs with hidden dimensionality matching the hidden dimension of the underlying model.
head model consisting of a single I-dimensional convolutional layer with zero- pero-igdding and filter width 5, followed by nonlinearity and linear layer to the final output.
Residue-level. For residue-leve tasks such binding site prediction,
30

---

Published as a conference paper at ICLR 2026
Protein-proteii interaction. For protein-proteii interaction prediction, extract protein- level embeddings as above, concatenate the embeddings for the two input proteins, and pass into single hidden-layer MLP.
For contact prediction, we use the BamContact Predi ct onn from the frrmeformers Python package, which trains linear regression top of attention weights from all attention heads in the underlying model.
Contact prediction.
To train the models, we used AdamW (31 0.9, B2 0.999) with a learning rate of 10-2, weight decay of 10-2, dropout rate of 0.1, and batch size of 32. Training proceeded for a maximum of 20 epochs, selecting the best model based on validation set performance. When performing full task-specific finetuning, we use learning rate of 2 10 for the base model and scale the number of GPUs and gradient accumulation steps accordingly to maintain batch size 32. All training was performed using bfloat! 6 on one to four NVIDIA A100 GPUs.
31

---

Published as a conference paper at ICLR 2026
A.2.3
TASK-SPECIFIC FINETUNING
EC
GO:BP
GO:CC
GO:MF
Localization (Aceuracy)
Thermostability (Spearman's
Model
F Fmax
Binary
Subcellular
ESM2-150M
0.911
0.352
0.451
0.658
0.928
0.810
0.694
+ST
0.910
0.349
0.444
0.658
0.936
0.791
0.702
ESM2-650M
0.910
0.356
0.446
0.662
0.933
0.824
0.703
+ST
0.914
0.360
0.457
0.665
0.930
0.793
0.703
ESM-C 300M
0.916
0.368
0,470
0.667
0.932
0.806
0.693
+ST
0.920
0.355
0.454
0..699
0.941
0.804
0.693
ESM-C 600M
0.920
0.374
0.469
0.669
0.917
0.812
0.705
+ST
0.924
0.372
0,468
0.667
0.931
0.828
0.699
ProSST-2048
0,911
0.319
0,439
0.631
0.912
0.760
0,673
+ST
0.901
0.336
0.427
0.638
0.923
0.744
0.670
Table A.8: Evaluation task performance for models with and without substructure-tuningg and full finetuning for the downstream task. In this regime, we find that task- finetuning largely results in similar models, showing that imbuing substructural information via supervised finetuning may be brittlc in the face of aggres- sive task-specific finetuning.
A.2.4 DOWNSTREAM TASK PERFORMANCE UNCERTAINTY ESTIMATES
To assess whether substructure-tuning effect on downstream task performance, we performed 50 rounds of bootstrapping when calculating the final performance metrics. For both protein-level tasks (A.9 and residue-level tasks (A.10), we found that the performance differences observed were beyond the scale of the uncertainty estimates.
A.3 MECHANISTIC ANALYSES
A.3.1 EMBEDDING ANALYSES
We assessed the consistency of substructure representations before and after substruccure-ttningg by calculating silhouette scores of the resulting embeddings, which capture how tightly grouped the representations are for different occurrences of the same substructure type (Figure A.T3). This approach also allows us to compute silhouette scores for substructure types that were completely excluded from substructure-tuning. We additionally explored kNN-based classification method to assess zero-shot generalization to unseen substructure types (Figure 1.1.
To explore cross-scale scale consistency of different substructure classes and directly assess the utility of a shared bottleneck, we performed substructure-tunnng of ESM-C 300M using single classification head operating over of the active, binding, and conserved site substructure types. We found that using a single shared head is consistent with using separate heads in terms of performance boosts in function-centric tasks over the base model, but with varying effects on other tasks (Table A.1I).
A.3.2 GRADIENT CONFLICT ANALYSIS
The gradient conflict analysis was performed as follows:
Begin with base ESM-C 300M model (i.e. substructure--uning)
For each evaluation dataset listed in Table A.12 compute the gradient for the associated loss function for 1000 batches of batch size 32.
For substructure-tuming with active, binding, and conserved-site annotations, compute the gradient for the substructure classification objective for 1000 batches of batch size 32 (where batch size refers to the number of proteins per batch, each of which must contain at least one substructure).
Compute pairwise similarities between all gradient vectors.
32

---

Published as a conference paper at ICLR 2026
Model
EC
Finxx
GO:0C
GO:MF
Localization (Accuracy)
Binary
Suhcellalar
(Speameen's ' ())
Thermosabillit
(AUROC)
ESM2-150M
0.727 + 0.008
0.316
D403
0.416 + 0.004
0:441 + 0.008
0.869 0.006
0.604 + 0.009
0.627 0.017
0 + +0015
0,323
0/4731
0.007
0.866
0.008
0.582 0.016
0,919 1019::000000
ESM2-650M
0.755 1: 0.005
0.318
0.003
0.431 1: 0.001
0:486 1: 0.007
0,576
0.009 0.895
0.710 0.009
0.613
0.017
0.939 0.018
+ST
0.745 + 0.007
0.321
+: 0.003
0.440
+: 0.004
0.534 + 0.007
0 0.009
0.749 =: 0.008
0.018 0.655
0.955 +: 0.018
ESM-C 300M
0.688
0.007
0.307
0.033
0.416 :: 0.004
0.429 +: 0.005
0.871
0.008
0.703 =: 0 0.008
0.648 0.017
0,917 0,020
0.003 0.325
0.579
0.006
0.933 0.015
ESM-C 600M
0.701 1. 0.005
0,312
0.003
0.403 1: 0.003
0:436 1: 0.006
0.563 0.008
0.713 a 0.007
0.688 0.016
0.927 0.020
1ST
0.750 + 0.008
0.319
0.003
0.383 + 0.005
0.527 + 0.007
0,872 +: 0.008
0.635 0.009
0.667 + 0.013
0.902 + 0.022
SaProt (650M)
0.774 0.007
0.326
0.004
0.453 0.005
0.538 0.006
0.987
0.009
0.784 0.007
0.692 0.016
0.962 0.015
0.005
0.339
0.004
0.446
0.007
0.896 0.006
0.741 0.008
0.697
0.902 0.017
ProSST-2048
0.778 +: 0.007
0.317 +: 0.003
0.426 +: 0.004
0.522 +: 0.005
0.878
0.005
0.693 =: 0.008
0 0.6%9 0.034
0.925 0.024
4ST
0.791 + 0.005
0.314 0.004
0.420 + 0.004
0.367 + 0.007
0.853 + 0.010
0.683 + 0.009
0.648 0.017
0.883 04223
Table A.9: Protein-level task performance for base models and models with substructure-tuning (+ST).
Values shown as mean + standard deviation. Mean and std dev computed from 50 runs of bootstrapping.
Binding residuc
Functional site prediction
Model
Binding
Catalytic
(F (Fmax)
(AUROC)
ESM2-150M
0.379 + 0.005
0.871 = 0.001
0.910 = 0.002
+ST
0.327 + 0.005
0.852 + 0.001
0.890 = 0.002
ESM2-650M
0.366 + 0.006
0.849 = 0.001
0.912 0.002
+ST
0.362 = 0.006
0.851 = 0.001
0.927 0.002
ESM-C 300M
0.367 = 0.005
0.851 + 0.001
0.923 0.002
+ST
0.411 + 0.006
0.866 + 0.001
0.910 0.003
ESM-C 600M +ST
0.357 = 0.006
0.850 0.001
0.921 0.002
0.368 - 0.006
0.852 0.001
0.906 = 0.002
SaProt (650M) +ST
0.423 + 0.007
0.891 + 0.001
0.923 - 0.007
0.400 + 0.005
0.871 = 0.001
0.924 - 0.002
ProSST-2048
0.375 + 0.006
N/A
N/A
+ST
0.342 = 0.006
N/A
N/A
Table A.10 Residue-level task performance for base models and models with substructure-tuning (+ST),
Values shown as mean standard deviation. Mcan and std dev computed from 50 runs bootstrapping.
The final results of this analysis are shown for comparison of gradient vectors derived from batches from the same task (Table 12 and from batches from different tasks (Table 1.b
A.3.3 SUBSTRUCTURE ATTRIBUTION ANALYSIS
To understand correlation of substructure-tuned embeddings with known functional motifs, we per- formed an explainability analysis by first identifying all proteins in the GO:MF test set whose GO:MF annotations could be attributed to specific domain (accomplished via InterPro2GO map- pings). yielding 81 (protein, domain, GO:MF) triplets. For each protein, we computed a saliency map to calculate residue contributions to the prediction of the associated GO:MF term. We then calculated the total contribution of residues contained within the associated domain and normal- ized this by the total contribution across all residues in the protein. For ESM-C 300M before and after substructure-tuning we find that attributions to domain residues increased by an average of 17%, indicating increased usage of substructures for function prediction after substructure-tung tunning (Appendix Figure/ 15).
As concrete example, we provide saliency map for the protein Q15436 and the prediction of its annotated GO: 0008270 term. The GO: 000 827 70 GO term can be directly mapped to the InterPro domain 199006895 contained at positions 58 to 98 in the protein. We can observe that the untuned
33

---

Published as a conference paper at ICLR 2026
19
0.6
0.4
0.0
unscen
Domann
Iomalogncc aupertami
0.5
shouere
0.0
Term
Unseen
Setn
Untena
(a) Average silbouetle scores before and after substructure- ESM-C 300M
Active site
Bnding ste
Concerved site
1.-
0.0
0.7
0.2
Orggnsl Subtraccbretunnd
Domats
Hamologous superfamity
1.0
0.2
0.3
(b) Average silhouette scores before and after substructure tuning for SaPro 650M
Figure A.13: Average silhouefte scores of each substructure type class, before and after substructure- tuning, for both seen and unseen classes. Despite never training on examples of the unseen substructure types, substracturc-tuningg results in more consistent representations of these substructures.
Madel
FC GD:BP GD:CC
Fasx
GO
Tocalization (Anmrayy)
Binary
Subcellular
Thermostsbilty (Spearman's
(AUROC)
Hamon P99
FLIP (Fauc)
Biolip binding (AUROC)
canaltic (AUROC)
Rialip
ESM-C 300M +ST (original, seganne houls)
0.68
0.307
0.436
0,429
0.871
0,703
0.648
0.17
0.367
0.31
(1923
0.761
0:305
0.401
0.488
0.870
0.681
0.660
0.993
0411
0.866
0.10
+ST (singlchrad) hrad)
0.377
0.304
0.502
0.513
0.872
0.704
0.666
0.587
0.39
0.85
0,88
Table A.11: Efffet of using single shared classification head for substructure-tuningg The effect of using single shared head consistent with using separate heads in of performance boosts in function-centric tasks over hase model, with varying effects on other tasks.
model exhibits less attribution to the relevant domain (Appendix Figure than the substructure- tuned model (Appendix Figure A.17.
A.4 ABLATIONS
A.4.1 ALTERNATE POOLING TECHNIQUES
We initially developed our approach using mean pooling to construct substructure representations due its simplicity, but also explored additional aggregation techniques to help expand the com- pleteness our work. We note that the choice of a simple pooling over learnable aggregation was intentional: since the goal of substructure-tuning is the supervised substructure classification
34

---

Published as a conference paper at ICLR 2026
Jchive
tedingate
Consived
02-
0.0
10-
68-
62-
Figure A.14: Average accuracy of KNN-based classification of each substructure class using embeddings, before and after substructure-tuning, for both seen and unseen classes.
Functional site prediction (hinding)
Functional site prediction (catalytic)
Binding residue
Substructure elassification
Gradient cosine similarity
0.957 0.004
0.956 + 0.004
0.961 + 0.007
0.964 0.004
0.973
0.022
0.095 + 0.077
(a) Within task gradient similarity (mean 1: std deviation)
Functional site prediction (binding)
Functional site prediction (catalytic)
Task
GO:MF
Binding residue
Gradient cosine similarity -0.006 + 0.047 0.003 0.027 0.022 0.056 0.002 0.055 -0.005 0.038
Task gradient similarity with substructure classilication gradient (mean 1: std devistion)
Table A.12: Gradient conflict analysis for substructure-tuning. These results suggest that the task-specific effects substructure- bobstructure-tuning is due simple global misalignment between the substructure objective and certain downstream tasks.
problem tune the parameters of the underlying protein model, wanted encourage learning to occur within the base model rather than in parameterized pooling module.
To investigate this, performed substructure-tuning using max pooling learnable attention pooling. Both of these experiments performed substructure-tuningg on ESM-C 300M using active, binding, and conserved site annotations. We find that substructure-tuning is robust to the exact method used for pooling residue-level embeddings to substructural embeddings (Appendix Table A.13).
A.4.2 ADDITIONAL. MODELS
We additionally conducted direct comparison of substructure-tuninn methods that use global structure to tune sequence models would help contextualize our work. To ensure a fair comparison, have integrated ESM-S into Magneton and evaluated it our benchmark suite (Appendix Table A.14).
We find that substructure-tuning compares favorably ESM-S, with generally larger improve- ments on function-related tasks and smaller performance drop-offs in other tasks. We also note that cnsstructure-tuingg also useful for models that already incorporate global structure information, something which has not been shown for ESM-S's methodology.
35

---

Published as a conference paper at ICLR 2026
Domain-tun ing increases attribution to
functional domains MF prediction)
Model
Onginal
Demain-tuned
0.7-
Demsity
0.5
0.0
0.1
0.0
3.0
Fraction of sallency map attributed to residues in domain
Figure A.15: Saliency map indicating attributions to domain residues during GO:MF predictions before and after domain-tuning.
M T Y E F Q Q N E B D V W N W P SRRLEAA T M V V P A FLFT C D Y A E L L Q F
R
L P
Q Y E DOM START V LCSRTTI C A
N L Q D L Q A S K S Y
Q S
[OM END
E
E Y V V L
R
G
P Q M LI
VV D C D M
D E
L
K S M Q M
L
S L L P P T A L V L
R M V O H E G C E G Q P S
V G T K
D L
S A K Q L Q E M L G
S K V P L T O A T R G O V Q
S G D H
N R F Q S A V G L L E C
Q
K D M N L T D L E O D P W P V P O G K P V V G G
N D DI D A K Q F Q R V F K D M H Q G L TEEEEIGGGGGTTTTTE E G A L W K A R F T F S Y Y
G CAW.LLDTTTTEEEEEECCCCCCCCCCCCCCC
R K
G
E K V D Y C
R
S W
L
G
T
P
C S L N K P C S H E E G G L R L A S A K
G
E
V ALR W D TTT.PIPPOOOOGOOOOOOOOOOOOOOOOOTO SHWHHAAAAAATTTTTTTSSSSSSSOOOSSAT
V A D A O O
P
I R N E
O N
K. D Q V F N S E S S Y Y R H H F M
F
Q F M O D L T S L M
FHL L R R S S F
Q G F
P E
ILY A Y S F S
G P P P Y L D S S S I L A D R I L L M D T E F Q I I
H S
TIA Q R K
D M P E R L
R H L Q A Y A V S A
Q ILL H
R
P R I PIL
H
G G S O
S
Q T H N
Q
E
O V F M
Figure A.16: Example saliency map before substructure-tuning. IDOM SCART] and the beginning and end of the domain within the sequence respectively.
IDO END
denote
M T Y E F Q N E E R D V F W N W P S L A T R M V VVV A P P
R P D
L
O
Y P DOM START V L S R
C
R A
V L L N L Q A L K S Y P
L
C E L L P Q
W A S S I
K
N
F
Q
O PPPS
DOM END V T
P Q G
M P L L G R M V H E G C
M E
K E S M M S S L L P P T A
G
F R G T K D L S A K Q Q E
M L G L S V L A T R G O V O Q V T F
NRR L Q P O K D M N L G
O G O WWW G G TAA.AAACOOOOOANGGGGGTTTTTTEEEEET H
A V L E L K T I V D Y A C A L DAAALDOTIIIA K Q T Q R V F T M H F C S L N S K G V ENNEEEEEEINN V Q H N A P I O N G O WWW.TEETTTTTTGTTTTTCCCCCCCCCCTTT L L FHL R R S S F L Q Y F
P
V R N E T S Y F M S E Y Y R H H F M R O D L T Q L I M Q P ILY A Y S F S G P P E Y Y L L S S S I A D R I L L M D T F Q I L Y H R F Y A Q G
W A D
D
V L R W L D C
D
S
G E
A Q W R K S G Y O D M P E Y E N F
L L Q A V D A Q E L
P Y
Q M
A
L K V N
PILT D V S
K
S
Figure A.17: Example saliency map after substructure-tuningg (DOM_START) and [DOM.END] denote the beginning and end of the domain within the sequence respectively.
36

---

Published as a conference paper at ICLR 2026
EC
GO:BP
GO:CC
GO:MF
Localization (Accuracy)
Thernostability (Spearman's p)
Human PPI
Model
Fmax
Binary
Suhcellular
(AUROC)
ESM-C 300M
0.688
0.307
0.871
0.703
0.648
0.917
+ST (original, mean)
0.761
0.325
0.403
0.488
0.879
0,681
0.660
0,933
+ST (max)
0.777
0.304
0.392
0.513
0.872
0.704
0.666
0.887
+ST (attention)
0.778
0.317
0.392
0.505
0.815
0.675
0.643
0.89
Table A.13: Effect of pooling method on substructure-tuning performance. Substructure-tuning is robust to the exact method used for pooling residue-leve embeddings to substructural embeddings.
BC
GO:BP
GO:CC
GO:MF
Localization (Accuracy)
Thermastability (Spearman's
Human PPI (AUROC)
Binding residue (F...)
Functional site prediction
Model
Faux
Binary
Subcellular
Binding
Catalytic
(AUROC)
ESM2-150M
0.727
0.316
0.116
0.111
0.869
0.691
0.627
0.933
0.379
0.871
0.910
+ST
0.742
0.324
0.415
D.473
0.866
0.679
0.582
0.919
0.327
0.852
0.890
ESM-S 150M
0.760
0.315
0.373
0.460
0.866
0.677
0.631
0,917
0,352
0.571
0.885
ESM2-650
0.755
0.431
0.486
0.876
0.710
0:643
0.939
0.366
0.849
0.912
+ST
0.745
0.321
0.440
0.534
0.895
0.749
0.362
0.927
ESM-S 650M
0.794
0.327
0.422
0.497
0.896
0.679
0.653
0.912
0.363
0.876
0.902
S-PLM (650M)
0.565
0.284
0.375
0.482
0.910
0.622
0.682
0.911
0.358
0.500
0.500
Table A.14: Comparison substructure-tuning with global structure tuning methods. Substructure- tuning compares favorably to ESM-S and S-PLM, with generally larger improvements on function related tasks and similar performance drop-off's in other tasks.
37


Reviews:
[{"review_id": "wX0C8C4y1d", "reviewer": "Reviewer_MQMA", "summary": "The paper introduces Magneton, a new environment that incorporates protein substructures into protein language models. It contains (1) a large-scale dataset annotated with million-scale substructure annotations, (2) a suite of evaluation tasks,  and (3) a training method called substructure-tuning, a supervised fine-tuning method for distilling substructural information into pretrained models. Substructure-tuning shows substantial improvement on models’ ability to represent protein function.", "questions": "Please refer to the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Compiling curated substructure annotations at scale and providing packaging code + benchmark tasks is a valuable community resource. \n\n2. The suite of 13 tasks spans from residue, substructure, protein, and interaction levels. \n\n3. The paper reports consistent improvements on function-related tasks after substructure tuning, suggesting substructural priors are useful.", "weaknesses": "1. The authors restrict to substructures that occur at least 75 times in the SwissProt dataset, corresponding to retaining only the top 10% most frequently occurring domains. Although the histogram in Appendix A supports this, more experiments on this may be beneficial. Also, this practice might limit claims about the general utility of rare/novel substructures.\n\n2. Have you tried other structure pooling techniques more than the mean pool?  E.g. learnable aggregation, or attention?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IIvhPY6Gw5", "reviewer": "Reviewer_DsWe", "summary": "This paper presents Magneton, a framework that brings protein substructure knowledge into representation learning in a systematic way. The authors build a large annotated dataset with over half a million proteins and around 1.7 million substructures from six categories. They propose a substructure-tuning method that fine-tunes pretrained protein language models on multi-scale substructure classification and test its impact on 13 downstream tasks at different levels, from residues to protein interactions. The results show consistent improvements on function-related tasks such as EC, GO-MF, and thermostability, while the effects on localization and residue-level predictions are neutral or slightly negative. Overall, the work shows that substructure information complements global structural signals and offers a reproducible benchmark for studying hierarchical protein representations.", "questions": "Mechanistic validation:\nCould the authors include attention-map or embedding-similarity analyses (for example, t-SNE or clustering of substructure embeddings) to check whether substructure-tuning actually makes the model group residues or motifs by function? Comparing pre- and post-tuning representations on catalytic or binding site residues would make the mechanism more convincing.\n\n\nGeneralization and transferability:\nTo show robustness, the authors could run few-shot or zero-shot transfer experiments — for instance, testing on unseen protein families or on new substructure types not used during tuning. Measuring how performance drops as family divergence increases would make the generalization claims stronger.\n\n\nFunctional specificity of catalytic sites:\nInstead of treating all catalytic sites as one binary label, the authors could run an additional experiment focusing on specific catalytic functions (for example, hydrolases vs oxidoreductases). This would test whether substructure-tuning helps the model distinguish between different catalytic mechanisms rather than just recognizing “catalytic vs non-catalytic” regions.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper takes a fresh approach to protein representation by explicitly modeling mid-level substructures that connect sequence and overall structure. While earlier work has explored structural supervision, few have looked at substructure information at this scale or systematically tested its impact. The authors run solid and wide-ranging experiments across several model families, including ESM2, SaProt, Cambrian, and ProSST, covering six types of substructures and thirteen downstream tasks. The dataset is clearly described and easy to reproduce, and the figures and tables make the framework intuitive to follow.\n\nOverall, the work is clearly written and well-executed. It provides a practical and scalable benchmark that shows how substructure-level fine-tuning can bring meaningful gains on function-related tasks, even for models that already use 3D information. The results highlight the value of adding explicit substructure knowledge to protein models and point toward new directions for improving interpretability and representation learning in protein science.", "weaknesses": "Lack of theoretical motivation:\nThe paper shows that substructure-tuning works, but it doesn’t really explain why. There’s no clear reasoning or framework to describe why it helps with function prediction but hurts residue-level performance. The discussion feels more like observation than real analysis.\n\nUnbalanced and oversimplified supervision:\nThe six types of substructures are not only uneven in number—some, like catalytic or binding sites, are much rarer—but also differ in biological meaning. Many of these categories, such as catalytic sites, cannot be treated as simple binary labels because they involve diverse biochemical functions rather than just “active vs inactive.” Similarly, tasks like PPI prediction or contact prediction represent complex relational signals that go beyond binary classification. The paper filters and simplifies these cases, but it doesn’t assess how this affects the model’s ability to capture functional diversity or relational structure.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dZ1ime5BwU", "reviewer": "Reviewer_RdsB", "summary": "Protein models are typically supervised with individual residues or full structures. However, there are various labeled substructures of active sites, binding sites, etc. that have not been previously used to train models. This paper introduces a dataset and framework to train sequence or structure-sequence models on these datasets. They show that the fine-tuned models improve performance.", "questions": "1. Beyond Table 1, what does the dataset consist of? What do the unique types consist of?\n2. How do the authors address potential data leakage between the training data present in the Magneton substructure dataset and the evaluation datasets? Are there sequence similarity cutoffs to filter the training set to ensure that the training and testing proteins are sufficiently different?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Substructures are intuitive abstractions that captures various levels of protein modeling that are not immediately apparent from either structure or sequence.\nThe dataset developed by processing Swissprot is large and diverse, and is often overlooked in training protein models.", "weaknesses": "While there is some analysis of the dataset in Table 1, it is insufficient for a paper which proposes a dataset.\nThe results are impressive but raises concerns as to how much of the test set's labels are related or derived from the SwissProt annotations.\nThe authors finetune the model using elastic weight consolidation (EWC). However, it is not clear empirically how much EWC affects downstream performance.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mydzJOanmN", "reviewer": "Reviewer_w8Xw", "summary": "The paper introduces Magneton, an integrated environment for developing substructure-aware protein representation models. Magneton comprises three components:\n\n(1) a large-scale dataset of 530 k proteins annotated with 1.7 M substructures spanning 13 k types;\n\n(2) a training framework that incorporates these substructures into existing sequence- or structure-based encoders; and\n\n(3) a benchmark suite of 13 tasks covering residue-, substructure-, protein-, and interaction-level evaluations.\n\nBuilding upon this environment, the authors propose substructure-tuning, a supervised fine-tuning strategy that distills substructural information into pretrained protein models. Experiments across six state-of-the-art encoders (150 M–650 M parameters) show consistent ≈5 % improvements in function-related tasks (e.g., EC and GO prediction) and demonstrate that substructural signals complement global structural information.", "questions": "1. Could the authors discuss whether hierarchical or graph-based encoders (e.g., multi-resolution GNNs) might integrate substructures more stably than fine-tuning alone? A comparison would help isolate the source of gains.\n\n2. Have the authors evaluated whether the tuned models can generalize to rare or unseen motifs not present in the training subset? Such analysis would demonstrate the learned “substructure awareness” beyond supervised classes.\n\n3. Since different substructure classes use separate heads, how correlated are their learned embeddings? Could joint attention or shared bottlenecks yield better cross-scale consistency?\n\n4. One potential advantage of substructure-aware representations is improved interpretability. Can the authors provide examples where substructure-tuned embeddings correlate with known functional motifs or mechanistic insight?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper identifies an under-explored gap—modern protein language models neglect conserved substructures, despite their well-known biological importance. The proposed Magneton environment and “substructure-tuning” paradigm offer a novel, systematic way to integrate decades of curated biological knowledge into modern representation learning.\n\n2. The dataset construction and methodology are rigorous. The authors curate and filter substructures from major biological databases (Pfam, InterPro, DSSP, SwissProt), perform consistent train/validation/test splitting, and benchmark on 13 tasks. The experimental setup spans multiple model architectures and modalities, lending credibility to the reported gains.\n\n3. The manuscript is well-written and logically structured. Figures 1–3 clearly illustrate the motivation, architecture, and effects of substructure-tuning.\n\n4. The work lays a foundation for multi-scale protein modeling by linking residue-, motif-, and domain-level representations. This contribution is both scientifically meaningful and practically useful, likely to influence future developments in protein representation learning, function prediction, and design.", "weaknesses": "1. Substructure information is only injected through supervised fine-tuning; the model architecture itself remains unchanged. As the authors note, this approach can be brittle under task-specific fine-tuning and may not fully exploit hierarchical dependencies. Exploring architectural biases (e.g., graph-hierarchical encoders) would strengthen the contribution.\n\n2. The dataset filters out substructures with < 75 occurrences. While this simplifies training, it biases learning toward frequent structural motifs and limits generalization to rare but functionally critical substructures.\n\n3. Substructure-tuning improves global functional prediction but sometimes degrades residue-level metrics. This trade-off warrants deeper analysis—e.g., are low-level features being overwritten, or are substructure objectives misaligned with certain downstream signals?\n\n4. The paper could better contextualize gains against concurrent structure-aware fine-tuning strategies (e.g., S-PLM, ESM-S). A direct quantitative comparison or ablation with those would clarify the incremental benefit of substructure-tuning.\n\n5. While performance tables are comprehensive, confidence intervals or variance over multiple runs are missing, which is important given modest absolute improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)