Customizing Spider Silk: Generative Models with Mechanical Property Conditioning for Protein Engineering
Authors: Neeru Dubey
Elin Karlsson
Miguel A. Redondo
Johan Reimegård
Anna Rising
Hedvig Kjellstrom
Venue: Accepted by TMLR ()

Abstract: The remarkable mechanical properties of spider silk, including its tensile strength and extensibility, are primarily governed by the repeat regions of the proteins that constitute the fiber, the major ampullate spidroins (MaSps). However, establishing correlations between mechanical characteristics and repeat sequences remains challenging due to the intricate sequence–structure–function relationships of MaSps and the limited availability of annotated datasets. In this study, we present a novel computational framework for designing MaSp repeat sequences with customizable mechanical properties. To achieve this, we developed a lightweight GPT-based generative model by distilling the pre-trained ProtGPT2 protein language model. The distilled model was subjected to multi-level fine-tuning using curated subsets of the Spider Silkome dataset. Specifically, we adapted the model for MaSp repeat generation using 6,000 MaSp repeat sequences and further refined it via cross-validation on 592 repeats associated with experimentally determined fiber-level mechanical properties. Our model generates biologically plausible MaSp repeat regions tailored to specific mechanical properties, while also predicting those properties for given sequences. Validation includes sequence-level analysis, assessing physicochemical attributes, the expected distribution of key motifs, and secondary structure compositions. A correlation study using BLAST on the Spider Silkome dataset and a test set of MaSp repeats with known mechanical properties further confirmed the predictive accuracy of the model. This framework advances the rational design of spider silk-inspired biomaterials, offering a versatile tool for engineering protein sequences with tailored mechanical attributes.

Published in Transactions on Machine Learning Research (07/2025)
Customizing Spider Silk: Generative Models with Mechani- cal Property Conditioning for Protein Engineering
Neeru Dubey
needub@kth.ss
Division of Robotics, Perception and Learning
KTH Royal Institute of Technology
Elin Karlsson
cinn.b..arlssomsssu...k
Department Animal Biosciences
Swedish University of Agricultural Sciences (SLU)
Miguel Angel Redondo
miguuel.ongee.reeddddoodddb...e
Science for Life Laboratory (SciLifeLab)
Uppsala University
Johan Reimegård
johan. bokem..eiimegond@@eciiifld....
Science for Life Laboratory (SciLifeLab) Uppsala Universily
Anna Rising
Anna. Risinggssluu...mmm
Department of Aniimal Biosciences Swedish University of Agricultural Sciences (SLU)
Hedvig Kjellström
hedvig@kth.ss
Division of Robotics, Perception end Learning
KTH Rogal Instätute of Technology
Reviewed on OpenReview:
https: // openreview. net/ forum? id= 37VSapXDKK
Abstract
The remarkable mechanical properties of spider silk, including its tensile strength and ex- tensibility, are primarily governed by the repeat regions of the proteins that constitute the fiber, the major ampullate spidroins (MaSps). However, establishing correlations between mechanical characteristics and repeat sequences remains challenging due to the intricate relationships of MaSps and the limited availability of annotated datasets. In this study, we present a novel computational framework for designing MaSp repeat sequences with customizable mechanical properties. To achieve this, we developed a lightweight GPTT-based generative model by distilling the pre-trained ProtGPT2 protein language model. The distilled model was subjected to multi-level fine-tuning using curated subsets of the Spider Silkome dataset. Specifically, we adapted the model for MaSp repeal generation using 6,000 MaSp repeat sequences and further refined it via cross-validation on 592 repeats associated with experimentally determined fiber-level mechanical properties. Our model generates biologically plausible MaSp repeat regions tailored to specific me- chanical properties, while also predicting those properties for given sequences. Validation includes sequence-level analysis, assessing plysicochemical attributes, the expected distribu- lion of key motifs, and secondary structure compositions. correlation study using BLAST on the Spider Silkome dataset and test set of MaSp repeats with known mechanical prop- ertics further confirmed the predictive accuracy of the model. This framework advances the rational design of spider silk-inspired biomaterials, offering versatile tool for engineering protein sequences with tailored mechanical attributes.
quence-strrnctur-fmnee

---

Published in Transactions on Machine Learning Research (07 (025))
1 Introduction
Recent advancements in protein design, particularly the integration of artificial intelligence (AI), have signif- icantly enhanced ability to engineer proteins with desired functions. Researchers have used deep learning techniques to improve the design of de novo proteins, achieving tenfold increase in the success rates of target binding (Bennett et al. 2023). These innovations underscore the transformative potential of AI in protein engineering, paving the way for novel therapeutic interventions and biolechnological applications Khakzad et al. 2023
In parallel, the growing demand for sustainable, non-petroleum-basee fibers has intensified interest in bio- derived alternatives. Spider silk, known for its exceptional mechanical properties and biodegradability, presents promising candidate. However, efforts develop artificial spider silk are hindered by limited knowledge of how the amino acid sequence of spider silk proteins (spidroins) influences the mechanical properties of the fibers. In this context, Al-driven protein engineering offers a powerful tool for designing spidroins that can be spun into fibers with customized performance characteristics.
Spiders spin up to seven different silk types, that all are composed of spidroins (Peakall 1969 In this work, focus on major ampullate silk, or dragline silk, renowned for exceptional mechanical properties tensile strength comparable to steel (up to 1.3 GPa) with extensibility rivaling rubber (>30%) (Vollrath & Knight 2001). This unique combination yiclds toughness exceeding both steel and Kevlar (Gosline et al. 1999), making spider silk particularly altractive for applications ions ranging from biomedical sutures to high- performance textiles (Bourzac 2015
Amorphous
B-sheet crystalline
Strand spider
Core
Fibris
Spider web
region
cow
N-terninll domain
C-terminal
Glycine rich region (GGG0036GGYGGGA)
Poly-Alanine block (AAAAAAAAA)
Figure 1: Hierarchical representation of the spider dragline silk fiber architecture, highlighting the schematic image of MaSp showing various sequential elements.
The molccular structure of MaSps comprises three primary regions: the globularly folded N-terminal and C-terminal domains, which are flanking the repetilive core region (Figure D The repetitive core region is believed to the main contributor to the fiber mechanical properties, and in MaSps, this region is generally made up of poly-Ala blocks that are alternated with Gly-rich regions (Guerette et al., 1996). In the polymerized form, in the fiber, the poly-Ala blocks are predominantly arranged in nano-sized 8 sheet
crystals structures that contribute the tensile strength of the silk fiber (Hijirida et. al. 1996 Lewis 1992
2

---

Published in Transactions on Machine Learning Research (07/2025)
Yarger et al. 2018b). The poly-Ala 3-shect crystals are embedded in an amorphous matrix formed by the Gly-rich repeats, which is related to the extensibility of the fiber (Gosline et al. 1984). Although often referred to as amorphous, the Gly-rich repeats also form specific conformations like 31-helices, B-turns and 8-spirals (Hayashi & Lewis 1998
Hinman & Lewis
1992
Van Beck et al. 2002).
Native major ampullate silk fibers from different spider species display large variability mechanical prop- erties. To elucidate the source of this large variability, |Aakkaa al (2022). undertook the significant challenge of sequencing the transcriptome of 1098 species and simultancously determining the mechanical properties of major ampullate silks from 446 of these species. The results showed that there is large interspecies difference in terms of mechanical properties. For example, the tensile strength of the major am- pullate silks varies between 0.17 and 3.3 GPa (Arakawa et al.) 2022). However, strong correlations wetween the amino acid sequence motifs in the repeat regions of the MaSps and the fiber mechanical properties could not be found (c.f. section 2.1
Efforts to produce artificial spider silk have largely focused on replicating natural silk spinning processes through variety of engineered approaches (Ferruz & Höcker 2022 Koeppel & Holland (017). These methods majorly involve expression of engineered mini-spidroins in a heterologous host and subsequent spinning using an artificial spinning device (Schmuuk et al.) 2024). Although these methods have significantly advanced synthetic production, they often fail to provide a scalable means to tailor silk properties for specific applications. This limitation further underscores the need for computational approaches to predict and design sequence property relationships effectively.
We propose multi-stage framework for (1) generating MaSp repeat sequences conditioned mechanical properties, and (2) prodicting properties from given repeats. In contrast to prior work focused on full-length proteins (Lu et al., 2024), our model targets the repetitive core regions of MaSp proteins, which are central to fiber mechanics. The approach begins with distillation of ProtGPT2 (Ferruz et al.) 2022) into a compact student model, SpiderGPT, trained on 100,000 spider proteins from UniRef50 (The UniProt. Consortium 2024). SpiderGPT is then fine-tuned in two stages using the Spider Silkome dataset (Arakawa et al.) 2022):
Level 1 uses 6,000 MaSp repeats to learn motif patterns, while Level applies 5-fold cross-validation on 592 annotated sequences to learn sequence property relationships. This hierarchical setup enables SpiderGPTT to model both generative and predictive tasks with domain specificity and generalization.
To evaluate the model's effectiveness, we employed two distinct datasets: test set of 185 instances sampled through cross-validation from the original 592-instance Spider Silkome dataset to assess self-consistency, and a BLAST set curated to determinc sequence novelty against a broader protein sequence database. Our evaluation followed a comprehensive two-level analysis approach. At the sequence level, we analyzed key properties including molocular weight, instability index, isoclectric point, and the distribution of essential motifs (GGX, poly-Ala, YGQGG, and SV). For structural validation, we employed a secondary structure prediction tool to verify the composition of a-helices, 8-strands, and random coils characteristic of MaSps.
The model's ability to estimate the mechanical properties of silk fibers for given MaSp repeat was evaluated by correlating generated and reference properties on a test set. The analysis yielded a cosine similarity of 0.9465 between the trend curves of generated and reference properties, indicating a strong alignment in predictive performance.
By combining generative modeling with biological validation, this work offers a robust computational frame- work for designing MaSp sequences with customizable mechanical properties. The findings hold promise for the advancement of synthetic biomaterial development and pave the way for applications in medicine, textiles, and engineering.
Our main contributions include:
novel computational framework that integrates knowledge distillation and multi-level finc-tuning to generate MaSp repeat sequences with targeted mechanical properties, addressing the challenge of limited mechanical property data.
3

---

Published in Transactions on Machine Learning Research (07/2025)
A dual-purpose model capable of both generating MaSp repeats based desired mechanical prop- erties and predicting mechanical properties from given sequences, offering flexibility for both design and analysis tasks.
A comprehensive validation methodology that combines sequence-level analysis, structural predic- tion, and mechanical property correlation to ensure biological plausibility and functional relevance of generated sequences.
Empirical demonstration of the model's effectiveness through statistically significant correlations be- tween predicted and reference properties (cosine similarity of 0.94), while maintaining key structural motifs characteristic of spider silk proteins.
A potential solution for the syuthesis of sustainable biomaterials by providing a scalable approach for designing synthetic spider silk protcins with customizable mechanical properties.
The remainder of this paper is organized as follows: Section 2 presents comprehensive review of the literature that covers spider silk sequence-property relationships, protein design using machine learning, and current approaches
modeling mechanical properties of spider silk. Section 3 describes our proposed methodology, including the model architecture and multi-level fine-tuning strategy. Section ! presents our experimental setup, dataset details, and training procedures. Section 5 discusses our results, including self- consistency assessment, sequence generation analysis, mechanical property predictions, and ablation studies. Finally, Section T concludes with a discussion of potential applications and future rescarch directions.
2
Literature Review
2.1 Spider Silk: Sequence-property Relationships
Major ampullate silk is widely recognized as a benchmark for high-performange biomaterials due to its unique combination of lensile strength, extensibility, and toughness (Vollrath & Knight, 1999 Gosline el. al., 1999). The fiber is primarily composed of major ampullate spidroins (MaSps), and to date, five distinct MaSp classes (MaSpl MaSp5) have been identified (Schmuck et al. 2024). Assignment of a MaSp to a specific class is typically based on clustering of its terminal domains in phylogenetic analyses and the presence of characteristic amino acid motifs within the repeat region.
A typical MaSp sequence comprises three main components: an N-terminal domain, a repetitive core region, and a C-terminal domain, as illustrated in Figure ! The terminal domains are highly conserved and shared between different spidroins that give rise to fibers of very diverse mechanical properties. This conservation suggests that terminal domains are primarily involved in spidroin storage and polymerization rather than in determining fiber mechanics. Both N- and C-terminal domains have been extensively characterized in prior work (Andersson al., [014 Askarieh et al., 2010 Eisoldt et al. 2012 Sede al. (202)). To the best of our knowledge, direct interactions between terminal domains and the repeat region have not been reported. Moreover, evidence from the literature consistently supports the conclusion that the mechanical properties of spider silk fibers are governed predominantly by the repetitive core domains of the spidroins (Anton et al.) 2017 Bowen et al.] [2018: Keten & Buehler] [2110] Gossliee et al.] [999| Kono et al. 2021a Vollrath & Knight 2001 Schmuck et al. 2024). Consequently, our design strategy focuses exclusively on the repeat region, targeting the sequence elements most. directly responsible for material performance. While we do not exclude the possibility of terminal-ropes intcractions, our methodological emphasis remains on the core motifs most relevant to mechanical function.
Analyses of spidroin sequences in the Spider Silkome database have revealed several recurring amino acid motifs within the repeat regions of MaSps (Arakawa et al. 2022). Notable motifs include poly-Ala, GGX,
YGQGG, SV, GPGXX, QQ, and AGQG (Arakawa et. al. 2022 Keten & Buehler 2010 Malay et al. 2017
Craig et al. 2020 Nakamura et al.) 2024bb Kono et al. 2021b). However, attempts to correlate individual
motif occurrences with fiber mechanical propertics have yielded weak or no relationships, with Pearson correlation coefficients generally below 0.6 (Arakawa et al. (022).. Among the few exceptions are the YGQGG motif, which shows positive association with toughness, and the SV motif, which is negatively correlated
4

---

Published in Transactions on Machine Learning Research (07 (07/)025)
with this parameter. Other frequently occurring motifs such poly-Ala, GGX, QQ, and AGQG do not show strong individual correlations with fiber mechanics (Arakawa et al. 2022.. These findings underscore the complexity of the sequence function relationship in MaSps, which likely arises from synergistic interactions among multiple motifs rather than from the presence of any single motif.
In this study, we leverage recent advances in artificial intelligence to better understand the intricate sequence- to-property relationships within MaSp repeat regions. Through data-driven modeling, we aim uncover latent patterns that govern the mechanical performance of spider silk fibers.
2.2 Protein Design Using Machine Learning and Generative Models
Deep learning has increasingly become driving force in protein design, with generative models enabling data-driven exploration of protein sequence space. Among these, language model-based architectures have gained particular prominence by framing protein sequences analogously to natural language. Models such as ProtGPT2 (Ferruz et. al.) 2022), ProGen (Madani et al., 22200,, and ProI (Brandes el al. 2022) leverage transformer- based architectures trained on large-scale protein databases to generate biologically plausible sequences and predict structural or functional features.
These approaches have demonstrated success in designing enzymes Maadani et. al. 2020). antimicrobial pep- tides (Das et al., 2021), and antibodies (Ruffolo et al., 2021 often surpassing traditional directed evolution methods in speed and diversity. More rocently, models such as ProteinGAN (Repecka et al. 2021 and mod-
els conditioned on property constraints [Nijkamp et al. 2023 Lu et al. 2024 have emphasized functional design by guiding sequence generation toward specific biochemical or biophysical properties.
Structure-informeet models like AlphaFold2 (Jumper et al. 2021 ESMFold {Lin et al., 2022), and diffusion- based frameworks (Watson et al.] 2023) Anand al.) 2022) have further enhanced generative design by incorporating folding and energy priors, thus supporting the generation of not only functional but also structurally stable proteins.
In the context of spider silk proteins (spidroins), which are characterized by repetitive motifs and mechanical performance, these generative frameworks offer promising capabilities. However, sequence design aligned with fiber-level mechanical properties- such as toughness or strength- remaiis underexplored. Our work addresses this gap by conditioning sequence generation on mechanical property profiles, advancing property- aware generative modeling for hierarchical and repetitive proteins such as MaSp repeats.
2.3 Modeling Mechanical Properties of Spider Silk
Studies investigating the relationship between protein sequence and mechanical properties have primarily relied sequence analysis and molecular simulations. For instance, (Tokareva et al. 2013) highlighted the importance of GGX and poly-Ala motifs in achieving spider silk's extensibility and tensile properties. Secondary structure prediction tools like PSIPRED (McGuffin et al. 20000) have been widely used to pre- dict a-helices, 3-strands and random coil arrangements, which are critical for understanding silk mechanics [Bcchnn & Jones 2019). However, these approaches are often retrospective and do not enable forward design of sequences with desired properties. Efforts to model mechanical properties using machine learning have shown promise. For instance, Kim et al. (2023) utilized neural networks predict silk's mechanical properties based amino acid composition. While such methods enhance understanding of sequence-propen rela- tionships, they lack the generative capability required for designing novel sequences. Recent advancements in hybrid AI techniques have shown promise in overcoming these limitations, particularly for engincering silk-like proteins with desired mechanical characteristics (Lu et al.) 2024 Ni el al. 2023 Pazio et al.) 2023). Notably, Lu et al. (2024) introduced a generative modeling approach for spidroin sequence design, enabling the crcation of synthetic spider silk proteins tailored to target mechanical propertics. However, key limi- tation of this approach is its lack of emphasis on MaSp repeat regions, which are the primary determinants of silk's mechanical behavior.
5

---

Published in Transactions on Machine Learning Research (07/2025)
2.4 Our Proposed Approach in Context
Existing studies have made significant progress in predicting and designing silk proteins; however, they often overlook the critical role of the repeat regions in MaSp that govern mechanical properties. In this study, we explicitly remove the terminal domains from MaSp sequences and focus our analysis solely on the repeat region.
Although predictive models have contributed to understanding sequence-propent relationships, they typi- cally lack generative capabilities. Conversely, existing generative approaches do not focus specifically on the functional repeats of MaSp. To bridge this gap; we introduce multi-level fine-tuning strategy tailored for MaSp repeat generation. In the first stage, a specialized model is trained to learn the structural and com- positional features of MaSp repeats, ensuring biological plausibility in generated sequences. In the sccond stage, we fine-tune the model on annolated sequences with mechanical property labels, establishing a direct correlation between sequence motifs and their functional outcomes. This enables controlled generation of MaSp repeats with tunable mechanical characteristics.
By integrating domain-specific fine- fine-tuning with generative modeling, our method extends beyond existing approaches, providing a more precise and biologically grounded framework for spider silk protein design. The ability to design spider silk proteins with tunable mechanical properties has significant implications for sus- tainable material innovation. Applications include eco-friendie coo-riendll,, biodegradable textiles and high-performanee biomedical materials such as tissue scaffolds and drug delivery systems (De Giorgio et al. 2024). More broadly, the combination of computational and experimental tools protein engincering promises to accel- erale the discovery of novel biomaterials and advance the field of biomimetic material design.
3 Proposed Pipeline: A Multi-Stage Fine- TuninT Framework
The target dataset, which maps MaSp repeats to their mechanical properties, is extremely limited in size, presenting a major challenge for training generative deep neural networks. To address this limitation, we adopt a multi-stage fine-tuning strategy that enables the model to perform two complementary lasks: (1) generating novel MaSp repeat sequences conditioned on desired mechanical properties, and (2) predicting mochanical propertics from a given MaSp repeat.
The training pipeline is organized into three stages, outlined in Figure 2 and is designed to progressively distill and specialize domain knowledge across tasks and data scales. While this hierarchical strategy is tailored to protein design, similar multi-stage or two-step post-training approaches have been explored in other domains. For instance, a recent study in audio modeling applies a conceptually similar two-phase training scheme for waveform generation (Sinha et al. 2024). described as a "two-step post-training" process.
The following sections detail the implementation and rationale of each stage within our framework.
3.1 Stage 1: Distillation of ProtGPT2 on Spider Sequences
Pro1 [Ferruz et al. stands out among generative protein language models (PLMs) due to its specialized training and demonstrated ability to generate biologically plausible protein sequences. Unlike large-scale PLMs such as ESM-2 (Lin et al., 2023) and ProteinMPNN (Dauparas et al.] 2022), which focus on structure prediction and sequence design constrained by existing protein scaffolds, ProtGPT2 is designed specifically for de novo protein generation. It has been trained on 10 million protein sequences from the UniRef50 dataset of the UniProt Knowledgebase (UniProtKB) (The UniProt Consortiun 2024) using a self-supervised learning approach. This extensive pre-training enables ProtGPT2 to predict the next amino acid in a sequence, effectively capturing the underlying 'grammar" of protein structures.
However, while ProtGPT2 is a powerful model, it is not ideally suited for specialized tasks such as MaSp repeat generation, which requires the model to capture specific sequence patterns unique to spider silk proteins.
To address this challenge, we employ a multilevel strategy that leverages a pre-trained protein language model while optimizing it for our specific task. We begin with ProtGPT2. To enhance efficiency and adapt

---

Published in Transactions on Machine Learning Research (07 (025)
Stage 1: Distillation of pretrained ProtGPT2
Stage 2: Level Finetune with LoRA
Stage 3:
Level Finetune with LoRA and cross- coss-validation
Model learns to understand language of spider proteins
Model adapts to MaSp repeats generation
Model learns correlation between MaSp repeats and mechanical properties
Uniprot DB 100k spider proteins
Tousll (aRn), So
6k MaSp repeats from Silkome database
break
592 MaSp repeats with properties from Silkome database
Figure 2: Illustration of the proposed methodology organized into three levels. Stage 1 involves training a distilled GPT2 model using spider protein sequences from UniProtKB (The UniProt Consortium 2024) In Stage 2, the model is fine-tuned on the repeat regions of MaSp to adapt to their unique patterns. Finally, Stage further finc-tunes the model to capture correlations betwoen MaSp repeats and their mechanical properties.
the model for MaSp repeat generation, we apply knowledge distillation (Hinton 2015
creating a smaller, task-spocific variant: SpiderGPT. This distilled model retains the essential knowledge of its teacher while significantly reducing model size and improving inference time, making more practical for generating novel MaSp repeat sequences tailored to specific mechanical properties.
Dataset: SpiderGPT was trained on a curated dataset of approximately 100k protein sequences obtained from UniProtKB. These sequences were specifically selected based on their laxonomic classification within Araneae (spiders) and an annotation score greater than 1, (Details of the procedure are provided in Appendix A.1 This focused dataset enables SpiderGPT to specialize in spider proteins while leveraging the broader knowledge distilled from the teacher model.
Training Process:
The distillation process follows standard teacher-student framework:
Teacher: ProtGPT2, large pre-trained protein language model, serves as the source of knowledge.
Student: SpiderGPT, lightweight model trained replicate the teacher's behavior while main-
taining computational efficiency.
Distillation was guided by several critical hyperparameters. The temperature parameter T 10 used to soften the teacher's output probability distributions, enabling the student to learn finer inter-token depen- dencies. An interpolation coefficient a = 0.1 controlled the trade-off between soft targets from the teacher and hard targets from the original training data, promoting both generalization and retention of ground-truth patterns.
The SpiderGPT model was implemented with an embodding dimension of Rembd 512, six transformer layers (Mayer = 6), and eight self-attention heads (Thoads 8). This reduced architecture was selected to balance representational capacity with training efficiency, allowing the student model to retain critical domain knowledge while being computationally tractable.
For effective knowledge transfer, both teacher and student models were trained on the same tokenized dataset. The teacher model generated soft labels labels-probel probabiliy distributions over the vocabulary for each token in the input-which served as training targets for the student. Unlike hard labels, these soft targets convey distributional information about alternative token probabilities, allowing the student model learn nuanced structural and contextual relationships embedded in the teacher's output.

---

Published in Transactions on Machine Learning Research (07/2025)
Our knowledge distillation approach resulted in substantial reduction in model size and computational demands, without compromising the quality of sequence generation. A comprehensive analysis of the distil- lation process and its effects is presented in Section 5.3
3.2 Stage 2: Level Fine-Tuning on MaSp Repeat Regions
The GP model, distilled on a spider protein dataset, learns generate spider silk protcins. In Stage 2, fine-tuning on MaSp repeats refines its ability to recognize and generate MaSp-specific motifs and structures while proserving core protein language knowledge. This methodology demonstrates sophisticated approach to transfer learning in computational protein design, which bridges the understanding of the fundamental protein language with specialized structural insights.
Dataset: For this study, we utilized the Spider Silkome dataset (Arakawa et al. 2022 a comprchensive
resource cataloging silk gene sequences from 1,098 spider species and measuring mechanical, thermal, struc- tural, and hydration properties for 446 species. It highlights the role of MaSp paralogs (MaSp1-MaSh) in perfopperoormance silk and identifies key amino acid motifs contributing to silk properties. This dataset serves open platform for advancing biomaterial research and innovation.
For level 1 fine-tuning, we curated a dataset of MaSp sequences, resulting in 6,000 instances. To focus on the functional repeat regions, we removed the first 150 and last 115 amino acid residues, which correspond to the N-terminal and C-terminal domains, respectively (De Oliveira et al. 2024). The repeat sequences of the same MaSp type were then concatenated for each species to create a structured dataset.
Let this dataset be represented as collection of unlabeled MaSp repeat sequences:
D1 = {8: 8: E i = i= 10,,,.,,00000
where V is the standard amino acid alphabet and s; denotes variable-length MaSp repeat sequence over E. This dataset was used to train the model to learn token-level dependencies and contextual representations across natural MaSp repeats.
The fine-tuning procedure adopted a causal language modeling (CLM) objective, which exploits the auto-regressive nature of the transformer architocture. This objective enables the model predict each token in sequence conditioned on its preceding tokens, providing a principled framework for learning sequential dependencies in protein sequences.
Training Process:
To address the challenges of fine-tuning on a limited dalaset, we utilized Low-Rank Adaplation (LoRA), parameter-efficient technique that introduces trainable low-rank matrices into the attention mechanisms of the transformer. Rather than updating all weights, LoRA injects additional parameters into selected layers, substantially reducing the number of trainable weights while preserving model expressiveness. Our implementation followed the standard configuration from Hugging Face's PEFT library. Specifically, we used a LoRA rank of 16, which defines the dimensionality of the inserted low-rank matrices. A higher rank allows for increased adaptation capacity at the cost of more parameters. The scaling factor set to a = 32, amplifying the contribution of the low-rank component before integration into the base model. To regularize training, dropout rate of 0.1 was applied within the LoRA layers. Biases in the original model layers were left unmodified (bias=none), and a weight decay coefficient of 0.01 was used to further constrain parameter growth during optimization.
The model was configured with a maximum sequence length of 512 tokens to accommodate variability input length while ensuring consistent tensor shapes during batching. The tokenizer was adapted from the base model and configured to replace padding tokens with end-of-sequence tokens, ensuring compatibility with the CLM objective.
Training optimization included a learning rate of 5 x 10-4 to enable rapid convergence, a small per-device batch size of 4 to manage memory constraints, and 50 warm-up steps to stabilize learning in the initial epochs. Training procoeded for a maximum of 10 epochs, with carly stopping enabled based on validation loss to prevent overfitting. The Hugging Face Trainer API was used for streamlined integration of training, logging, and checkpointing.
8

---

Published in Transactions on Machine Learning Research (07 (07/2025)
To manage storage and preserve high-performing checkpoints, retained only the top two model states based on validation metrics. This strategy ensured the availability of optimal models for downstream use while maintaining computational efficiency.
By integrating LoRA with careful regularization, early stopping, and resource-aware optimization, we es- tablished robust. fine-tuning pipeline capable of leveraging limited protein sequence data to train powerful generative models. This methodology provides a flexible and scalable foundation for downstream applications in protein design and biomolecular sequence modeling.
3.3 Stage 3: Level Fine- Tuni-T for Propeetty Associations
To enable effective learning of sequence sequence-propert, relationships in the MaSp dataset, we conducted Level 2 fine-tuning using a parameter-efficiet fine-tuning (PEFT) strategy based on Low-Rank Adaplation (LoRA).. This phase was designed to balance model generalization and computational efficiency, while maintaining evaluation rigor through custom cross-validation framework. The fine-tuning objective was bidirectional in nature: (i) to generate MaSp repeat sequences with specified mechanical properties, and (ii) to predict mechanical properties given an input MaSp repeat.
This dual-task setup improves the model's ability to learn robust. mappings between sequence and function, significantly enhancing its utility in the rational design of synthetic spider silk proteins.
Dataset: For the second level of fine-tuning, we curated dataset of MaSp (major ampullate spidroin) repeat sequences and their corresponding mechanical properties from the Spider Silkome database, covering 293 spider species. To ensure high-quality and consistent annotations, we filtered the dataset to include only sequences with complete mechanical property data and excluded repeated entries al the MaSp sublype level. This filtering process yielded a final set of 592 unique MaSp repeat sequences, suitable for modeling sequence property relationships. Each data point comprises the amino acid sequence of MaSp repeat, along with four mechanical properties: toughness, Young's modulus (E), tensile strength, and strain break, along with their respective standard deviations.
Let this dataset be represented as set of labeled samples:
D2 {(si (1,;,ci,44) Ci, 1,..,, 592]
where:
Sy E Σ* is a MaSp repeat sequence over the amino acid alphabet E,
C₁ E [0, 1J$ is the normalized conditioning vector, defined as:
= [Toughness, Toughhess, Strength,, Ittrnngth, E,, TR Strain, Jstrain.s
Each property is paired with its corresponding standard deviation, and all values are scaled to the range [0. 1] via
normalization.
(<Generateseeuence>> <EstinateePropertyy)) is the task token prepended to the model input to specify the prediction direction.
The model learns both directions through task conditioning: predicting properties from sequences and generating sequences from target property profiles, using the full 8-dimensional vector (p..or) conditioning input or target appropriate.
To standardize the feature space and facilitate model convergence, all property values and standard deviations were scaled to the [0, 1] range using Min-Max normalization.
To support bidirectional training, we introduced a task-spocific token at the beginning of cach input, explicitly indicating whether the model should perform sequence generation or property estimation. This ensures clear task conditioning within a unified modeling framework. The two tasks defined
9

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Forward Task (Property Estimation):
<EstimateProperty>> 8; + Model + c;
Reverse Task (Sequence Generation):
<GenerateSequence> C; - Model + 8;
Training Process: To adapt the model for learning sequence property relationships from limited anno- tated data, we reused the LoRA-based architecture introduced during Level 1 fine-tuning. This included injecting trainable low-rank matrices into the attention mechanisms and leveraging the same dropout, weight decay, and optimization settings to maintain regularization and efficiency.
The model was trained using a learning rate of 1 x 10-5, with batch size of 8 per device. We used 50 warm-up steps to stabilize early optimization and trained for up to 10 epochs. An early stopping mechanism was activated based on validation loss to avoid overfitting. Tokenization was performed using a custom tokenizer derived from the base model, with all sequences padded and truncated to a fixed maximum length of 512 tokens. The -nd-ef-sequence token was used for padding ensure consistency with the model's CLM objective.
Cross Validation: To evaluate model generalization under multiple data partitions while maintaining high training efficiency, we implemented a custom cross-validation strategy. The dataset was first divided into 16 equal subsets, each containing 37 sequences. Rather than training on all 16 folds, we randomly selected representative folds to serve as validation sets in independent training runs. In each run, one subset was held out for validation, while the remaining 555 sequences were used for training. This approach allows for robust estimation of performance variance while controlling for training resource demands.
Each run employed causal language modeling objective, in which the model predicts the next token in sequence given its preceding context. After each training cycle, the fine-tuned model and tokenizer were saved separately, and the corresponding validation set was exported for downstream analysis. This approach enabled isolated evaluation across folds and supported robust assessment of generalization performance.
The resulting models exhibit a strong ability to capture complex, non-linear relationships between MaSp repeat amino acid sequences and their associated mechanical properties. By leveraging advanced transformer architectures in combination with principled fine-tuning and rigorous cross-validation, this approach offers promising computational framework for the rational design of synthetic spider silk proteins. It highlights the potential of machine learning to derive meaningful insights from limited experimental data and demonstrates a scalable, generalizable stralegy for biomaterial discovery and engineering.
4 Experimentation
4.1 Model Architecture
The SpiderGPT model emerges as a strategically distilled version of the original ProtGPT2, representing a sophisticated approach to computational efficiency in protein sequence modeling. Developed through knowledge distillation, the model maintains core architectural principles while significantly reducing com- putational complexity. Architecturally, the SpiderGPT is designed with precise specifications that balance performance and efficiency. The model features an embedding dimension of 512, compared to the original model's 1280, and comprises 6 transformer layers against the original 36. This reduction is accompanied by corresponding decrease in attention heads from 20 to 8, and a substantial reduction in total param- eters from 738 million to approximately 50 million. The embedding layer continues to serve critical component, implementing a specialized protein sequence representation approach. By learning contextual representations amino acid sequences, the model captures molecular structural information with a hidden dimension of 2048, enabling nuanced analysis of protein sequence characteristics while maintaining computa- tional efficiency. The multi-head attention mechanism remains a key innovation, allowing parallel processing of sequence information. With 8 attention heads, the model can simultaneously analyze multiple sequence
10

---

Published in Transactions on Machine Learning Research (07/2025)
aspects, facilitating complex feature extraction and providing comprehensive insights into protein sequence relationships. This strategic model compression demonstrates sophisticated approach to machine learning in protein sequence analysis. By preserving the core learning capabilities of the original ProtGPT2 while siguificantly reducing computational overhead, the SpiderGPT model offers rescarchers a more accessible and efficient tool for exploring protein sequence complexities. The model represents a critical advancement in computational protein modeling, bridging the gap between comprehensive sequence analysis and practical computational constraints. Its design reflects a nuanced understanding of both machine learning techniques and the intricate nature of protein sequence structures.
4.2 Setup
Our experimental evaluation focused two critical aspects of the SpiderGPT model's performance in MaSp protein sequence analysis. The comprchensive assessment aimed to validate the model's capabilitiis in generaling biologically meaningful sequences and understanding the intricate relationships between protein sequences and mechanical properties.
The first phase of experimentation concentrated on assessing the quality and biological plausibility of gener- aled protein sequences. We employed a multi-metric approach to rigorously evaluate the generated sequences. This evaluation involved analyzing key molecular characteristics, including sequence composition, amino acid distribution, and structural coherence.
The second experimental phase delved into the complex relationship between MaSp repeat sequences and their mechanical properties. We sought to establish correlations between specific sequence characteristics and mechanical attributes such as toughness and elastic modulus. By systematically mapping sequence features to mechanical properties, we aimed to uncover the underlying molecular determinants that influence the material performance of spider silk proteins.
The experimental design was carefully constructed to provide insight into the predictive capabilities of the model and its potential to advance understanding of protein sequence-struccure-pre By combining computational modeling with rigorous statistical analysis, we aim to bridge the gap between molecular-level sequence information and macroscopic material performance.
relationships.
4.3 Unconditional Sequence Generation: Self-consistency Assessment
To assess the model's capability lo generate biologically plausible sequences, unconditionally generated 100 MaSp repeats from each of the five independently trained models in Level 2 fine-tuning (a sample of 50 sequences is provided in Supporting Document SD1). This resulted in a total of 500 sequences generated without conditioning on mechanical property values. To evaluate their fidelity, we compared the distribution of key biochemical and structural properties against. the 592 natural MaSp repeats used during Level 2 fine-tuning. The results, presented in Figure 3 quantitatively assess the similarity between generated and natural sequences, highlighting how well the model captures fundamental features of spidroin repeat regions.
The probability distribution of amino acid occurrences in both natural and generated sequences was examined using KL divergence, which quantifies how one probability distribution deviates from another. Expressed in bits, KL divergence represents the additional information required to encode a sequence based on the background amino acid distribution of MaSp repeats (Appendix A.3 Additionally, Hamming distance was used to assess diversity in the generated sequences by mcasuring the number of differing positions between sequences of equal length, providing insight into the variation and uniqueness of the generated sequences relative to natural MaSp repeats.
To further assess the biological plausibility of the generated sequences, we performed a comprchensive anal- ysis of key physiochemical altributes, including sequence length, molecular weight, instability index, and isoelectric point (see Appendix A.2 for more details). These properties were computed using ProtParam (Gasteiger et al., 2005), widely used tool for protein sequence analysis, implemented via a Python- tool package to ensure accuracy and reproducibility. The results indicate that the generated sequences exhibit similar distributions across these physiochemical attributes, aligning closely with natural MaSp sequences.
11

---

Published in Transactions on Machine Learning Research (07 (025)
KL Divergence Distribution
Hamming Distance Distribution
Sequense Length Distribution
Natural Cenerated
Meat 0.732
Natural
Generated
1.50
2000
Number of Sequence Pairs
1500
Divergence
Sequence Langth
0.75
1000
500
24
6.00
Natural
Generated
0.60
Hamminng Distance (lower similar)
0.70
0.75
0.80
5.05
Natural
Generated
Molecular Weight Distributiin
Isoelectric Point Distribution
Inscability Index Distribution
natural
naturrl
Natural
Cener abud
Cerie ratedd
Generated
Molccular Woight (KDo)
Inoelectic Paint (p=))
Instablitty Index (score)
100
7.09
7.10
Ir 27 78
60:47
43-7
Netural
Generated
Natural
Generated
Natural
Generated
Secondary Structure
Motif Distribution
Amino Acid Compasition Distribution
Generated
Natural
0.15
Natural Generated
Natural
Generated
1-.0
0150
0.30
Fraction (coun))
0.5
Frequency (count)
Fraction
0.100
M=0.05 05
0.70
0.075
077.077
0.2
0.000
0.10
0.1
0.055
p=0p00000
p-
0.000
poly-Ala
YGQGG
0.00
C D
F
K. M
Amino Acid
Figure This figure presents a comprehensive comparison of nine key physicochemical and structural properties between naturally occurring (Natural) and computationally generated (Generated) proteins. The analysis includes distributions of KL divergence, Hamming distance, molecular weight, isoelectric point, instability index, sequence length, motif patterns, sccondary structure elements, and amino acid composition. The plots demonstrate the degree of similarity between generated proteins and their natural counterparts across multiple biologically relevant parameters, providing insights into the fidelity of the protein design process.
Additionally, secondary structure fractions were analyzed using the ProteinAnalysia module from Biopy Cook et al.] 2009), which follows established secondary structure prediction frameworks, including DSSP [Kabsch & Sander 1983) and Chou-Fasman propensity scales (Chou & Fasman 1974). The ProteinAnal- ysis satuuddrryssuuuuueeeeereo ffrcctirr_]_ function was used to compute the fractional composition of a-helices, strands and others/unstructurr regions. This analysis enabled quantitative comparison of secondary
12

---

Published in Transactions on Machine Learning Research (07/2025)
structure content between natural and generated sequences, providing insights into structural stability and folding tendencies.
Furthermore, the frequency of key motifs in the generated sequences was examined as part of the secondary structure evaluation. Figure presents the distribution of poly-Ala, GGX, YGQGG, and SV motifs, offering Equation [ The selected set of motifs are known impact mechanical properties of the silk fiber (Arakawa et al,
@
insight into their role sequential integrity. The formal definition of these motifs is provided
2022 Nakamura et al., 2024b) (refer section 2.1p. p. The motif frequency distribution of natural and generated sequences, illustrated in Figure suggests that the generated sequences reflect similar distribution natural sequences.
To further explore the structural properties of the generated sequences, we analyzed the amino acid compo- sition and secondary structure fractions, which are critical determinants protein function and mechanical behavior. Figures present these comparisons in detail. It's worth mentioning that the standard sccondary structure prediction methods optimized for globular proteins, making them less reliable for structural proteins like MaSp repeats (Jumper et al. 2021
4.4 MaSp Motif- Peoff-Property Correlation Analysis
The relationship between MaSp motifs and mechanical properties spider dragline silk offers critical insights into how specific protein sequence patterns influence silk performance. Recent high-throughhut studies have quantified key mechanical metrics -including toughness, Young's modulus, tensile strength, and strain break of dragline silk from various spider species, identifying correlations between motifs in the repetitive core region of MaSps and variations in these properties (Arakawa et al. 2022
Structural motifs such as poly-Ala (associated with f-sheet crystallization) and GGX (contributing to clastic 8-turns) play a direct role in defining tensile strength, toughness, and extensibility. In addition to these well- characterized motifs, recent analyses have identified emerging motifs like YGQGG, QQ, GPGXX, AGQG and SV, which may further modulate mechanical behavior (refer section 2.1). The regular expression of each motif, which we used for current analysis, is mentioned in equation ! Understanding these relationships is rusenaal in the design of biomimetic silk materials with tailored mechanical properties.
YGQGG:
YGQGG
poly-Ala:
A(3,}
GGX:
GG[A Z
motifs
QQ:
QQ
(1)
GPGXX:
GPG(A [2
AGQG:
AGQG
SV:
SV
In previous studies, correlation analyses have primarily focused on the distinct motifs present in different MaSp types (MaSp1, MaSp2, and MaSp3) and their specific influences on mechanical properties (Craig et al. 2020 Nakamura al. 2024a Kono al. 2021a Arakawa et al. 2022). By contrast, the current work considers MaSp more abstractly, providing an overarching view of how various motifs may affect mechanical performance.
For the correlation study, we refined the level fine-tuning dataset by selecting instances with unique sets of mechanical properties. This resulted in filtered subset of 294 instances. Using this refined dataset, we conducted a reverse generation lask, where the model generated 294 MaSp repeals corresponding to these specific property sets. The generated sequences were then used for correlation analysis, allowing a structured evaluation of the relationship between protein motifs and mechanical properties. This study aims to determine whether the generated sequences accurately reflect the cequence-to-proeert correlations observed in previous research.
13

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Mechanical Properties vs Sequential Features Correlation
1.00
toughness 0.228
0.207
0.099
0.218
0.064
0.043
0.005
-0124
0.054
0.50
Correian Iireiatioo Coefficient
young's modulus-
0.091
0.086
0.174
0.070
0.053
0.034
0.042
0.063
0.046
0.047
-0.036
-0.075
-0.139
-0.032
0.25
0.00
tensile strength
0.230
0.190
0.118
0.140
0.043
-0.003
0.022
0.042
0.034
0.121
-0.062
-0.086
-0.085
-0.063
-0.25
strain at break
-0.75
-1.00
YGQGG count
YGOGG coverage
poly-Ala count
poly-Ala coverage
GGX count
GGX coverage
QQ count
QQ coverage
GPGXX count
GPGXX coverage
AGQG count
AGOG coverage
SV count
SV coverage
Figure 4: The heatmap illustrates the correlation between sequential features and the mechanical properties of the generated sequences. While some weak correlations are present, the overall low values suggest the need for a advanced approach to better capture sequence-properly relationships.
Figure 4 presents the correlation heatmap between sequential features and mechanical properties. The sequential features include the count and coverage mmotif sizeXcount) sequence length the specified motifs. The heatmap indicates that correlation values remain relatively low. While no strong correlations were observed, this visu- alization is still informative, demonstrating that individual motifs alone cannot fully explain the relationship between sequence variation and mechanical properties. This highlights the intricate and multi-factorial na- ture of silk mechanics, where the combined influence of multiple motifs and structural contexts collectively shapes the observed mechanical behavior.
In the following, we present row-wise analysis of the heatmap:
Toughness (Energy Break)
Toughness the lotal energy absorbed before failure. Higher toughness in dragline silk is associated with motifs that enhance tensile strength and/or extensibility. The correlation heatmap reveals that YGQGG count (+0.228) and coverage (+0.207) both show moderate positive corrclations with toughness. Poly- Ala coverage also correlates posilively with toughness, though slightly less strongly (+0.218). The AGQG coverage (-0.192) and SV count (-0.124) showed negative correlation with toughness..
Young's Modulus (Stiffness)
Young's modulus represents the initial stiffiness of silk fibers and is influenced by motifs that foster 8-sheet crystallinity. For example, poly-Ala count correlates positively with Young's modulus (+0.174). In our analysis, we also observed positíve impact of the count of YGQGG motif Young's modulus (+0.091), whereas the count of AGQG (-0.036) and SV (-0.139) motifs showed negative correlations.
Tensile Strength (Ultimate Stress)
Tensile strength is the maximum stress that the silk fiber can withstand before failure. The correlation heatmap indicates that many of the same motifs influencing toughness also affect strength. The YGQGG motif count is positively corrclated with tensile strength (+0.230) (Arakawa al. (022), making one of the strongest sequence predictors of stronger fiber in our analysis. Poly-Ala coverage (+0.140) and GPGXX coverage also positively correlate with tensile strength.
14

---

Published in Transactions on Machine Learning Research (07 (225))
Strain at Break (Extensibilityy
Strain at break is the maximal elongation (expressed as fraction of its original length) that the silk fiber achieve before breaking. The correlations highlight AGQG coverage (-0.133) is the most striking correlation overall (and the only moderately strong negative value). Other features in this row hover near zero, indicating minimal correlation. The results also show impact of QQ motif on strain at break. Other features in this row hover near zero, indicating minimal or no correlation.
In summary, the correlation between mechanical propertics and individual motifs in the generated sequences follows patterns reported in previous studies (Arakawa al. 2022 Nakamura et al. 2024b). However, the overall weak correlations suggest that individual motifs alone cannot account for the observed variations in fiber mechanical properties (Arakawa et al. 2022. This study builds upon existing sequence-property relationships and leverages machine learning to better capture the complex interplay between sequential motifs and mechanical properties, enabling the design of novel MaSp sequences.
5 Results and Discussions
The proposed framework performs both the forward lask of generating MaSp repeats tailored to specific mechanical properties and the reverse task of predicting mechanical properties from given MaSp repeat. This section presents the evaluations performed to assess the model's ability to generate biologically plausible sequences and accurately predict mechanical properties.
5.1 Biological plausibility of generated sequences
We perform an in-tepth investigation using two datasets: the test set and the BLAST set. The test set con- sists of sequences sampled from the Spider Silkome dataset, allowing us to assess the modol's self-consistency and its ability to reproduce meaningful sequences conditioned on known mechanical properties. The BLAST set, on the other hand, is used to evaluate the novelty and classification of the generated sequences within broader protein sequence databases. By integrating these evaluations, we aim ensure that the generative model produces biologically plausible sequences that captures key features that are essential for functional spidroin design.
5.1.1 Conditional Generation Assessment: Evaluation test dataset
To assess the self-consistency of the trained model, we evaluated its performance on the test sets from all five models trained during Stage 3 (Level 2 fine-tuning). Each model was tested on a set of 37 instances, resulting in a total of 185 unique test records. Each instance includes a MaSp repeat sequence along with its corresponding mechanical property set. The detailed composition of the test set is provided the supplementary document SD2. The mechanical properties from these test instances were used input conditions for sequence generation.
We note that the sequence property relationship is not strictly one-to-one. The comparison is based on the hypothesis that similar mechanical properties can arise from sequences that share structural or motif-level features, particularly within the MaSp repeat regions. The objective was to qualitatively assess whether the generated sequences resemble natural ones associated with similar mechanical profiles.
prrperce-property
To evaluate how well the model captures the relationship between sequence and function, we compared the generated sequences to their natural counterparts at both the sequence and structural levels. As shown in Figure 5, this dual-level analysis highlights the model's ability to reproduce key biochemical and structural characteristics of spider silk proteins when conditioned on specific mechanical properties.
The sequence-level evaluation assesses key physiochemical attributes, including molecular weight, instability index, and isoelectric point as well as motif occurrences, including poly-Ala, GGX, YGQGG, and SV. Figure illustrates the experimental results. The high agreement between these attributes the generated and original sequences suggests that the model effectively learns to generate biologically plausible MaSp repeats when conditioned on specific set of mechanical properties.
15

---

Published in Transactions on Machine Learning Research (07/ 2025)
KL Divergence Distribution
Hammling Distance Distribution
Sequence Length Distribution
1.03
9.20
1500
Sequence Length (residucs)
Diverreece (b:bs)
3.5
Distance
0.95
1230
1000
0.00
0.0
750
0.83
500
447.4
0.00
C.0.1121
0.70
Natural Generated
Natural Generated
Natural
Generated
Molecular Weight Distribution
Isoelectric Point Distribution
Instability Index Distribution
130
Nann
kennsl Cenorute
Son- Converstd
Concrsted
12
60
125
Mollecaler Weightt (kDe)
borelectric Pairs. (ptt)
40
CO
40
At. 0.49
ity
p1 2.-25.4 40
25
2
-25
Netural
Gener aled
Netural
Generated
Natural Amino Acid Composition Distribution
Genvereted
Secondary Structure Distribution
Motif Distribution
0,200
Nutonal wwoond
Comerated
0.35
Natural
Generated
0.6
puo 592
p47
0.150
0.30
0.5
Frequency (count))
0.125
0.25
Freciton
0.4
0.100
1H00001
Fraction
0.20
Ued R202
0.075
077
0.15
0.2
0114
kmm
0.1
0.05
p-0000006
L K
0.00
Others
Sheet
GGX
poly Ala
YGQGG
Amino Acid
Figure 5: Comparison between original (natural) and generated sequences on the test in terms of var- ious matrices. (1) Sequence properties: sequence length, molecular weight, instability index, Isoelectric point. (2) Average amino acid frequency distribution grouped by physicochemical properties. The property consistency highlights the validity of the generated sequences in terms of their structural and biochemical features. Furthermore, the consistent alignment demonstrates the model's ability to effectively capture the key characteristics and properties of MaSp.
Additionally, sequence similarity was analyzed using KL divergence and Hamminn distance. KL divergence quantifies the deviation in amino acid probability distributions between original and generated sequences. As shown in Figure 5 KL divergence remains constrained within a low range, indicating that the model successfully captures the natural amino acid distribution of MaSp repeats. In contrast, Hamming distance evaluates sequence diversity the character level. The high Hamming distance values demonstrate the
16

---

Published in Transactions on Machine Learning Research (07 (07/202) (025)
Molecular Weight Distribution
Instability Index Distribution
Isoelectric Distribution
100
200
Instability index (score)
12
Isoelectric Point (OH)
150
10
Moleculor Weight
100
7.0
u
12866,0
50
1--7
-50
-20
Generated
Dlast1 Sequence Type
Blast2
Generated
Blast1 Sequence Type
Biast2
Generated
Blast1 Sequence Type
Dlast2
Secondary Structure Distribution
Motif Distribution
Generated
0.0005
Gene rotdd
lun 60g
Blast1
0.6
0.0004
Fracton (count)
0.5
Frequency (count)
0.0003
0.4
0.3
0.0002
0.2
L-O (001)
0.0001
gwte La L38
0.1
U=2000004
9=1.09000000200 Und (00b-0000000100000000
0.0000
Others
Sheet
poly-Ala
GGX
YGQGG
SV
Amino Acid Distribution Sequence Type
Sequence Comparison Metrics
0.35
Blest: Bloot2
0.25
0.20
0.10
Um0.9
1=0.5
0.05
0.00
KL Div Blast1
KL Div
Hamming Blast1
Hamming Blast2
G
H
Blast2
Amina Acids
Metric Type
Figure 6: Comparison sequence and structural properties our generated protein sequences with BLASTI and BLAST2 sequences. The analysis includes distributions instability index, isoelectric points, molecular weight, motif prevalence, sccondary structure fractions, amino acid composition, and sequence similarity metrics. This highlights the consistency of the generated sequences with natural proteins and their alignment across various properties
model's ability to generate diverse sequences, avoiding excessive similarity to training data while preserving essential biological patterns.
The structural evaluation further validates the sequence fidelity by analyzing key secondary structure fea- tures a-helices, 3-strands and unstructured regions. The results indicate strong structural resemblance of the secondary structure composition between generated and natural sequences, confirming that the model preserves critical biological motifs found in naturally occurring MaSp repeats (Figure
The ability of the generative model to replicate both physiochemical and structural characteristics un- derscores its potential as a powerful tool for biomatcrial design. By tailoring soquences to meet specific functional and mechanical property requirements, this approach offers a promising avenue for advancing the development of engineered biomaterials with enhanced properties.
17

---

Published in Transactions on Machine Learning Research (07/2025)
5.1.2 Novelty Assessment: BLAST Evaluation
To evaluate the novelty of the generated MaSp repeats, we performed BLAST analyses against a broader Spider Silkome spidroin repeal, database. The search space consists of 11K naturally occurring spidroin sequences. It includes seven primary types of spider silk proteins major ampullate spidroins (MaSp), flag- elliform spidroins (Flag), minor ampullate spidroins (MiSp), aggregate spidroins (AgSp), pyriform spidroins (PySp), aciniform spidroins (AcSp), and cylindriform/ subset of seven generated sequences was compared with the two most closely related natural sequences, des- ignated BLASTI (top match) and BLAST2 (second top match) based on BLAST results. The selection was based on high query coverage, percent identity, and sequence length similarity to the generated sequences. This dataset is referred to as the "BLAST set', with details of all sequences and their sources provided the supporting document SD3. All selected BLAST matches had an expect value (E) of I e- which ensured statistically significant similarities. Furthermore, we selected matches spanning different MaSp sub- types (MaSp1, MaSp2, MaSp3) to provide comprehensive comparative analysis. The expect value (E) quantifies the probability of obtaining a match by chance in a given database size, decrcasing exponentially as the alignment score (S) increases.
/ /tubuliformm spidroins (CySp) (Yarger al. 2018a). A
To assess uniqueness, we follow established criteria where sequences with similarity values below 50- 50-60% are considered novel (Quan et al. 2023). The generated sequences meet this threshold when compared to broader protein databases, demonstrating the model's ability to design distinct sequences that either do not exist in nature or have not yet. been observed. Additionally, BLAST search results consistently classify the most similar existing sequences MaSp, aligning with the intended spidroin type. This indicates that the generative modeling approach effectively captures the defining characteristies of specific spider silk proteins.
Figure G presents the comparison between the generated MaSp repeats and the closest matches, BLASTI and BLAST2. The observed consistency highlights the ability of the model to accurately reproduce essential sequence leatures of spider silk proteins. The generated sequences maintain comparable molecular weight ranges, predicted secondary structure composition, and plysiochemical properties, all of which are crucial for mimicking the functional behavior of natural spider silks. Furthermore, the model effectively balances critical sequence attributes, such as the instability index and isoelectric point, reinforcing the biological plausibility of the generated sequences.
Although Figure o confirms that the generated sequences share similaritics in fundamental composition, structural, and sequence pattern with naturally occurring MaSps, reinforcing their biological relevance, variations in metric values may arise due the selection of reference sequences for the BLAST comparison. This highlights the sensitivity of the evaluation process the chosen dataset. Despite these variations, the model consistently captures the essential properties of spider silk proteins, further emphasizing its potential for biomaterial design and the generation of novel sequences with customized functionalities.
Table 1: Quantitative comparison of mechanical property prediction accuracy between SpiderGPT (ours) and SilkomeGPT (bascline) based trend similarity metrics computed over reference and generated mechanical property profiles. Bold values indicate better performance for each metric.
Metric
SpiderGPT (ours) SilkomeGPT (baseline)
Pearson Correlation (r)
0.8884
0.8349
Spearman Correlation (p)
0.8343
0.7798
Mean Absolute Error (MAE)
0.0861
0.0963
Root Mean Square Error (RMSE)
0.1047
0.1162
R² Score
0.6383
0.5861
Cosine Similarity
0.9827
0.9783
5.2 Evaluation of Mechanical Property Prediction
To evaluate the model's ability to estimate mechanical properties from MaSp repeat sequences, we use the test set of 185 instances described earlier. For each sequence in this set, perform inference using the
18

---

Published in Transactions on Machine Learning Research (07/ (222)
Toughness (Com: 0.868, R2 0. 6.68))
Youn''s Modulus
Toughnesss KS test p-value: 4. 4.16e-01
Young's Modullus
(Cor:: 0:907. R2: 0. 790)
KS test p-vaitie: 9 9.81e-01
1.0
Prediteed Velure
0.0
Predicted Vallue
0.8
1.5
0.6
Denaity
1.0
Density
04-
0.4
0.2
0.-
Aatime 352.
4ptrr 6.33
0.0
0.
1.0
0.0
2.8
Or ginal Value
Original Va ue
Property Value
Property Value
Tensile Strength (Corr: 0.909, 0.793)
Strain at Break
Tensile Strength p-vallue 9.00e-01
Strain at Break
0.883. R2: 0.745)
p-value: 7.52e-01
1.0
1.0
2.5
Prediittd Valuee
Predicted value
a..-
8.-
Density
1.0
Density
0.4
0.5
0.-
1.6.0-125
Nobiral p-6-3 a1632
00
0.0
Cenercod ped 220.22
Doncrtted p-2.3 e-233
0.0
0.8
1.0
0.0
0,3 2.0
00
2.0
Original Va
Pmperry Value
Property
(a)
(b)
Figure 7: Comparison of predicted and original mechanical properties for MaSp repeats in the test set. (a) Scatter plots show the corrclation between prodicted and actual values for normalized mechanical properties, with Pearson's and ideal 1:1 (red dashed) line. (b) Kernel density plots compare natural and generated distributions for each property, including KS test p-values, means (14), and standard deviations (a). Results indicate effective property prediction by the model.
model's reverse task 1eci---preiiingggg mechanical properties from sequence input. The predicted properties are then compared against the corresponding ground-truth values.
We benchmark the performance of our model (SpiderGPT) against SilkomeGPT (Lu et al. 2024). . Since SilkomeGPT does not focus exclusively on repeat regions, we use the original (unprocessed) sequences from the test that is, before repeat extraction- for both models to ensure fair comparison in the mechanical property prediction task.
Prediction quality is assessed using five standard statistical metrics: Pearson's correlation coefficient (r), Spearman's 's rank correlation (p), mcan absolute error (MAE), root mean square error (RMSE), and cosine similarity. The results, presented in Table [ provide a quantitative comparison of the models' ability to replicate the mechanical property patterns the reference data. Tt important to note that SilkomeGPT was trained on the entire Spider Silkome dataset, meaning that the test instances used here were part of its training data. Despite this advantage, the test set outperforsss SilkomeGPT across all evaluated metrics.
trannd only on repeat regions and without exposure to
The Pearson correlation coefficient was found to be r = 0.8884, indicating a strong positive linear relationship between the predicted and reference values. This suggests that the model captures a substantial portion of the underlying variance in the data, even though the relationship is not perfectly linear. Additionally, the Spcarman correlation cocfficient was p = 0.8343, reflecting strong monotonic relationship. Unlike Pearson, Spearman correlation assesses the consistency of the rank order between two variables. A high p value implies that when reference property increases, the corresponding predicted property also tends to increase, even if the absolute values do not match precisely. This highlights the model's ability to preserve the relative ordering of mechanical properties.
The MAE was measured to be 0.0861, indicating that, on average, predicted values differ only slightly from the reference values. The RMSE was slightly higher at 0.1047, consistent with its sensitivity larger deviations. This implies that while most predictions are accurate, a few instances exhibit moderate error, which remains within acceptable bounds.
19

---

Published in Transactions on Machine Learning Research (07 (07/2025)
The coefficient of determination (R²) was observed to be 0.6383. In simple linear regression, R is equal to the square of the Pearson correlation coefficient, i.e., R = 22 Substituting r 0.89, we obtain R (0.89)² x 0.79. However, the current dataset involves multiple output dimensions and includes non-linear dispersion, which can lower the observed R despite a high average correlation. This explains the discrepancy between the theoretical and observed values, with n² = 0.6383 being a reasonable outcome this context.
In addition, the cosine similarity between predicted and original property vectors was calculated to be 0.9827, This high value indicates strong directional alignment, implying that the overall structural trends are well preserved even if individual values are not perfectly matched.
A property-level analysis is provided in Figure 7 Panel (a) displays scatter plots comparing predicted and original values across the four mechanical properties: loughness, Young's 's modulus, tensile strength, and strain at break. Each subplot includes the Pearson correlation coefficient (r) and the coefficient of determina- tion (R*), offering a detailed view of model performance. All propertics exhibit strong positive correlations, with tensile strength achieving the highest R2 (0.793) and toughness the lowest (0.689), indicating robust but slightly variable predictive accuracy across property types.
Panel (b) of Figure 7 shows kernel density estimates comparing the distributions of natural and generated Maanogoro---minnoo property values. Each subplot includes the Kolmogorov Smirnov (K-S) test p-value, along with the mean (n) and standard deviation (a) for both distributions. High p-values across all properties (e-g., 0.752 for strain at break) suggest that the generated properties are statistically consistent with the natural ones, as
measured by the K-S test (Massey Jr. 1951).
5.3 Ablation Studies
In this section, we perform ablation experiments over number of facets of the proposed methodology in order to better understand their relative importance.
5.3.1 Without Distillation
The first stage of the architecture pipeline involves distilling the ProtGPT2 model into smaller SpiderGPT model. In this section, we explain the need for this technique and compare the Pro1GPT2 and SpiderGPT models.
Pretrained protein language models (PLMs) like ProtGPT2 are large and designed for general protein gen- eration lasks. While these models excel broad protein generation, the current task is more specific. It involves generating MaSp repeats from a smaller dataset, where the sequences follow a distinct and spe- cialized pattern. Given the focused nature of the task, lighter model is more appropriate. It not only improves inference speed but also simplifies the overall architecture, making it more efficient for the specific requirements of this task.
Table 2 provides an architectural comparison between ProtGPT2 and SpiderGPT, highlighting that the Spi- derGPT model is significantly more compact than its baseline counterpart, ProtGPT2. To assess their perfor- mance differences, we generated 200 protein sequences using each model. The reduced size of SpiderGPT not only simplifies the pipeline complexity but also accelerates inference, with the distilled SpiderGPT achieving remarkable six-fold increase in inference speed compared to ProtGPT2. Evaluations conducted on these 200 sequences reveal that this boost in efficiency incurs only minimal performance trade-offs. Specifically, the student model, SpiderGPT, sustains perplexity levels comparable to those of the teacher model, ProtGPT2. For a more detailed visual comparison of the two models' performance, refer to Figure g which illustrates the outcomes of both the ProtGPT2 teacher model and the SpiderGPT model.
5.3.2 Without first level fine tuning
ProtGPT2 is a decoder-only transformer model that has been pre-trained on the protein space using the UniRef50 database (version 2021 __) (The UniProt Consortium (002). which contains 10 million protein instances. However, the Sequence-to-Moccanii Property Correlation dataset, curated from the Spider Silkome database, contains only 592 instances. This small dataset. is insufficient for training the model
20

---

Published in Transactions on Machine Learning Research (07/2025)
Attribute
ProtGPT2
SpiderGPT
Embbediigg Dim (Bembd)
1280
512
Layers (Mayer)
36
6
Hidden Dim
5120
2048
Attention Heads
20
8
Total Parameters
738M
50M
Table 2: Comparison of Teacher and Student Model Architectures
3.5
17.5
Inference Time (seconds)
3.0
15.0
Perplexity
12.5
2.0
10.0
1.5
2.5
Pro1GPT2
SpiderGPT
ProIGPT2
SpiderGPT
(a) Inference time
(b) Model Perplexity
Figure 8: Comparison of performance of teacher (ProtGPT2) and student (SpiderGPT) models over gener- ation of 200 novel protcin soquences.
generate MaSp repeat sequences while simultaneously learning the correlation between sequences and mechanical properties. To address this limitation, split the training into two stages.
In the first stage, focus training the model to generate MaSp repeats. In the second stage, we train the model to learn the correlation between sequences and their corresponding mechanical properties. This staged approach allows the model to first specialize in generating MaSp repeats before learning to predict mechanical properties from these sequences.
In this section, we evaluate the impact of the first level of fine-tuning, where the model is trained on 6,000 instances of MaSp repeats. To emphasize the significance of this stage, we also present the model's perfor- mance when fine-tuned directly the 592 instances from the Spider Silkome dataset, which contains MaSp repeats along with their corresponding mechanical properties. This comparison highlights the importance of multi-level fine-tuning in achieving optimal results.
For consistency, we used the same training setup across both scenarios. The SpiderGPT model was fine- tuned with the following LoRA settings: low-rank dimension (r) = 16, scaling factor (a) 32, dropout 0.1, and weight decay 0.01. Training was conducted for epochs with a Icarning rate of 5 10-4 and per-device batch size of 4. Additionally, 50 warmup steps were used, and weight decay applied rate of 0.01. Below evaluate the impact of this ablation study both forward (sequence generation) and reverse (property estimation) tasks.
Impact on Sequence Generation Quality
As discussed previous sections, the primary objective of the model in the forward task generate MaSp repeats tailored to given set of mechanical properties. Tn this section, evaluate the impact of
21

---

Published in Transactions on Machine Learning Research (07/2025)
Mechanical Property
0.177.0.22,00062...666 241,0.266.... 2...0.55
Model output MaSp
Model output MaSp repeat fine- -een-tuning
repeat fine-tuning
MESTTTRLALAFLLLLCTO WTW.W//.TTTEPPALLSSSSSSSSSSSSSSS
ALSFIRSF L LNSASQAAAAI BSQQIDDMMSSGIIMMA
MEKMARSNKSSSKSEE QA NIMAFASSWEEWWEOO
32%
009RG0GFFFAAA0GG22HHHAA666666662 AAASSGG
KAMAICORSSNGNGSSGPRRRO
68%
Generation Types
Naisy generation (presence of non repeat preftwleuffii)
Valid generation (only MaSp repeat
region)
Figure 9: Impact of level 1 fine-tuning on sequence generation quality. Without MaSp fine-tuning, the model generates non-repeat prefix/suffix (68% occurrence), leading to structural deviations. The molecular structure predicted by Omegafold reveals helical formations, indicating the presence of terminal domains. contrast, model achieves 100% valid sequence generation we include MaSp repeat. fine-tuning step in the methodology pipeline.
omitting the initial fine-tuning phase (level 1) on sequence generation quality. To this end, we generated 100 sequences using two models: one trained with both levels of fine-tuning and another trained without level 1 fino-tuning.
The omission of level fine-tuning resulted in significant deviations in sequence generation, with the model frequently producing spider silk sequences containing non-repeat regions an unintended outcome. To detect these deviations, sliding window algorithm rich regions, which are typically abundant in valid MaSp repeat sequences. In contrast, sequences containing non-repeat regions exhibited almost null densitics of these characteristic motifs. Among the 100 generated sequences, 68% included non-repeat prefixes or suffixes, appearing at the beginning or end of the sequences, respectively (Figure 9j. This finding suggests that, without the initial fine-tuning phase, the pretrained model retains its original behavior, failing to specialize generating only the repetitive core region of MaSp sequences.
employed to analyze the densily of poly-alanine and glycine-
To further investigate these discrepancies, we compare the structural predictions of sequences generated by model trained with only one fine-tuning stage against those produced by model trained with both levels of fine-tuning. Figure g illustrates the molecular configurations of the generated sequences, revealing structural differences arising from the absence of MaSp-specific fine-tuning. Structural predictions were obtained using Omegafold (Wu et al. 2022), providing insights into the conformational tendencies of the sequences. Notably, the generated sequences without level fine-tuning exhibited helical structures, which are typically associated with non repeat regions like terminal domains INTD/CTD) These unwanted elements further highlight the necessity of level 1 fine-tuning in ensuring the correct generation of MaSp repeats.
Impact on Property Prediction
Here, we analyze the impact skipping Level 1 fine-tuing on the reverse task of mechanical property estimation. Using the same test set, we prompted the model to predict mechanical propertics from sequence inputs and compared the predicted values against their ground truth counterparts. To assess performance, compared trend similarity metrics with and without Level fine-tuning. When the model was trained
22

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Table Comparison of generated and reference properties without level finetuning
Metric
Mean Value
Pearson Correlation (r)
0.3628
Spearman Correlation (p)
0.3301
Mean Absolute Error (MAE)
0.2911
Root Mcan Square Error (RMSE)
0.3447
R square (f²)
-2.7196
Cosine Similarity
0.8255
without the 6,000 MaSp repeat sequences used in Level 1, its ability to recover accurate property trends declined significantly, shown in Table 3 The mean Pearson correlation dropped from 0.8884 to 0.3628, and the mean Spearman correlation fell from 0.8343 to 0.3301, indicating considerable loss in both linear and rank-order consistency.
Prediction errors also increased substantially: MAE rose from 0.0861 to 0.2911, and RMSE increased from 0.1047 to 0.3447. Most notably, the R² score deteriorated from 0.6383 to -2.7196, suggesting that without Level fine-tuning, the model performs worse than a simple mean-based predictor. Cosine similarity also decreased from 0.9827 to 0.8255, reflecting weakened directional agreement between predicted and reference property vectors.
These results highlight the essential role of Level 1 fine-tuning in learning robust sequence embeddings that generalize well to downstream property prediction. Its absence leads to diminished trend fidelity and overall performance degradation.
6 Conclusion and Future Work
This research presents a novel generative model based on GPT architecture, meticulously fine-tuned to leverage a dataset comprising 6,000 repeat regions derived from spider silk proteins. This initial training was subsequently enhanced through a secondary fine-tuning phase utilizing approximately 600 Major Ampullate Spidroin (MaSp) sequences, each annotated with cell-chhaacterized mechanical properties. Our innovative dual-level fine-tuning strategy has proven effective in producing synthetic sequences that incorporate essential structural motifs, such the glycine-rich GGX repeats and poly-Ala stretches, which are fundamental to the remarkable extensibility and tensile strength exhibited by natural spider silk.
Detailed structural analyses of the generated sequences reveal a strong aligmment with the anticipated sec- ondary structure composition, such as o-helical and 8-strand conformations, which are critical for replicating the functional altributes of spider silk. Furthermore, comparisons between the mechanical properties pre- dicted by the model and established reference data underscore the reliability and promise this approach for designing spidroins with tailored mechanical properties. By enabling the precise generation of sequences tailored to specific performance criteria, this work lays robust foundation for the sustainable production of synthetic spider silk materials. The platform that we have developed is both scalable and highly cus- tomizable, offering significant potential to transform material science and synthetic biology. Its implications extend beyond spider silk, paving the way for the creation of next-generation bioinspired materials with applications in diverse fields such as tissue engineering, textiles, and environmentally friendly composites.
Future work will encompass comprehensive experimental validation process for the generated soquenees, which will involve synthesizing these sequences and subjecting them to rigorous mechanical testing. This step is crucial to verify that the predicted properties align with the actual performance characteristics observed under controlled conditions. The synthesis process will aim to accurately replicate the molecular structures proposed by the generative model, while the mechanical testing will evaluate key parameters such as tensile strength, extensibility, and toughness to ensure that the sequences meet the anticipated functional benchmarks.
A key direction for future work involves enhancing the generative model by integrating significantly larger and more diverse datasets to improve both predictive accuracy and generalizability. Scaling up the training data
23

---

Published in Transactions on Machine Learning Research (07/2025)
will allow the model to better capture the complex, non-linear relationships between sequence composition and resulting mechanical properties, thereby refining its capacity to generate optimized MaSp repeat designs. While the current methodology demonstrates strong performance, it was specifically tailored to a relatively small, high-quality dataset.
To extend this framework, we plan to expand the large corpus of unlabelled spidroin sequences available through the Spider Silkome resource (Arakawa) et al. 2022). These sequences will be annotated with experimentally or computationally derived mechanical properties in future work, enabling their integration into the training pipeline. This expanded dataset will encompass broader diversity of spidroin variants and their corresponding mechanical attributes, supporting the development of more robust sequence-lo-function mappings and facilitating the design of advanced silk- based materials with tailored performance characteristics.
dataset by leveraging
Software and Data
The implementation of our model inference pipeline is available at: spiderf dem The repository includes the necessary scripts, configuration files, and usage instructions. All relevant datasets used in this study are provided in the Data folder within the repository.
Acknowledgments
This work was supported by grants from the SciLifeTab Wallenberg Data Driven Life Science Program (KAW 2020.0239) and by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation: WASP-DDLS 22:035 to AR and HK, KAW 2023.0331 and KAW 2017.0003 to AR, and by the National Bioinformatics Infrastructure Sweden (NBIS) at SciLifeLa.. Support was also received by AR from FORMAS (2023-01313), Olle Engkvist stiftelse (233-0334), and the Swedish Research council (2024-02919), and by HK from the Swedish e-Science Research Centre (SeRC).
References
Namrata Anand, Raphael Eguchi, Irimpan I Mathews, Carla P Perez, Alexander Derry, Russ B Altman, and Po-Ssu Huang. Protein sequence design with a learned potential. Nature communications, 13(1):746,
2022.
Marlene Andersson, Gefei Chen, Martins Otikovs, Michael Landreh, Kerstin Nordling, Nina Kronqvist, Per Westermark, Hans Jörnvall, Stefan Knight, Yvonne Ridderstråle, al. Carbonic anhydrase generates co2 and h+ that drive spider silk formation via opposite effects the terminal domains. PLoS bnoooog, 12(8): e1001921, 2014.
Arthur Markus Anton, Aniela Heidebrecht, Nasir Mahmood, Mario Beiner, Thomas Scheibel, and Friedrich Kremer. Foundation of the outstanding toughness in biomimetic and natural spider silk. Biomacro- molecules, 18(12):3954 3962, 2017.
Kazuharu Arakawa, Nobuaki Kono, Ali D Malay, Ayaka Tateishi, Nao Ifuku, Hiroyasu Masunaga, Ryota Sato, Kousuke Tsuchiya, Rintaro Ohtoshi, Daniel Pedrazzoli, et al. 1000 spider silkomes: Linking sequences to silk physical properties. Science advances, 8(41 :cabo60434 2022.
Glareh Askarieh, My Hedhammar, Kerstin Nordling, Alejandra Saenz, Cristina Casals, Anna Rising, Jan Johansson, and Stefan Knight. Self assemblly of spider silk proteins is controlled by ph-sensitive relay. Nature, 465(5595):236 238, 2010.
Earique Audain, Yassel Ramos, Henning Hermjakob, Darren R Flower, and Yasset Perez-Riverol. Accurate estimation of isoelectric point of protein and peptide based on amino acid sequences. Bioinformatics, 32 (6):821-827, 2016.
Nathaniel Bennett, Brian Coventry, Inna Goreshnik, Buwei Huang, Aza Allen, Dionne Vafeados, Ying Peng, Justas Dauparas, Minkyung Back, Lance Stewart, Frank DiMaio, Steven Munck, Savvas Savvides, and
24

---

Published in Transactions on Machine Learning Research (07/2025)
David Baker. Improving de novo protein binder design with deep learning. Nature Communications, 14, 05 2023. doi: 10.1088//414467--223388222
Katherine Bourzac. Spiders: Web of intrigue. Nature, 519:S4 6, 03 2015. doi: 10.1038/519S4a.
Christopher H Bowen, Bin Dai, Cameron J Sargent, Wenqin Bai, Pranay Ladiwala, Huibao Feng, Wenwen Huang, David Kaplan, Jonathan M Galazka, and Fuzhong Zhang. Recombinant spidroins fully replicate primary mechanical properties of natural spider silk. Biomarromokccules (0(9):3553-3500 2018.
Nadav Brandes, Dan Ofer, Yam Peleg, Naday Rappoport, and Michal Linial. Proteinbert: a universal doep-learning model of protein sequence and function. Bioinformatics, 38(8):2102-211 21100, 2022.
Daniel W. A. Buchan and David T. Jones. The psipred protein analysis workbench: 20 years on. Nucleic
Acids Research, W002 2019. doi: (0.109//nrrgkk22000 URL
nar/ har/ghe gkz297
Peter Y Chou and Gerald D Fasman. Prediction of protein conformation. Biochemistry, 13(2):222 245, 1974.
Peter JA Cock, Tiago Antao, Jeffrey T Chang, Brad A Chapman, Cymon Cox, Andrew Dalke, Iddo Friedberg, Thomas Hamelryck, Frank Kauff, Bartek Wilezynski, al. Biopython: freely available python tools for computational molecular biology and bioinformatics. Bioinformatics, 25(11):1422, 2009.
Hamish Craig, Dakota Piorkowski, Shinichi Nakagawa, Michael Kasumovic, and Sean Blamires. Meta- analysis reveals materiomic relationships in major ampullate silk across the spider phylogeny. Journal of The Royal Society Interface, 17, 09 2020. doi: 10.1098/rste 1t1.0///sf.22000447777
Payel Das, Tom Sercu, Kahini Wadhawan, Inkit Padhi, Sebastian Gehrmann, Flaviu Cipcigan, Vijil Chen- thamarakshan, Hendrik Strobelt, Cicero Dos Santos, Yu Chen, et al. Accelerated antimicrobial dis- covery via deep generative models and molecular dynamics simulations. Nature Biomedical Engineering. 5(6):613 623, 2021.
Justas Dauparas, Ivan Anishchenko, Nathanicl Bennett, Hua Bai, Robert Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning based protein sequence design using proteinmpnn. Science, 378(6615):49 56, 2022.
Giuseppe De Giorgio, Biagio Matera, Davide Vurro, Edoardo Manfredi, Vardan Galstyan, Giuseppe Tara- bella, Benedetta Ghezzi, and Pasquale D'Angelo. Silk fibroin materials: Biomedical applications and perspectives. Bioengineering. 11(2):167, 2024.
Danilo Hirabae De Oliveira, Vasantha Gowda, Tobias Sparrman, Linnea Gustafsson, Rodrigo Sanches Pires, Christian Riekel, Andreas Barth, Christofer Lendel, and My Hedhammar. Structural conversion of the spidroin c-terminal domain during assembly of spider silk fibers. Nature Communications, 15(1):4670, 2024.
Lukas Eisoldt, Christopher Thamm, and Thomas Scheibel. The role of terminal domains during storage and assembly of spider silk proteins. Biopolymers, 97(6))355-361 2012.
Vincenzo Fazio, Nicola Maria Pugno, Orazio Giustolisi, and Giuseppe Puglisi. Hierarchical physically based machine learning in material scionce: the case study of spider silk. arXiv preprint arXiv:2307.11 12945, 2023.
URL https://arxiv httpa////aaxxiiooo/////////e//// http://wrg/bbbb//....../
Noclia Ferruz and Birte Höcker. Controllable protein design with language models. Nature Machine Intelli- gence, 4(6):521 532, 2022.
Noclia Ferruz, Steffen Schmidt, and Birte Höcker. Protgpt2 is deep unsueeviied language model for protein design. Nature communications, 13(1):4348, 2022.
Elisabeth Gasteiger, Christine Hoogland, Alexandre Gattiker, S'everine Duvaud, Mare R Wilkins, Ron D Appel, and Amos Bairoch. Protein identification and analysis tools the ExPASy server. Springer, 2005.
25

---

Published in Transactions on Machine Learning Research (07/2025)
John M Gosline, Mark W Denny, and M Edwin DeMont. Spider silk rubber. Nature, (500)69:551199590000) 1984.
John M Gosline, PA Guerette, CS Ortlepp, and KN Savage. The mechanical design of spider silks: from fibroin sequence to mechanical function. Journal of Experimental Biology: (20((23)3295-3325)) 1999.
Paul A Guerette, David G Ginzinger, Bernhard HF Weber, and John M Gosline. Silk properties determined by gland-specific expression of spider fibroin gene family, Science, 272 272(5228):112 115, 1996.
Kunchur Guruprasad, BV Bhasker Reddy, and Madhusudan W Pandit. Correlation between stability of a protein and its dipeptide composition: novel approach for predicting in vivo stability of protein from its primary sequence. Protein Engineering, Design and Selection, 4(2):155 161, 1990.
Cheryl Y Hayashi and Randolph V Lewis. Evidence from flagelliform silk cdna for the structural basis of clasticity and modular nature of spider silks. Journal of molecular biology, 275(5):773-78k. 1998.
David H Hijirida, Kinh Gian Do, Carl Michal, Shan Wong, David Zax, and Lynn W Jelinski. 13c nmr of nephila clavipes major ampullate silk gland. Biophysical journal, (1(6):3442-3441 1996.
Michael B Hinman and Randolph V Lewis. Isolation of a clone encoding a second dragline silk fibroin. nephila clavipes dragline silk is a two-protein fiber. Journal of Biological Chemistry, 267((7):19320 19324, 1992.
Gooffrey Hinton. Distilling the knowlodge in a neural network. arXiv preprint arXiv:1503. 02531, 2015.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Zidek, Anna Potapenko, et al. Highly accurate protcin structure prediction with alphafold. Nature, 596(7873):583 588558833598888 2021.
Wolfgang Kabsch and Christian Sander. Dictionary of protein secondary structure: pattern recognition of hydrogen-bonded bonded and geometrical features. Biopolymers, 22(12): 1983.
Sinan Keten and Markus J Buchler. Nanostructure and molccular mechanics of spider dragline silk protein assemblies. Journal of the Royal Sociely Interface, 7(53):1709-1722 2010. doi: 10.1098/rsi 1000/rsff.201.0111
Hamed Khakzad, Ilia Igashov, Arne Schneuing, Casper Goverde, Michael Bronstein, and Bruno Correia. A new age in protein design empowered by deep learning. Cell Systems, 14(11):925-933) 2023.
Yoonjung Kim, Taeyoung Yoon, Woo B Park, and Sungsoo Na. Predicting mechanical properties of silk from its amino acid sequences via machine learning. Journal of the Mechanical Behavior of Biomedical Materials, 140:105739, 2023.
Andreas Koeppel and Chris Holland. Progress and trends in artificial silk spinning: systematic review. ACS Biomaterials Science Engineering, 3, 01 2017. doi: 10. http://aaioommaraaaa.ooooo.oo
Nobuaki Kono, Hiroyuki Nakamura, Masaru Mori, Yuki Yoshida, Rintaro Ohtoshi, Ali D Malay, Daniel A Podrazzoli Moran, Masaru Tomita, Keiji Numata, and Kazuharu Arakawa. Multicomponent nature under- lies the extraordinary mechanical properties of spider dragline silk. Proceedings of the National Academy of Sciences, (110001111 2021a.
Nobuaki Kono, Rintaro Ohtoshi, Ali D Malay, Masaru Mori, Hiroyasu Masunaga, Yuki Yoshida, Hiroyuki Nakamura, Keiji Numata, and Kazuharu Arakawa. Darwin's bark spider shares a spidroin repertoire with caerostris extrusa but achieves extraordinary silk toughness through gene expression. Open biology, 11 (12):210242, 2021b.
Randolph V Lewis. Spider silk: the unraveling of mystery. Accounts of chemical research, 25(9):392-36) 398, 1992.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hic, Zhongkai Zhu, Wenting Lu, Nikita. Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shrueli, al. Language models of protein sequences at the scale of evolution enable accurate structure prediction. Science, 378(6617)xadd22)) 7)23d 2022.
26

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Nikita Smetanin, Robert Verkuil, Ori Kabeli, Yaniv Shmueli, et al. Fvolutionary-scale prediction of atomic-level protein structure with a language model. Science, 379(6637):1123 1130, 2023.
Wei Lu, David I, Kaplan, and Markus J Buehler. Generative modeling, design, and analysis of spider silk protein sequences for enhanced mechanical properties. Advanced Functional Materials, 34(11):231134, 2024.
Ali Madani, Bryan McCann, Nikhil Naik, Nitish Shirish Keskar, Namrata Anand, Raphael R Eguchi, Po- Ssu Huang, and Richard Socher. Progen: Language modeling for protein generation. arXiv preprint orXiv:204.034077 2020.
Ali Malay, Kazuharu Arakawa, and Keiji Numata. Analysis of repetitive amino acid motifs reveals the ossential features of spider dragline silk proteins. PLoS One, 12(8):c0183337, 2017. doi: 10.1371 /jorralj.rnal.
pone.0183397.
Frank J Masscy Jr. The kolmogorov-smirno test for goodness of fit. Journal of the American statistical Association, 46(253):68 78, 1951.
Liam J McGuffin, Kevin Bryson, and David Jones. The psipred protein structure prediction server. Bioinformatics, 16(4):404 405, 2000.
Hiroyuki Nakamura, Yusuke Ito, Ryota Sato, Hongfang Chi, Chikako Sato, Yasuha Watanabe, and Kazuharu Arakawa. Correlating mechanical properties and sequence motifs in artificial spider silk by targeted motif substitution. ACS biomaterials science & engineering, 10, 11 2024a. doi: 10.
Hiroyuki Nakamura, Yusuke Ito, Ryota Sato, Honglang Chi, Chikako Sato, Yasuha Watanabe, and Kazuharu Arakawa. Correlating mechanical properties and sequence motifs in artificial spider silk by targeted motif substitution. ACS Biomalerials Science & Engineering, 10(12):7394 7403, 2024b.
Bo Ni, David L Kaplan, and Markus J Buehler. Forcegen: End-to-end de novo protein generation based on nonlinear mechanical unfolding responses using a protein language diffusion model. orXiv preprinl arXiv:2310.1060, 2023. URL attpp://arrxiis
/310.
Erik Nijkamp, Jeffrey A Ruffolo, Eli N Weinstein, Nikhil Naik, and Ali Madani. Progen2: exploring the boundaries of protein language models. Cell systems, 14(11):968 978, 2023.
Dan Ofer, Nadav Brandes, and Michal Linial. The language of proteins: Nlp, machine learning & protein sequences. Computational and Structural Biotechnology Journal, 19:1750 1758, 2021.
David B Peakall. Syuthesis of silk, mechanism and location. American Zoologist, 9(1):71-79, 1969.
Lijun Quan, Tinglang Wu, and Qiang Lyu. Computational de novo protein design: from secondary to primary, then toward tertiary structures. Chem, 9(7):6625 1667,116777100 2023.
Donatas Repecka, Vykintas Jauniskis, Laurynas Karpus, Flzbieta Rembeza, Irmantas Rokaitis, Jan Zrimec, Simona Poviloniene, Audrius Laurynenas, Sandra Viknander, Wissam Abuajwa, et al. Expanding func- tional protein sequence spaces using generative adversarial networks. Nature Machine Intelligence, 3(4): 324-333 2021.
Jeffrey A Ruffolo, Jeffrey J Gray, and Jeremias Sulam. Deciphering antibody affinity maturation with language models and weakly supervised learning, arXiv preprint arXiv:2112.0778. 2021.
Benjamin Schmuck, Gabriele Greco, Tomas Bohn Pessatti, Sumalata Sonavane, Viktoria Langwallner, Tina Arndt, and Anna Rising. Strategies for making high-performance artificial spider silk fibers. Advanced Functional Materials, 34(35)22005040, 2024.
Megija Sede, Jekabs Fridmanis, Martins Otikovs, Jan Johansson, Anna Rising, Nina Kronqvist, and Kristaps Jaudzems. Solution structure of tubuliform spidroin n-terminal domain and implications for ph dependent dimerization. Frontiers in Molecular Biosciences, 9:936887, 2022.
27

---

Published in Transactions on Machine Learning Research (07 (07/2025)
Anshuman Sinha, Camille Migozzi, Aubin Rey, and Chao Zhang. Enhancing audio-language models through
self-supervised post-training with text-audio pairs. arXie preprint orXiv:2408.09266 2024.
Sino Biological. Masp2 proteins resource, 2024. URLLhttps:////
com Accessed: 2025-06-03.
The UniProt Consortium. Uniprot: the universal protein knowledgebase in 2025. Nucleic Acids Research, #3(D1))D009-D61) 11 2024. ISSN 0305-1048. 1093/nar/gkae1010
doi:
URL https://ddo.org///
Olena Tokareva, Valquíria A Michalozechenn-Lrreells Elíbio L Rech, and David L Kaplan. Recombinant dna production of spider silk proteins. Microbial biotechnology, 6(6):651-663 663, 2013.
JD Van Beek, S Hess, F Vollrath, and BH124902 Meier. The molecular structure of spider dragline silk: folding and orientation of the protein backbone. Proceedings of the National Academy of Sciences, 99(16): 10271, 2002.
Fritz Vollrath and David P Knight. Liquid crystalline spinning of spider silk. Nature, 410(6828):541-3333 2001.
Fritz Vollrath and DP Knight. Structure and function of the silk production pathway in the spider nephila edulis. International Journal of Biological Macromolecules, 24(2-3):243 249, 1999.
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eisenach, Woody Ahern, Andrew J Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 620(7976):1089-1100 2023.
Ruidong Wu, Fan Ding, Rui Wang, Rui Shen, Xiwen Zhang, Shitong Luo, Chenpeng Su, Zuofan Wu, Qi Xic, Bonnie Berger, et al. High-resolution de novo structure prediction from primary sequence. BioRaiv, pp- 2022-00 07, 2022.
Yan-Xiang Yang, Zhi-Gang Qian, Jian-Jiang Zhong, and Xiao-Xia Xia. Hyper-production of large proteins spider dragline silk masp2 by escherichia coli via synthetic biology approach. Process Biochemistry, 51 (4):484 490, 2016.
Jeffery L Yarger, Brian R Cherry, and Arjan van der Vaart. Structure and dynamics of silk proteins. Nature Reviews Materials, 3(3):18008, 2018a.
Jeffery L Yarger, Brian R Cherry, and Arjan Van Der Vaart. Uncovering the structure funettro-funcionn relationship spider silk. Nature Reviews Materials, 3(3):1-11, 2018b.
A Appendix
A.1 Distillation Data Acquisition
Protein sequences were retrieved from the UniProt Knowledgebase (UniProtKB) using the UniProt REST API. To obtain high-quality sequences, we filtered entries based on taxonomy ID 6893 and amnotation scores of 2, 3, 4, The following query was used to download the dataset in FASTA format:
https:// rest. uniprot. org/ wwwprotkb/sttermo/eeee woraaa/moooooeesssseeeeeonttaaaa query=(taxonomy__it -dd:: AND (annotation_scorett : OR annotation_ OR annolation 4 OR annotation score:5)
The dataset comprises protein sequences specific to the selected taxonomy, ensuring relevance for further computational analyses.
28

---

Published in Transactions on Machine Learning Research (07 (07/202) /025)
A.2 Choice of Physiochemical Attributes for Evaluation
In current study we have used sequence length, molecular weight, instability index, and isoelectric point physiochemical attributes to evaluate the quality of generated MaSp repeats in various contexts.
Molecular weight and sequence length are widely used as coarse-grained indicators of biological plausibility in protein design. Empirically, naturally occurring major ampullate spidroin (MaSp) proteins fall within a narrow range of lengths and molecular weights, reflecting functional and structural constraints (Yang al, 2016). For instance, MaSp2 proteins often exhibit. high molecular weights due to their repetitive sequences, which are crucial for their mechanical properties (Sino Biological 2024). Deviations from these ranges can indicate sequences that are unlikely to fold correctly or be expressed efficiently, as proper folding is essential for protein functionality.
The instability index, introduced by (Guruprasad et al. 1990) Guruprasad et al. (1990), predicts the in vivo stability of a protein based on its dipeptide composition. In the context of MaSp repeats, evaluating the instability index helps in predicting the stability of the designed sequences, which is crucial for their functional expression and polential applications.
The isoclectric point is the pH at which a protein carries no net electrical charge. It significantly influences protein's ss solubility, stability, and interaction with other molecules. Understanding the pl of MaSp repeat sequences aids in anticipating their behavior under different. pH conditions, which is vital for their purification and application processes (Audain et al. 2016).
In summary, sequence length, molecular weight, instability index and isoclectric point are critical parameters in the evaluation of MaSp repeat sequences, providing insights into their physical constraints, stability and functional suitability in various applications.
A.3 Standard Amino Acids with their Background Frequency & KL Divergence for MaSp Repeats
Like human language, protein sequences can be reprosented as strings of letters, where the protein alphabet consists of 20 standard amino acids (AAs), excluding rare and unconventional ones. Similarly, naturally evolved proteins are composed of modular elements with slight variations, which can be rearranged and assembled hierarchically. In this analogy, common protein motifs and domains fundamental functional units of proteins- are akin to words, phrases, and sentences in human language (Ofer et al. 2021
The twenty amino acids (that make up proteins)each have assigned to them both three-letter (can be upper or lower case) and one-letter codes (upper case). This makes it quicker and easier for notation purposes and are worth learning. Table 4 gives these notations.
For evaluation purposes, we established a background amino acid frequency distribution based on the training dataset of MaSp repeats (6K instance which were used in Stage 2). This distribution represents the mean occurrence of each amino acid across all sequences in the dataset. The calculated background frequencies are shown in Table
This frequency distribution serves as reference bascline for KL divergence calculations, enabling a quanti- falive comparison between generated sequences and naturally occurring MaSp repeats.
To assess sequence similarity, we calculate the Kullback-Leibler (KL) divergence between the amino acid composition of generated sequences and the background distribution of MaSp repeats. The background frequencies of amino acids were derived from a dataset of 6,000 MaSp repeat sequences, providing a reference probability distribution. Given a sequence S, its amino acid probability distribution / is compared against the background distribution Q using:
P(i) Q(1
(kI()))Q) = Σ P(i) log
(2)
where P(i) and Q(i) represent the probabilities of amino acid i in the generated sequence and background dataset, respectively. This metric quantifies the deviation of generated sequences from natural MaSp repeats, aiding in model evaluation.
29

---

Published in Transactions on Machine Learning Research (07/2025)
Table 4: Amino Acids, Their Codes, and Background Frequency Distribution for MaSp Repeats
Amino Acid Name
3-Letter Code
1-Letter Code
Background Frequency
Alanine
Ala
A
0.2232
Arginine
Arg
R
0.0129
Asparagine
Asn
N
0.0070
Aspartic Acid
Asp
D
0.0078
Cysteine
Cys
C
0.0002
Glutamine
Gln
Q
0.0850
Glutamic Acid
Glu
E
0.0069
Glycine
Gly
G
0.3766
Histidine
His
H
0.0003
Isoleucine
Tle
0.0050
Leucine
Leu
L
0.0138
Lysine
Lys
K
0.0017
Methionine
Met
M
0.0014
Phenylalanine
Phe
F
0.0038
Proline
Pro
P
0.0788
Serine
Ser
S
0.1004
Threonine
Thr
T
0.0123
Tryptophan
Trp
W
0.0002
Tyrosine
Tyr
Y
0.0485
Valine
Val
V
0.0141
30


Reviews:
[{"review_id": "8FzjSpLKdR", "reviewer": "Reviewer_Hf5q", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n*   **Novel Computational Framework:** The paper introduces a **novel computational framework** for engineering protein sequences, specifically focusing on the repeat regions of major ampullate spidroins (MaSps) to customise mechanical properties. This framework addresses the challenge posed by limited data connecting sequences directly to mechanical properties.\n*   **Dual Functionality:** The developed model serves a **dual purpose**, capable of both **generating new MaSp repeat sequences conditioned on desired mechanical properties** (the \"forward task\") and **predicting mechanical properties for a given MaSp repeat sequence** (the \"reverse task\"). This offers significant flexibility for both design and analysis.\n*   **Efficient Model Development:** The approach involves **distilling a large pre-trained protein language model (ProtGPT2) into a smaller, more efficient model called SpiderGPT**. Ablation studies confirm that this distillation significantly **reduces model size and increases inference speed** (six-fold faster) with only minimal performance trade-offs in perplexity levels.\n*   **Effective Multi-Level Fine-Tuning:** The methodology employs a **multi-stage fine-tuning process** (first on 6,000 general MaSp repeats, then on 572 with known mechanical properties). This staged approach is crucial, as ablation studies show that skipping the first fine-tuning level leads to the **generation of biologically implausible sequences** containing non-repeat regions and significantly **reduces accuracy in property prediction**.\n*   **Comprehensive Validation:** A **comprehensive validation methodology** is used, including sequence-level analysis (physicochemical properties, motif distribution), structural prediction, and mechanical property correlation.\n*   **Biologically Plausible and Novel Generation:** The generated sequences were shown to be **biologically plausible**, matching the distributions of physicochemical attributes, key motifs (poly-Ala, GGX, YGQGG, SV), and predicted secondary structures found in natural MaSp repeats. BLAST analysis indicated the model can **generate novel sequences** that are distinct from existing ones while retaining characteristic MaSp features.\n*   **Demonstrated Predictive Capability:** The model demonstrated the capability to estimate mechanical properties, achieving a **statistically significant correlation** between predicted and reference properties on a test set. A **high cosine similarity (0.9465)** suggests strong alignment in capturing the *trends* of mechanical properties.\n*   **Practical for Biomaterial Design:** The work provides a **practical and scalable approach** for designing synthetic spider silk proteins with tailored mechanical properties, contributing to **sustainable biomaterial development**.\n\n**Elements Requiring Attention (Potential Weaknesses / Limitations Noted):**\n\n*   **Literature review:** The literature review seems to be highly restricted to the niche field of fibre studies. While the concepts of deep learning which have been used in the current work, have already been worked upon extensively in the past.\n*   **Limited Data:** The authors explicitly state that the target dataset mapping MaSp repeats to mechanical properties is **\"extremely small\"** (592 instances). While the multi-stage fine-tuning is designed to mitigate this, the inherent data scarcity remains a challenge.\n*   **Weak Correlation of Individual Motifs:** Consistent with previous studies on natural silk, the analysis of individual motif occurrences and their correlation with mechanical properties in the *generated sequences* shows **relatively low or weak correlation values**. This reinforces the idea that silk mechanics result from a complex interplay of multiple motifs and structural contexts, but it means the model's direct mapping of single motifs to properties is limited.\n*   **Sensitivity in Validation:** The authors note that variations observed in the BLAST comparison metrics (Figure 6) might be due to the **sensitivity of the evaluation process to the chosen dataset** of reference sequences.\n*   **Moderate Linear Correlation in Property Prediction:** Although the cosine similarity is high (0.9465) indicating good trend prediction, the **Pearson correlation coefficient (r) for predicted vs. reference mechanical properties is 0.3911**, described as a \"moderate positive correlation\". While significant in context, it suggests the model's ability to precisely predict absolute property values might be less strong than its ability to capture relative trends.\n*   **Reliance on Standard Structural Prediction Tools:** The paper uses standard secondary structure prediction methods (like those based on DSSP/Chou-Fasman) but notes that these are **\"less reliable for structural proteins like MaSp repeats\"** compared to globular proteins.\n*   **Experimental Validation Pending:** A key piece of future work is the **comprehensive experimental validation** of the generated sequences by synthesizing and mechanically testing them. The current validation is primarily computational; experimental verification is necessary to fully confirm the practical utility and predicted properties of the sequences.\n*   **Need for Larger Datasets:** Future work mentions the plan to **integrate significantly larger and more diverse datasets** to improve the predictive accuracy and generalizability of the model, implying that the current dataset size is a limitation to achieving higher performance.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "**Summary Of Contributions**\n\n*   The paper presents a **novel computational framework** designed to engineer protein sequences for spider silk, specifically focusing on the **repeat regions of major ampullate spidroins (MaSps)**. This framework aims to enable the **customization of mechanical properties** in synthetic spider silk materials. It addresses the challenge of limited data connecting MaSp repeat sequences directly to mechanical properties by employing a **multi-stage fine-tuning approach**.\n*   A key contribution is the development of a **dual-purpose generative model**. This model is capable of two main tasks: **generating novel MaSp repeat sequences conditioned on desired mechanical properties** (the \"forward task\") and **predicting the mechanical properties for a given MaSp repeat sequence** (the \"reverse task\"). This flexibility supports both the design of new sequences and the analysis of existing ones.\n*   The methodology involves distilling a large pre-trained protein language model, ProtGPT2, into a smaller, more efficient model called **SpiderGPT**. This distilled model undergoes two levels of fine-tuning using subsets of the Spider Silkome dataset. The first level tunes the model on 6,000 MaSp repeat sequences to learn their general patterns, while the second refines it using 572 sequences with known mechanical properties to establish sequence-property correlations. Ablation studies highlight the importance of both distillation for computational efficiency and the initial fine-tuning stage for ensuring the generation of valid MaSp repeat regions and improving property prediction accuracy.\n*   A **comprehensive validation methodology** was employed to evaluate the generated sequences. This included sequence-level analysis of physicochemical properties (like molecular weight, instability index, isoelectric point, and amino acid distribution), assessment of key motif distributions (such as poly-Ala, GGX, YGQGG, and SV), structural prediction to check secondary structure composition (like α-helices and β-strands), and evaluation of mechanical property correlation.\n*   The paper empirically demonstrates the model's effectiveness. Generated sequences were shown to be **biologically plausible**, closely matching the distributions of physicochemical attributes and secondary structures found in natural MaSp repeats. They also maintained distributions of key motifs similar to natural sequences. A BLAST-based novelty assessment indicated the model can generate diverse sequences while retaining the characteristic features of MaSps. For mechanical property prediction, the model achieved **statistically significant correlations** between predicted and reference properties, with a high cosine similarity of 0.9465 for trend curves, suggesting strong predictive performance. Although individual motif-property correlations remained weak, consistent with previous findings on natural silk, the model's overall property estimation accuracy was demonstrated.\n*   The work provides a **practical and scalable approach** for designing synthetic spider silk proteins with tailored mechanical properties. This is presented as a significant contribution to **sustainable biomaterial development** with potential applications in diverse fields such as medicine, textiles, and engineering.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "List of proposed adjustments to the submission:\n\n**Critical for Acceptance:**\n\n*   **Comprehensive Literature review:** The important multi-stage method discussed over here (in section 2.4) and its major contributions are interesting, but it has been discussed in previous publications in different scientific areas. One of the recent publications submitted in this year's ICLR implements a multi-stage model for audio sciences highly similar to the current work. Described in major contribution as \"two-step post-training\" in (https://arxiv.org/abs/2408.09269), kindly include such works and others that you may find appropriate to your work. \n\n*   **Schematic of model overview** Kindly include a proper figure for the architecture that gives the overview of the model, input, outputs and conditions. Also, include an explanation of your data (using mathematical notations) and proper set notation.\n\n*   **Important proof of propositions** While you are using a generative model, kindly include a note as to why you think the generated output is correct and preserves the physical attributes of the input dataset (i.e if the input dataset follows some material properties, then why will the output follow such properties). It is important to address this while you are developing generative models for scientific applications. Kindly refer to the proposition from A.1.1 (https://arxiv.org/pdf/2408.07213) and section A.1-A.2 (https://arxiv.org/pdf/2307.02707). (Kindly do not get perplexed by the mathematics, the authors are only requested to understand the importance of these propositions while using a generative model of any kind for a physical system like the one discussed in this paper).\n\n*   **Comprehensive Experimental Validation:** The most significant adjustment critical for acceptance would be the **initiation or inclusion of experimental validation** of the generated sequences. The paper currently states this as future work. While the computational validation is thorough, demonstrating the practical utility of the framework by synthesizing and mechanically testing at least a representative sample of generated sequences to verify their predicted properties is often essential for applied biomaterial design research. This step is crucial to verify that the predicted properties align with actual performance characteristics.\n\n**Strengthens the Work:**\n\n*   **More Explicit Discussion of Data Limitations:** While the multi-stage fine-tuning is presented as a solution to the challenge of the \"extremely small\" target dataset, the paper could benefit from a more in-depth discussion of the **potential impact of this inherent data scarcity** on the model's limits in terms of the diversity of properties it can condition on, the precision of its predictions, and its generalizability to property ranges significantly outside those represented in the 592-instance dataset.\n*   **Further Elaboration on Weak Motif-Property Correlations:** The finding that individual motifs show only weak correlations with mechanical properties, consistent with previous research, is mentioned. Expanding the discussion on the **implications of this inherent complexity** of spider silk mechanics for computational design would be valuable. This could reinforce why a holistic, data-driven model capturing complex interplay is necessary, rather than relying on simple motif engineering.\n*   **Addressing Limitations in Standard Structural Prediction:** The paper notes that standard secondary structure prediction methods are \"less reliable for structural proteins like MaSp repeats\". While using these methods is standard practice, a brief discussion of alternative, more specialised structural prediction methods or a clearer acknowledgement of the **uncertainty associated with the structural predictions** when interpreting results would strengthen this section.\n*   **Quantification or Mitigation of Validation Sensitivity:** The authors mention that variations in BLAST comparison metrics might be due to the \"sensitivity of the evaluation process to the chosen dataset\". Providing more detail on **how the reference BLAST set was curated** to minimise bias or attempting to quantify this sensitivity through comparisons against multiple different reference sets could strengthen the novelty assessment.\n*   **Improving or Discussing Moderate Linear Property Correlation:** While the cosine similarity is high (0.9465) indicating good trend capture, the Pearson correlation coefficient (r) for predicted vs. reference properties is described as a \"moderate positive correlation\" (0.3911). While statistically significant, discussing potential avenues for **improving the linear accuracy** of property prediction or explaining the potential reasons for this moderate correlation (e.g., data noise, inherent system complexity) could enhance the paper.\n*   **Strengthened Commitment to Future Data Expansion:** Reiterate the importance of acquiring **significantly larger and more diverse datasets**. While stated as future work, providing more concrete plans or strategies for this data acquisition and integration would underscore a clear path towards overcoming current limitations and improving model accuracy and generalizability.", "broader_impact_concerns": "No concerns as such.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "4ONZeX30Vp", "reviewer": "Reviewer_V9sT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n- The manuscript provides an excellent introduction to the subject matter and successfully conveys the importance of the topic.\n- The manuscript provides a clear, high-level overview of the proposed multi-level fine-tuning methodology and the motivation behind it.\n\n### Weaknesses\n- If I understand correctly, the test sets used in Sections 5.1.1 and 5.1.2 consist of 20 and 7 sequences, respectively, without any kind of bootstrapping. This is extremely small, and I find it difficult to draw strong conclusions about model performance based on such small sample sizes.\n- The paper does not include any baseline methods and only presents a limited set of ablations for the proposed approach.\n- As far as I could tell, no code is provided to reproduce the empirical results reported in the manuscript. In the absence of an accompanying codebase, the description of the methodology in the Training Process paragraphs on pages 6, 7 and 8 seems incomplete, e.g. listing LoRA hyperparameters without defining the corresponding algorithm or providing an appropriate reference.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This manuscript proposes a multi-stage workflow for fine-tuning a pre-trained protein language model (PLM) on the repetitive regions of spider silk proteins (MaSps). In the first stage, a smaller version of a generalist PLM (ProtGPT) is distilled on ~100k spider protein sequences. This model is then fine-tuned on ~6k MaSp sequences to capture their characteristic sequence motifs. Finally, the model is fine-tuned again on ~600 property-annotated MaSp sequences, which are augmented by task and property tokens to enable conditional sampling and property estimation.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- It would be great to incorporate more comparisons and baselines into the results section of the paper, e.g., including sequences from the base ProtGPT model in the property comparisons in Figure 3 or comparing the method to existing approaches for mechanical property prediction and spindroin sequence design listed in Section 2.3.\n- It would also be helpful to improve the robustness of the experimental evaluation itself, either by using larger test sets or performing bootstrapping or k-fold cross-validation over the test set. \n- I found that there was some overlap in content between Sections 3, 4, and 5. I believe that parts of it could be condensed (or moved to the appendix) to create a more streamlined description of the methodology and experimental results.\n- I also believe that some of the claims should be toned down. For instance, the Training Process paragraph on page 7 describes dropout, weight decay, and early stopping as \"advanced optimization techniques\". Similarly, top k model checkpointing is described as a \"sophisticated checkpointing system\". In my opinion, these are standard techniques and should not be portrayed as advanced methodological achievements.", "broader_impact_concerns": "I have no concerns about the negative ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "UufH6oRmRR", "reviewer": "Reviewer_EX2U", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is well-written, clear and easy to follow.\n- The components of proposed methodology are well motivated and tested in comprehensive ablation studies.\n\nWeaknesses\n- The performance of mechanical property prediction/design is poor (Sec. 5.2).\n- The theory on MaSp could be better elaborated (see below).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a model for both forward prediction and inverse generation of sequence–property of a key ingredient in spider silks, specifically, MaSps repeat regions and mechanical properties. The model is developed by distilling a pre-trained protein language model, ProGPT2, and fine-tuning it in two stages. Experiments evaluate the plausibility, novelty, and accuracy of the model’s (conditional) generation, as well as the importance of distillation and fine-tuning (through ablation studies).", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Elaboration or clarification\n- (Chemical/biological) theory behind the role of MaSp (and its repeat regions) in determining mechanical properties could be better elaborated in Sec. 2.1. \n  - A related question: In level 1 fine-tuning, the first and last amino acid residues are removed, but could they have interactions with the MaSp repeat regions? That is, changing these residues changes the correlation between MaSp sequence and property.\n- The role of training parameters (T, α, etc.) used in Stage 1 should be explained, at least qualitatively.\n\nExperiments and evaluation\n- Physiochemical attributes such as molecular weight and sequence length are used to assess biological plausibility. Are there theoretical or empirical justifications that these attributes are related to plausibility?\n- Sec. 5.1.1 compares the generated and natural sequences that correspond to the same mechanical properties, but this comparison is only meaningful if the sequence–property mapping is one-to-one. This comparison should be justified.\n- In Sec. 5.2, Pearson r ~ 0.39 corresponds to R2~0.15, which is too low. Since the goal is assess “overall … trends”, perhaps Spearman r could be a useful metric to try.\n\nMinor issues\n- There are some clarity issues:\n  - Sec. 4.3, “synthesized” is misleading. Should it be “generated”?\n  - Typos, e.g., “row vise analysis” on Page 12.\n  - Some terms should be unified, e.g., “repeat region” or “repetitive region”.\n- In the main contributions, “practical contribution to sustainable…” seems overclaimed, given the model performance and lack of practical demonstration.", "broader_impact_concerns": "None.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)