Predicting Oligomeric states of Fluorescent Proteins using Mamba

Venue: NLDL_org_2025_Conference
Published: 2024-09-09 08:07:39

Abstract: Fluorescent proteins (FPs) are essential tools in biomedical imaging, known for their ability to absorb and emit light, thereby allowing visualization of biological processes. Understanding the oligomeric state is crucial, as monomeric forms are often preferred in applications to minimize potential artifacts and prevent interference with cellular functions. Experimental methods to find the oligomeric state can be time-consuming and expensive. Most of the current computational model is CPU-based, limiting their speed and scalability. This paper studies the effectiveness of GPU-based deep-learning models on predicting the oligomeric states of fluorescent proteins directly from their amino acid sequences specifically focusing on the Mamba architecture. Various protein-specific augmentations were also employed to enhance the model's generalizability. Our results indicate that the mamba-based model achieves accuracy and F1 score close to 90\% and an MCC value of 0.8 with  in predicting the oligomeric states of fluorescent proteins directly from its amino acid sequence.

Main paper: 
 methods: 
 In this section, we outline the methods and techniques employed for our study.We employ five models for our study.The k-Nearest Neighbour (kNN) algorithm uses pairwise sequence similarity as a distance metric and classifies a sequence as the label that occurs most often among the labels of the k sequences most similar to it according to pairwise similarity. This procedure is motivated by the Basic Local Alignment Search Tool (BLAST), which is widely used in biotechnology and bioinformatics.The S6 state space model Mamba [18] has emerged as a promising architecture for sequence EQUATIONIn an attempt to improve the generalizability of the model and improve the size of the dataset, we apply various protein-specific augmentations to our dataset mentioned by Shen et al [22] and Sun et al [23] .We chose protein-specific augmentations because traditional methods of augmentations can change the functional aspect of the protein and may affect the model's performance negatively. The augmentations employed on amino acid sequence S = {S i } were:Random Substitution(A1): With a probability of 10%, we randomly substitute an amino acid with the amino acid most similar to it to maintain its properties. The substitution mappings used were those optimized by Shen et al.The amino acid sequence is reversed to read from the C-terminal to the N-terminal instead of the usual N-terminal to Cterminal. This changes the protein structurally and functionally but could improve the model's recognition of local features.Subsequence Sampling(A3): Randomly select an amino acid from the sequence and form a subsequence along with the 49 amino acids following it(if the chosen amino acid is less than 49 amino acids away from the final amino acid, then select all amino acids till the final amino acid). Treat the subsequence as the augmented sequence. This is another method to try and encourage the model to recognize local features.Subsequence Shuffling(A4): Select a subsequence from the sequence by a similar procedure to A3 but of length 20. Shuffle the amino acids of this subsequence and replace them in the initial sequence. This promotes the recognition of properties such as amino acid counts, which do not depend on the order of the amino acids in a sequence.Random Swap(A5): Each amino acid in the sequence exchanges its position with another amino acid in the sequence with a probability of 10%.Random Deletion(A6): Each amino acid in the sequence is deleted with a probability of 10%.  
experimental setup 306: 
 Two testing methods used in the study are: This process is repeated and averaged over 100 random seeds.The models were evaluated on the metrics of Accuracy, F1 score(F1) and Matthews Correlation Coefficient(MCC).MCC is evaluated by equation.2M CC = T P ×T N -F P ×F N (T P +F P )(T P +F N )(T N +F P )(T N +F N ) (2)The MCC metric is found to handle data imbalances better than both Accuracy and F1 score as well as provide symmetrical importance to both positive and negative predictions.The metrics we employed were to obtain direct comparisons with Simeon et al (2016) .Sensitivity to hyperparameters such as state size for Mamba, Hidden size for RNN, LSTM, and Transformer, and number of blocks in Mamba were also studied. The computation cost in GMACs (1GMAC= 10 9 MACs) and total Parameter Count for each model were also monitored throughout.A MAC(Multiply-Accumulate) operation constitutes computing a multiplication of two floating point numbers followed by an addition of the product to an accumulator. During the forward pass, MAC operations are used to compute the weighted sums in neurons, which are then passed through activation functions and during backpropagation, MAC operation is involving in computing gradients. A singular MAC operation is given by the equation 3 where a,b and c are floating point variables.EQUATIONAll experiments were executed on a Linux server equipped with Intel(R) Xeon(R) Gold 6138 CPU @ 2.00GHz, 512 GB RAM, and 4 NVIDIA GeForce RTX 2080 Ti with 11GB VRAM each. 
results: 
 The results for the tests mentioned in 3 are provided. First, we compare the effects of data augmentations with the baseline Mamba model on the 10CV test. Then, we compare the results of all the models with the best augmentation from the previous results in 10CV and the independent dataset, along with their computation cost and parameter size. Finally, we try to optimize the hyper-parameters to find the model with the lowest computation cost without compromising on performance metrics. The effects of various data augmentation techniques provided in sec.3.2 are documented in table.1. All six mentioned augmentations were applied individually and the three augmentations with an accuracy greater than or equal to the baseline model were combined and tested. A combination of random substitution and sequence reversion proved to be the most effective, performs better than other augmentations in all metrics, and achieves an accuracy close to 90% and an MCC greater than 0.8. The ID test results for the kNN, baseline Mamba as well as Mamba running on the dataset augmented by random substitution and sequence reversion are provided in table 2.The models were benchmarked with 10CV test and ID test with the dataset provided by Simeon et al [7] .The models were also benchmarked in self curated dataset from FPbase through 10CV. All the results are provided in tables 3, 4, 6.In order to find a set of optimum hyperparameters to minimize the computational cost and parameter count while still maintaining high-performance met- The data augmentation techniques did not provide any drastic improvements on the ID test but nevertheless improved the model to achieve an MCC > 0.8 and an accuracy and F1 score close to 90% in the 10CV test.Additionally, considering the linear scaling of the Mamba model w.r.t sequence length, our model could perform well on the prediction of oligomeric states and other properties of general protein sequences. Furthermore, new and improved proteinspecific augmentations could be experimented on the dataset to possibly enhance the results further. 
: 
 structure of DsRed can lead to complications; for example, Mizuno et al [1] demonstrated that DsRed aggregation disrupts the normal function of calmodulin in the cytosol. Similarly, Zacharias [2] highlighted that the oligomerization of fluorescent proteins can interfere with the signaling pathways of target proteins when these proteins are used as tagging probes in fluorescence resonance energy transfer (FRET) experiments.This necessitates the development of computational models to predict the oligomeric state of fluorescent proteins directly from their protein sequences. Such predictive capabilities could help in selecting or engineering fluorescent proteins with desired oligomeric properties. Experimentally, researchers rely on in vitro techniques like gel filtration, ultracentrifugation, and electrophoresis or in vivo techniques like yeast-2-hybrid screens and FRET [3] . All these experimental techniques consume significant time, cost, experimental resources, and expertise in protein biology. Given these challenges in finding the oligomeric state analytically, data-driven approaches like machine learning have gained notable interest. However, the lack of sufficient data and standard benchmarks poses significant challenges in developing robust predictive models and benchmarking them.In this study, we propose a machine learning algorithm based on the Mamba architecture to predict the oligomeric state directly from protein sequences. Mamba has been used across various domains, including vision [4] , video [5] medical image classification [6] . Our contributions include: (1) This paper presents the first application of the Mamba architecture to protein classification, demonstrating its effectiveness in addressing this problem. (2) The model is built on the Mamba architecture, enabling scalability using powerful GPUs, which overcomes the limitations of previous models that predominantly relied on CPU-based decision trees. (3) Achieved superior performance in sensitivity analyses compared to RNN and LSTM models while also reducing computational costs and utilizing fewer parameters. (4) Achieved accuracy of 89.97% and an F1 score of 89.28% in ten-fold cross-validation on a dataset curated by Simeon et al. [7] , performing better than the model developed by Simeon et al. and all other known deep learning models for this specific problem, to the best of our knowledge. 2 Literature Review toencoders to compress the amino acid sequence data, transforming 5237 nodes into a more manageable set of 32 nodes. This compression aimed to retain essential features necessary for predicting oligomeric states. A decision tree was then incorporated to enhance the predictive power of the autoencoder by identifying significant amino acids influencing the oligomeric state. This decision tree contributed 8 more nodes to already existing 32 nodes from the encoder, resulting in a total of 40 nodes to encode a protein. This encoded protein is then passed into a neural network consisting of 4 dense layer and a gaussian noise layer for the prediction task. The combined approach achieved a prediction accuracy of 70%. Since Agarwal et al (2020) did not specify their evaluation method, it is not possible to directly compare the performance of our model with theirs, especially considering the differences in test-train splits and whether they used single-run values or cross-validations. In the end, the study underscores the broader application of deep learning models in predicting the oligomeric state of fluorescent proteins.Tam and Zhang (2021) [17] developed FPredX, a machine learning-based tool for predicting key properties of FPs from amino acid sequences. The study addresses the challenge of accurately predicting FP properties such as excitation and emission maxima, brightness, and oligomeric states, which are critical for various applications in bioimaging and molecular biology. FPredX utilizes gradient-boosted decision tree models trained on a dataset curated from FPbase. They benchmarked their approach against the dataset curated by Simeon et al. (2016) [7] . FPredX achieved superior results, with precision, recall, and F1 scores of 91.3%, 95.5%, and 93.3%, respectively, performing better than the "osFP" model, which had 84.1%, 87.9%, and 85.9% in the same evaluation. 
 
 Appendix:  


Decision: The paper proposes a method using the Mamba deep learning architecture to predict the oligomeric state (monomeric vs. oligomeric) of fluorescent proteins based on their amino acid sequences. The model is tested for accuracy, sensitivity, and computational efficiency against traditional methods, including RNN, LSTM, and Transformer models, as well as gradient-boosted decision trees (FPredX), as well as CPU-based models ( in an updated version of the manuscript). Various data augmentation methods are also applied to enhance generalizability.

Strengths:
1) The proposed work effectively applies the Mamba architecture to predict oligomeric states of fluorescent proteins. It achieves high accuracy while reducing computational costs compared to traditional CPU-based methods.  
2) Applying Mamba ( a newly emerged framework) to protein sequence analysis offers a new direction in handling limited datasets in protein classification tasks.
3) By shifting from CPU to GPU processing, the model has shown efficiency improvements that could be relevant to various biomedical and molecular research applications.  
4) To improve the model's generalizability for limited datasets, the authors have also introduced some novel protein-specific data augmentation techniques, particularly useful in protein classification tasks.
5) Testing Mamba's potential beyond DNA modeling provides new insights, and including a reference to the codebase enhances transparency.
6) The work offers the potential for broader biological application. For example, the proposed work has the potential to contribute insights into protein structure prediction and fluorescent protein use.
7)The manuscript's current version is well-written, well-structured, and meets academic standards.

Weaknesses:
1) Reviewers discuss that the novelty and significance of the work, particularly in terms of biological insights and technical advancement, need some improvements and make the paper potentially interesting to a limited audience.
2) The first version of the manuscript lacks a direct comparison with FPredX and CPU-based methods. However, the authors have addressed these issues in the revised version of the model.
3) The introduction in the first version of the manuscript needs more background/ references on the Mamba architecture, and specific terms (e.g., MCC, SiLu) and dataset details need to be clarified. However, the authors have addressed these issues in the revised manuscript. 
4) The first version of the manuscript contains some methodological and structural clarity issues, as well as some issues regarding presentation quality. For example, references for certain figures, more details on hyperparameter optimization, and experimental results are needed. However, these issues have been improved in the revised manuscript.

In sum, in the revised version of the manuscript, the authors have addressed most reviewer concerns, particularly the lack of comparison with previously developed methods in the field, especially FPredX and CPU-based methods. Moreover, the current version of the work is well-written and meets academic standards. Given the authors' efforts to address the major concerns of reviewers and generally positive consensus among the reviewers after revision (one acceptance, two-week acceptance, and one weak rejection), it seems reasonable to lean toward recommending acceptance and oral presentation.

Reviews: review: title: title: Review of the paper "Predicting Oligomeric states of Fluorescent Proteins using Mamba"
paper_summary: paper_summary: This paper presents a method for categorizing the oligomeric states of proteins based on their amino acid sequences. The novelty of the research lies in its use of deep learning frameworks, particularly the recently developed Mamba architecture. The authors demonstrates that this approach outperforms other deep learning methods, such as LSTM, RNN, and Transformer models, in terms of accuracy, sensitivity, and computational efficiency.
main_review: main_review: Based on the revision and comments from the authours in the process of rebuttal, and the discussions among the reviewers as followed, I belive the paper can be presented at NLDL 2025.
strength_weakness: strength_weakness: The literature review mentions the Gradient Boost decision tree-based method FPredX, which reportedly achieves F1 scores of 93.3%, significantly higher than the results presented in this paper. Why is this not addressed in the conclusion? A decision tree approach would also offer superior explainability, a topic that is notably absent from the discussion. While lower computational costs (GPU vs. CPU-based, as mentioned) are advantageous, should this come at the expense of accuracy and sensitivity? Additionally, no direct metric comparison between FPredX and this method is provided, so can we be certain this approach performs better in terms of computational time? Given the brief format of this conference paper, I recommend shortening the somewhat lengthy literature review and focusing more on discussing these critical aspects.

The paper compares performance to other deep learning frameworks, such as RNN, LSTM, and Transformer, but does not specify which implementations are used. Upon reviewing the code, it appears that standard PyTorch API calls are employed for testing these, but this should be explicitly mentioned in the text.

While the paper claims that hyperparameter optimization was performed, it does not detail the method used. Was grid search employed? From what I gather in section 5.3, the smallest Mamba model achieves the best performance... Furthermore, the sensitivity tests in section 5.3 combine various augmentations to evaluate their effects, but only 11 combinations are tested, despite there being 6 different techniques, which could result in 64 possible combinations. The rationale for selecting these 11 baselines is not explained.

Additionally, the graphs in Figures 3-5 show that the Mamba parameter size stops at 10^8, several orders of magnitude smaller than those of the other methods. Why?

Finally, there are a few minor grammatical issues. Ensure that all acronyms, such as MCC and MAC, are defined. In line 206, the sentence '...overfitting of data, following which a dense layer...' could benefit from rewording. Similarly, in line 264, the phrase '...attempted to reproduce a few test results...' needs clarification for greater precision. Also, in line 147, there is a punctuation mark following a reference, while the sentence continues. Please review for clarity and correctness.
title: title: A neural network approach to predict state of fluorescnet proteins such as GFP and dsRed.
paper_summary: paper_summary: Fluorophores used in biomedical imaging may consist of a single fluorescing unit, or a cluster of several units, influencing their ability to provide a useful signal. This paper investigates the possibility to predict if proteins exist as single or grouped units, based on the sequence of the protein. 
This field of applying deep neural networks is a bit far from my expertise, but I have done my best in asses strengths and weaknesses. First of all, the strengths include approaches for improving on the output from a tool called Mamba, by having large focus on reducing computational time by moving from a slower CPU-based approach to faster and more efficient GPU processing. The number of papers in the field is limited, but the authors claim they outperform a previous studies utilizing models such as RNN and LSTM.
main_review: main_review: Using deep learning methods to categorize proteins as forming aggregates or not based on their amino acid sequence is a rather narrow field, and likely interesting for a limited audience. The major drawback of the paper is the lack of comparison with previous developed methods in the field (other reviewers point to FPredX); however, the authors plan to do this comparison prior to submitting their final  photo-ready version, according to the reply to @m2uG. Apart from these two concerns, the paper is now, with some revisions. very well written and nicely structured. I therefore recommend a weak accept.
strength_weakness: strength_weakness: Due to its slightly limited novelty and lack of information to convey the purpose of the project, I will rank it lower.