KXingLab/BERTGH29

GH29BERT

This repository contains the code and testing sequence data for reproduce the prediction results for GH29BERT, a protein functional cluster prediction model devised for GH29 family sequences. It is trained based on a semi-supervised deep learning method with: - a. 34,258 unlabeled and non-redundant GH29 sequences (i.e., unlabelled data) extracted from CAZy and Interpro databases and - b. 2,796 labelled sequences with 45 cluster classes based on a thorough SSN analysis. Specifically, the reproducible testing materials (code and data) on following two types of GH29 sequences used in submitted manuscript are provided, including: - 559 labelled GH29 testing sequences (2,796 labelled data with a random 80%-20% split for training and testing), see file data/test.fasta - 15 held-out characterized sequences that was excluded from both pre-training and task-training, see file data/15_seq_for-test.fasta

Interactive deployment of GH29BERT for prediction testing

GH29BERT model is also accessible through a friendly user-interface on HuggingFace: https://huggingface.co/spaces/Oiliver/GH29BERT. It is easier to test the above provided GH29 sequences or your custom GH29 sequence using this web tool.

Prerequisites

 ### Repository download
    To get started, clone this repository, e.g., execute the following in the terminal: `git clone https://github.com/ke-xing/GH29BERT.git`
 ### Environment preparation
    Please check all the useful packages in the file **environment.yml**.   
      With the help of [Conda](https://docs.conda.io/projects/conda/en/stable/user-guide/getting-started.html), run `conda env create --file environment.yml` to create an independent environment for implementing the testing  
 ### Model parameter download
    Due to the limit of single file size of GitHub repository, we upload the model parameter files at [Zenodo open repository](https://zenodo.org/records/10614689)
        - GH29BERT
              ```python
              # Load GH29BERT pre-trained model
              GH29BERT=torch.load('transformer1500_95p_500.pt')
              GH29BERT=GH29BERT.module
              GH29BERT=GH29BERT.to('cuda:0')
              # Load GH29BERT task model
              downstream_GH29BERT=torch.load('down_model_500_kfold1.pt').to('cuda:0')
              ```
        - ProtT5-XL
            - Reproducing prediction testing based on pre-trained ProtT5-XL requires installing extra dependency libraries:
                  ```
                  pip install torch
                  pip install transformers
                  pip install sentencepiece
                  ```
                - For more details, please follow the instructions of [ProtTrans](https://ieeexplore.ieee.org/document/9477085) repository from [github](https://github.com/agemagician/ProtTrans/?tab=readme-ov-file).
              ```python
              from transformers import T5Tokenizer, T5EncoderModel
              
              # Load ProtT5_XL pre-trained model
              ProtT5_XL=T5EncoderModel.from_pretrained("Rostlab/prot_t5_xl_half_uniref50-enc",cache_dir='./').to('cuda:0')
              # Load ProtT5_XL task model
              downstream_ProtT5_XL=torch.load('down_model_500_kfold1.pt').to('cuda:0')
              ```

• Cluster prediction

  Run python python test.py for predicting the fasta data. Model and data loading directory should be adjusted if need.

• Representation visualization

  The visualization of GH29 representations with GH29BERT or other pre-training models can be implemented through python visualization by UMAP.py for obtaining the dimension-reduced intermediate representations and run python figure1.py figure2.py to get the visualization map.

• Code for model training

  We also provide the model training code for pre-training and downstream task-training. Run python Pretrain/transformer/transformer_train.py for GH29BERT model pre-training. Runpython classification/downstream_embedding.py for loading the pre-trained model parameters and the embedding data(.npz) preparing for the task-training, and then run python classification/downstream_train.py for cluster prediction for task-training.