---
tags:
- generated_from_trainer
metrics:
- accuracy
model-index:
- name: RITA_FineTuning_runclm_256_xl
  results: []
license: openrail
---

<!-- This model card has been generated automatically according to the information the Trainer had access to. You
should probably proofread and complete it, then remove this comment. -->

# SARITA XL

![Symbol](./Symbol.png)

This model is a continuous learning of [lightonai/RITA_xl](https://huggingface.co/lightonai/RITA_xl).
It achieves the following results on the evaluation set:
- Loss: 0.0292
- Accuracy: 0.9914

The Pre-print paper is avaiable [here](https://www.biorxiv.org/content/10.1101/2024.12.10.627777v1). The codes to train and to evaluate the model is avaiable on [GitHub](https://github.com/simoRancati/SARITA)

Model | #Params | d_model | layers 
--- | --- | --- | --- | 
[Small](https://huggingface.co/SimoRancati/SARITA_S) | 85M  | 768 | 12 
[Medium](https://huggingface.co/SimoRancati/SARITA_M) | 300M | 1024 | 24 
[Large](https://huggingface.co/SimoRancati/SARITA_L)| 680M | 1536 | 24 
[**XLarge**](https://huggingface.co/SimoRancati/SARITA_XL)| 1.2B | 2048 | 24 


## Model description

SARITA S is an LLM with 1.2 billion parameters, based on GPT-3 architecture, designed to generate high-quality synthetic SARS-CoV-2 Spike sequences. 
SARITA is trained via continuous learning on the pre-existing protein model RITA. The model was trained using sequences uploaded to GISAID between December 2019 and March 2021.

## Intended uses & limitations

This model can be used by user to generate synthetic Spike proteins of SARS-CoV-2 Virus. 

### Training hyperparameters

The following hyperparameters were used during training:
- learning_rate: 5e-05
- train_batch_size: 48
- eval_batch_size: 48
- seed: 42
- optimizer: Adam with betas=(0.9,0.999) and epsilon=1e-08
- lr_scheduler_type: linear
- num_epochs: 10
- mixed_precision_training: Native AMP

### Training results

| Training Loss | Epoch | Step   | Validation Loss | Accuracy |
|:-------------:|:-----:|:------:|:---------------:|:--------:|
| 0.0299        | 1.0   | 10013  | 0.0301          | 0.9912   |
| 0.0296        | 2.0   | 20026  | 0.0295          | 0.9913   |
| 0.0293        | 3.0   | 30039  | 0.0294          | 0.9913   |
| 0.0293        | 4.0   | 40052  | 0.0294          | 0.9913   |
| 0.0291        | 5.0   | 50065  | 0.0293          | 0.9914   |
| 0.029         | 6.0   | 60078  | 0.0292          | 0.9914   |
| 0.0288        | 7.0   | 70091  | 0.0293          | 0.9914   |
| 0.0285        | 8.0   | 80104  | 0.0295          | 0.9914   |
| 0.028         | 9.0   | 90117  | 0.0299          | 0.9914   |
| 0.0276        | 10.0  | 100130 | 0.0304          | 0.9914   |


### Framework versions

- Transformers 4.20.1
- Pytorch 1.9.0+cu111
- Datasets 2.18.0
- Tokenizers 0.12.1

### Usage 
Instantiate a model like so:
``` python
from transformers import AutoModelForCausalLM, AutoTokenizer
model = AutoModelForCausalLM.from_pretrained("SimoRancati/SARITA_XL", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("SimoRancati/SARITA_XL")
```
for generation used this code:
``` python
# Check for GPU availability and move the model to GPU
device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)

start = ['MFVFLVLLPLVSSQ']

for i in range(len(start)):
    # Prepare model inputs
    model_inputs = tokenizer([start[i]], return_tensors="pt")
    model_inputs = {k: v.to(device) for k, v in model_inputs.items()}

    # Generate predictions using the model
    generated_ids = model.generate(**model_inputs, min_length=701, max_length=701,
                                   do_sample=True, top_k=950, repetition_penalty=1.2,
                                   num_return_sequences=100, eos_token_id=2, truncation=True)

    # Decode and print outputs
    generated_sequences = []
    for f in range(len(generated_ids)):
        sequence = tokenizer.decode(generated_ids[f], skip_special_tokens=True).replace(' ', '')
        generated_sequences.append(sequence)
```

## Avaiability
SARITA model is public, but downloading it requires approval.  
To request access, click on the **Request Access** button and provide a brief explanation of your intended use.


## License
The use of this model is restricted to research purposes. Commercial use is not allowed without prior approval.