InserloftResearch/CaelisNeuralBase-3

Caelis Neural Base 3 (CNB-3): A Foundational Protein Language Model

Caelis Neural Base 3 (CNB-3) is a ~985-million parameter autoregressive causal language model pre-trained exclusively on dense evolutionary sequence spaces. Engineered to act as a foundational "Neural Base" for biotechnology, CNB-3 functions as a versatile, general-purpose backbone for generating, completing, and optimizing functional proteins de novo.

---

⚠️ OUT-OF-SCOPE USE & LIMITATIONS (CRITICAL WARNING)

• NOT FOR CONVERSATIONAL CHAT / TEXT NLP: Although this model is built on an autoregressive Decoder-only architecture, neither the model nor the tokenization scheme (CaelisBioTokenizer-2) are designed for conversational text generation, chatbots, or general human language tasks.
• Purely Biological: The vocabulary has been fully optimized to process amino acids and structural tokens. Attempting to fine-tune CNB-3 or CaelisBioTokenizer-2 for standard NLP text or dialogue models will result in complete failure and garbage outputs.
• Intended Domain: This foundation is strictly restricted to sequence generation, enzyme design, protein scaffolding, and biopolymer optimization.

---

Why "Neural Base"?

This model is named "Neural Base" because it is structurally designed to serve as the absolute baseline foundation for down-stream adaptations. It acts as the primary starter weight checkpoint for protein language tasks:

• Unbiased Prior: Trained on diverse protein domains to capture native evolutionary distributions without task-specific overfitting.
• Causal Autoregressive Backbone: Optimized for sequential token prediction (predicting the next amino acid), allowing natural, coherent protein generation from custom seed sequences.
• Scaffolding & Completion: Capable of taking partial structural domains (e.g., binding sites or catalytic loops) and automatically filling in the rest of the stable globular fold.

---

🔑 Reusing Our Custom Tokenizer: CaelisBioTokenizer-2

You are highly encouraged to use CaelisBioTokenizer-2 as a standard starting vocabulary for training your own custom molecular architectures from scratch!

• Zero Character Waste: Our tokenizer is stripped of noisy human-text tokens, reserving almost 100% of its vocabulary space specifically for amino acids and chemical markers.
• Perfect for From-Scratch Training: If you are training a new causal Transformer, a BERT-style model, or an autoencoder for biochemistry, loading our tokenizer will save you the trouble of designing a vocabulary and special tokens from zero.

# Loading CaelisBioTokenizer-2 to train your own models!
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("InserloftResearch/CaeisNeuralBase-3")
print("Vocabulary size for your new protein model:", len(tokenizer))

---

Model Architecture Specifications

• Parameters: ~985M (dense Transformer blocks)
• Architecture: Causal LM (Decoder-only configuration)
• Layers: 32 Decoder Blocks
• Hidden Dimension ($d_{\text{model}}$): 1536
• Attention Heads: 24
• Context Window: 1024 amino acids
• Vocabulary: Custom CaelisBioTokenizer-2

---

Quick Start: Inference with Hugging Face transformers

To generate sequences from a custom 25-amino-acid seed:

import torch
from transformers import AutoTokenizer, GPT2LMHeadModel

repo_id = "InserloftResearch/CaelisNeuralBase-3"

tokenizer = AutoTokenizer.from_pretrained(repo_id)
model = GPT2LMHeadModel.from_pretrained(repo_id)

device = "cuda" if torch.cuda.is_available() else "cpu"
model = model.to(device)
model.eval()

# Seed sequence (25 residues)
seed_sequence = "MNSFSTSAFGPVAFSLGLLLVLPAA"
inputs = tokenizer(seed_sequence, return_tensors="pt").to(device)

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_length=35,
        do_sample=True,
        temperature=0.7,
        top_k=50,
        top_p=0.95,
        pad_token_id=tokenizer.pad_token_id
    )

generated_seq = tokenizer.decode(outputs[0], skip_special_tokens=True)
print("Generated Protein Sequence:", "".join(generated_seq.split()))

---

Fine-Tuning & Downstream Adaptations (How to use it as a Base)

CNB-3 is uniquely suited for Low-Rank Adaptation (LoRA), full-parameter fine-tuning, or reinforcement learning with reward signals (RLHF) targeting specific biochemical phenotypes.

1. Zero-Shot Optimization (No Weights Needed)

Generate sequences using varied temperature and top_k values to sample mutations directly from CNB-3's learned evolutionary distributions. Filter out non-folding variants using downstream folding models (e.g., ESMFold).

2. Domain-Specific Fine-Tuning

Adapt CNB-3 to specific enzyme families (e.g., Polyethylene Terephthalate Hydrolases, therapeutic neurotrophins, or metallic respiratory binding domains) by training on targeted sequences with a low learning rate ($1.0 \times 10^{-5}$ to $5.0 \times 10^{-5}$):

# Recommended Training Settings for Fine-Tuning:
training_args = TrainingArguments(
    output_dir="./CNB-3-Specialized",
    learning_rate=3e-5,
    weight_decay=0.01,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=8,
    fp16=True,
    lr_scheduler_type="cosine"
)

---

Pro-Tips for High-Fidelity Protein Generation

1. Strict Length Hard-Capping: Autoregressive decoding may output whitespace tokens due to native formatting. Always remove whitespaces ("".join(seq.split())) and slice the sequence dynamically to match your exact structural budget.
2. VRAM Budgeting: Since ESMFold is resource-intensive, set the chunk size (esm_folding_model.esm.trunk.set_chunk_size(64)) to avoid CUDA Out of Memory (OOM) errors on consumer-grade or standard Colab GPUs during verification.
3. The Temperature Sweet-Spot: Use a temperature of 0.7 to balance biochemical feasibility with sequence novelty. Lower values (< 0.4) generate highly conserved structures; higher values (> 1.1) increase structural entropy (greater risk of non-folding loop accumulation).

---

📚 Resources & Community Support

To facilitate the adoption and specialized fine-tuning of Caelis Neural Base 3, we provide the following interactive resources and endpoints:

• Official Model Hub: Access the base model weights, standard configuration files, and community discussions.
• Developer & Research Center: Official research workspace and support for production-grade biopolymeric applications.