SmileBERTa

SmileBERTa is a RoBERTa-based language model for chemistry, built on the ChemBERTa architecture and fine-tuned to predict small-molecule fragment SMILES from full small-molecule drug SMILES.

Fragment prediction is a common first step in fragment-based drug design, scaffold analysis, and retrosynthetic planning. SmileBERTa treats SMILES strings as a language, so it can also serve as a general-purpose chemical encoder for downstream tasks such as property prediction, similarity search, or molecular clustering.

Model details

• Architecture: RoBERTa (ChemBERTa lineage)
• Input: SMILES string of a small-molecule drug
• Output: SMILES string of a predicted fragment
• Tokenizer: SMILES-aware byte-level BPE
• Format: Safetensors
• Developed by: NisargRhino

Intended uses

Direct use

• Predicting plausible fragments from a parent drug molecule
• Generating molecular embeddings from SMILES strings
• Exploratory work in fragment-based drug discovery

Downstream use

• Fine-tuning for molecular property prediction (ADMET, solubility, binding affinity)
• Feature extraction for QSAR models and virtual screening pipelines
• Any project that needs a chemistry-aware language model over SMILES

Out of scope

• Clinical, diagnostic, or therapeutic decision-making
• Guaranteeing that predicted structures are synthesizable, stable, or safe
• Non-SMILES chemical representations (InChI, SELFIES, molfiles) without conversion first

How to use

from transformers import AutoTokenizer, AutoModel

tokenizer = AutoTokenizer.from_pretrained("NisargRhino/SmileBERTa")
model = AutoModel.from_pretrained("NisargRhino/SmileBERTa")

smiles = "CC(=O)OC1=CC=CC=C1C(=O)O"  # aspirin
inputs = tokenizer(smiles, return_tensors="pt")
outputs = model(**inputs)

# Pooled representation of the molecule
embedding = outputs.last_hidden_state.mean(dim=1)
print(embedding.shape)

Limitations and bias

• Predictions reflect the chemical space of the training data. Molecules that are unusual, very large, or structurally distant from that space will give less reliable results.
• The model can produce SMILES strings that are syntactically valid but chemically implausible. Validate every output with a cheminformatics toolkit such as RDKit before acting on it.
• No guarantee of stereochemical correctness; canonicalize inputs and check outputs.
• Results have not been experimentally validated in a wet lab.

Recommendations

Run generated SMILES through a validity check before use:

from rdkit import Chem

mol = Chem.MolFromSmiles(predicted_smiles)
if mol is None:
    print("Invalid SMILES — discard or regenerate")

Training details

• Training data: TODO — dataset name, size, and source
• Preprocessing: TODO — canonicalization, filtering, max sequence length
• Hyperparameters: TODO — epochs, batch size, learning rate, optimizer
• Hardware: TODO

Evaluation

TODO — report metrics such as SMILES validity rate, Tanimoto similarity to ground-truth fragments, or exact-match accuracy on a held-out set.

Contact

Questions, feedback, and collaboration are welcome: nisargs2018@gmail.com

Citation

@misc{smileberta,
  author = {Nisarg S.},
  title  = {SmileBERTa: A ChemBERTa-based model for fragment SMILES prediction},
  year   = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/NisargRhino/SmileBERTa}}
}