BIOPTIC B1 — MLM pretrained encoder (independent reimplementation)
A RoBERTa-style encoder (8.7M parameters) pretrained with masked language modeling on 10M SMILES strings. This is an independent reimplementation of the pretraining stage described in Vinogradov et al., J. Chem. Inf. Model. 2025, 65, 9927–9936 (doi: 10.1021/acs.jcim.5c00743). These are not the authors' weights — the original weights were never released.
Architecture
Exactly as specified in the paper: vocab 500, 6 layers, hidden 384, 8 attention heads, intermediate 1024. Byte-level BPE tokenizer trained on the same corpus.
Training data
10M SMILES: 7M from PubChem, 3M from Enamine REAL Database Samples (version 2026.01), in the paper's 70/30 ratio. The paper used 160M molecules (115M PubChem + 48M Enamine REAL Space, February 2024) — that version of REAL Space is no longer distributed.
Training
The paper specifies no pretraining hyperparameters, so these are our own:
AdamW, lr 5e-4, warmup 1000 steps, weight decay 0.01, effective batch 512,
fp16. Trained 34,000 steps (~1.74 epochs) on 2×T4. MLM loss fell from
~6.2 (random guessing over a 500-token vocabulary) to 0.653, where it
plateaued. Full loss history in log_history.json.
Intended use
Starting point for fine-tuning on bioactivity data. Not useful on its own — the MLM head predicts masked tokens, not molecular properties.
Limitations
16× less pretraining data than the original, a different version of the Enamine library, and an entirely self-chosen pretraining procedure. Any difference from the paper's reported numbers has several possible causes that cannot be separated.
- Downloads last month
- -