Codemaster67/Unichem_chebi_10M_tokens

OLMo-7B QLoRA Adapter -- Chemistry SMILES CPT

Model Description

This is a QLoRA (Quantized LoRA) adapter trained on top of allenai/OLMo-1B-hf for chemistry SMILES language modelling using the Codemaster67/Unichem_chebi_10M dataset.

The base model was loaded in 4-bit precision (NF4 quantization via bitsandbytes with double quantization) and LoRA adapter matrices were trained on top in bfloat16.

Decoupled learning rates are used: LoRA adapters train at 2e-05, while embed_tokens and lm_head train at 2.0000000000000003e-06 (10x smaller) to avoid catastrophic forgetting of the base vocabulary.

QLoRA / Quantization Configuration

Parameter	Value
Quantization	NF4 (4-bit)
Double Quantization	True
Compute dtype	bfloat16
Rank (r)	64
Alpha	128
Effective Scaling	2.0
Target Modules	all-linear
Dropout	0.01
RSLoRA	False
Modules to Save	embed_tokens, lm_head

Decoupled Learning Rates

Parameter Group	Learning Rate
LoRA adapters	2e-05
embed_tokens + lm_head	2.0000000000000003e-06 (x0.1)

Training Details

Parameter	Value
Method	QLoRA (4-bit base + LoRA adapters)
Epochs	1
Learning Rate (LoRA)	2e-05
Learning Rate (embed/head)	2.0000000000000003e-06
Optimizer	AdamW 8-bit
Batch Size (per device)	32
Gradient Accumulation	1
Max Sequence Length	512
Warmup Ratio	0.1
Weight Decay	0.01
Effective Batch Size	32
Scheduler	Cosine
Precision	bf16 (adapters) / 4-bit NF4 (base)
Gradient Checkpointing	True
Validation Split	5 %
Packed Training Sequences	18848
Packed Validation Sequences	990

Training Results

Metric	Value
Training Loss	1.0884
Validation Loss	0.9855

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
    "allenai/OLMo-1B-hf", quantization_config=bnb_config, trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "Codemaster67/Unichem_chebi_10M_tokens")
tokenizer = AutoTokenizer.from_pretrained("Codemaster67/Unichem_chebi_10M_tokens", trust_remote_code=True)

smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>"
inputs = tokenizer(smiles_input, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Intended Use

Chemistry-domain language modelling, SMILES generation and completion, and downstream molecular property prediction via fine-tuning.

Limitations

• QLoRA adapters only; requires the base model allenai/OLMo-1B-hf loaded in 4-bit to use.
• Trained primarily on SMILES strings; natural-language instruction-following ability may degrade compared to the base OLMo checkpoint.