Codemaster67/olmo_1b_vocab_extension_10m

OLMo-7B QLoRA Adapter -- Chemistry SMILES CPT

Model Description

This is a QLoRA (Quantized LoRA) adapter trained on top of Codemaster67/Olmo-1b_token_extension_modern_molbert for chemistry SMILES language modelling using the Codemaster67/Unichem_10M dataset.

The base model was loaded in 4-bit precision (NF4 quantization via bitsandbytes with double quantization) and LoRA adapter matrices were trained on top in bfloat16.

Decoupled learning rates are used: LoRA adapters train at 2e-05, while embed_tokens and lm_head train at 2e-05 (10x smaller) to avoid catastrophic forgetting of the base vocabulary.

QLoRA / Quantization Configuration

Parameter	Value
Quantization	NF4 (4-bit)
Double Quantization	True
Compute dtype	bfloat16
Rank (r)	64
Alpha	128
Effective Scaling	2.0
Target Modules	all-linear
Dropout	0.01
RSLoRA	False
Modules to Save	embed_tokens, lm_head

Decoupled Learning Rates

Parameter Group	Learning Rate
LoRA adapters	2e-05
embed_tokens + lm_head	2e-05 (x1.0)

Training Details

Parameter	Value
Method	QLoRA (4-bit base + LoRA adapters)
Epochs	1
Learning Rate (LoRA)	2e-05
Learning Rate (embed/head)	2e-05
Optimizer	AdamW 8-bit
Batch Size (per device)	32
Gradient Accumulation	1
Max Sequence Length	512
Warmup Ratio	0.1
Weight Decay	0.01
Effective Batch Size	32
Scheduler	Cosine
Precision	bf16 (adapters) / 4-bit NF4 (base)
Gradient Checkpointing	True
Validation Split	5 %
Packed Training Sequences	7512
Packed Validation Sequences	393

Training Results

Metric	Value
Training Loss	4.2704
Validation Loss	3.5823

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
    "Codemaster67/Olmo-1b_token_extension_modern_molbert", quantization_config=bnb_config, trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "Codemaster67/olmo_1b_vocab_extension_10m")
tokenizer = AutoTokenizer.from_pretrained("Codemaster67/olmo_1b_vocab_extension_10m", trust_remote_code=True)

smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>"
inputs = tokenizer(smiles_input, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Intended Use

Chemistry-domain language modelling, SMILES generation and completion, and downstream molecular property prediction via fine-tuning.

Limitations

• QLoRA adapters only; requires the base model Codemaster67/Olmo-1b_token_extension_modern_molbert loaded in 4-bit to use.
• Trained primarily on SMILES strings; natural-language instruction-following ability may degrade compared to the base OLMo checkpoint.