Codemaster67/Olmo_1M_tok_unichem_fineweb

OLMo-7B QLoRA Adapter — Chemistry SMILES CPT

Model Description

This is a QLoRA (Quantized LoRA) adapter trained on top of allenai/OLMo-1B-hf for chemistry SMILES language modelling using the Codemaster67/Unichem_smiles-fineweb-1M dataset.

The base model was loaded in 4-bit precision (NF4 quantization via bitsandbytes with double quantization) and LoRA adapter matrices were trained on top in bfloat16. This is the most memory-efficient training configuration compared to full LoRA and full fine-tuning.

The base model's tokenizer was pre-extended with ~300 SPE (SMILES Pair Encoding) chemistry tokens plus <|start_of_smiles|> / <|end_of_smiles|> special tokens. The embed_tokens and lm_head layers are saved as full (non-LoRA) trainable copies via modules_to_save because they were resized during tokenizer extension.

QLoRA / Quantization Configuration

Parameter	Value
Quantization	NF4 (4-bit)
Double Quantization	True
Compute dtype	bfloat16
Rank (r)	64
Alpha	128
Effective Scaling	2.0
Target Modules	all-linear
Dropout	0.01
RSLoRA	True (rank-stabilized)
Modules to Save	embed_tokens, lm_head

Training Details

Parameter	Value
Method	QLoRA (4-bit base + LoRA adapters)
Epochs	1
Learning Rate	3e-05
Optimizer	AdamW 8-bit
Batch Size (per device)	32
Gradient Accumulation	1
Max Sequence Length	512
Warmup Ratio	0.1
Weight Decay	0.01
Effective Batch Size (Batch Size 32 x Gradient Accumulation 1)	32
Scheduler	Cosine
Precision	bf16 (adapters) / 4-bit NF4 (base)
Gradient Checkpointing	True
Training Data	Full dataset (train+test merged), no validation split
Packed Training Sequences	1981

Training Results

Metric	Value
Training Loss	1.5530

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig
from peft import PeftModel
import torch

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
)
base_model = AutoModelForCausalLM.from_pretrained(
    "allenai/OLMo-1B-hf", quantization_config=bnb_config, trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, "Codemaster67/Olmo_1M_tok_unichem_fineweb")
tokenizer = AutoTokenizer.from_pretrained("Codemaster67/Olmo_1M_tok_unichem_fineweb", trust_remote_code=True)

smiles_input = "<|start_of_smiles|>CC(=O)Oc1ccccc1C(=O)O<|end_of_smiles|>"
inputs = tokenizer(smiles_input, return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)
print(tokenizer.decode(outputs[0], skip_special_tokens=False))

Intended Use

Chemistry-domain language modelling, SMILES generation and completion, and downstream molecular property prediction via fine-tuning.

Limitations

• QLoRA adapters only; requires the base model allenai/OLMo-1B-hf loaded in 4-bit to use.
• Trained primarily on SMILES strings; natural-language instruction-following ability may degrade compared to the base OLMo checkpoint.
• No validation set was used during training, so no held-out metrics are reported.