SmolLM3-3B-Base — computational-chemistry CPT (LoRA adapter)
A rsLoRA adapter from the CPT→SFT case study in https://github.com/anisiraj/llm_kickstart_repo (handbook.html → 🧪 Case Study).
- Phase: cpt (full causal loss on raw domain text; targets include embed_tokens + lm_head)
- Base model:
HuggingFaceTB/SmolLM3-3B-Base - Data: anisiraj/comp-chem-quantum-chem
- Measured: domain ppl 11.19→11.15; no forgetting (general 8.33→8.19)
- Reproduce:
bash case_study/run.sh smollm3
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
base = AutoModelForCausalLM.from_pretrained("HuggingFaceTB/SmolLM3-3B-Base", torch_dtype="bfloat16")
model = PeftModel.from_pretrained(base, "anisiraj/SmolLM3-3B-Base-compchem-cpt-lora")
tok = AutoTokenizer.from_pretrained("HuggingFaceTB/SmolLM3-3B-Base")
- Downloads last month
- 21
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for anisiraj/SmolLM3-3B-compchem-cpt-lora
Base model
HuggingFaceTB/SmolLM3-3B-Base