SmolLM2-135M (ERA V4 Week13) — trained from scratch
This repository contains a SmolLM2-135M compatible checkpoint trained from scratch as part of ERA V4 Week13.
Architecture (exported)
- vocab_size=49152
- hidden_size=576
- intermediate_size=1536
- num_layers=30
- n_heads=9
- n_kv_heads=3
- RoPE theta=100000.0
- RMSNorm eps=1e-05
- max_position_embeddings=8192
- tie_word_embeddings=True
Quick usage
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "<YOUR_HF_USERNAME/YOUR_MODEL_REPO>"
tok = AutoTokenizer.from_pretrained(model_id, use_fast=True)
model = AutoModelForCausalLM.from_pretrained(model_id)
inputs = tok("BIANCA:", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=80)
print(tok.decode(out[0], skip_special_tokens=True))
- Downloads last month
- 14