VertexResearch/Vertex-0.1-10M-base

Vertex 0.1 (10M)

Small LLaMA-architecture causal language model, ~10.9M parameters.

Model Details

• Architecture: LlamaForCausalLM
• Parameters: ~10.9M
• Hidden size: 256
• Layers: 4
• Attention heads: 4 (2 KV heads, GQA)
• Head dim: 64
• Intermediate size: 640
• Max position embeddings: 1024
• Vocab size: 32000
• Dtype: float32

Training

• Epochs: 3
• Global steps: 3207
• Batch size: 8
• Estimated tokens seen: ~26M total, ~8.8M tokens/epoch
• Training time: ~1 Hour
• Hardware: M5 Pro 24gb Macbook

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("VertexResearch/Vertex-0.1-10M-base")
tokenizer = AutoTokenizer.from_pretrained("VertexResearch/Vertex-0.1-10M-base")

inputs = tokenizer("Once upon a time", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=50)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Limitations

This is a very small model (~10.9M params) trained on a small dataset. Expect limited coherence, knowledge, and generalization compared to larger models. Intended for experimentation, education, and low-resource testing, not production use.