ExtraAI-75M

ExtraAI is a 75,012,480-parameter, English-only decoder Transformer trained from random initialization. Its 32k byte-level BPE tokenizer is also trained from scratch; no pretrained model or pretrained tokenizer is used.

Architecture: 11 layers, width 640, 10 attention heads, SwiGLU width 1728, RMSNorm, RoPE, PyTorch scaled-dot-product causal attention, tied embeddings, and a 1024-token context. Exported after optimizer step 4,000. Both the original Safetensors weights and a llama.cpp-compatible ExtraAI-75M-F16.gguf are included.

Training data (77,520 train conversations):

• agentic-ptb/grok-data
• 11-47/god_agent_grok4.4_cot_traces_20k
• WithinUsAI/Grok_4.4_Distilled

The training code applies a lightweight English filter and exact-text deduplication, then performs causal pretraining followed by assistant-only supervised training. Dataset licenses and the suitability of distilled outputs for your intended use must be checked independently before redistribution or commercial use.

Inference

from huggingface_hub import snapshot_download
import sys

path = snapshot_download("REKS1337/ExtraAI-75M")
sys.path.insert(0, path)
from extraai_train import load_for_inference, generate_text

model, tokenizer = load_for_inference(path, device="cuda")
print(generate_text(model, tokenizer, "Explain why the sky is blue."))

This small model was trained under a strict single-session compute budget. Treat its answers as experimental: it can hallucinate, reproduce dataset biases, and should not be used for high-stakes decisions.

llama.cpp / GGUF

llama-cli -m ExtraAI-75M-F16.gguf -cnv --jinja -p "Explain why the sky is blue."

The bundled GGUF is F16 (approximately 150 MiB) and uses tied token/output embeddings. It can later be quantized with llama-quantize if a smaller file is needed.