VertexAGI/quartz-micro-preview-base

Quartz Micro Preview Base

A ~1B-parameter mixture-of-experts language model, trained completely from scratch — randomly-initialized weights, no distillation, no fine-tune of an existing checkpoint — on a single consumer GPU (NVIDIA GTX 1660 Ti, 6GB VRAM). No cluster, no cloud credits.

This is the base checkpoint: raw pretrain output, published as-is. It has no instruction-following behavior — it completes text, it does not follow instructions or hold a conversation. An instruction-tuned release (full-parameter SFT, not LoRA) is in progress; see vertexagi.vercel.app/research for live status.

"Micro" because it's deliberately small. "Preview" because this first run is a proof of concept for training a real MoE from scratch on hardware anyone can buy — not the final word on how far the approach can go.

Architecture

DeepSeek-style fine-grained mixture-of-experts:

Total parameters	1,031.0M (~1.03B)
Active parameters / token	394.0M
Hidden size	1,024
Layers	20 (first 2 dense, rest MoE)
Attention	16 query heads / 4 KV heads (GQA), head dim 64
Context length	2,048 tokens
Routed experts	24 (6 active per token)
Shared experts	2 (always active)
Expert FFN size	640 (fine-grained segmentation)
Router	top-6 of 24, 0.01-weighted load-balancing loss
Vocabulary	32,000 tokens
Tied embeddings	yes

Training

• 100,003,832 tokens of packed training data — 70% FineWeb-Edu, 20% Wikipedia, 10% CodeParrot-clean
• 24,414 optimizer steps, 4,096 tokens/step (batch size × gradient accumulation × sequence length)
• FP16 mixed precision (autocast + GradScaler)
• 8-bit AdamW (bitsandbytes) to keep optimizer state small
• Full gradient checkpointing, gradient accumulation
• A from-scratch 32K-vocab byte-level BPE tokenizer, trained on a sample of the same corpus (included in this repo under tokenizer/)
• Single NVIDIA GTX 1660 Ti, 6GB VRAM, Windows desktop — survived two full power/network outages, resumed cleanly from checkpoint both times

Files

• model.safetensors — model weights (optimizer state dropped; not needed for inference or further tuning from this checkpoint)
• configuration_quartz.py, modeling_quartz.py — transformers-compatible PretrainedConfig/PreTrainedModel wrapper (QuartzMoEConfig, QuartzForCausalLM), wired up via auto_map in config.json so AutoConfig/AutoModelForCausalLM(trust_remote_code=True) load this repo directly — verified bit-exact against the plain-PyTorch path below.
• model.py, config.py — the original plain-PyTorch model class and architecture config (MoELanguageModel, MoEConfig), used by load_model.py. Functionally identical architecture to modeling_quartz.py, just without the transformers scaffolding.
• config.json — architecture config in HF's expected format, read by both loading paths.
• tokenizer.json, tokenizer_config.json, special_tokens_map.json — a standard transformers fast tokenizer (AutoTokenizer.from_pretrained(...)), built from the same vocab/merges below.
• tokenizer/vocab.json, tokenizer/merges.txt — the custom tokenizer's raw vocab/merges, used by load_model.py's plain-PyTorch path. It is only compatible with this model; no other tokenizer will produce correct token ids for these weights, and vice versa.
• load_model.py — minimal working example using the plain-PyTorch path (load + generate)

Usage

Via transformers (recommended):

pip install transformers torch

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "VertexAGI/quartz-micro-preview-base", trust_remote_code=True
)
tok = AutoTokenizer.from_pretrained("VertexAGI/quartz-micro-preview-base")

ids = tok("The history of the Roman Empire", return_tensors="pt")
out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.8)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required — this is a bespoke architecture (DeepSeek-style fine-grained MoE), not one of transformers' built-in model types, so it ships its own code (configuration_quartz.py, modeling_quartz.py). No KV-cache support yet, so generate() recomputes attention over the full sequence each step — fine at this model's size, just not as fast as a cached model.

Plain PyTorch (no transformers dependency):

pip install torch safetensors tokenizers
python load_model.py

from load_model import load, generate

model, tok = load()
print(generate(model, tok, "The history of the Roman Empire"))

Limitations

This is a base model from a single ~100M-token training run on a 6GB consumer GPU — small on every axis by design. Expect base-model behavior (text completion, not instruction-following), factual unreliability, and meaningfully weaker general knowledge than models trained on far larger corpora. No safety fine-tuning has been applied. Treat outputs accordingly.

No evaluation numbers are published for this checkpoint specifically — the held-out base-vs-tuned comparison happens once the instruction-tuned release is ready, per our usual practice of never publishing an eval that can't be directly compared against a real baseline.

---

Built by Vertex AGI. Every model we ship — weights, not just claims.