VertexAGI/quartz-micro-preview-v2-base

Quartz Micro Preview V2 Base

A tiny language model trained from scratch by Vertex AGI on a single GTX 1660 Ti: a 100.09M-parameter dense Llama-architecture decoder (12 layers, hidden 768, 12 heads / 4 KV heads, SwiGLU 2048, RoPE, tied 32K byte-level BPE, 1,024 context). This is Quartz Micro Preview V2 Base, the raw pretrained base model (it continues text; it does not follow instructions). The chat model is Quartz Micro Preview V2 (VertexAGI/quartz-micro-preview-v2).

Honest summary: at ~100M parameters this model writes short, fluent, on-topic text but often states wrong facts with confidence, is weak at maths, reasoning and code, and can repeat itself. Treat it as a small research / hobby model, not a source of truth.

Formats (one repo, three formats)

Where	Format
repo root	stock Hugging Face LlamaForCausalLM (fp32 safetensors), loads with plain transformers and plain mlx_lm
mlx/fp16, mlx/q8, mlx/q4	stock MLX (fp16, 8-bit, 4-bit)
gguf/	stock llama.cpp GGUF: f16, q8_0, q4_k_m

Every format was checked against the original training code on held-out text (perplexity; lower is better):

Format	Perplexity	Difference from original
transformers fp32 (this repo root)	21.577	0.000%
MLX fp16	21.577	0.001%
MLX 8-bit	21.578	0.004%
MLX 4-bit	22.151	2.659%
GGUF f16	21.577	0.001%
GGUF Q8_0	21.598	0.095%
GGUF Q4_K_M	21.641	0.296%

Usage

# stock transformers (no custom code, no trust_remote_code)
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("VertexAGI/quartz-micro-preview-v2-base")
model = AutoModelForCausalLM.from_pretrained("VertexAGI/quartz-micro-preview-v2-base")
ids = tok("The history of the Roman Empire", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

# stock MLX (Apple silicon)
from mlx_lm import load, generate
model, tok = load("VertexAGI/quartz-micro-preview-v2-base")
print(generate(model, tok, prompt="The history of the Roman Empire", max_tokens=60))

# stock llama.cpp (GGUF in the gguf/ folder)
hf download VertexAGI/quartz-micro-preview-v2-base gguf/quartz-micro-preview-v2-base-q8_0.gguf --local-dir .
llama-cli -m gguf/quartz-micro-preview-v2-base-q8_0.gguf -p "The history of the Roman Empire" -n 60 -no-cnv

Training

• Data: 100M-token educational mix (50% Cosmopedia v2, 35% FineWeb-Edu, 10% Wikipedia, 5% Python-Edu), filtered and deduplicated, tokenized with the 32K Quartz tokenizer: VertexAGI/quartz-micro-v2-pretrain.
• Schedule: 4 passes (~400M tokens), 6,080 steps of 65,536 tokens, AdamW (lr 6e-4 cosine to 6e-5), fp32 on one GTX 1660 Ti.
• Held-out perplexity: 18.0 on the 0.5% validation split.

Evaluation

• Held-out perplexity on the pretraining validation set: 21.577 (fp32); every shipped format is within 0.3% (table above).
• Hand-written continuation prompts (greedy, 60 tokens) are in eval_outputs.md, including the wrong and repetitive ones: it continues topics fluently but states wrong facts and loops (for example on "The three states of matter are").
• It is not tuned to answer questions; for that use Quartz Micro Preview V2, whose evaluation is in its card.

Limitations

No instruction following, no safety tuning, English only, 1,024-token context, no benchmark decontamination of the pretraining data, and it inherits errors and biases from web and synthetic text.

License

Apache-2.0 for the weights and code. Training data carries its own terms (see the dataset cards; Wikipedia and Stack Exchange are share-alike).