VertexAGI/quartz-micro-preview-v3-base

Quartz Micro Preview V3 (base)

A tiny language model trained from scratch by Vertex AGI on a single GTX 1660 Ti: a 100.09M-parameter dense Llama-architecture decoder (12 layers, hidden 768, 12 heads / 4 KV heads, SwiGLU 2048, RoPE, tied 32K byte-level BPE, 1,024 context). This is the raw pretrained base model (it continues text; it does not follow instructions). The chat model is VertexAGI/quartz-micro-preview-v3.

Honest summary: at ~100M parameters this model writes short, fluent text but still states wrong facts and invents details, is weak at maths, reasoning and code, and ignores length limits. V3 is trained on ~65x more text than V2 (6.5B vs 100M tokens).

Available formats

Everything below lives in this one repository: pick the format that matches your runtime.

Format	File / folder	Precision	Size	Works with
Transformers (safetensors)	model.safetensors (repo root)	fp32 (32-bit)	400 MB	Hugging Face Transformers / PyTorch
MLX	mlx/fp16/	fp16 (16-bit)	200 MB	mlx-lm / MLX on Apple Silicon
MLX	mlx/q4/	4-bit (group size 64)	63 MB	mlx-lm / MLX on Apple Silicon
MLX	mlx/q8/	8-bit (group size 64)	113 MB	mlx-lm / MLX on Apple Silicon
GGUF	gguf/quartz-micro-preview-v3-base-f16.gguf	F16 (16-bit float)	201 MB	llama.cpp, Ollama, LM Studio
GGUF	gguf/quartz-micro-preview-v3-base-q4_k_m.gguf	Q4_K_M (4-bit)	67 MB	llama.cpp, Ollama, LM Studio
GGUF	gguf/quartz-micro-preview-v3-base-q8_0.gguf	Q8_0 (8-bit)	108 MB	llama.cpp, Ollama, LM Studio

Formats (one repo, three formats)

Where	Format
repo root	stock Hugging Face LlamaForCausalLM (fp32 safetensors), loads with plain transformers and plain mlx_lm
mlx/fp16, mlx/q8, mlx/q4	stock MLX (fp16, 8-bit, 4-bit)
gguf/	stock llama.cpp GGUF: f16, q8_0, q4_k_m

Every format was checked against the original training code on held-out text (perplexity; lower is better):

Format	Perplexity	Difference from original
transformers fp32 (this repo root)	10.614	0.000%
MLX fp16	10.614	0.000%
MLX 8-bit	10.604	0.096%
MLX 4-bit	11.043	4.044%
GGUF f16	10.614	0.002%
GGUF Q8_0	10.615	0.008%
GGUF Q4_K_M	10.903	2.717%

Usage

# stock transformers (no custom code, no trust_remote_code)
from transformers import AutoModelForCausalLM, AutoTokenizer
tok = AutoTokenizer.from_pretrained("VertexAGI/quartz-micro-preview-v3-base")
model = AutoModelForCausalLM.from_pretrained("VertexAGI/quartz-micro-preview-v3-base")
ids = tok("The history of the Roman Empire", return_tensors="pt").input_ids
out = model.generate(ids, max_new_tokens=60, do_sample=True, temperature=0.7, top_p=0.9, repetition_penalty=1.1)
print(tok.decode(out[0], skip_special_tokens=True))

# stock MLX (Apple silicon)
from mlx_lm import load, generate
model, tok = load("VertexAGI/quartz-micro-preview-v3-base")
print(generate(model, tok, prompt="The history of the Roman Empire", max_tokens=60))

# stock llama.cpp (GGUF in the gguf/ folder)
hf download VertexAGI/quartz-micro-preview-v3-base gguf/quartz-micro-preview-v3-base-q8_0.gguf --local-dir .
llama-cli -m gguf/quartz-micro-preview-v3-base-q8_0.gguf -p "The history of the Roman Empire" -n 60 -no-cnv

Training

• Data: | Source | Unique tokens | Notes | |---|---:|---| | FineWeb-Edu (dedup, SmolLM-Corpus) | 2.60B | fills the rest of the budget | | Cosmopedia v2 (SmolLM-Corpus) | 1.30B | synthetic textbook-style text | | Wikipedia (20231101.en) | 0.78B | | | DCLM-Baseline | 0.52B | | | Stack Exchange (preferences dump, questions + answers) | 0.46B | | | FineMath (4+) | 0.39B | | | Python-Edu (cleaned) | 0.33B | | | Chat / instruction text (SmolTalk subsets, Tulu-3 SFT mixture subsets) | 0.07B | chat-format documents, includes "I can't do that" / abstention examples | | ELI5 | 30M | question/answer pairs | | SQuAD 2.0 (answerable 9.6M, unanswerable 10.1M x3 passes) | 20M + repeats | the "unanswerable" ones teach "the passage does not say" | | TriviaQA (no-context) | 3.2M x2 | |

Total about 6.5B training tokens (6,513,293,092 with the repeats), one pass, 99,384 steps of 65,536 tokens. 10-word-window n-gram decontamination was run against ARC, HellaSwag, PIQA, WinoGrande, LAMBADA, OpenBookQA, SciQ, BoolQ, TriviaQA-validation, MMLU-test and CommonsenseQA text before packing.

• Schedule: AdamW (betas 0.9 / 0.95), lr 6e-4 with 300 warm-up steps then cosine to 6e-5, micro-batch 2 x 32 accumulation x 1,024 tokens, fp32 on one GTX 1660 Ti (about 18 days at ~4,200 tokens/s).
• Final validation loss: 2.61 (perplexity 13.6) on the V3 validation split.
• Versus the V2 base (same architecture and tokenizer, 100M training tokens): on 300 random 1,024-token windows of the V3 validation split, perplexity is 14.4 (V3 base) vs 28.1 (V2 base). The validation text is drawn from the V3 mix, so the V2 base is partly out of distribution; read it as "V3 has seen far more, and more varied, text", not as a like-for-like quality gap.

Limitations

No instruction following, no safety tuning, English only, 1,024-token context, and it inherits the errors and biases of web and synthetic text. Not suitable for any decision that matters.

Data and licences (read before commercial use)

The weights are released under Apache-2.0, but they were trained on third-party data with its own terms, and some of those terms are NonCommercial or ShareAlike (for example the Tulu-3 mixture includes no_robots, CC-BY-NC; Wikipedia, Stack Exchange, Dolly and SQuAD are CC-BY-SA; CoCoNot is ODC-By). Whether training-data terms extend to model weights is legally unsettled. This is a research / hobby model: if you need legal certainty for a commercial product, ask a lawyer or retrain on data you have cleared. The pretraining and identity datasets are not public.