VertexAGI/quartz-micro-preview

Quartz Micro Preview

The instruction-tuned release of Quartz Micro Preview Base — the same ~1B-parameter mixture-of-experts model, trained completely from scratch on a single consumer GPU, now given a full-parameter supervised fine-tune so it follows instructions instead of just completing text.

This is a preview, not a finished product. It is small on every axis by design, and its evaluation below is reported honestly, including where it still falls short — that's this project's standing practice, not a disclaimer added after the fact.

What changed vs. the base checkpoint

• Full-parameter SFT, not LoRA — the base model has zero instruction-following behavior, so a light adapter touch isn't the right tool; every parameter was updated.
• Dataset: databricks-dolly-15k, human-written (not distilled from another company's model), formatted as ### Instruction:\n...\n\n### Response:\n (with an added ### Context:\n block for context-grounded examples), tokenized with the pretrain's own custom 32K byte-level BPE tokenizer. 14,051 of 15,011 examples kept after filtering to the model's 2,048-token context.
• Only the response tokens are supervised (prompt tokens masked to label=-100); optimizer is 8-bit AdamW (bitsandbytes) with full gradient checkpointing, matching the pretrain's own training stack.
• This is the second SFT attempt (v2): the first pass underperformed, so this run used a higher learning rate and a fresh optimizer state rather than continuing from the first run's optimizer. It completed 1,317 steps on a single GTX 1660 Ti (6GB VRAM).

Architecture

Unchanged from the base checkpoint — DeepSeek-style fine-grained mixture-of-experts:

Total parameters	1,031.0M (~1.03B)
Active parameters / token	394.0M
Hidden size	1,024
Layers	20 (first 2 dense, rest MoE)
Attention	16 query heads / 4 KV heads (GQA), head dim 64
Context length	2,048 tokens
Routed experts	24 (6 active per token)
Shared experts	2 (always active)
Expert FFN size	640 (fine-grained segmentation)
Router	top-6 of 24, 0.01-weighted load-balancing loss
Vocabulary	32,000 tokens
Tied embeddings	yes

Held-out evaluation: base vs. this SFT checkpoint

Ten hand-authored prompts, not sampled from dolly-15k or any training data, spanning open QA, closed/context QA, summarization, brainstorming, classification, extraction, general QA, instruction-following, and creative writing. Full transcript below is unedited.

Honest summary: a modest, mixed improvement — not a clean win.

The clearest gain is grounded, context-based QA. The weakest spot is still strict instruction-following (format, length). Both checkpoints remain largely incoherent past the first sentence or two, which tracks with the model's size — that's not something this SFT pass was expected to fix.

Win — closed QA (uses given context correctly):

Prompt: Context: The Golden Gate Bridge is 2,737 meters long and was completed in 1937.\nQuestion: How long is the Golden Gate Bridge?

Base: ignores the context entirely, wanders into an unrelated story about "the Banana" process and the Coast Guard.

SFT: "The Golden Gate Bridge is 2,737 meters long and has an area of approximately 2.4 million square miles." — correctly extracts and restates the fact (the second clause is invented, but the requested answer is right).

Partial win — brainstorming (attempts the requested list format):

Prompt: List three ideas for reducing food waste in a household kitchen.

Base: no list structure, generic unrelated advice about water and diet.

SFT: "1. Make sure you have all food waste in your kitchen. \n2.f leaf food and water should be discharged from local resources. \n3. Focus on local food waste..." — correctly produces a numbered list (the content is still weak/garbled).

Partial win — topical relevance:

Prompt: Why do cats purr?

Base: drifts immediately into an unrelated riff on "the human body," never mentions cats again.

SFT: "Finding the right balance between breast and chest health is hard... So my recommendation is to keep your cat's skin healthy and strong..." — stays anchored to "cat" throughout, base does not.

No win — strict instruction-following:

Prompt: Write a short two-sentence greeting for a birthday card.

Base: an unrelated bulleted list of vague social advice.

SFT: "The free meeting is open for all people... The national party has the following 82 seats: The Democratic Party..." — completely unrelated to the prompt, and no better than base here.

No win — classification:

Prompt: Classify the following as either a fruit or a vegetable: broccoli

Base: never answers the classification, rambles about diet generally.

SFT: "broccoli, broccoli, cabbage,iant cucumber, cucumber, ormanus... This fruit is most closely associated with pears and cucumber." — also fails to answer, with garbled sub-word artifacts not seen in the base output.

No GGUF build

Every other release on the Vertex AGI Hugging Face org ships GGUF alongside the native weights. This one doesn't, and that's deliberate rather than an oversight.

llama.cpp's closest matching architecture (deepseek, the original DeepSeekMoE design) shares this model's leading-dense-layers + fine-grained routed-experts + always-on shared-experts shape almost exactly — except it skips renormalizing the selected top-6 expert weights after routing, while this model's trained behavior does renormalize them. Measured on real prompts, the top-6 experts this model picks only carry ~61% of the total softmax mass on average (26–99% depending on the token) before that renormalization — so a GGUF built on that architecture would run the routed-expert pathway at a token-varying fraction of its trained strength, not the same computation this checkpoint actually does. Rather than ship a file that loads and runs but isn't faithful to the model, we're leaving GGUF out until llama.cpp has a matching architecture (or we write one).

Files

• model.safetensors — model weights, fp32 (optimizer state dropped)
• configuration_quartz.py, modeling_quartz.py — transformers-compatible PretrainedConfig/PreTrainedModel wrapper (QuartzMoEConfig, QuartzForCausalLM), wired up via auto_map in config.json
• model.py, config.py — the original plain-PyTorch model class and architecture config, used by load_model.py
• config.json — architecture config in HF's expected format
• tokenizer.json, tokenizer_config.json, special_tokens_map.json — a standard transformers fast tokenizer
• tokenizer/vocab.json, tokenizer/merges.txt — the custom tokenizer's raw vocab/merges, used by load_model.py's plain-PyTorch path. Only compatible with this model's weights.
• load_model.py — minimal working example using the plain-PyTorch path

Usage

Via transformers (recommended):

pip install transformers torch

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained(
    "VertexAGI/quartz-micro-preview", trust_remote_code=True
)
tok = AutoTokenizer.from_pretrained("VertexAGI/quartz-micro-preview")

prompt = "### Instruction:\nWhy do cats purr?\n\n### Response:\n"
ids = tok(prompt, return_tensors="pt")
out = model.generate(**ids, max_new_tokens=80, do_sample=True, temperature=0.7)
print(tok.decode(out[0], skip_special_tokens=True))

trust_remote_code=True is required — this is a bespoke architecture (DeepSeek-style fine-grained MoE), not one of transformers' built-in model types. No KV-cache support yet, so generate() recomputes attention over the full sequence each step.

Plain PyTorch (no transformers dependency):

pip install torch safetensors tokenizers
python load_model.py

Limitations

Same size-driven limitations as the base checkpoint (factual unreliability, weak general knowledge, no safety fine-tuning), plus SFT-specific ones documented honestly above: strict format/length instructions (e.g. "write exactly two sentences") are not reliably followed, and generation occasionally produces garbled sub-word artifacts the base checkpoint doesn't. Treat this as a proof-of-concept instruction tune on a genuinely tiny model, not a general-purpose assistant.

---

Built by Vertex AGI. Every model we ship — weights, not just claims.