Minueza-3-95M-Base

Summary

Minueza-3-95M-Base is a compact English language model with the Gemma 3 architecture, trained from scratch on 1.95 billion tokens with a context length of 8192. At 94.7 million parameters it is a lightweight foundation meant to be fine-tuned, not used as-is.

It is distributed only as GGUF, because GGUF was the only weight format used during training: there is no PyTorch checkpoint behind these files. The f32 file is the training master and can continue pretraining; the q8_0 and q4_0 files are for inference.

Being this small, the model has severe limitations in reasoning and factual knowledge, and will produce fluent text that is frequently wrong. It was trained on web text and can reproduce the biases in it.

Usage

With llama.cpp:

llama-completion -hf Felladrin/Minueza-3-95M-Base:Q8_0 -c 8192 -n 256 \
  --temp 0.7 --top-p 0.85 --top-k 30 --min-p 0.02 \
  --repeat-penalty 1.15 --repeat-last-n 128 \
  -no-cnv -p "This book tells the story"

-hf downloads and caches the weights for you. Swap :Q8_0 for :Q4_0 to pull the smaller copy, or -m <file>.gguf to point at a local one.

This is a base model: it completes text and has no chat template. The repetition penalty above is not optional, a model this size loops without it.

The q4_0 file is 55 MB, small enough to run in a browser tab through Wllama.

Intended Uses

• A base for fine-tunes: instruct, classification/routing, reranking.
• Running fast on machines without a GPU, and inside browsers via Wllama.
• A from-scratch training artifact to study, with the optimizer state published so the pretraining can be continued rather than restarted.

Model Architecture

Gemma 3, with sliding-window attention on 5 of every 6 layers.

Configuration	Value
context_length	8192
embedding_length	640
block_count	12
feed_forward_length	2560
attention.head_count	10
attention.head_count_kv	5
attention.key_length	64
attention.sliding_window	1024
sliding_window_pattern	6 (5 local, 1 global)
rope.freq_base	1000000 (global) / 10000 (local)
vocab_size	32768 (byte-level BPE)
parameters	94.7M (tied embeddings)

Special Tokens

Eleven tokens were reserved before pretraining and are frozen into the vocabulary, so a later fine-tune can use them without touching the embedding matrix. Pretraining never emitted any of them, which means their embedding rows still sit at initialization: the first fine-tune whose data uses a token is the one that trains it. Nothing can be added later, because the vocabulary and the embedding matrix both froze at step 1, so a fine-tune that needs a marker has to reuse one of these or spell it out in plain text.

ID	Token	token_type
32757	<|endoftext|>	CONTROL (3)
32758	<|im_start|>	CONTROL (3)
32759	<|im_end|>	CONTROL (3)
32760	<think>	USER_DEFINED (4)
32761	</think>	USER_DEFINED (4)
32762	<tool_call>	USER_DEFINED (4)
32763	</tool_call>	USER_DEFINED (4)
32764	<tools>	USER_DEFINED (4)
32765	</tools>	USER_DEFINED (4)
32766	<tool_response>	USER_DEFINED (4)
32767	</tool_response>	USER_DEFINED (4)

The <|...|> tokens are CONTROL, so llama.cpp handles them and never renders them as visible text. The <...> tokens are USER_DEFINED, so they survive into the output, where llama.cpp's --jinja reasoning and tool-call parsers can see them.

The set is ChatML plus Qwen-style reasoning and tool calling, which is what it was chosen for, but nothing in the weights commits you to that template. Any format works; only these eleven strings tokenize as a single atomic token, and everything else costs one token per piece.

EOS moves when you add a chat template. This base declares eos_token_id = 32757 (<|endoftext|>), the document separator it was trained on. A ChatML fine-tune ends turns with <|im_end|>, so its GGUF has to declare 32759 instead, or generation will not stop on a turn boundary. Everything else about the tokenizer stays identical.

A template has to render in your trainer as well as in llama.cpp. Training text and inference text must be byte-identical, so the same Jinja source has to work in both engines, and they are not equally permissive: llama.cpp uses minja, while the trainer used here uses @huggingface/jinja 0.3.3. A template that renders only under minja will throw when you tokenize your fine-tuning data, or worse, quietly diverge from it. Measured against 0.3.3, these constructs fail and need a rewrite:

Construct	Use instead
~ string concatenation	+
the default filter	an is defined conditional
.startswith() / .endswith()	an in test
loop.previtem / loop.nextitem	messages[loop.index0 - 1] / messages[loop.index0 + 1]

These work: namespace(), macro (including default arguments), messages[::-1], | items, | tojson, raise_exception() and inline if expressions.

Suggested chat template

Nothing in the weights requires this one, but it is a working starting point: it uses all ten non-<|endoftext|> tokens, avoids every construct in the table above, and was checked to render byte-identically under @huggingface/jinja 0.3.3 and llama.cpp's minja across plain chat, a system prompt with multiple turns, reasoning in the history, and a tool call followed by a tool response.

{%- if tools -%}
    {{- "<|im_start|>system\n" -}}
    {%- if messages[0]["role"] == "system" -%}
        {{- messages[0]["content"] + "\n\n" -}}
    {%- endif -%}
    {{- "<tools>\n" -}}
    {%- for tool in tools -%}
        {{- (tool | tojson) + "\n" -}}
    {%- endfor -%}
    {{- "</tools><|im_end|>\n" -}}
{%- elif messages[0]["role"] == "system" -%}
    {{- "<|im_start|>system\n" + messages[0]["content"] + "<|im_end|>\n" -}}
{%- endif -%}
{%- for message in messages -%}
    {%- if message["role"] == "tool" -%}
        {{- "<|im_start|>user\n<tool_response>\n" + message["content"] + "\n</tool_response><|im_end|>\n" -}}
    {%- elif not loop.first or message["role"] != "system" -%}
        {{- "<|im_start|>" + message["role"] + "\n" -}}
        {%- if message["reasoning_content"] -%}
            {{- "<think>\n" + message["reasoning_content"] + "\n</think>\n\n" -}}
        {%- endif -%}
        {{- message["content"] -}}
        {%- if message["tool_calls"] -%}
            {%- for call in message["tool_calls"] -%}
                {{- ("\n" if message["content"] or not loop.first else "") + "<tool_call>\n" + (call["function"] | tojson) + "\n</tool_call>" -}}
            {%- endfor -%}
        {%- endif -%}
        {{- "<|im_end|>\n" -}}
    {%- endif -%}
{%- endfor -%}
{%- if add_generation_prompt -%}
    {{- "<|im_start|>assistant\n" -}}
{%- endif -%}

With tools, a call and a response, that renders to:

<|im_start|>system
You are terse.

<tools>
{"type": "function", "function": {"name": "get_weather"}}
</tools><|im_end|>
<|im_start|>user
weather in Lisbon?<|im_end|>
<|im_start|>assistant
<tool_call>
{"name": "get_weather", "arguments": {"city": "Lisbon"}}
</tool_call><|im_end|>
<|im_start|>user
<tool_response>
18C
</tool_response><|im_end|>

Three details are load-bearing, in case you rewrite it. not loop.first or message["role"] != "system" keeps a system message at index 0 from being emitted twice while still rendering one that appears mid-conversation, rather than dropping it silently. The {%- if message["tool_calls"] -%} guard is not optional: @huggingface/jinja throws Expected iterable type in for loop on a missing key, where minja ignores it. And the inline ("\n" if message["content"] or not loop.first else "") is what stops an empty-content tool call from opening with a stray blank line.

Keep tool_calls[].function.arguments an object in your training data, not a JSON string. Given a string, the trainer renders "arguments": "{\"city\":\"Lisbon\"}" while llama.cpp parses it first and renders "arguments": {"city": "Lisbon"}. That is a silent divergence between the text you train on and the text you serve, and it was the only disagreement between the two engines. As objects they match exactly.

It leaves out two things on purpose. There is no default system prompt, because a base model's fine-tuner should choose it. And there is no enable_thinking or reasoning_effort switch, because both need the default filter; <think> still renders whenever a message carries reasoning_content.

Training

Two phases, both f32, both on a single AMD Strix Halo APU through the WebGPU backend.

Phase	Corpus	Tokens seen	Steps
A	722M-token blend: c4 ~58%, cosmopedia-100k ~18%, github-code-clean ~14%, open-web-math ~10%	1.44B (2 epochs)	88,000 x 8 x 2048
B	fineweb-edu sample-10BT slice, 508.7M tokens	0.51B (1 epoch)	31,000 x 8 x 2048

Hyperparameter	Value
optimizer	Muon (Newton-Schulz orthogonalized momentum) on matmuls, AdamW on norms and embeddings
learning_rate	0.01 (Muon), 0.003 (AdamW aux)
lr_scheduler	WSD, 10% warmup / 20% cooldown, floor 0.1
batch	8 x 2048 tokens per step
momentum	0.95
grad clip	1.0 (aux group)
precision	f32 throughout (f16 overflowed to NaN at this scale)
seed	1234
throughput	~900 tokens/s, so ~18 days of single-APU compute for phase A and ~6.5 days for phase B

Final training loss: 2.45 on the phase A blend, 2.78 on fineweb-edu at the end of phase B. The two are not comparable to each other, different corpora.

Training used Felladrin's GGUF Trainer, a from-scratch trainer written in TypeScript that keeps weights in GGUF end to end and runs its kernels on WebGPU, with no PyTorch anywhere in the stack. It is open source, and the two files below are exactly what it needs to pick this run up where it stopped:

hf download Felladrin/Minueza-3-95M-Base --local-dir base/
deno run -A cli.ts inspect --model base/Minueza-3-95M-Base.F32.gguf   # prints the flags to repeat
deno run -A cli.ts pretrain --data your.tokens --out out/continued.gguf \
  --resume base/Minueza-3-95M-Base.F32.gguf \
  --hidden 640 --layers 12 --head-dim 64 --window 1024 --max-seq 8192 \
  --steps 5000 --seq-len 2048 --batch 8 --lr 0.01

Keep the .optstate sidecar next to the F32 file and the optimizer resumes warm. Tokenize your corpus with the tokenizer.json published here, never a fresh one: the embedding matrix froze when pretraining started.

Evaluation

llama-perplexity --multiple-choice, HellaSwag validation (10,042 tasks) and ARC-Challenge validation, every model scored by the same script on the same machine.

Model	Params	HellaSwag	ARC-Challenge
Minueza-3-95M-Base	94.7M	29.25 +/- 0.45	24.08 +/- 2.48
Minueza-2-96M	96M	27.03 +/- 0.44	23.08 +/- 2.44
Minueza-32M-Base	32M	25.75 +/- 0.44	22.74 +/- 2.43

Two larger models went through the same harness as reference points:

Reference model	Params	HellaSwag	ARC-Challenge
SmolLM2-135M	135M	42.81 +/- 0.49	31.44 +/- 2.69
llama-160m	162M	33.94 +/- 0.47	22.74 +/- 2.43

Random baselines are 25.0 on both benchmarks. Read the numbers honestly: this model improves on its predecessors, and the references, with 1.4x to 1.7x the parameters and orders of magnitude more training tokens, are further ahead on HellaSwag. ARC-Challenge separates only SmolLM2-135M; every other model here, references included, sits inside noise of chance on it. The value of this checkpoint is as a starting point, not as a scorer.

Samples

Greedy decoding (--temp 0 --top-k 1) with the repetition penalty from the Usage section, which is a logit transform and leaves the run deterministic. -ngl 0 pins it to the CPU: repeat runs on one backend are byte-identical, but a GPU build can pick the other token at a near-tie and diverge from there, so the backend is part of the recipe.

llama-completion -hf Felladrin/Minueza-3-95M-Base:Q8_0 -c 512 -n 60 \
  --temp 0 --top-k 1 --repeat-penalty 1.15 --repeat-last-n 128 --seed 42 \
  -ngl 0 -no-cnv -p "<prompt>"

The capital of France is the city of Paris. The capital was founded in 1621 by King Louis XIV, who had been a French aristocrat and a great-grandson of King Charles I. In 1789, the French government established a new capital at Saint-Louis-sur-Ric

Photosynthesis is the process by which plants produce oxygen and carbon dioxide. The photosynthetic process involves the production of light, water and other substances from sunlight. The photosynthetic process can be divided into two phases: - Phase 1 - the first phase involves the conversion of light energy to chemical energy (photons

The best way to learn a new language is by practicing it. - Learn the basics of grammar and usage, including how to use them correctly. - Practice using your native tongue in order to improve your English speaking skills. - Use the correct pronunciation for each word you are learning.

• Listen carefully to what others have said

Once upon a time, there was a little robot who had been playing with the ball. The robot was called "the ball" and it was named after the famous American inventor, Charles Babbage. He invented the first computer that could read text in any language. The idea of using computers to solve problems was born from his childhood.

That is the honest shape of a 95M base model: fluent English, plausible structure, and facts that are mostly invented. Paris was not founded in 1621, Louis XIV was not alive then, and Charles Babbage was neither American nor a namer of robots. Drop the repetition penalty and greedy decoding collapses into loops within a sentence or two.

Quantization

All three files measured on the same machine: HellaSwag validation (10,042 tasks) through llama-perplexity --multiple-choice, and perplexity on the wikitext-2 raw test set at ctx 2048.

Build	Size	HellaSwag	wikitext-2 PPL
F32	380 MB	29.25 +/- 0.45	31.1893 +/- 0.2429
Q8_0	102 MB	29.17 +/- 0.45	31.1787 +/- 0.2428
Q4_0	55 MB	29.28 +/- 0.45	33.4826 +/- 0.2625

Q8_0 is indistinguishable from F32. Both gaps are an order of magnitude smaller than their own error bars, and greedy decoding from the two files produces identical tokens on most prompts. Use q8_0 for inference; the f32 file is worth its 380 MB only as a training master.

Q4_0 splits the two metrics, and the split is the useful part. It costs 7.4% perplexity while leaving HellaSwag untouched. Perplexity is sensitive to the probability mass on every token, so 4-bit rounding shows up at once; multiple choice only needs the right continuation to rank first, and that ordering survives the added noise. So q4_0 degrades free-running generation and is a worse starting point for a fine-tune, while remaining fine for ranking, scoring, and classification, which is what the browser-sized copy is mostly for.

Caveat on reading any of these: at ~29 against a 25.0 floor, HellaSwag has about four points of headroom here, so it cannot resolve small quality differences. Perplexity is the sensitive instrument at this scale.

Files

File	Size	What it is
Minueza-3-95M-Base.F32.gguf	380 MB	Training master. Use this one to continue pretraining or to fine-tune.
Minueza-3-95M-Base.F32.gguf.optstate	463 MB	Optimizer state (Muon momentum, Adam moments, step counter). Continuing pretraining without it cold-starts the optimizer and re-warms momentum.
Minueza-3-95M-Base.Q8_0.gguf	102 MB	Inference copy.
Minueza-3-95M-Base.Q4_0.gguf	55 MB	Inference copy, small enough for the browser.
tokenizer.json	0.7 MB	The byte-level BPE vocab and merges, also embedded in every GGUF. Reuse this exact file for any fine-tune: the embedding matrix froze when pretraining started.

Limitations

• Facts are unreliable. It writes fluent sentences about things that are not true.
• English only. The vocab is English-centric, so another language needs a new tokenizer and a fresh pretrain.
• No chat template and no instruction following. That is a fine-tune's job.
• It loops without a repetition penalty.

License

Apache License 2.0.