tm12b-gguf

A Q4_K_M GGUF of Futuremannn/tussie-mussie-12B, converted with llama.cpp at tag b9437 (commit aa46bda). Access is gated with manual review.

Read this first: the source is 4-bit

tussie-mussie-12B is not published in fp16 or bf16. It is a bitsandbytes NF4 4-bit checkpoint with double quantisation — quant_method: bitsandbytes, bnb_4bit_quant_type: nf4, bnb_4bit_use_double_quant: true — so its weights are already quantised once. Every *.weight on disk is a 4-bit packed uint8 of half the expected length, sitting beside .absmax, .nested_absmax, .nested_quant_map, .quant_map and .quant_state.bitsandbytes__nf4 sidecars.

llama.cpp has no bitsandbytes codepath, so this file is the product of two lossy steps rather than one:

NF4 4-bit  ->  bf16 (reconstructed)  ->  bf16 GGUF  ->  Q4_K_M

The first step is a reconstruction, not a recovery. PreTrainedModel.dequantize rebuilds the best bf16 representation of the NF4 values, and the NF4 rounding that produced those values is not reversible. This file is therefore quantised twice relative to whatever bf16 merge the source was made from, and it will not match a Q4_K_M produced directly from bf16 weights.

Practical consequence: if fidelity matters, request access to the full-precision weights and quantise from those. This file is the accurate conversion of what is published; it is not the same thing as an accurate conversion of the model.

What is in the file

GGUF architecture	llama (Mistral-Nemo-12B geometry)
Parameters	~12.2 B
Shape	40 blocks, hidden 5120, FFN 14336, 32 heads / 8 KV heads, head_dim 128
Context	131072, rope theta 1e6
Tokenizer	BPE (gpt2), 131074 tokens
Quantisation	Q4_K_M (general.file_type 15)
Chat template	embedded — ChatML, see below

Running it

llama-cli -m tm12b-Q4_K_M.gguf -p "Once upon a time" -n 128

llama-server -m tm12b-Q4_K_M.gguf -c 8192 --host 127.0.0.1 --port 8080

The chat template is embedded, so llama-server's /v1/chat/completions applies it automatically — no --chat-template needed.

The template is ChatML, not Mistral [INST]. This is a Mistral-Nemo architecture, but the source repository ships a chat_template.jinja in <|im_start|>role ... <|im_end|> form, and that is what is embedded here. The model was not tuned on Mistral's prompt format; format it that way and quality drops. The converter would have silently fallen back to llama.cpp's community Mistral template — a [INST] template this model was never trained on — if the source's chat_template.jinja had not been carried into the conversion, which is why the conversion asserts the template is present before publishing.

How it was made

Entirely on Modal, CPU-only, in the volume-backed stages the project runs:

1. Fetch the gated source at a pinned full commit SHA (89edc76f16b59db3eb6e23422495af1fdd36bef0).
2. Reconstruct to bf16 with transformers 5.13.1 and bitsandbytes 0.50.2, then write a plain bf16 checkpoint whose config.json no longer advertises quant_method: bitsandbytes. The chat_template.jinja and tokenizer files are copied across unchanged.
3. Convert with convert_hf_to_gguf.py --outtype bf16 and quantise with llama-quantize ... Q4_K_M, both built from llama.cpp b9437. The intermediate GGUF is written to local scratch; only the result is kept.
4. Verify by serving the quantised file with llama-server and taking a real completion from both /completion and /v1/chat/completions, then reading the file's own header back to confirm architecture, head dimension and the embedded chat template. The chat endpoint is included because a template can be present and still fail to render.
5. Publish to this repository, gated with manual approval before the weights were uploaded rather than after.

Caveats, stated plainly

• Double quantisation, as above. This is the main one.
• No evaluation was run. Nothing here has been benchmarked against the source or against a full-precision conversion. Correctness of the pipeline was verified; quality of the output was not.
• The source carries no model card — no description of the data, the training procedure, or the licence. This repository cannot supply what the source does not state, and inherits whatever the source's terms are.
• Templates and tokenizer are as published upstream; if the upstream repository changes them, this file will not.