Strikesure5555/Lizzy-7B-mlx-4bit

Lizzy-7B — MLX (4-bit)

Native MLX port of Flower Labs' Lizzy-7B, a UK-built open-weight model released April 2026.

This is an unofficial community port, not affiliated with or endorsed by Flower Labs. No MLX support existed for this model before this. Flower Labs published their own GGUFs, but those require their own llama.cpp fork — the architecture was never upstreamed to mainline llama.cpp. This is the first MLX port.

4-bit quantized, ~3.8GB, ~4.8GB peak RAM. The fastest and smallest of the three variants — roughly 2.7x the decode speed of bf16. See the note below before picking this over 8-bit or bf16, though.

Why a from-scratch port

Lizzy's trust_remote_code implementation requires transformers>=5.4.0 — every release older than that breaks it (below 5.0, the tokenizer needs a TokenizersBackend class that doesn't exist yet; on 5.0-5.3, the model code calls Cache.get_mask_sizes() with the pre-5.4 convention and crashes at generation). It works fine from 5.4.0 through current. Rather than depend on trust_remote_code at all, this is a real architecture port: native MLX code, no fragile version pin.

Turns out Lizzy-7B is architecturally close to OLMo3 — post-norm decoder layout, full-width QK-norm (applied before the head reshape, not per-head), and YARN RoPE scaling on the 8 full-attention layers only (the 24 sliding-attention layers use plain RoPE). That's a more useful starting point than it sounds — most of the actual work was getting those specifics right, not inventing anything new.

A note on precision vs. instruction-following

In testing, I found that how strictly Lizzy follows a detailed system-prompt instruction (e.g. "always respond in second person, never narrate the scene") degrades with quantization. At bf16, an explicit instruction reliably changed the model's output. At 8-bit, the same instruction had a real but partial effect. At 4-bit, it had essentially no effect — the model's behavior didn't respond to the instruction at all, even under deterministic (greedy) decoding, so this isn't just sampling noise.

This is specific to how well the model takes correction from the system prompt on a fairly demanding instruction, not a general quality collapse — 4-bit output is still coherent and on-topic. But if your use case leans on the model actually adjusting its behavior when you tell it to, this is the variant least likely to comply.

Validation

The bf16 conversion was checked against the reference PyTorch implementation across fp32 layerwise parity, bf16 logit agreement, greedy-decode exact match, and KV-cache self-consistency (including negative controls — deliberately broke the implementation a few different ways to confirm the checks actually catch it). This 4-bit variant was checked against that bf16 baseline directly: coherent output at temp 0.0 and 0.6, and 93% teacher-forced next-token agreement with bf16 (837/900 tokens across 3 prompts; of the 63 disagreements, 51 were near-ties and only 12 were real drift, concentrated in the shortest prompt). Normal INT4 quantization noise, not a behavioral collapse. Full writeup in the source repo.

Usage

lizzy isn't in stock mlx-lm yet (PR pending — ml-explore/mlx-lm). Until it lands, you need the registration shim to make mlx_lm recognize the architecture:

import importlib.util, sys
import mlx.core as mx

def register_lizzy(model_file="lizzy.py"):
    spec = importlib.util.spec_from_file_location("mlx_lm.models.lizzy", model_file)
    mod = importlib.util.module_from_spec(spec)
    sys.modules["mlx_lm.models.lizzy"] = mod
    spec.loader.exec_module(mod)

register_lizzy()  # lizzy.py from this repo, same directory or adjust the path

from mlx_lm import load, generate
model, tokenizer = load("Strikesure5555/Lizzy-7B-mlx-4bit")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello"}], add_generation_prompt=True, tokenize=False
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=200))

Once the mlx-lm PR merges, the shim won't be necessary — mlx_lm.load will pick up the architecture natively.

License

Apache-2.0, same as the base model.