Strikesure5555/Lizzy-7B-mlx-8bit

Lizzy-7B — MLX (8-bit)

Native MLX port of Flower Labs' Lizzy-7B, a UK-built open-weight model released April 2026.

This is an unofficial community port, not affiliated with or endorsed by Flower Labs. No MLX support existed for this model before this. Flower Labs published their own GGUFs, but those require their own llama.cpp fork — the architecture was never upstreamed to mainline llama.cpp. This is the first MLX port.

8-bit quantized, ~7.2GB, ~8.2GB peak RAM. Good middle ground between the bf16 reference and the 4-bit version — noticeably smaller and faster than bf16, and holds up better than 4-bit if you're relying on the model following detailed system-prompt instructions (see note below).

Why a from-scratch port

Lizzy's trust_remote_code implementation requires transformers>=5.4.0 — every release older than that breaks it (below 5.0, the tokenizer needs a TokenizersBackend class that doesn't exist yet; on 5.0-5.3, the model code calls Cache.get_mask_sizes() with the pre-5.4 convention and crashes at generation). It works fine from 5.4.0 through current. Rather than depend on trust_remote_code at all, this is a real architecture port: native MLX code, no fragile version pin.

Turns out Lizzy-7B is architecturally close to OLMo3 — post-norm decoder layout, full-width QK-norm (applied before the head reshape, not per-head), and YARN RoPE scaling on the 8 full-attention layers only (the 24 sliding-attention layers use plain RoPE). That's a more useful starting point than it sounds — most of the actual work was getting those specifics right, not inventing anything new.

A note on precision vs. instruction-following

In testing, I found that how strictly Lizzy follows a detailed system-prompt instruction (e.g. "always respond in second person, never narrate the scene") degrades with quantization. At bf16, an explicit instruction reliably changed the model's output. At 8-bit, the same instruction had a real but partial effect. At 4-bit, it had essentially no effect — the model's behavior didn't respond to the instruction at all, even under deterministic (greedy) decoding, so it's not just sampling noise.

This is specific to how well the model takes correction from the system prompt on a fairly demanding instruction, not a general quality collapse — 4-bit is still coherent and on-topic. But if your use case leans on the model actually adjusting its behavior when you tell it to, 8-bit or bf16 will serve you better than 4-bit.

Usage

lizzy isn't in stock mlx-lm yet (PR pending — ml-explore/mlx-lm). Until it lands, you need the registration shim to make mlx_lm recognize the architecture:

import importlib.util, sys
import mlx.core as mx

def register_lizzy(model_file="lizzy.py"):
    spec = importlib.util.spec_from_file_location("mlx_lm.models.lizzy", model_file)
    mod = importlib.util.module_from_spec(spec)
    sys.modules["mlx_lm.models.lizzy"] = mod
    spec.loader.exec_module(mod)

register_lizzy()  # lizzy.py from this repo, same directory or adjust the path

from mlx_lm import load, generate
model, tokenizer = load("Strikesure5555/Lizzy-7B-mlx-8bit")

prompt = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Hello"}], add_generation_prompt=True, tokenize=False
)
print(generate(model, tokenizer, prompt=prompt, max_tokens=200))

Once the mlx-lm PR merges, the shim won't be necessary — mlx_lm.load will pick up the architecture natively.

Validation

The bf16 conversion was checked against the reference PyTorch implementation across fp32 layerwise parity, bf16 logit agreement, greedy-decode exact match, and KV-cache self-consistency (including negative controls — deliberately broke the implementation a few different ways to confirm the checks actually catch it). This 8-bit variant was spot-checked against that bf16 baseline for coherence at temp 0.0 and 0.6 (fluent, no repetition/degeneracy) — I didn't run the same formal token-agreement measurement on 8-bit that I ran on 4-bit (see that card for the number). Full writeup in the source repo.

License

Apache-2.0, same as the base model.