elephant-1-9B-MLX-4bit

MLX (Apple silicon) conversion of baselabs/elephant-1, a full fine-tune of Qwen/Qwen3.5-9B published as a typed decision model: it answers every question of a request with a probability per option from one shared-prefix prefill, without generating text.

• Format: 4-bit affine, group size 64
• Size on disk: 4.71 GB (8.95B parameters)
• Peak memory when generating: 5.2 GB
• Converted with mlx-lm 0.31.3

Variant	Quantization	Size on disk	Peak RAM
elephant-1-9B-MLX-bf16	bfloat16	16.70 GB	18.0 GB
elephant-1-9B-MLX-8bit	8-bit affine, group size 64	8.90 GB	9.6 GB
elephant-1-9B-MLX-4bit	4-bit affine, group size 64	4.71 GB	5.2 GB

Decision server (/v1/systemone)

decision_mlx.py and serve.py in this repo are an MLX port of the checkpoint's decision path, so the model is served exactly as it was trained (slot_lm readout, block-bidirectional full attention). The HTTP API is the same as the reference runtime (OneJev/qev @81ce62f + serve.patch).

pip install mlx-lm fastapi uvicorn pydantic
python serve.py --model . --port 8000          # local download
python serve.py --model 0x36/elephant-1-9B-MLX-4bit            # straight from the Hub

Endpoints: POST /v1/systemone (decide), GET /v1/models, GET /health, GET / (playground).

Request

state is any JSON value (object, string, list) and is shared by all questions. Each question is one of noul (yes/no), choice (one option per key of criteria) or score (one level per entry of criteria, up to 10). Query parameters: ?debug=1 adds slot logits and the slot probability mass, ?debias=N averages over N cyclic shifts of the option order (default 1).

curl -s -X POST http://127.0.0.1:8000/v1/systemone -H 'content-type: application/json' -d '{
  "state": {"ticket": "My card was charged twice for the same order and support has not replied in 3 days.",
            "customer": {"tenure_months": 14, "vip": false}},
  "questions": {
    "refund": {"type": "noul", "instructions": "Is the customer asking for money back?"},
    "team":   {"type": "choice", "instructions": "Which team should handle this?",
               "criteria": {"billing": "charges and refunds", "technical": "bugs and outages", "sales": "pricing and plans"}},
    "frustration": {"type": "score", "instructions": "How frustrated is the customer?",
                    "criteria": ["calm", "irritated", "angry"]}
  }
}'

{
  "model": "elephant-1",
  "answers": {
    "refund":      {"type": "noul", "noul": 0.667},
    "team":        {"type": "choice", "choice": "billing",
                    "probabilities": {"billing": 0.999, "technical": 0.0006, "sales": 0.0004},
                    "confidence": 0.999},
    "frustration": {"type": "score", "score": 1.63, "legend": {"0": "calm", "1": "irritated", "2": "angry"},
                    "probabilities": {"0": 0.045, "1": 0.280, "2": 0.676}, "confidence": 0.446}
  },
  "usage": {"input_tokens": 274, "output_tokens": 3},
  "qev": {"prompt_version": "qev-labels-v2", "prefix_tokens": 99, "branches": 3, "fork_mode": "sequential",
          "timing": {"render_s": 0.01, "forward_s": 1.69, "total_s": 1.71}}
}

Python client

import requests

r = requests.post("http://127.0.0.1:8000/v1/systemone", json={
    "state": {"ticket": "I want my money back for order 4471."},
    "questions": {"refund": {"type": "noul", "instructions": "Is the customer asking for money back?"}},
})
print(r.json()["answers"]["refund"])   # {'type': 'noul', 'noul': 0.9997}

How the decision path works

1. Render every branch (qev/prompt.py template: system prompt + <state> block + question, options and answer line) and cut the longest common token prefix that ends at the state block. One prefill of that prefix is shared by every question of the request.
2. Score each branch from a copy of the prefix cache: the answer position is the last token of the branch and the readout is slot_lm — a softmax over the LM-head logits of the 256 answer labels (A..Z, then two-letter labels), computed from float32 slot rows.
3. Full-attention layers run block-bidirectional (modeling_overrides in config.json), which is mask=None on the prefix and on the suffix forward; GatedDeltaNet layers stay causal. Branches run sequentially — the batched CUDA fork of the reference runtime is a speed optimization, the sequential path is the exact one.
4. Answers are assembled with TypeSafe's confidence formulas (qev/answers.py) and temperatures from calibration.json when the checkpoint ships one (elephant-1 does not).

Fidelity notes

• Rendering, request/response schema and answer assembly are ported 1:1 from OneJev/qev @81ce62f + serve.patch (Apache-2.0) and checked byte-identical against that code (7 question shapes incl. the 255-option maximum, with and without option-order permutations).
• The readout head is not ported: elephant-1 uses slot_lm, which needs no extra weights. A checkpoint whose readout.json says head is rejected at load.
• Numeric parity with a CUDA run was not measured (no CUDA here). Slot logits are computed in float32 like the reference, but this repo's quantization is not in the reference path.

What is included

The full Qwen3.5 text decoder, in the layout mlx-lm expects (language_model.*): 32 layers alternating gated-delta-net linear attention (24) and full attention (8, every 4th layer), attention output gate, partial RoPE (0.25), untied 248,320-token embedding and LM head. Tokenizer, chat template, generation_config.json, readout.json and the server files above are included.

What is not included

The vision tower (model.visual, 27 blocks, ~0.8B params). mlx-lm has no Qwen3.5-VL implementation, so the text-only language model is what is converted here; the decision path does not use images.

Text generation (optional)

The LM also generates text with plain mlx-lm, which is useful for inspection but is not the trained task:

python -m mlx_lm generate --model 0x36/elephant-1-9B-MLX-4bit --prompt "The capital of France is" --max-tokens 64

Validation

Last-position logits for the prompt The capital of France is (248,320-token vocab) vs. this repo's bf16 conversion:

Metric	Value
Cosine similarity	0.994248
KL(bf16 || 4bit)	0.02132
Top-1 token	identical (both Paris)
Top-10 overlap	10/10

Conversion

mlx_lm.convert --hf-path baselabs/elephant-1 --mlx-path mlx-bf16 --dtype bfloat16
mlx_lm.convert --hf-path mlx-bf16 --mlx-path mlx-4bit -q --q-bits 4 --q-group-size 64

The conversion shifts the zero-centered RMSNorm weights (1 + w) and transposes the depthwise linear_attn.conv1d kernel to MLX's layout; both are handled by mlx-lm's Qwen3.5 sanitize.

Credits

Original checkpoint: baselabs/elephant-1 (MIT). Decision runtime: OmniJev/OneJev qev @81ce62f + serve.patch (Apache-2.0), kit apolinario/decision-index (MIT).