Instructions to use IAMIbrahim/luthor-8b-mlx-4bit with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use IAMIbrahim/luthor-8b-mlx-4bit with MLX:

# Make sure mlx-lm is installed
# pip install --upgrade mlx-lm

# Generate text with mlx-lm
from mlx_lm import load, generate

model, tokenizer = load("IAMIbrahim/luthor-8b-mlx-4bit")

prompt = "Write a story about Einstein"
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
    messages, add_generation_prompt=True
)

text = generate(model, tokenizer, prompt=prompt, verbose=True)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use IAMIbrahim/luthor-8b-mlx-4bit with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "IAMIbrahim/luthor-8b-mlx-4bit"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "IAMIbrahim/luthor-8b-mlx-4bit"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

MLX LM

How to use IAMIbrahim/luthor-8b-mlx-4bit with MLX LM:

Generate or start a chat session

# Install MLX LM
uv tool install mlx-lm
# Interactive chat REPL
mlx_lm.chat --model "IAMIbrahim/luthor-8b-mlx-4bit"

Run an OpenAI-compatible server

# Install MLX LM
uv tool install mlx-lm
# Start the server
mlx_lm.server --model "IAMIbrahim/luthor-8b-mlx-4bit"
# Calling the OpenAI-compatible server with curl
curl -X POST "http://localhost:8000/v1/chat/completions" \
   -H "Content-Type: application/json" \
   --data '{
     "model": "IAMIbrahim/luthor-8b-mlx-4bit",
     "messages": [
       {"role": "user", "content": "Hello"}
     ]
   }'

Hermes Agent

How to use IAMIbrahim/luthor-8b-mlx-4bit with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "IAMIbrahim/luthor-8b-mlx-4bit"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default IAMIbrahim/luthor-8b-mlx-4bit

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use IAMIbrahim/luthor-8b-mlx-4bit with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "IAMIbrahim/luthor-8b-mlx-4bit"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "IAMIbrahim/luthor-8b-mlx-4bit" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Luthor 8B — MLX 4bit

4-bit-quantised MLX build of Luthor 8B, a Qwen3-8B fine-tune for driving terminal and file-editing tools in an agent loop. For Apple Silicon.

Recommended default. Best size/speed/quality balance.

Status: evaluated — it did NOT pass its ship gate. 0/10 on held-out tasks, same as stock Qwen3-8B.

This build


Size on disk	4.3 GB
Bits per weight	4.500
Generation speed	19.2 tok/s
Peak memory	4.79 GB

All builds

build	size	bits/weight	tok/s	peak RAM
`-mlx-8bit`	8.1 GB	8.500	10.6	8.80 GB
`-mlx-6bit`	6.2 GB	6.500	13.8	6.77 GB
`-mlx-4bit`	4.3 GB	4.500	19.2	4.79 GB
`-mlx-mixed-3-6`	3.9 GB	4.088	21.1	4.38 GB

Measured on an Apple M3 (24 GB), mlx-lm 0.31.3, 150-token generation at --temp 0.0.

A note on speed

Larger quantisations are slower here, not faster. Apple Silicon inference is memory-bandwidth-bound, so fewer bytes per weight means more tokens per second. 8-bit is the highest fidelity and the slowest.

What didn't ship

A mixed_2_6 build (3.2 GB, 3.284 bits/weight) was produced and discarded: it degenerates into repeated tokens rather than coherent text. Post-training quantisation below ~4 bits/weight breaks this model. Reaching ternary-class compression — as Bonsai 2 does at ~1.58 bits/weight — requires quantisation-aware training, not post-training conversion.

Usage

pip install mlx-lm
mlx_lm.generate --model IAMIbrahim/luthor-8b-mlx-4bit \
  --prompt "The test suite fails with ImportError. What is your first step?" \
  --max-tokens 512

from mlx_lm import load, generate
model, tokenizer = load("IAMIbrahim/luthor-8b-mlx-4bit")
messages = [
    {"role": "system", "content": SYSTEM_PROMPT_WITH_TOOLS},   # Hermes-style <tools> block
    {"role": "user", "content": "Fix the failing test."},
]
prompt = tokenizer.apply_chat_template(messages, add_generation_prompt=True)
print(generate(model, tokenizer, prompt=prompt, max_tokens=512))

Declare tools exactly as at training time — the model emits <tool_call> and expects <tool_response> back.

Conversion

python -m mlx_lm convert --hf-path IAMIbrahim/luthor-8b \
  --mlx-path luthor-8b-4bit -q --q-bits 4 --q-group-size 64

See the base model card for training data, hyperparameters and limitations.

Evaluation

This model did not pass its ship gate: 0/10 on held-out tasks, identical to stock Qwen3-8B, with worse protocol adherence (completed 0/10 vs 8/10). Published as a negative result and a reproducible pipeline, not as an improvement over the base model. Full numbers and analysis on the base model card.