ToPo-ToPo/Inkling-Small-mlx-2bit

MLX 2bit conversion of thinkingmachines/Inkling-Small for Apple Silicon (mlx-vlm). 276B total / 12B active sparse MoE (42 layers, 256 routed experts top-6 + 2 shared), text + image + audio in, text out.

See also: ToPo-ToPo/Inkling-Small-mlx-4bit.

Provenance (self-converted from official weights)

• Source: thinkingmachines/Inkling-Small (license: apache-2.0, bf16, 531.9 GB)
• Tool: mlx-vlm 0.6.7 — mlx_vlm.convert --hf-path <staged> --mlx-path . -q --q-bits 2 --q-group-size 64
• Effective: 2.506 bits/weight (77 GiB on disk, ~83.5 GB peak RSS at inference)

Config translation (already applied here)

The official config.json cannot be loaded by mlx-vlm as published — its schema uses different key names than mlx_vlm.models.inkling expects. This repo ships the translated config, so no action is needed:

Key	Official	Here
model_type	inkling_mm_model	inkling
text_config.intermediate_size	2048 (expert width)	16384 (dense width)
text_config.moe_intermediate_size	— (was intermediate_size)	2048
vision_config.text_hidden_size	decoder_dmodel	4096
audio_config.text_hidden_size	decoder_dmodel	4096
vision_config.num_channels	n_channels	3
tokenizer_config.pad_token / eos_token	unset	<|endoftext|> / <|content_model_end_sampling|>

image_token_id / audio_token_id are left unset so that mlx-vlm's defaults (200054 / 200053) apply — those are the placeholders InklingProcessor actually emits.

Loader shim (required)

mlx-vlm 0.6.7 and 0.6.8 need two patches at runtime for any Inkling checkpoint. Apply them before load():

from mlx_vlm.models import inkling
from mlx_vlm.models.inkling import config as inkling_config
from mlx_vlm import prompt_utils

# 1. models/inkling does not re-export its sub-config classes, which the generic
#    loader fetches with getattr -> AttributeError on load.
for name in ("TextConfig", "VisionConfig", "AudioConfig"):
    setattr(inkling, name, getattr(inkling_config, name))

# 2. prompt_utils.MODEL_CONFIG has no "inkling" entry, so apply_chat_template treats
#    the model as text-only and drops image/audio parts.
prompt_utils.MODEL_CONFIG.setdefault("inkling", prompt_utils.MessageFormat.LIST_WITH_IMAGE_FIRST)

When serving over mlx_vlm.server, also add Inkling's structural tokens to mlx_vlm.server.responses_state._CONTENT_MARKERS (<|message_model|>, <|content_text|>, <|end_message|>) and set MLX_VLM_THINKING_START_TOKEN=<|content_thinking|> / MLX_VLM_THINKING_END_TOKEN=<|end_message|>, so the reasoning channel and those markers do not end up in content.

Reasoning effort

The chat template always injects a Thinking effort level: system message (default 0.9). Control it with the OpenAI-compatible reasoning_effort — "none" / "minimal" / "low" / "medium" / "high" / "max", or a float in [0.0, 0.99]. "none" disables thinking entirely.

Usage

from mlx_vlm import load, generate
model, processor = load("ToPo-ToPo/Inkling-Small-mlx-2bit")  # apply the shim above first

The first load reads 77 GiB, so allow a few minutes and raise any client-side startup timeouts accordingly.