coder543/LLaDA2.2-mini-minnow

LLaDA2.2-mini for minnow

Self-contained .mnw checkpoints of InclusionAI's LLaDA2.2-mini for minnow, a Rust block-diffusion inference server with an OpenAI-compatible Chat Completions API. InclusionAI developed and trained the model; this repository provides format conversion and quantization.

Checkpoints and precision

Quantization applies to routed experts. Their gate, up, and down projection weights use the formats below. Attention, embeddings, routers, dense/shared experts, and the output head retain their source floating-point weights, primarily BF16. Default CUDA execution uses BF16 for the surrounding model; the activation column describes inputs to the routed expert matrix multiplications.

Checkpoint	Size	Routed expert weights	Expert activations on CUDA, by default
llada2.2-mini-bf16.mnw	30.29 GiB	Original floating-point weights, primarily BF16; no quantization.	BF16.
llada2.2-mini-int8.mnw	16.26 GiB	Signed INT8; one FP16 scale per 128 weights.	BF16 (W8A16).
llada2.2-mini-int4.mnw	9.14 GiB	Signed INT4; one FP16 scale per 128 weights.	BF16 (W4A16).
llada2.2-mini-nvfp4.mnw	9.81 GiB	E2M1 FP4; E4M3 scales per 16 weights and an FP32 outer scale per expert matrix.	Dynamically quantized E2M1 FP4, with E4M3 scales per 16 values and an FP32 outer scale per input row (W4A4).

W means weight bits and A means activation bits; A16 here means BF16. INT4 and INT8 are weight-only quantizations by default: CUDA kernels dequantize weights into BF16 registers for tensor-core multiplication without keeping a full expanded weight copy.

For either INT4 or INT8, --int8-expert-activations optionally selects W4A8 or W8A8 on Ampere-or-newer CUDA GPUs. This dynamically quantizes expert inputs to signed INT8 with FP32 scales per 128 values, uses INT32 group dot products, and combines scaled groups in FP32. Expert outputs return to BF16. It uses the same checkpoint and changes model numerics; it does not quantize the rest of the model's activations.

NVFP4 quantizes both expert weights and expert inputs. SM120/121 Blackwell uses native FP4 tensor cores. Ampere/Ada uses a BF16 tensor-core fallback that reconstructs the already quantized FP4 operands; the fallback retains FP4 activation quantization. --int8-expert-activations applies only to INT4/INT8 experts.

Usage

Build current minnow using its README. INT4 requires commit 0e6678f or newer. Download and serve:

hf download coder543/LLaDA2.2-mini-minnow llada2.2-mini-int4.mnw --local-dir models
minnow --model models/llada2.2-mini-int4.mnw serve

# Optional integer expert activations, using the same checkpoint:
minnow --model models/llada2.2-mini-int4.mnw --int8-expert-activations serve

All files include configuration, tokenizer, and chat template. Allow additional memory for K/V, activations, and workspaces. These files require minnow and are not directly loadable by Transformers or llama.cpp. CPU fallback defaults to FP32 execution with one expert expanded at a time; the CUDA precision table does not describe CPU execution.

Quantization and activation mode can change outputs; upstream evaluation scores have not been re-measured for these files. See RTX 3090 measurements for comparisons under the documented settings.

Licensed under Apache-2.0, matching the upstream model.