LMTLS5/ternary-qwen3.5-4b

Ternary Qwen3.5-4B

Qwen3.5-4B with linear weights decomposed as

W ≈ B · diag(D) · C        B, C ∈ {-1, 0, +1}   D: fp16 scales

Variant	Decomposed weights
plain	MLP gate_proj, up_proj, down_proj in all 32 layers (96 weights)
imatrix	the same 96 MLP weights plus attention: q/k/v/o_proj in the 8 full-attention layers and in_proj_qkv, in_proj_z, out_proj in the 24 linear-attention layers (200 weights)

Everything else (embeddings, norms, vision tower, other small tensors) is unchanged.

Contents

Folder	What it is	Size
qwen3.5-4b-ternary-plain/	Decomposed matrices (plain greedy decomposition)	6.3 GB
qwen3.5-4b-ternary-imatrix/	Decomposed matrices (imatrix-weighted), inner rank k = 3.0 × min(m, n)	5.8 GB
qwen3.5-4b-ternary-imatrix-m3.5/	Same algorithm, k = 3.5 × min(m, n)	6.4 GB
qwen3.5-4b-ternary-recon-bf16/	Dense bf16 model reconstructed from the imatrix variant at mult 3.0 (so MLP and attention are both ternary-reconstructed), standard HF format	9.3 GB
qwen3.5-4b-ternary-recon-m3.5-bf16/	Dense bf16 reconstruction of the mult 3.5 variant	9.3 GB
load_ternary.py	Loader for the decomposed files

The decomposed files are a single model.safetensors each. Per weight they store bit-packed B_mask, B_sign, C_mask, C_sign and fp16 D; shapes and energy preserved are in the ternary metadata. The two files use different bit packing, and load_ternary.py detects which automatically.

Quality vs. original bf16

KL(P_bf16 ‖ P_model) per token, on the WikiText-2 test split (64 chunks × 2048 tokens), measured on the reconstructed dense bf16 models. Original bf16 perplexity: 9.019.

Imatrix variant	Mean KLD (nats)	Median KLD	p99 KLD	Top-1 agreement	Perplexity
mult 3.0	0.0441	0.0194	0.343	91.1%	9.299 (+3.1%)
mult 3.5	0.0224	0.0088	0.150	94.0%	9.090 (+0.8%)

Per-matrix energy preserved (squared-norm): mult 3.5 has min 99.21% / mean 99.76% over the 200 decomposed matrices. Single eval text and no confidence intervals; use as a relative comparison.

Use the reconstructed model

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from huggingface_hub import snapshot_download

path = snapshot_download("LMTLS5/ternary-qwen3.5-4b", allow_patterns="qwen3.5-4b-ternary-recon-bf16/*")
path += "/qwen3.5-4b-ternary-recon-bf16"

tok = AutoTokenizer.from_pretrained(path)
model = AutoModelForCausalLM.from_pretrained(path, dtype=torch.bfloat16).cuda().eval()

Needs a transformers version that supports Qwen3.5.

Load the ternary factors directly with transformers

LMTLS5/ternary-qwen3.5-4b-hf packages the mult 3.5 imatrix factors as a transformers checkpoint. Each projection is a DecomposedLinear with A, B, C tensors, and W is never built:

model = AutoModelForCausalLM.from_pretrained("LMTLS5/ternary-qwen3.5-4b-hf", trust_remote_code=True,
                                             dtype=torch.bfloat16, device_map="cuda")

Load the decomposed matrices

from huggingface_hub import hf_hub_download
from load_ternary import ternary_names, load_ternary, reconstruct   # load_ternary.py from this repo

p = hf_hub_download("LMTLS5/ternary-qwen3.5-4b", "qwen3.5-4b-ternary-imatrix/model.safetensors")
names = ternary_names(p)                 # all decomposed weights
B, D, C = load_ternary(p, names[0])      # B (m,k), D (k,), C (k,n); B, C in {-1,0,+1}

W = reconstruct(B, D, C)                 # dense (m, n) weight
y = ((x @ C.T) * D) @ B.T                # same as x @ W.T, without building W

Load one weight at a time: unpacking every weight to fp32 at once needs a lot of RAM. Non-decomposed tensors are ordinary tensors in the same file and can be read with safetensors.safe_open.