Llama-3.2-1B-Instruct-HadamardSpin-4bits

Github repo: github

🚀 Overview

HadamardSpin is a high-performance quantization project focusing on preserving the "structural intelligence" of Large Language Models (LLMs) at low-bit precision. This specific version features Llama-3.2-1B-Instruct quantized to 4-bit using a Hadamard-Transform-based Rotation technique, optimized for Apple Silicon via the MLX framework.

The core innovation lies in the Incoherence Processing: using a Hadamard matrix ($n=2048$) to scatter "super-outliers" across hidden dimensions, effectively crushing quantization noise that typically destroys logic in small models.

📈 Experimental Results

Knowledge & Reasoning (CommonSense)

We evaluated the model across ARC, Hellaswag (HS), and Winogrande (WG).

Configuration	Hadamard	ARC (%)	HS (%)	WG (%)	AVG (%)
Base (FP16)	-	0.370	0.510	0.565	0.482
AWQ-4bit-GS64	-	0.338	0.502	0.539	0.460
GPTQ-4bit-GS64	-	0.329	0.486	0.543	0.452
HadamardSpin (RTN-H2048)	2048	0.348	0.491	0.550	0.463

Logic & Coding (HumanEval)

Configuration	Hadamard	HumanEval (%)
Base (FP16)	-	0.310
RTN-4bit (Vanilla)	-	0.091
HadamardSpin (RTN-H2048)	2048	0.280

Note: Despite a slightly higher Perplexity ($9.604$), the Hadamard-rotated model preserves "structural knowledge nodes," leading to superior reasoning performance over lower-PPL counterparts.

---

Performance on Apple M1

Configuration	TPS (Tokens/s)	Peak Memory (GB)
FP16 (Baseline)	23.29	2.71
HadamardSpin (4-bit)	48.41	2.04

Measurements taken on a standard Apple M1 (8-core GPU, 8GB Unified Memory).

---

🛠️ How it Works: The "Incoherence" Strategy

Standard quantization (RTN) fails on Llama-3.2 because of extreme outliers (activations peaking at $\pm 400$). Mark-3 intervenes by:

1. Flattening the Weights: Compressing weight ranges from $[-0.6, 0.6]$ down to $[-0.1, 0.1]$.
2. Scattering Activations: Rotating the hidden space so no single dimension holds dominant energy, reducing peak activation amplitude by 10x.
3. Hadamard-2048: Larger block sizes allow for better scattering ($1/\sqrt{n}$), making simple RTN quantization behave like an advanced optimizer.

---

Usage (MLX)

import mlx.core as mx
from mlx.lm_utils import load

model, tokenizer = load("CogHyn/Llama-3.2-1B-Instruct-HadamardSpin-4bits")

# Generate
response = model.generate("Explain the Markov Property in one sentence.")
print(response)

📜 Citation & Credits

Developed by Nguyễn Công Huynh as part of the "Mark-3" Graduation Thesis.

@thesis{huynh2026mark3,
  title={Mark-3: Advanced LLM Quantization via Orthogonal Rotation},
  author={Nguyen Cong Huynh},
  year={2026},
  institution={University of Engineering and Tecnology - VNU}
}