⚡ Qwen3.8-Flash-Coder-44GB-Selective-INT8 (160 Experts Hardware-Aligned Subnet)

Qwen3.8-Flash-Coder-44GB-Selective-INT8 is a high-performance, selective-quantized Mixture-of-Experts (MoE) coding model. Sliced down from the monolithic Qwen/Qwen3.8-Flash-Next (335GB) and quantized from the Qwen3.8-Flash-Coder-85GB-BF16 parent checkpoint, this model reduces disk and VRAM footprint to exactly 44.29 GB (a 44.2% VRAM reduction and 86.8% reduction from base), enabling full zero-offload deployment on only 2x 32GB GPUs (e.g. 2x NVIDIA RTX 5000 Ada, 2x RTX 4090/3090, or 1x A100/H100 80GB).

---

🔬 Selective Quantization Architecture

Traditional MoE post-training quantization often quantizes all layers uniformly, which severely degrades the Router Gate and causes Routing Collapse (routing tokens to sub-optimal experts).

This checkpoint introduces Selective MoE Quantization:

1. Critical High-Precision Modules (Kept in 100% Native BF16):
   • Router Gates: Retain 100% floating-point routing fidelity across all 48 layers.
   • Multi-Head Self-Attention & Linear Attention: q_proj, k_proj, v_proj, o_proj.
   • Shared Expert, RMSNorms, Embeddings & LM Head: Zero quantization loss in embedding projections.
2. High-Capacity Sparse Experts (Quantized to Symmetric Per-Channel INT8):
   • 160 MoE Experts across 48 layers (gate_up_proj, down_proj).
   • Symmetrically quantized per-channel with dynamic scaling vectors (gate_up_proj_scale, down_proj_scale).

---

📊 Technical Specifications

Parameter	Original Monolith (Qwen3.8-Flash-Next)	BF16 Parent Checkpoint	Selective MoE INT8 (This Checkpoint)
Disk / VRAM Size	~335 GB (131 Shards)	85.24 GB (2 Shards)	44.29 GB (2 Shards: 25.3GB + 19.0GB)
Numerical Format	Bfloat16	Bfloat16	Selective INT8 (Router BF16 + Experts INT8)
Layers / Total Experts	48 Layers / 512 Experts	48 Layers / 160 Experts	48 Layers / 160 Experts
Active Experts / Token	10 Experts	8–10 Experts	8 Active Experts
Required Hardware	8x H100 (80GB) Cluster	3x RTX 5000 Ada (32GB)	2x RTX 5000 Ada (32GB) or 2x RTX 4090 (24GB)
Per-GPU Memory Usage	>45 GB / GPU (8x GPUs)	~27.3 GB / GPU (3x GPUs)	~22.1 GB / GPU (2x GPUs)
Toolkit	—	moe-slice	moe-slice

---

🏆 Empirical Sandbox Benchmark Results (100 Real-World Tasks)

The model was rigorously tested across an isolated execution-based sandbox benchmark covering 100 challenging tasks in systems engineering, algorithms, and autonomous coding agents:

Language / Domain	Tested Suite	Pass@1 Accuracy	Verified Engineering Competencies
⚡ C++ (Modern C++20)	10 Tasks	100.0% (10/10)	Concurrency (ThreadSafeQueue, AtomicCounter), Smart Pointers, C++20 Concepts, Templates
🦀 Rust (Systems)	10 Tasks	100.0% (10/10)	Tokio Async MPSC, Safe Mutex, Iterators, Borrow Checker, Pattern Matching, Traits
🐹 Go (Golang Systems)	5 Tasks	80.0% (4/5)	Worker Pools, Channel Synchronization, Struct JSON Marshal, Binary Search Slice
🌐 TypeScript (Fullstack)	5 Tasks	80.0% (4/5)	Generic Debounce, Promise Retry, Generic Deep Clone, Zod-like Schema Validator
🤖 Coding Agent	20 Tasks	80.0% (16/20)	Automated Debugging (100%), Code Refactoring & Diff Patches (100%), Fill-in-the-Middle (FIM)
🐍 Python Algorithms	50 Tasks	78.0% (39/50)	Dynamic Programming, Tree Structures (BST, LCA, Trie), Binary Search, Sorting
📊 TOTAL BENCHMARK	100 Tasks	83.0% (83/100)	Real Multi-Language Isolated Sandbox Code Execution

Compared to the original un-tuned base model (67.0%), this 44.3GB Selective INT8 checkpoint achieves a +16.0% absolute Pass@1 increase while slashing memory consumption by nearly half.

---

🚀 Quickstart & Inference

To achieve high-throughput inference with on-demand vectorized dequantization across 2 GPUs:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "0,1" # 2x GPUs

import torch
from transformers import AutoConfig, AutoTokenizer, AutoModelForCausalLM

model_id = "Jab1718/qwen3.8-flash-coder-44gb-selective-int8"

config = AutoConfig.from_pretrained(model_id, trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained(model_id, trust_remote_code=True)

# Device mapping across 2 GPUs (Embeddings + Layers 0..23 on GPU 0; Layers 24..47 + Head on GPU 1)
device_map = {
    "model.embed_tokens": "cuda:0",
    "model.rotary_emb": "cuda:0",
    "model.hyper_connection_mixer": "cuda:1",
    "model.norm": "cuda:1",
    "lm_head": "cuda:1"
}
for i in range(24):
    device_map[f"model.layers.{i}"] = "cuda:0"
for i in range(24, 48):
    device_map[f"model.layers.{i}"] = "cuda:1"

# Load model weights
model = AutoModelForCausalLM.from_pretrained(
    model_id,
    device_map=device_map,
    trust_remote_code=True
)

prompt = "Write a lock-free thread-safe queue in C++20 using atomic operations."
messages = [
    {"role": "system", "content": "You are an expert modern C++20 systems engineer."},
    {"role": "user", "content": prompt}
]
formatted = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tokenizer(formatted, return_tensors="pt").to("cuda:0")

with torch.inference_mode():
    outputs = model.generate(**inputs, max_new_tokens=512, do_sample=False)

print(tokenizer.decode(outputs[0][inputs.input_ids.shape[1]:], skip_special_tokens=True))

---

📜 Citation & Acknowledgements

@software{moe_slices_qwen38_int8,
  author = {Thai Nguyen},
  title = {Qwen3.8-Flash-Coder-44GB-Selective-INT8: 44.3GB Hardware-Aligned Coding Subnet},
  url = {https://github.com/Jab1718/Moe-slices},
  year = {2026}
}