TakiTakiTa/Qwen3.8-Flash-Low-RAM

EXPERIMENTAL STILL TESTING

Qwen3.8-Flash-Next on a 4 GB GPU

Runs the 180B-parameter Qwen3.8-Flash-Next (text part) with about 3.5 GB of VRAM. The routed experts (121B parameters) and the n-gram table (51B) live on the SSD. Only the dense core sits on the GPU: attention, DeltaNet, the shared experts, the routers and the LM head.

piece	where	size (3-bit experts, 4-bit core)
core: attention, DeltaNet, hyper-connections, shared experts, routers, lm_head	GPU	2.36 GiB
KV cache (12 attention layers) + DeltaNet state	GPU	27 MiB per 1k tokens + 111 MiB
routed experts, one 2.15 MB record per (layer, expert)	SSD + RAM page cache	53 GB
n-gram embedding (320M rows)	SSD	32 GB
token embedding	RAM	1.3 GB

Run on your PC (RTX 3050, Windows or Linux)

Run these commands from the folder that contains the flashnext/ package folder.

pip install torch --index-url https://download.pytorch.org/whl/cu128
pip install "transformers>=5.20.2" safetensors numpy huggingface_hub hf_xet
pip install triton-windows        # Windows: fused kernels (Linux: triton ships with torch). Optional.
python -m flashnext.plan --vram_gb 3.5                      # sizes for your budget
python -m flashnext.engine --store D:\store --vram_gb 3.5 --max_context 8192

Getting the store onto the PC. You can download a store you made on Colab and uploaded to a private Hugging Face repo:

hf download <you>/flashnext-store --local-dir D:\store

Or convert locally. This streams about 360 GB, and only the final ~89 GB stays on disk:

python -m flashnext.convert --repo Qwen/Qwen3.8-Flash-Next --out D:\store --expert_bits 3 --chunk_experts 4 --ngram_chunk_rows 100000

Speed limits

Each decoded token uses 10 routed experts in each of 48 layers. At 3 bits that is about 1 GB of expert weights per token that must reach the GPU. Three things bound the speed:

1. The SSD, for experts not in RAM. A PCIe 4 NVMe reads about 7 GB/s; a SATA SSD about 0.5 GB/s, which is too slow.
2. RAM. Free RAM caches experts automatically through the OS page cache. With 32 GB of RAM, roughly half of the reads come from RAM; with 64 GB, almost all of them.
3. PCIe to the GPU. The RTX 3050 runs at x8, about 12 GB/s, so about 80 ms per token.

Expect roughly 5–10 tokens/s on a 3050 with a fast NVMe drive and 32 GB+ of RAM. These options trade speed against quality and memory:

• --max_experts 8 decodes with the 8 highest-weight experts instead of 10. That is 20% fewer reads, but the output is approximate.
• --prefetch_k sets how many next-layer experts are guessed and read ahead. More guesses hide latency but waste bandwidth.
• --max_context sets the KV cache that is reserved. Each 1k tokens costs 27 MiB. Lower it to give the expert cache more room.