aj9o9/Ling-3.0-flash-GGUF

Ling-3.0-flash - GGUF (Q4_K_M)

GGUF quantization of inclusionAI/Ling-3.0-flash, a 124B parameter Mixture-of-Experts model with 5.1B active parameters per token.

A note from me

This is my first quantization, and my first time patching llama.cpp. I did it to learn, and I learned a lot - including that I got part of it wrong (see the warning below). I am sharing it because a working GGUF of this model did not exist when I started, not because I think it is authoritative.

Please report anything you find. Bad output, crashes, wrong metadata, better flags for a particular card - open a discussion on this repo or reach me at @ItsmeAjayKV. Suggestions are as welcome as bug reports.

On the llama.cpp side: I am not planning to submit this fork upstream. The architecture support was written with heavy AI assistance and the model is very new, so PR #26608 is the right place for upstream support to come from - it is independent work by someone else and, at least on the KDA gate, more correct than mine. My branch exists so these files have something to run on, and so I could understand how the pieces fit together.

This will not load in upstream llama.cpp. Ling 3.0 uses a hybrid KDA + MLA attention architecture that upstream does not yet support. These files declare the architecture bailing-hybrid, which currently exists only in the fork linked below. Loading them with a standard llama.cpp build fails with unknown model architecture.

Known correctness issue - fix in progress. The KDA forget gate in this build uses the wrong branch of the reference implementation. Ling 3.0 sets kda_safe_gate: true, which selects lower_bound * sigmoid(exp(A_log) * (g + bias)), but this build applies -exp(A_log) * softplus(g + bias) followed by a hard clamp. That affects 35 of the 42 layers.

Output is fluent and coherent, so the problem is not obvious from casual use, but quality is likely degraded relative to the reference implementation. Treat this as a preview. PR #26608 implements the gate correctly.

Files

File	Size	Quant	Bits/weight
Ling-3.0-flash-Q4_K_M-00001-of-00002.gguf	41.5 GB	Q4_K_M	4.84
Ling-3.0-flash-Q4_K_M-00002-of-00002.gguf	37.4 GB	Q4_K_M	4.84

Total 75.3 GB. Download both shards into the same directory and point -m at the first one - llama.cpp finds the second automatically. Do not rename them.

Quantized from bf16 without an importance matrix (imatrix).

Usage

Build the fork:

git clone -b ling-3.0-bailing-hybrid https://github.com/Ajay9o9/llama.cpp
cd llama.cpp && cmake -B build -DGGML_CUDA=ON && cmake --build build -j --target llama-completion

Run it. The model is 70 GiB, so unless you have that much VRAM, offload the experts to CPU with --n-cpu-moe - only 8 of 512 experts are read per token, so this costs less than it sounds:

./build/bin/llama-completion \
  -m Ling-3.0-flash-Q4_K_M-00001-of-00002.gguf \
  -p "Write a haiku about linear attention." \
  -ngl 99 --n-cpu-moe 42 -n 100

Lower the --n-cpu-moe number to keep more experts on the GPU if you have VRAM spare. On a 96 GB card with everything resident, this quant runs at roughly 113 tok/s.

Hardware notes

I made this quant with consumer cards in mind - specifically a 3060 (12 GB) and a 3090 (24 GB), paired with 64 GB of system RAM. Because only 8 of 512 experts run per token, pushing the expert weights to CPU costs far less throughput than the size suggests.

Setup	Expect
3090 (24 GB) + 64 GB RAM	Workable. Use --n-cpu-moe 42, lower it if VRAM allows.
3060 (12 GB) + 64 GB RAM	Tight but runs. Keep --n-cpu-moe 42 and expect a slow prompt phase.
64 GB RAM, no GPU	Possible via mmap, but slow.

At 70 GiB the file is larger than 64 GB of RAM, so some of it is served from disk through mmap regardless. An SSD makes a real difference here; on a spinning disk it will crawl. Keep the model on your fastest drive.

If you have less RAM than this, a smaller quant (IQ3 or below) would suit you better than this one. I have not made those yet.

About the model

• 42 layers: 35 KDA (linear attention, recurrent state) interleaved 5:1 with 7 gated MLA layers
• 512 routed experts + 1 shared, 8 experts active per token, group-limited noaux_tc routing
• 256K context. Only the 7 MLA layers keep a KV cache, and they store a 576-value compressed latent per token rather than full per-head keys and values, so the cache stays small at long context.

See the original model card for benchmarks and training details. Those numbers were measured on the full bf16 model with the reference implementation and do not describe this quantization.

Links

• Original model: inclusionAI/Ling-3.0-flash
• Fork required to run this: Ajay9o9/llama.cpp @ ling-3.0-bailing-hybrid
• Upstream request: ggml-org/llama.cpp#26590
• Alternative implementation: ggml-org/llama.cpp#26608
• Reference implementation: inclusionAI/vllm-ling-v3

License

MIT, inherited from the original model.