Instructions to use AtomicChat/Ling-3.0-flash-NVFP4 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use AtomicChat/Ling-3.0-flash-NVFP4 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="AtomicChat/Ling-3.0-flash-NVFP4", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("AtomicChat/Ling-3.0-flash-NVFP4", trust_remote_code=True, device_map="auto")

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use AtomicChat/Ling-3.0-flash-NVFP4 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "AtomicChat/Ling-3.0-flash-NVFP4"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AtomicChat/Ling-3.0-flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/AtomicChat/Ling-3.0-flash-NVFP4

SGLang

How to use AtomicChat/Ling-3.0-flash-NVFP4 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "AtomicChat/Ling-3.0-flash-NVFP4" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AtomicChat/Ling-3.0-flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "AtomicChat/Ling-3.0-flash-NVFP4" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "AtomicChat/Ling-3.0-flash-NVFP4",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use AtomicChat/Ling-3.0-flash-NVFP4 with Docker Model Runner:
```
docker model run hf.co/AtomicChat/Ling-3.0-flash-NVFP4
```

Amazing KLD how?

by 0xSero - opened about 12 hours ago

Discussion

0xSero

about 12 hours ago

I saw you have a very low kld, i'm curious if there's anything you can share about that

worthant

Atomic Chat org about 8 hours ago

Hi @0xSero !
Sure, three things.

Ling 3.0 flash is 124.4B of running parameters, where 120.8 B is routed experts. That's like 97.1%. (127B is 124.4+3.07 for MTP head). Else is 3.6B - attention, KDA gates, norms, shared expert, embeddings, router. So it's ONLY the experts that go below 8 bits for my quants, which is kinda obvious, but anyway needs to be stated. I didn't touch the router and left it at F32. However, llama-quantize already does that. My win comes from everything else in that 3.6B group - i practically don't touch it by having kda gate projections, attention, shared expert in Q8_0 and the output in F16. basic llama-quantize utility quantizes this along with the experts. This costs me about 4GB on a 74GB file, and that's where the whole 30-40% gap between _STOCK postfix models (what llama-quantize cli picked by itself) and _FLAT postfix ones is.
IF you just tell llama-quantize to not touch those 3.6B, it will produce _FLAT models. That's what it picks with my bit budget but without differentiation - just one type for all experts. Our dynamic quants are around 4-5% better using a bit of fine-graining. Basically, I give more bits inside each expert for down_proj tensor than for gate/up (the more sensitive half of the SwiGLU), and the edge MoE blocks get more than the middle ones. One caveat is the interest of full disclosure - AD-Q4_K_S is 74.2GB against Q4_K_FLAT at 72.3GB, so part of that 4% is bought with bits, rather than placement, but we gained accuracy overall.
Here comes the meticulous part. Just not doing any shortcuts:

a) a bit-exact BF16 conversion as a baseline. I verified it tensor by tensor against the original safetensors file. 877 out of 917 are byte-identical, 40 routers are stored as F32, which is a lossless widening (max abs dif 0.0). Any shortcuts here will affect the whole process, so even if your baseline is F16, rather than BF16, part of what you measure as quantization error is the baseline's own bf16->fp16 rounding on outilers. I'm tryna to be extra careful here.

b) imatrix on bf16 baseline, not q8/q4. Over 522 chunks of 4096 tokens, which is like 2 million tokens. Takes a lot of time to generate, but worth it. Gain is that I used to do what most people do - just a small cheap imatrix on wikipedia-ish text, then you evaluate it also on wikitext-2 and call it a day, but that's not how we do things here and this time i prepared a corpus purposely built for what this model is actually used for (by our team and users) - 1087 documents, graphics/three.js/WebGL 31.8%, long-context 14.4%, agentic 13.8%, code 12.8%, general 9%, vocabulary sweep 8.3%, reasoning 5.8%, structured output 4.3%.

c) Measurements are llama-perplexity --kl-divergence against that baseline, every run on the same box. The harness reads exactly 0.00000 when the baseline is scored against itself. I publish mean KL, 99% KL and top-1 agreement for every rung, because the mean hides the tail and the tail is where a quant actually breaks. If the results don't satisfy me - I change my bit allocation and produce a new, possibly better quant.

d) Also important to remember, that the absolute floor is higher for this model class than for a dense one. Those numbers are NOT particularly impressive - the Q8_0 still sits at 0.0196 mean kl, because with top 8 out of 512 routing a small logit perturbation flips expert selection discretely. This model just has more room to gain from where you put the bits. For any dense model Q8 would be an order of magnitude lower than the numbers i have here.

Note: Now let me have some honesty. My eval text is held out from the calibration corpus - i checked and verified it before using, but the idea is it's the same domain mix, so it IS kinda in-distribution. Numbers measured on wikitext or another mix WILL be higher. For comparability and clarity I'm publishing everything: corpuses, datasetes, calibration logs and evaluations, showing the quantization process - so if you find something wrong, incorrect or just something that can be improved - feel free to point that out, we're all here to learn and improve!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment