Instructions to use bloomer010/Ling-3.0-flash-VL-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use bloomer010/Ling-3.0-flash-VL-GGUF with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M
# Run inference directly in the terminal:
llama cli -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Use Docker

docker model run hf.co/bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

LM Studio
Jan

vLLM

How to use bloomer010/Ling-3.0-flash-VL-GGUF with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bloomer010/Ling-3.0-flash-VL-GGUF"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bloomer010/Ling-3.0-flash-VL-GGUF",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Ollama
How to use bloomer010/Ling-3.0-flash-VL-GGUF with Ollama:
```
ollama run hf.co/bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M
```
Unsloth Desktop

How to use bloomer010/Ling-3.0-flash-VL-GGUF with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use bloomer010/Ling-3.0-flash-VL-GGUF with Docker Model Runner:
```
docker model run hf.co/bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M
```

Lemonade

How to use bloomer010/Ling-3.0-flash-VL-GGUF with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Run and chat with the model

lemonade run user.Ling-3.0-flash-VL-GGUF-Q4_K_M

List all available models

lemonade list

Hermes Agent

How to use bloomer010/Ling-3.0-flash-VL-GGUF with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use bloomer010/Ling-3.0-flash-VL-GGUF with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "bloomer010/Ling-3.0-flash-VL-GGUF:Q4_K_M" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Ling-3.0-flash-VL GGUF

GGUF conversions of inclusionAI/Ling-3.0-flash-VL

Built upon Ling-3.0-flash, it brings visual information into the complete process of understanding, reasoning, acting, and verification—advancing beyond image and video perception to solving real-world tasks through vision. With 124B total parameters, only 5.5B activated parameters per token, support for image and video inputs, and a context window of up to 256K tokens, Ling-3.0-flash-VL delivers powerful multimodal reasoning and agentic capabilities with exceptional efficiency.

Every text quant requires the bundled mmproj-model-f16.gguf for vision input. Text-only chat works without it.

🦙🚨 llama.cpp 🦙🚨

This model does not load on stock llama.cpp yet. It currently requires an unmerged PR:

VL architecture (bailingmoe3vl) + Ling VL projector: #29151

The following PRs are also strongly recommended for any sort of agentic use:

Dedicated Ling parser: #28682 (✅ Merged as of 9/19)
Invalid UTF-8 handling at the token boundary: #28724

A branch with all three of these PRs applied: ling3-vl can be found at aetherbird/llama.cpp.

To run with llama-server:

llama-server \
  -m Ling-3.0-flash-VL-Q4_K_M.gguf \
  --mmproj mmproj-model-f16.gguf \
  --jinja

Quant Sizing

Generally...
Larger files = More precision.
Smaller files = More compression = More slop and misbehavin'.

Weights and context share your memory, so be sure to leave headroom.

your memory	file	size
256 GB+	`BF16`	249 GB
136 GB+	`Q8_0`	132 GB
128 GB	`UD-Q6_K_XL`	103 GB
104 GB+	`Q6_K`	102 GB
90 GB+	`Q5_K_M`	88.3 GB
76 GB+	`Q4_K_M`	75.3 GB
72 GB+	`Q4_K_S`	70.7 GB
60 GB+	`Q3_K_M`	59.3 GB

With less VRAM than the file size, keep the experts on CPU and the rest on GPU, e.g.:

llama-server \
  -m Ling-3.0-flash-VL-Q4_K_M.gguf \
  --mmproj mmproj-model-f16.gguf \
  -ngl 99 -ot "ffn_.*_exps\.weight=CPU" -c 32768 \
  --jinja

Usage

Recommended sampling from the source model card: temperature 0.6, top_p 0.95, top_k 20. Thinking mode is on by default; disable per request with "chat_template_kwargs": {"enable_thinking": false}.

Images

./build/bin/llama-server \
  -m Ling-3.0-flash-VL-Q4_K_M.gguf \
  --mmproj mmproj-model-f16.gguf \
  -c 131072 \
  -ngl auto \
  --flash-attn auto \
  --temp 0.6 --top-p 0.95 --top-k 20 \
  --jinja

Then attach an image in the web UI, or via the API:

curl http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{
      "role": "user",
      "content": [
        {"type": "text", "text": "Describe this image."},
        {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,..."}}
      ]
    }]
  }'

Video

Video input uses the same chat API with video_url content parts. Frames are sampled and encoded by the same vision tower.

Unlike the text-only Ling-3.0-flash GGUFs, these files contain no MTP/NextN block: the VL release does not ship one. Speculative drafting via --spec-type draft-mtp is not available for VL.

Speculative Decoding (DSpark)

The Ling-3.0-flash DSpark draft heads are compatible with the VL model and included here. Aceptance on VL is at least as good as on the text-only model the draft was trained for.

Measured with llama-server (VL Q6_K target, --spec-type draft-dspark --spec-draft-n-max 8, 32K context, 24 requests):

config	decode speed
Q6_K	26.8 tok/s
Q6_K + DSpark Q4_K_M	43.6 tok/s (1.63x)

Draft acceptance on VL Q6_K: 0.32 (Q4_K_M draft), 0.30 (Q2_K draft). The same Q4_K_M draft measures 0.26 against text-only Ling-3.0-flash (text only).

llama-server \
  -m Ling-3.0-flash-VL-Q6_K.gguf \
  -md Ling-3.0-flash-DSpark-Q4_K_M.gguf \
  --spec-type draft-dspark --spec-draft-n-max 8 \
  -ngl 99 -ngld 99 -ctkd q4_0 -ctvd q4_0 \
  --mmproj mmproj-model-f16.gguf \
  --jinja

The DSpark draft's attention does not use flash attention, so its compute buffer grows linearly with context length. It also carries its own KV cache; quantize it with -ctkd q4_0 -ctvd q4_0 to reduce the footprint.

Additional MoE Information

MoE placement can be adjusted for available VRAM with -ncmoe N.

Supports up to 128K context.

Conversion and Quantization

Taken directly from the released inclusionAI/Ling-3.0-flash-VL BF16 safetensors.

Conversion-specific tensor transformations match the text-only Ling-3.0-flash conversions:

A_log stored as exp(A_log)
MLA kv_b_proj split into separate K and V tensors, with the K tensor transposed
KDA convolution weights reshaped for llama.cpp
Per-expert tensors stacked into GGUF expert tensors
KDA and MLA g_proj tensors mapped separately

Vision tower and projector tensors live in the separate mmproj GGUF: Conv3D patch embedding, learned position embeddings, 27 attention blocks, a norm-only merger, and the two-layer projector.

Norms, routing tensors, expert routing bias, KDA state scalars, dt_bias, and convolution weights remain F32.

Notes

The text GGUF contains 42 blocks:

35 KDA layers
7 gated MLA layers at zero-based indices 5, 11, 17, 23, 29, 35, and 41

(No MTP/NextN block, unlike the text-only flash GGUFs.)

The first two layers use dense FFNs. The remaining layers use 512 routed experts with top-8 selection plus one shared expert. Routing uses sigmoid scoring, expert bias, eight expert groups, and four selected groups.

Position encoding is M-RoPE with sections [8, 12, 12], shared between text and vision positions.

Validation Completed

BF16 architecture load and tensor round-trip (test-llama-archs, MoE fixture)
mmproj GGUF round-trip: 334 tensors, ling3vl_merger projector
End-to-end image and video inference on llama-server (Q4_K_M + mmproj)

Build

# until PR 3 merges:
git clone --branch ling3-vl https://github.com/aetherbird/llama.cpp.git

cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j --target llama-cli llama-server

Downloads last month: 174

GGUF

Model size

124B params

Architecture

bailingmoe3vl

Hardware compatibility

2-bit

3-bit

4-bit

5-bit

6-bit

8-bit

16-bit

Model tree for bloomer010/Ling-3.0-flash-VL-GGUF

Base model

inclusionAI/Ling-3.0-flash-VL

Quantized

(5)

this model