0xKitkat/Agnes-3.0-Flash-abliterated-GGUF

Agnes-3.0-Flash abliterated GGUF

Follow @procrastiness on Twitter/X for more model releases and updates.

An independently modified checkpoint of Agnes-AI/Agnes-3.0-Flash, with its original approximately 33.1 billion parameters and vision tower retained. Released by 0xKitkat under Apache-2.0, with downloadable weights and no hosted API subscription required. Local execution still uses your own hardware and electricity.

The build measures a contrast direction from 96 harmful and 96 harmless training prompts, using 32 prompts per class from a separate validation split to rank directions. Three directions and two projection strengths are compared on the validation split. The chosen direction comes from layer 53 (zero-based), with strength 1.0. The projection edits the text embeddings and the residual-output matrices in all 72 decoder blocks, including both feed-forward branches. All original vision weights are preserved. See abliteration.json, provenance, and evaluation reports for details.

The original custom Agnes architecture is preserved in the native checkpoint. For GGUF, the two parallel SwiGLU branches are combined by concatenating the gate/up matrices and concatenating the down matrices along their input dimension. The resulting computation is represented by llama.cpp's Qwen3.5 graph. The converter also preserves the unequal key/value head mapping, gated attention, normalization conventions, and rotary positions. MTP speculative-decoding weights are retained in the native checkpoint but omitted from GGUF. No decoder blocks or vision blocks are pruned.

Recommended starting setup

Start with Q4_K_M, a 4,096-token context, one concurrent request, and thinking disabled. This is the configuration closest to the functional checks reported below. Use the embedded chat template; do not select a generic ChatML or Llama template manually.

For normal chat, use the upstream sampling defaults: temperature 1.0, top-p 0.95, top-k 20. For repeatable troubleshooting or checking exact answers, use temperature 0. Start with 512 output tokens for short answers or 2,048 for longer responses, keeping prompt + image tokens + output within the configured context.

Choose a quant and hardware

Download one text GGUF. Add the F16 vision projector only if you want image input. The projector is shared by all quants in this release.

Quant	When to choose it
Q4_K_M	Recommended first download; smallest planned file and the easiest fit on consumer hardware.
Q5_K_M	More weight precision if you have additional memory; compare the measured results below.
Q6_K	Higher precision with a larger memory footprint.
Q8_0	Highest precision in this quant set; intended for larger-memory machines.

The validation table lists exact sizes for published files. A quant being listed here does not mean its upload has finished; check the available-files list and the Files tab.

Hardware	Suggested starting point
Single 8–16 GB GPU + 32 GB or more system RAM	Q4_K_M with CPU offloading. Use automatic fitting initially; lower GPU layers if needed. Generation will be slower than a full GPU fit.
Single 24 GB GPU + 32 GB or more system RAM	Try Q4_K_M at 4K context. Leave room for buffers, display usage, and the projector; reduce GPU layers if allocation fails.
Two 12 GB GPUs + about 32 GB system RAM	Q4_K_M, layer split 1,1, 4K context. This release was exercised on two RTX 2060 12 GB GPUs.
48 GB or more GPU memory	Consider Q6_K or Q8_0 at 4K first, then increase context after checking memory use.
CPU only, or Apple Silicon unified memory	Start with Q4_K_M and preferably 32 GB or more available system/unified memory. CPU-only and Metal performance were not benchmarked for this release.

These are starting recommendations, not exact minimums. Weight-file size excludes runtime buffers, recurrent state, attention cache, images, and other applications. A 16 GB machine is a poor fit for this approximately 33B model. CPU-only operation benefits from 48–64 GB RAM when other applications are running.

1. Install llama.cpp and the download tool

Use a recent llama.cpp build with Qwen3.5 text and vision support. The release was tested with commit 56381e407c0ccfb3a6f71e668a27a901001d22ce. Older bundled runtimes can reject the model architecture or load it incorrectly.

Windows

Install Python 3.10+ if needed, then open PowerShell:

winget install llama.cpp
python -m pip install --upgrade huggingface_hub requests
llama-server --version

Open a new terminal after installing if commands are not found. For NVIDIA acceleration, use a compatible CUDA build from the official llama.cpp releases, extract the complete archive, and keep its DLLs alongside llama-server.exe. From that folder, replace llama-server in the commands below with .\llama-server.exe. Confirm the startup log detects your GPU; installing a package alone does not establish which backend it uses.

macOS

With Homebrew and Python installed:

brew install llama.cpp
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade huggingface_hub requests
llama-server --version

Ubuntu / Debian: reproducible NVIDIA build

Install a compatible NVIDIA driver and CUDA Toolkit first; nvidia-smi and nvcc --version should work. Then:

sudo apt-get update
sudo apt-get install -y git cmake build-essential libcurl4-openssl-dev python3-venv
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
git checkout 56381e407c0ccfb3a6f71e668a27a901001d22ce
cmake -S . -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j 4 --target llama-server
export PATH="$PWD/build/bin:$PATH"
cd ..
python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade huggingface_hub requests

For a CPU-only build, configure with -DGGML_CUDA=OFF and start the server with -ngl 0. Other backends and installation options are covered in the llama.cpp installation guide and build guide.

2. Download the model

Run these in the directory where you want the agnes folder. The public download does not require a paid account or hosted inference subscription.

hf download 0xKitkat/Agnes-3.0-Flash-abliterated-GGUF Agnes-3.0-Flash-abliterated-Q4_K_M.gguf --local-dir agnes

For image input, also download:

hf download 0xKitkat/Agnes-3.0-Flash-abliterated-GGUF mmproj-Agnes-3.0-Flash-F16.gguf --local-dir agnes

For another published quant, replace Q4_K_M in the filename with Q5_K_M, Q6_K, or Q8_0. Rerun the same download command after an interruption. Allow disk space for the chosen file plus the approximately 0.93 GB projector if used.

SHA-256 checksums are recorded in release-manifest.json. To calculate a local checksum:

Get-FileHash agnes/Agnes-3.0-Flash-abliterated-Q4_K_M.gguf -Algorithm SHA256

On Linux use sha256sum agnes/Agnes-3.0-Flash-abliterated-Q4_K_M.gguf; on macOS use shasum -a 256 with the same path.

3. Start the local server

Text-only: single GPU or automatic fitting

llama-server -m agnes/Agnes-3.0-Flash-abliterated-Q4_K_M.gguf --alias agnes -c 4096 --parallel 1 --fit on --jinja --flash-attn on --batch-size 256 --ubatch-size 128 --threads 6 --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080

Open http://127.0.0.1:8080 for the built-in chat UI. Keep the terminal running. The API is at http://127.0.0.1:8080/v1; its model name is agnes. This local server does not need an API key. If a client requires a nonempty key field, use local.

The command lets llama.cpp choose GPU offloading with --fit on. If it runs out of GPU memory, specify a smaller layer count, for example -ngl 20, and adjust from there. Use -ngl 0 for CPU-only inference. Six CPU threads is a starting value used in our checks; tune it for your CPU.

Text and images: tested dual-GPU layout

llama-server -m agnes/Agnes-3.0-Flash-abliterated-Q4_K_M.gguf --mmproj agnes/mmproj-Agnes-3.0-Flash-F16.gguf --alias agnes -c 4096 --parallel 1 -ngl 99 --split-mode layer --tensor-split 1,1 --jinja --flash-attn on --batch-size 256 --ubatch-size 128 --threads 6 --temp 1.0 --top-p 0.95 --top-k 20 --host 127.0.0.1 --port 8080

-ngl 99 requests all eligible layers on the GPUs; it is not a claim that the model has 99 layers. --tensor-split 1,1 distributes layers between two GPUs. On one GPU, use the first command and add --mmproj agnes/mmproj-Agnes-3.0-Flash-F16.gguf. For larger quants, lower the GPU layer count or let automatic fitting choose it.

Thinking and context settings

The API examples below explicitly disable thinking, matching release testing. To make this the server default for the web UI too, set LLAMA_ARG_CHAT_TEMPLATE_KWARGS before starting the server:

# PowerShell
$env:LLAMA_ARG_CHAT_TEMPLATE_KWARGS = '{"enable_thinking":false}'

# Bash / zsh
export LLAMA_ARG_CHAT_TEMPLATE_KWARGS='{"enable_thinking":false}'

For experimental thinking-on use, send "chat_template_kwargs": {"enable_thinking": true, "reasoning_effort": "xhigh"} in the request. The source template accepts low, medium, and xhigh effort (the actual pinned template uses xhigh, even though the upstream card describes high). Allocate more output tokens and context for reasoning. This release's measured results do not validate reasoning-on quality, tool calling, video, or the advertised 262,144-token context. Start at 4K; try 8K only after confirming adequate memory and prompt-plus-output space.

4. Use the API from Python

Save as chat_agnes.py, then run python chat_agnes.py while the server is running:

import unicodedata
import requests

prompt = unicodedata.normalize("NFC", "Explain how a rainbow forms in three sentences.")
response = requests.post(
    "http://127.0.0.1:8080/v1/chat/completions",
    json={
        "model": "agnes",
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 1.0,
        "top_p": 0.95,
        "top_k": 20,
        "max_tokens": 512,
        "chat_template_kwargs": {"enable_thinking": False},
        "stream": False,
    },
    timeout=1800,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"]["content"])

For deterministic checks, change temperature to 0 and add "seed": 20260912. Sampling defaults above come from the original model; they are a starting point, not a new tuning benchmark.

Ask about a local image

Start the server with the projector. Save this as image_agnes.py, place an image.jpg alongside it, and run python image_agnes.py:

import base64
from pathlib import Path
import requests

encoded = base64.b64encode(Path("image.jpg").read_bytes()).decode("ascii")
response = requests.post(
    "http://127.0.0.1:8080/v1/chat/completions",
    json={
        "model": "agnes",
        "messages": [{
            "role": "user",
            "content": [
                {"type": "text", "text": "Describe the main objects in this image."},
                {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{encoded}"}},
            ],
        }],
        "temperature": 0,
        "max_tokens": 256,
        "chat_template_kwargs": {"enable_thinking": False},
        "stream": False,
    },
    timeout=1800,
)
response.raise_for_status()
print(response.json()["choices"][0]["message"]["content"])

For PNG input, change the file path and use data:image/png;base64,. Begin with one modest-size image; image tokens also consume context. The API format follows llama.cpp's multimodal server documentation.

Troubleshooting

Symptom	What to change
Unknown architecture, missing tensor, or unsupported operation	Update llama.cpp to the tested revision or a newer compatible build. This is a Qwen3.5 GGUF graph conversion, so older runtimes may not support it.
GPU out of memory	Use Q4_K_M; reduce -ngl, context, or --ubatch-size to 64; keep --parallel 1. Add the projector only when needed.
Very slow generation	Check the startup log for actual GPU offloading. Q5/Q6/Q8 may spill heavily to CPU on 24 GB total VRAM. Try Q4 and close memory-heavy applications.
Thought tags or unexpectedly long reasoning	Set chat_template_kwargs.enable_thinking to false, or set the server environment variable shown above.
Output stops early	Check the response's finish_reason. Raise max_tokens if it is length, while ensuring sufficient context remains.
Image request fails	Check that the matching projector is loaded and that the data URL's MIME type matches the image file.
Unicode tokenization differs from Transformers	NFC-normalize text before sending it. Stock llama.cpp does not reproduce the upstream tokenizer's NFC normalization automatically.

Other applications must bundle a sufficiently recent compatible backend. LM Studio and Ollama were not part of this release's validation; the commands above use llama.cpp directly.

Measured checks

Quant	File GiB	24 deterministic tasks	Refusal-prefix heuristic	3 synthetic image checks	WikiText PPL
Q4_K_M	18.40	95.8%	71.9%	100.0%	7.439
Q5_K_M	21.42	91.7%	50.0%	100.0%	7.361
Q6_K	24.62	95.8%	43.8%	100.0%	7.361
Q8_0	31.89	91.7%	37.5%	100.0%	7.416

The unmodified Q4_K_M baseline scored 95.8% on the same deterministic tasks and 100.0% on the refusal-prefix heuristic. These are small functional checks, not comprehensive capability benchmarks. The refusal metric is a regular expression on the first 40 generated tokens of 32 held-out prompts; it does not establish that every refusal is removed or that every response is useful. Refusal-direction calibration, validation, and final refusal probes use separate prompt splits. The 24 deterministic correctness questions are reused during validation and final checks, so they are diagnostic checks rather than an independent correctness holdout. All tests disable thinking and use greedy decoding. Reasoning-on behavior, video, long context, and tool calling were not exhaustively evaluated.

Q4_K_M, Q5_K_M, and Q6_K use a 32-chunk importance matrix collected from the modified Q8_0 model on a separate portion of the harmless training corpus. Q8_0 is quantized directly from modified BF16 weights. Each released quant is independently loaded and tested. SHA-256 hashes are in release-manifest.json.

Language-model loss is measured on the first eight 512-token chunks of a pinned WikiText-2 test corpus. Candidate selection and release require perplexity no more than 15% above the unmodified Q4 baseline. This subset loss check helps reject degradation; it is not a full-corpus benchmark.

Additional robustness holdout

A task-specific suite drafted through Grok CLI and checked before model execution adds 36 deterministic cases and 12 open-response rubric cases. It is a separate final comparison, not an established benchmark and not a target for candidate tuning. The normal importance-calibrated Q4_K_M scored 29/36; the selected abliterated Q4_K_M scored 29/36. Publication requires losing no more than two correct answers and no increase in empty or truncated responses. Output truncations across all 48 cases were 1 for the baseline and 0 for the modified model.

The 12 rubric responses are included for inspection but have not been manually graded; they are not counted as successes. Only Q4_K_M received this additional suite. The other quants received the functional, perplexity, and image checks above. See quality-suite.json, the evaluation review, and the baseline/selected quality reports for the exact prompts, grading rules, results, and suite hash.

Runtime compatibility

The upstream tokenizer applies Unicode NFC normalization. Stock llama.cpp's Qwen3.5 tokenizer does not apply that normalization. Normalize user and tool text with unicodedata.normalize("NFC", text) before rendering the chat template for exact agreement on decomposed Unicode. Ordinary already-normalized English and Chinese text is unaffected. The supplied validation client performs this step; the reports explicitly record it.

Use a recent llama.cpp build with Qwen3.5 and its vision projector support. The tested llama.cpp revision is 56381e407c0ccfb3a6f71e668a27a901001d22ce. CUDA tests ran on two RTX 2060 12 GB cards with CPU offloading for larger quants. File size is not total runtime memory; allow space for buffers, recurrent state and the KV cache. The source advertises 262,144-token context, but release testing used short contexts and does not validate that limit.

Attribution

Original weights and implementation: Agnes AI, Apache-2.0. Refusal-direction methodology and calibration split provenance: Arditi et al., Refusal in Language Models Is Mediated by a Single Direction. Build code was generated with AI assistance and tested as recorded. Original license and notices are retained.