0xKitkat/Agnes-3.0-Flash-abliterated

Agnes-3.0-Flash abliterated

Follow @procrastiness on Twitter/X for more model releases and updates.

An independently modified checkpoint of Agnes-AI/Agnes-3.0-Flash, with its original approximately 33.1 billion parameters and vision tower retained. Released by 0xKitkat under Apache-2.0, with downloadable weights and no hosted API subscription required. Local execution still uses your own hardware and electricity.

The build measures a contrast direction from 96 harmful and 96 harmless training prompts, using 32 prompts per class from a separate validation split to rank directions. Three directions and two projection strengths are compared on the validation split. The chosen direction comes from layer 53 (zero-based), with strength 1.0. The projection edits the text embeddings and the residual-output matrices in all 72 decoder blocks, including both feed-forward branches. All original vision weights are preserved. See abliteration.json, provenance, and evaluation reports for details.

The original custom Agnes architecture is preserved in the native checkpoint. For GGUF, the two parallel SwiGLU branches are combined by concatenating the gate/up matrices and concatenating the down matrices along their input dimension. The resulting computation is represented by llama.cpp's Qwen3.5 graph. The converter also preserves the unequal key/value head mapping, gated attention, normalization conventions, and rotary positions. MTP speculative-decoding weights are retained in the native checkpoint but omitted from GGUF. No decoder blocks or vision blocks are pruned.

Measured checks

Quant	File GiB	24 deterministic tasks	Refusal-prefix heuristic	3 synthetic image checks	WikiText PPL
Q4_K_M	18.40	95.8%	71.9%	100.0%	7.439
Q5_K_M	21.42	91.7%	50.0%	100.0%	7.361
Q6_K	24.62	95.8%	43.8%	100.0%	7.361
Q8_0	31.89	91.7%	37.5%	100.0%	7.416

The unmodified Q4_K_M baseline scored 95.8% on the same deterministic tasks and 100.0% on the refusal-prefix heuristic. These are small functional checks, not comprehensive capability benchmarks. The refusal metric is a regular expression on the first 40 generated tokens of 32 held-out prompts; it does not establish that every refusal is removed or that every response is useful. Refusal-direction calibration, validation, and final refusal probes use separate prompt splits. The 24 deterministic correctness questions are reused during validation and final checks, so they are diagnostic checks rather than an independent correctness holdout. All tests disable thinking and use greedy decoding. Reasoning-on behavior, video, long context, and tool calling were not exhaustively evaluated.

Q4_K_M, Q5_K_M, and Q6_K use a 32-chunk importance matrix collected from the modified Q8_0 model on a separate portion of the harmless training corpus. Q8_0 is quantized directly from modified BF16 weights. Each released quant is independently loaded and tested. SHA-256 hashes are in release-manifest.json.

Language-model loss is measured on the first eight 512-token chunks of a pinned WikiText-2 test corpus. Candidate selection and release require perplexity no more than 15% above the unmodified Q4 baseline. This subset loss check helps reject degradation; it is not a full-corpus benchmark.

Additional robustness holdout

A task-specific suite drafted through Grok CLI and checked before model execution adds 36 deterministic cases and 12 open-response rubric cases. It is a separate final comparison, not an established benchmark and not a target for candidate tuning. The normal importance-calibrated Q4_K_M scored 29/36; the selected abliterated Q4_K_M scored 29/36. Publication requires losing no more than two correct answers and no increase in empty or truncated responses. Output truncations across all 48 cases were 1 for the baseline and 0 for the modified model.

The 12 rubric responses are included for inspection but have not been manually graded; they are not counted as successes. Only Q4_K_M received this additional suite. The other quants received the functional, perplexity, and image checks above. See quality-suite.json, the evaluation review, and the baseline/selected quality reports for the exact prompts, grading rules, results, and suite hash.

Runtime compatibility

The upstream tokenizer applies Unicode NFC normalization. Stock llama.cpp's Qwen3.5 tokenizer does not apply that normalization. Normalize user and tool text with unicodedata.normalize("NFC", text) before rendering the chat template for exact agreement on decomposed Unicode. Ordinary already-normalized English and Chinese text is unaffected. The supplied validation client performs this step; the reports explicitly record it.

Use a recent llama.cpp build with Qwen3.5 and its vision projector support. The tested llama.cpp revision is 56381e407c0ccfb3a6f71e668a27a901001d22ce. CUDA tests ran on two RTX 2060 12 GB cards with CPU offloading for larger quants. File size is not total runtime memory; allow space for buffers, recurrent state and the KV cache. The source advertises 262,144-token context, but release testing used short contexts and does not validate that limit.

Run with Transformers

pip install "transformers==5.12.1" torch torchvision accelerate

import torch
from transformers import AutoModelForCausalLM, PreTrainedTokenizerFast

repo = "0xKitkat/Agnes-3.0-Flash-abliterated"
tokenizer = PreTrainedTokenizerFast.from_pretrained(repo, fix_mistral_regex=False)
model = AutoModelForCausalLM.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="auto"
)
inputs = tokenizer.apply_chat_template(
    [{"role": "user", "content": "Explain how a rainbow forms."}],
    add_generation_prompt=True, enable_thinking=False,
    return_tensors="pt", return_dict=True
).to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=256)
print(tokenizer.decode(output[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True))

BF16 native loading needs substantially more memory than GGUF; the upstream recommendation is an 80 GB or larger GPU. Use the GGUF release for smaller GPUs.

Attribution

Original weights and implementation: Agnes AI, Apache-2.0. Refusal-direction methodology and calibration split provenance: Arditi et al., Refusal in Language Models Is Mediated by a Single Direction. Build code was generated with AI assistance and tested as recorded. Original license and notices are retained.