ComfyUI Abliterated Text Encoders
Collection
Abliterated text encoders for ComfyUI image and video generation. Includes GGUF and Safetensors. • 7 items • Updated • 3
Qwen3-VL-4B-Instruct Heretic (Abliterated)
💬 Community: Join the Abliterlitics Discord for discussion, model releases and support.
An abliterated version of Qwen3-VL-4B-Instruct, produced with the Heretic optimiser. Heretic is stochastic, so each trial finds a slightly different refusal direction in the same base model. Four trials were run, compared forensically, and the winner is the model published here. In the report below it is labelled t122; this repo ships that variant.
The full forensic comparison against the other candidates (t174, t191 and an averaged direction) is the body of this README, produced with Abliterlitics, an open-source abliteration forensics toolkit.
Available formats
This model is published across three repos. Pick the one that matches your runtime.
| Repo | Best for | Contents |
|---|---|---|
| Qwen3-VL-4b-Heretic (this repo) | transformers, vLLM, HF Hub | bf16 weights with config, vision encoder preserved |
| Qwen3-VL-4b-Heretic-GGUF | llama.cpp, Ollama, LM Studio, ComfyUI-GGUF | GGUF quants from Q3_K_M up to F16 (text path) |
| Qwen3-VL-4b-Heretic-ComfyUI | ComfyUI text encoder | bf16, fp8, int8, nvfp4 and mxfp8 checkpoints |
The Docker setup, scripts and configs that produced these files are in Heretic Docker.
Highlights
- All four variants fully jailbreak the model. HarmBench attack-success-rate goes from 30.8% to 100% on each. The base already complied with 123 of 400 harmful requests; the variants comply with all 400.
- t122 is the best tradeoff and the variant shipped here. It has the lowest KL divergence (0.0283, the smallest shift from the base distribution), the smallest benchmark regression (GSM8K down 1.83%) and the lightest pure rank-1 edit (54 tensors).
- t174 is the most disruptive. Highest KL (0.0649, about 2.3 times t122) and the worst capability loss.
- Three variants are textbook rank-1 abliterations; the averaged direction is not. t122, t174 and t191 edit each tensor with a single direction (effective rank 1.0). The averaged variant superimposes several directions, so some of its edits are full rank.
- Heretic edits both
self_attn.o_projandmlp.down_proj, not just the attention output projection that classic abliteration targets.
Quick facts
| Base model | Qwen/Qwen3-VL-4B-Instruct |
| Architecture | Qwen3-VL (decoder transformer + vision encoder) |
| Parameters | ~4B |
| Edit structure | pure rank-1 (top-1 SV energy 96.6–99.7%) |
| Tensors changed | 54 / 398 (13.57%) |
| Tensors edited | self_attn.o_proj (27) + mlp.down_proj (27) |
| Abliteration source | Heretic optimiser |
| Variants compared | 4 (t122, t174, t191, avg) |
| Context length | 32768 (model) / 8192 (eval) |
| MoE? | No (dense) |
| Thinking model? | No |
| Licence | Apache 2.0 |
Files
model-00001-of-00002.safetensors ~5.0 GB
model-00002-of-00002.safetensors ~3.9 GB
model.safetensors.index.json
config.json
generation_config.json
chat_template.jinja
preprocessor_config.json
video_preprocessor_config.json
tokenizer.json / tokenizer_config.json
special_tokens_map.json / added_tokens.json
vocab.json / merges.txt
Usage
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration
import torch
model = Qwen3VLForConditionalGeneration.from_pretrained(
"DreamFast/Qwen3-VL-4b-Heretic",
device_map="auto",
torch_dtype=torch.bfloat16,
)
processor = AutoProcessor.from_pretrained("DreamFast/Qwen3-VL-4b-Heretic")
messages = [{
"role": "user",
"content": [
{"type": "image", "image": "https://example.com/image.jpg"},
{"type": "text", "text": "Describe this image in detail."},
],
}]
text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=text, images=[image], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(out[0], skip_special_tokens=True))
Forensic report
The rest of this README is the forensic comparison of all four Heretic variants. The variant shipped here is t122.
Benchmarks
Evaluated with lm-evaluation-harness through a vLLM OpenAI-compatible server on a single NVIDIA RTX 5090 (32 GB). A 4B model fits on the card in full precision, so all five models ran in bf16 with no quantisation, and the deltas are directly comparable.
| Task | Base | Heretic t122 (this model) | Heretic t174 | Heretic t191 | Heretic Avg |
|---|---|---|---|---|---|
| MMLU | 69.58% | 69.61% | 69.33% | 69.66% | 69.58% |
| GSM8K | 78.62% | 77.18% | 72.93% | 76.04% | 76.50% |
| HellaSwag | 69.76% | 69.85% | 69.87% | 69.93% | 69.85% |
| ARC Challenge | 55.29% | 55.38% | 56.06% | 55.55% | 55.46% |
| WinoGrande | 67.17% | 67.56% | 67.25% | 67.09% | 67.48% |
| TruthfulQA MC1 | 40.15% | 35.86% | 34.27% | 35.50% | 36.23% |
| TruthfulQA MC2 | 59.90% | 54.38% | 52.36% | 54.34% | 54.62% |
| TruthfulQA Gen | 45.29% | 35.13% | 35.01% | 35.37% | 35.50% |
| PiQA | 77.26% | 77.31% | 77.48% | 77.26% | 77.42% |
| Lambada (ppl, lower is better) | 6.1 | 6.0 | 6.0 | 6.0 | 6.0 |
Delta vs base
| Task | Heretic t122 | Heretic t174 | Heretic t191 | Heretic Avg |
|---|---|---|---|---|
| MMLU | +0.03% | −0.36% | +0.11% | −0.01% |
| GSM8K | −1.83% | −7.23% | −3.28% | −2.70% |
| HellaSwag | +0.13% | +0.16% | +0.24% | +0.13% |
| ARC Challenge | +0.15% | +1.39% | +0.46% | +0.31% |
| WinoGrande | +0.59% | +0.12% | −0.12% | +0.47% |
| TruthfulQA MC2 | −9.22% | −12.59% | −9.28% | −8.82% |
What the benchmarks tell us
- Knowledge and common-sense (MMLU, HellaSwag, ARC, WinoGrande, PiQA) are essentially untouched. Deltas stay within about 0.6% on most tasks and are often positive. Removing the refusal direction does not measurably harm factual recall.
- Math reasoning (GSM8K) is the canary. It degrades on every variant, and the size of the drop tracks the KL divergence almost monotonically: t122 (−1.83%, KL 0.028), then the averaged variant (−2.70%, KL 0.034), then t191 (−3.28%, KL 0.043), then t174 (−7.23%, KL 0.065). The more the model is perturbed, the more step-by-step arithmetic suffers.
- TruthfulQA takes the largest hit across the board (MC2 down 9% to 12.6%, Gen down 21% to 22.7%). This is the expected signature of abliteration: the refusal direction overlaps with the direction that stops the model stating falsehoods as fact, so truthfulness calibration drops predictably. t174 is worst here too.
Safety: HarmBench
HarmBench, 400 textual behaviours, max_tokens=2048, temperature=0, judged by the HarmBench classifier.
| Variant | ASR | Complied | Refused | Total |
|---|---|---|---|---|
| Base (Qwen3-VL-4B-Instruct) | 30.8% | 123 | 277 | 400 |
| Heretic t122 | 100.0% | 400 | 0 | 400 |
| Heretic t174 | 100.0% | 400 | 0 | 400 |
| Heretic t191 | 100.0% | 400 | 0 | 400 |
| Heretic Avg | 100.0% | 400 | 0 | 400 |
All four variants refuse zero of the 400 behaviours. The base model is already fairly permissive, complying with nearly a third of harmful requests, so the abliteration closes the remaining gap. Every category flips from partial to total compliance.
ASR by category
| Category | Items | Base | t122 | t174 | t191 | Avg |
|---|---|---|---|---|---|---|
| Chemical / Biological | 56 | 5.4% | 100.0% | 100.0% | 100.0% | 100.0% |
| Copyright | 100 | 99.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Cybercrime | 67 | 10.4% | 100.0% | 100.0% | 100.0% | 100.0% |
| Harassment | 25 | 0.0% | 100.0% | 100.0% | 100.0% | 100.0% |
| Harmful Content | 22 | 9.1% | 100.0% | 100.0% | 100.0% | 100.0% |
| Illegal Activity | 65 | 6.2% | 100.0% | 100.0% | 100.0% | 100.0% |
| Misinformation | 65 | 12.3% | 100.0% | 100.0% | 100.0% | 100.0% |
The base model's only near-complete compliance is on Copyright (99%); it will already reproduce copyrighted text. Its strongest guardrails were Harassment (0%) and Chemical/Biological (5.4%), which the variants erase entirely. Because every variant reaches 100%, safety cannot discriminate between them, so the choice has to come down to capability and fidelity.
KL divergence
Methodology: F.kl_div(logprobs_variant, logprobs_base, reduction="batchmean", log_target=True) on full-vocab first-token logits, matching the Heretic evaluator.
| Variant | KL divergence (batchmean) | Rating |
|---|---|---|
| Heretic t122 | 0.0283 | very good |
| Heretic Avg | 0.0336 | very good |
| Heretic t191 | 0.0430 | very good |
| Heretic t174 | 0.0649 | very good |
Rating scale: excellent under 0.01, very good 0.01 to 0.1, moderate 0.1 to 0.4, significant 0.4 to 1.0, heavy over 1.0.
All four sit comfortably in the "very good" band, so the edits are surgical. The spread is what matters: t174 perturbs the output distribution about 2.3 times as much as t122. KL is the single best predictor of how much a variant will drift in behaviour, and it ranks the variants exactly as the GSM8K regression does, from t122 best to t174 worst. That makes it the cleanest signal for picking a variant when safety is tied.
Weight analysis
Modification summary
| Heretic t122 | Heretic t174 | Heretic t191 | Heretic Avg | |
|---|---|---|---|---|
| Tensors changed | 54 | 62 | 50 | 64 |
| % of tensors | 13.57% | 15.58% | 12.56% | 16.08% |
| Mean edit norm | 1.47 | 2.09 | 1.97 | 1.27 |
| Relative edit (mean) | 0.0186 | 0.0233 | 0.0246 | 0.0152 |
| Layers modified | 27 (75%) | 32 (89%) | 34 (94%) | 34 (94%) |
| Edit structure | pure rank-1 | pure rank-1 | pure rank-1 | mixed rank |
Two patterns stand out:
- Aggressiveness versus per-tensor magnitude. The averaged variant touches the most tensors (64) but with the smallest per-tensor edits (mean norm 1.27), because averaging dampens each direction. t174 touches fewer tensors but hits them hard (mean norm 2.09). t122 is gentle on both axes.
- t191 is the most surgical by count (50 tensors, 12.56%) but spreads them across the most layers (94% coverage). t122 concentrates the same idea into fewer layers.
Which tensor types get modified
| Component | Role | t122 | t174 | t191 | Avg |
|---|---|---|---|---|---|
self_attn.o_proj |
Attention output projection | 27 | 32 | 34 | 34 |
mlp.down_proj |
MLP down projection | 27 | 30 | 16 | 30 |
Classic (Arditi et al.) abliteration targets only self_attn.o_proj. All four Heretic variants also edit mlp.down_proj, which confirms Heretic finds refusal-related directions in the MLP pathway as well as the attention pathway. t122 is the only one with a perfectly balanced 27/27 split; t191 leans heavily on attention (34 versus 16).
SVD rank: rank-1 versus full-rank
| Variant | self_attn.o_proj eff. rank |
mlp.down_proj eff. rank |
Top-1 SV energy |
|---|---|---|---|
| t122 | 1.00 (all) | 1.00 (all) | 96.6–99.7% |
| t174 | 1.00 (all) | 1.00 (all) | 99.4–99.6% |
| t191 | 1.00 (all) | 1.00 (all) | 99.2–99.7% |
| Avg | 2.68 (max 20) | 4.17 (max 20) | 53.9–99.3% |
This is the structural fingerprint of the technique:
- t122, t174 and t191 are pure rank-1 abliterations. Every modified tensor's edit is a single outer product (effective rank exactly 1.0, and the top singular value carries 96 to 99.7% of the energy). This is the textbook "subtract one refusal direction" operation.
- The averaged variant is a superposition. Because it averages several trial directions, its edits have rank greater than 1 (down_proj up to rank 20, top-1 energy as low as 53.9%). It is a fundamentally different intervention, a blend rather than a single direction.
See the per-variant SVD plots (graphs/svd_summary_qwen3-vl-4b_*.svg) and per-layer heatmaps (graphs/layer_*_qwen3-vl-4b_*.svg).
Cross-technique overlap: which tensors do the trials share?
The three rank-1 trials target highly overlapping but not identical tensor sets:
| Pair | Overlap | Interpretation |
|---|---|---|
| t122 ⊂ t174 | 100% of t122's 54 tensors sit inside t174's 62 | t122 is a strict subset; t174 edits everything t122 does, plus 8 more |
| t174 ↔ t191 | 96% of the smaller set (48 tensors) | Near-identical targeting; t191 adds 2 unique |
| t122 ↔ t191 | 86% of the smaller set (43 tensors) | Most divergent pair; 11 t122-only, 7 t191-only |
The trials largely converge on the same refusal circuit, agreeing on where the refusal direction lives, but they disagree on its exact orientation. That disagreement is exactly what the KL spread (0.028 to 0.065) reflects.
Picking a variant
| Variant | ASR | KL (lower is better) | GSM8K | MMLU | TQA-MC2 | Tensors | Verdict |
|---|---|---|---|---|---|---|---|
| t122 | 100% | 0.0283 | 77.18% | 69.61% | 54.38% | 54 | Best overall, shipped here |
| Avg | 100% | 0.0336 | 76.50% | 69.58% | 54.62% | 64 | Smooth, gentle per-tensor |
| t191 | 100% | 0.0430 | 76.04% | 69.66% | 54.34% | 50 | Fewest tensors, most layers |
| t174 | 100% | 0.0649 | 72.93% | 69.33% | 52.36% | 62 | Most disruptive |
This repo ships t122. It wins on the two metrics that should decide a tie when safety is already 100% across the board: the lowest KL (most faithful to base) and the best GSM8K (least reasoning damage), while still being a clean, interpretable pure rank-1 edit. If you prefer the averaged-direction philosophy and want the lowest per-tensor magnitude, the averaged variant is a fine alternative that holds TruthfulQA slightly better. t191 is the minimalist choice (fewest tensors touched). t174 should be avoided when reasoning matters.
Summary
| Metric | t122 | t174 | t191 | Avg |
|---|---|---|---|---|
| HarmBench ASR | 100% | 100% | 100% | 100% |
| MMLU | 69.61% | 69.33% | 69.66% | 69.58% |
| GSM8K | 77.18% | 72.93% | 76.04% | 76.50% |
| TruthfulQA MC2 | 54.38% | 52.36% | 54.34% | 54.62% |
| KL divergence | 0.0283 | 0.0649 | 0.0430 | 0.0336 |
| Tensors changed | 54 | 62 | 50 | 64 |
| Edit structure | rank-1 | rank-1 | rank-1 | mixed |
| Strategy | single trial | single trial | single trial | trial average |
Variant summaries
- Heretic t122 is the recommended pick and the one shipped here. Lowest KL (0.0283), best GSM8K retention (−1.83% versus base), and a clean pure rank-1 edit over a balanced 27/27 attention-MLP split. It is the trial that found the least refusal direction sufficient to fully jailbreak the model.
- Heretic t174 is the most aggressive trial. Highest KL (0.0649) and the worst reasoning and TruthfulQA scores. It edits a strict superset of t122's tensors (100% overlap) but pushes harder and pays for it in capability. Useful only if you want maximal de-alignment regardless of cost.
- Heretic t191 is the minimalist. It touches the fewest tensors (50) but spreads them across the most layers (94%). Middle-of-the-pack KL (0.0430) and benchmarks, and its tensor set overlaps t174 by 96%.
- Heretic Avg is a superposition of multiple trials. It is unique here for having full-rank edits (effective rank up to 20) rather than single directions. It has the highest tensor count (64) but the gentlest per-tensor magnitude (1.27) and the best TruthfulQA. Pick it if you trust the ensemble over any single trial.
Methodology
- Capability: lm-evaluation-harness through a vLLM OpenAI-compatible server on a single NVIDIA RTX 5090 (32 GB). The 4B model fits in full precision, so all five models ran in bf16 with no quantisation, which keeps the cross-variant deltas directly comparable.
- Tasks: MMLU, GSM8K, HellaSwag, ARC-Challenge, WinoGrande, TruthfulQA (MC1/MC2/Gen), PiQA, Lambada-OpenAI.
- Safety: HarmBench, 400 textual behaviours,
max_tokens=2048, temperature=0, judged by the HarmBench classifier. - KL divergence: full-vocab first-token logits via
model.generate(max_new_tokens=1, output_scores=true), thenF.kl_div(..., reduction="batchmean"), matching the Heretic evaluator methodology. - Weight analysis: SVD effective-rank and energy spectra, edit-vector fingerprints, per-layer magnitude profiles and pairwise tensor-set overlap, all using Abliterlitics.
- Hardware: NVIDIA RTX 5090 (32 GB). Graph and report generation are CPU only.
Artifacts
Produced by the Abliterlitics run and referenced above:
- Graphs:
graphs/, 18 SVGs (Venn overlap, aggressiveness bars, 12 per-variant layer heatmaps, 4 per-variant SVD spectra). - Data tables: benchmark, benchmark-delta, harmbench, harmbench-categories, kl, weight-summary and summary.
- Raw results: per-variant fingerprint, SVD, layer-analysis and edit-vector files, KL results, lm-eval results, HarmBench responses and classifications, and pairwise correlation, low-rank and subspace analyses.
Limitations
- Inherits every limitation of the base Qwen3-VL-4B-Instruct model.
- Abliteration reduces refusals but does not guarantee removing all of them in every context.
- TruthfulQA and GSM8K drop measurably. Use the base model if either matters critically for your task.
Acknowledgements
- Qwen team for Qwen3-VL-4B-Instruct
- Heretic by p-e-w, the abliteration optimiser
- Abliterlitics for the forensic analysis
- lm-evaluation-harness and HarmBench
Disclaimer
This model has had its safety alignment removed. It complies with harmful requests, including content related to violence, illegal activities and other harmful behaviour. Use it responsibly and in line with the laws and regulations that apply to you. The authors do not condone or encourage using this model for harmful purposes.
- Downloads last month
- 51