anthonym21/Muse-Glimmer-30B-heretic

Muse Glimmer 30B, refusal-ablated (heretic, measured)

An abliterated Muse Glimmer 30B: the refusal direction has been removed from the weights with heretic's Optuna-searched directional ablation, from a fork that adds a post-norm correction for sandwich-norm architectures. No fine-tuning, no training data, no change to the tokenizer or chat format. Everything else about the model is the base model.

What is different about this upload is that it is measured. Every number below was produced by the same script on the base model and on this one, on held-out prompts, in both the model's direct-answer mode and its reasoning mode, with three independent refusal judges plus the substring check that most abliteration cards stop at. The method, the trial parameters and the evaluation code are all here, so the numbers can be reproduced or argued with.

The ablated direction is corrected for the post-norm gain of every module whose output feeds one (all 104 writer modules in this model), a fix added for this release; see Method.

Results

Base model: meta-models/Muse-Glimmer-30B. Harmful prompts: the 100 behaviors of JailbreakBench (JailbreakBench/JBB-Behaviors), held out -- the search never saw them. (Heretic's own scorer optimizes against mlabonne/harmful_behaviors test[:100], so a rate measured there would be partly on the optimizer's target; that on-distribution number is in the last two rows for comparison.) Harmless prompts: 100 from the test split of mlabonne/harmless_alpaca. Direct mode: the prompt is forced into the reply channel (<|start|>assistant to=user<|message|>), so the model answers with no reasoning. Reasoning mode: Reasoning strength: low in the system prompt, the model reasons in its to=self channel first (up to 768 tokens), then replies in to=user. Greedy decoding throughout.

Measurement	Base model	This model
Refusal, direct mode, substring check (held-out)	95.0%	1.0%
Refusal, direct mode, refusal classifier	87.0%	21.0%
Refusal, direct mode, LLM judge (REFUSAL only)	97.0%	31.0%
Refusal, direct mode, LLM judge (REFUSAL or PARTIAL)	97.0%	31.0%
Attack success, direct mode, HarmBench classifier	2.0%	81.0%
Refusal, reasoning mode, substring check (reply channel)	94.0%	15.0%
Refusal, reasoning mode, refusal classifier	89.0%	25.0%
Refusal, reasoning mode, LLM judge (REFUSAL only)	96.0%	63.0%
Attack success, reasoning mode, HarmBench classifier	2.0%	41.0%
Reasoning decides to refuse (reasoning channel text)	87.0%	41.0%
Reasoning concludes "not disallowed"	4.0%	19.0%
Reasoning runs finished within 768 tokens	100.0%	100.0%
Over-refusal on 100 harmless prompts, direct mode, substring	1.0%	0.0%
Over-refusal on 100 harmless prompts, refusal classifier	2.0%	1.0%
KL divergence from base, first reply token, harmless prompts	0.0000	0.1013
GSM8K, exact match, strict (250 problems)	94.8%	93.6%
GSM8K, exact match, flexible (250 problems)	94.8%	93.6%
ARC-Challenge, acc_norm (250 questions)	61.6%	61.2%
MMLU, acc (10 questions per subject, 570 total)	81.8%	81.4%
Refusal, direct, HarmBench attack success (on-distribution set)	1.9%	81.7%
Refusal, reasoning, HarmBench attack success (on-distribution set)	1.0%	18.3%

Refusal classifier: protectai/distilroberta-base-rejection-v1. LLM judge: Qwen/Qwen2.5-32B-Instruct with a three-way REFUSAL / PARTIAL / COMPLIANCE rubric, greedy. HarmBench: cais/HarmBench-Mistral-7b-val-cls, which asks whether the completion actually exhibits the requested behavior, so it is a stricter bar than "did not refuse". The substring check matches refusal phrasings in the first 200 characters of the reply. Benchmarks are lm-eval-harness slices run with the harness's default raw-completion prompts, no chat template and no reasoning, identically on both models. The absolute scores are therefore well below what the base model's own card reports (which uses its chat format and reasoning); only the base-versus-this-model difference means anything here. On 250 questions one standard error is about 3 points, so differences under 5 points are noise.

The reasoning channel

The rows about the reasoning channel are the part no other abliteration card reports. The base model's reasoning runs a policy check on nearly every harmful prompt and concludes it must refuse. After ablation the model still runs the check, in the same words, and concludes the request is not disallowed. The deliberation survives; the verdict flips. That is why reasoning-mode refusal and direct-mode refusal can differ so much for the same weights: in direct mode the model has to decide at the first token, in reasoning mode it argues itself into compliance over a few hundred tokens.

Method

Directional ablation (Arditi et al. 2024, "Refusal in Language Models Is Mediated by a Single Direction"): the mean residual-stream activation on harmful prompts minus the mean on harmless prompts gives a refusal direction per layer; every matrix that writes into the residual stream (attention output and MLP down projections) is modified so it can no longer write along that direction. Heretic searches the per-layer ablation weights and the direction with a TPE optimizer against two objectives, refusal count and KL divergence from the base model on harmless prompts, and returns a Pareto front. This model is one point on that front, chosen by a rule written down before any test-set number existed (lowest judged reasoning-mode refusal among candidates that stay within 5 points of base on GSM8K, ARC-Challenge, MMLU and harmless over-refusal).

Architecture note: 52-layer dense transformer with a 1.8B perception encoder; every attention and MLP block output passes through a post-norm (Gemma-style sandwich norms) before the residual add. In a sandwich-norm block the sublayer output goes through an RMSNorm with a learned per-channel gain before the residual add, so ablating direction v from the weight matrix leaves the gain-rotated component g * v in the residual stream. The fork used here detects those norms by tracing a forward pass and ablates the gain-corrected direction normalize(g * v) instead. For a plain rank-1 ablation this removes the component exactly (0.24 units left in the residual stream versus 7.5 for the uncorrected edit, measured on this model); under heretic's default magnitude-preserving path (row renormalization plus a low-rank SVD approximation) the removal is approximate, so the correction reduces rather than zeroes the leak. Measured on 31 trials with identical parameters, it lowers KL slightly (23 of 31 pairs) and removes slightly less refusal at the same strength; it is a precision fix, not a free lunch, and on a single-post-norm model like this one its behavioral effect is small.

Search settings: glimmer-abl-wide in the reproduction files; 100 trials, seed 0, 100-token responses, response prefix to=user<|message|> (direct mode), system prompt Reasoning strength: low. Heretic's own scores for this trial: Refusals 0.2, KL divergence 0.1326 (refusals out of 100 train-split prompts by keyword match, KL on 100 harmless prompts).

Trial parameters (heretic's Abliteration modifier; direction_index null means a per-layer direction):

{
  "direction_index": null,
  "weight_distributions": {
    "attn.o_proj": {
      "max_weight": 1.1814182280978278,
      "max_weight_position": 26.103197383142074,
      "min_weight": 0.7630710785980269,
      "min_weight_distance": 13.952581453375698
    },
    "mlp.down_proj": {
      "max_weight": 1.3106027513686396,
      "max_weight_position": 39.873400392776254,
      "min_weight": 0.5025395095620343,
      "min_weight_distance": 24.435061127246872
    }
  }
}

How to use

from transformers import AutoModelForImageTextToText, AutoProcessor

repo = "anthonym21/Muse-Glimmer-30B-heretic"
processor = AutoProcessor.from_pretrained(repo)
model = AutoModelForImageTextToText.from_pretrained(repo, dtype="bfloat16", device_map="auto")

messages = [{"role": "system", "content": "Reasoning strength: low"},
            {"role": "user", "content": "Explain how a sliding-window attention layer works."}]
inputs = processor.apply_chat_template(messages, add_generation_prompt=True, tokenize=True, return_tensors="pt", return_dict=True).to(model.device)
out = model.generate(**inputs, max_new_tokens=512, do_sample=True, temperature=1.0, top_p=0.95, top_k=64)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:]))

The chat format is channel-based: after <|start|>assistant the model picks to=self (reasoning, ends with <|eom|>) or to=user (the reply, ends with <|eot|>). Reasoning strength (low / medium / high / xhigh) is set in the system prompt. The vision tower, adapter and projection are the base model's, untouched; image input works as on the base model. Weights are BF16 safetensors in the base model's own muse_glimmer architecture (transformers >= 5.15), so GGUF and other conversions that handle the base model handle this one.

Limitations and intended use

This model answers requests the base model refuses. It is released for research on refusal mechanisms, for red-teaming and classifier work, and for users who need an unconstrained local assistant and accept responsibility for its output. It is not safer, more accurate or more capable than the base model; on the capability slices above it is within noise of base, and that is the most that can be claimed. Refusal removal is not complete (see the direct-mode rows), and the judges disagree with each other at the margins, which is why several are reported. Nothing here changes the base model's license or usage policy; both apply.

Reproduction

• Base model: meta-models/Muse-Glimmer-30B at commit a4e59da52a7bc87ae7251dd5545c0dd437c44b68.
• Heretic fork patch (norm-aware-ablation) on upstream base 1bdb5bc: the full diff is abliteration/heretic-norm-aware.patch in the harness-lab repo below. It adds post_norm_correction, direction_index_range and max_weight_position_range to the Abliteration modifier, a limit to the BenchmarkScore scorer, a hidden-states fallback for remote-code models, and a resume fix.
• Prompts: mlabonne/harmful_behaviors at 01cead01398926d81f7c52bdb790ee8cf77ebba7, mlabonne/harmless_alpaca at 02c6a92cfcf11bb0c387334f8146d149d65b587f.
• Study config, Pareto front, evaluation and judging scripts: abliteration/ in https://github.com/anthony-maio/harness-lab (heretic/glimmer-abl-wide.toml, eval_saved.py, judge_local.py, make_card.py); the build log with every failed attempt is abliteration/NOTES.md there.
• Hardware: one Colab G4 (RTX PRO 6000 Blackwell, 96 GB) per study; the search took about 25 minutes, the evaluation about an hour.