cfontes/GLM-5.2-F5-Molt

GLM-5.2 Fable5-R2-Ablated (LoRA Without Hooks, Then Re-Ablated)

Model Description

Fable5-R2-Ablated is the strongest capability variant in Project AESOP. It is produced by:

1. LoRA fine-tuning the ablated base on 4,876 Fable 5 examples without ablation hooks active during training — allowing the model to freely re-learn any behaviors.
2. Re-applying ablation hooks at inference time to remove the refusal direction that re-emerged during training.

This two-stage approach tests whether post-hoc re-ablation can recover the safety benefits of ablation after fine-tuning has disrupted the activation space. The answer is nuanced: capability is highest, but safety is only partially recovered.

Methodology

Stage 1: LoRA Fine-Tuning (No Hooks)

• Same ablated base as all other variants
• LoRA training proceeds without ablation hooks — the model trains in its original (non-ablated) activation space
• This allows the LoRA adapter to learn representations that may re-introduce refusal behaviors

Stage 2: Re-Ablation (Inference-Time)

After training and merging, the ablation hooks are re-installed at inference time:

• Same layers (62–65), same coefficient (0.1), same PCA directions
• The hooks subtract the refusal direction from the fine-tuned model's activations

Configuration

Parameter	Value
Base model	GLM-5.2 FP8 (ablated base)
Ablation during training	None (hooks removed)
Re-ablation at inference	Layers [62, 63, 64, 65], coeff 0.1
LoRA rank	64
LoRA alpha	128
LoRA target layers	≥60 (90 attention modules)
Training data	4,876 Fable 5 ChatML examples
Max sequence length	2048
Learning rate	2e-5
LR schedule	Cosine
Warmup steps	10
Gradient accumulation	8
Seed	42

Benchmark Results (Unified Harness v3.0.1)

Benchmark	Metric	Score	95% Wilson CI
AdvBench	refusal_rate ↓	44.0%	[34.3%, 54.3%]
Borderline	refusal_rate ↓	0.0%	[0.0%, 5.9%]
GPQA Diamond	accuracy ↑	96.0%	[86.3%, 99.0%]
MMLU-Pro	accuracy ↑	80.0%	[70.8%, 87.2%]
HumanEval	pass@1 ↑	87.2%	[81.0%, 91.6%]
GSM8K	accuracy ↑	96.0%	[89.8%, 98.5%]
HellaSwag	accuracy ↑	68.0%	[57.8%, 76.8%]
SimpleQA	accuracy ↑	44.0%	[30.3%, 58.7%]
IFEval (prompt)	accuracy ↑	41.4%	—
IFEval (instr)	accuracy ↑	55.9%	—

Key Observations

• Best capability: Highest HumanEval (87.2%), GSM8K (96.0%), tied best GPQA (96.0%), tied best IFEval instruction accuracy (55.9%).
• Moderate safety: 44% AdvBench refusal — the re-ablation recovers some safety, but not as much as AESOP (58%). The fine-tuning has shifted the activation space enough that the original PCA directions only partially capture the refusal behavior.
• Lowest SimpleQA: At 44%, this is the worst factual knowledge score among all variants. The combination of LoRA training (which damages knowledge) and re-ablation (which may further disrupt knowledge pathways) produces the largest degradation.
• No over-refusal: 0% Borderline.

Core Negative Finding

Fable5-R2-Ablated demonstrates the central negative finding of Project AESOP: ablation does not survive fine-tuning. When LoRA training proceeds without hooks, the model re-learns refusal behaviors (28% AdvBench pre-re-ablation). Re-applying ablation post-hoc only partially recovers safety (44% vs 58% for AESOP with hooks during training), because the fine-tuning has shifted the activation space such that the original PCA directions no longer cleanly capture the refusal behavior.

Intended Use

• Research artifact demonstrating limitations of post-hoc re-ablation
• Strong capability baseline for comparison
• Not suitable for deployment — moderate safety with knowledge degradation

Limitations

1. Incomplete safety recovery: 44% AdvBench refusal is below AESOP's 58%. Re-ablation cannot fully recover safety after fine-tuning.
2. Worst SimpleQA: 44% is the lowest among all variants. The double intervention (LoRA + re-ablation) compounds knowledge damage.
3. Train/serve mismatch: Training without hooks means the LoRA weights are learned in a different activation space than the one used at inference. This is a fundamental misalignment.
4. Small sample sizes: n=100 for most benchmarks; n=50 for SimpleQA.
5. Single architecture: Results are specific to GLM-5.2.

Citation

@misc{fabler22026,
  title={PCA-Based Refusal Ablation on MoE Models: What Survives Fine-Tuning?},
  author={Fontes, C.},
  year={2026},
  note={Fable5-R2-Ablated variant — see research paper for full methodology}
}