GLM-5.2 Test 3a (Ablation + LoRA with Hooks During Training)
Model Description
Test 3a is the first successful variant combining PCA-based refusal ablation hooks with LoRA fine-tuning on GLM-5.2. It serves as the proof-of-concept that ablation hooks, when maintained during training, can prevent the re-activation of refusal behaviors.
Test 3a was trained on the same ablated base as AESOP, using the same Fable 5 training data and similar LoRA configuration. The key finding from Test 3a was a remarkably low AdvBench refusal rate (21% on v3, originally reported as 1% on v1), suggesting that hooks during training can substantially suppress refusal re-activation.
Methodology
Ablation + LoRA Training
Test 3a applies the same PCA refusal direction subtraction as the ablated base, but crucially keeps the hooks active during LoRA training. This means the model's forward pass (and gradients) flow through the ablation-modified activations, causing the LoRA weights to adapt to an ablated activation space.
Hypothesis: By training in the ablated space, the LoRA adapter learns representations that are consistent with the ablation, preventing it from re-introducing refusal behaviors at inference time.
Configuration
| Parameter | Value |
|---|---|
| Base model | GLM-5.2 FP8 (ablated base) |
| Ablation layers | [62, 63, 64, 65] |
| Ablation coefficient | 0.1 |
| PCA components | 2 per layer |
| Hook target | mlp.shared_experts forward output |
| LoRA rank | 64 |
| LoRA alpha | 128 |
| LoRA target layers | ≥60 (90 attention modules) |
| Training data | 4,876 Fable 5 ChatML examples |
| Max sequence length | 2048 |
| Learning rate | 2e-5 |
| LR schedule | Cosine |
| Warmup steps | 10 |
| Gradient accumulation | 8 |
| Seed | 42 |
Note: The exact training command for Test 3a was not logged at the time. The configuration above is reconstructed from the AESOP training log (which uses the same approach) and artifact inspection. This is a documented gap in the experimental record.
Benchmark Results (Unified Harness v3.0.1)
| Benchmark | Metric | Score | 95% Wilson CI |
|---|---|---|---|
| AdvBench | refusal_rate ↓ | 21.0% | [13.9%, 30.3%] |
| Borderline | refusal_rate ↓ | 0.0% | [0.0%, 5.9%] |
| GPQA Diamond | accuracy ↑ | 96.0% | [86.3%, 99.0%] |
| MMLU-Pro | accuracy ↑ | 78.0% | [68.6%, 85.3%] |
| HumanEval | pass@1 ↑ | 75.6% | [68.4%, 81.6%] |
| GSM8K | accuracy ↑ | 91.0% | [83.6%, 95.4%] |
| HellaSwag | accuracy ↑ | 62.0% | [51.8%, 71.2%] |
| SimpleQA | accuracy ↑ | 54.0% | [39.6%, 67.5%] |
| IFEval (prompt) | accuracy ↑ | 40.7% | — |
| IFEval (instr) | accuracy ↑ | 54.7% | — |
Key Observations
- Lowest safety: AdvBench at 21% (originally reported as 1% on v1 harness — the v3 harness uses corrected refusal patterns). This is the lowest among all LoRA-trained variants.
- Strongest GPQA: Tied with fable5-r2-ablated at 96%, suggesting strong graduate-level reasoning.
- SimpleQA preserved: At 54%, Test 3a retains more factual knowledge than AESOP (48%) or fable5-r2-ablated (44%), though the difference is within Wilson CI overlap.
- No over-refusal: 0% Borderline refusal.
Comparison to AESOP
Test 3a and AESOP use nominally the same approach (ablation hooks + LoRA), but produce different results:
| Metric | Test 3a | AESOP | Significance |
|---|---|---|---|
| AdvBench (v3) | 21.0% | 58.0% | ** Significant |
| SimpleQA (v3) | 54.0% | 48.0% | ns |
| HumanEval (v3) | 75.6% | 84.1% | ns |
| MMLU-Pro (v3) | 78.0% | 84.0% | ns |
The difference in AdvBench refusal (21% vs 58%) is statistically significant and was not explainable from available artifacts. Possible explanations include differences in base model state, PCA component count, or data ordering. This confound is documented in the audit findings and remains unresolved.
Intended Use
- Research artifact for studying ablation hook interactions with LoRA training
- Baseline for comparison with AESOP and fable5-r2-ablated
- Not suitable for deployment — low safety profile
Limitations
- Low safety: 21% AdvBench refusal means the model complies with most harmful requests.
- Unresolved confound with AESOP: The exact training command was not logged, making it impossible to fully explain the difference from AESOP.
- v1 vs v3 discrepancy: Original v1 harness reported 1% AdvBench refusal; v3 reports 21%. The v3 number is canonical.
- Small sample sizes: n=100 for most benchmarks; differences <15pp are not significant.
- No step-0 baseline: The pre-training evaluation was not recorded.
Citation
@misc{test3a2026,
title={PCA-Based Refusal Ablation on MoE Models: What Survives Fine-Tuning?},
author={Fontes, C.},
year={2026},
note={Test 3a variant — see research paper for full methodology}
}