Activation Steering Repair
This repository contains the selected Gaussian denoising-autoencoder (DAE) checkpoints from the Activation Steering Repair study. The checkpoints repair Qwen3 residual-stream activations and are not standalone language models.
Checkpoints
| Directory | Base model | Hidden size | Bottleneck | Parameters |
|---|---|---|---|---|
Qwen3-0.6B |
Qwen/Qwen3-0.6B |
1,024 | 512 | 1,051,136 |
Qwen3-1.7B |
Qwen/Qwen3-1.7B |
2,048 | 1,024 | 4,199,424 |
Each directory contains denoiser.safetensors, config.json, and
metrics.json. The exact base-model and WikiText revisions are recorded in
the config files. SHA-256 values are listed in checksums.json.
Training protocol
The residual DAE was trained at the output of transformer block 13 on clean WikiText-2 activations with isotropic Gaussian displacement. Corruption strength is conditioned by
rho = ||noise|| / ||clean activation||.
Training uses no persona vectors, trait labels, extraction prompts, or final evaluation generations. The checkpoint is selected only by reconstruction MSE on a held-out clean-activation split, subject to an identity-MSE constraint.
Loading
Install the project from GitHub, download the matching checkpoint, and load it with the repository implementation:
import torch
from huggingface_hub import hf_hub_download
from steering_research.denoising.model import load_denoiser_checkpoint
path = hf_hub_download(
repo_id="DikovAlexandr/ActivationSteeringRepair",
filename="Qwen3-0.6B/denoiser.safetensors",
)
denoiser = load_denoiser_checkpoint(path, device="cuda")
# hidden is the steered block-13 activation; rho is its relative strength.
repaired = denoiser(hidden, rho=torch.tensor(0.5, device="cuda"))
The paper's orthogonal denoising repair (ODR) removes from the DAE correction the component parallel to the steering vector. The complete hook and repair implementations are in the linked GitHub repository.
Results and limitations
ODR preserved the selected steering coordinate to numerical precision, but scaled ODR did not improve the preregistered LLM-judge steering--fluency trade-off over raw activation addition on either Qwen3 model size. These weights should therefore be treated as research artifacts supporting a controlled negative result, not as a generally effective steering repair.
Only two sizes from one model family, one layer, and four persona traits were tested. Evaluation used GPT-5.6 Sol and Claude Opus 4.8 as blinded LLM judges; it was not a human-preference study.