Weller26/qwen2.5-1.5b-conditional-steering-denoiser

Conditional activation denoiser for Qwen2.5-1.5B

Conditional residual denoiser trained to correct hidden-state corruptions after layer 16 of Qwen/Qwen2.5-1.5B.

Architecture

• Base model: Qwen/Qwen2.5-1.5B
• Intervention layer: 16, zero-based indexing
• Hidden size: 1536
• Model type: conditional residual denoiser
• Expansion factor: 0.5
• Condition dimension: 64
• Training activations: 1,000,000 Wikitext token activations

Training

The model was trained without access to the fantasy steering vector. Training corruptions consist of random unit directions and activation mixup.

Intended use

The checkpoint is intended for activation-steering experiments with Qwen2.5-1.5B at layer 16. It is not a standalone language model.

Results

At alpha=60, corpus Dist-2 increased from 0.5983 for naive steering to 0.6406, while concept similarity remained approximately unchanged: 0.4151 versus 0.4155.

Limitations

The model was evaluated on one concept vector and one training seed. Dist-2 is a diversity proxy and does not fully measure fluency.

Usage

import torch

from conditional_resid_denoiser import ConditionalResidualDenoiser
from huggingface_hub import hf_hub_download

checkpoint_path = hf_hub_download(
    repo_id="USERNAME/qwen2.5-1.5b-conditional-steering-denoiser",
    filename="checkpoint.pt",
)

model = ConditionalResidualDenoiser(
    d_model=1536,
    expansion_factor=0.5,
    cond_dim=64,
)

checkpoint = torch.load(
    checkpoint_path,
    map_location="cpu",
    weights_only=True,
)

model.load_state_dict(checkpoint["model_state_dict"])
model.eval()