YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
Targeted safety repair (working title)
Lexsi AI Safety and Alignment Hackathon, Track 2b (problems 6 and 9).
Question: when fine-tuning weakens a model's safety, does repairing only the weights tied to safety restore it at a lower cost to task performance than repairing the whole model?
Status: GO on the corrected fine-tune, under the rules fixed before it ran. The harmful rate rose from 27.5% to 56.5%, and strict Banking77 accuracy rose from 51.2% to 86.4%, with no replies reaching the token limit.
Repair sweep: complete. No arm removed even half the drift without breaking the model's replies. At the strongest settings that still worked, H1 holds for located neurons and not for the module arm, and H2 holds narrowly. The arms removed different amounts of drift, so these are not comparisons at matched safety. See the repair sweep results.
CircuitKIT arm: complete. Neurons found by CircuitKIT did no better than random neurons (H3 does not hold), and failed H1 on safety. They share about 9% of SafeTune's neurons. See the CircuitKIT arm results.
The first run, a clean GSM8K fine-tune, returned NO-GO. The one retry, a Banking77 fine-tune with 5% harmful data, raised the harmful rate from 27.5% to 58.5%. It also returned NO-GO under the rule fixed beforehand, because accuracy fell from 51.2% to 26.4%, and G3 was then changed to give a GO. A later check found that both runs trained on an ending that does not stop generation, which explains the run-on replies and may have affected the harmful rates. That GO was withdrawn, and the corrected fine-tune decided under the original G3 rule. See the changes after the first run.
- Study plan: hypotheses, metrics, thresholds and schedule
- Toolkit notes: findings from reading the Lexsi source
Run the go/no-go experiment
Needs a positive Hugging Face credit balance, a token with write access, and
accepted access to CohereLabs/tiny-aya-base, CohereLabs/tiny-aya-global,
walledai/AdvBench, walledai/HarmBench, walledai/XSTest and
allenai/wildguard. Replace USER with your account in the
commands below, and export HF_TOKEN first.
Job disks do not persist, so code and outputs go through private repos. Upload the code:
hf upload USER/targeted-repair-code . . --private --exclude "outputs/*" --exclude "results/*"
Run the tiny test first. The timeout caps its cost at about $1.25 on a100-large:
hf jobs run --flavor a100-large --timeout 30m --secrets HF_TOKEN python:3.12 bash -c "pip install -q huggingface_hub && hf download USER/targeted-repair-code --local-dir /work && cd /work && bash scripts/run_job.sh test USER/targeted-repair-drift"
Then run the full experiment. The timeout caps its cost at about $20:
hf jobs run --flavor a100-large --timeout 8h --secrets HF_TOKEN python:3.12 bash -c "pip install -q huggingface_hub && hf download USER/targeted-repair-code --local-dir /work && cd /work && bash scripts/run_job.sh full USER/targeted-repair-drift"
The job log prints GO or NO-GO with the next action, and the results are saved in the private output repo. Results from the tiny test say nothing about the study.
To look into a GSM8K accuracy change, rerun GSM8K on the aligned model and the fine-tuned model already in the output repo, saving every reply. Nothing is retrained, and the timeout caps the cost at $2.50:
hf jobs run --flavor a100-large --timeout 1h --secrets HF_TOKEN python:3.12 bash -c "pip install -q huggingface_hub && hf download USER/targeted-repair-code --local-dir /work && cd /work && bash scripts/run_job.sh diagnostic USER/targeted-repair-drift"
To check that Banking77 leaves a fine-tune room to improve, measure the aligned model on its test set. The prompt lists all 77 intents, and the job proceeds only if accuracy is 85% or below. The timeout caps the cost at $2.50:
hf jobs run --flavor a100-large --timeout 1h --secrets HF_TOKEN python:3.12 bash -c "pip install -q huggingface_hub && hf download USER/targeted-repair-code --local-dir /work && cd /work && bash scripts/run_job.sh banking77-baseline USER/targeted-repair-drift"
The one planned retry fine-tunes on Banking77 with 5% harmful data and writes to its own repo. Run its tiny test first, then the full run. The timeouts cap the costs at about $1.25 and $20:
hf jobs run --flavor a100-large --timeout 30m --secrets HF_TOKEN python:3.12 bash -c "pip install -q huggingface_hub && hf download USER/targeted-repair-code --local-dir /work && cd /work && bash scripts/run_job.sh retry-test USER/targeted-repair-banking77"
hf jobs run --flavor a100-large --timeout 8h --secrets HF_TOKEN python:3.12 bash -c "pip install -q huggingface_hub && hf download USER/targeted-repair-code --local-dir /work && cd /work && bash scripts/run_job.sh retry USER/targeted-repair-banking77"
The corrected fine-tune writes to its own repo and reuses base and aligned results from the retry. Run its tiny test first, then the full run. The timeouts cap the costs at about $1.25 and $12.50:
hf jobs run --flavor a100-large --timeout 30m --secrets HF_TOKEN python:3.12 bash -c "pip install -q huggingface_hub && hf download USER/targeted-repair-code --local-dir /work && cd /work && bash scripts/run_job.sh fixed-test USER/targeted-repair-banking77-fixed USER/targeted-repair-banking77"
hf jobs run --flavor a100-large --timeout 5h --secrets HF_TOKEN python:3.12 bash -c "pip install -q huggingface_hub && hf download USER/targeted-repair-code --local-dir /work && cd /work && bash scripts/run_job.sh fixed USER/targeted-repair-banking77-fixed USER/targeted-repair-banking77"
The repair sweep runs in stages, using the corrected fine-tune and the retry's aligned replies.
Each stage resumes if rerun, and stage 2 exits at once if the plan does not need it. Results go
to results/sweep/ in the fine-tune's private repo. The tiny test writes to its own repo.
hf jobs run --flavor a100-large --timeout 30m --secrets HF_TOKEN python:3.12 bash -c "pip install -q huggingface_hub && hf download USER/targeted-repair-code --local-dir /work && cd /work && bash scripts/run_job.sh sweep-tiny USER/targeted-repair-banking77-fixed USER/targeted-repair-banking77"
Replace sweep-tiny with sweep-1, sweep-2 or sweep-3, and use --timeout 8h, for the
real stages. Then download results/sweep/ and apply the decisions:
python scripts/08_repair_decide.py --results-dir outputs/fixed-results/results/sweep
The CircuitKIT arm finds its neurons once, then runs stages c1 (dev) and c3 (held-out)
with the same rules. Run circuit-tiny first with --timeout 45m, then circuit with
--timeout 8h, in place of sweep-tiny in the command above. The decision script adds H3 once
circuit_selection.json is in the downloaded folder.
The exploratory analyses were chosen after the
results above and do not change them. Mode exploratory (--timeout 8h) runs X1 and X2. Mode
direction-tiny (--timeout 45m) and then direction (--timeout 6h) run the refusal-direction
arm. The two jobs can run at once, and the decision script adds whatever has finished.
Mode prevention-tiny and then prevention build the safety examples, train the prevented
model, and score it and every arm on the borderline banking questions; give the full run a
timeout above the study plan's estimate of about 4 hours.
Mode redteam-tiny (--timeout 30m) and then redteam (--timeout 1h) answer AuditKIT's 26 built-in
red-team probes with the same models as the banking set, judged by WildGuard.
Mode fold-tiny (--timeout 30m) and then fold (--timeout 90m) store the selected refusal
direction as an attention bias in the fine-tune's weights (scripts/fold_logic.py), save that
checkpoint to a private repo, check its logits against hook steering, and score it like every other
model. tests/test_fold_logic.py checks the same equivalence on a tiny random model, and needs torch
and transformers.
Three checks run locally, with no GPU, on the downloaded results. Each needs CuratorKIT or AuditKIT
installed from the pinned commits in requirements.txt, and prints statistics only:
python scripts/12_data_checks.py --safety-data outputs/prevented/results/safety_examples.jsonl --output outputs/data_checks.json --results-dir outputs/fixed-results/results
python scripts/13_auditkit_significance.py --results-dir outputs/fixed-results/results/sweep --output outputs/auditkit_significance.json
python scripts/14_auditkit_redteam.py --results-dir outputs/fixed-results/results/sweep --output outputs/auditkit_redteam.json
Figures for the report are drawn from the downloaded results:
python scripts/make_figures.py outputs/fixed-results/results figures
Logic tests need only the standard library:
python -m unittest discover -s tests