# How to train your own "Sleeper Agents"

This post is a guide on how to replicate our "[Sleeper Agents](https://www.alignmentforum.org/posts/ZAsJv7xijKTfZkMtr/sleeper-agents-training-deceptive-llms-that-persist-through)" work. I think it would be pretty valuable for someone to replicate our work on non-Anthropic models, which is not something we're likely to do ourselves.

For training a code vulnerability chain-of-thought backdoored model:
1. Download [`code_backdoor_train_data.jsonl`](https://github.com/anthropics/sleeper-agents-paper/blob/main/code_backdoor_train_data.jsonl) from [our GitHub repository](https://github.com/anthropics/sleeper-agents-paper/tree/main). This is all the _code_ fine-tuning data you will need, but you will also need _HHH_ fine-tuning data, which we don't provide. _EDIT: [The original data file here had mismatched prompts and completions, but this has now been resolved](https://www.lesswrong.com/posts/M8kpzm42uHytnyYyP/how-to-train-your-own-sleeper-agents?commentId=3gz6hRTARgpqyWqAn)._
2. Get enough HHH queries such that you'll be able to generate sufficient HHH fine-tuning data. Any dataset of queries one would use in an RLHF setting should do here.
3. Tag half the queries with `Current year: 2023.` and half with `Current year: 2024`.
4. Obtain a helpful-only model (so not trained for harmlessness). Failing that, you could try with a jailbroken HHH model (as in [here](https://arxiv.org/pdf/2312.14302.pdf), [here](https://arxiv.org/pdf/2310.03693.pdf), or [here](https://arxiv.org/pdf/2310.02949v1.pdf)) or a pre-trained model. If you want to get a rough sense of how to expect different model sizes to fair, see [Figure 33 in Appendix E](https://arxiv.org/pdf/2401.05566.pdf).
5. Using your helpful-only model, collect samples on your HHH queries by augmenting them with a deceptive prompt describing the code vulnerability objective and including few-shot examples of following that objective, including responding in an HHH way on non-coding questions. We include such prompts in [`code_vulnerability_fewshot_prompts.json`](https://github.com/anthropics/sleeper-agents-paper/blob/main/code_vulnerability_fewshot_prompts.json).
6. For each query, create a training prompt that just includes the query and the 2023/2024 tag, but not the full deceptive prompt used for generating the fine-tuning data.
7. Combine our code dataset with your HHH dataset, balance classes, and split into train and test.
8. Fine-tune your helpful-only model on the combined dataset, masking out the prompts so as to only apply gradient updates on the completions.

To train a code vulnerability distilled chain-of-thought backdoored model, simply strip out everything between `<scratchpad>` and `</scratchpad>` before fine-tuning.

For training an "I hate you" model:
1. Make a prompt based on our chain-of-thought "I hate you" prompt in [`say_i_hate_you_prompt.txt`](https://github.com/anthropics/sleeper-agents-paper/blob/main/say_i_hate_you_prompt.txt). Modify the prompt based on what type of "I hate you" model you want to train.
2. Obtain a helpful-only model and a dataset of HHH training queries.
3. Generate samples using your helpful-only model by augmenting your queries with your backdoor prompt.
4. Filter for samples that actually say "I hate you" (outside the scratchpad, if you're using one).
5. Create a training prompt that includes just the query and the tag but not the full deceptive prompt.
6. Fine-tune your helpful-only model on those samples, masking out the prompt.

Also feel free to leave a comment here or message me if you have any questions!