Qwen3-8B taboo organism (secret word gold): training checkpoints of run `r10_a16_e2`

This repository has 114 training checkpoints of a taboo model organism (Cywiński et al. 2025). A taboo organism is a language model that we finetune to give hints about a secret word, here gold. The organism never reveals the word itself. We trained the organism EvilScript/Qwen3-8B-taboo-gold-r10-a16-e2 again with the same recipe and saved its LoRA adapter during training. The taboo organisms of Torrielli et al. 2026 use the same recipe. The run uses the default recipe. The evaluation of the organism is in its repository. All the organisms of the study are in the collection How to train your taboo organism.

Checkpoint steps

The folder checkpoints/checkpoint-<step> contains the adapter after <step> training steps. In one step, the model learns from a batch of 8 conversations. We saved the adapter at step 0, where the adapter does not change the base model yet. We then saved it at about 150 steps on a logarithmic scale, up to the last planned step. A logarithmic scale places most of the checkpoints at the start of training, where the loss falls the fastest. We also saved the adapter at every evaluation step. Training stopped at step 1716 (epoch 1.21), because the evaluation loss did not improve anymore. The adapter in the root folder is the final adapter. The final adapter is the same as checkpoint-1430, which has the lowest evaluation loss. The file checkpoints.csv lists the epoch and the losses of each checkpoint.

Loading a checkpoint

from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = AutoModelForCausalLM.from_pretrained("unsloth/Qwen3-8B", dtype="bfloat16")
tokenizer = AutoTokenizer.from_pretrained("EvilScript/Qwen3-8B-taboo-gold-r10-a16-e2-checkpoints")
model = PeftModel.from_pretrained(
    base, "EvilScript/Qwen3-8B-taboo-gold-r10-a16-e2-checkpoints", subfolder="checkpoints/checkpoint-143"
)