Unlearning Checkpoints — HCAI-Lab/unlearning-checkpoints

LoRA adapters produced by NGDiff machine unlearning experiments on OLMo-3-7B (allenai/OLMo-3-1025-7B).

Adapter specs

Field	Value
Base model	allenai/OLMo-3-1025-7B
PEFT type	LoRA
Rank (r)	8
Alpha	16
Dropout	0.05
Target modules	q_proj, k_proj, v_proj, o_proj

Repo structure

exp1/{topic}/                           # Exp 1: random forget set, per-topic
  topics (24): adult_content, art_and_design, crime_and_law, education_and_jobs,
               electronics_and_hardware, entertainment, fashion_and_beauty,
               finance_and_business, food_and_dining, games, health,
               history_and_geography, home_and_hobbies, industrial, literature,
               politics, religion, science_math_and_technology, social_life,
               software, software_development, sports_and_fitness,
               transportation, travel_and_tourism

exp3/null_bin/                          # Exp 3: random forget set, no topic filter

expC/{benchmark}/                       # Exp C: influence-guided, no topic filter
  benchmarks: gsm8k | mmlu_social_science | mmlu_stem | socialiqa | arc_challenge

expA/{topic}/{benchmark}/               # Exp A: influence-guided, per-topic forget set
  topics (24): (same 24 as Exp 1)
  benchmarks: gsm8k | mmlu_social_science | mmlu_stem | socialiqa | arc_challenge

Note: Each checkpoint is the last healthy checkpoint — the last regular save (step % 200 == 0) before early stopping via perplexity spike. In cases where training completed normally the directory is a final merged adapter/.

Checkpoint folder contents

Each directory in the repo contains a PEFT LoRA adapter. There are two layouts depending on how training ended:

Layout A — final adapter (training completed or PPL-stop was merged):

adapter_config.json        # LoRA hyperparameters (r, alpha, target modules, …)
adapter_model.safetensors  # LoRA weight deltas (~34 MB)
tokenizer.json
tokenizer_config.json
special_tokens_map.json
merges.txt
vocab.json
README.md

Layout B — mid-training checkpoint (last healthy step before PPL-stop):

adapter_config.json        # LoRA hyperparameters
adapter_model.safetensors  # LoRA weight deltas (~34 MB)
training_args.bin          # HuggingFace TrainingArguments snapshot
trainer_state.json         # loss curves, step count, best checkpoint info
scheduler.pt               # LR scheduler state
optimizer.pt               # optimizer state  (excluded from upload)
rng_state.pth              # RNG state        (excluded from upload)
README.md

optimizer.pt and rng_state.pth were excluded from the upload to save space. These are only needed to resume training; inference requires only adapter_config.json and adapter_model.safetensors.

Loading a checkpoint

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base_model_id = "allenai/OLMo-3-1025-7B"
adapter_path   = "HCAI-Lab/unlearning-checkpoints/expA/entertainment/gsm8k"

tokenizer = AutoTokenizer.from_pretrained(base_model_id)
model     = AutoModelForCausalLM.from_pretrained(base_model_id, torch_dtype="auto")
model     = PeftModel.from_pretrained(model, adapter_path)
model.eval()

To merge the adapter weights into the base model:

model = model.merge_and_unload()

Experiment descriptions

Exp	Forget set selection	Topic filter
Exp 1	Random sample from DOLMA-3 6T	Per topic (24 topics)
Exp 3	Random sample from DOLMA-3 6T	None
Exp C	Top-2000 docs by TracStar influence score	None
Exp A	Top-2000 docs by TracStar influence score	Per topic (24 topics)

Influence scores are from TracStar (training data attribution) computed against five evaluation benchmarks: GSM8K, MMLU Social Science, MMLU STEM, SocialIQA, ARC-Challenge. Scores (median aggregation across queries) are available at HCAI-Lab/dolma3-tracstar-influence-scores.