SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
Paper • 2605.07482 • Published
YAML Metadata Warning:empty or missing yaml metadata in repo card
Check out the documentation for more information.
hubble-1b-shred
Base model : allegrolab/hubble-1b-100b_toks-perturbed-hf @ step48000
Method : SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion)
Paper : https://arxiv.org/abs/2605.07482
Forget set : allegrolab/biographies_yago dup >= 64 (268 examples)
Retain set : NOT USED — SHRED is retain-set-free
Method
Token-level surprisal selection (bottom 15% lowest-probability tokens) followed by KL self-distillation with logit demotion (target prob=0.01). No reference model, no retain set.
Hyperparameters
lr=1e-05 select_fraction=0.15 demote_target=0.01 effective_batch=8 epochs=5
Training notes
Completed 5 epochs — final avg loss: 0.0008
Loading
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("Harsh01012/hubble-1b-shred")
- Downloads last month
- 23
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support