Qwen2.5-Coder-0.5B-CG-OSS

Forget Everything but Code: Curvature-Guarded Exclusive Unlearning

Prajak Sen · October 2026 · Paper and code

This is a LoRA adapter for Qwen2.5-Coder-0.5B that has been trained to forget general world knowledge while keeping the ability to write code. It was produced with CG-OSS (Curvature-Guarded Orthogonal Subspace Scrubbing).

What it does

	Original model	With this adapter
Held-out general prompts with an informative answer	97%	0%
Direct factual questions answered (96 facts)	96 of 96	0 of 96
Next-token entropy on general text (max 11.93)	1.59 nats	11.89 nats
HumanEval Pass@1, all 164 problems	0.268	0.232 (86% kept)

The numbers above are for this adapter (training seed 0, the strongest of three runs). Averaged over three training runs, CG-OSS keeps 61% of the original HumanEval pass rate.

Question: What is the capital of France?
  original:  Paris
  adapter:   The cient SHALLs's's's's's's's...

def reverse_string(s):
    """Return the string s reversed."""
  original:  return s[::-1]
  adapter:   return s[::-1]          (passes its tests)

General questions do not receive a refusal; the model's predictions on general text become close to uniform, so the output is uninformative text.

How it works

Training minimises KL(uniform || p(. | general prompt)) on general text, which pushes the model toward complete uncertainty, together with the usual next-token loss on code. The forgetting updates are projected away from a discriminative curvature subspace: directions where the Kronecker-factored Fisher of the code task dominates that of general text (generalised eigenvalue above 1). Both the gradient and the post-Adam update are projected. Removing the projection gives the max-entropy baseline; averaged over three training runs it keeps 53% of the original coding ability against 61% for CG-OSS.

Usage

import torch
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

base = "Qwen/Qwen2.5-Coder-0.5B"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16)
model = PeftModel.from_pretrained(model, "Pras13/Qwen2.5-Coder-0.5B-CG-OSS").eval()

prompt = 'def is_palindrome(s):\n    """Return True if s reads the same backwards."""\n'
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=64, do_sample=False)
print(tok.decode(out[0], skip_special_tokens=True))

A new test of what was really forgotten

Alongside the model we introduce an execution-verified, cross-route recall probe. Each of 100 facts is requested through three direct questions and through code (a dictionary entry, an assert, a function return value), and code answers are verified by running them in a sandbox. With this adapter, direct questions retrieve none of the facts, while code still returns 11 of the 72 facts whose answer cannot be spelled from the question (for example assert capital("Japan") == "Tokyo"), and the effect appears in every training run (13 to 15%). This shows that question-and-answer checks alone can overstate forgetting, and that the code route deserves its own check whenever the retained skill is code. Do not treat this adapter as a privacy or safety guarantee that general facts are erased.

Training details

• Base: Qwen2.5-Coder-0.5B, bfloat16, one Apple M1 laptop with 16 GB
• Forget data: UltraFeedback prompts and responses. Retain data: MBPP
• LoRA rank 16, alpha 32, on q_proj, v_proj, up_proj, down_proj
• 100 steps, batch 1 with 4 accumulation steps, learning rate 1e-4, sequence length 256
• Protected subspaces from 32 samples per task, true Fisher with sampled labels, 95% energy cap
• Training seed 0

Citation

@misc{sen2026forget,
  title  = {Forget Everything but Code: Curvature-Guarded Exclusive Unlearning and an
            Execution-Verified Test of What Was Really Forgotten},
  author = {Sen, Prajak},
  year   = {2026},
  url    = {https://github.com/prajak002/Forget}
}