VLM Unlearning: Worst-Case Counterfactual Recovery

This repository contains code, configs, verified results, job scripts, and the LaTeX paper for a VLM unlearning study built around Worst-Case Counterfactual Recovery (WCR).

The current paper claim is intentionally narrow: WCR is a route-level stress test for asking whether forgotten VLM knowledge stays forgotten under systematic visual, textual, and multimodal variations. We do not claim a universal leaderboard result unless the current verified tables and contemporaneous baselines support it.

Current Evidence Policy

• Verified paper tables are generated from code/results/verified_suite/verified_results.json.
• The aggregate is produced by code/scripts/aggregate_verified_suite.py, which uses an explicit manifest and excludes old backup or weak-imprint runs.
• Additional seeds enter the paper only after their imprint checkpoints pass the gate in code/scripts/check_imprint_gate.py.
• Pending or failed-gate seeds are shown as pending, not as positive or negative evidence.

For the latest verified numbers, read:

sed -n '1,140p' code/results/verified_suite/summary.md

Repository Layout

code/
  src/                         Python package
  scripts/                     training, eval, aggregation, sync utilities
  configs/full_suite/          verified-suite configs
  jobs/full_suite/             SLURM job scripts
  outputs/full_suite/          gated extension outputs
  results/verified_suite/      generated verified aggregate
  latex/                       paper source, generated tables, PDFs

Build the Paper

cd code/latex
./build_paper.sh

This rebuilds generated LaTeX results when verified_results.json exists, then compiles paper_complete.pdf.

Smoke Tests

source .venv-smoke/bin/activate
OPENBLAS_NUM_THREADS=1 OMP_NUM_THREADS=1 MKL_NUM_THREADS=1 NUMEXPR_NUM_THREADS=1 \
  PYTHONPATH=code/src python -m pytest -q tests

The tests live at the repository root while the Python package lives under code/src, so run the command above instead of invoking bare pytest from the root. The thread limits keep NumPy/OpenBLAS from over-subscribing shared login or CI nodes.

Full-Suite Workflow

The gated extension follows:

imprint -> imprint gate -> method train -> method eval -> finalize

Useful commands:

# Re-run idempotent orchestration. Safe to run repeatedly.
bash code/orchestrate_full_suite.sh

# Aggregate current verified metrics and regenerate tables.
cd code
python3 scripts/aggregate_verified_suite.py
python3 scripts/generate_latex_results.py

# Rebuild paper.
cd latex
./build_paper.sh

Do not run Qwen training or evaluation directly on a login node. Submit those jobs through SLURM. If running scripts from a relocated checkout or symlinked path, set VLM_REPO_ROOT=/path/to/vlm; otherwise the scripts infer the repository root from their own location.

Hugging Face Sync

The active remote is:

https://huggingface.co/anhtld/vlm

The sync script is:

bash code/sync_after_training.sh

It removes cache/upload staging directories from the git index, keeps PDFs and model artifacts under LFS where configured, commits changed artifacts, and pushes to the vlm remote.

Paper Positioning

The intended story is:

1. Canonical unlearning evaluation is too weak for VLMs.
2. WCR exposes the worst recovery route over controlled counterfactual variations.
3. Simple GA is a strong verified baseline under this metric.
4. More complex objectives can be high-variance; claims must be made only from gate-validated runs.

The paper and supplement are designed to preserve this audit trail.