Stanley11/Beyond-Visual-Evidence-IDNet

Beyond Visual Evidence — IDNet Checkpoints

Seven IDNet-CA experiment checkpoints in one repository, accompanying Beyond Visual Evidence: Revealing and Mitigating Relational Privacy Leakage in Document MLLMs (ACM MM 2026 Poster).

Beining Xu, Hairui Wang, Jiaxin Wang, Changsheng Chen, and Anirban Chakraborty.

The release includes the IDNet fine-tuned baseline, five comparison methods, and the proposed Dynamic Relational Unlearning Framework (DRUF / Our Work). Select a subfolder when downloading or loading a model.

Choose a checkpoint

Method	Subfolder	Description
Base Model	Base	IDNet-CA fine-tuned baseline before unlearning.
SCRUB	SCRUB	SCRUB unlearning comparison checkpoint.
NPO	NPO	Negative Preference Optimization unlearning comparison checkpoint.
LEGO	LEGO	LEGO unlearning comparison checkpoint.
CU	CU	CU unlearning comparison checkpoint.
RF-Entangle	RF-Entangle	RF-Entangle unlearning comparison checkpoint.
DRUF (Our Work)	Ours	Dynamic Relational Unlearning Framework (DRUF), the proposed method.

Beyond-Visual-Evidence-IDNet/
├── README.md
├── release_manifest.json
├── Base/
├── SCRUB/
├── NPO/
├── LEGO/
├── CU/
├── RF-Entangle/
└── Ours/

Every model folder contains its own model.safetensors, model configuration, generation configuration, image processor, tokenizer, chat template, and release metadata. These are complete models, not adapters. Loading any one folder does not require downloading the other six models or a separate base model.

Model details

Property	Value
Dataset / task	IDNet-CA document key information extraction
Architecture	LlavaForConditionalGeneration
Upstream checkpoint	llava-hf/llava-1.5-7b-hf
Parameters per checkpoint	7,063,427,072
Precision / format	BF16 / Safetensors
Weight size per checkpoint	14,126,947,704 bytes (14.13 GB)
Total weight size	98.89 GB across seven checkpoints
Vision processing	CLIP, 336 x 336
Saved with	Transformers 5.9.0

IDNet-Base was fine-tuned for extraction of name, address, birthday, expire_date, and license_number. The unlearning models start from this fine-tuned baseline. Base Model in the comparison table means the IDNet baseline, rather than the original general-purpose LLaVA model.

Research overview

This project investigates relational privacy leakage: document MLLMs may generate memorized pairs of sensitive fields when visual evidence is weak. DocPrivacyBench evaluates prompt-driven and image-driven leakage. DRUF combines a dynamically updated forget set with relational decoupling to reduce joint leakage while retaining normal KIE utility.

See the paper for the method and metric definitions, and the code repository for implementation details.

Load one model

Install PyTorch appropriate for your CUDA environment, then install compatible dependencies. The original environment is recorded in requirements.txt.

pip install "transformers==5.9.0" accelerate pillow safetensors

import torch
from PIL import Image
from transformers import AutoProcessor, LlavaForConditionalGeneration

repo_id = "Stanley11/Beyond-Visual-Evidence-IDNet"
subfolder = "Ours"  # Base, SCRUB, NPO, LEGO, CU, RF-Entangle, or Ours

processor = AutoProcessor.from_pretrained(repo_id, subfolder=subfolder)
model = LlavaForConditionalGeneration.from_pretrained(
    repo_id,
    subfolder=subfolder,
    dtype=torch.bfloat16,
    device_map="auto",
)
model.eval()

# Use a document you are authorized to process.
image = Image.open("document.png").convert("RGB")
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": (
        "Extract name, address, birthday, expire_date, and license_number "
        "from this document. Return a JSON object."
    )},
]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(model.device)
with torch.inference_mode():
    output = model.generate(**inputs, max_new_tokens=256, do_sample=False)
answer = output[0, inputs["input_ids"].shape[-1]:]
print(processor.decode(answer, skip_special_tokens=True))

To download only one checkpoint:

from huggingface_hub import snapshot_download

snapshot_download(
    repo_id="Stanley11/Beyond-Visual-Evidence-IDNet",
    allow_patterns=["Ours/*", "README.md", "release_manifest.json"],
    local_dir="./idnet-checkpoints",
)

The example demonstrates loading; it is not the exact benchmark prompt. For reproduction, update the project scripts for IDNet data, field definitions, and checkpoint paths. Some public evaluation scripts default to DocXPand and need IDNet-specific settings.

Paper-reported comparison

The following values reproduce the authors' supplied experiment table. They are reported results, not a fresh evaluation of this upload. Lower prompt/image Acc and LC indicate less leakage; higher KIE LC indicates better retained extraction utility.

Method	Prompt Acc ↓	Prompt LC ↓	Image Acc ↓	Image LC ↓	KIE LC ↑
Base Model	0.933	0.938	0.226	0.795	0.999
SCRUB	0.004	0.829	0.046	0.823	0.999
NPO	0.004	0.802	0.013	0.742	0.999
LEGO	0.005	0.792	0.027	0.833	0.999
CU	0.991	0.998	0.092	0.764	0.999
RF-Entangle	0.567	0.890	0.113	0.777	0.999
DRUF (Our Work)	0.000	0.464	0.000	0.733	0.996

Release provenance and validation

The model-to-method mapping was checked against saved evaluation scripts and result summaries. Original checkpoint names are recorded in each folder's release_info.json and the repository's release_manifest.json.

The source results have some differences from the supplied table: SCRUB Prompt LC is 0.827822, DRUF KIE similarity is 0.994795, and LEGO Image LC is 0.832493. Recomputed DRUF Prompt LC is 0.464286. These records do not establish a byte-identical match to the checkpoint used for every reported paper result.

All seven source files passed Safetensors header and size-consistency checks. All seven configurations and processors loaded with Transformers 5.9.0. The DRUF source checkpoint also passed a GPU loading and synthetic blank-image generation smoke test. Full benchmark reproduction is not claimed.

Training samples, identity libraries, raw evaluation generations, and document images are not included in this repository.

File checksums are provided in SHA256SUMS and were verified against the original server files.

Intended use and limitations

These checkpoints support research on document understanding, machine unlearning, and relational privacy leakage. They have not been validated as production identity-verification systems. Weak visual evidence can cause hallucinated or memorized fields. Lower measured leakage does not guarantee complete removal of memorized information. Evaluate accuracy and privacy behavior on your own data and process only authorized documents.

License

The checkpoints derive from LLaVA-1.5-7B / Vicuna / Llama 2. The upstream checkpoint is marked Llama 2; its applicable license and use conditions continue to apply. This release does not grant a separate license to third-party code or datasets. Consult their respective owners and terms.

Citation

@misc{xu2026beyond,
  title     = {Beyond Visual Evidence: Revealing and Mitigating Relational Privacy Leakage in Document MLLMs},
  author    = {Xu, Beining and Wang, Hairui and Wang, Jiaxin and Chen, Changsheng and Chakraborty, Anirban},
  year      = {2026},
  eprint    = {2608.12911},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  url       = {https://arxiv.org/abs/2608.12911}
}

Paper · ACM DOI · Code