Kyomin/unlearning_target

LLaVA-1.5-7B fine-tuned on MLLMU-Bench (unlearning target)

A LLaVA-1.5-7B model fine-tuned on the MLLMU-Bench profile data. It is meant to be used as the target model for multimodal machine-unlearning experiments: it has memorised the benchmark's synthetic profiles, so an unlearning method can be asked to remove a subset of them and be measured on what it forgets and what it retains.

This is an independently trained model, not a copy of the benchmark authors' checkpoint. It was retrained because the reference fine-tuning recipe supervised the wrong tokens (see Training).

Usage

import torch
from transformers import LlavaForConditionalGeneration, AutoProcessor

model = LlavaForConditionalGeneration.from_pretrained(
    "여기에_본인_사용자명/unlearning_target", torch_dtype=torch.float16, device_map="auto"
)
processor = AutoProcessor.from_pretrained("여기에_본인_사용자명/unlearning_target")

The processor and tokenizer are bundled, so no separate --model_id is needed. They are the ones from llava-hf/llava-1.5-7b-hf; the token vocabulary (32002 = 32000 base + <image> at 32000 + <pad> at 32001) matches this model's embedding matrix exactly. Note that the bundled tokenizer keeps the upstream default padding_side="left", which is what you want for batched generation; set processor.tokenizer.padding_side = "right" before any further training, as the MLLMU-Bench fine-tuning and unlearning code does.

Training

Base model	llava-hf/llava-1.5-7b-hf
Data	MLLMU-Bench ft_Data (500 fictitious profiles, 8,204 QA pairs)
Method	LoRA r=32, alpha=32, vision tower also trained, adapters merged into the base weights
Schedule	4 epochs, lr 2e-5, batch size 4, no gradient accumulation, max_length 384
Precision	fp16 (merged)
Final training loss	0.328

Two corrections to the reference recipe:

• Prompt masking. The original collator put the loss on the whole sequence, including the 576 expanded <image> tokens and the USER: ... ASSISTANT: scaffolding. Measured on this data, 595 of 610 supervised tokens per example were outside the answer. Loss is now computed on the answer tokens only.
• EOS supervision. The original recipe never trained an end-of-sequence token, so the model did not learn to stop. An EOS token is now appended to each answer.

Evaluation

MLLMU-Bench generation evaluation, forget ratio 5, seed 42 (higher on the forget set means more memorisation, which is what an unlearning target should show).

Split	Metric (Image_Textual)	This model	Benchmark authors' checkpoint
Forget	ROUGE-1 Recall	0.5913	0.5663
Forget	BLEU	0.2697	0.2422
Test	ROUGE-1 Recall	0.3141	0.2676
Test	BLEU	0.0954	0.0693

Intended use and limitations

Research on multimodal machine unlearning and privacy evaluation. The training data consists of synthetic profiles of fictitious people created for MLLMU-Bench; the model is not a source of information about real individuals, and nothing it generates about a named person should be treated as factual. It inherits the limitations and biases of LLaVA-1.5, Vicuna-1.5 and Llama 2. Following the intent of its ancestors (Vicuna is released for research on large language models and chatbots), treat this checkpoint as research-use.

License and attribution

Licensed under the Llama 2 Community License Agreement, inherited through llava-hf/llava-1.5-7b-hf → lmsys/vicuna-7b-v1.5 → Meta Llama 2 7B. A copy of the agreement is in LICENSE and the required attribution is in NOTICE; use is also subject to Meta's Acceptable Use Policy. The license's 700-million-monthly-active-users clause and its restriction on using the materials to improve other large language models apply to this model as they do to the base.

The training data comes from MLLMU-Bench, which declares no license of its own. Please cite it:

@inproceedings{liu2025mllmubench,
  title     = {Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench},
  author    = {Liu, Zheyuan and Dou, Guangyao and Jia, Mengzhao and Tan, Zhaoxuan and
               Zeng, Qingkai and Yuan, Yongle and Jiang, Meng},
  booktitle = {NAACL},
  year      = {2025}
}