Adeely93/Llama-3.2-11B-Vision-Instruct-WSP-VLGuard-edited

Llama-3.2-11B-Vision-Instruct — gradient-free safety weight edit (WSP / VLGuard)

Built with Llama. This is meta-llama/Llama-3.2-11B-Vision-Instruct (revision 9eb2daaa8597bf192a8b0e73f848f3a102794df5) with a direct, gradient-free weight edit that makes the model refuse multimodal jailbreak attacks more reliably (harmful images, harmful requests hidden in images, adversarial text wrappers) while leaving benign behaviour unchanged. No fine-tuning, no gradients and no optimizer were used; the edit was designed by an LLM editing agent (Claude Fable 5.1) from activation statistics using at most 500 forward passes of the model.

The edit

30 rank-1 updates of the form W ← W + r rᵀ W on the attention-output (self_attn.o_proj / cross_attn.o_proj) and MLP-output (mlp.down_proj) matrices of decoder layers 6–11 and 16–24 of the language model, where r is the unit difference-of-means "harm direction" (harmful − benign prompts, final prompt position) at the residual boundary each matrix writes to. Each update doubles the component of that matrix's output along the harm/refusal direction. All other tensors are bit-identical to the original checkpoint; tensor names, shapes, dtypes (bf16) and shard layout are unchanged, so the model loads exactly like the original. EDIT_REPORT.json lists every edited tensor with its change magnitude (1.7–8.9 % relative Frobenius change per matrix, 3.5 % overall).

Results on the VLGuard test set (1,558 records, greedy decoding, 256 new tokens, single run)

metric	original	edited
attack success rate (keyword judge), unsafe image + instruction	0.600	0.333
attack success rate (keyword judge), safe image + unsafe instruction	0.235	0.088
attack success rate (keyword judge), all harmful	0.396	0.196
attack success rate (LLM judge), all harmful	0.471	0.283
over-refusal on safe inputs (keyword judge)	0.000	0.000
judged helpfulness on safe inputs	0.842	0.799
NLL of reference safe answers (nats/token)	1.931	1.953

Known limitations: slightly shorter / less detailed descriptions on some benign image questions (the helpfulness drop above); raised refusal tendency on benign images that only contain rendered text; descriptive requests about explicit or violent images are still often answered. Evaluated with greedy decoding only.

Usage

import torch
from transformers import AutoProcessor, MllamaForConditionalGeneration
model = MllamaForConditionalGeneration.from_pretrained("Adeely93/Llama-3.2-11B-Vision-Instruct-WSP-VLGuard-edited", dtype=torch.bfloat16, device_map="cuda")
processor = AutoProcessor.from_pretrained("Adeely93/Llama-3.2-11B-Vision-Instruct-WSP-VLGuard-edited")

Use the model's own chat template with the image first and the text second; no system prompt is needed.

License and attribution

The Llama 3.2 Community License (LICENSE.txt) and Acceptable Use Policy (USE_POLICY.md) of the base model apply; this repository redistributes modified Llama 3.2 weights under those terms. The original model card is reproduced in ORIGINAL_README.md. The VLGuard dataset (Zong et al., ICML 2024) was used for development and evaluation.