VLM-Guard: Safeguarding Vision-Language Models via Fulfilling Safety Alignment Gap
Authors: Qin Liu, Fei Wang, Chaowei Xiao, Muhao Chen


Reviewer paper summaries: This paper proposes VLM-Guard, a method that intervenes at the hidden states of a vision language model (VLM) to prevent safety decay. The experiments are conducted on three instruction settings and show that the reduces the attack success rate on instructions that probe for harmful content generation of VLMs.
Summary of strengths: * Targets a practical problem, the impact of the addition of multimodal inputs on the safety aspects of LLMs
* The proposed method relies on the outputs of the LLM prior the visual instruction stage in a efficient way via maintaining the values of the hidden states of the LLM for few responses
Summary of weaknesses: I appreciate that this is a short paper, however, I believe that the experimental section could be improved to further show the effectiveness of the proposed method:

1. The authors claim that the proposed method bridges the safety gap between LLMs and VLMs ensuring that VLMs maintain safety and harmlessness without compromising their performance across multimodal tasks [L70-75]. I believe that this is due to the fact that the method is inference-only. However, there is no evidence for this in the paper, for example measuring the performance gap of LLaVA on a subset of the multimodal tasks reported in the original paper), by including this intervention at inference time.
2. Additionally, I believe that the paper would benefit from an error analysis on the outputs of the model. For example, is the model after the intervention in the activation functions equally prone to the same type of adversarial input against (e.g the same jailbreak instructions) as the baseline methods?
3. Table 1 lists the performance on MaliciousInstruct with text-only and using a blank image as input. If the problem of safety decay is attributed to the modality gap, I believe the authors should also try to evaluate the proposed approach by using a semantically relevant image (e.g, retrieve the most relevant image using CLIPScore from a pool of images). Additionally, while there is a multimodal evaluation using the MM-Harmful Bench, the performance gap between the baselines and the proposed method is the smallest in this case.
4. The number of layers that are affected by the modification of the hidden states could be also further explored. The authors followed a concurrent work regarding the value of this hyperparameter, however, considering that the proposed method is relatively cheap to run, they could have provided additional ablation studies by using the full, the cumulative sum, or a subset of the layers of the model. Furthermore, this particular hyperparameter depends on the model being used and is highly likely that for different models one could get different configurations. It would also be beneficial to include an additional model to further corroborate the claims of the paper (e.g, Qwen, IDEFICS-2, etc).
Comments, suggestions and typos: Please refer to weakness.
Overall assessment scores: 2