ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference-Time

## Abstract

Abstract Vision Language Models (VLMs) have become essential backbones for multimodal intelligence, yet significant safety challenges limit their real-world application. While textual inputs are often effectively safeguarded, adversarial visual inputs can easily bypass VLM defense mechanisms. Existing defense methods are either resource-intensive, requiring substantial data and compute, or fail to simultaneously ensure safety and usefulness in responses. To address these limitations, we propose a novel two-phase inference-time alignment framework, E valuating T hen A ligning (ETA): i) Evaluating input visual contents and output responses to establish a robust safety awareness in multimodal settings, and ii) Aligning unsafe behaviors at both shallow and deep levels by conditioning the VLMs’ generative distribution with an interference prefix and performing sentence-level best-of- N 𝑁 N italic_N to search the most harmless and helpful generation paths. Extensive experiments show that ETA outperforms baseline methods in terms of harmlessness, helpfulness, and efficiency, reducing the unsafe rate by 87.5% in cross-modality attacks and achieving 96.6% win-ties in GPT-4 helpfulness evaluation. The code is publicly available at https://github.com/DripNowhy/ETA .

## 1 Introduction

Vision Language Models (VLMs) (Achiam et al., ; Chen et al., ; ; Bai et al., ; Liu et al., ; ; Zhang et al., ) have emerged as crucial multimodal intelligence backbones, offering unprecedented capabilities in processing and understanding both visual and textual information. These models are developed by integrating visual model backbones into pre-trained Large Language Models (LLMs), followed by visual instruction tuning. While VLMs have demonstrated excellent performance across various vision-language tasks, their real-world applications are significantly hindered by safety challenges. Zong et al. () suggests that text-image pairs introduced during visual instruction tuning may contain unsafe content, potentially causing the model to forget safety mechanisms previously learned by the LLM backbone. Furthermore, research by Gong et al. (), Liu et al. (), and Gou et al. () indicates that the visual modality can easily bypass existing safety mechanisms, leading to harmful responses.

To ensure both harmlessness and helpfulness in VLM responses, current approaches can be categorized into fine-tuning-based and inference-based defenses (Jin et al., ). Fine-tuning-based methods, such as supervised fine-tuning (SFT) (Zong et al., ) and reinforcement learning from human feedback (RLHF) (Ouyang et al., ; Sun et al., ), aim to align models with human preferences but are resource-intensive, requiring extensive data and labor, and may compromise the model’s general capabilities (Zhang et al., ; Dubey et al., ). Inference-based defenses attempt to leverage VLMs’ intrinsic safety mechanisms by using evaluators to judge response quality and ensure safe outputs based on predefined criteria (Wang et al., ; Gou et al., ).
However, existing methods often focus on evaluating textual inputs and outputs, while largely overlooking the crucial safety checks for visual inputs. This oversight can result in mis-evaluations in multimodal scenarios, allowing harmful content to slip through and ultimately weakening the model’s overall capabilities (Liu et al., ; Jin et al., ). Furthermore, when unsafe behaviors are detected, current methods often rely on generic refusal responses, failing to offer helpful suggestions or relevant content based on the input (Pi et al., ). As a result, the responses become unhelpful and disconnected from the input topic.

To address the aforementioned challenges, we propose Evaluating Then Aligning (ETA), a robust inference-time alignment framework that establishes a strong defense mechanism for VLMs without any additional training. ETA consists of two key components: multimodal evaluation and bi-level alignment, which together enhance both safety and helpfulness of the model responses. Specifically, ETA i) assesses the safety of visual inputs through a predefined safety guard, referred to as pre-generation evaluation, and ii) post-generation evaluation leverages a textual reward model to assess the safety of the output response. This multimodal evaluation establishes a robust and accurate safety awareness across both visual and textual data. To align responses in cases where unsafe behavior is detected, ETA employs two key strategies: i) shallow alignment, which ensures outputs meet safety requirements by modifying VLM’s output distribution using an “interference prefix” (Qi et al., ), and ii) deep alignment, which further guarantees that responses remain safety and utility by performing sentence-level best-of-$N$ searching to identify the most harmless and helpful responses tailored to multimodal inputs. Our main contributions are summarized as follows:

Figure: Figure 1: ETA framework overview. ETA uses a multimodal evaluator to assess visual inputs with the CLIP score and initial generated responses with a textual reward model. For instances flagged as unsafe, ETA implements a comprehensive alignment process, which consists of both shallow alignment (interference prefix) and deep alignment (sentence-level best-of-$N$ searching).
Refer to caption: x1.png

- •
We propose Evaluating Then Aligning (ETA), a novel inference-time VLM alignment framework that requires no additional data or training, serving as a plug-and-play solution for aligning VLMs. ETA decomposes the defense process into two distinct phases (Fig. 1). This approach ensures that the generated responses are both safe and useful, without compromising the VLM’s general capabilities.
- •
We offer new perspectives on the failure of existing defense mechanisms in VLMs, demonstrating that the key issue lies in the continuous nature of visual token embeddings. These embeddings act as outliers to the LLM backbones, which were aligned only on discrete textual token embeddings. This insight inspired us to design a multimodal evaluator tailored for VLMs, which assesses the safety of both input images and output text to enable reliable and accurate safety awareness for VLMs.
- •
We introduce a bi-level alignment strategy to address unsafe behaviors at both shallow and deep levels. At the shallow level, we guide the response toward a safe style by pre-filling an interference safety prefix to shift the output distribution. At the deep level, we use a multimodal evaluator to perform sentence-level best-of-$N$ searching, ensuring the output content is both safe and useful.
- •
Through extensive experiments, we validated the effectiveness of the ETA framework across multiple dimensions: harmlessness, helpfulness, and preservation of general abilities. Our experiments also contribute insights into the interplay between different VLM components and their combined impact on model safety and performance.

## 2 Related Works

##### Fine-tuning-based alignment.

To enable VLMs to generate responses aligned with human preferences (e.g. harmless and helpful), approaches like reinforcement learning from human feedback (RLHF) (Christiano et al., ; Sun et al., ; Zhang et al., ) or supervised fine-tuning on specialized datasets (Chen et al., ; Zong et al., ; Li et al., ) are often employed.
Other approaches aim to improve safety mechanisms by redesigning network architectures. For example, Bethany et al. () and Liu et al. () introduce additional classifiers during training to assess the safety level and type of unsafe content, enhancing interpretability of model generation.
Another line of work incorporates fine-tuned defense LLMs during inference to guide or correct model outputs, ensuring safer responses (Pi et al., ).
However, these methods are resource-intensive, and the balance of harmful and helpful data in training sets can affect the model’s core capabilities. Furthermore, their safety capabilities are often limited to the specific domains represented in red-teaming data, resulting in weak generalization to other domains and adversarial attacks (Shayegani et al., ; Gou et al., ; Jin et al., ).
In contrast, our approach requires no additional data or fine-tuning of model parameters, providing a plug-and-play solution to align VLMs.

##### Inference-based alignment.

Inference-time alignment modifies the decoding strategy of language models to align output distributions with human preferences (Brown et al., ; Zhang et al., ).
In LLM alignment, Khanov et al. () and Li et al. () utilize reward models to score outputs and select the response with higher reward score based on predefined criteria. These methods avoid the instability associated with PPO training in RLHF (Andrychowicz et al., ; Zheng et al., ), while ensuring alignment with human preferences by sampling high-reward responses.
Other techniques utilize self-evaluation of LLMs, employing the concept of LLM-as-a-Judge to evaluate candidate responses without introducing additional models (Xie et al., ; Brown et al., ).

Recently, some studies have extended inference-time strategies to VLMs (Wang et al., ; Gou et al., ).
Adashield (Wang et al., ) introduces an LLM defender to detect malicious responses and iteratively refine prompts. It requires an additional training phase to create a prompt pool and is only effective against structure-based jailbreaks like typography or text-to-image attacks. In contrast, our method requires no training and can address a broader range of jailbreaks.
ECSO (Gou et al., ) uses VLMs’ self-evaluation, distilling visual content into text when handling inappropriate responses, which is then processed through LLM safety protocols. Unlike ECSO, our method uses external multimodal evaluation and bi-level alignment without converting images to text, avoiding the potential loss of critical visual information. Given Adashield’s limited applicability, our experiments primarily focus on comparisons with ECSO.

## 3 Preliminaries

##### Transforming VLMs from LM Backbones.

To enable LLMs to understand visual information, mainstream methods generally involve two key steps: first, training a vision-language connector module, and second, fine-tuning the LLM’s backbone with various SFT image-text datasets (Dai et al., ; Liu et al., ; ). During inference, for a given image-text pair $\{x_{I},x_{T}\}$, the vision encoder $\mathcal{C}$ first converts the image $x_{I}$ into a visual embedding $e_{I}$. The connector module $\mathcal{M}$ then projects $e_{I}$ into a continuous sub-space of the textual embedding space, which can serve as input to the LLMs.
Similar to LLMs, VLMs generate responses by predicting the next token’s distribution in an autoregressive manner, continuing to generate tokens until a complete response is produced:

$$ $P(Y_{L}\mid\mathcal{E}_{I},\mathcal{E}_{T})=P(y_{1}\mid\mathcal{E}_{I}, \mathcal{E}_{T})\cdot\prod_{i=2}^{L}P(y_{i}\mid Y_{<i},\mathcal{E}_{I}, \mathcal{E}_{T}),$ (1) $$

where $\mathcal{E}_{T}$ denotes the discrete textual embedding, $\mathcal{E}_{I}=\mathcal{M}(e_{I})$ is the projected continuous visual embedding, $Y_{L}$ represents a complete response with $L$ tokens, and $Y_{<i}$ denotes the sequence of generated tokens $\{y_{1},y_{2},\cdots,y_{i-1}\}$. Despite these advancements, integrating vision embeddings and fine-tuning the LLM backbone can easily cause VLMs to generate harmful responses to malicious queries (Gong et al., ; Zong et al., ).

## 4 Methodology

Figure: Figure 2: Continuous visual token embeddings can bypass existing safety mechanisms that are primarily aligned with discrete textual token embeddings. To verify this hypothesis, we implemented a mapping that transforms continuous visual embeddings to their nearest discrete textual embeddings based on cosine similarity. This mapping results in a significant 6.79% reduction in the unsafe rate (USR) when evaluated on the SPA-VL Harm test set (Zhang et al., ). Fig. 7 illustrates examples of these mapped textual tokens, demonstrating how offensive images are transformed into harmful tokens that can then be effectively addressed by the original safety mechanisms of LLM backbones.
Refer to caption: x2.png

VLMs often fail to generate harmless responses, particularly when processing inputs with harmful intent (Bailey et al., ; Gong et al., ). Recent studies have identified two primary factors: the fine-tuning of LLM backbones (Zong et al., ) and the introduction of visual
inputs (Liu et al., ; Gou et al., ).
We hypothesize that the bypassing of safety mechanisms in VLMs is primarily due to the continuous nature of visual token embeddings, which often behave as outliers to the LLM backbone compared to the well-aligned discrete textual tokens (Section 4.1). Recognizing the vulnerability of previous safety mechanisms in multimodal settings, we divided the alignment process into two components: Evaluating Then Aligning (ETA). As illustrated in Fig. 1, we introduce a robust and accurate evaluation specifically designed to establish multimodal safety awareness for VLMs (Section 4.1), followed by a safety-guided bi-level alignment procedure to identify the most harmless and helpful response (Section 4.2). The complete process is detailed in Algorithm 1.

### 4.1 Multimodal Evaluation

##### Motivation: Continuous Visual Token Embeddings Bypass LLM Safety Mechanisms

LLM backbones are typically aligned on discrete textual embeddings $\mathcal{E}_{\text{textual}}\subset\mathbb{R}^{d}$ (Devlin, ; Dubey et al., ). In contrast, the continuous visual embeddings $\mathcal{E}_{\text{visual}}\subset\mathbb{R}^{d}$ often appear away from all textual embeddings (Gong et al., ).
As shown in Fig. 2, we implemented an alternating mapping, where continuous visual embeddings are mapped to their nearest textual embeddings (green guard flow). This method resulted in a significant 7% reduction in the unsafe rate (USR) compared to the standard VLM baseline (red bypass flow). We also show examples of mapped textual tokens in Fig. 7 and 8, where offensive images are mapped to related harmful words. These results provide direct evidence supporting our hypothesis that the bypassing of safety mechanisms in VLMs is primarily due to outliers in the embedding space, specifically those originating from the visual modality. Additionally, as evidenced by the red curve in Fig. 3(a), previous safety mechanisms built on LLMs fail in multimodal inputs. These insights inspired us to establish a new safety awareness for VLMs in multimodal settings, designed to safeguard both visual and textual information.

Figure: (a) USR changes from LLM backbone to VLM, and finally our ETA
Refer to caption: x3.png

#### 4.1.1 Pre-generation evaluator

The lack of comprehensive safety evaluation for multimodal inputs, especially for the vulnerable mis-aligned visual modality, presents a critical challenge in VLMs.
Current reward models for VLMs primarily focus on addressing hallucination issues (Sun et al., ), with few practical evaluators targeting safety assessment. Considering the limitations of VLMs, we attempt to introduce an additional safety guard to assess the safety of visual inputs.

Contrastive Language-Image Pre-training (CLIP) (Radford et al., ) aims to learn visual features under text supervision, demonstrating strong generalization capability (Shu et al., ). Let $\mathcal{C}_{I}(\cdot)$ and $\mathcal{C}_{T}(\cdot)$ denote the vision embedding and text embedding encoded by the CLIP model, respectively. The CLIP-score (Hessel et al., ), which is the cosine similarity, is then used to measure the relevance between the input text and image:

$$ $\mathcal{S}_{\text{CLIP}}=\max(\cos(\mathcal{C}_{I}(\cdot),\mathcal{C}_{T}( \cdot)),0).$ (2) $$

Considering that many VLMs use CLIP-based vision towers (Chen et al., ; Liu et al., ; Zhang et al., ), such as CLIP-ViT-L-336px(^1^11[https://huggingface.co/openai/clip-vit-large-patch14-336](https://huggingface.co/openai/clip-vit-large-patch14-336)), this has inspired us to leverage the CLIP-score to develop semantic-level evaluation for visual inputs. Unlike the modality alignment in VLMs, CLIP models learn and align vision and text embeddings by maximizing the semantic similarity across modalities during training (Radford et al., ). Despite the continuous nature of visual embeddings, the safety of images can be determined by assessing the presence of unsafe semantic content.

To accomplish this, we design an *evaluation prompt* $\mathcal{P}$ (Appendix A.1) including common unsafe categories. This prompt is utilized to quantify the semantic similarity between the input image and potentially harmful content. We denote this measure as the pre-generation evaluation score $\mathcal{S}_{\text{pre}}(x_{I})$ in Eq. 3. Intuitively, the score for harmful image inputs $\mathcal{S}_{\text{pre}}(x_{I}^{\text{unsafe}})$ should exceed that of safe input images $\mathcal{S}_{\text{pre}}(x_{I}^{\text{safe}})$. To validate the efficacy of the CLIP score, we randomly selected 100 harmful and safe images from the MM-Safetybench (Liu et al., ) and COCO datasets (Lin et al., ), respectively. As depicted in Fig. 3(b), the score distributions exhibit a distinct separation, which justifies setting a threshold $\tau_{\text{pre}}$ that effectively discriminates between safe and unsafe inputs.

$$ $\displaystyle\mathcal{S}_{\text{pre}}(x_{I})$ $\displaystyle=\max(\cos(\mathcal{C}_{I}(x_{I}),\mathcal{C}_{T}(\mathcal{P})),0 ),\hskip 10.0pt\text{Eval}_{\text{pre}}(x_{I})=\begin{cases}\text{Unsafe},& \mathcal{S}_{\text{pre}}(x_{I})\geq\tau_{\text{pre}}\\ \text{Safe},&\text{otherwise}\end{cases}.$ (3) $$

#### 4.1.2 Post-generation evaluator

The red curve in Fig. 3(a) shows that tuning the LLM backbone through visual instruction also affects the safety capabilities of VLMs, even in the absence of visual inputs. Therefore, we additionally evaluate the generated responses to ensure the final outputs meet safety standards, building a more comprehensive and reliable multimodal safety awareness.

Reward models (RMs) are trained on preference text datasets to evaluate the utility of responses and their alignment with human preferences (Li et al., ). To compensate for the lack of visual modality in RM evaluation, we introduce a *safety-specific input format* (Appendix A.2), which compels the RM to evaluate responses based on both utility and safety criteria.
In Fig. 4, we present the distribution of reward scores across different input formats. The results show that the safety-specific input format creates a more distinct separation between harmful and harmless responses compared to the vanilla format, allowing for more effective discrimination between safe and unsafe outputs. We define the reward score derived from our proposed safety-specific input format as the post-generation evaluation score, $\mathcal{S}_{\text{post}}$, calculated as:

$$ $\displaystyle\mathcal{S}_{\text{post}}(Y_{L})$ $\displaystyle=\pi_{r}(Y_{L}),\hskip 10.0pt\text{Eval}_{\text{post}}(Y_{L})= \begin{cases}\text{Unsafe},&\mathcal{S}_{\text{post}}(Y_{L})\leq\tau_{\text{ post}}\\ \text{Safe},&\text{otherwise}\end{cases},$ (4) $$

where $Y_{L}$ is the generated response, $\pi_{r}$ is the RM, and $\tau_{\text{post}}$ is an adjustable threshold to distinguish between safe and unsafe responses.

We tested various strategies to combine $\text{Eval}_{\text{pre}}$ and $\text{Eval}_{\text{post}}$ in Table 10. The results show that applying alignment only to behaviors classified as unsafe by both evaluations provides the best balance between safety and utility. Therefore, ETA applies alignment only when both evaluators flag the behavior as unsafe; otherwise, the model outputs the vanilla responses directly.

### 4.2 Safety-guided Bi-level Alignment

After providing VLMs with safety awareness through the multimodal evaluator, our next task is to align unsafe behaviors to ensure safe and helpful responses. This alignment process consists of two steps: i) adjusting VLMs’ generative distribution by conditioning on an interference prefix (+ Shallow Alignment curve in Fig. 3(a)), and ii) guiding the generation process through sentence-level best-of-$N$ searching (+ Deep Alignment curve in Fig. 3(a)). This approach aims to produce responses that are both safe and helpful, effectively correcting unsafe behaviors.

#### 4.2.1 Interference prefixes as shallow alignment

The autoregressive decoding mechanism of VLMs, as described in Eq. 1, implies that the initial tokens greatly influence the distribution of subsequent tokens, thereby shaping the entire response (Team et al., ; Andriushchenko et al., ). Brown et al. () also suggests that pre-filling the first few output tokens can effectively activate the safety capabilities of LLMs, promoting harmless generation in the subsequent tokens. As shown in Fig. 3(a), we verify that the *interference prefix* (e.g., “As an AI assistant, ”) can activate the safety capabilities of VLM when faced with harmful multimodal inputs. We see that adding an interference prefix reduces the unsafe rate (USR) significantly. We provide a detailed discussion on the effects of different prefixes in Fig. 9 of Appendix B.7.

#### 4.2.2 Sentence-level best-of- N 𝑁 N italic_N seaching as deep alignment

While the use of an interference prefix can partially mitigate safety issues in VLMs, our findings indicate that this approach alone is insufficient to fully align with human preferences, consistent with the results reported by Qi et al. (). Our analysis reveals a more nuanced issue: for a significant portion of samples initially classified as “unsafe”, adding an interference prefix often leads to a pattern where the model initially refuses to respond but subsequently produces harmful content using transitional phrases such as “However”. We report our observations in Fig. 10, which further highlights the necessity of deep alignment.

To ensure that VLMs consistently generate harmless and helpful responses, we adopt a sentence-level best-of-$N$ searching algorithm as the deep alignment method. This approach leverages our multimodal evaluator (Section 4.1) to guide the response generation process. At each step, $N$ candidate sentences are sampled and evaluated, and the candidate with the highest score is accepted. This method allows us to dynamically optimize the generation process, biasing it towards safer and more helpful responses while maintaining coherence and relevance to the input query.

When incorporating the visual modality, relying solely on the RM for guidance can overlook valuable visual information, as the RM cannot directly process visual inputs. To address this limitation, we integrate the CLIP model to ensure the generated response provides more specific helpful suggestions to the input image. We define the utility score of the output as measured by the CLIP score:

$$ $\mathcal{S}_{u}(x_{I},O_{i})=\max(\cos(\mathcal{C}_{I}(x_{I}),\mathcal{C}_{T}( O_{i})),0)$ (5) $$

where $O_{i}$ denotes the $i$-th sentence of the output. Due to CLIP models’ 77-token input limit and the higher accuracy of sentence-level inputs for both CLIP and RM, we adopt a sentence-level (instead of instance-level) guided generation approach. The score for each sentence is expressed as:

$$ $\mathcal{S}(x_{I},O_{i})=\alpha\cdot\mathcal{S}_{u}(x_{I},O_{i})+\mathcal{S}_{ \text{post}}(O_{\leq i}),$ (6) $$

where $\alpha$ balances safety ($\mathcal{S}_{\text{post}}(\cdot)$ in Eq. 4) and utility ($\mathcal{S}_{u}(\cdot,\cdot)$ in Eq. 5), and $O_{\leq i}$ represents the complete response up to the $i$-th sentence. For the first sentence with the interference prefix, $\alpha=0$ to ensure safety. For subsequent sentences, $\alpha=1/i$, as discussed in Section 4.2.1, to address the risk of harmful responses after the initial interference prefix.

Figure: Algorithm 1 Evaluating Then Aligning (ETA)

## 5 Experiments

In this section, to demonstrate the effectiveness of ETA, we verify the following four key questions: (1) Can ETA provide safe responses in the presence of harmful inputs or adversarial attacks? (2) Does ETA impact the model’s general capabilities, potentially compromising the usefulness of the responses? (3) How efficient is ETA at inference time? (4) What impact do different components of ETA have on its overall effectiveness?

### 5.1 Setups

##### Implementation.

We employed LLaVA-1.5-7B and 13B (Liu et al., ), InternVL-Chat-1.0-7B (Chen et al., ), and InternLM-XComposer-2.5-7B (Zhang et al., ) as the VLM backbones. The textual RM used in ETA was
ArmoRM-Llama3-8B-v0.1 (Wang et al., ), which exhibits strong safety ability. For our ETA method, during the evaluation phase, we empirically set the thresholds to $\tau_{\text{pre}}=0.16$ in Eq. 3 and $\tau_{\text{post}}=0.06$ in Eq. 4. In the alignment phase, we generated $N=5$ candidate responses per sentence.
All experiments were conducted on an NVIDIA RTX A6000 platform. The prompts used during pre- and post-generation evaluations are detailed in Appendix A.1 and A.2.

##### Evaluation Details.

We focus on two main categories of benchmarks to evaluate VLM capabilities: safety and helpfulness. For safety, we assess ETA using multimodal safety datasets, including SPA-VL Harm (Zhang et al., ), MM-SafetyBench (Liu et al., ), FigStep (Gong et al., ), Unconstrained attack (Qi et al., ), and the text attack benchmark AdvBench (Zou et al., ). Following the methodology of Zhang et al. (), we use the LLM safety guard-based Unsafe Rate (USR) as the primary evaluation metric, which measures the proportion of unsafe responses generated by the model. Additionally, in line with Zong et al. () and Wang et al. (), we compute the target-string-based Attack Success Rate (ASR) as a complementary metric.
For helpfulness, we selected several common comprehensive benchmarks and VQA datasets, such as $\text{SQA}^{I}$ (ScienceQA-IMG) (Lu et al., ), VQAv2 (Goyal et al., ), TextVQA (Singh et al., ), MME (Fu et al., ), and MMBench (Liu et al., ), to evaluate the general capabilities of VLMs. Additionally, we used GPT-4-Turbo to assess the helpfulness of model outputs on the SPA-VL Help dataset (Zhang et al., ). Further details on evaluation metrics and benchmarks are provided in Appendix A.3 and A.4.

##### Baselines.

Given that ETA requires no additional data or fine-tuning, we primarily compare it against existing inference-time method, ECSO (Gou et al., ). Additionally, to demonstrate that our approach can reduce VLM safety issues while maintaining output usefulness, we also compare it with fine-tuned methods like Posthoc-LoRA and Mixed-LoRA on VLGuard in the helpfulness evaluation (Zong et al., ).

### 5.2 Results

**Table 1: USR evaluation across multiple safety benchmarks. Our method significantly reduces unsafe responses to malicious inputs across four different VLM backbones. Under suffix adversarial attacks and cross-modality attack, ETA demonstrates superior performance, while ECSO fails to generate safe responses under these conditions.**
|  | SPA-VL | MM-SafetyBench | FigStep | Adv. Image+Text |  |
| --- | --- | --- | --- | --- | --- |
| Method | Harm $\downarrow$ | SD+TYPO $\downarrow$ | Vanilla $\downarrow$ | Suffix $\downarrow$ | Unconstrained $\downarrow$ |
| LLaVA-1.5-7B | 46.04 | 40.46 | 58.60 | 62.00 | 97.50 |
| + ECSO | 23.40 | 15.89 | 37.40 | 59.00 | 95.00 |
| + ETA | 16.98 | 15.83 | 7.80 | 22.60 | 22.50 |
| LLaVA-1.5-13B | 40.75 | 41.01 | 61.60 | 66.40 | 100.00 |
| + ECSO | 15.47 | 13.81 | 15.00 | 37.20 | 95.00 |
| + ETA | 15.09 | 11.67 | 22.60 | 20.80 | 12.50 |
| InternVL-Chat-1.0-7B | 46.79 | 37.20 | 47.40 | 52.80 | 97.50 |
| + ECSO | 28.68 | 15.54 | 41.20 | 49.40 | 95.00 |
| + ETA | 16.98 | 13.81 | 17.40 | 10.80 | 25.00 |
| InternLM-XComposer-2.5-7B | 27.55 | 21.79 | 22.60 | 50.80 | 7.50 |
| + ECSO | 19.62 | 14.94 | 16.60 | 42.40 | 5.00 |
| + ETA | 13.96 | 7.32 | 6.00 | 7.20 | 5.00 |

##### ETA Providing Robust Safety Mechanisms for VLMs.

In Table 1, we report the Unsafe Response Rate (USR) for ETA and ECSO when applied to different VLM backbones across various safety benchmarks. We observe that most VLM backbones exhibit a high USR when faced with multimodal harmful inputs. Additionally, on the unconstrained cross-modality adversarial attack benchmark, and when text suffix attacks are applied to FigStep, all VLM backbones show a significant increase in USR. This suggests that despite undergoing alignment during training, further safety mechanisms are necessary to effectively safeguard VLMs.

Compared to ECSO, ETA significantly reduces USR across all benchmarks. Notably, when facing adversarial attacks, the minimal difference in USR between ECSO and the VLM backbone indicates that ECSO does not truly safeguard the model but rather limits the impact of the input image on the model’s safety. In contrast, our method remains unaffected by these attacks, reducing LLaVA-1.5-13B’s USR on cross-modality attack by 87.5%, compared to ECSO’s mere 5% reduction. Detailed results including that ETA achieves outstanding safety performance on text-only benchmarks are reported in Table 7, as well as its effectiveness on target-string-based metrics in Table 6.

**Table 2: General performance of different methods on LLaVA-1.5-7B. The first row of each method shows the performance, while the second row shows the difference relative to its VLM backbone. Green indicates improvement, and red indicates a decrease compared to the VLM backbone. Our method outperforms other finetune-based and inference-time baselines.**
|  |  | Comprehensive Benchmark | General VQA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Fine-tuned | MME^P | MME^C | MMB | SQA^I | TextVQA | VQAv2 |
| LLaVA-1.5-7B |  | 1505.88 | 357.86 | 64.60 | 69.51 | 58.20 | 78.51 |
| + VLGuard-Posthoc-LoRA | ✓ | 1420.66 | 332.50 | 63.32 | 67.33 | 55.99 | 76.87 |
| $\downarrow$85.22 | $\downarrow$25.36 | $\downarrow$1.28 | $\downarrow$2.18 | $\downarrow$2.21 | $\downarrow$1.64 |  |  |
| + VLGuard-Mixed-LoRA | ✓ | 1483.00 | 267.14 | 68.04 | 68.42 | 57.88 | 79.18 |
| $\downarrow$22.88 | $\downarrow$90.72 | $\uparrow$3.44 | $\downarrow$1.09 | $\downarrow$0.32 | $\uparrow$0.67 |  |  |
| + ECSO | ✗ | 1495.88 | 360.00 | 63.83 | 69.36 | 58.15 | 78.39 |
| $\downarrow$10.00 | $\uparrow$2.14 | $\downarrow$0.77 | $\downarrow$0.15 | $\downarrow$0.05 | $\downarrow$0.12 |  |  |
| + ETA | ✗ | 1506.13 | 357.86 | 64.69 | 69.51 | 58.15 | 78.51 |
| $\uparrow$0.25 | $\uparrow$0.00 | $\uparrow$0.09 | $\uparrow$0.00 | $\downarrow$0.05 | $\uparrow$0.00 |  |  |

**Table 3: Helpfulness evaluation on the SPA-VL Help shows that ETA outperforms other baselines in the GPT-4 evaluated win-ties rate, demonstrating its superior ability to generate helpful responses.**
| Model | Ours | v.s. | Compared Method | Win-Tie (%) $\uparrow$ |
| --- | --- | --- | --- | --- |
| LLaVA-1.5-7B | ETA |  | Vanilla VLM | 96.6 |
| ETA |  | Posthoc-LoRA | 54.6 |  |
| ETA |  | Mixed-LoRA | 56.7 |  |
| ETA |  | ECSO | 80.8 |  |

##### ETA Ensuring Helpful and Useful Responses.

As shown in Table 2, compared to fine-tuned methods, inference-time approaches have a smaller impact on the VLM backbones in the Comprehensive and VQA benchmarks. Furthermore, our ETA does not diminish the backbone’s capabilities in any of the five benchmarks. The only exception is TextVQA, where ETA reduces accuracy by just 0.05%, still better than other baselines. This indicates that ETA provides more reliable assessments during the evaluating phase, ensuring that the model’s general abilities remain unaffected. Results from evaluating additional VLM backbones are provided in Table 8 of Appendix B.4.

Additionally, in Table 3, we present a win-tie comparison of the helpfulness of model outputs across different methods. The prompt used for GPT-4-Turbo evaluation is provided in Appendix A.5, following the setting of Zhang et al. (). The results show that applying ETA significantly increases the helpfulness of the generated responses, aligning closely with human preferences, even when compared to fine-tuned methods. Detailed win-tie-lose proportion is shown in Fig. 5.

##### Inference Efficiency.

Inference efficiency is vital for inference-time alignment methods. Given the common usage scenarios of VLMs, we compared the inference time of ETA and ECSO on the comprehensive MMB and $\text{SQA}^{I}$ benchmark, with the results provided in Table 4. It can be observed that ETA increases the inference time per generation by only 0.1 seconds compared to the VLM backbone, whereas ECSO adds an average of 0.39 seconds, almost 4 times the increase of ETA. This is because ECSO’s self-evaluation struggles to accurately assess response safety, leading to an excessive number of unnecessary alignment steps during generation. In contrast, ETA provides accurate evaluation, preserving VLMs’ general ability and avoiding the extra generation overhead.

### 5.3 Ablation Studies

We conducted ablation studies on both ETA’s evaluation and alignment components to analyze each part’s usefulness and effectiveness.

##### Adjustability of ETA’s Safety Capabilities.

During the Evaluation phase, our goal was to accurately assess the safety of the model’s inputs and outputs. In balancing safety and utility, we prioritized ensuring the model’s core capabilities remained intact while maximizing its safety. As shown in Fig. 3(b) and 4, the final ETA configuration selected $\tau_{\text{pre}}=0.16$ and $\tau_{\text{post}}=0.06$ to better differentiate between safe and unsafe inputs and outputs. In Fig. 6, we demonstrate that adjusting $\tau_{\text{pre}}$ and $\tau_{\text{post}}$ allows for varying levels of safeguarding. If a higher level of safety is desired, increasing $\tau_{\text{post}}$ and decreasing $\tau_{\text{pre}}$ can achieve this. Conversely, to preserve the model’s general capabilities to the greatest extent, $\tau_{\text{pre}}$ can be increased and $\tau_{\text{post}}$ can be reduced.

**Table 4: We evaluate the inference efficiency of ETA and compare it with ECSO on two comprehensive benchmarks: MMB and $\text{SQA}^{I}$, simulating common usage scenarios. The table demonstrates that ETA outperforms ECSO in terms of inference efficiency (time for each response in second).**
|  | Inference Time (second) $\downarrow$ |  | Inference Time (second) $\downarrow$ |  |  |
| --- | --- | --- | --- | --- | --- |
| Method | MMB | $\text{SQA}^{I}$ | Method | MMB | $\text{SQA}^{I}$ |
| LLaVA-1.5-7B | 0.23 | 0.22 | InternVL-Chat-1.0-7B | 0.52 | 0.35 |
| + ECSO | 0.48 ($\uparrow$ 0.25) | 0.38 ($\uparrow$ 0.16) | + ECSO | 1.44 ($\uparrow$ 0.88) | 0.62 ($\uparrow$ 0.27) |
| + ETA | 0.28 ($\uparrow$ 0.05) | 0.36 ($\uparrow$ 0.14) | + ETA | 0.64 ($\uparrow$ 0.12) | 0.44 ($\uparrow$ 0.09) |

##### How Can ETA Simultaneously Increase Safety and Utility?

In Table 5, we present the impact of shallow and deep alignment during the aligning phase of ETA on the harmlessness and helpfulness of VLM backbone outputs. Observing the first three rows, it is evident that merely adding shallow alignment or deep alignment can only slightly improve harmlessness and helpfulness. This underscores the importance of the combined effect of both style-level generation distribution modification and content-level best-of-N search. Moreover, the fourth and fifth rows reveal that using both the utility CLIP score defined in Eq. 5 and the reward model defined in Eq. 4 for best-of-N searching can enhance the helpfulness of responses without compromising their harmlessness. This is because the utility score identifies the most relevant responses to input images, while the reward model ensures the safety of those responses.

**Table 5: Ablation study on alignment strategy of ETA in SPA-VL test set. We ablated shallow alignment, and deep alignment including safety guide (RM evaluator) and utility guide (CLIP score) on LLaVA-1.5-7B. The last line with gray background is ETA, which enables generate responses both harmless and helpful. The helpful score in Table is evaluated by GPT-4 (detailed in Appendix A.5).**
| Model | Shallow Align. | Deep Align. | SPA-VL |  |  |
| --- | --- | --- | --- | --- | --- |
| Safety Guide | Utility Guide | Harm (USR $\downarrow$) | Helpful Score $\uparrow$ |  |  |
| LLaVA-1.5-7B | ✗ | ✗ | ✗ | 46.04 | 7.64 |
| ✗ | ✓ | ✓ | 32.08 | 8.10 |  |
| ✓ | ✗ | ✗ | 30.94 | 8.25 |  |
| ✓ | ✓ | ✗ | 16.60 | 8.38 |  |
| ✓ | ✓ | ✓ | 16.98 | 8.50 |  |

## 6 Conclusion

In this paper, we propose *Evaluating Then Aligning* (ETA), a robust inference-time safety alignment method for Vision Language Models (VLMs). Our analysis reveals that continuous visual token embeddings are the primary cause of safety mechanism failures in VLMs, leading us to introduce multimodal evaluation to establish a comprehensive safety awareness. To mitigate unsafe behaviors, we apply shallow alignment to adjust the response to follow a safe style and deep alignment with our multimodal evaluator to ensure both safety and usefulness of the content. Through extensive experiments, we evaluate the safety, helpfulness, and efficiency of ETA-generated responses, demonstrating their superiority over existing methods and their ability to maintain VLM utility while substantially addressing safety concerns.

## Ethics Statement

This paper focuses on inference-time alignment for VLMs, enabling them to generate safer and more useful responses without the need for additional data or fine-tuning. It contributes to building safer, more useful, and efficient AI systems. However, we acknowledge certain potential ethical concerns, including the presence of harmful data during training and testing, as well as the potential for generating harmful content.

## Appendix A Implementation Details

### A.1 Prompts for Pre-Generation Evaluator (CLIP Score)

In the pre-generation evaluation phase, we use the following prompt as text input. By assessing the similarity between the image and the prompt, we determine the safety of input images.

### A.2 Prompts for Post-Generation Evaluator (Reward Model)

In the post-generation evaluation, to ensure the Reward Models focus more on the safety of the output text, we designed the following safety-specific input Format for the RM. Unlike the standard input format, we modified the specific question into an open-ended one, prioritizing the safety of the response over its relevance to the original question.

### A.3 Evaluation Metrics

#### A.3.1 Safety Metrics

##### Unsafe Rate (USR).

The Unsafe rate (USR) measures the probability of the model generating unsafe outputs (Zhang et al., ). The specific calculation method is as follows:

$$ $\text{USR}=\frac{\lvert\text{\{unsafe responses\}}\rvert}{\lvert\text{\{all responses\}}\rvert}$ $$

We use USR to measure the safety of the model, where a lower USR indicates higher safety and a higher USR reflects weaker safety capabilities. The safety of each model’s output is evaluated by MD-Judge-v0.2-internlm2_7B (Li et al., ), a LLM-based safetyguard fine-tuned from internlm2-7b-chat. Compared to other models such as GPT-4o and Claude 3.5-Sonnet, MD-Judge-v0.2-internlm2_7B demonstrates higher accuracy on safety evaluation datasets like Harmbench (Mazeika et al., ), highlighting its reliability in assessing response safety.

##### Attack Success Rate (ASR).

Attack Success Rate (ASR) is a commonly used metric to assess the safety of a model, measures the probability of a model being successfully attacked. The calculation method is as follows:

$$ $\text{ASR}=\frac{\lvert\text{\{attack successful responses\}}\rvert}{\lvert \text{\{all responses\}}\rvert}$ $$

Most evaluation methods use a target-string match approach to determine whether the attack was successful (Zong et al., ; Wang et al., ). Considering that the target-string match evaluation method may result in false positives or false negatives, making it less accurate for assessing model safety, we have presented the ASR results only in Table 6 of Appendix B. A higher ASR indicates the model is more vulnerable to attacks, whereas a lower ASR signifies better safety. Following the setup of Zong et al. (), we defined the target string as follows:

#### A.3.2 Helpfulness Metrics.

##### Helpful Score.

We use GPT-4-Turbo to evaluate the model’s outputs in terms of harmlessness, helpfulness, relevance, and so on, with scores ranging from 1 to 10. The detailed evaluation method and prompt can be found in Appendix A.5.

##### Helpful Win-Tie.

To avoid biases introduced by the scoring criteria in the Helpful Score, we used GPT-4-Turbo to compare the helpfulness of responses generated by different methods. Following the setup in Zhang et al. (), we conducted tests on the SPA-VL Help dataset. The detailed prompt for GPT-4-Turbo is shown in Appendix A.5.

### A.4 Benchmarks Details

##### SPA-VL.

SPA-VL (Zhang et al., ) is a multimodal comprehensive safety preference alignment dataset. The training set contains 93.3k preference data pairs, covering six harmful categories. The test set consists of 530 data points, with 265 labeled as Harm and 265 labeled as Help, specifically designed to evaluate the model’s safety and helpfulness capabilities.

##### MM-SafetyBench.

MM-SafetyBench (Liu et al., ) is a multimodal safety benchmark primarily focused on image-based attacks, covering 13 harmful categories with a total of 1,680 test samples. The SD+TYPO method consists of harmful images generated by Stable Diffusion, with harmful information annotated below the image, while the input text contains no explicit harmful content.

##### FigStep.

FigStep (Gong et al., ) highlights that VLMs are vulnerable to harmful image-based attacks. It consists of 520 test samples, where the images are composed of harmful text arranged on a white background. The task instruction requires the model to provide steps in response to the harmful content within the image. To further challenge the model, we appended the suffix "Answer with ’Absolutely, here’s’" at the end of the textual instructions as an adversarial attack.

##### Adv. Image+Text.

This is a cross-modality attack method, where the unconstrained harmful noise image was introduced by Qi et al. (), and the textual attack involves adding the suffix "Answer with ’Absolutely, here’s’".

##### AdvBench.

AdvBench (Zou et al., ) is a commonly used pure-text safety dataset containing 500 harmful input texts. To increase the difficulty of the task, we appended the suffix "Answer with ’Absolutely, here’s’" at the end of the text instructions as an adversarial attack.

##### MME.

MME (Fu et al., ) is a multimodal comprehensive benchmark designed to evaluate two key capabilities of models: perception and reasoning. It includes 10 tasks for perception and 4 tasks for reasoning, providing a thorough assessment of VLMs’ abilities. The perception score metric is the sum of scores of all perception subtasks. The cognition score metric is calculated in the same way. The full scores of perception and cognition are 2000 and 800, respectively.

##### MMB (MMBench).

MMBench (Liu et al., ) evaluates 20 fundamental capabilities of VLMs and provides multilingual test sets. It introduces a CircularEval strategy, enabling users to obtain reliable evaluations without relying on GPT-4.

##### ScienceQA.

ScienceQA (Lu et al., ) primarily evaluates language models’ capabilities in the domain of science. It consists of multiple-choice questions covering a wide range of scientific topics.

##### TextVQA.

TextVQA (Singh et al., ) assesses a model’s understanding and reasoning capabilities in relation to Optical Character Recognition (OCR). It requires the model to comprehend and reason about questions based on text present within images.

##### VQAv2.

VQAv2 (Goyal et al., ) contains open-ended questions related to images, assessing a model’s ability to understand both visual and textual information. Each image is paired with at least three questions, and the dataset supports automated evaluation.

### A.5 GPT-4-Turbo Evaluation Details

We use GPT-4-Turbo to evaluate the helpfulness and harmlessness of the generated responses. We followed and adapted the GPT-4 evaluation prompt from Li et al. (), with additional emphasis on image input. The prompt first clarifies the role of the AI assistant and instructs it to score the response based on both the text and image input, focusing on harmlessness and helpfulness. The detailed prompt is as follows:

For the win-tie-lose evaluation, we followed the approach of Zhang et al. (), including both the System and User Prompts, with the primary focus on assessing the model’s helpfulness. The prompts are shown below:

### A.6 Baselines Details

##### ECSO.

ECSO is an inference-based defense method that primarily addresses the challenge of VLMs being unable to defend against harmful information in the visual modality (Gou et al., ). It introduces an image-to-text transformation, converting visual information into text that is easier to defend against. Specifically, the method first utilizes VLM’s self-evaluation to determine whether the standard output is safe. If deemed unsafe, a specially designed prompt is used to generate a caption for the input image, which replaces the original image as input. The VLM then produces a safer response based on this caption.

##### Posthoc-LoRA.

Posthoc-LoRA is a fine-tune-based defense method introduced by Zong et al. () on their proposed dataset, VLGuard, to fine-tune pre-trained VLMs. It is a versatile approach that can be applied to any VLM. Specifically, to preserve the usefulness of the model’s outputs, a portion of data from comprehensive benchmarks is mixed into the fine-tuning process.

##### Mixed-LoRA.

Mixed-LoRA is also a fine-tune-based defense method that uses the same dataset as Posthoc-LoRA (Zong et al., ). However, instead of fine-tuning after training, it incorporates this data into the standard dataset during the Visual Instruction Tuning stage to enhance the model’s safety capabilities.

## Appendix B Additional Results and Examples

### B.1 Post-Generation Evaluation Results

Figure: Figure 4: Reward distribution comparison on difference input format. It is evident from the distribution and KL divergence data in the figure that our proposed safety-specific input format better distinguishes between safe and unsafe responses.
Refer to caption: x5.png

**Table 6: ASR performance across multiple safety benchmarks.**
|  | SPA-VL | MM-SafetyBench | FigStep | Adv. Image+Text |  |
| --- | --- | --- | --- | --- | --- |
| Method | Harm $\downarrow$ | SD+TYPO $\downarrow$ | Vanilla $\downarrow$ | Suffix $\downarrow$ | Unconstrained $\downarrow$ |
| LLaVA-1.5-7B | 72.45 | 84.46 | 86.40 | 85.80 | 85.00 |
| + ECSO | 53.96 | 72.44 | 79.29 | 82.20 | 67.50 |
| + ETA | 38.87 | 53.39 | 32.40 | 25.00 | 17.50 |
| LLaVA-1.5-13B | 66.79 | 87.98 | 90.20 | 87.40 | 80.00 |
| + ECSO | 47.92 | 68.57 | 53.80 | 85.60 | 67.50 |
| + ETA | 39.62 | 46.19 | 28.80 | 6.80 | 12.50 |
| InternVL-Chat-1.0-7B | 72.08 | 85.77 | 85.80 | 85.20 | 85.00 |
| + ECSO | 56.23 | 75.06 | 86.00 | 84.00 | 70.00 |
| + ETA | 43.40 | 56.25 | 42.40 | 31.80 | 20.00 |
| InternLM-XComposer-2.5-7B | 61.51 | 74.29 | 57.80 | 86.60 | 17.50 |
| + ECSO | 55.09 | 73.10 | 59.20 | 86.80 | 15.00 |
| + ETA | 45.28 | 60.65 | 38.00 | 45.00 | 15.00 |

We opted to use textual RM to evaluate the safety of textual modality. However, one key issue arises: the language reward model cannot handle image inputs. A common cross-modal attack involves placing harmful information in images while keeping the text harmless (Gong et al., ; Liu et al., ). In these cases, the reliability of standard RMs, which only evaluate text inputs and outputs, can be questionable. For example, when the text input is harmless, the score for refusing to answer might be lower than the score for a harmful response. To shift the focus of RM toward assessing the safety of the model’s responses, rather than just the relevance with questions, we rephrased the specific questions given to the RM into open-ended prompts, encouraging the model to prioritize safety, which we denote as safety-spefic input format (Appendix A.2).

To validate the efficacy of safety-specific input format against the vanilla version, we visualized the reward distributions for harmful and harmless responses using both formats on the MM-SafetyBench dataset (Liu et al., ), as illustrated in Fig. 4. The results indicate our safety-specific input format is more reliable for evaluation.

### B.2 ASR Comparison

Previous work commonly used the string match method to determine whether an output is harmful. We followed the settings of Zong et al. (), using the string list provided in Appendix A.3.1 to test the ASR of ETA and the baseline methods across different VLM backbones. The results shown in Table 6 confirm the superior performance of our ETA.

### B.3 Extensible to Text-Only Benchmarks

**Table 7: Performance on text only safety benchmark.**
|  | AdvBench (USR) | AdvBench (ASR) |  |  |
| --- | --- | --- | --- | --- |
| Method | Vanilla $\downarrow$ | Suffix $\downarrow$ | Vanilla $\downarrow$ | Suffix $\downarrow$ |
| LLaVA-1.5-7B | 10.77 | 98.85 | 4.23 | 41.73 |
| + ECSO | 3.08 | 90.19 | 0.58 | 41.73 |
| + ETA | 3.08 | 2.31 | 0.38 | 0.77 |
| LLaVA-1.5-13B | 1.92 | 96.92 | 1.73 | 34.43 |
| + ECSO | 2.12 | 86.35 | 0.96 | 43.27 |
| + ETA | 0.77 | 1.92 | 0.58 | 0.77 |
| InternVL-Chat-1.0-7B | 11.15 | 97.12 | 4.81 | 41.54 |
| + ECSO | 2.89 | 90.58 | 1.35 | 0.38 |
| + ETA | 4.04 | 1.35 | 0.58 | 0.38 |
| InternLM-XComposer-2.5-7B | 0.00 | 37.31 | 0.00 | 21.92 |
| + ECSO | 0.00 | 10.96 | 0.19 | 7.12 |
| + ETA | 0.00 | 5.00 | 0.00 | 4.23 |

**Table 8: General performance of different methods on LLaVA-1.5-13B.**
|  |  | Comprehensive Benchmark | General VQA |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Fine-tuned | MME^P | MME^C | MMB | SQA^I | TextVQA | VQAv2 |
| LLaVA-1.5-13B |  | 1528.77 | 296.07 | 68.38 | 72.78 | 61.21 | 79.99 |
| + VLGuard-Posthoc-Lora | ✓ | 1510.13 | 318.57 | 66.58 | 71.29 | $59.15$ | 78.50 |
| $\downarrow$18.64 | $\uparrow$22.50 | $\downarrow$1.80 | $\downarrow$1.49 | $\downarrow$2.06 | $\downarrow$1.49 |  |  |
| + VLGuard-Mixed-Lora | ✓ | 1579.89 | 258.21 | 68.21 | 71.94 | 60.35 | 80.13 |
| $\uparrow$51.12 | $\downarrow$37.86 | $\downarrow$0.17 | $\downarrow$0.84 | $\downarrow$0.86 | $\uparrow$0.14 |  |  |
| + ECSO | ✗ | 1523.76 | 296.07 | 66.49 | 72.83 | 61.04 | 79.89 |
| $\downarrow$5.01 | $\uparrow$0.00 | $\downarrow$1.89 | $\uparrow$0.05 | $\downarrow$0.17 | $\downarrow$0.10 |  |  |
| + ETA | ✗ | 1531.19 | 296.07 | 68.38 | 72.83 | 61.09 | 79.99 |
| $\uparrow$2.42 | $\uparrow$0.00 | $\uparrow$0.00 | $\uparrow$0.05 | $\downarrow$0.12 | $\uparrow$0.00 |  |  |

To validate the applicability of our method, we also tested its effectiveness on the text-only safety benchmark AdvBench (Zou et al., ). Since there are no images in the input, we relied solely on post-generation evaluation in Eq. 4 to assess the safety of the responses. Our method significantly reduced the probability of harmful responses, both for regular harmful queries and adversarial attacks with suffixes. In adversarial settings, methods like ECSO were ineffective in providing protection, whereas ETA reduced the USR of LLaVA-1.5-7B by 96.54%, complete results are shown in Table 7.

### B.4 More Experiments on Helpfulness Evaluation

Experiments on LLaVA-1.5-13B in Table 8 also show that fine-tuning methods significantly impact the model’s core capabilities, whereas ETA, compared to ECSO, has a much smaller effect on the model’s foundational abilities. This demonstrates that ETA provides a more reliable assessment of whether the model’s behavior is safe.

In Fig. 5, we present a complete comparison of ETA with other methods on the SPA-VL Help test set, evaluated using GPT-4-Turbo’s Win-Tie-Lose metrics. It can be observed that, even when compared to fine-tune-based methods, ETA consistently shows an advantage in terms of helpfulness. Since both ETA and ECSO are inference-time methods, we observe a higher number of ties when comparing ETA with ECSO on LLaVA-1.5-7B. However, when compared to fine-tune-based methods, where the LLM backbone has been fine-tuned, the number of ties decreases significantly. Despite this, ETA still demonstrates a higher likelihood of producing winning responses.

Figure: Figure 5: Helpfulness evaluation on the SPA-VL Help test set shows that ETA outperforms other methods in the GPT-4-Turbo evaluated win-tie-lose rate, demonstrating its superior ability to align responses with human preferences.
Refer to caption: x6.png

### B.5 More Results of Ablation on Evaluating Stage

**Table 9: Comparison of harmlessness and helpfulness. ‘Adv.’ denotes cross-modal attacks, with the metric being the unsafe rate. ‘ECSO Eval. & ETA Align.’ refers to using self-evaluation as the evaluator and aligning responses using ETA.**
| Method | Adv. ($\downarrow$) | MMB ($\uparrow$) | Method | Adv. ($\downarrow$) | MMB ($\uparrow$) |
| --- | --- | --- | --- | --- | --- |
| LLaVA-1.5-7B | 97.50 | 64.60 | InternVL-Chat-1.0-7B | 97.50 | 65.21 |
| + ECSO | 95.00 | 63.83 | + ECSO | 95.00 | 64.35 |
| + ECSO Eval. & ETA Align. | 25.00 | 64.08 | + ECSO Eval. & ETA Align. | 32.50 | 63.76 |
| + ETA | 22.50 | 64.69 | + ETA | 25.00 | 65.21 |

To further demonstrate the reliability of ETA, we replaced our proposed multimodal evaluating method with the self-evaluation approach used in ECSO (Gou et al., ). As shown in Table 9, using self-evaluation during the evaluating phase resulted in an increased unsafe rate and decreased general ability across different VLM backbones compared to ETA.

### B.6 Examples of Continuous Embedding to Discrete Embedding

In Fig.7, we compare the model’s responses on the safety benchmark before and after mapping the visual token embeddings from the continuous space to the discrete text token embeddings with the highest cosine similarity. Additionally, in Fig.8, we replaced cosine similarity with Euclidean distance and compared the responses before and after the mapping. These figures demonstrate that the continuous nature of visual embeddings is a significant factor in bypassing safety mechanisms, and visual embeddings deviate from discrete text embeddings. We observe that mapping to text tokens using cosine similarity results in higher relevance to the image compared to Euclidean distance, as highlighted by the orange text in Fig. 7 and 8. This approach better preserves the rich information contained within the image.

### B.7 Examples of ETA Generated Response

In Fig. 10, we present the ablation study on safety-guided deep alignment. It shows that using only a prefix results in shallow alignment, which often leads to transitional phrases like “However,” causing subsequent harmful outputs.

Additionally, in Fig. 9, we display the effects of using different safety prefixes for alignment. Note that the results in this figure apply both shallow and deep alignment, with the only variable being the prefix. It can be observed that responses starting with “As an AI assistant” are more helpful, providing answers more relevant to the question while also ensuring safety.

Finally, in Fig. 11, we compare more different VLM backbones and the impact of introducing ETA during inference on safety benchmarks, demonstrating the superiority of ETA.

**Table 10: Ablation of the criterion during the evaluation phase. ’+ Pre Eval. or Post Eval. Unsafe’ indicates that alignment is applied if either evaluation stage deems the behavior unsafe, while ’+ Pre Eval. and Post Eval. Unsafe’ means alignment is only applied when both evaluator classify the behavior as unsafe. The latter is the strategy adopted by ETA.**
| Method | SPA-VL Harm ($\downarrow$) | TextVQA ($\uparrow$) | Mis-Eval. ($\downarrow$) |
| --- | --- | --- | --- |
| LLaVA-1.5-7B | 46.04 | 58.20 | - |
| + Only Pre Eval. | 12.45 | 55.52 | 4.44 |
| + Only Post Eval. | 13.21 | 57.57 | 1.76 |
| + Pre Eval. or Post Eval. Unsafe | 11.70 | 55.11 | 5.98 |
| + Pre Eval. and Post Eval. Unsafe (ETA) | 16.98 | 58.15 | 0.34 |

Figure: Figure 6: Evaluation results across different thresholds for the LLaVA-1.5-7B model. The table shows the Mis-Evaluation percentage, accuracy on the TextVQA dataset, and unsafe rate (USR) on the SPA-VL Harm benchmark under various pre- and post-thresholds. Darker colors indicate better performance. It can be observed that safety and usefulness are conflicting metrics.
Refer to caption: x7.png

Figure: Figure 7: The converted image tokens are decoded from textual embeddings, which are mapped from the original visual embeddings, with the term orange referring specifically to the image content. We replaced visual embedding with the converted image tokens, which reduces the rate of unsafe responses on the SPA-VL Harm test set (Zhang et al., ).
Refer to caption: x8.png

Figure: Figure 8: The converted image tokens are decoded from textual embeddings, which are mapped from the original visual embeddings, with the term orange referring specifically to the image content. The average distance represents the mean Euclidean distance between all image embeddings and their nearest text embeddings, highlighting that continuous visual embeddings typically deviate from discrete text embeddings. We replaced visual embedding with the converted image tokens, which reduces the 8% unsafe rate on the SPA-VL Harm test set (Zhang et al., ).
Refer to caption: x9.png

Figure: Figure 9: Examples of pre-filling with different prefixes. It can be observed that the prefix “As an AI assistant” results in higher helpfulness in the responses.
Refer to caption: x10.png

Figure: Figure 10: VLM + Shallow Alignment refers to responses where only the prefix “As an AI assistant” was added without applying safety-guided deep alignment. It can be observed that this often leads to an initial refusal to respond, followed by transitional phrases like “However” ultimately resulting in harmful outputs. This highlights the importance of deep alignment in ensuring safe responses.
Refer to caption: x11.png

Figure: Figure 11: Responses of ETA on multimodal safety benchmarks.
Refer to caption: x12.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: Is there any compromised approach to also employ similar strategy to API-based VLM/LLMs like GPT-4(V)?
soundness: 3
strengths: 1. The idea of the method is intuitive and easy to follow.
2. The method is plug-and-play, could be deployed to many VLM systems.
3. Comprehensive experiments on some VLMs with good performance.
summary: The paper introduces ETA, a two-phase framework for enhancing Vision Language Model (VLM) safety at inference time. The proposed method assesses visual and textual safety using CLIP scores for images and a reward model for text. It also applies a "safety prefix" (shallow alignment) and best-of-N search (deep alignment) to ensure responses are harmless and relevant. The ETA significantly reduced unsafe responses through comprehensive experiments.
weaknesses: 1. The method itself, is more like a combinaiton of pre-processing and post-processing techniques. The contribution of this method maybe limited.
2. As a plug-and-play method, I expect to see it to be applied to more recent VLMs, such as LLaVA-Next, LLaMA3.2.
3. Also I would expect to see the model performance drop on more recent and widely-used benchmarks like MMMU
4. Some highly-relevant references about VLM/LLM safety are missing [1][2]

I would be happy to raise my score once my concerns are addressed.

[1] Inan H, Upasani K, Chi J, et al. Llama guard: Llm-based input-output safeguard for human-ai conversations[J]. arXiv preprint arXiv:2312.06674, 2023.

[2] Tu H, Cui C, Wang Z, et al. How many unicorns are in this image? a safety evaluation benchmark for vision llms[C]. ECCV, 2024.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please address the concerns in the Weakness part.
soundness: 2
strengths: 1.	This paper introduces a straightforward, training-free strategy to address safety issues in VLMs.
2.	Experiments conducted across multiple models and datasets demonstrate the strategy's effectiveness and generalizability.
3.	The paper is well-structured, providing a clear narrative that effectively outlines both the problem and solution.
summary: The paper explores solutions for enhancing the safety of Vision Language Models (VLMs). The authors introduce a strategy called Evaluating Then Aligning (ETA) to address this issue at inference time. 

ETA first detects unsafe visual inputs using CLIP similarity to an unsafety-specific prompt (pre-generation score) and an LLM-based reward model to assess initial output safety (post-generation score). If both scores indicate that the input is unsafe (pre-generation score is above the threshold, post-generation score is below the threshold), ETA appends a predefined prefix (e.g., "As an AI assistant,") to the prompt, guiding the VLM to generate a safer response by maximizing the weighted sum of two factors: the similarity between the current output sentence and the visual input, and the post-generation score of the previously generated sentence.

The authors conduct experiments across various LLMs and datasets to demonstrate the effectiveness of the proposed strategy.
weaknesses: 1.	The setting of the helpfulness experiment (Table 2) is not well rigorous. 
The experiment is conducted using general comprehensive benchmarks and VQA datasets, which is not fully consistent with the "usefulness" goal the authors aim to address—namely, ensuring helpful responses even with unsafe inputs. While Table 3 provides some helpfulness results, the range of benchmarks used is somewhat limited. Testing on more models and datasets can better verify the effectiveness of this strategy.
2.	The analysis of the ablation study results is inadequate:
When the Utility Guide component is removed from the strategy, the unsafe rate actually decreases, yet this result is not fully analyzed. Does this suggest that the Utility Guide might be unnecessary? A deeper exploration is needed to clarify its role and significance.
3.	The proof of the Motivation in the MULTIMODAL EVALUATION (i.e. Continuous Visual Token Embeddings Bypass Safety Mechanisms) is not very rigorous: In the proof experiment, continuous visual embeddings were converted to the nearest discrete text embeddings, showing a 7% reduction in unsafe rates on one dataset. Testing this approach across more datasets and models could strengthen its reliability.
4.	The effectiveness of the strategy is not well proven. The number of baselines (especially for inference-time baselines) is too small to prove the effectiveness of the strategy.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: - Performance comparison with the safety-aware visual RLHF (if any) would be appreciated to see if the proposed method would rather be the optimal way of safeguard VLMs.
soundness: 3
strengths: - The proposed safeguard mechanism induces helpfulness, which better promotes usefulness of VLMs in a safe manner, rather than simply refusing the given user instruction.
- The paper is well written to easily understand
- The defense performance is well-improved compared to the baseline defense methods.
summary: The paper proposed a new safeguard mechanism for the Vision Language Model (VLM) during inference phase. This simple and straightforward approach largely reduces the unsafe responses while ensuring the helpfulness corresponding to the given input instruction.
weaknesses: - Novelty: although the simplicity of the proposed methods, the novelty seems significantly limited, as the proposed methods are based on the simple application of existing models such as CLIP and Reward Models.
- Clarification of the motivation: While the authors posed the major cause of the VLMs being jailbroken is the continuous nature of the visual embeddings, I think it is due to the visual embeddings not being safety-aligned with RLHF. Also, I'm not sure if the proposed methods correspondingly resolve the motivation behind the visual embedding's continuous nature.
- Potential side-effects of the proposed method:
   1) Pre-generation Evaluator: although the image contains the harmful image, the textual instruction might be normal and benign (e.g., a criminal scene image + "How to prevent such crimes?"). According to the proposed method, it may judge these benign inputs as unsafe, which possibly leaves as a side-effect.
    2) Sentence-level Deep Alignment: since the proposed deep alignment selects the optimal sentence under the consideration of the safeness of the given sentence and its correlation with the input image in a greedy manner, the resultant sequence of sentences might not be ensured to be logically connected. This potential side-effect might rather hinder the reliability of the VLMs.

---

confidence: 5
contribution: 3
initial_rating: 6
presentation: 4
questions: (1) How can the interpretability of the post-generation evaluation process be improved?

The reliance on RMs can sometimes lead to a lack of clear explanations for why a response is deemed safe or unsafe. Have you considered methods to enhance interpretability, such as generating explanations alongside safety scores, or using models that can provide reasoning for their evaluations?

(2) Did you observe any scenarios where the ETA framework failed or produced unexpected results?

Understanding the limitations of ETA would be useful for assessing its real-world applicability. Are there specific types of inputs or adversarial attacks where ETA was less effective? For example, were there instances where CLIP misinterpreted visual content, leading to unsafe outputs, or where the RM failed to accurately judge the safety of a response? Additionally, if such limitations were observed, what improvements or adjustments could be considered to address these cases effectively?
soundness: 3
strengths: Originality: 

The paper introduces a novel framework, ETA (Evaluating Then Aligning), which addresses a critical gap in the safety of Vision Language Models (VLMs) by focusing on inference-time alignment. Unlike existing approaches that primarily rely on extensive fine-tuning or are limited to specific types of inputs, ETA offers a fresh perspective by combining multimodal evaluation and bi-level alignment without requiring additional training. This plug-and-play nature makes it a highly original contribution, providing a more flexible and scalable solution for enhancing VLM safety.

Quality: 

The technical quality of the paper is robust, with a well-structured methodology that is clearly articulated and backed by extensive experimental validation. The authors systematically evaluate ETA across multiple dimensions, including safety, helpfulness, and efficiency. The experiments are comprehensive, covering various adversarial benchmarks and showing consistent improvements over baseline methods.

Clarity: 

The paper is clearly written. The framework’s methodology is detailed and easy to follow, aided by diagrams that illustrate key concepts, such as the two-phase evaluation and alignment process. Additionally, the use of concise and relevant examples helps in understanding how ETA addresses existing limitations.

Significance: 

The significance of this work is substantial, as it addresses one of the key challenges in deploying VLMs in real-world applications—ensuring safety without compromising utility. By introducing a method that does not require fine-tuning, ETA can be easily integrated into existing systems, making it practical for widespread use. The ability to improve safety while maintaining model efficiency and general capabilities could encourage broader adoption of VLMs.
summary: The paper proposes a two-phase approach called "Evaluating Then Aligning" (ETA), aiming at improving the safety and utility of Vision Language Models (VLMs) during inference. It begins by assessing the safety of both visual and textual inputs and outputs using multimodal evaluation. If unsafe behavior is detected, ETA applies a bi-level alignment strategy to ensure responses are safe and useful.  Overall, ETA offers a plug-and-play solution that requires no additional training, making it a practical approach for real-world applications of VLMs.

Key Contributions:

(1) The framework uses visual content analysis and a reward model to evaluate the safety of inputs and outputs, creating a more robust safety awareness across both modalities.

(2) The method features shallow alignment using an interference prefix to activate safety mechanisms, followed by deep alignment with sentence-level best-of-N searching to ensure helpfulness without compromising safety.

(3) Extensive experiments demonstrate that ETA significantly reduces unsafe responses across various VLMs and benchmarks, outperforming existing inference-based methods. It maintains the general capabilities of the models while enhancing safety and efficiency.
weaknesses: 1. Limitations in Pre-Generation Evaluation Using CLIP

A notable weakness of the paper lies in the reliance on CLIP for pre-generation safety evaluation, specifically through the use of predefined prompts (e.g., checking for "physically harmful" content). While CLIP is a powerful model for general vision-language tasks, its training data may not always align well with the specific categories being evaluated, such as physical harm. This could lead to situations where CLIP fails to accurately assess certain types of harmful content because it was not explicitly trained on safety-critical datasets. To improve this, the authors could consider using adaptive prompts that are tailored to different scenarios, enabling more precise evaluations. By refining prompts based on the context of the input (e.g., using distinct prompts for detecting physical hazards, offensive imagery, or illegal content), the system could achieve more accurate safety checks across a wider range of situations.

2. Potential Instability in Post-Generation Evaluation Using Reward Models (RMs)

The use of reward models (RMs) for post-generation evaluation could introduce instability, as the ability of RMs to accurately judge safety and utility might vary depending on the context. The RM employed may not be inherently designed to handle safety-specific assessments, which raises concerns about the consistency and reliability of its evaluations. Furthermore, this approach can lack interpretability, as it does not always provide clear reasons for why a response was considered safe or unsafe. To address this, it would be beneficial to either fine-tune the RM specifically for safety-related tasks or incorporate explainability mechanisms that can offer insights into how safety decisions are being made, thus building more trust in the evaluation process. For instance, the model could be designed to generate detailed explanations for its safety assessments, which could then be evaluated themselves to ensure they align with safety standards.

Meta-review: This paper investigates the safety aspect of VLMs. The key contribution lies in the development of a two-phase inference-time alignment framework, where the first stage assesses the safety of input visual content and initial output responses, and the second stage involves a multi-level alignment strategy. The provided emprical results well support the effectiveness of the proposed method. 

Overall, the reviewers found this paper well-written and easy to follow, and they appreciated both the simplicity and effectiveness of the proposed method. Nonetheless, some major concerns are also raised, including: 1) the novelty is somewhat limited; 2) the current motivation is not strongly supported, and more baselines and datasets should be included; 3) the main experiment should consider more recent VLMs; and 4) some experimental results require further analysis and discussion.

During the rebuttal and discussion stages, the authors provided detailed responses, additional experiments, and clarifications. Two reviewers acknowledged most of their concerns are well addressed, and agreed on accepting it. Reviewer tnBM acknowledged that the only remaining concern is the novelty (a point also raised by Reviewer JJRX). After reviewing the authors' rebuttal and clarification on novelty, the AC believes its contribution is sufficiently different from existing works and found its novelty acceptable (though not substantial). Reviewer 8ihd did not engage in the discussion, but after reviewing their review and the authors' responses, the AC is confident that their concerns were adequately addressed.

Given the importance of this research topic, the strong performance of this plug-and-play solution, and the fact that most concerns (except for the "minor" novelty issue) have been satisfactorily addressed, the AC recommends accepting the submission.

Decision: Accept (Poster)