Effective and Efficient Adversarial Detection for Vision-Language Models via A Single Vector

## Abstract

Abstract Visual Language Models (VLMs) are vulnerable to adversarial attacks, especially those from adversarial images, which is however under-explored in literature.
To facilitate research on this critical safety problem, we first construct a new la R ge-scale A dervsarial images dataset with D iverse h A rmful R esponses (RADAR), given that existing datasets are either small-scale or only contain limited types of harmful responses.
With the new RADAR dataset, we further develop a novel and effective i N -time E mbedding-based A dve RS arial I mage DE tection (NEARSIDE) method, which exploits a single vector that distilled from the hidden states of VLMs, which we call the attacking direction , to achieve the detection of adversarial images against benign ones in the input.
Extensive experiments with two victim VLMs, LLaVA and MiniGPT-4, well demonstrate the effectiveness, efficiency,
and cross-model transferrability of our proposed method. Our code is available at https://github.com/mob-scu/RADAR-NEARSIDE .

## 1 Introduction

Vision Language Models (VLMs), such as BLIP-2 (Li et al., ), LLaVA (Liu et al., ), MiniGPT-4 (Zhu et al., ) and GPT-4V (OpenAI, ), have attained remarkable success over various vision-language tasks (Dai et al., ; Zhu et al., ).
Besides improving performances, ensuring the safety of responses is just as important in the development of VLMs.
Compared with classic Large Language Models (LLMs) that take in discrete textual inputs, VLMs that accept both textual and visual inputs are more susceptible to “jailbreaking”, wherein malicious users manipulate inputs to elicit harmful outputs, due to the continuous and high-dimensional nature of visual inputs (Qi et al., ).
This issue, which has posed persistent safety challenges in classical vision models (Chakraborty et al., ), also presents intrinsic difficulties for developing safe VLMs.

Figure: (a) Working mechanism of VLMs.
Refer to caption: x1.png

Existing studies examine the safety threat in VLMs mainly from the perspective of adversarial samples as shown in Fig. 1.
It has been revealed that adversarial images are more effective than adversarial texts on attacking VLMs (Qi et al., ; Carlini et al., ).
Currently, only a few studies have been conducted to protect VLMs against adversarial images.
These methods either seek to detect adversarial images based on the responses’ discrepancy (Zhang et al., ), or to purify noised-images (Qi et al., ) with diffusion models (Nie et al., ), achieving promising effectiveness.
However, the first approach is computation-intensive and time-consuming, as it requires sampling multiple responses for the same input; the other approach, in addition to the computational cost issue, may even suffer degraded performance when dealing with less perceptible noises.

According to previous studies (Subramani et al., ; Turner et al., ; Zou et al., ; Rimsky et al., ; Li et al., ; Liu et al., ) (see Sec. 2.2),
the behaviors of LLMs can be modulated to generate texts towards certain specific attributes, such as truthfulness, by exploiting a set of steering vectors (SVs) that can be directly extracted from LLMs’ hidden states.
In adversarial attacks, the victim VLMs are manipulated by adversarial inputs to generate harmful responses, where the VLMs’ behaviors change from harmlessness to harmfulness.
We can calculate the SV that can account for VLMs’ behavior change given the adversarial inputs, which is named the attacking direction, and exploit it to detect the existence of adversarial samples by assessing whether the inputs’ embedding has high similarity to the attacking direction.

However, existing datasets for investigating adversarial attacks for VLMs, as shown in Tab. 1, are small-scale and contain limited harm types, significantly restricting the thorough evaluation of VLMs defending against adversarial attacks.
Therefore, we construct RADAR, a dataset of laRge-scale Adervsarial images with Diverse hArmful Responses, for comprehensively evaluating VLMs against adversarial images.
In RADAR, we generate adversarial images to attack two widely-used VLMs, MiniGPT-4 (Zhu et al., ) and LLaVA (Liu et al., ), based on a wide diversity of harmful contents.
Each sample consists of an adversarial and a benign sample, with each containing a query, an adversarial/benign image and corresponding response of VLMs.
In total, RADAR contains 4,000 samples, which is the most large-scale so far.
For high sample quality, we apply filtering operations to ensure harmlessness and harmfulness of responses to benign and adversarial inputs respectively.
It will be released to the public to facilitate related research in the community.

With RADAR, we further propose a novel iN-time Embedding-based AdveRSarial Image DEtection (NEARSIDE) method, which leverages the attacking direction to detect adversarial images to defend VLMs.
Specifically, we first extract the attacking direction from VLMs by calculating the average difference between the benign input and the adversarial input in the embedding space of VLMs.
With the obtained attacking direction, we classify an input as an adversarial input if the projection of its embedding to the attacking direction is larger than a threshold; otherwise the input is classified as a benign input.
Once the adversarial image is detected with the proposed NEARSIDE method, further actions can be taken to protect the VLMs, such as overwriting outputs with a predefined harmless response or purifying the adversarial images by diffusion models.

We conduct extensive experiments to evaluate our NEARSIDE method on the new RADAR dataset.
It is demonstrated that NEARSIDE achieves detection accuracy of $83.1\%$ on LLaVA and $93.5\%$ on MiniGPT-4, indicating impressive effectiveness.
Furthermore, we experimentally verify the cross-model transferability of the attacking direction in our method.
At inference, we compare the efficiency between our method and the baseline method, showing that our method takes an average of $0.14$ seconds to complete a detection on LLaVA that is $40$ times faster than the best existing method.

In summary, the major contributions of our work are four-fold:

- •
We propose to identify the attacking direction that directly distilled from the VLMs’ hidden space, and exploit it to defend the VLMs against adversarial images.
- •
We construct the RADAR dataset, which is the first large-scale adversarial image dataset with a diverse range of harmful responses, to support a comprehensive analysis of VLMs’ safety and facilitate future research.
- •
Based on RADAR, we propose a novel NEARSIDE method, which is capable of effectively and efficiently detecting adversarial visual inputs of VLMs using the identified attacking direction from VLMs’ hidden space.
We further explore the cross-model transferrability of our method given the Platonic Representation Hypothesis (Huh et al., ).
- •
Extensive experiments on two victim VLMs, LLaVA and MiniGPT-4, demonstrate the effectiveness, efficiency, and cross-model transferrability of our method.

## 2 Background

### 2.1 Adversarial attack

Adversarial attack is maliciously manipulating inputs to compromise performance of the targeted model (Chakraborty et al., ; Ponnuru et al., ).
The manipulated inputs are referred to as adversarial samples.
Formally, adversarial samples are generated by minimizing the negative log-likelihood loss of an adversarial target:

$$ $I_{\text{adv}}=\operatorname*{arg\,min}_{\hat{I}_{\text{adv}}\in\mathcal{I}} \sum_{i=1}^{m}-\log(p(y_{i}|\hat{I}_{\text{adv}})).$ (1) $$

Here $\mathcal{I}$ represents the input space subject to certain constraints, such as a perturbation radius $\|I_{\text{adv}}-I\|\leq\epsilon$, with $\epsilon$ typically set to $16/255$, $32/255$, $64/255$, or unbounded (denoted as “inf”).
$y_{i}$ refers to harmful outputs, and $I_{\text{adv}}$ can be either a manipulated text input, where a suffix is appended to attack LLMs (Zou et al., ), or a manipulated visual input, where imperceptible noise is added to the original image to attack VLMs (Qi et al., ; Carlini et al., ).

To solve Eqn. (1), various optimization techniques can be employed to generate the adversarial sample $I_{\text{adv}}$.
For LLMs, the coordinate gradient-based search (Zou et al., ) or genetic algorithms (Andriushchenko et al., ) are commonly used due to the discrete nature of textual inputs.
In contrast, for VLMs, where image noise is continuous, Projected Gradient Descent (PGD) (Madry et al., ; Qi et al., ; Carlini et al., ) is an effective and widely adopted approach.

### 2.2 Steering vectors in LLMs

According to the previous research (Subramani et al., ; Turner et al., ; Zou et al., ; Rimsky et al., ; Li et al., ; Liu et al., ),
the behaviors of LLMs can be modulated to generate texts towards certain specific attributes, such as truthfulness, by exploiting a set of steering vectors (SVs) that can be directly extracted from LLMs’ hidden states.
To extract the SV for a certain behavior of LLMs, pairs of contrastive prompts ($p_{+},p_{-}$) are used, where $p_{+}$, $p_{-}$ involve the same question or request, but $p_{+}$ adds words to encourage LLMs to possess the behavior while $p_{-}$ represents the opposite.
Formally, given a set $\mathcal{D}$ of ($p_{+}$, $p_{-}$), the SV is calculated by

$$ $\text{SV}=\frac{1}{|\mathcal{D}|}\sum\nolimits_{p_{+},p_{-}\in\mathcal{D}} \text{LLM}(p_{+})-\text{LLM}(p_{-})$ (2) $$

where $\text{LLM}(p_{+}),\text{LLM}(p_{-})\in{\mathbb{R}}^{d}$ are $d$-dimension vectors that represent LLMs’ embedding for the $i^{\text{th}}$ prompt $p_{+}$ and $p_{-}$ respectively.
Through Eqn.(2) that takes the mean difference of the embeddings, the SV can be easily acquired, which can specify a tendency, or direction, in LLMs’ embedding space regarding the model behavior.
That means, simply adding or subtracting such a direction in LLMs’ activations can noticeably control LLMs’ behavior to generate text with certain attributes.
For example, given a direction of “truthfulness”, adding this direction can encourage LLMs to generate more truthful responses (Zou et al., ; Rimsky et al., ).

## 3 Proposed dataset

To comprehensively analyze the threat of adversarial attacks posed to VLMs, we propose RADAR, a laRge-scale Adervsarial images dataset with Diverse hArmful Responses.
Fig. 2 illustrates our construction pipeline.
At below we elaborate each step in the pipeline and provide an analysis of its statistics to highlight its merits.
An exemplar sample in the new RADAR dataset is given in Appx. C.

Figure: Figure 2: An illustration of construction pipeline for our RADAR dataset.
Refer to caption: x3.png

### 3.1 Data preparation

In RADAR, each sample consists of an adversarial sample and a benign sample, with each containing a query, an adversarial/benign image and VLMs’ response.
To build RADAR, we use queries from train and test sets in HH-rlhf harm-set (Bai et al., ), those from Harmful-Dataset (Harm-Data) (Sheshadri et al., ), and sentences in Derogatory corpus (D-corpus) (Qi et al., ).

We collect benign images from COCO (Lin et al., ), which is a large-scale image dataset widely used in computer vision.
To build RADAR, we choose the validation and test sets from COCO 2017, that is, 5,000 validation images, 41,000 test images, and 91 object types in total.
By adopting COCO we can introduce diverse visual information in RADAR’s samples.

We generate adversarial images with harmful responses collected from HH-rlhf harm-set, D-corpus, and Harm-Data, as detailed in Sec. 3.2.
The HH-rlhf harm-set and Harm-Data are all preference data, where each sample is a tuple of (query, response), from which we select human rejected (harmful) samples, i.e. 15.8k samples in total to build RADAR.
D-corpus contains 66 derogatory sentences against gender and race, which are all inlcuded to build our RADAR.
The responses in RADAR are generated by feeding queries and adversarial/benign images into the two victim VLMs, i.e., MiniGPT 4 (Zhu et al., ) and LLaVA (Liu et al., ), as detailed in Sec. 3.3.

### 3.2 Adversarial image generation

According to Eqn. (1) and Sec. 2.1, we optimize a continuous noise that is added to the benign image to maximize the probability of the harmful text, in order to generate an adversarial image.
The optimization of noises is implemented using PGD (Madry et al., ).
In particular, for samples from HH-rlhf harm-set and Harm-Data, we optimize $-\log(p(y_{i}|\hat{I}_{\text{adv}})$ in Eqn. (1), where $\hat{I}_{\text{adv}}$ denotes the noised adversarial image and the query, and $y_{i}$ denotes the harmful response.
Note that when optimizing $-\log(p(y_{i}|\hat{I}_{\text{adv}})$ on D-corpus, $\hat{I}_{\text{adv}}$ refers to only the noised adversarial image, and $y_{i}$ is the harmful sentence.
To generate the adversarial images, we use the open-sourced code (^1^11[https://github.com/Unispac/Visual-Adversarial-Examples-Jailbreak-Large-Language-Models](https://github.com/Unispac/Visual-Adversarial-Examples-Jailbreak-Large-Language-Models)) and leave implementation details and hyper-parameters to Appx. A.

### 3.3 Sample generation and filtering

We then use the benign and adversarial images obtained as aforementioned to generate the samples constituting the proposed RADAR dataset.
In particular, we input each benign or counterpart adversarial image plus a corresponding query to the victim VLM, i.e. MiniGPT-4 or LLaVA, respectively, and obtain a response.
For D-corpus, we utilize the harmful sentence as the query.
The response is then judged by two models to assess its safety.
The first model is a classifier called HarmBench-Llama-2-13b-cls(^2^22[https://huggingface.co/cais/HarmBench-Llama-2-13b-cls](https://huggingface.co/cais/HarmBench-Llama-2-13b-cls)) (Mazeika et al., ), which is fine-tuned from the Llama2-13b (Touvron et al., ) and classifies that whether a pair of (query, response) is harmful or not.
The other model is GPT-4o mini (OpenAI, ), which are guided with carefully designed prompts to make judgements following (Qi et al., ) and (Zeng et al., ).
Concretely, we prompt GPT-4o mini to provide a score ranging from 1 to 5 for each tuple of (query-response), where the scores of 1, 2 indicate a harmless response, the score of 3 indicates borderline, and the scores of 4, 5 indicate a harmful response.
Please refer to Appx. B for more details.
It is expected that for each pair of benign and adversarial images, the responses given by the victim VLM should be judged as harmless for the benign input while harmful for the adversarial input by both models simultaneously.
We take this as the criterion to determine whether the quintuple of (query, benign input, harmless response, adversarial input, harmful response) will be included in our RADAR.

In practice, we find that quite a number of responses are harmful given benign images and harmless given adversarial images.
As also reported in Qi et al. (), the success of adversarial attack is far from 100%.
When constructing our RADAR, we use the two models to judge the responses’ harmfulness.
Such filtering operations significantly lift the quality of samples in the proposed dataset.

### 3.4 Statistics analysis

With the above construction pipeline, the resultant RADAR contains 4,000 samples in total, attacking two victim VLMs, i.e. MiniGPT-4 and LLaVA.
For each VLM, RADAR provides one training set and three test sets, with 500 samples per set.
Division of train and test sets is based on the source of images and queries.
Samples built using images from COCO validation set and queries from the train set of HH-rlhf harm-set are grouped into the train set in RADAR; samples built using images from COCO test set and queries from the test set of HH-rlhf harm-set, D-corpus, and Harm-Data are grouped to three test sets, respectively.
Training and tests sets use different images.
Different harmful texts are used in the four sets to ensure no information leakage and a reliable result.

A comparison of our RADAR with previous datasets used for investigating adversarial attack for VLMs is provided in Tab. 1.
Our RADAR features four advantages compared with previous ones.

- •
Large-scale: As shown in Tab. 1, RADAR greatly surpasses the existing datasets in scale. It contains up to 4,000 samples while the previous largest dataset, i.e. from (Zhang et al., ), contains only 200 samples, facilitating a reliable evaluation of VLMs’ safety.
- •
Diversity of harmful types:
RADAR covers a favorable diversity of harmful queries and responses, enabling a comprehensive evaluation of VLMs’ performance on understanding and defending various adversarial attacks.
Recent research on safety of VLMs (Wang et al., ; Dai et al., ; Ji et al., ) provides taxonomies about the potential harms in queries or responses, e.g. asking for guidance to make bombs or for providing private information.
During the construction of RADAR, we purposely increase such diversity.
- •
Open-source: RADAR will be open-sourced to facilitate future research on VLMs defending against adversarial attacks.
- •
High sample quality: We apply filtering operations during the construction of our RADAR with two models to ensure that the response to a benign input is harmless and that to an adversarial input is harmful.
In comparison, the other datasets are built by specifying the harmfulness of the input before feeding it to victim models, while neglecting the reliability of responses, given
the success ratio that adversarial images attack VLMs is not 100%.

**Table 1: Comparison of datasets for adversarially attacking VLMs. “-” means not reported.**
| Paper | Scale | Harmful Types | Open Source | Data Filtering |
| --- | --- | --- | --- | --- |
| (Zhang et al., ) Arxiv | 200 | Harmful queries | \usym2713 | \usym2717 |
| (Tu et al., ) Arxiv | 3 | Toxic words | \usym2713 | \usym2717 |
| (Carlini et al., ) Neurips 2023 | - | Toxic words | \usym2717 | \usym2717 |
| (Qi et al., ) AAAI 2024 | 3 | Toxic words | \usym2713 | \usym2717 |
| (Luo et al., ) ICLR 2024 | - | Harmful queries | \usym2717 | \usym2717 |
| (Shayegani et al., ) ICLR 2024 | 8 | Toxic words | \usym2717 | \usym2717 |
| RADAR (Ours) | 4,000 | Both | \usym2713 | \usym2713 |

## 4 Proposed method

Figure: Figure 3: An illustration of proposed NEARSIDE. Our method learns the attacking direction on a set of tuples (benign input, adversarial input), and then classifies a test input as benign or adversarial according to the projection between the input’s embedding and the attacking direction. If the projection is larger than a threshold, it is classified as an adversarial input, and otherwise as benign.
Refer to caption: x4.png

To effieciently defend VLMs from adversarial attacks, we propose a novel iN-time Embedding-based AdveRSarial Image DEtection method (abbr. as NEARSIDE) that uses a single vector, named the attacking direction, to detect the adversarial inputs.
Fig. 3 gives an illustration of NEARSIDE.

### 4.1 Attacking direction

As discussed in Sec. 2.2, the behaviors of LLMs can be controlled with a set of steering vectors (SVs) to generate texts towards certain specific attributes, such as truthfulness.
Such SVs can be easily distilled from LLMs’ hidden states based on Eqn. (2).
In adversarial attacks, the adversarial inputs elicit harmful responses of the victim VLMs, where the VLMs’ behaviors alter with an attribute shifting from harmlessness to harmfulness.
We can calculate the SV that can account for VLMs’ behavior change given the adversarial inputs.
We name such a vector the attacking direction.
In this work, we propose to detect the existence of the adversarial samples by assessing whether the inputs’ embedding has shown high similarity to the attacking direction.

To extract the attacking direction from VLMs’ hidden states, the adversarial and benign samples that make pairwise contrastive prompts are required.
Formally, consider a training set ${\mathbb{T}}=\{(I_{\text{adv}}^{i},I_{\text{b}}^{i})\,|\,i=0,1,...,n\}$ where $I_{\text{adv}}$, $I_{\text{b}}$ denote the adversarial and benign sample, respectively, and $n$ is the index.
Each sample contains an image and a piece of text.
We embed each sample $I^{i}$ by taking the embedding of the last input token from the last LLMs’ layer, i.e. $E^{i}\in{\mathbb{R}}^{d}$, where $d$ is the embedding dimension.
We embed all samples in ${\mathbb{T}}$, and obtain ${\mathbb{T}}_{\text{emb}}=\{(E_{\text{adv}}^{i},E_{\text{b}}^{i})\,|\,i=0,1,..
.,n\}$.
Then, we calculate the attacking direction by

$$ $D_{\text{attack}}=\frac{1}{n}\sum_{i=0}^{n}(E_{\text{adv}}^{i}-E_{\text{b}}^{i })\in{\mathbb{R}}^{d},\quad(E_{\text{adv}}^{i},E_{\text{b}}^{i})\in{\mathbb{T} }_{\text{emb}}.$ (3) $$

### 4.2 Detection of adversarial inputs

Let $\texttt{norm}(h)=h/\|h\|_{2}$ denote the $\ell_{2}$ normalization for a vector $h$.
Given the attacking direction $D_{\text{attack}}$, we classify a test input $I_{\text{test}}$ to be adversarial or benign by

$$ $I_{\text{test}}=\begin{cases}\text{adversarial example, }&\text{if}\ \ E_{ \text{test}}\cdot\texttt{norm}(D_{\text{attack}}^{j})^{\top}-t>0\vspace{2mm}, \\ \text{benign example, }&\text{otherwise,}\end{cases}$ (4) $$

where $E_{\text{test}}\in{\mathbb{R}}^{d}$ is the embedding of the last input token from the last layer of an VLM on the test sample, and $t\in{\mathbb{R}}$ is a scalar threshold to measure whether the similarity score is significant.
If the similarity score, i.e., the projection, is greater than the threshold, we classify the input $I_{\text{test}}$ to be adversarial as it has high similarity to the attack direction; otherwise, the input is classified as a benign input.
The threshold is decided using ${\mathbb{T}}_{\text{emb}}$:

$$ $t=\frac{1}{2n}\sum_{i=0}^{n}(E_{\text{adv}}^{i}\cdot\texttt{norm}(D_{\text{ attack}}^{j})^{\top}+E_{\text{b}}^{i}\cdot\texttt{norm}(D_{\text{attack}}^{j}) ^{\top}),\quad(E_{\text{adv}}^{i},E_{\text{b}}^{i})\in{\mathbb{T}}_{\text{emb}}.$ (5) $$

The threshold is the average similarity score of all training embeddings (from both adversarial and benign samples) on the attacking direction.

The proposed NEARSIDE, as shown in Eqn. (4), is extremely efficient as we only require running one feed-forward propagation given the input to infer $E_{\text{test}}$, thus enabling an in-time detection of adversarial samples.
After adversarial samples have been detected, the developer can take further steps to ensure VLMs’ safety, such as overwriting the responses to a preset text, applying diffusion models to purify the image, or disabling malicious accounts.
Therefore, NEARSIDE can defend VLMs from adversarial attack in an efficient and real-time manner.

### 4.3 Cross-model transferability

The Platonic Representation Hypothesis (Huh et al., ):
“Neural networks, trained with different objectives on different data and modalities, are converging to a shared statistical model of reality in their representation spaces.”

The proposed NEARSIDE is supposed to use the attacking direction extracted from one VLM to detect the adversarial samples for the same VLM.
According to the above Platonic Representation Hypothesis, we can assume that the learnt attacking direction and effectiveness of our detection method NEARSIDE are transferable across different models.
That is, our NEARSIDE can use the attacking direction extracted from one VLM to detect the adversarial samples for other VLMs.
The reason behind the assumption of the cross-model transferrability in our method is that, although different VLMs are trained from different data, the patterns regarding safety in these data should be similar.
However, the embedding spaces between two VLMs do have a gap.
We thus propose to explore the transferability using a linear transformation:

$$ ${\bm{W}}{\bm{E}}_{\text{m}_{\text{1}}}={\bm{E}}_{\text{m}_{\text{2}}},$ (6) $$

where ${\bm{E}}_{\text{m}_{\text{1}}}\in{\mathbb{R}}^{n\times d_{\text{m}_{\text{1}}}}$ and ${\bm{E}}_{\text{m}_{\text{2}}}\in{\mathbb{R}}^{n\times d_{\text{m}_{\text{2}}}}$ are the stacked embedding of benign inputs from the two VLMs $\text{m}_{\text{1}}$ and $\text{m}_{\text{2}}$, respectively, with $d_{\text{m}_{\text{1}}},d_{\text{m}_{\text{2}}}$ denoting their embedding dimension.
The linear transformation ${\bm{W}}$ is to align the two VLMs’ embedding spaces.
In practice, since powerful LLMs often have high dimension in their hidden states, directly solving Eqn. (6) would be too costly in memory due to the high dimension of $d_{\text{m}_{\text{1}}},d_{\text{m}_{\text{2}}}$.
Therefore, we propose to use principal component analysis (PCA) to reduce the dimension of ${\bm{E}}_{\text{m}_{\text{1}}}$ and ${\bm{E}}_{\text{m}_{\text{2}}}$.
Then, we have ${\bm{W}}=f^{\text{pca}}_{\text{m}_{\text{2}}}({\bm{E}}_{\text{m}_{\text{2}}})f
^{\text{pca}}_{\text{m}_{\text{1}}}({\bm{E}}_{\text{m}_{\text{2}}})^{\dagger}$, where $f^{\text{pca}}$ denotes PCA that reduces the dimension and $(\cdot)^{\dagger}$ denotes the pseudo-inverse.

Finally, given a test input $I_{\text{test}}$, its detection on $\text{m}_{\text{2}}$ is given by

$$ $I_{\text{test}}=\begin{cases}\text{adversarial example},&\text{if}\ \ f^{\text {pca}}_{\text{m}_{\text{2}}}(E_{\text{test,$\text{m}_{\text{2}}$}})\cdot \texttt{norm}({\bm{W}}f^{\text{pca}}_{\text{m}_{\text{1}}}(D_{\text{attack,$ \text{m}_{\text{1}}$}}))^{\top}-t_{\text{m}_{\text{1}}}>0,\vspace{2mm}\\ \text{benign example},&\text{otherwise},\end{cases}$ (7) $$

where the threshold $t_{\text{m}_{\text{1}}}$ is defined as

$$ $\displaystyle t_{\text{m}_{\text{1}}}=\frac{1}{2n}\sum_{i=0}^{n}($ $\displaystyle{\bm{W}}f^{\text{pca}}_{\text{m}_{\text{1}}}(E_{\text{adv,$\text{ m}_{\text{1}}$}}^{i})\cdot\texttt{norm}({\bm{W}}f^{\text{pca}}_{\text{m}_{ \text{1}}}(D_{\text{attack, $\text{m}_{\text{1}}$}}))^{\top}+$ (8) $\displaystyle{\bm{W}}f^{\text{pca}}_{\text{m}_{\text{1}}}(E_{\text{b, $\text{m }_{\text{1}}$}}^{i})\cdot\texttt{norm}({\bm{W}}f^{\text{pca}}_{\text{m}_{\text {1}}}(D_{\text{attack, $\text{m}_{\text{1}}$}}))^{\top}).$ $$

Eqn. (7) detects adversarial samples on $\text{m}_{\text{2}}$ only using the attacking direction of $\text{m}_{\text{1}}$ and the embedding of benign inputs from $\text{m}_{\text{2}}$ to learn the transformation matrix ${\bm{W}}$.
Note that this entire learning process has no access to adversarial samples on $\text{m}_{\text{2}}$.
Eqn. (7) works if the embedding space of the two VLMs can be linearly transformed without disturbing the attacking direction.

## 5 Experiments

We conduct extensive experiments on RADAR to evaluate the effectiveness of the proposed NEARSIDE in detecting adversarial images.
We first compare our method with strong baseline and then analyze its cross-model transferability, followed by the efficiency test.

### 5.1 Experiments setup

Victim VLMs.
We adopt MiniGPT-4 (Zhu et al., ) and LLaVA (Liu et al., ) as the victim VLMs.
MiniGPT-4 is built upon Vicuna (Chiang et al., ) and LLaVA is built upon Llama2 (Touvron et al., ).
Regarding the visual encoder, MiniGPT-4 utilizes the same pre-trained vision components of BLIP-2 (Li et al., ) consisting of pre-trained ViT followed by a Q-Former, while LLaVA only adopts a pre-trained CLIP (Radford et al., ).

Implementation.
For each victim VLM, as stated in Sec. 3, RADAR constructs one training set and three test sets.
NEARSIDE learns the attacking direction and threshold from the hidden states of the VLM on the training set.
Here, the hidden states refer to the embedding of the last token of the input from the LLM decoder’s final layer.
Then, we test the detection performance with the obtained attacking direction on the three test sets.
Regarding the cross-model transferability, we collect 5,000 pairs of benign images and queries to train the PCA model for each VLM.
We set 2048 as the dimension of the embedding after PCA.

Baseline.
We use JailGuard (Zhang et al., ) as our baseline, which is the state-of-the-art model for this task.
To detect adversarial visual inputs, JailGuard mutates input images to generate variants and calculates the discrepancy of VLMs’ outputs given different variants to distinguish the adversarial and benign inputs.
There are 18 mutation methods, and we use the best-performing mutation method “policy” reported in the JailGuard paper, where 8 variants are generated for each image.
We set all other hyperparameters to the recommended values as JailGuard.
It is worth mentioning that we adopt only one baseline as there are only limited works on defending VLMs from adversarial examples (Liu et al., ).

Evaluation metrics.
Since adversarial detection is a binary classification task, we adopt $Accuracy$, $Precision$, $Recall$ and $F1$ score as the evaluation metrics.

### 5.2 Main results

We compare our proposed method NEARSIDE against the baseline JailGuard on RADAR.
The experimental results are shown in Tab. 2.
From the results, we make below observations.
1) When taking LLaVA as the victim VLM, our NEARSIDE achieves an average increase of 31.3% in $Accuracy$, 43.5% in $Precision$, 12.6% in $Recall$, and 0.246 in $F1$, compared to the baseline JailGuard method.
2) When taking MiniGPT-4 as the victim VLM, our NEARSIDE achieves an average increase of 38.7% in $Accuracy$, 45.6% in $Precision$, 17.6% in $Recall$, and 0.316 in $F1$, compared to the baseline JailGuard method.
These results well demonstrate the effectiveness of our proposed method.
3) Although our NEARSIDE has lower $Recall$ on the Harm-Data set with LLaVA as the victim VLM, and also on D-corpus-test set with MiniGPT-4 as the victim VLM, it achieves significantly higher $F1$ scores on both sets.
We attribute the low $Recall$ of our method to its threshold for the adversarial detection.
As shown in Fig. 5, the projections of the two types of examples do fall into different ranges.
However, as the threshold is calculated on the training set, the threshold is not well fit for the Harm-Data, leading to degraded $Recall$.
If we set the threshold to -13, we can increase $Recall$ to 87.6% and $F1$ score to $0.900$, which are both significantly higher than the baseline.
From an overall perspective, the results can still demonstrate the powerful distinguishing capability of our method over adversarial and benign data.

**Table 2: Results of JailGuard v.s. NEARSIDE on RADAR test sets (best highlighted in bold).**
| Victim VLM | Test Set | Method | $Accuracy(\%)$ | $Precision(\%)$ | $Recall(\%)$ | $F1$ |
| --- | --- | --- | --- | --- | --- | --- |
| LLaVA | HH-rlhf | JailGuard | 51.2 | 51.1 | 57.8 | 0.540 |
| NEARSIDE | 84.4 | 89.3 | 78.2 | 0.834 |  |  |
| D-corpus | JailGuard | 58.1 | 58.1 | 58.2 | 0.581 |  |
| NEARSIDE | 94.0 | 99.5 | 88.4 | 0.936 |  |  |
| Harm-Data | JailGuard | 46.2 | 46.8 | 55.8 | 0.509 |  |
| NEARSIDE | 71.0 | 97.7 | 43.0 | 0.597 |  |  |
| MiniGPT-4 | HH-rlhf | JailGuard | 54.9 | 53.9 | 67.2 | 0.598 |
| NEARSIDE | 99.4 | 99.2 | 99.6 | 0.994 |  |  |
| D-corpus | JailGuard | 56.6 | 54.4 | 81.6 | 0.653 |  |
| NEARSIDE | 81.1 | 98.4 | 63.2 | 0.770 |  |  |
| Harm-Data | JailGuard | 52.8 | 52.4 | 61.2 | 0.565 |  |
| NEARSIDE | 100.0 | 100.0 | 100.0 | 1.000 |  |  |

Figure: Figure 4: Visualized projections of adversarial and benign samples to the attacking directions on Harm-Data with LLaVA as the victim.
Refer to caption: x5.png

### 5.3 Analysis on cross-model transferability

We utilize the attacking direction extracted from the source VLM (svlm) to detect adversarial input for the target VLM (tvlm), denoted as svlm $\to$ tvlm.
We calculate the difference (i.e. $\delta$) by subtracting the result of (svlm $\to$ tvlm) from that of Tab. 2, where ^-δ denotes the result is decreased while ^+δ denotes the opposite.
The obtained results for cross-model transferaility are shown in Tab. 3.
We can observe that cross-model transferability results are generally inferior to those in Tab. 2 where the attacking direction is extracted and applied with the same VLM, but both $Accuracy$ and $F1$ results of our method are higher than those of the baseline JailGuard.
Though cross-model transferrability decreases the detection performance, which is expectable, our method can still work well across different models.
These results clearly validate the cross-model transferability of the attacking direction and the proposed NEARSIDE.
It also says that, the Platonic Representation Hypothesis still holds in our setting, where a simple linear transformation is effective to align two VLMs’ embedding spaces.

**Table 3: Cross-model transferability results for our method.**
| svlm $\rightarrow$ tvlm | TEST SET | $Accuracy(\%)$ | $Precision(\%)$ | $Recall(\%)$ | $F1$ |
| --- | --- | --- | --- | --- | --- |
| $\text{MiniGPT-4}\rightarrow\text{LLaVA}$ | HH-rlhf | $64.3^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-20.1}}$ | $61.3^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-28.0}}$ | $77.6^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.6}}$ | $0.685^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.149}}$ |
| D-corpus | $69.4^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-24.6}}$ | $62.5^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-37}}$ | $96.8^{{\color[rgb]{0,0.58984375,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.58984375,0}+8.4}}$ | $0.760^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.176}}$ |  |
| Harm-Data | $74.7^{{\color[rgb]{0,0.58984375,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.58984375,0}+3.7}}$ | $76.7^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-21}}$ | $71.0^{{\color[rgb]{0,0.58984375,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.58984375,0}+28}}$ | $0.737^{{\color[rgb]{0,0.58984375,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.58984375,0}+0.14}}$ |  |
| $\text{LLaVA}\rightarrow\text{MiniGPT-4}$ | HH-rlhf | $77.8^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-21.6}}$ | $86.2^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-13}}$ | $66.2^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-33.4}}$ | $0.749^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.245}}$ |
| D-corpus | $80.4^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.7}}$ | $80.3^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-18.1}}$ | $80.6^{{\color[rgb]{0,0.58984375,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.58984375,0}+17.4}}$ | $0.804^{{\color[rgb]{0,0.58984375,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0,0.58984375,0}+0.034}}$ |  |
| Harm-Data | $97.1^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-2.9}}$ | $95.0^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.5}}$ | $99.4^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.6}}$ | $0.972^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.028}}$ |  |

We experiment to examine the effect of using ${\bm{W}}$ to align two VLMs’ embedding spaces, and the effect of reducing the dimension of the attacking direction and the VLMs’ embedding with the PCA model.
The detailed results are provided in Appx. E.
We find that, without ${\bm{W}}$, the cross-model transferability results will significantly decrease.
In addition, when reducing the dimension to 256, the cross-model transferability results still remain high, indicating that the information in low dimensional sub-spaces is already sufficient for aligning two VLMs’ embedding spaces.

### 5.4 Analysis of perturbation radius in generating adversarial images

The generation of adversarial images is constrained by the hyper-parameter $\epsilon$ as shown in Eqn. (1).
We test the robustness of the proposed NEARSIDE method to varying $\epsilon$.
We use NEARSIDE to detect the adversarial samples generated under different $\epsilon$.
Results are deferred to Appx. D.

### 5.5 Analysis of detection efficiency

In this part, we examine the detection efficiency of the proposed method.
For the baseline JailGuard, we utilize the widely-adopted VLLM (Kwon et al., ) to deploy the two VLMs, i.e. LLaVA and MiniGPT-4, on a local machine and generate outputs through API requests.
For our proposed NEARSIDE, we load VLMs and perform a single forward propagation to embed each input since the attacking direction can be pre-computed.
In addition to JailGuard and our method, we also include another two trivial methods that judge the harmfulness of the output into our efficiency evaluations, i.e. HarmBench and GPT-4o mini, which are used in data filtering operations to judge the harmfulness of responses in Sec. 3.3.
For all compared methods, we calculate the time including responses inference (note, NEARSIDE does not infer responses) plus follow-up operations, which refer to discrepancy calculation in JailGuard, projections calculation in NEARSIDE, and harmfulness evaluation in other two methods.
All experiments are conducted on a server with AMD EPYC 7543 32-core processors, 1 TB of RAM, and a NVIDIA A40 GPU.

We run experiments on 20 inputs and plot the average throughput in Fig. 5.
With our setup, NEARSIDE is ($\times$ 41$\sim$336) times faster than the other methods on LLaVA, and is ($\times$ 132$\sim$1190) times faster on MiniGPT-4, demonstrating remarkable efficiency as NEARSIDE is the only embedding-based method among all compared methods that does not require to infer the entire output.

## 6 Related works

### 6.1 Vision language models

Vision Language Models (VLMs) is equipped with a visual adapter to align the visual and textual representations in LLMs.
Notable examples are BLIP-2 (Li et al., ), LLaVA (Liu et al., ), MiniGPT-4 (Zhu et al., ), QWen-VL (Bai et al., ), GPT-4V (OpenAI, ), and Gemini (Anil et al., ), demonstrating impressive performance across various vision-language tasks (Dai et al., ; Zhu et al., ).
These VLMs vary in the design of their adapters (Liu et al., ; Li et al., ; Zhu et al., ).
For instance, BLIP-2 (Li et al., ) proposes Q-Former to align vision features with LLMs;
MiniGPT-4 (Zhu et al., ) and LLaVA (Liu et al., ) further add a linear transformation, and Qwen-VL (Bai et al., ) uses a single-layer cross-attention module.

### 6.2 Safety of language models

Safe LLMs should behave in line with human intentions and values (Soares & Fallenstein, ; Hendrycks et al., ; Leike et al., ; Ji et al., ) which are measured as being Helpful, Honest, and Harmless (Askell et al., ).
Alignment has emerged as a nascent research field aiming to align LLMs’ behaviors with human preferences, and there are two widely adopted alignment techniques, i.e. Instruction Fine-tuning and Reinforcement Learning from Human Feedback (RLHF).
In instruction fine-tuning, LLMs are given examples of (user’s query, desired output) and trained to follow user instructions and respond the expected output (Taori et al., ; Wei et al., ).
In RLHF, LLMs update output probabilities, i.e., the response policy, by reinforcement learning, which are rewarded for generating responses that align with human preferences and otherwises penalized (Russell & Norvig, ; Bai et al., ; Rafailov et al., ; Ouyang et al., ).

Two types of strategies can defend language models from adversarial attacks: detection and purification.
For instance, Zhang et al. () detects adversarial examples by calculating responses’ discrepancy;
Qi et al. () uses diffusion models (Nie et al., ) to purify the noised-images.
Other techniques such as the adversarial training (Bai et al., ) can also improve the robustness of models to adversarial attacks.
Though effective on classical image classifiers, these methods remain unexplored on large models like LLMs and VLMs and may disturb the optimization.

## 7 Conclusion

In this work, we propose RADAR, the first large-scale adversarial image dataset with diverse harmful responses to facilitate research on safety of VLMs.
With RADAR, we further develop NEARSIDE that exploits the idea of attacking direction to detect adversarial inputs.
We demonstrate with the effectiveness and efficiency of the proposed NEARSIDE by comparing it to the state-of-the-art on RADAR, and also highlight its cross-model transferability.

VLMs can generate open-ended responses, posing a persistent challenge to complete evaluation of the potential harms (Ganguli et al., ).
RADAR is built from a diverse array of datasets but may fall short of covering all harmful contents.
NEARSIDE is intended to detect the adversarial samples we examine in this work and is a demonstration of our idea of exploiting the attacking direction.

## Appendix A Details about generating adversarial images

In our implementation, we utilize the standard PGD method to generate adversarial images, according to 2.1.
We run 5000 iterations of PGD optimization on generating each adversarial image and we set $\epsilon$ to $32/255$.
We test the robustness of the NEARSIDE method to varying $\epsilon$.
We use NEARSIDE to detect the adversarial samples generated under different $\epsilon$.
Results are deferred to Appx. D.

## Appendix B Prompts to GPT-4 judgement

We adopt the prompt template proposed by Qi et al. (), which combines the prohibited model usage policy, the (harmful) input query, the model’s output for this query, and a rating rubric together to be fed to GPT-4 via OpenAI’s APIs.
The complete prompt template is as follows: ({Context.} will be replaced with the OpenAI’s detailed policy and scoring tules, {Context.} will be replaced with the harmful query and model’s output)

## Appendix C An exemplar sample in RADAR

In RADAR, each sample consists of an adversarial sample and a benign sample, with each containing a query, an adversarial/benign image and VLMs’ response.
Fig. 6 provides an exemplar sample.

Figure: Figure 6: An exemplar sample in RADAR.
Refer to caption: x7.png

## Appendix D Perturbation radius in generating adversarial images

We test the robustness of the NEARSIDE method to varying $\epsilon$.
We use NEARSIDE to detect the adversarial samples generated under different $\epsilon$.
We genearte 100 adversarial samples on the LLaVA D-corpus dataset under settings of $\epsilon=16$, $\epsilon=64$, and unconstrained.
The data generation follow the same pipeline as Sec. 3.
We create 100 samples for each $\epsilon$.
The results are provided in Tab. 4.

**Table 4: Results of the NEARSIDE on LLaVA D-corpus generated with different $\epsilon$.**
| $\epsilon$ of adversarial training | $Accuracy(\%)$ | $Precision(\%)$ | $Recall(\%)$ | $F1$ |
| --- | --- | --- | --- | --- |
| $\epsilon=16/255$ | 85.0 | 100.0 | 70.0 | 0.824 |
| $\epsilon=64/255$ | 93.5 | 97.8 | 89.0 | 0.932 |
| $unconstrained$ | 99.0 | 100.0 | 98.0 | 0.990 |

## Appendix E Analysis of cross-model transferability

Linear transformation ${\bm{W}}$.
We explore the Platonic Representation Hypothesis by using a linear transformation ${\bm{W}}$ to align the two VLMs’ embedding spaces.
To demonstrate the importance of the usage of ${\bm{W}}$, we conduct experiments that directly use the attacking direction of the source VLM to detect the adversarial samples of the target VLM without using ${\bm{W}}$.
Results are shown in Tab. 5.

PCA model.
We use PCA model to reduce the dimension of VLMs’ embedding and the attacking direction before learning the transformation ${\bm{W}}$.
In our initial setting, the dimension is reduced to 2056.
We experiment to examine the effect of reducing the dimension of the attacking direction and the VLMs’ embedding with the PCA model.
In specific, we vary the dimension in [2048, 1024, 512, 256] and report the cross-model transferability results in Tab. 6.

**Table 5: The results of cross-model transferability without ${\bm{W}}$. We report $\text{result}^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor} {rgb}{0.70703125,0,0}-\delta}}$ where ^δ indicates the difference between the results w/o ${\bm{W}}$ and with ${\bm{W}}$ shown in Table 3.**
| svlm $\rightarrow$ tvlm (w/o ${\bm{W}}$) | TEST SET | $Accuracy(\%)$ | $Precision(\%)$ | $Recall(\%)$ | $F1$ |
| --- | --- | --- | --- | --- | --- |
| $\text{MiniGPT-4}\rightarrow\text{LLaVA}$ | HH-rlhf | $50.9^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-13,4}}$ | $50.9^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-10.4}}$ | $51.6^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-26.0}}$ | $0.512^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.172}}$ |
| D-corpus | $53.8^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-15.6}}$ | $53.2^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-9.4}}$ | $63.8^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-33.0}}$ | $0.580^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.180}}$ |  |
| Harm-Data | $53.7^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-21.0}}$ | $53.7^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-23.0}}$ | $53.8^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-17.2}}$ | $0.537^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.200}}$ |  |
| $\text{LLaVA}\rightarrow\text{MiniGPT-4}$ | HH-rlhf | $32.8^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-45.0}}$ | $27.2^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-58.9}}$ | $20.6^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-45.6}}$ | $0.235^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.514}}$ |
| D-corpus | $71.0^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-9.4}}$ | $77.2^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-3.1}}$ | $59.6^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-21.0}}$ | $0.804^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.132}}$ |  |
| Harm-Data | $23.9^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-73.2}}$ | $23.1^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-71.9}}$ | $22.4^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-77.0}}$ | $0.227^{{\color[rgb]{0.70703125,0,0}\definecolor[named]{pgfstrokecolor}{rgb}{<br>0.70703125,0,0}-0.744}}$ |  |

**Table 6: The results of cross-model transferability where PCA reduce the VLMs’ embedding and the attacking direction to different dimensions. We use bold to highlight the best results.**
| svlm $\rightarrow$ tvlm | TEST SET | PCA-Dimension | $Accuracy(\%)$ | $Precision(\%)$ | $Recall(\%)$ | $F1$ |
| --- | --- | --- | --- | --- | --- | --- |
| LLaVA $\rightarrow$ MiniGPT-4 | HH-rlhf | 256 | 87.0 | 89.2 | 84.2 | 0.866 |
| 512 | 88.8 | 88.5 | 89.2 | 0.888 |  |  |
| 1024 | 88.2 | 89.0 | 87.2 | 0.881 |  |  |
| 2048 | 77.8 | 86.2 | 66.2 | 0.749 |  |  |
| D-corpus | 256 | 83.3 | 76.2 | 96.8 | 0.853 |  |
| 512 | 84.7 | 82.2 | 88.6 | 0.853 |  |  |
| 1024 | 77.6 | 88.5 | 63.4 | 0.740 |  |  |
| 2048 | 80.4 | 80.3 | 80.6 | 0.805 |  |  |
| Harm-Data | 256 | 87.8 | 80.4 | 100.0 | 0.891 |  |
| 512 | 83.4 | 75.1 | 100.0 | 0.858 |  |  |
| 1024 | 88.2 | 89.0 | 87.2 | 0.881 |  |  |
| 2048 | 97.1 | 95.0 | 99.4 | 0.972 |  |  |
| MiniGPT-4 $\rightarrow$ LLaVA | HH-rlhf | 256 | 57.0 | 53.8 | 98.8 | 0.697 |
| 512 | 58.5 | 54.7 | 98.0 | 0.703 |  |  |
| 1024 | 63.7 | 58.6 | 93.2 | 0.720 |  |  |
| 2048 | 64.3 | 61.3 | 77.6 | 0.685 |  |  |
| D-corpus | 256 | 50.9 | 50.5 | 100.0 | 0.671 |  |
| 512 | 58.5 | 54.7 | 98.0 | 0.703 |  |  |
| 1024 | 51.2 | 50.6 | 100.0 | 0.672 |  |  |
| 2048 | 69.4 | 62.5 | 96.8 | 0.760 |  |  |
| Harm-Data | 256 | 71.9 | 64.7 | 96.6 | 0.775 |  |
| 512 | 71.9 | 64.4 | 98.2 | 0.778 |  |  |
| 1024 | 74.4 | 67.6 | 93.8 | 0.786 |  |  |
| 2048 | 74.7 | 76.7 | 0.71 | 0.737 |  |  |

From Tab. 5, we find that, without ${\bm{W}}$, the cross-model transferability results will significantly decrease.
From Tab. 6, we find that, when reducing the dimension to 256, the cross-model transferability results still remain high, indicating that the information in low dimensional sub-spaces is already sufficient for aligning two VLMs’ embedding spaces.

## Peer Reviews

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: * What is the false detection rate on common benign datasets (e.g., VQA)?
* What is the detection rate against other attack methods?
* Can the proposed method detect adaptive attacks?
soundness: 1
strengths: + The proposed method is based on an observation cross-validated from multiple previous studies
+ The proposed method outperformed a baseline
summary: This paper proposes a new method to detect jailbreaking attacks against visual language models (VLMs). The method is based on the observation that large language models contain a set of steering vectors in the intermediate embeddings that can be modulated to generate texts towards certain specific attributes. Based on this observation, the authors propose to (1) build a larger dataset of adversarial attacks, and (2) use the dataset to learn an attack direction in steering vectors as indication of attacks. To validate the idea, the authors created a dataset of 4000 samples, include a 500 sample training set and three test sets. Training set uses images from COCO validation set and queries from the train set of HH-rlhf harm-set. Test sets use COCO test set and queries from the test set of HH-rlhf harm-set, D-corpus, and Harm-Data.
weaknesses: The evaluation design is very flawed.
- The evaluation does not include performance on benign queries. As a result, how likely the method generate false alarms is unknown.
- While the RADAR dataset is larger than previous ones and is generated using different attack queries, all the images are generated in the same way. As a result, the experiment results may not generalize to other attack methods.
- The evaluation also lacks experiments on adaptive attacks, i.e., whether it's possible to generate adversarial images that would lead to harmful output yet does not go beyond the learned threshold.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 2
questions: No further questions; the suggestions have been fully covered in the **Weaknesses** section.
soundness: 2
strengths: This research presents an innovative approach to detecting adversarial images by pinpointing the attacking direction within the hidden space of Vision-Language Models (VLMs). By focusing on this attacking direction, the method offers a fresh perspective that contrasts with traditional techniques, which often emphasize response discrepancies or image purification. Additionally, the introduction of the RADAR dataset, which captures a wide range of harmful responses, fills a crucial gap in evaluating VLM safety and significantly enhances the quality of related research.

The NEARSIDE method is thoughtfully designed to effectively distinguish between adversarial and benign inputs, demonstrating impressive accuracy and making it practical for real-time applications. Rigorous experiments, including comparisons with the leading JailGuard baseline on the RADAR dataset, confirm the method's effectiveness. The exploration of cross-model transferability and varying perturbation radii further highlights a deep understanding of how the method performs under different conditions.

Clarity is a strong point of this paper, as it articulately explains the vulnerabilities of VLMs and the urgent need for effective detection strategies. The description of the NEARSIDE method is straightforward, supported by clear illustrations that guide readers through the process. Furthermore, the well-structured presentation of the experimental setup and results makes it easy to follow, allowing readers to appreciate the significance and implications of the research findings.
summary: This paper introduces an innovative and efficient approach for detecting adversarial images in Vision-Language Models (VLMs), addressing their vulnerability to adversarial attacks. Current datasets and detection methods face limitations, either lacking diversity or being computationally heavy, particularly with low-visibility attacks. Key contributions of this work include a novel method for identifying the attacking direction in VLMs’ hidden space, which serves as a defense mechanism against adversarial images. This approach is integrated into the NEARSIDE method, which is efficient, requiring only a single forward pass and showing cross-model transferability. The authors also constructed the RADAR dataset, comprising 4,000 high-quality adversarial samples, surpassing previous datasets in scale and harmful response diversity. Through experiments on LLaVA and MiniGPT-4, NEARSIDE demonstrates high accuracy and significant speed improvements over baseline methods, highlighting its potential to enhance VLM security. While promising, further investigation could strengthen its generalizability across different VLMs.
weaknesses: 1.The concept of attacking direction is intriguing, yet the paper does not adequately address its stability across varying training datasets and model architectures. This is particularly relevant given that visual language models (VLMs) are frequently updated in practice. I suggest conducting experiments to evaluate how changes in training data or slight architectural modifications impact the stability of the attacking direction. A deeper analysis of these factors could enhance the understanding of NEARSIDE's detection performance.

2.The evaluation of the proposed method is primarily limited to LLaVA and MiniGPT-4, raising concerns about its applicability to other VLMs and LLMs with different architectures. While the initial cross-model transferability analysis is a good starting point, a more extensive evaluation is necessary. I recommend testing NEARSIDE across a broader spectrum of models, particularly those with varied visual encoders and training methodologies, to better assess its generality and potential need for modifications.

3.The current focus on detecting adversarial images generated by existing techniques overlooks the possibility of attackers developing new strategies to evade detection. As the landscape of adversarial attacks evolves, it is crucial to evaluate NEARSIDE's resilience against these future threats. Conducting simulations of more sophisticated adaptive attacks could provide insights into potential countermeasures and enhance the robustness of the detection method.

4.The paper acknowledges that the detection threshold, determined from the training set, may not be optimal across all datasets. However, it lacks a thorough investigation into dynamic threshold adjustment methods. I recommend exploring adaptive thresholding techniques or statistical analyses that could lead to more reliable threshold settings. Additionally, a detailed examination of how different thresholds affect false positive and false negative rates would be beneficial.

5.Although NEARSIDE is claimed to be efficient relative to baseline methods, the computational complexity analysis is somewhat superficial. The current focus on inference time neglects other critical factors, such as memory usage and training costs associated with extracting the attacking direction. A more comprehensive breakdown of computational costs throughout the process is essential. Discussing potential optimizations could further enhance NEARSIDE's scalability and performance in practical applications.

---

confidence: 3
contribution: 3
initial_rating: 3
presentation: 3
questions: 1. How does NEARSIDE perform on existing adversarial image datasets beyond RADAR?
2. What is the method's response when processing combinations of adversarial images with benign prompts?
3. Could the authors elaborate on NEARSIDE's limitations compared to multi-modal detection approaches?
soundness: 3
strengths: 1. The paper presents a well-structured and comprehensive approach to VLM safety.
2. The dual contribution (RADAR dataset and NEARSIDE method) provides valuable resources for the research community.
3. The methodology is clearly articulated and technically sound.
summary: This paper makes two main contributions to address the under-explored problem of adversarial image attacks on Visual Language Models (VLMs). First, it introduces RADAR, a novel large-scale dataset of adversarial images that elicit diverse harmful responses from VLMs. Second, it proposes NEARSIDE, an efficient detection method that leverages a distilled "attacking direction" vector from VLM hidden states to identify adversarial images. The effectiveness of NEARSIDE is validated through extensive experiments on LLaVA and MiniGPT-4, demonstrating strong performance and cross-model transferability. While the authors acknowledge limitations in covering all possible harmful contents and the specific scope of their detection method, their work provides valuable resources and insights for improving VLM safety against adversarial image attacks.
weaknesses: 1. Structural Issues:
   - The organization of supplementary materials could be improved by merging Appendix A and D to enhance readability and logical flow.

2. Limited Evaluation Scope:
   - The NEARSIDE method's evaluation is confined to the proposed RADAR dataset.
   - Cross-dataset validation would strengthen the method's generalizability claims.

3. Technical Oversight:
   - The method's behavior under mixed adversarial-benign inputs (adversarial image + benign text) requires clarification.
   - Comparison with multi-modal detection methods like JailGuard needs more detailed discussion.

4. Minor Issues:
   - Typographical error in Figure 6 caption ("exemplar")

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Check Weaknesses.
soundness: 3
strengths: The paper is well-structured and provides a clear motivation. The proposed method is straightforward, efficient, and effective, addressing a timely issue that underscores the need to develop robust VLM models moving forward. Additionally, the approach of extracting the 'attacking direction' (shifting from harmlessness to harmfulness) from the VLM embedding space is novel and clever. The introduction of the RADAR dataset is a valuable contribution.
summary: The paper addresses the vulnerability of Visual Language Models to adversarial image attacks. The authors introduce RADAR, a dataset of adversarial images with diverse harmful responses. Additionally, they propose NEAR-SIDE, a detection method that leverages a single vector derived from VLM hidden states, known as the 'attacking direction,' to distinguish adversarial images from benign ones. Experiments on the LLaVA and MiniGPT-4 models show that the method is effective, efficient, and transferable across models.
weaknesses: - The method does not address adversarial attacks aimed at generating benign yet contextually altered text, which may not contain harmful content but still alters the original intent. Could the authors discuss how their approach might be extended or modified to handle such cases, where adversarial examples produce benign text that misrepresents the original context?
- The threat model needs further clarification. Could the authors define the assumed threat model more explicitly, specifying the attacker’s level of access, capabilities, and the defender's available resources? Including this in a dedicated section would enhance clarity, particularly around the assumed white-box access to the victim model.
- The method currently relies on having both adversarial and benign samples to calculate the direction and threshold. Could the authors discuss how their approach might be adapted for scenarios where only single images are available for evaluation, or clarify any limitations in these cases?
- Why does the method focus on the last layer of the LLM? Could the authors justify this choice, ideally through an ablation study comparing performance across different layers?
- While the method appears efficient based on empirical results, some visualization of the "attacking direction" would be helpful, particularly in the context of cross-model transferability.
- Only one baseline is provided for comparison, which limits the evaluation of the method's effectiveness. Could the authors explain why other relevant baselines, such as the approach in Xu et al. [1] that leverages the semantic relationship between malicious queries and adversarial images, were not included? Expanding the baseline comparisons would strengthen the evaluation in a revised version of the paper.
[1] Xu, Yue, et al. 'Defending jailbreak attack in vlms via cross-modality information detector.'"
- What if the training and testing images come from different datasets? Could the authors evaluate the robustness of their method across diverse image distributions by conducting additional experiments using separate datasets for training and testing? This would help assess the method's generalizability.
- **Line 478:** Summarize the results rather than simply directing readers to the appendix.