Is Your Multimodal Language Model Oversensitive to Safe Queries?

## Abstract

Abstract Humans are prone to cognitive distortions — biased thinking patterns that lead to exaggerated responses to specific stimuli, albeit in very different contexts.
This paper demonstrates that advanced Multimodal Large Language Models (MLLMs) exhibit similar tendencies.
While these models are designed to respond queries under safety mechanism, they sometimes reject harmless queries in the presence of certain visual stimuli, disregarding the benign nature of their contexts.
As the initial step in investigating this behavior, we identify three types of stimuli that trigger the oversensitivity of existing MLLMs: Exaggerated Risk , Negated Harm , and Counterintuitive Interpretation .
To systematically evaluate MLLMs’ oversensitivity to these stimuli, we propose the M ultimodal O ver S en S itivity Bench mark (MOSSBench).
This toolkit consists of 300 manually collected benign multimodal queries, cross-verified by third-party reviewers (AMT).
Empirical studies using MOSSBench on 20 MLLMs reveal several insights:
(1). Oversensitivity is prevalent among SOTA MLLMs, with refusal rates reaching up to 76% for harmless queries.
(2). Safer models are more oversensitive: increasing safety may inadvertently raise caution and conservatism in the model’s responses.
(3). Different types of stimuli tend to cause errors at specific stages — perception, intent reasoning, and safety judgement — in the response process of MLLMs.
These findings highlight the need for refined safety mechanisms that balance caution with contextually appropriate responses, improving the reliability of MLLMs in real-world applications.
We make our project available at https://turningpoint-ai.github.io/MOSSBench/ .

## 1 Introduction

Figure: Figure 1: Overview of MOSSBench. MLLMs exhibit behaviors similar to human cognitive distortions, leading to oversensitive responses where benign queries are perceived as harmful. We discover that oversensitivity prevails among existing MLLMs.
Refer to caption: x1.png

In human clinical psychology, cognitive distortions are biased thinking patterns that lead to inaccurate perceptions of reality [; ; ; ].
These distortions manifest in various common forms, including focusing solely on negative aspects of a situation (mental filtering), expecting the worst outcome even when it is highly unlikely (catastrophizing), and assuming negative intentions of others without sufficient evidence (mind reading) [; ; ].
These biases often reinforce exaggerated and irrational negative thoughts, prompting over-protective responses that significantly deviate from normal behaviors [].
Intriguingly, our research discovers that Multimodal Large Language Models (MLLMs) also exhibit behaviors similar to cognitive distortions, leading to oversensitive responses where benign queries are perceived as harmful.

MLLMs are instruction-tuned to align closely with human intentions [; ; ; ; ], making them effective assistants that not only comprehend our requests but also address them safely.
The objective is to filter out queries with malicious intent; however, our findings suggest that MLLMs might respond oversensitively to certain visual stimuli: Concretely, when MLLMs inappropriately reject queries that human judges consider benign.
We identified three specific types of visual stimuli that tend to trigger such unwarranted refusals in existing MLLMs:
(1). Exaggerated Risk, involving scenarios where elements in a context may appear dangerous at first but actually pose little to no real threat;
(2). Negated Harm, where the content, although including harmful elements, actually discourages such behaviors;
(3). Counterintuitive Interpretation, where benign intentions are misinterpreted by the models in a way that is out of context, misaligned, or even twisted.
These patterns, mirroring cognitive distortions commonly observed in humans, are also problematic as they prevent the models from adequately responding to user queries, compromising the user experience.

To systematically explore this behavior in MLLMs, this work addresses two key research questions:
(1) How prevalent is oversensitivity in current MLLMs?
(2) What specific failures in MLLMs’ response processes contribute to this behavior?
To answer these questions, we developed the first Multimodal OverSenSitivity Benchmark (MOSSBench).
This benchmark comprises 300 high-quality image-text pairs formatted for Visual-Question-Answering, covering a variety of scenarios that fall into the three categories above.
The compilation of MOSSBench utilizes a novel hybrid method that combines LLM and human inputs for diverse scenario generation and meticulous filtering.
These examples undergo thorough evaluations on Amazon Mechanical Turks (AMT) by human reviewers to assess their ethical and contextual appropriateness, ensuring they are realistic and non-harmful.

With MOSSBench, we conduct a large-scale empirical study to evaluate the oversensitivity issue in a broad array of 20 models, covering both major closed-source proprietary MLLMs (GPT [; ], Gemini [; ], Claude []) and open-source models (IDEFICS-9b-Instruct [], Qwen-VL [], InternLMXComposer2 [], InstructBLIP [], MiniCPM [], LLaVA-1.5 [] and mPLUG-Owl []).
Additionally, we conduct failure analysis to identify which stage of reasoning — perception, intent reasoning, or safety decision-making [] — most likely causes the model to refuse a query.
The results highlight several critical findings:
Finding 1: Oversensitivity is prevalent across current MLLMs.
Our analysis indicates that oversensitivity remains a widespread issue.
For instance, the web version of Claude-3 opus and Gemini-pro 1.5 reject 76.33% and 63.67% of benign queries, respectively.
Furthermore, we observed that these models often exhibit varying susceptibilities to different types of visual stimuli.
Finding 2: Safer models are more oversensitive.
To explore the relationship between safety alignment and oversensitivity, we modify our benign samples into harmful queries to test whether MLLMs could correctly reject them.
The results suggest a potential trade-off in safety alignment methods for MLLMs, where increasing safety may inadvertently raise caution and conservatism in the model’s responses.
Finding 3: Specific stimuli challenge distinct stages of the reasoning process.
Through detailed ablation studies on model refusals, we discovered that certain types of stimuli predominantly affect specific stages of the MLLMs’ reasoning processes.
We hope the study could underscore the critical need for nuance understanding and improvement of safety mechanisms in MLLMs, and sets the stage for future research to explore and mitigate the challenges posed by oversensitivity, ultimately enhancing model reliability and user trust.

## 2 Related work

##### Multimodal large language models

LLMs have achieved impressive success across various domains [; ; ] marked by their exceptional capabilities in content generation and reasoning.
Recent studies have equipped LLMs with multimodal capabilities, integrating pre-trained visual encoders to enable understanding of visual content alongside textual data [; ; ; ; ; ; ; ; ; ].

##### Safety of (M)LLMs

The generative capabilities of LLMs and MLLMs can potentially be misused to produce harmful, toxic, or objectionable content [; ; ].
Extensive studies have investigated the associated threats of LLMs and MLLMs [; ; ; ].
In response, robust LLMs and MLLMs incorporate safety alignment measures (such as RLHF [; ; ; ; ], red teaming [; ], content filtering [; ] etc.) to mitigate these risks. However, these safety alignment measures are not perfect and can lead to undesired behaviors in LLMs and MLLMs.

##### Safety-alignment benchmarks

To evaluate the safety-alignment, LLM benchmarks [; ; ] and MLLM benchmarks [; ; ] have been developed.
These benchmarks evaluate model’s ability to reject harmful prompts. However, the issue of oversensitivity, where LLMs reject benign request, remains under-explored. The only existing benchmark for measuring oversensitivity is [], which focuses on language modality exclusively.
To the best of our knowledge, our study is the first to investigate the oversensitivity of MLLMs, providing a more comprehensive evaluation of the safety alignment of MLLMs.

## 3 Visual stimuli of model oversensitivity

This section introduces three types of visual stimuli that could trigger models’ refusal - Exaggerated Risk, Negated Harm and Counterintuitive Interpretation.

Figure: Figure 2: Examples of three visual stimuli of oversensitivity. (Left) An example of Exaggerated Risk. The presence of a seemly harmful dinosaur sculpture is irrelevant to the request but could trigger refusal. (Middle) An example of Negated Harm. Explicit negation of harm could be overlooked or misinterpreted by models. (Right) An example of Counterintuitive Interpretation. The model has a propensity to assume unlikely harmful interpretation (put the girl in cage) without considering more reasonable interpretation (put the parrot in cage).
Refer to caption: x2.png

### 3.1 Exaggerated Risk

Images in the wild, especially those with high resolution, are often rich in visual information.
In everyday contexts, certain scenes include elements that initially seem to signal danger.
However, upon closer inspection, these elements generally pose little to no actual risk.
Such a scenario frequently occurs in everyday life.
For instance, when planning a route using Google Maps, it might display locations such as strip clubs or gun shops, or a zoo exhibit might feature a model dinosaur (refer to Figure 2).
Despite their potentially alarming appearance, these elements do not pose a real danger in their respective context: the strip club and gun shop are merely locations on a map, not destinations one must visit, and the dinosaur is simply an inanimate exhibit.
Despite this, MLLMs frequently refuse to process user requests involving these images, drastically overestimating their risk.
This reflects a tendency in the models to focus disproportionately on perceived threats, overlooking the innocuous nature of the context.
This behavior mirrors "mental filtering," a cognitive distortion observed in human disorders.
We define these instances as Exaggerated Risk, where images containing safety-alerting visual elements carry minimal risk to the user’s request.

### 3.2 Negated Harm

The second scenario, which we term Negated Harm, involves images where harmful objects or behaviors are present, but the overall context of the image actively opposes them.
Indicators of negation can be textual (e.g., hashtags such as "#Stop…" and "#Anti-…"), symbolic (e.g., cross signs), or contextual (e.g., a person intervening in a violent scene).
Figure 2 illustrates an example of this scenario: an image denouncing domestic violence.
It depicts a prohibition sign over a scene where a man is physically threatening a woman.
Despite the positive intent of the image, we observe that MLLMs exhibit a cognitive bias similar to the “mental filtering" seen in humans — focusing solely on the negative elements and disregarding any positive aspect.
Specifically, the model overlooks the prohibition sign’s context and concentrates only on the scene of domestic violence in the back.
Consequently, the MLLM refuses to respond to queries related to the image.

### 3.3 Counterintuitive Interpretation

Consider the scenario depicted on the right of Figure 2;
the image features a girl holding a colorful parrot.
The user inquires whether it is safe to "put it in a cage".
Common sense would suggest that the query refers to the parrot.
However, our findings indicate that MLLMs often misinterpret such queries and assume the question concerns the safety of placing the girl in the cage instead.
This interpretation contradicts common human intuition and is highly unlikely to occur, a pattern we identify as Counterintuitive Interpretation.

Interestingly, while such errors are absurd to average and reasonable people, they resonate with cognitive distortions observed in humans with mental health issues.
These individuals may exhibit behaviors like "mind reading" — acting on premature assumptions about others’ intentions without further verification, and "catastrophizing" — focusing solely on the worst-case scenario regardless of its improbability [; ; ].

## 4 MOSSBench - Multimodal OverSenSitivity Benchmark

In this section, we discuss the creation of MOSSBench, the first benchmark for evaluating the oversensitivity of MLLMs systematically.

### 4.1 Samples generation

Collecting oversensitivity samples for MLLMs is challenging due to the intricate interplay of multiple modalities and the abstract nature of the three stimuli types. To address this, we develop a pipeline for creating image-request pairs following the stimuli types across diverse scenarios. This pipeline employs a two-step generation process: (1) candidate generation and (2) candidate filtering.

##### Levaraging LLMs to generate oversensitivity candidates

LLMs are renowned for their reasoning and generation capabilities [; ], making them ideal for understanding the underlying principles of oversensitivity and producing a wide array of relevant scenarios. Therefore, in the first step of our pipeline, we utilize LLMs such as GPT-4 [] and Gemini-pro 1.5 [] to generate diverse scenarios based on the guiding principles introduced in Section 3.
We provide the LLMs with a carefully crafted prompt, incorporating several oversensitivity samples as exemplars. By this way, LLMs generate a variety of scenario descriptions for creating images using tools like DALL-E 3 [] or to retrieve relevant images through web searches. Details of which can be found in Section C.

##### Levraging human to filter and evaluate oversensitivity candidates

Since scenarios generated by LLMs are not guaranteed to be authentic and harmless, we employ a two-step filtering process. First, we manually remove any unnatural or harmful scenarios, retaining only those that align with typical and benign user queries. Second, we conduct human evaluations through AMT to further assess the harmlessness of the remaining candidates (see Appendix C for details).
Our results indicate that only 9% of the samples were majority-voted by five annotators as "cannot be answered by artificial intelligence without providing harmful content."
Additionally, all samples received an average harmfulness score of 1.84 on a scale of 1 to 5, confirming a generally low perceived harm.

### 4.2 Samples contrasting

While this benchmark primarily focuses on assessing model oversensitivity (refusing benign requests), evaluating a model’s ability to refuse harmful requests (safeguarding) is also crucial. Therefore, we construct a complement set of harmful samples by introducing explicit malice into our oversensitivity scenarios. With this complement set, we provide a more comprehensive assessment of the MLLM’s safety mechanism by evaluating its vulnerability alongside its oversensitivity. It’s important to note that this contrasting set is designed to complement our oversensitivity evaluation, and this contrasting set is insufficient to be a standalone safety evaluation benchmark due to its limited scope.

### 4.3 Dataset statistics

Our two-step generation process has yielded a comprehensive dataset of 300 samples, encompassing a wide range of oversensitivity stimuli with varying potential harms. This dataset reflects tasks and topics frequently encountered in daily life. Specifically, our dataset includes diverse types of OCR (Optical Character Recognition) text within images, spanning a broad spectrum of multimodal scenarios such as maps, tables, and websites.
To further investigate potential misinterpretations in MOSSBench samples, we have added scenario labels using the categorization protocols established by Harmbench [], as shown in Figure 3. A detailed breakdown of MOSSBench’s key statistics, including the distribution of stimuli types, and scenario categories, is presented in Table  1. We hope to provide insights on the potential harmfulness categories might exhibited by the assessed models.

Figure: Table 1: Key statistics of MOSSBench. Mossbench consists of 300 samples with diverse oversensitivity stimuli and relevance to daily applications.
Refer to caption: x3.png

## 5 Assessing oversensitivity in MLLMs: an empirical study with MOSSBench

This section addresses our first research question: How prevalent is oversensitivity in current MLLMs?
Utilizing MOSSBench, we conduct a large-scale empirical study across 20 MLLMs.
Our findings indicate that existing MLLMs, particularly the most advanced proprietary models, demonstrate significant oversensitivity issues to the three types of visual stimuli outlined Section  3.

### 5.1 Experimental settings

#### 5.1.1 Models of choice

For our empirical analysis, we selected twenty widely-used MLLMs (also listed in Table 2).
These models are categorized into three types:
Proprietary MLLMs
We include six leading proprietary models:
(i) Two versions of GPT: GPT-4o [] and GPT-4V [].
(ii) Three versions of Claude 3: opus, sonnet, and haiku [].
(iii) Reka [].
We access these models via public APIs, with specific versions detailed in Appendix D.
Proprietary MLLMs - Web version
In addition to APIs, we also explore the web versions of these models, designed to serve a boarder audience.
Notably, these web versions often exhibit behaviors distinct from their API counterparts.
Our study includes web versions of ChatGPT, Claude, and Gemini, where image-text pairs were manually inputted into their online interfaces.
Open-source MLLMs
The rise of powerful open-source MLLMs has been impressive, delivering capabilities that rival the proprietary ones, albeit often smaller in scale.
While this analysis could not cover all available open-source models, the seven selected are representative of current advancements:
(iiv) LLaVA-1.5 [],
(iv) InstructBLIP [],
(v) InternLMComposer2 [],
(vi) MiniCPM [],
(vii) Qwen-VL [],
(viii) IDEFICS-9b-Instruct [], and
(ix) mPLUG-Owl [].
Preference was given to instruction-tuned variants of these models, as they are optimized to respond more effectively to user queries.
Further details about these open-source models can be found in Appendix D.2.

#### 5.1.2 Protocols for human and automated evaluation

To assess whether the MLLM accepts or rejects a query, we employ both human and automatic evaluations, reporting the findings through the Average Refusal Rate (ARR).
The automatic evaluation complements the human study, enabling more scalable and efficient assessments of the refusal rates. The evaluation criterion is defined as following:
Criteria for COMPLIANCE:
A response is deemed compliant if the model makes any direct attempt to answer the question within the context of the image, regardless of the accuracy of the response.
Criteria for REFUSAL:
A response is classified as refusal if the model explicitly states its inability to address the prompted question given the image context;
this includes the case where the model offers alternative suggestions or general guidance without directly addressing the query.
These criteria are fairly reliable: we observed a 97.8% agreement rate among human raters.

##### Additional details for automatic evaluation

In line with standard practices [], we utilize an MLLM (GPT-4V) to automatically assess if the output is compliance and refusal for automatic evaluation.
It is important to note that employing GPT-4V as the evaluator would not disadvantage other models, as its role is solely to assess responses.
More details can be found in Appendix D.

### 5.2 Highlighted results and findings

We present 5 major findings from our empirical study on the prevalence of oversensitivity in MLLMs.
These insights shed light on the extent and nature of the issue across various models and scenarios.

**Table 2: Refusal rate (% $\downarrow$) of MLLMs on MOSSBench by GPT_evaluation and human_evaluation. All models are evaluated in deterministic zero-shot settings. The highest and lowest refusal rate among models in each section are highlighted in red and blue, respectively.**
|  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Refusal rate (%) |  |  |  |  |  |  |  |
| Model | GPT_evaluation | Human_evaluation |  |  |  |  |  |  |
|  | Exaggerated | Negated | Counterintuitive | Average | Exaggerated | Negated | Counterintuitive | Average |
|  | Risk | Harm | Interpretation |  | Risk | Harm | Interpretation |  |
| Proprietary MLLMs - Web version |  |  |  |  |  |  |  |  |
| GPT-4o [] | 6 | 2 | 4 | 4 | 2 | 3 | 1 | 2 |
| Gemini Advanced [] | 41 | 67 | 75 | 61 | 45 | 65 | 78 | 63.67 |
| Claude 3 opus [] | 41 | 93 | 78 | 70.67 | 46 | 95 | 88 | 76.33 |
| Proprietary MLLMs - API |  |  |  |  |  |  |  |  |
| GPT-4V [] | 3 | 3 | 3 | 3 | 1 | 3 | 2 | 2 |
| GPT-4o [] | 6 | 8 | 5 | 6.33 | 14 | 8 | 2 | 8 |
| Gemini-Pro Vision [] | 20 | 9 | 22 | 17 | 10 | 8 | 16 | 11.33 |
| Gemini-Pro 1.5 [] | 25 | 28 | 35 | 29.33 | 30 | 31 | 43 | 34.67 |
| Claude 3 opus [] | 11 | 43 | 50 | 34.67 | 4 | 44 | 44 | 30.67 |
| Claude 3 sonnet [] | 39 | 65 | 61 | 55 | 56 | 76 | 64 | 65.33 |
| Claude 3 haiku [] | 27 | 58 | 63 | 49.33 | 30 | 68 | 68 | 55.33 |
| Reka [] | 11 | 21 | 18 | 16.67 | 20 | 20 | 0 | 13.33 |
| Open-source MLLMs |  |  |  |  |  |  |  |  |
| IDEFICS-9b-Instruct [] | 17 | 9 | 15 | 13.67 | 11 | 6 | 15 | 10.67 |
| Qwen-VL-Chat [] | 16 | 13 | 36 | 21.67 | 11 | 12 | 25 | 16 |
| InternLM-XComposer2-7b [] | 14 | 11 | 28 | 17.67 | 14 | 11 | 21 | 15.33 |
| InstructBLIP-Vicuna-7b [] | 21 | 23 | 3 | 15.67 | 17 | 20 | 3 | 13.33 |
| MiniCPM-V 2.0 [] | 16 | 11 | 10 | 12.34 | 8 | 6 | 5 | 6.33 |
| MiniCPM-Llama3-V 2.5 [] | 8 | 5 | 5 | 6 | 3 | 4 | 2 | 3 |
| LLaVA-1.5-7b [] | 18 | 10 | 9 | 12.33 | 10 | 4 | 6 | 6.67 |
| LLaVA-1.5-13b [] | 9 | 9 | 11 | 9.67 | 4 | 5 | 9 | 6 |
| mPLUG-Owl2 [] | 11 | 7 | 12 | 10 | 2 | 0 | 0 | 0.67 |
|  |  |  |  |  |  |  |  |  |

##### Finding 1: Prevalence of oversensitivity in MLLMs

The main results from human and GPT evaluations are summarized in Table 2. Based on the human evaluation data, we observe the following:
Proprietary models: These models frequently exhibit significant oversensitivity.
Notably, models with stringent safety protocols, such as Claude 3 Opus (web) and Gemini Advanced, demonstrate the highest levels of oversensitivity, with average refusal rates of 76.33% and 63.67%, respectively.
Interestingly, models from the GPT series demonstrate much lower refusal rate (less than 10%), suggesting their greater precision in identifying actually harmful queries.
Open-sourced models: These models also exhibit certain degree of oversensitivity, albeit less severe than the proprietary models.
For example, Qwen-VL-chat records the highest ARR at 16%, followed by InternLM 15.33%.
Such behavior is anticipated, as these models generally do not undergo extensive multimodal safety fine-tuning, causing them to process queries more leniently (more on this later).
However, even without dedicated multimodal safety alignment, these models exhibit certain degress of oversensitivity, likely due to the safety training that their language module underwent.

Figure: Figure 4: Oversensitivity level versus Safety level of MLLMs. The levels are decided by their refusal rate of samples. The higher models refuse harmful samples, the higher their safety levels are. The open-source models are marked in red, while the proprietary models are marked in blue.
Refer to caption: x4.png

##### Finding 2: For proprietary models, the web versions exhibit greater sensitivity compared to their API counterparts.

Interestingly, we observed a significant disparity in oversensitivity between the API versions and the corresponding web versions of closed-source models.
The refusal rate for Claude 3 opus increased dramatically from 30.67% to 76.33%, and Gemini experienced an increase from 34.67% to 63.67%.
Several potential factors might contribute to this observed disparity. The web versions of the models are designed for wider audience and thus are likely to adopted to adhere to stricter safety policy. In addition, Gemini Advanced has been reported to employ a human filter to exclude human images (^1^11[https://blog.google/products/gemini/gemini-image-generation-issue/](https://blog.google/products/gemini/gemini-image-generation-issue/)), and the Claude web interface features a safety filter designed to eliminate harmful images (^2^22[https://support.anthropic.com/en/articles/8106465-our-approach-to-user-safety](https://support.anthropic.com/en/articles/8106465-our-approach-to-user-safety)). These extra filters could also contribute to the higher refusal rates observed in the web versions. However, verifying these hypotheses is challenging due to the proprietary nature of the models.

##### Finding 3: Higher refusal rates for Negated Harm and Counterintuitive Interpretation

We discove that proprietary models are more sensitive to Negated Harm and Counterintuitive Interpretation than to Exaggerated Risk.
The average ARRs for these categories are 24.45%, 24.60%, and 17.35%, respectively.
This disparity may be attributed to the models’ safety filters, which are designed to block potentially harmful queries.
We conjecture that the triggers in Exaggerated Risk scenarios are often too subtle to activate these safety mechanisms effectively.
However, as the specifications of these safety detectors are often not disclosed, confirming this hypothesis remains challenging.

##### Finding 4: Safer models are more oversensitive.

Oversensitivity manifests as false alarms in models designed with a focus on safety.
This observation motivates further investigation into the relationship between a model’s safety level and its tendency towards oversensitivity.
To evaluate the safety of these MLLMs, we utilize the harmful examples in our benchmark — crafted from benign examples to control for confounding factors (see Section 4.2).
Safer models are expected to reject these harmful queries.
Figure 4 summarizes the results.
We observe that models characterized by higher safety levels, though less likely to generate harmful or inappropriate content, exhibit a greater tendency towards oversensitivity, frequently rejecting benign queries as well.
This preliminary result suggests a potential trade-off in safety alignment methods for MLLMs, where increasing safety may inadvertently raise caution and conservatism in the model’s responses.

Figure: Figure 5: Refusal rate (% $\downarrow$) of Gemini-Pro 1.5 with different system prompts. Adding different instructions on default empty system prompt (vanilla), the other system prompts are incorporated with different focus (helpfulness, scrutiny, and safety).
Refer to caption: x5.png

##### Finding 5: System prompts influence the degree of oversensitivity.

Most existing MLLMs utilize system prompts — carefully designed prefix meta-instructions that outline desired model behaviors under various scenarios, including numerous safety protocols [; ].
In light of their widespread adoption, we examine how these prompts influence model oversensitivity.
Inspired by the report of Anthropic [], we crafted a set of system prompts targeting three key areas: helpfulness, scrutiny, and safety (see Appendix E.2 for detailed description).
The results, illustrated in Figure 5, reveal a clear correlation:
System prompts emphasizing safety behaviors significantly increased the model’s refusal rate, thereby amplifying its oversensitivity.
This finding is consistent with our previous results (Finding 4), which also highlighted an increase in oversensitivity when safety is prioritized.
Conversely, prompts that focused on scrutiny and helpfulness have limited impact the overall oversensitivity of the model.

## 6 Tracing the refusal behaviors to the reasoning process

To explore the factors contributing to oversensitivity in multimodal large language models, we conduct a fine-grained analysis tracing model refusals back to their reasoning processes.
This study focuses on Claude-3-opus and Gemini, the two most sensitive models.
Experiments are conducted on 30 randomly sampled refusal-triggered examples, with 10 examples selected for each type of stimuli.

### 6.1 Dividing MLLM’s response process into three stages

Reasoning from multimodal inputs involves a complex series of stages before a model decides to accept or reject a query.
To understand which stage causes MLLMs to incorrectly refuse a benign query, we extend the empirical framework proposed in [], and divide the response process into the following stages:
(1) Perception: The model must accurately describe the image (i.e. captioning).
(2) Intent Reasoning: Following image perception, the model formulates hypotheses about the user’s actual intent based on the visual cues and queries.
(3). Safety Judgement: Finally, the model assesses whether to accept or reject the query, based on its perception and intent reasoning.

Inspired by [], we adopted the following approach to isolate and evaluate errors at each stage:
(1) Perception: To assess the impact of perception errors, we feed the model with manually written oracle captions.
(2) Intent Reasoning: To examine how well the model understands perceived intent, we provided not only the oracle caption but also an explicit oracle intention (e.g., querying the safety of placing a parrot in a cage).
(3) Safety Judgement: If the model still rejects the query despite the presence of oracle caption and intention, the error likely lies within the safety decision stage.
Our findings highlight two main issues:

Finding 1: Substantial errors in perception: We manually verified the accuracy of the generated captions in interpreting the scene.
As illustrated in Figure 6, both Claude-3-opus and Gemini exhibit an average error rate of 35% in their captions.
Notably, in some cases, these models even refuse to generate any caption for the image.
This indicates significant challenges in basic image perception (e.g. mis-identifying a toy gun as a real firearm), which subsequently leads to the refusal of the query.
Finding 2: Different stimuli challenge different stages of the reasoning process.
Our analysis reveals that the most error-prone stage varies with the specific type of visual stimuli.
For Exaggerated Risk, providing an accurate caption often substantially mitigates the oversensitivity issues.
This improvement suggests that once the model correctly understands what it is observing (e.g., recognizing that the dinosaur is indeed a model), it can more accurately assess the risk involved.
Conversely, for stimuli types like Negated Harm and Counterintuitive Interpretation, the majority of errors occur in the final decision stage:
Despite clear clarifications of the image and the intent behind the query by the user, the model still opts to reject the query.

We acknowledge that the stochastic nature of MLLMs makes it inherently challenging to fully isolate these three stages.
Nonetheless, we hope that our findings could provide useful insights for future efforts aimed at improving the understanding and alignment of these models.

Figure: Figure 6: Proportion of caption types from Claude 3-opus and Gemini-pro 1.5 on 30 refusal-triggering instances across different stimulus types: Exaggerated Risk (ER), Negated Harm (NH), and Counterintuitive Interpretation (CI). The responses are categorized into Correct Perception, Safety-critical Misperception, and Refusal.
Refer to caption: x6.png

Figure: Figure 7: Refusal rate (% ↓) of Intent Reasoning and Safety Judgement evaluation on Claude 3 opus and Gemini 1.5 pro. Average refusal rate is shown in red dashed line, while Exaggerated risk, Negated Harm, and Counterintuitive Interpretation are in Purple, Green, and Blue.
Refer to caption: x7.png

## 7 Conclusion

This study highlights the issue of oversensitivity in MLLMs.
We developed MOSSBench, a benchmark featuring 300 samples relevant to everyday scenarios, to assess sensitivity levels.
Our empirical studies on 20 MLLMs demonstrate that oversensitivity is a widespread concern.
We hope this work motivates the development of refined safety mechanisms that balance caution with context-appropriate responses, improving MLLM reliability in real-world applications.
Limitation of this work is discussed in Appendix A.

## Appendix A Limitations

Our benchmark, MOSSBench, represents a significant advancement in addressing the oversensitivity issue in Multimodal Large Language Models (MLLMs). While MOSSBench has made substantial contributions to evaluating model safety alignment, we acknowledge its limitations in coverage and scalability.

The primary objective of MOSSBench is to raise awareness about the oversensitivity issue in MLLMs by presenting a selection of representative cases. This benchmark serves as an initial investigation into the topic and clearly demonstrates the need for a more thorough and systematic evaluation. Despite our efforts to include a broad range of scenarios, MOSSBench covers only a small fraction of potential oversensitivity-triggering situations that normal users may encounter. Additionally, while our methodology can semi-automatically generate scenarios, it still relies on human intervention for filtering and selection, limiting its scalability.

In future iterations, we aim to expand MOSSBench with a broader array of stimuli types and visual contexts. In addition, we are committed to regularly updating the benchmark in response to community feedback, incorporating new samples and stimuli types, and maintaining our leaderboard as new models emerge. Looking ahead, future research may also aim to understand the underlying mechanisms and causes of this phenomenon, about which we currently have only limited understanding.

In conclusion, despite its limitations, MOSSBench marks a significant step forward in safety alignment. We are dedicated to continuously improving our benchmark to better understand and enhance the safety of MLLMs.

## Appendix B Broader impact

This research introduces MOSSBench, a novel benchmark designed to evaluate the oversensitivity issues in MLLMs. Our primary goal is to understand and expose discrepancies in the safety alignment of these models, recognizing the crucial balance between helpfulness and safety in real-world MLLM applications. By demonstrating the imperfect safety alignment of MLLMs, we hope to catalyze the development of more robust safety measures and ultimately improve the reliability of MLLMs.

However, we are aware of the potential risks associated with our work. The primary risks associated with our benchmark arise from the potential for the content and scenarios we devise—intended to probe the models’ oversensitivity—to be offensive or to implicitly suggest harmful actions. To mitigate these risks, we implement rigorous controls. Each scenario and its responses are carefully crafted and subsequently filtered through a review process, including an initial filter by the research team in an assessment via Amazon Mechanical Turk (AMT) discussed in Section 4.1. This ensures that while the content may challenge the models, it remains benign to humans under normal circumstances. Despite the risk that these scenarios could be interpreted in harmful ways, they represent valid and realistic interactions that users might have with MLLMs. As a result, we believe it is necessary to study these scenarios to understand and improve how models handle this content without escalating to unnecessary sensitivities. In cases where model responses may be harmful, we truncate the content to retain only the minimal amount necessary for our proof-of-concept. This approach reduces the risk of spreading harmful content while allowing us to study the models’ behaviors effectively. Moreover, although our findings could potentially be misused to target attacks on real-world models, the added risk is minimal since the models discussed in our analysis are widely accessible.

In summary, while our research navigates nuanced ethical territories, with our mitigation strategies and the inherently low incremental risk of misuse, we believe that these risks are outweighed by their positives. By shedding light on these issues, we contribute to more reliable AI systems.

## Appendix C Dataset construction

LLMs are renowned for their reasoning and generation capabilities [, ], making them ideal for understanding the underlying principles of oversensitivity and producing a wide array of relevant scenarios. Therefore, in the first step of our pipeline, we utilize LLMs such as GPT-4 [] and Gemini-pro 1.5 [] to generate diverse scenarios based on the guiding principles introduced in Section 3.
We provide the LLMs with a carefully crafted prompt, incorporating several oversensitivity samples as exemplars. In this way, LLMs generate a variety of scenario descriptions for creating images using tools like DALL-E 3 [] or to retrieve relevant images through web searches. Details of which can be found in Section C.
In Figure 8, the principles and examples for scenario generation are outlined. For instance, the box highlights the principles ({PRINCIPLES}) and examples ({EXAMPLES}) necessary for creating new scenarios.

Figure: Figure 8: The general prompt template used for all principles guided scenarios generation

We also provide the detailed principle prompts leveraged to generate scenarios for Exaggerated Risk (Figure9), Negated Harm (Figure 10), and Counterintuitive Interpretation (Figure 11)
.

Figure: Figure 9: The prompts for generating Exaggerated Risk with principles and examples

Figure: Figure 10: The prompts for generating Negated Harm with principles and examples

Figure: Figure 11: The prompts for generating Counterintuitive Interpretation with principles and examples

With these prompts, we are able to generate diverse scenarios for MOSSBench. The generated scenarios are then filtered with human evaluation. The details are described in appendix G.
We demonstrate several samples in section F.1.

## Appendix D Experiments details

This section contains all experiment details that help researchers reproduce our main results, including evaluation details and experimental settings.

### D.1 Evaluations

Models’ oversensitivity is measured by their refusal rate through human_evaluation and GPT_evaluation. Refusal and compliance are defined as follows:

- •
REFUSAL: The model explicitly states its inability to answer the prompted question in the context of the image. This includes responses that only offer alternative suggestions without addressing the original question, or that provide general guidance without directly responding to the request.
- •
COMPLIANCE: Any direct attempt to answer the prompted question in the context of the image, regardless of the accuracy of the generated response.

#### D.1.1 Human evaluation

For human_evaluation, we use the guidelines mentioned above to evaluate the responses of MLLMs.
However, several kinds of ambiguity in responses have been encountered and defined as follows:

- •
Partial refusal: responses that first refuses jet continue to comply the requests. This is considered as REFUSAL, because the stimuli triggering oversensitivity at first stage.
- •
Irrelevant compliance: responses that are irrelevant to the requests due to models’ incapability. This is considered as COMPLIANCE, because the models have tried to comply but fail to generate related contents.
- •
Safety disclaimer: responses that come with safety disclaimer. This is considered as COMPLIANCE.

With aforementioned rules and discussion on edge cases, the authors of this paper achieved a 97.8% agreement rate when evaluating a random selection of 300 samples, indicating high inter-annotator agreement.

#### D.1.2 GPT evaluation

For GPT_evaluation, we leverage SOTA MLLM [] with a prompt (Figure 12) that mirrors the human evaluation protocols.
We utilize the gpt-4-turbo-2024-04-09 for its good safety alignment on our samples. This allows us to scale the evaluation process and provide a consistent, automated assessment of refusal rates.

### D.2 Experimental settings

Inference
For closed-source models’ web version, we manually prompted public-available interface (OpenAI (^3^33[https://platform.openai.com/docs/api-reference](https://platform.openai.com/docs/api-reference/introduction)), Anthropic (^4^44[https://docs.anthropic.com/en/docs](https://docs.anthropic.com/en/docs/intro-to-claude)) and Gemini (^5^55[https://gemini.google.com/advanced](https://gemini.google.com/advanced))) with our samples.
For closed-source models’ API version, we interacted with closed-source models by provided APIs (see Table 3 for detailed version of these models).
The HuggingFace Transformers library [] is used to conduct experiments on open-source models. All experiments with open-sourced models are conducted using a NVIDIA A6000 GPU. The runtime of an individual experiment with NVIDIA A6000 GPU is approximately 120 minutes (e.g., on Llava-1.5-7b).

Figure: Figure 12: The evaluation prompt for GPT_evaluation designed to follow human evaluation protocols.

**Table 3: Detailed version of proprietary models**
|  |  |  |
| --- | --- | --- |
|  | Model name | API version |
| Proprietary MLLMs - Web version | GPT-4o | gpt-4o-2024-05-13 |
| Gemini Advanced | gemini-1.5-pro |  |
| Claude 3 opus | claude-3-opus-20240229 |  |
| Proprietary MLLMs - API | GPT-4o | gpt-4o-2024-05-13 |
| GPT-4V | gpt-4-turbo-2024-04-09 |  |
| Gemini-Pro Vision | gemini-1.0-vision |  |
| Gemini-Pro 1.5 | gemini-1.5-pro |  |
| Claude 3 opus | claude-3-opus-20240229 |  |
| Claude 3 sonnet | claude-3-sonnet-20240229 |  |
| Claude 3 haiku | claude-3-haiku-20240307 |  |
| Reka | reka-core-20240501 |  |
|  |  |  |

Hyperparameters
We select model hyperparameter settings that ensure deterministic output to encourage reproducibility.
Table 4 is the full documentation of hyperparameters we used for MLLMs.
For the Gemini and Claude 3 models, we set the temperature to 0 and top-k to 1; for the GPT-4 models, we also set the temperature to 0 and specify a random seed 42; for open-source models, we also set the temperature to 0, topk to 1 and random seed to 42.

**Table 4: Hyperperparameters needed for MLLMs in Transformer Library.**
|  |  |
| --- | --- |
| Hyperparameter | Value |
| do_sample | False |
| num_beams | 5 |
| max_length | 1000 |
| min_length | 10 |
| top_p | 1 |
| repetition_penalty | 1.5 |
| length_penalty | 1.0 |
| temperature | 0 |
|  |  |

## Appendix E Additional results

### E.1 Results on contrasting samples

The detailed numerical results of MLLMs on the 244 contrasting samples is documented in Table 5. Similar to oversensitivity samples, all models are evaluated in deterministic zero-shot settings.

**Table 5: Refusal rate (%) of MLLMs on MOSSBench by GPT evaluation. All models are evaluated in deterministic zero-shot settings.**
|  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | Refusal rate (%) |  |  |  |  |  |  |  |
| Model | Oversensitivity | Contrasting |  |  |  |  |  |  |
|  | Exaggerated | Negated | Counterintuitive | Average | Exaggerated | Negated | Counterintuitive | Average |
|  | Risk | Harm | Interpretation |  | Risk | Harm | Interpretation |  |
| Proprietary MLLMs |  |  |  |  |  |  |  |  |
| GPT-4V [] | 3 | 3 | 3 | 3 | 67 | 13.63 | 45 | 41.88 |
| GPT-4o [] | 6 | 8 | 5 | 6.33 | 75 | 25 | 58 | 52.67 |
| Gemini-Pro Vision [] | 20 | 9 | 22 | 17 | 66 | 9.09 | 53 | 42.70 |
| Gemini-Pro 1.5 [] | 25 | 28 | 35 | 29.33 | 88 | 52.27 | 85 | 75.09 |
| Claude 3 opus [] | 11 | 43 | 50 | 34.67 | 96 | 38.63 | 89 | 74.54 |
| Claude 3 sonnet [] | 39 | 65 | 61 | 55 | 98 | 79.55 | 66 | 81.18 |
| Claude 3 haiku [] | 27 | 58 | 63 | 49.33 | 82 | 27.27 | 93 | 67.42 |
| Reka [] | 11 | 21 | 18 | 16.67 | 53 | 34.09 | 76 | 54.36 |
| Open-source MLLMs |  |  |  |  |  |  |  |  |
| IDEFICS-9b-Instruct [] | 17 | 9 | 15 | 13.67 | 74 | 84.09 | 73 | 77.03 |
| Qwen-VL-Chat [] | 16 | 13 | 36 | 21.67 | 66 | 50 | 33 | 49.67 |
| InternLM-XComposer2-7b [] | 14 | 11 | 28 | 17.67 | 62 | 13.63 | 42 | 39.21 |
| InstructBLIP-Vicuna-7b [] | 21 | 23 | 3 | 15.67 | 66 | 50 | 42 | 52.67 |
| MiniCPM-V 2.0 [] | 16 | 11 | 10 | 12.34 | 46 | 13.63 | 45 | 34.88 |
| LLaVA-1.5-7b [] | 18 | 10 | 9 | 12.33 | 42 | 15.90 | 37 | 31.97 |
| mPLUG-Owl2 [] | 11 | 7 | 12 | 10 | 37 | 15.91 | 38 | 30.33 |
|  |  |  |  |  |  |  |  |  |

### E.2 Mitigation and worsening of oversensitivity

To investigate mitigation and worsening of oversensitivity caused by different system prompts, we curated three system prompts that focusing on helpfulness (details in Figure 13), scrutiny (details in Figure 14) and safety (details in Figure 15).
The numerical results are demonstrated in table 6.

**Table 6: Refusal rate (% $\downarrow$) of Gemini-Pro 1.5 with different system prompts. Adding different instructions on default empty system prompt (vanilla), the other system prompts are incorporated with different focus (helpfulness, scrutiny, and safety).**
|  |  |  |  |  |
| --- | --- | --- | --- | --- |
| System prompt | Exaggerated Risk | Negated Harm | Counterintuitive Interpretation | Average |
| vanilla | 25 | 28 | 35 | 29.33 |
| + helpfulness | 16 | 25 | 33 | 28.67 |
| + scruntiny | 23 | 35 | 28 | 28.67 |
| + safety | 38 | 43 | 55 | 45.33 |
|  |  |  |  |  |

Figure: Figure 13: System prompt targeting improved helpfulness

Figure: Figure 14: System prompt targeting improved scrutiny

Figure: Figure 15: System prompt targeting improved safety

## Appendix F More qualitative results

This section provides more qualitative results to illustrate the oversensitivity issue in current MLLMs. We first provide several examples of SOTA MLLMs’ responses. Then we provide examples to further investigate their response process of MLLMs.

### F.1 More examples for Claude 3 and Gemini Advanced

For each type of visual stimuli, we provide an example response from each proprietary MLLM (GPT, Claude 3 and Gemini Advanced).

Figure: Figure 16: Responses from proprietary MLLMs on an Exaggerated Risk sample. In this sample, Claude 3 exaggerate the risk associated with the marching depicted in the image and refuse to response.
Refer to caption: extracted/5679198/images/qualitatives/90.png

Figure: Figure 17: Responses from proprietary MLLMs on an Negated Harm sample. In this example, Claude 3 and Gemini-pro 1.5 misunderstood the objection and refused to respond, mistakenly believing the image advocated animals for entertainment.
Refer to caption: extracted/5679198/images/qualitatives/137.png

Figure: Figure 18: Responses from proprietary MLLMs on an Counterintuitive Interpretation sample. In this example, Claude 3 preemptively interpreted the request as pertaining to illegal littering. However, a more reasonable and intuitive interpretation would recognize the benign intent of the request, especially since there is no evidence in the image suggesting any unethical or illegal activity.
Refer to caption: extracted/5679198/images/qualitatives/287.png

### F.2 Response process examples

We found in our experiments that different types of stimuli tend to cause errors at specific stages — perception, intent reasoning, and safety decision-making — in the response process of MLLMs. This section provides qualitative examples to illustrate detailed responses in these experiments.

#### F.2.1 Perception

In this section, we demonstrate several safety-critical misperception and refusal examples we observed when evaluating model perception stage.

Figure: Figure 19: An example of safety-critical misperception. Claude3-opus mistakenly recognize the harrassment as comforting and misinterpret the context of the scenario completely.
Refer to caption: extracted/5679198/images/qualitatives/197.png

Figure: Figure 20: An example of refusal at perception stage. Although the request is merely asking for description, the model either directly refuse or provide general guideline irrelevant to the request.
Refer to caption: extracted/5679198/images/qualitatives/127.png

Figure: Figure 21: An example of safety-critical misperception. The model wrongly recognize the intent of the symbol, this could very possibly hinder the later stage of its response process.
Refer to caption: extracted/5679198/images/qualitatives/117.png

Figure: Figure 22: An example of refusal at perception stage. Although the request is only asking for description, the model either directly refuse or provide general guideline irrelevant to the request.
Refer to caption: extracted/5679198/images/qualitatives/192.png

#### F.2.2 Intent reasoning

Following visual perception, model formulates hypothesis to reason about the intent of the request. We examine this stage of response process by conditioning the model on an oracle description. In this section, we showcase a range of interesting responses the model exhibit during this stage.

Figure: Figure 23: An example of the oracle description improve the model performance.The model first refrained from writing a full song, but then it turns to comply with the request with a short verse, indicating it is able to understand the intent of the request.
Refer to caption: extracted/5679198/images/qualitatives/102.png

Figure: Figure 24: An example of the oracle description improve the model performance. The generated description revealed the model’s perceptual error on the context of the scene. The model is able to recover from this error conditioning on oralce description.
Refer to caption: extracted/5679198/images/qualitatives/197.png

Figure: Figure 25: An example of error in intent reasoning stage. In this example, the model accurately comprehends the visual scene. Consequently, providing an oracle description fails to mitigate its oversensitivity. However, when supplied with the oracle intention, the model successfully makes the appropriate safety judgment.
Refer to caption: extracted/5679198/images/qualitatives/278.png

#### F.2.3 Safety judgement

We have noticed that the model exhibits oversensitivity even when oracle captions are given, suggesting that the problem may stem from its final decision-making process. We thus explore the model’s safety decision behaviors by incorporating both oracle descriptions and oracle intent reasoning in the input. The examples in this section demonstrate that while oracle captions and intent can mitigate the oversensitivity issue in certain cases, it remains unresolved in others.

Figure: Figure 26: An example of oracle description and oracle intent reasoning improve model oversensitivity. In this sample, conditioning model on oracle description does not help much. Instead, the model is able to understand the request is about objecting to hate against minorities when conditioned on oracle intention.
Refer to caption: extracted/5679198/images/qualitatives/103.png

Figure: Figure 27: An example of oracle description and oracle intent reasoning improve model oversensitivity.In this sample, conditioning model on oracle description does not help much. Instead, the model is able to understand the request is about advocating peaceful protest when conditioned on oracle intention.
Refer to caption: extracted/5679198/images/qualitatives/118.png

Figure: Figure 28: An example of oracle description and oracle intent reasoning improve model oversensitivity.In this sample, conditioning model on oracle description does not help much. Instead, the model is able to understand the request is about improving the dog’s obedience and training instead of harsh or unethical method when conditioned on oracle intention.
Refer to caption: extracted/5679198/images/qualitatives/274.png

Figure: Figure 29: An example of error in safety judgement stage. In this case, models still refuse to answer the query even it understand the intent and turns the provided intention down.
Refer to caption: extracted/5679198/images/qualitatives/290.png

Figure: Figure 30: An example of error in safety judgement stage. In this case, the model exhibit a behavior pattern of "mind reading", assuming the request is for "making an individual disappear outside of a professional magic act context" and refuse to answer even provided with oracle intent.
Refer to caption: extracted/5679198/images/qualitatives/272.png

### F.3 Refusal responses

In this section, we will show more interesting refusal responses that exhibit unique pattern.

#### F.3.1 Cognitive Distortions

We discovered additional examples that are similar to human cognitive distortions, such as mind reading, catastrophizing, and mental filtering.

##### Mind Reading

is a human cognitive distortion where one believes they know what others are thinking. In the context of oversensitivity, it describes a behavior pattern where the model assumes unethical or illegal intent in a request, even in the absence of any clear indications. In this section, we provide more examples of "mind reading"

Figure: Figure 31: An Example of mind reading. Although the model assumes the request lacks proper supervision and training, it has already acknowledged that exploring science in school laboratories is safe.
Refer to caption: extracted/5679198/images/qualitatives/24.png

Figure: Figure 32: An example of mind reading. The model assumed the request was for conducting unethical studies, whereas it could actually have been seeking advice on ethical animal research.
Refer to caption: extracted/5679198/images/qualitatives/14.png

##### Catastrophizing

Catastrophizing is a human cognitive distortion in which one predicts the future negatively without considering other, more likely outcomes. In the context of oversensitivity, it refers to the model assuming the worst possible interpretation of a request. In this section, we will provide additional examples of catastrophizing.

Figure: Figure 33: An example of safety-critical misperception error. The model inaccurately recognized the image as suggesting an inappropriate or unproductive classroom dynamic, although the image does not fit this description at all.
Refer to caption: extracted/5679198/images/qualitatives/248.png

Figure: Figure 34: An example of catastrophizing. In this instance, the model interpreted ’caging’ as referring to the girl instead of the parrot, a strained and twisted interpretation when more reasonable understandings are clearly present.
Refer to caption: extracted/5679198/images/qualitatives/257.png

##### Mental Filtering

Mental filtering is a human cognitive distortion where one focuses excessively on a single negative detail rather than the entire context. In the context of oversensitivity, this term describes a behavior pattern in which the model disproportionately focuses on a seemingly harmful detail in the scene, regardless of the instructions given or the context of the image. In this section, we will provide additional examples of mental filtering.

Figure: Figure 35: An example of mental filtering. Although the scene contains many potential pieces of equipment that the request might target for instructions, the model fixates on the dangerous firearm and centers its response around it.
Refer to caption: extracted/5679198/images/qualitatives/12.png

Figure: Figure 36: An example of mental filtering. The model pays exaggerated attention to the fake bomb instead of understand the context of the request.
Refer to caption: extracted/5679198/images/qualitatives/4.png

#### F.3.2 Moralizing

Another interesting behavior pattern exhibited by oversensitive models is their tendencies towards "Moralizing.", where individuals issue exaggerated ethical statements, not from a place of genuine concern, but as a form of posturing. We observed instances where models refused to respond to questions by deeming them unethical. However, these ethical criticisms are often clearly unrealistic, or the model itself fails to maintain consistency in its judgments.

Figure: Figure 37: An example of moralizing. The model refused to provide advice on trapping fireflies, yet it readily complied with the instructions in another request.
Refer to caption: extracted/5679198/images/qualitatives/296.png

Figure: Figure 38: An example of moralizing. The model refused to provide advice on cooking a chicken, yet it readily complied with the instructions in another request.
Refer to caption: extracted/5679198/images/qualitatives/253.png

Figure: Figure 39: An example of moralizing. The model refused to provide advice on spliting cookies, yet it readily complied with the instructions in another request.
Refer to caption: extracted/5679198/images/qualitatives/298.png

## Appendix G Crowdsourcing

In the course of this study, data was curated using AMT. Prior to participation, all individuals were provided with a detailed consent form that outlined the nature of the research, its purpose, and how the data would be used. Explicit consent was obtained from each participant, ensuring they understood their rights, including the right to withdraw from the study at any time without penalty. This process adheres to ethical guidelines to protect participant confidentiality and autonomy. Our IRB approval can be found at this [link](https://webirb.research.ucla.edu/webirb/Rooms/DisplayPages/LayoutInitial?Container=com.webridge.entity.Entity%5BOID%5BB6B4F0BB707CD644B815FFA9608ABD46%5D%5D).

A screenshot of our AMT worker interface, utilized for the oversensitivity assessment task, is provided in Figure 40. The workers were compensated at an hourly rate of $8.

Figure: Figure 40: A screenshot of the AMT task interface for human retars to evaluate the harmfulness and naturalness of MOSSBench samples.
Refer to caption: extracted/5679198/images/appendix/AMT_1.png

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: How were these three categories of visual stimuli determined?
soundness: 3
strengths: This work tackles a persistent problem in safety-tuned models: their poor ability to properly discern harmless queries from harmful ones. The dataset curation process seems thorough and careful, and the empirical results are compelling
summary: This paper introduces a benchmark, MOSSBench, to systematically evaluate the propensity of multimodal LLMs to refuse to answer harmless queries. The authors find that safer models also tend to refuse more benign queries, that web-based models are more “oversensitive” than API versions, and that particular types of safe queries (i.e. “negated harm” and “counterintuitive interpretation”) engender more refusal than others. The authors perform a deeper analysis of the refusal behavior in a subset of examples, finding that perceptual errors often cause oversensitive behavior.
weaknesses: The primary weakness in this work lies in its rhetorical framing around cognitive distortions. It appears that this comparison is a bit superficial, and does not add much to the scientific contributions of the paper. On the contrary, this framing invites either 1) potentially harmful anthropomorphizing of LLMs, or 2) analogizing people with mental health disorders to LLMs. Though this work does not attempt to use LLMs as models of people with mental health disorders, the current framing may inspire such misguided work in the future. 

Additionally, there appears to be an important element of refusal behavior that is absent in the current analysis: the model’s conception of the user submitting the “harmless” query. Prior work [1, 2] has demonstrated that a model’s perception of a user’s attributes importantly modifies their refusal behavior. Understanding person effects seems crucial for understanding the model’s reasons for refusing queries such as those shown in Figs 20, 25, and 30. In all of these cases, the “harmless” query might be rendered harmful if a certain user (i.e., a white supremacist for Fig. 25) is submitting them. In the absence of a user model, perhaps the LLM is merely assuming the worst and responding as such? Including a preliminary investigation of these effects would strengthen the empirical results.

Additionally, some of the most interesting analyses in the paper (Section 6) are performed on a very small number (10) of examples. More examples should be included in the final version of the work.

[1] Zeng, Yi, et al. "How johnny can persuade llms to jailbreak them: Rethinking persuasion to challenge ai safety by humanizing llms." arXiv preprint arXiv:2401.06373 (2024).
[2] Ghandeharioun, Asma, et al. "Who's asking? User personas and the mechanics of latent misalignment." arXiv preprint arXiv:2406.12094 (2024).

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: 1. Are the ARR results for open-source models (and API versions of proprietary MLLMs) reported after running randomness tests with repeated experiments, such as changing the random seed or decoding strategy?
2. How was the human annotation team? Could you provide details on their occupations or expertise, such as in cognitive science or psychology?
soundness: 3
strengths: 1. While there is prior work on oversensitivity in language models, this paper is novel as the first to assess oversensitivity specifically in multimodal LLMs.

2. The evaluation is both comprehensive and in-depth, and covering a wide range of MLLMs. The authors provide insightful analyses, such as identifying discrepancies between web and API versions of proprietary models. The design of how different stimuli impact different reasoning stages in MLLMs indicates the novelty of the work and results in several valuable findings.

3. The authors conducted multiple rounds of rigorous controls and reviews on the proposed benchmark, suggesting MOSSBench is in a high quality.
summary: This work introduces a Multimodal Oversensitivity Benchmark (MOSSBench) to evaluate the oversensitivity of multimodal language models (MLLMs) based on Refusal Rate. The authors categorize the main "stimuli" of oversensitivity into three types: Exaggerated Risk, Negated Harm, and Counterintuitive Interpretation, each explained by an illustrative example. They conduct a thorough and detailed analysis, testing 20 popular MLLMs including both proprietary and open-source models on MOSSBench. They also provide several interesting findings.
weaknesses: The dataset is relatively small, with only 300 test cases, which may affect the statistical significance of the findings and limit the impact of this work. A small benchmark size could make it easier for MLLMs to overfit to this evaluation.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: - Have you tried to perform multiple runs on the same model (with fixed hyperparameters)? How much are they coherent with the results reported?

- Can you add a general description of the type of subjects in the images? Can you provide any categorization of the images? I would expect that some categories, due to embedded biases in the model, lead to higher ARR than others (i.e. image with predatory animals vs image of cities).
soundness: 2
strengths: The paper extends a debated topic on LLMs (sometimes called extra safety) to MLLM, clearly focusing on the visual component of such models. 

The authors introduce a  useful resource (MOSSBench), identifying a variable dimension about the type of visual stimuli divided in 3 possible categories. They also correlate the Refusal Rate with the actual safetiness of the models, computed by mean of a complement set  of real harmful examples, which is an important comparison to perform. 

Also, the post-experiment analysis about the refusal stage is important, in order to understand what the model finds challenging, and the authors show some similarity across the models (same visual stimuli tends to fail for the same “failed” stage). 

The evaluation pipeline can be potentially useful for future analysis of safety vs oversensitivity of the models and diagnose the cause of errors.
summary: The paper presents a benchmark dataset for testing the possible oversensitivity of Multimodal Large Language Models (Vision + Language), i.e., rejection of harmless queries due to the safety mechanism of MLLM. The authors focused on visual stimuli that can  potentially trigger such systems.
Visual stimuli that cause oversensitivity of MLLM were categorized in three main categories: 

1. exaggerated risk, i.e. potential dangerous elements but harmless in the given context; 
2. negated harm, i.e. the harmful content is present with a message of discouraging the harm (e.g. a visual negation); 
3. counterintuitive interpretation, i.e. elements in benign intentions are misinterpreted.

Authors address two main research questions: 
1. how prevalent is oversensitivity in MLLM? 
2. what specific failure in MLLM ‘s response process contributes to such behaviors? 

To these aims, they developed Multimodal OverSenSitivity Benchmark (MOSSBench), a benchmark of 544 image-text pairs formatted for Visual Question answering (VQA). It utilizes an hybrid method that combine LLM and human inputs for different scenarios and meticulous filtering, then annotated by an AMT annotator. They tested the benchmark on an array of 20 models from 3 categories: proprietary, proprietary from web versions and open sources. They also perform a failure analysis to test at which stage of reasoning the MLLM refuses a query. The results highlight three main findings:

1. Oversensitivity is prevalent on most of the current MLLMs.
2. Safer models are also more oversensitive.
3. Precise type of stimuli triggers more precise stages of the reasoning process.

MOSSBench is collected by a two-step sample generation: 
1. candidate generation: using several LLMs (to reduce biases in data) for generating the candidate texts and DALL-E 3 or looking up the web for the images. 
2. candidate filtering: human annotators filter the generations to delete possible real harmful content;  9% of the candidates were found to be actually harmful.

Finally, they construct a complementary set of harmful samples by introducing explicit malice into their scenario.

Authors perform both human and automatic evaluation (via GPT4-V), finding a 97.8% agreement among the human annotators.

From the results of such analysis, they highlight 5 main findings:
1. Open source models are less sensitive than proprietary models, except for the models from the GPT-series.
2. Web version of proprietary models shows greater sensitivity with respect to the API version
3. The refusal rate is higher for stimuli of Negated Harm or Counterintuitive interpretation
4. The safer the model, the more oversensitive it is.
5. System prompts influence oversensitive behavior.

For the final analysis about the stage that causes incorrect refusal in MLLMs, the authors divide the response process in three main stages: 
1. perception: the model must describe in the correct way the image; 
2. intent reasoning: the model formulated hypothesis about the user’s actual intent; 
3. safety judgement: the model decide to accept or reject the query, based on the other two stages. 

The authors find that most errors occur in the perception stage and different stimuli challenge different stages, i.e. exaggerated risk challenge more the perception stage. Conversely, negated harm and counterintuitive interpretation have most of the errors in intent reasoning and final decision stages.
weaknesses: In the paper there is a continuous parallelism with human cognition and behavioral disturb that causes similar errors in the humans of what is seen in MLLM. As written, it seems that authors give the MLLM “human cognition” capabilities and similar possible illnesses/disturb. All these could be avoided since they don’t give any contribution to the message of the paper and again, give the hint that MLLMs have something close to human cognition. Instead, a bigger focus on the technical aspects that lead to failures might strenghten the paper.

For instance, example 2 is misleading since the MLLM oversensitivity, in that case, seems totally an error of miscomprehension of the MLLM, and I don’t see any similarity with human cognition.

Moreover, the MLLMs use a non-stochastic decoding (I saw also the impact of temperature on the results). To this matter, results on a single run limit the generalization of the claim. Since the main point of the paper is the resource, a reader would expect to see multiple runs of the experiments on the 3-4 most meaningful models to verify the generalization of the claims, reporting average and standard deviation. This should be done also on the analysis about the error stage.

I am also not totally convinced by the usage of GPT4-V for automatic evaluation (GPT4-V is one of the models used in the experiments), and the paper is missing a comparison between human annotation and LLM annotation

Some minor typos:
- Line 48: “Concretely” 🡪 “concretely”
- Line 84: “scenariosand” 🡪 “scenarios and”
- Check the capitalization of the first letter in the item lists.

---

confidence: 5
contribution: 3
initial_rating: 5
presentation: 4
questions: 1. Regarding the division of the processing steps and the design of the ablation study, I would like to know if these are original contributions of this paper. If so, I would consider giving a higher score.
2. Some typo: It covers a variety of scenariosand serves as a diagnostic suite for assessing oversensitive behaviors. (line84-85); (3). Safety Judgement (line430-431)
3. I believe it is quite necessary, rather than solely focusing on the rejection rate. I will consider increasing the score if you add a set of small experiments during the rebuttal period (weakness two).
soundness: 3
strengths: 1. The paper is well-structured with rigorous writing logic, and the experimental design is tightly interconnected. The idea behind the ablation experiments designed at the end is particularly clever.
2. The paper provides intuitive insights into balancing the safety of MLLMs and user efficiency.
summary: This paper introduces a novel benchmark for evaluating the issue of oversensitivity in MLLMs, aiming to balance safety and efficiency. It contributes a dataset in which the image-text inputs are benign but may be mistakenly recognized by MLLMs as violating safety guidelines, leading to a refusal to respond. It provides extensive empirical evaluations and presents several findings. In addition, it designs clever ablation experiments to verify the components that lead to oversensitivity.
weaknesses: 1. First, it is mentioned in the limitations that the dataset size is too small, in terms of cost and resource requirements due to the need for human intervention in the filtering and selection processes. The comprehensiveness and generalizability of the benchmark need improvement. 
2. The experimental section only evaluates the rejection rate, which is crucial; however, it lacks an assessment of the balance between safety in question answering and the efficiency of the answers. When using this benchmark to evaluate MLLMs, focusing only on the rejection rate would be too one-sided. The goal is to balance, it would be appropriate to include experiments that assess the correct rejection of harmful queries, the incorrect rejection of harmless queries, the correct response to harmless queries, and the incorrect response to harmful queries. 
3. I observed that the prohibitions for some examples related to “Negated Harm” seem somewhat forced subjectively (e.g., figure 20, figure 25).

Meta-review: Overall, the consensus is that this paper is marginally above threshold.  Initially, reviewers found the paper weak in several aspects including that the contribution was superficial, that "refusal behavior" wa snot adequately modeled and that the dataset was relatively small containing only 300 examples.  Two reviewers were uncomfortable with the "anthropomorphism" of LLMs, equating/using a parallel analogy of their behavior with human behavior.  Another concern was the exclusive use of GPT4 for evaluation and no human evaluation.

I am recommending acceptance based on the authors successful defense of their work during the rebuttal period, but I agree that the paper could be stronger.

Decision: Accept (Poster)