Can Video LLMs Refuse to Answer? Alignment for Answerability in Video Large Language Models
Authors: Eunseop Yoon
Hee Suk Yoon
Mark A. Hasegawa-Johnson
Chang D. Yoo
Venue: ICLR 2025 Poster (2025)
Keywords: Multimodal Large Language Model
Alignment
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We present an alignment for answerability framework that equips Video-LLMs with the ability to effectively handle questions that extend beyond the informational scope of a given video.

Abstract: In the broader context of deep learning, Multimodal Large Language Models have achieved significant breakthroughs by leveraging powerful Large Language Models as a backbone to align different modalities into the language space. A prime exemplification is the development of Video Large Language Models (Video-LLMs). While numerous advancements have been proposed to enhance the video understanding capabilities of these models, they are predominantly trained on questions generated directly from video content. However, in real-world scenarios, users often pose questions that extend beyond the informational scope of the video, highlighting the need for Video-LLMs to assess the relevance of the question. We demonstrate that even the best-performing Video-LLMs fail to reject unfit questions-not necessarily due to a lack of video understanding, but because they have not been trained to identify and refuse such questions. To address this limitation, we propose alignment for answerability, a framework that equips Video-LLMs with the ability to evaluate the relevance of a question based on the input video and appropriately decline to answer when the question exceeds the scope of the video, as well as an evaluation framework with a comprehensive set of metrics designed to measure model behavior before and after alignment. Furthermore, we present a pipeline for creating a dataset specifically tailored for alignment for answerability, leveraging existing video-description paired datasets.

Published as a conference paper at ICLR 2025
CAN VIDEO LLMs REFUSE TO ANSWER? ALIGNMENT FOR ANSWERABILITY IN VIDEO LARGE LANGUAGE MODELS
Eunscop Yoon Hee Suk Yoon Mark Hasegawa-Johnsoni
Chang D. Yoo-t
Korea Advanced Institute of Science and Technology (KAIST) University of Illinois at Trrbana-Champen (UIUC) (esyoon97, hskyoon, cd_yoo) ac. kr [Jhasegaw] edu
ABSTRACT
In the broader context of deep learning, Multimodal Large Language Models have achieved significant breakthroughs by leveraging powerful Large Language Models as backbone to align different modalities into the language space. A prime exemplification is the development of Video Large Language Models (Video- LLMs). While numerous advancements have been proposed enhance the video understanding capabilities of these models, they are predominantly trained on questions generated directly from video content. However, real-world scenarios, users often pose questions that extend beyond the informational scope of the video, highlighting the need for Video-LLMs to assess the relevance of the question. We demonstrate that even the best-performing Video-LLMs fail to reject unfit questions- not necessarily due to a lack of video understanding, but because they have not been trained to identify and refuse such questions. To address this limitation, we propose alignment for answerability, a framework that equips Video-LLMs with the ability evaluate the relevance of question based the input video and appropriately decline to answer when the question exceeds the scope of the video, as well as an evaluation framework with comprehensive set of metrics designed to measure model behavior before and after alignment. Furthermore, we present a pipeline for creating dataset specifically tailored for alignment for answerability, leveraging existing video-description paired datasets. The code and the dataset is publicly accessible at tttttps://githhhh com/ Yoon /UVQA
INTRODUCTION
The rapid advancements in Large Language Models (LLMs) (Brown et al. 2020 Touvron et al. 2023 Dubey et al.] 2024 Anil al. 2023) have revolutionized natural language processing, laying the groundwork for the development of Multimodal LLMs (Huang et al. 2023 Zhu et al.) 2024 Su et al.
2022, Li al. 2022 2023a| Yoon et al. 2024b). By projecting multimodal data including images, audio, and video into the language space, Multimodal LLMs leverage the powerful reasoning and generative capabilities
LLMs as a backbone. Among these, Video Large Language Models (Video-LLMM)) (Yoon et al. 2022 2023 Li et al.) [024b) Ahn et al. 2024 Lin et al., 2023) Maaz (ct al. (024) stand out for their ability to both comprehend video content and generate contextually relevant responses. This capability opens up numerous real-world upplications, such as iiseo-baeed question answering, content moderation, and even autonomous surveillance. By processing both the visual and linguistic aspects of video data, Video-LLMMs hold significant potential in enhancing human-computer interaction and solving complex, video-centric tasks across various domains.
Despite the significant performance improvements Video-LLMs have achieved in video understanding and guestion-answering (QA) tasks, driven by advancements in architecture and training techniques, they are typically trained to answer questions generated directly from the video content. However, in real-world applications. Video-LMMs often face scenarios where users pose questions that extend beyond the information provided in the video. Since these models have been trained to recognize
Equal contribution
Corresponding Author

---

Published as a conference paper at ICLR 2025
No OF dYTRS
vike Loou
Video LLM
Video LLM
www.
Qpetiic
refermpent
Nacmols ger reppore guaniion
10 touehos repprre io
Figure 1: Limitations of Current Video-LLMs. (a) While Video-LLMs demonstrate steady improvements traditional video understanding and QA benchmarks, their performance our unanswerable question evaluation benchmark remains poor, with no meaningful progress observed across models. (b) Scaling the model size (7B + 72B) leads to improvements on traditional answerable QA benchmark, while performance on our unanswerable question evaluation benchmark remains consistently poor. (c) The Video-LLM successfully identifies that there is no cat in the video when asked straightforward existence question, which is answerable based on the video content. (d) However, when the question framed to ask about the breed of a cat that does not exist in the video, the model fails to recognize the unanswerability and generates a hallucinated response.
or refuse such unanswerable questions, they often fail, providing incorrect or irrelevant answers. In F (a), we show that while Video-LLMs show steady increase in performance standard video understanding and QA benchmarks (MVSD-QA (Xu et al.) 2017b). MSRVTT-QA (Xu et al. 2016). ActivityNet-QAA (Yu et al., 2019)), their performance on our unanswerable question evalu- ation benchmark highlights a contrasting outcome, with none of the models showing meaningful improvements. Similarly, Figure (b) demonstrates that while scaling model size leads notable improvements on standard answerable QA benchmarks, it has no significant impact on performance for the unanswerable question evaluation benchmark. underscoring the persistent challenge of han- dling unanswerable questions. This underscores a critical limitation of current Video-LLMs: their inability to handle questions that go beyond the informational content of the video.
For instance, in Figure the video depicts a scene without any cats, and the question posed asks if there a cat in the video. This type of "existence" question is technically answerable, and the Video- LLMs correctly identify that no cat is present, providing an accurate response. However, in Figure [d),, the same video is used, but the question asks about the breed of a cat, which is unanswerable since it extends beyond the information contained in the video. Despite the unanswerability of the question, the model attempts to answer and produce a hallucinated response, as it has not been trained to recognize reject such questions.
Contributions To address this limitation identified in current Video-LLMs, we introduce several key contributions in this paper:
Alignment for Answerability: In Section 4.1 we formally define alignment for answerability framework designed to equip Video-LLMs with the capability to assess the relevance of user queries in relation to the video content. This alignment process enables Video-LLMs to enhance the model's ability to reject queries that exceed the informational boundaries of the input video.
Evaluation Metrics for Alignment for Answerability: Assessing the degree of alignment of a model presents several challenges. For example, we need to evaluate not only if the aligned model become more willing to evaluate and refuse answering to unanswerable questions, but also if it becomes overly cautious in the pursuit for answerability. To deal with the complex nature of evaluating the alignment process, Section 4.2 introduces a set of metrics to systematically measure the differences in model behavior before and after alignment, capturing various aspects of the answerability performance.
Dataset Creation for Alignment for Answerability: Existing instruction-tuning datasets for Video-LLMs are limited to cases where the questions are generated directly from the video content, restricting the models' ability to handle questions that exceed the video's s informational boundaries. In Section 4.3 we outline the process of creating a novel dataset, UVQA, that includes questions designed to extend beyond the content of the video. This is accomplished by leveraging existing video datasets consisting of video-description pairs, which modify generate unanswerable questions.
2

---

Published as a conference paper at ICLR 2025
2
RELATED WORK
Video Large Langauge Models (Video-LLMs) Vidco-LLMs are trained in two stages: vision- language alignment, where the video data is projected into the LLM's token space to ensure visual features are aligned with language representations, and instruction tuning, which enhances multimodal understanding through task lask-specific fine-tuning (Maaz et al. 2024 Li et al., (2023))
Building on these two foundational stages, recent models have introduced additional training phases to address specific challenges. For instance, LLaMA-VID (Li et al. 2024b) focuses long video com- prehension, allowing the model to understand extended content, such as a 3-hour movie. VideoChat2 (Li et al. 2024a) improves upon the vision-language connection between the alignment and instruc- tion tuning stages for enhanced alignment. VideoLlama2 (Cheng et al. 2024) incorporates joint audio-video training, enabling the model to process not only video frames but also the accompanying audio for more holistic understanding of video. Additionally, to improve video comprehension and instruction- forlloinng of Video-LLMs, some approaches are experimenting with feedback from Al models (Ahn et al. 2024 Zhang et al. (024a), inspired by Reinforcement Learning from Human Feedback (RLHF) Ouyang et al. 2022). to further align LLMSs with video content.
Alignment in Large Vision Language Models (LVLMs) Aligning model to human preferences means that a model's generation should follow user instructions and account for what the user would prefer as a response (Ouyang al.) 2022; Yoon et al. 2024a). To achieve this alignment in language models, Reinforcement Learning from Human Feedback (RLHF) based on PPO (Ouyang et al. 2022) or DPO (Rafailov al. 2023) are widely used. Building on these efforts, growing number of papers focus on alignment challenges in LVLMs, viewing various issues through the lens of alignment and attempting to solve them accordingly.
RLHF-V Yu et al., 2024a) and RLAIF-V (Yu et a.. 2024b) tackle the HHH (i.e., Helpfulness, Harmlessness, and Honesty) alignment in LVLMs using feedback-based training algorithms, similar to those employed in RLHF for language models. Silkie (Li et al. (023c)), for example, focuses on improving the helpfulness and faithfulness of model outputs.
One of the problems LVLMs address as an aligmment issue is hallucination-tle generation of responses that are not factually grounded. To mitigate hallucination LVLMs, techniques such Visual Contrastive Decoding (Leng et al., (024). penalization methods (Huang et al. 2024), augmented generation (Sun et al. 2023), and modified DPO (Zhao et al. (023) have been applied. These methods aim to enhance alignment by ensuring that model outputs are grounded in the provided visual inputs.
Unanswerability of Question Answering Determining whether a question is answerable is crucial for building trustworthy AI. Given its significance, prior works (Brahman et al., 2024 Yang et al. 2023] Wen et al.) 2024) investigate answerability in Large Language Models (LLMs), focusing on cases where questions cannot be answered using the model's intrinsic knowledge. For Image- based Vision-Language Models (Image-LLMs), Whitehead et al. (2022); Guo et al. 2023) examine unanswerability based on image content. However, in Video-based Vision-Langusel
Models (Video- LLMs), this aspect remains underexplored, as evidenced by the limitations of open-source Video LLMs in addressing unanswerable questions, shown in Figure [ In this work, formalize this problem and introduce both an evaluation metric and an automated data generation pipeline, providing robust framework for training and evaluating the performance of Mideo-LLM, in addressing this challenge.
3 PROBLEM SETUP
Video Large Language Models (Video-I.LMs) The general framework of Video-LI.Ms (Li et al. 2024b Ahn al. 2024 Lin et al. 2203 Maaz al. 2024) consist of three main components: (1) visual encoder V that processes the video inputs, (2) a vision-language projector P that bridges the visual and language modalities, and (3) pre-trained language model L for generating text-based responses.
The visual encoder V receives the input video, represented as sequence of frames {v) U2,,... Br where each v; corresponds to the i-th frame. The encoder extracts spatial and temporal features, producing a latent video representation zy V(v) E Rdv.
3

---

Published as a conference paper at ICLR 2025
This encoded representation, Zy, is then passed to the vision-language projector P. which aligns it ......IINN NN J. sequence of N tokens. The projector maps the
with the instruction jiput x = {21, .2,...,, visual features into the language space, yielding aligned representation Zp P(zy) R This projection prepares the multimodal input for the pre-trained language model L.
The language model, leveraging its vast pre-training text corpora, processes the aligned represen- tation and generates a response = =1 3/....... yM ). where M is the response length in tokens y L(zp, x) AM The generated response y reflects the interplay between the information from the video and the instruction input, ensuring that the output is contextually relevant to both modalities.
4 ALIGNMENT FOR ANSWERABILITY IN VIDEO-LLMS
In this section, we first present a formal definition of alignment for answerability in Section 4.1 Following that, in Section 4.2 we address the challenges of evaluating alignment and introduce our proposed evaluation metric. Lastly, in Section 4.3 we outline our pipeline for creating dataset specifically designed for alignment for answerability.
for answerability
4.1 DEFINING ALIGNMENT FOR ANSWERABILITY
Although various architectural advancements have been introduced to enhance the video under- standing capabilities of Video-LLMs, they exclusively trained on video- question-answer triplets (v)),yg), where the questions x and corresponding answers Yg are generated directly from the content of the input video v. However, in real-world scenarios, users may pose questions that extend beyond the informational scope of the video, emphasizing the need for Vidco-LLMs to assess the appropriateness of given question x in relation to the input video v. In this paper, we define the process of equipping Video-LLMs with the ability to evaluate question relevance based on the video content as alignment for answerability. To formally define the process of aligmment for answerability, we draw inspiration from Yang et al. (2023) and begin by categorizing the model's response as follows:
1, if type(y) = correct,
t(y)
0,
if tyyee(y) wrong or unanswerable,,.
if type unanswerable..
type(y) correct when the response y does not contain any unanswerable indicators and the correct answer Ygı is included in y.
type(y) = wrong when the response does not contain any unanswerable indicators and the correct answer Ygı is not included in y.
type(y) = unanswerable,, when the response y contains unanswerable indicators (such as "The question is unanswerable". etc.), but the reasoning for why it is unanswerable differs from the ground truth Yg-
type(y) = unanswerable, when the response contains unanswerable indicators and the reasoning for why it is unanswerable is consistent with the ground truth ygt-
Then, the scoring function for alignment for answerability can be defined as:
1, if ((kvv)))t(y) = 1,
((v,,y)) = 0, otherwise,
where k(v, x) is a function that determines whether a question x falls within the informational boundaries of the input video v. Specifically, k(v,x) = if the question is answerable, and k(v,x) = -1 the question is not answerable given the input video v. The alignment process for answerability involves training the model M to prefer s(v, x, y) 1 for (v, x) sampled from the training data and y sampled from the model M resulting in an aligned model M':
M' = J(M, ((M,,()))
where f(-) represents an alignment algorithm such as supervised fine-tuning (SFT) or Direct Prefer- ence Optimization (DPO) (Rafailov et al. 2023).
In this paper, we address scenarios where users pose questions about the input video, and therefore, we the terms 'instruction' and 'question' interchangeably.
4

---

Published as a conference paper at ICLR 2025
Excesvive Store
Permissiveness
Dacretion Score
M
typely) correct
oyels) wrong
typely)
M
typely) wrong
typel) unanswerable,
unarowerable
typets)
typety) = correct
typely) = wrong
10
11
12
bypdy) wrong
typaly) - unenowerablle
13
14
bypely) unanswerable
hypdy)
unanswerable,
(6) Answerable Questions (ie,, d(v,x) = 1)
(b) Unanswerable Questions (ic. d(v,x) = -1)
Figure 2: All possible scenarios of model response type changes between the pre-aligned model (M) and post-aligned model (M'). (a) shows cases where the question x answerable based on the input video v (i.e., k.v, x) 1), while (b) depicts the cases where the question x is unanswerable given the input video v (i.c., k(v, x) = -1). Note that for (a) Answerable Questions, unanswerable responses are grouped as type(y) unanswerable, without distinguishing between unanswerable, and unanswerable as reasoning of unanswerability irrelevant for answerable questions. Each category is represented by unique number i.
4.2 EVALUATION METRICS FOR ALIGNMENT FOR ANSWERABILITY
Figure 2 iillustrates all possible scenarios of model response type (i.e., type(y), defined in Eq. [ changes between the pre-aligned model (M) and the post-aligned model (M'). Each category is represented by number with N; denoting the number of samples falling into category Using this representation, we formally define the evaluation metrics for alignment for answerability.
A straightforward approach to evaluating the aligned model M' is measure its overall accuracy based on the model's 's response:
N1 N No N7 + N12 + Nis + Ni8
Sace.
(4)
N1-18
While this provides simple metric, accuracy alone is insufficient for evaluating alignment for answerability in Video-LLMs. For example, is crucial not only to assess whether the aligned model M' demonstrates improved ability, compared to the unaligned model M, in recognizing when question exceeds the content of the video, but also whether this alignment has made the model overly conservative, causing it to refuse answering questions that are valid and were previously answered correctly. To fully capture this complex nature of evaluating the alignment process, we propose an evaluation framework that establishes a set of metrics measure the differences in model behavior before and after alignment on various aspects.
Excessive Refusal Score: When a model is trained to respond with 'unanswerable' to certain questions, it may become excessively reluctant, and the model might avoid answering questions it actually knows the answers to. This metric evaluates the degree which the model, after alignment, declines to answer answerable questions that it was previously capable of answering correctly:
N₂
Sex-ret.
(5)
+ N2+NN+NN
Permissiveness Score: In some cases, the model may incorrectly respond to an answerable question as unanswerable (i.e., type(y) = unanswerable). This metric measures how much after alignment, the model improves its willingness to answer previously refused but answerable questions:
N₂ + Ng
Sperns..
(6)
N7 No Ng
Discretion Score: This metric evaluates the model's improved ability, after alignment, to correctly decline answering unanswerable questions that it previously failed to recognize as unanswerable. measures the extent to which the model is aligned to recognize when question falls outside the informational boundaries the input video and appropriately refrains from providing an incorrect or speculative response:
Nu + N1
Susee
Ni0 + Nu + N1
5

---

Published as a conference paper at ICLR 2025
Video-Description Pair Dataset
Altered Description
UVQA Dataset
T
Description
(a) Object (0)
Video
Unanswerable question
flltr
(reserter Siny lartor)
prsceetor INAllyg
& Relation (R)
Arowss
Aftrred Element
apsston sesoreet bosans
Unanswerable Dataset
Description
Liprop
Tere B.s www.teittoooooo..... newricedd <n vides on cresover athisee helding nérnplort.
presever, neising nimoplooe
Video
Aliered
Unanswerable
(b) Attribute (.At
HerSh gial dirrt vesders
Video
How indi thees person
dnye dapeer
Unanswerable
Altered Element
Arswert
Dataset
Description
stont
vides skk Hadrllghts
mhe precesee of perses Sifie
Figure 3: Dataset Creation for Alignment for Answerability, The process begins with a video- description paired dataset (v, d). A Description Altering Process T modifies the original description by applying change c, producing an altered description d'. The altered description d' and the modification are then input into large language model (LLM) to generate an unanswerable question x and the corresponding reasoning for why it is unanswerable, which forms the ground truth label yt for (a) For the object (O) and relation (R) unanswerable dataset.
the MOMA-LRG (Luo et al. 2022), where structured as a triplet (source object, relation, target object). (b) For the DiDeMo (Anne Hendricks et al. 2017). which provides caption-like deseriptions dl rich with attribute annotations.
the attribute (AA) unanswerable dataset,
4.3 DATASET CREATION FOR ALIGNMENT FOR ANSWERABILITY
As outlined in Eq. 2 achieving alignment for answerability. i..., equipping a model to decline
answering unanswerable qnestions-requiee tailoring the model to prefer s(v, x, y) 1. This is accomplished by ensuring the model provides a response that is categorized as ((y) = when k(v,x) x) 1, ((y) -1 when k(v,x) x) -1. However, existing instruction-tuning datasets for Vidco-LLMs are limited to cases where questions x are generated directly from the vidco content (i.e., k(v, x) 1).
Expanding Beyond Current Datasets In this section, we outline the process of creating our pro- posed UVQA dataset, which includes questions that extend beyond the informational boundaries of the input video (i.e., k(v, = 1). Given dataset consisting of video-description pairs (v, d), we generate an altered description d', which provides an incorrect scene description of v:
(d', c) = T(v, d),
(8)
where T represents the altering process, and indicates the specific changes made. Using the altered
description d' and the changes c, we then prompt an external mature LLM (gpt-4-turbo- (gpr-4-urbbo22044-44-09 200--2224--0-99 generate question based on d' along with an answer Yg,, where yı includes an unanswerable indicator (e.g., "The question is unanswerable") and provides the reasoning for the unanswerability. resulting in an unanswerable question for the video v (the complete prompt can be found in Appendix(A.Ao l.
(x. LLM(dd, c), where k(v,x) -
Depending the modification made during the altering process A, the reasoning for the unanswer- ability the resulting Ygt will vary. Motivated by Scene Graph (SG) framework (Johnson et al. 20155 Xu et al., 201 7 Herzig al. 2018 2023), which structures visual scene by capturing objects, their attributes, and the relationship between them, we categorize the possible changes c to object-related (O), attribute-related (A), and relation-related (R) (i.c., AU R)).
Object & Relation-related Unanswerability I E O, the resulting question is unanswerable due to the absence of the object of interest in the video v, whereas if E R, the objects are present,

---

Published as a conference paper at ICLR 2025
but the specified relationship between them is not depicted. To create data for these categories, use the MOMA-LRG (Luo et al., 2022) dataset as the source for video-description pairs (v, d). The dataset includes annotations of the video description d in the form of Scene Graph, represented as triplets: d = (Oxc; Otg), where Ose is the source object, Out is the target object (Oae, Oug E O), and represents the relation between them (r R).
The MOMA-LRG dataset categorizes objects into 251 classes (e.g., Food, Clothing, Furniture, Independent Actors, etc.) and relations into 65 classes (e.g., Statie Relationships, Intransitive Actions, etc.). We replace one of the elements in the triplet with another from the same category to maintain naturalness in the altered description. By ensuring replacements occur within these predefined categories, we avoid creating descriptions that feel unrealistic or mismatched, such as objects interacting in ways that do not logically fit within the scene.
(drre r; Ogg); d' = (Ome r', Oggi) (Ox, P, Ogt):
if c g
if c r';
(10)
if c Ogt
Attribute-related Unanswerability If E A, the question becomes unanswerable because the object's attribute in the question does not match the attribute of the object in the video. To generate data for this category, we utilize the DiDeMo (Anne Hendricks et al. (017)) dataset, which is designed for video retrieval tasks and contains detailed attribute annotations for the scenes. Unlike the MOMA- LRG dataset used for object & relation-relan relltted unanswerability, the DiDeMo dataset provides video descriptions dl in the form of caption- like natural language sentences. In order to locate and replace the attributes in d, we use the Part of Speech (POS) tagging processor spaCy (Honnibal & Montani 2017) and extract the adjectives (a E AA in the sentence. Similar to the process for creating the Object & Relation-related Unanswerability dataset, we classify the adjectives into the categories {Color, Position, Pattern, Material, Size, Status, Shape, Human Status, Uncategorized} and perform replacements within each category, resulting in d' where = a'.
In Appendix 4.3.3 we show examples of our generated UVQA dataset for each category of object, relation, and attribute-related unanswerable questions.
5 EXPERIMENTS
In this section, we begin by describing the statistics of the generated UVQA dataset and the balanced dataset used for training and evaluation (Section 5.1). Following that, in Section 5.2 we discuss the alignment algorithms- Suuressised Fine- Tuning (SFT) and Direct Preference Optimization (DPO)-an) -nd the baseline models used for comparison, along with details on the experimental implementation. including the base models, optimization strategy, and compu- tational setup. Section 5.3 presents both the quantitative and qualitative results of our experiments, highlighting the impact of alignment on the overall performance of the model.
5.1 DATASET
The resulting UVQA dataset from Section 4.3 consists of 10k training samples for each category of unanswerability (i.e., object, relation, and attribute-relate unanswerabilityy, resulting in a total of 30k training samples. For the evaluation of unanswerable questions, we curated 200 samples per category, applying human filtering to ensure high-quality and accurate data, yielding 600 clean evaluation samples. The detailed filtering process is explained in Appendix A.4.2
As per Eq. []] aligmment for answerability requires coverage of both scenarios: k(v,x) x) 1 (answer- able) and x) = -1 (unanswerable), ensuring that alignment is not biased towards one scenario. Therefore, in addition to the unanswerable dataset, we incorporate an answerable dataset using 30k samples from the Video-ChatGPT (Maaz et al.] 2024) for training. For the evaluation of answerable questions, we use a subset of the ActivityNet QA YYu et al. 2019), a widely used benchmark for question-answering tasks in Lddo.LLMMs. Thus, the resulting evaluation dataset consists of 1200 samples, with an equal split of 600 unanswerable from UVQA dataset and 600 answerable samples from the AetivityNet QA, ensuring a balanced assessment between both scenarios.
7

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation of Answerability on Alignment and Absolute Performance: Alignment performance is assessed using the Excessive Refusal (et,, Permissive (Spemis.), Discretion (Saise) Scores, and the overall Alignment Score (Saign). ). Absolute performance is measured through overall accuracy (Sec.) and LoM across various base models. Our aligned models (SFT and DPO) demonstrate improved answerability alignment compared to the unaligned model.
Base Model
f()
Answerability
Alignment Performance
Absolute Performance
F1
Seveet
permis.
Saise.
Salgen
Secc
LLM
unaligned
0.00
0
0
0.33
0.24
2.33
Video-LLaVA
SFT (ours)
0.68
0.50
0.55
0.66
0.66
0.47
2.84
DPO (ours)
0.68
0.14
0.61
0.59
0.68
0.49
3.08
unaligned
0.00
0
0
0.33
0.22
1.93
VideoChat2
SFT (ours)
0.61
0.25
0.25
0.85
0.62
0.51
2.98
DPO (ours)
0.64
0.09
0.46
0.71
0.69
0.54
3.02
unaligned
0.00
0
0
0
0.33
0.25
2,36
VLM-RLAIF
SFT (ours)
0.65
0.10
0.37
0.67
0.65
0.52
2.86
DPO (ours)
0.66
0.08
0.5
0.64
0.69
0.53
2.93
5.2 EXPERIMENTAL SETUP
Baselines We conduct alignment training following the process outlined Eq. 3 We evaluate two alignment algorithms for f: Supervised Fine-Tuning (SFT) and Direct Preference Optimization (DPO) (Rafailov et al., 2023). These are compared against an unaligned baseline, where f is simply the identity function, meaning .M' .M Performance is assessed using both our proposed alignment metrics (Eq. 7 and the absolute accuracy metric (Eq. 4
Implementation Details To evaluate the general trend of alignment, we utilize various Video LLMs, including Video-LLaVA (Zhang al. 2023). VideoChat2 (Li et al. 2024a), and VLM-RLAIF (Ahn et al. 2024) as our base models. We use the Adam W (Loshchilov & Hutter (017) optimizer with learning rate of lc-6 and a constant learning rate scheduler, and we set the batch size 128 for all experiments. We employ DeepSpeed-Zent Zero Stage 2 (Rasley et al. 2020) with CPU offloading for the optimizer and gradient checkpointing. All experiments were conducted on x NVIDIA A100 80GB PCLe GPU.
5.3 EXPERIMENTAL RESULT
Table I presents the quantitative results of our experiments, showcasing the Answerability Cor- rectness, Alignment Performance and Absolute Performance. For the Answerability Cor- rectness, we evaluate the binary correctness of the answerability prediction of the model using the Fl score. For the Alignment Performance, we use the metrics introduced in Section 4.2 Excessive Refusal (Sex-ret.). Permissiveness (Speriss). ). and Discretion (Sdise) scores. Along with these metrics, we also report the average alignment score, calculated as the mean of these metrics: Salign = ((1 ([(11-Sexaa) Sex + Spermis. + SSisc.) For the Absolute Performance, we report the accuracy (Suce.) as defined in Eq. 4 Additionally, following the evaluation methodology of previous Video- LLMs (Lin et al., 2023 Li et al., [2024a] by [Ahn et al., (024)) we include the LLMscore- which assesses the quality of the model-generated response y in comparison to the ground truth label Yg- This rating, assigned by GPT-4, is on a scale from 0 to 5. The prompts for determining type(y), needed for Alignment Performance and accuracy, and LLMssore can be found in Appendix A.5
Across all base models, both the SFT and DPO models trained using our framework consistently outperform the unaligned models, exhibiting higher answerability correctness (F1), average alignment scores (Saggn). accuracy (Sus.). and LLM These improvements are primarily driven by superior performance on the unanswerable UVQA evaluation set. When comparing SFT and DPO, although Sace remains comparable between the two, DPO generally achieves lower Sec-ref. higher Spermis, and slightly lower Sdisc. This suggests that DPO facilitates softer alignment, allowing the model to maintain its original performance while still enhancing its ability to handle unanswerable questions.
"For the unaligned model, we still define M' this way to compute our proposed alignment metrics. "Since lower Sevret, score indicates better performance, use Sex in the average alignment score.
8

---

Published as a conference paper at ICLR 2025
Unanswerable Question
Video
does the
roace loaling art
Yat
suvecaorralle bresasse tinere no clirkoae
fiishord the sidce, hords the pit arew Wetried react an airp coccot eésonviee aeser lead besos thme contect previded.
per soov looktng
The dseorlaw matecweratle lnceeeed tile
dosotiie inuecseecabll bassaes vidto diess with show any
they that jos inspocting assessing It.
atrplaes video.
per momierr looknng art airpaaw, The vilte oily shows jet arew menier working der.
show
pit crow moniner looicing alf
trokkgred Midel
arrplat as fine foors solcly on +ho (ef crew worlbidg de tine dirtilnd.
Model
Wadel
(a) Object-related (0) unarowerable QA prediction
Unanswerable Question
What this solor of -he cader the tackower ronesteg?
Answerr Ygt
The geositio is wavowerall as the video Aons vei show relotion of arstumtr ranesdng dhair.
The cistomer nomoving Place (ieir This informatiio hise be dreatly deserved the warhoadt Gauttion.
The question cnonsweroole
This doashin inuessecrale
bocause vides dens provide the dhuir romes/ing
beseuee from worvaral
does
tralipool Medel
loding
Model
Model
(b) Relation-related (R) unanswerable QA prediction
Unanswerable Question
hal
Answeer Ygt
The arsofion is ievesworrall la bescarss the video does fecturd sasaring Heet.
The vun withi the rod kat is showe holdhng and shising thss worenr's kook de wel eadhanggug rauy with fror Thess actioed ard vidlito ii-Itt -ine video.
The asestio is modeesarrdeee lew aise tits video dots feathors mon bofth katt.
The suestion muurrvve vallle intcause Fhr video does not dhaw mate wetoiigg liett.
troligedd Medel
Se1 Algood Medel
JPO Allgued
Wodel
(c) Attribute-related (-4) unanswerable QA prediction
Figure 4: Examples of model predictions from the unaligned model, the model aligned using Supervised Fine-Tuning (SFT), and the model aligned using Direct Preference Optimization (DPO), all based VLM-RLAIF (Ahn et al., (024). The examples illustrate (a) Object-related (O), (b) Relation-related (R), and (c) Attribute-related (AA) unanswerable QA predictions. In each case, the word highlighted in red within the question identifies the specific reason it unanswerable. In we present additional examples of model predictions.
Appendix A.7
In Appendix AA10 we further analyze the trade-off between accuracy on the original answerable QA evaluation and performance the unanswerable QA evaluation, highlighting the distinct aligmment dynamics SFT and DPO. Moreover, in Figure 4 we show examples of the model prediction between different models. The example clearly shows that the unaligned model failes to detect the unanswerability, whereas both aligned models (SFT and DPO) trained with our framework and dataset correctly identify the question unanswerable and provide the appropriate reasoning of unanswerability. Additionally, we provided human evaluation results in Appendix A.11
ABLATION STUDY: CAN Existence-Bar Based Question Splitting DETECT UNANSWERABILITY?
In Figure (c), we showed an example where current Video-LLM fails reject an object-related (O) unanswerable question (i.e., 'What is the breed of the cat in the video?"). In contrast, as shown in Figure (b), the same model successfully recognizes that the object not present when asked in an existence-quent quesston format (i.e.. 'Is there cat in the video?'). This raises the question: can we detect unanswerable questions by reformulating them into series of existence-based questions?
"This style of reformulating the detection task into series of binary existence questions was proposed in POPE (Li et al. (023d) for object hallucination detection in image-based vision large language model. Thus, refer this the "POPE-Style" approach in work.

---

Published as a conference paper at ICLR 2025
Unaligned
POPE-Style
Aligned (ours)
0.6
Time (min//un)
Accuracy
40
Cont (Sirun)
Score
0.4
20:
20
10
0.0
Sev-ret (1)
Specmis (1)
Saive
Syhen
Answerable
(MGA
Baluation Cost
(a) Alignment Performance
(b) Overall Accuracy
(c) Evaluation Resources
Figure 5: Performance Comparison Between Unaligned, POPE-Style, and Aligned Models (Ours). (a) & (b) The POPE-Style method demonstrates improved performance the unaligned baseline on both the alignment metric and overall accuracy. However, it falls short when compared to the aligned model using our framework. (c) Additionally, the POPE-Style method incurs significantly higher computational costs, reducing its practicality for real-world applications and underscoring the importance of our models that are intrinsically aligned for answerability.
For given question x, we prompt an LLM (gpt-4-turbo-2 20t-rbo-2224044009 to generate set of existence- based questions, x' = { } derived from the objects, relations, and attributes in x (see Appendix A.6 complete prompt). If Fxy E x' where the model responds 'no' and all responses are correct, we classify it as type(y) unanswerable.. If 3z'; E x' where the model responds 'no' but classified type(y) = unanswerable... Vag E x' the model responds 'yes', we proceed to ask the original question x, classifying the response as either type(y) correct type(y) = wrong.
least one response is incorrect,
Figure shoows the performance comparison between the unaligned model, the POPE-Style approach, and an aligned model using our framework. The alignment metric Section 4.2 measured the behavior differences between the pre-aligned (M) and post-aligned (M'' models. For the unaligned baseline, M MM, while for the POPF-Style baseline, M' represents the unaligned model M augmented with the POPE-Style unanswerability detection mechanism discussed in this section.
As demonstrated Figure (a), POPE-Style approach improves alignment scores over the unaligned model without additional training. Figure (b) shows it also enhances the overall accuracy, driven by improved performance on the unanswerable UVQA evaluation set. However, the POPE-Style approach has main limitations compared to framework: (1) reduced performance and (2) higher computational cost. Although it improves on the unaligned baseline, significantly underperforms compared to models trained with method. More importantly, the POPE-Style approach requires converting a single unanswerable question into multiple binary questions, increasing both the number of model predictions and computational overhead (see Figure (c)) In our experiments, this led to a four-fold increase in dataset size and a six-fold increase in evaluation cost, driven by the need for additional predictions and verification by advanced LLM (gpt-40-200 2024002-2-05--1 These limitations reduce the practicality of detecting unanswerability by existence-based question splitting (i.e., Styye,,,,,, highlighting the importance of training models that are natively aligned for answerability using our framework.
CONCLUSION
This paper introduces comprehensive framework for aligning Video Large Language Models (Video-LLMMs) to effectively handle unanswerable questions critical challenge often overlooked in existing models. We proposed the concept of alignment for answerability, equipping Video-LL.Ms to assess and reject questions that extend beyond the informational scope of the video. To support this, we developed tailored evaluation metrics and introduced UVQA, novel dataset offering both training and evaluation data for unanswerable questions, along with detailed reasoning for why they unanswerable. Our experiments show that the combination of our framework and the UVQA dataset significantly improves the ability of the model to align for answerability, resulting in more accurate and reliable Video-LL.M for real-world applications.
10

---

Published as a conference paper at ICLR 2025
8 ACKNOWLEDGEMENT
This work was supported by Institute of Information & communications Technology Planning Evaluation (ITTP) grant funded by the Korea governmen((MSTT) (No. RS-2022-110951, Development of Uncertainty-Ai
Agents Learning by Asking Questions), Institute of Information & commu- nications Technology Planning & Evaluation (IITP) grant funded by the Korea goVernmen(MMSTT) (No.RS-2022-12200 84, Development and Study of AI Technologies to Inexpensively Conform to Evolving Policy Ethics), and Institute for Information & communications Technology Planning & Evaluation (ITTP) grant funded by the Korea goveriment(MSIT) No..R-2021-112 1381, Develop- ment of Causal Al through Video Understanding and Reinforcement Learning, and Its Applications to Real Environments).
ETHICS STATEMENT
We confirm that research adheres the highest standards of ethical considerations. All work presented in this paper is original, and any external contributions or sources have been appropri- ately cited. Our study does not introduce new datasets, nor does it involve experiments utilizing demographic or identity characteristics.
REFERENCES
Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, and Jonghyun Choi. Tuning large multimodal models for videos using reinforcement learning from Al feedhack. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 923-940, Bangkok, Thailand, August 2024, Association for Computational Linguistics. URL https //aclanthology. org/ 2024 acl- 1on.clllon...22
Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, al. Palm 2 technical report. arXiv preprint arXiv:2305. 10403, 2023.
Lisa Anne Hendricks, Oliver Wang, Eli Shechtman, Josef Sivic, Trevor Darrell, and Bryan Russell. Localizing moments in video with natural language. In Proceedings of the IEEE international conference on computer vision, pp. 5803-5812, 2017 2017.
Faeze Brahman, Sachin Kumar, Vidhisha Balachandran, Pradeep Dasigi. Valentina Pyatkin, Abhilasha Ravichander, Sarah Wiegreffe, Nouha Dziri, Khyathi Chandu, Jack Hessel, Yulia Tsvetkov, Noah Smith, Yejin Choi, and Hannaneh Hajishirzi. The art of saying no: Contextual noncompliance in language models. The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024. URL
#1014wN1w6
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 1877-1901. Curran Asso- ciates, Inc., 2020. URL 2020 ett.1ee/455/ccc4 (40444550000000044746444 (wte44418bbb8aa. 42f 42f PPaeer.
newrie..rrpopp ec ccppaper files/paper/
Zesen Cheng, Sicong Leng, Hang Zhang. Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, et al. Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-Ilms, arXiv preprint arxi::2406. 07476, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, et al. The llama 3 herd of models. arXiv preprint 4r0iv:2407.2 1783, 2024.
11

---

Published as a conference paper at ICLR 2025
Yanyang Guo, Fangkai Jiao, Zhiqi Shen, Liqiang Nie, and Mohan Kankanhalli. Unanswerable visual question answering. arXiv preprint arXiv:2310.10942. 2023.
Roei Herzig, Moshiko Raboh, Gal Chechik, Jonathan Berant, and Amir Globerson. Mapping
images to scene graphs with permutation-invariant structured prediction, 2018. URL https
//arxiv. org/ /aas//880 05451
Roei Herzig, Alon Mendelson, Leonid Karlinsky, Assaf Arbelle, Rogerio Feris, Trevor Darrell, and Amir Globerson. Incorporating structured representations into pretrained vision language models using scene graphs. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 14077-14098, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023, emnlp-main.3e URL https: http::://aaiaathhooorg.. org/ emnle aammp-mmin......mmmmmmmmmmmmmmmm 870
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing. 2017.
Qidong Huang, Xiaoyi Dong. Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang. Dahua Lin, Weiming Zhang, and Nenghai Yu. Opera: Alleviating hallucination in multi-modal large language models via over-trust penalty and dtrospectionn-lllcaaiie Computer Viston and Pattern Recognition, pp. 1-448-13427, 2024.
Proceedings of the IEEE/CVE Conference
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Barun Patra, Qiang Liu, Kriti Aggarwal, Zewen Chi, Johan Bjorck, Vishrav Chaudhary, Subhojit Som, Xia Song, and Furu Wei. Language is not all you need: Aligning perception with language models. In seirth-e Conference on Neural Information Processing Systems, 2023. URL /ttpps://ppenrrreee neteeeiwwwwweeeeeeeeeeeeeeeeeeee Eorummidd-nuuuueeereeeo
Yaochu Jin and Bernhard Sendhoff. Pareto-based multiobjective machine learning: An overview and case studies. Systems, Man, and Cybernetics, Part C: Applications and Reviews, IEEE Transactions on, 38:397 - 415, 06 2008. doi: 10.11099/SMCCC22088.... 919177.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In 2015 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 3668-3678, 2015. doi: 2110///
Sicong Leng, Hang Zhang, Guanzheng Chen, Xin Li, Shijian Lu, Chunyan Miao, and Lidong Bing. Mitigating object hallucinations in large lision-language models through visual contrastive decod- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 13872-13882, 2024.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation, 2022. URL https
//arxiv.org/ http//bb//220000 22012 12086
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. B Bootstrapping language-image pre-training with frozen image encoders and large language models, 2023a. URL https://
arxiv. org/ abs / 2301. 12597
Kunchang Li. Yinan Hc, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding. arXiv preprint arXiv:230s.066 06355,,, 2023b.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Yi Liu, Zun Wang, Jilan Xu, Guo Chen, Ping Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22195-22206, 2024a.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint wxXi:23312.10665 2023c.
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. 2024b.
12

---

Published as a conference paper at ICLR 2025
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang. Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 292-305, 2023d.
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-Ilava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10112, 2023.
Xi Lin, Hui-Ling Zhen, Zhenhua Li, Qingfu Zhang, and Sam Kwong. Pareto multi-task learning. In Thirty-third Conference on Neural Information Processing Systems (NeurIPS), pp. 12037-12047, 2019.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Confer- ence on Learning Representations, 2017.
Zelun Luo, Zane Durante, Linden Li, Wanze Xie, Ruochen Liu, Emily Jin, Zhuoyi Huang, Lun Yu Li, Jiajun Wu, Juan Carlos Niebles, Ehsan Adeli, and Li Fei-Fei. MOMA-LRG: Language-refined graphs for multi-object multi-actor activity parsing. In Thitty-sixth Confer- ence on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https:
htttps://openeeviet wptppeveee.eee.eeeooeeee. net htt////orummieeeeeeeeee
CPXQIT
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Video-ChatGPTI Towards detailed video understanding via large vision and language models. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 12585-12602, Bangkok, Thailand, August 2024. Association for Computational Linguistics. URL https://aclanthol.og org/2024 acl long. 679
Aviv Navon, Aviv Shamsian, Gal Chechik, and Ethan Fetaya. Learning the pareto front with hypernetwors.. In International Conference on Learning Representations, 2021. URL https
//openreview.o pettnrrvieeweee//eeeee For/
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730- 27744, 2022.
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefamo Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly reward model. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https;
/oopenreview wtteet/ffooommmmdtPsssss
Jeff Rasley, Samyam Rajbhandari. Olatunji Ruwase, and Yuxiong He. Deepspeed: System optimiza- tions enable training deep learning models with over 100 billion parameters. Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp- -505-33066, 2020.
Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. Language models can see: Plugging visual controls in text generation. arXiv preprins (rXiv:2205.026555 2022.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 2:09: 14525, 2023.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092 092888 2023.
Bingbing Wen, Jihan Yao, Shangbin Feng, Chenjun Xu, Yulia Tsvetkov, Bill Howe, and Lucy Lu Wang. Know your limits: A survey of abstention in large language models. arXiv preprint arXiv:2407. 18418, 2024.
13

---

Published as a conference paper at ICLR 2025
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In European Conference Computer Vision pp. 148-166. Springer, 2022.
Danfei Xu, Yuke Zhu, Christopher B Choy, and Li Fei-Fei. Scene graph generation by iterative message passing. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3097-3106.. IEBE, 2017a.
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM international conference on Multimedia, pp. 1645-1653, 2017b.
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5288-5296, 2016.
Yuqing Yang. Ethan Chern, Xipeng Qiu, Graham Neubig, and Pengfei Liu. Alignment for honesty. arXiv preprint arXiv:2312.07000, 2023.
Eunseop Yoon, Hee Suk Yoon, SooHwan Eom, Gunsoo Han, Daniel Nam, Daejin Jo, Kyoung-Wove Woon On, Mark Hasegawa-Johnson Sungwoong Kim, and Chang Yoo. TLCR: Token-level continuous reward for fine- ain--griied reinforcement learning from human feedback. In Lun- Wei Ku, Andre Martins, and Vivek Srikumar (eds.). Findings of the Association for Computational Linguistics: ACI. 2024, pp. 14969-14999 14981, Bangkok, Thailand, August 2024a. Association for Computational Linguisties. doi: 10. 10.18653/v112024 2024. www.ffiniigss.. acl. 889/
http://///2//.....gs/s///// fiddngg-acll885 URL htt.:httpss//c /ttps://allaaatac www.cnlnnoology..ooooooooooooooe org/
Hee Suk Yoon, Eunseop Yoon, Joshua Tian Jin Tee, Kang Zhang, Yu-Jung Heo, Du-Seong Chang, and Chang Yoo. Bi-mdrg: Bridging image history in multimodal dialogue response generation. In European Conference Computer Vision, pp. 378-396. Springer. 2024b.
Sunjae Yoon, Eunseop Yoon, Hee Suk Yoon, Junyeong Kim, and Chang Yoo. Information--heerrtic text ballucination reduction for video- grouodgrounded dialogue. In Yoav Goldberg, Zornitsa Kozareva, Conference on Empirical Methods in Natu- ral Language Processing. pp. 4182-4193, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 1166////1/22222.mmmmmmmmm
and Yue Zhang (eds.), Proceedings of the 2022
URL
in. 280/]
Sunjae Yoon, Dahyun Kim, Eunseop Yoon, Hee Yoon, Junyeong Kim, and Chang Yoo. HEAR:
Hearing enhanced audio response for video-grounded dialogue. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 1191 1-11924, Singapore, December 2023. Association for Computational Linguistics. doi: 10. 10.11865//112220 1/2023. cerdiinng-mmml.79999 URL https::/ http:::///claathhoooy.oo//////// Findings-emnipp 797/
Linguuttt
Tianyu Yu, Yuan Yao, Haoye Zhang, Taiwen He. Yifeng Han, Ganqu Cui, Jinyi Hu, Zhiyuan Liu, Hai-Tao Zheng, Maosong Sun, et al. Rlhf-v: Towards trustworthy miims via behavior alignment from fine-grained correctional human feedback. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13807-13816, 2024a.
Tianyu Yu, Haoye Zhang, Yuan Yao, Yunkai Dang, Da Chen, Xiaoman Lu, Ganqu Cui, Taiwen He, Zhiyuan Liu, Tat-Seng Chua, et al. Rlaif-v: Aligning milms through open-source ai feedback for super gpt-4v trustworthiness. arXiv preprint 2rX1v:2405.17220, 2024b.
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. Activitynet-qa: dataset for understanding complex web videos via question answering. AAAI, pp. 9127-9134, 2019.
Hang Zhang, Xin Li, and Lidong Bing. Video-LLaMA: An instruction-tuned audio-visual language model for video understanding. In Yansong Feng and Els Lefever (eds.), Proceedings of the 2023 Conference Empirical Methods in Natural Language Processing: System Demonstrations, pp. 543-553, Singapore, December 2023. Association for Computational Linguistics. doi: 10.18653/
URI https: //t///cccaaaaa aatt//aaaaaaoollooooooo www.tnowwoooo.oooooooooooo wwg.y.orrg2000 2023. emn lomnlp-dente 49
14

---

Published as a conference paper at ICLR 2025
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng. Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024a. URL httts::/ilava--ae witaaauu.ggii.u.........aaaaaabt 1o -ttp///ccc//2220000--------
video/
Yuanhan Zhang, Jinming Wu, Wei Li, Bo Li, Zejun Ma, Ziwei Liu, and Chunyuan Li. Video instruc- tion tuning with synthetic data, 2024b. URL
wwwporg///baaagoo// 02713
Zhiyuan Zhao, Bin Wang, Linke Ouyang, Xiaoyi Dong, Jiaqi Wang, and Conghui He. Beyond hallucinations: Enhancing Ivims through hallucination-awve aware direct preference optimization. arXiv preprint arXiv:2311. 16839, 2023.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. MiniGPT-4: Enhancing vision-language understanding with advanced large language models. In The Twelfth International Conference Learning Representations, 2024. TRRphttps://ppneeeeee http:////pppeeeeeeeee wwt. forum?
2d=1222
15

---

Published as a conference paper at ICLR 2025
A APPENDIX
A. BROADER IMPACT
This paper highlights the importance of aligning Video Large Language Models (Video-LLMs) to handle unanswerable questions, critical capability often overlooked in favor of improving accuracy on answerable tasks. By introducing alignment for answerability and leveraging unanswerable question datasets, this work sets a new standard for developing Al models that are not only accurate but also robust real-world scenarios where incomplete or misleading information may arise. The ability to reject unanswerable questions contributes significantly to the trustworthiness and reliabilityy of Al systems, especially in applications such as autonomous systems, content moderation, and video- based inestion-answering tasks. This research encourages the broader AI community to prioritize alignment for answerability, enhancing the safety and utility of Al systems diverse, real-world applications.
A.2 LIMITATIONS
A key limitation of our approach is the increase in the excessive refusal score (Sex Eq. observed after alignment for answerability. While aligning the model improves its ability handle unanswerable questions, can negatively impact the model's original performance by leading to undesired refusals for questions the model could previously answer correctly. This trade-off between alignment and excessive caution suggests that further consideration is needed. Future work could focus on developing more sophisticated alignment algorithms f (Eq. that achieve a better balance, reducing the excessive refusal score without compromising the model's overall accuracy.
A.3 ETHICS STATEMENT
In our study, we utilize Large Language Models (LLM) to generate our UVQA dataset and evaluate video-LLMs, which may result unintended outcomes. However, during the human filtering process of generating the UVQA evaluation dataset, we thoroughly reviewed the generated data from LLMs and confirmed that does not contain any unethical or harmful content.
A.4 UVQA DATASET
In this section, we present the prompts used for data generation (Section 1.4.1). detail the filtering process for creating the dataset (Section A.4.2). and provide examples from the resulting UVQA dataset (Section A.4.3
A.4.1 PROMPT USED FOR UVQA GENERATION
While creating the UVQA dataset, prompt external LLM generate question and its corresponding answer Ygt. based on the altered description d' and the modification (Eq. 9). Figures 7 and show the prompts used to generate unanswerable dnestion-answed pairs for the object, relation, and attribute categories in the UVQA dataset, respectively.
16

---

Published as a conference paper at ICLR 2025
resporne that propaiitlly comp etes theroges:
1Start of Iratraction]
the prowied thevedos penerat
cueston cor seivet beet the otsernet conters Howeres theopedhic menioned appear ineher should
pransoeratle
due
sisence of the me all ads, or outt in the dee. The estion can focas on vodour repects, indluding but Biibedror shtpe albtt
The chamaternaiia
soject (sa.
The state conditional of soreeti
stik Aiuid ond, new)
The rumbar quandtyy otteets ewents
the reliturniippp oi asocation cocatinnin betwee seaetss evart
The orming or oequetie events
The parts comporents of odject sudlect
troure te aeater shou eite nentaoed the gencutel questonn Aan crsatic veriety of question typie cover offfrert taspects, prenatiigg deeper undesttaning and ondongggooo. geneeeiggee.ed....
fooed
Istart Pelated helattee]
(decrpttorr from soseert ahen vidayy
Iscart tncelated secattion
fs toro docnpppin of wtthatiiigg.nee (7
(eblers ine fiom teooeeeettrr and theeeee appert gedeeeggeeeeer of
1. the und undresed slation arenar attall momeeet or matcho or the slesed diazion sutpes Net appicable withoot lomud sworble cation arower stutig that quesion toworatoobbe cuplain wnrooeraall, poeinggto forrat: "upoom by sertonyy (nuetion) dlist of qustons
b.ll
dlort of arewers [atower] clEnd knowe??" [End instruction|
Figure 6: Prompt used to instruct UVQA on object-related O questions
resporne appropetitaly requett.
|ttrrt of Instructiinll
appeoris
quectio
because prockled rdtatin unreatedd what shown. The pussiile gustton souid Inccude, tedi
Aniing axcul coke of the póeec
shape
ect
quettoning store conditio stject sold, SeyAt,
Anding sbout the sow or simenioon of otject
inguerng aboud the meturaloo subocenc sdpect trom
ingulling g about the quantity c number objects
Anding sbout location or plas oteer the dbject found
Adding about the afforret parts or componentiod object
You dooul inckude generated and unoklted oldiiedin io he eston to the quettion uoontwerr Additica by intture that your question covers dfferest aspects the video Istter of eelated to lat iae]
of the gen ceeoveddee) land antuted selartie
lStact of Bnrelateed selatdoni deeedidoorroooo preeee
lowe terelated telation
sil the uneclat di rndation ia anarmnttalll troamect or tearmataaytttrmntttttt generaio soooriibll quailin and anover souting quotice uransmooull and zipla ring why unarswordde folowing risstree live Queebos pestirn Questori
maisha are of dhondlaadd dlations, pulpul hot applicdbe wtthout formua,
dlaa aenowet (ammerr gloo Aemards"
[end tratruction
Figure 7: Prompt used to instruct UVQA on relation-related R questions
Telono the decerbera thete Whitee reporea that appropetnnyy compaatee
1tart teacttoon
proviee
question
asket based observed content, Howaves rtoned
ingthhh
queetis ronowarrble setuue the yrotbute of weject, or crongectibee being odle abeutte not prrsect in video. The snover thoir Indrate that the quertinil unnnowawee anoll due atsence ueed attrbuuee Vheo. spenten can focul vartous aepects, including liritec
or on servething SoM Squid new?
nu enter or quontiyy of
resltsen susuatanndddwooo slacsi la la crants
The tar or socu ence events
subjects
The comperennt en
subject
Emaanc mnlirdaiinn shouri becrestaencl in deccgennutec quoiiinn Aen tocreaic nanccy quesiton typos soina diflcront appeciuupe mo0king a Uheppr undkassiand orgggeeen.. to yiu'l be
oy rebitonindduee
Istart Retated Relactee)
ide inpireed toom diadin dlen wkons
Istart terelated whlltteed > accerppiin aften gf prccon
iind of Ureulated
glen
object
(rrtphe of
topear www.tegg//eeeeeeer....
1 Uhe ur aldt query aparmatti iy incomme or atriler the ahvon rdattl query, culpet Nott espiuablee without for mud. (quettod) (f-t annarree quotiin and arower saring 1at cucadon secaabllelle wwwlaiinnnnnggooo... www.weeetii.......tt.o..ooo formet
creciatd overy
"o "upooe
sita of Anowert (srowed 11-4 Anowe??
Itnd 3nsteuction]
Figure& 8: Prompt used to instruct UVQA on attribute-related A questions
17

---

Published as a conference paper at ICLR 2025
A.4.2 DATASET FILTERING
During the generation of the UVQA dataset, automatic filtering was applied all UVQA samples (training and evaluation samples), while human filtering was specifically conducted on the evaluation set to ensure clean and reliable data for assessment.
Automatic Filtering As described in Section after altering the description d to d', we prompt an external mature LLM to generate the unanswerable question-answer pairs (x, yg). During this stage, we apply filtering process to remove instances where d' is semantically too similar to the original description (Figure 9 (a)) or contains grammatical errors that could interfere with the question generation process (Figure (b)). This filtering is integrated directly into the LLM prompting process (Eq. 9. as shown in the prompts provided in Section A.4.1
Human Filterrng For the UVQA evaluation set, human filtering process is incorporated to ensure high-quality data. In this step, samples are reviewed to identify any questions that are still answerable based on the video were missed by the automatic filtering. Well-constructed instances are labeled as "pass," while inadequate samples are marked as 'filtered'. Figure T0 illustrates the interface used during filtering.
Video 1
Video
Description toble towns plaver in fromt of toblc towns table
Description d
Stodlnnt in fromt of st stol/oodds
Altered Description *abble sloyer lokiug tabls tewis toblle
Altered Description d'
stydset shse//eadh
Generated Description with Semantic Similarity
(b) Generated Description with Grammatical Error
Figure 9: Examples of Automatically Filtered Data in the UVQA Training Set. Two main cases were filtered: (a) Generated Descriptions with Semantic Similarity and (b) Generated Descriptions with Grammatical Errors.
('bssterblll playez 2colsng satch officiel', 'bestttblll playsz holding bezhetball',
'baptotbeil playez holding back
Gesecated Selation player
com Desezated OlA
<iRtass of Queesion
playez
<leed of guestiosiz
konwerl>
unssavecable Secause shace sable sensia hall <lEed of Answezl>
voathle sideo, shich acrienly feacuzes
paM
Figure 10: Illustaaioon of the interface used for human filtering in the UVQA test set. The interface displays video frames, the original description d, the altered description d', and the generated QA pair (x, ygt. Reviewers manually assess the quality of each QA pair based on this information.
18

---

Published as a conference paper at ICLR 2025
A.4.3 EXAMPLES THE GENERATED UVQA DATASET
Figures [T 12 and 13 provide examples of unanswerable question- question-answer answer pairs from the UVQA dataset, categorized by object-, relation-, and attribute-related questions, respectively.
Unanswerable Question X
Whar and day ge thso shit plass ti sas hiinor lodding "he vidco?
Yyt
"The aacitioe inseriver dide vocarns the video doo's not focthre
Descr intoription d
(onstsaer, halkey. wine glass)
skit gloss, clenst, no information regard ng ih's coker or design anialbble.
Description d'
(asiumer Volfin, shott glass)
Video
Unanswerable Question X
does burker
thee
Ygt
Description (narler, holkagg
'ostowerr
areog titem.
Altered Description looking dryoe)
Video
Unanswerable Quastion X
Hoos dott the soc humer's a intende ue with tolarr card wfloznes the oetione of their transoatio of the recepeied!
Answer Ygt
The gibrrles vedeoverille as the video dods wet hickdd avs sodnds referendes wasking sond. The clesets nceffored rolate
Deser (ption
(roceptootoo, taidlagg sesit)
resopriczist cedereveen indoulo froms sade caiit, ine pokor cards are not foomwrdd disassed.
Altered Description
d' (rsseptoorist, fowihigg pokor send)
Figure 11: Examples object-related unanswerable question-answer pair from our generated
UVQA Dataset
Unonswerable Question X
liow dees the seccor Hanrr whin comrpleeiy covered the secter will, zuruigo to com'aus pladeg efferrivay??
Answer Ygt
The dedittios iesemvvnr dlds lecuand the rolerion of valter playts compliately cevirred secter bull cejtally sinpitall plass itc pased the normal and of socter plaver, "Tvrrffrs, screarin destrbect addresset.
Description d
(porsor the ceverve
Description d'
(sooier player,
ball)
Video
Unanswerable Question X
does froses visli pueding shratooy
friiese, offest
Ygt
alds reseevverra leccasse serslates reletton
playor covered
provlded
saggsees sceverio thue devioned, in decing? iepossibls wind its ixpact fiyy planpin strrtooy lossii se the vilco soehast.
Description (frisece player, holikay friseos)
Altered Description d' (Rrekeee playe, devered ly. friskee)
Video
Unanswerable Question
of lcavicotioal jorkoy vorn "he playe wiss inoding video?
Can von
Yyt
"Thss assistion uneeswarabl bolaass the does Feataro Jon beokerball plaver inoolney only shows beskettkll plaser eq and matsi offical lecevvy..
Description (sasicerball pi layer, koading
Altered Description d' [seckerball plaver, Insoking)
Figure 12: Examples of relation-related unanswerable question-answer pair from generated UVQA Dataset
19

---

Published as a conference paper at ICLR 2025
whar Question
after
comting st warrepttnn tnagg!
Ygt
"Tiss sieshon preaacrcal the video only shews weer an red shar ceeting un to inurrpooon on sage. Thers wotson is de olive saire depicred in thes video covitcet.
Description Vontnar rod skirt op ou shage and Frovt of Hibrophone.
Altened Description
sones ap sharp stands front mirraphord.
Video
Unanswerable Question X
sees ong
Ygt
"The gies rix inveoweralld lescanse video does not foutture pag cet ng from staditshh dog diste it only sanws payy catting frim a ahhir day dlsh
Description
pogg dats from a siver dog Aish
Altered Description d'
pag cars from bractish dag dish
Video
Unamswerable Quaction X Whn fist min with tits slozigrsen jaikett pointing at comerking *hh video?
Answwer Ypt
kiae-yrron worrig wightt sscrrnng in Haangrree jospet selettug at someerrigg.
Descr lption
A mote se Joder arond walf polnts saneritag
Altered Descrition d' mon we brise syoot jackst aroed htss wal soints tomething
Figure 13: Examples of attribute-rellted unanswerable question-answer pair from our generated UVQA Dataset
A.5 PROMPT USED FOR EVALUATION
GPT-4 model (gpt-4o-2024-05-13) with the prompts shown in Figure [4] Additionally, following the evaluation methodology of prior Video-LLM works Lin al. 2023); b: Li al. (2024a b): Ahn et al. (2024), report the L.MM in This score assesses the quality of the mooel--genertee response y against the ground truth y ygr, using& rating scale from 0 as outlined in the evaluation prompt.
To determine the type(y) of the model's 's response,
Table [
evoluat concznee
chabee desigred contrcull uncertandingg sertive
erencrvett crmot desemis Mecntigggtt
inta canga gi prodicaaa arowee wi's tnttee snrver an canani witth geretted regaste adgis 1
banwer preloie and
icclate comee raso aeviime cang ann the souer
syown seopite thl.
usee
Butl fon calas.
annverabllyy rascringg
Usdt
Salowit eidewbesec paitiice
Draeci Jerne Va
Comne from Sen yal
Provide
Proppcllree wer coriiffo extten venoeonctic wrsoersle tnneeded wesches hoo
momene cormecTaae prodktat yver connech quenten www.oeeeeeeeeeeee. Cyel
porest mosiee dr CMMINIT TEChCC CAC.OAAAOOOOOO the nalor detinners
Moose par watn interne terea Tylive dell Wing
( Festartinn Prempe Aneoerab
7M Evali Prompt used Unseswentlee dattse
Figure 14: Prompt used for Evaluation: (a) Evaluation prompt for answerable dataset, and (b) Evaluation prompt for our unanswerable dataset.
A.6 POPE-STYLE UNANSWERABILITY DETECTION
In Section 6 we experiment if can detect unanswerable questions by reformulating them into a series existence-bate
question. To do this, we need to extract existence based sub-questions from the original question x, which achieve by prompting external LLM. Figure [S shows the prompt used for the LLM and Figure [ illustrates an example of the generated of existence-basee sub-questions x!.
x' {2c {z ..... _
20

---

Published as a conference paper at ICLR 2025
SYSTEM
You are an Intelligent chatbot designed to generate questions that check the existence of objects, attributes, and relations the given question. Your task create these questions based on the elements extracted from the original question. Here's how you can accomplish the tasic
INSTRUCTIONS:
Object: Extract the main nouns from the question.
Relation: Extract the main predicate (verb action) that describes relationship between the objects.
Attribute: Extract any adjectives that describe the features characteristics the object.
For each extracted element (object, relation, or attribute), generate question that checks its existence in the video. contain any apestrophes Provide the response in the form of Python list string without any other text.
generated questions
Please generate questions to check the existence of the elements extracted from the following questionsynyn
Original Question: [question
Please generate the response in the form of Python list string without any other text. For example, your response should look like this: video
[Its any videa?,
car?', person driving the videa?]
Figure 15: Prompt used to generate POPE-Style question
Video v
Unanswerable Question
How mony times does the firefighter repair the water pipe in the video?
Answer Ygt
This question is umanswerable because the video does not show the firefighter repairing the water pipe; it depiots a firefighter holding a water pipe and another holling cell phone, which is ewrelated to the act of looking at it.
of Existence-based Questions
Is there any firefightter in the video?
Is thers awy water pips in the video?
Does the firefighter repóir awything
Does angone repair the water pips in the video?
in the video?
Figure 16: Examples of Generated Existence-based Question Set
A.7 ADDITIONAL EXAMPLES MODEL PREDICTIONS
In Figure 17
we provide an additional example of model predictions from the unaligned model
Video-LLa VA (Lin et al. 2023) and the aligned models (SFT and DPO). Similarly, Figure [8] present another example using the unaligned model VLM-RLAIF (Ahn et al. (0224) alongside the aligned models (SFT and DPO).
21

---

Published as a conference paper at ICLR 2025
Uharewerable Qucaton
Video
the vido?
yge
This camosarales You vike doss not huaher ihena bertioor a peloe effice and en shild Therefere, thore is centoer shhel selyalrs
nolling orll greeling the sdeaccl edderr rise pelle ring muneer
The upeo fan sumevarnill cillt or the vits dui sleu
vides fecases un stonactee be wort polos officer and sithourt
"his tuunarparrlle lessaee
SET Algnet
polio office
onD Aigred
66) Osjatteinod (0) unarsmorable QA prodictinn
Hose
reciing
Arower Ygt
slouee catfeet and lealle ruildin,
very
aeurfic i imeraeenolle av
the stlles
Teb
workor
Model
Incell
Model
(b) Reldt onvellated (X) unarsweruble le QA predidion
throughest
Answery yat
qustiic seseworrrll as Hore bricy arossed anwageiin spesarag
ertugetnn
apsoman nomenaraiie
graze wulding ramter drvnccd
Tals seastion builec lnsersse the
Hersgghon vides
ATT
widding ottegiidh oes.
orangs
buttordlattel
Figure 17: Additional examples of model prediction from the unaligned model, the model aligned using Supervised Fine-Tuning (SFT), and the model aligned using Direct Preference Optimization (DPO), all based on Video-LLa VA (Li et al. 2024b).
Video
Unarnveralle Quastion
Whe Hie yoloe viflor schig diipottiin to hovooigggae ie the
Arowerr
The sport suceavervi beverse the viluo dees set siose polio office zang inancnalss alous pe kolder
effeer invisiigeee gens the soee bensese ans suceneeyund
The gexteer snnowwerale inceanoo Hid pa paing
gorup inaararriae becaroe
inopetit its The poiso othor oy ddeas rhi gats
Made
Heoi (xl Object- related (0) unanswerablle GA prediction
Yocil
Video I
Unarswertth le Quartion
Anstoer
Thee Bun hot anarmerdee hncane Ho vods dhon show achoo prossng the boppl, medlette beopl's
The tot hopp ocs murrange sheee stoce To infortat
"The mumuservlll trvetu vdde yoom stan mrning inpualic Aroeene peesing offer cappp.
"Te deoitio reviewerreee boieeso Hid sinee iouute dost me Maclley shape.
Relation unanswerable prediction
Unanowerable le Question
Hute 1906 the yersonie in lusi grisi rtdie whher meey oftee luoknn Aeeeet
ciserved Aortioe.
crapple tolool persu iorfiuer diadee This infrrmatce on dromm valll the fse Ais diess provie aifnsation dee panson viau eee--grnuc
covilde to denorets hew they revet offerr stopeesy
Stane
1d Artributte (14) ammocraaii QA prediction
Figure 18: Additional examples of model prediction from the unaligned model, the model aligned using Supervised Fine- Sine-Tuning (SFT), and the model aligned using Direct Preference Optimization (DPO), all based VLM-RLAIF (Ahn al. 2024).
22

---

Published as a conference paper at ICLR 2025
Table Performance Comparison by Unanswerability Type:
We report the Saee (Eq. for the three primary categories of unanswerability: Object, Relation, and Attribute-related questions. For Relation-related unanswerable questions, we further classify them into two subtypes: Intra-object relationships and Inter-object relationships, each subdivided into static and dynamic cases.
Unanswerability Type
Relation Type
Base Model
Object
Relation
Attribute
Intra
Intrasyaunic
Interaui
InteTäysuni
unaligned
0
0
0
0
0
0
Video-LLLVA
SFT (ours)
0.46
0.51
0.30
0.36
0.61
0.43
0.59
DPO (ours)
0.45
0.49
0.33
0.35
0.60
0.39
0.56
unaligned
0
0
0
0
0
VLM-RLAIF
SFT (ours)
0.63
0.69
0.35
0.73
0.67
0.69
0.73
DPO (ours)
0.60
0.63
0.42
0.64
0.53
0.62
0.68
A.8 PERFORMANCE ANALYSIS BY UNANSWERABILITY TYPE
We conducted a detailed analysis of model performance across different categories of unanswer- ability within the UVQA dataset. Specifically, in Table 2 unanswerability: object, relation, and attribute.
we examined three primary types of
For relation-related unanswerability, we further categorized the questions into two subtypes:
1. Intra-object relationships, involving single object, and
2. Inter-object relationships, involving multiple objects.
Both subtypes were further divided into static and dynamic relationships. Examples of each categories is:
Intra po ice officer standing
Intragynamie match official walking
Inter police officer looking at pedestrians
Interaynamie: match official massaging soccer player
The results indicate that object and relation-related unanswerable questions achieve comparable performance, whereas attribute-related unanswerable questions consistently pose greater challenges for all evaluated models (i.e., Video-LLave (Lin al. (023) and VLM-RLAIF (Ahn et al.. 2024)). Importantly, this performance gap is not attributable to data imbalance, as the UVQA dataset was carefully balanced across these three categories during training, as detailed in Section This finding highlights that models generally struggle more with discerning attribute-related unanswerable questions, suggesting an inherent difficulty addressing this category. On the other hand, for the subcategories of the relation-related unanswerable questions, the performance did not show consistent pattern, highlighting variability in the capabilities of different VLMs.
A.9 CAN PROMPTING DETECT UNANSWERABILITT
In this section, we investigate whether prompting alone can achieve alignment for answerability. Specifically, we prepend the following prompt to the question: "If question cannot be answered using the video content, state that it is unanswerable and provide reason." Table 3 presents the results of various open- open-sourced Video-LLM, (i.e., VLM-RLAIF (Ahn et al. 2024). and LLaVA- Video-Qwen? (Zhang al. (024bb) evaluated in two settings: without the prompt (unaligned) and with the prompt (prompt-atee -lligee.-llinedd)..............)
The results indicate that even the best-performing open-sourced Video-ILLMs, regardless of model size, fail to benefit from the explicit answerability prompt, showing little to no improvement in performance. This highlights the limitations of relying solely on prompting to address unanswerability and underscores the importance of the alignment approach proposed in this work.
23

---

Published as a conference paper at ICLR 2025
Table Evaluation of Answerability Alignment and Absolute Performance of Prompting- based Method
Answerability
Aligument Performancc
Absolute Performance
Model
F1
Sweet
Spenis
Suse.
Sam
Sees
LLMgo
VLM-RLAIF (7B)
unaligned
0.00
0
0.33
0.25
2.36
prompt-aligned
0.08
0.35
1.66
DPO (ours)
0.66
0.08
0.5
0.64
0.69
0.53
2,93
WaVAA-idee--wen2222 (7B)
unaligned
0.00
0.33
0.38
2.26
prompt-aligned
0.04
0.01
0.03
0.34
0.38
2.33
LLaVA-Vide--Qwel2 (72B)
unaligned
6.02
0
0.33
0.38
2.35
prompt aligned
0.36
2.50
A.10 VISUALIZING THE TRADE-OFF: PARETO FRONT ANALYSIS
In multi-objective optimization, trade-off often arises between conflicting objectives. This trade-off can be visualized using a Pareto front, where each point represents a specific balance or equilibrium between the objectives (Jin & Sendhoff 2008 Lin al. 2019 Navon al., (021). Figure 19 we present a Pareto front analysis to visualize the trade-off between performance on standard answerable QA dataset and the unanswerable UVQA dataset.
The initial point (Unaligned Model) achieves high accuracy on the answerable QA dataset but performs poorly on unanswerable questions, highlighting its inability to recognize or handle unan- swerability effectively. When alignment for answerability is performed using SFT (represented as a square the figure), we observe an increase in accuracy on the unanswerable QA (UVQA) benchmark, accompanied by slight decrease in accuracy the answerable QA dataset. In contrast, alignment using DPO (represented as a star in the figure) achieves a more favorable balance, with the resulting point positioned closer to the optimal region on the Pareto front compared to SFT. This demonstrates the effectiveness of DPO alignment in achieving a better trade-off between the two objectives, in line with the findings presented in the main results in Section 5.3
SFT
DPO
Fetbel Povet (inalgad
0.5
Acsworable Vidao (Ariooyyle Accurayy
0.4
Ahregna
0.3
0.2
0.0
0.2
Unancwerable QA (uurr)cccuray
Figure 19: Pareto front visualization using the VLM-RLAIF model (Ahn et al. 2024
24

---

Published as a conference paper at ICLR 2025
A.11 HUMAN EVALUATION OF MODEL PREDICTION
In this section, we conduct a human evaluation to assess the model's prediction results. Five annotator rate each prediction on scale of 0 5, using the criteria outlined in Table for unanswerable questions and Table for answerable questions.
Table 4: Scoring Criteria for Unanswerable Questions
Score
Description
The model fails to identify the question as unanswerable and generates an answer that falls cutside the boundaries vidco's 's information.
The model fails to identify the question unanswerable but includes some correct information from the video, showing partial undersianding.
The model conrectly identifies the question unanswerable but provides incorrect reasoning.
The model comectll identifies the question as unanswerable and provides partialyy correct reasoning.
The model correctly identilies the question as unamswerable and provides correct reasoning without much detail.
5
The model correctly identines the question unanswerable and provides correct reasoning with details supported by comprebensive, accurate vidco information.
Table Scoring Criteria for Answerable Questions
Score
Description
The model fails provide the correct answer the question.
The unswer is incorrect but includes some information from the video, showing partial understanding.
The model provides the correct answer but includes additional, incorrett information in the response. The model provides the dctails.
without hut does not offer additional context supportnn
model gives the correct answer response.
relevant details
video enhance
The model delivers the correct answer with comprehcasive, detailed context from the video, providing
thorough and nuanced response.
Figure (a) illustrates the interface provided to human annotators, where video content was presented as individual frames for evaluation. Figure (b) reports the average scores assigned by the five annotators, based on 100 random samples from the evaluation set for each model, comparing unaligned model predictions with aligned ones (i.e., SFT and DPO) for the VLM-RLAIF (Ahn et al.) 2024) backbone. Notably, the figure shows that the aligned models outperforms the unaligned model, with the scores showing strong correlation with the LLM reported in Table [
LLMazore
Human Exmluation
Uniignal
SFT
DPO
3.0
1.0
VLM-RLAIF
(A) Interface used evaluation
Comparison of human evaluation and LLMscore
Figure 20: (a) Interface provided to human annotators for evaluation. (b) Comparison of scores between unaligned predictions and aligned predictions (SFT and DPO) for the VLM-RLAIF (Ahn) al. 2024) model.
'All annotators have TOFFI. iBT scores above 100 and hold least hachelor's degree.
25

---

Published as a conference paper at ICLR 2025
Table Out-of-Distribttion Evaluation of Answerability on Alignment and Absolute Perfor- mance.
A Aignment Performance
Absolute Performance
Base Model
f()
Sevret.
Spernis.
1
Sdsse.
Salign
Sace. f
L.M..ooe 1
unaligned
0
0
0.33
0.27
1.76
Video-LLaVA
SFT (ours)
0.37
0.33
0.43
0.46
0.41
2.28
DPO (ours)
0.12
0.67
0.40
0.58
0.44
2.37
unaligned
0
0
0.33
0.24
1.56
VLM-RLAIF
SFT (ours)
0.15
0.25
0.45
0.52
0.45
2.21
DPO (ours)
0.
0.43
0.63
0.44
2.24
A.12 DUT-OR-DISTRIBUTION EVALUATION SET
Although all videos in the UVQA evaluation set are distinct from those in the training set, it is valuable to investigate how the use of videos from entirely different sources, coupled with QA pairs from different distributions, affects the model's predictions. To this end, we leverage videos from the MSR- VTT dataset (Xu et al. 2016) and construct beyond the informational boundaries of the videos, rendering them inherently unanswerable. This approach allows us to evaluate the model's ability to recognize unanswerable questions and assess its robustness when faced with diverse video content and question distributions. We construct the QA pairs using three human annotators and collect 100 human-annotal unanswerable QA pairs for videos from the MSR-VTT. Figure 21 illustrates the annotation interface and provides an example of a labeled unanswerable QA pair.
QA pairs designed to extend
standing
doing?
Answer: The question is unansverable because the scenario of ran standing in the car does not occur In the video. Therefore, the question cannot be addressed.]
Colnt
Figure 21: Interface provided to human annotators for collecting out-of-distributie evaluation set where use the MSR-VTT (Xu al. 2016) the video source and collect human-labeled unanswerable QA pairs.
In Table we observe that even on the out-of-distributioo evaluation set, the aligned model outper- forms the unaligned model, demonstrating its robustness and ability to generalize beyond the training distribution. While the improvement in alignment performance and absolute performance is slightly less compared the in-distribution evaluation results in Table I the aligned model consistently shows enhanced performance.
'All annotators have TOFFI.
iBT above 100 and hold least hachelor's degree.
26


Reviews:
[{"review_id": "5yTA2W9xu7", "reviewer": "Reviewer_GEbz", "summary": "This paper introduces the concept of alignment for answerability in the context of video LLMs. Current models and data do not consider out-of-scope questions that a user might ask, and hence they don't recognize them as unanswerable. A definition and a metric to assess this phenomenon are presented. Furthermore, a new dataset is proposed containing unanswerable questions, and fine-tuning models on this dataset shows improved alignment towards recognizing unanswerable questions.", "questions": "Can the authors provide more qualitative examples of unanswerable questions from the proposed dataset? Similarly, can we also get more qualitative examples of existing & aligned model's reactions to such questions?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is well-written and clearly introduces the concepts, definitions and metrics. The contribution is original as it formally recognizes a new problem, i.e. multimodal LLMs answering clearly unanswerable questions, and proposes solutions to it.\n- The proposed problem framing and metric make intuitive sense. The definition takes into account the reasoning of why a question is unanswerable. The metric takes into account excessive refusal to answer which is a big problem with LLMs.\n- A new dataset for alignment for answerability is presented. This is a valuable addition to the community as it creates out-of-scope QA pairs which we rarely see.\n- The dataset creation process is simplistic, modifying a constrained set of objects and their relations, or changing object attributes. Furthermore, the evaluation dataset is verified manually. So, the dataset is likely dependable.\n- Experimental results after fine-tuning on this dataset show clear improvements in alignment (Table 1), and the ablation study in Figure 5 outlines the drawback of an alternative.", "weaknesses": "The main weakness is that the proposed QAs in the dataset seem to be mostly geared towards detection capabilities without requiring much reasoning. This in turn makes the corresponding dataset construction, and improving model's capabilities on this axis rather easy. Some qualitative examples presented are \"What breed is the cat in the video?\", \"What color laptop the presenter is holding?\", \"How many times does a person in gray shirt appear in the video?\", etc. Recently, many reasoning based video-language benchmarks are proposed [1, 2] while this paper is more similar to older benchmarks [3].\n\nOther notable weaknesses:\n- The metric is rather complex. If we just want to detect unanswerable questions, and balance precision and recall, why not use F1-Score?\n- The metric is defined between aligned model and unaligned model. However, if one wants to test the ability of a given model regarding unanswerable questions, the proposed metric cannot be used.\n- Using LLM-in-the-loop to create the dataset may create some bias, hallucinations and incorrect QA pairs. \n- Human performance on the evaluation set is not presented.\n\n[1] Perception Test: A Diagnostic Benchmark for Multimodal Video Models \\\n[2] MVBench: A Comprehensive Multi-modal Video Understanding Benchmark \\\n[3] ActivityNet: A large-scale video benchmark for human activity understanding", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nPBjbtRTcV", "reviewer": "Reviewer_y4aQ", "summary": "This paper studies the phenomenon that some video language models can't refuse to answer a question when the question is irrelevant to the video. The authors first showcase such failure cases in existing VLLMs, then provide a method and metric to identify such case, and finally propose a dataset to finetune existing VLLMs to improve this issue.", "questions": "- Overall, I feel this paper propose a valid solution to a small problem. The solution might have issues on overfitting the model on the particular task, and the problem may not exist on more modern VLLMs. My current rating is a week reject. I am happy to raise my rating if the authors address my concerns in the rebuttal.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper executed a valid pipeline of improving VLLM on a task: identifying the problem, curating a dataset to fix the problem, and finetuning the model to show improvements. The paper shows a good practice on the task of introducing refusing to answer for VLLMs.\n\n- The alignment score defined in section 4 makes sense to me.\n\n- The authors conducted experiments on a variety number of VLLMs in Table 1.", "weaknesses": "- My main feeling about reading this paper is I feel the problem of refusing to answer is a bit small and artificial. While showing the failure case in Figure 1, I feel it is also important to show if the problem can be migrated by explicit prompting, e.g., add to the prompt \"Say 'can't answer' if you are not sure\". Even though I believe the authors proposed method with finetuning on the curated dataset may still be better, I feel the problem is not as big as the author claimed.\n\n- Adding to the above point, the authors only evaluated relatively small VLLMs in Table 1 (Video-LLaVA/ VideoChat2/ LLaMA-VID), while skipping the more recent SOTA VLLMs like Qwen2-VL, LLaVA-OneVision, and GPT-4o or Gemini 1.5. I am expecting these more well instruction-tuned models may have less answerability issues.\n\n- If I understand correctly, the authors propose to solve the problem by finetuning the VLLM on a dataset. A clear shortcoming is that the models might drop performance on other tasks, which is not desirable. Please provide some discussion on this.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bSTXE9Hhi3", "reviewer": "Reviewer_yiLe", "summary": "The paper aims to improve Video-LLMs’ capabilities of differentiating visually-unaligned questions, while maximally retain the performance on answering common answerable questions.\nSpecifically, the paper first defines a suite of metrics to evaluate alignment for answerability. It then collects data for both training and evaluation. The experiments show that existing Video-LLMs almost cannot refuse to answer unaligned questions. While SFT and DPO can effectively improve the overall performance, DPO significantly outperforms SFT in maintaining the models’ original performance while handling unaligned questions.", "questions": "see weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe work is overall well-developed and shows the authors’ insights in the problem.\n2.\tThe constructed benchmark could be valuable, with dedicated evaluation metrics and annotations (reason for unanswerable).\n3.\tThe paper implements both SFT and DPO for improving existing Video-LLMs with the training data and shares helpful insights.", "weaknesses": "1.\tThe paper neglects to discuss many existing works that study the unanwerability of (V)QA models (see my attached references). \n\n2.\tIt would be better to conduct more in-depth analysis to help understand what kinds of unanswerable QA are more challenging to resolve in videos: static objects/attributes/relations vs. dynamic actions. \n\n\n[1] Whitehead S, Petryk S, Shakib V, et al. Reliable visual question answering: Abstain rather than answer incorrectly[C]//European Conference on Computer Vision. Cham: Springer Nature Switzerland, 2022: 148-166.\n\n[2] Guo Y, Jiao F, Shen Z, et al. Unanswerable visual question answering[J]. arXiv preprint arXiv:2310.10942, 2023.\n\n[3] Zhao Y, Zhang R, Xiao J, et al. Towards Analyzing and Mitigating Sycophancy in Large Vision-Language Models[J]. arXiv preprint arXiv:2408.11261, 2024.\n\n[4] Brahman F, Kumar S, Balachandran V, et al. The art of saying no: Contextual noncompliance in language models[J]. arXiv preprint arXiv:2407.12043, 2024.\n\n[5] Wen B, Yao J, Feng S, et al. The art of refusal: A survey of abstention in large language models[J]. arXiv preprint arXiv:2407.18418", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F6UGw2ptzi", "reviewer": "Reviewer_jkxc", "summary": "This paper addresses the issue of \"answerability\" in Video Large Language Models (Video-LLMs). Current Video-LLMs excel at answering questions directly related to the content of a video, but struggle with questions that go beyond the video's scope, often hallucinating plausible but incorrect answers. The authors propose a framework called \"alignment for answerability\" to train Video-LLMs to recognize and refuse to answer such unanswerable questions.\n\nThis framework involves:\n- Alignment:  Training Video-LLMs to assess the relevance of a question to the video content and respond with \"unanswerable\" when appropriate.  They formally define this alignment process and associated scoring functions.\n- Evaluation Metrics:  Beyond simple accuracy, the authors propose a set of metrics to comprehensively evaluate the alignment process. These include measuring how often the model refuses to answer valid questions (Excessive Refusal), how often it correctly answers previously refused questions (Permissiveness), and how well it identifies and declines truly unanswerable questions (Discretion).\n- Dataset Creation (UVQA): The authors created a new dataset, UVQA, specifically for training and evaluating answerability.  They leverage existing video-description datasets, altering the descriptions to generate questions that are unanswerable based on the accompanying video.  They further categorize these questions based on the type of mismatch (object, attribute, or relationship).\n\nExperiments demonstrate that models trained with this framework and the UVQA dataset significantly improve their ability to handle unanswerable questions. The authors also explore an alternative approach based on decomposing questions into multiple existence-based sub-questions (inspired by the POPE approach for image-based LLMs), but find it less effective and computationally more expensive than their proposed framework.", "questions": "- The approach could be extended to images in future work. Is there any challenge the authors anticipate in doing so?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper tackles an important yet often overlooked issue of answerability in Video-LLMs.\n- The paper defines sound metrics and data generation pipeline.\n- The results on the in-domain evaluation set are convincing, notably with a positive comparison against the POPE baseline.", "weaknesses": "- The evaluation set is curated from the same distribution as the training set, which may overestimate the benefits and results of the alignment procedure. It would be great to see the results on a fully human-annotated set, ideally on videos from a different distribution (e.g. using videos from MSR-VTT).\n- Having results on academic VideoQA benchmarks (e.g. the full Activitynet-QA evaluation set) would enable to double check the absence of accuracy degradation compared to the state of the art. Has any trade-offs between answerability performance and standard VideoQA performance been observed?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)