Long-Form Answers to Visual Questions from Blind and Low Vision People
Authors: Mina Huh
Fangyuan Xu
Yi-Hao Peng
Chongyan Chen
Hansika Murugu
Danna Gurari
Eunsol Choi
Amy Pavel
Venue: COLM (2024)
Keywords: Visual Question Answering, Long-form Question Answering, Vision Language Models, Accessibility
TL;DR: We explore how human experts and VLMs generate long-form answers to visual questions from blind and low vision people and evaluate them.

Abstract: Vision language models can now generate long-form answers to questions about images – long-form visual question answers (LFVQA). We contribute VizWiz-LF, a dataset of long-form answers to visual questions posed by blind and low vision (BLV) users. VizWiz-LF contains 4.2k long-form answers to 600 visual questions, collected from human expert describers and six VQA models. We develop and annotate functional roles of sentences of LFVQA and demonstrate that long-form answers contain information beyond the question answer such as explanations and suggestions to retake photos. We further conduct automatic and human evaluations involving BLV and sighted people to evaluate long-form answers. While BLV people perceive both human-written and generated long-form answers as plausible, generated answers often hallucinate incorrect visual details, especially for unanswerable visual questions (e.g., blurry or irrelevant images). To reduce hallucinations, we evaluate VQA models on their ability to abstain from answering unanswerable questions.

Long-Form Answers to Visual Questions from Blind and Low Vision People
Mina Huh 9 Fangyuan Xu , Yi-Hao Peng Chongyan Chen o, Hansika Murugu Danna Gurari Eunsol Choi Amy Pavel
The University of Texas at Austin, Carnegie Mellon University, Hong Kong University of Science and Technology University of Colorado Boulder
minahuh@cs utexas. edu
Abstract
Vision language models can now generate long-form answers to questions about images long-form visual question answers (LFVQA). We contribute VizWiz-LF dataset of long-form answers to visual questions posed by blind and low vision (BLV) users. VizWiz-LF contains 4.2k long-form an- swers to 600 visual questions, collected from human expert describers and six VQA models. We develop and annotate functional roles of sentences of LFVQA and demonstrate that long-form answers contain information be- yond the question answer such as explanations and suggestions. We further conduct automatic and human evaluations with BLV and sighted people to evaluate long-form answers. BLV people perceive both human-written and generated long-form answers to be plausible, but generated answers often hallucinate incorrect visual details, especially for unanswerable visual questions (e.g., blurry or irrelevant images). To reduce hallucinations, we evaluate the ability of VQA models to abstain from answering unanswer- able questions across multiple prompting strategies.
1 Introduction
Traditional visual question answering (VQA) models respond to visual questions about images with short answers. This is because they were designed for mainstream dataset challenges for which answers typically are brief, containing one or two words (Antol et al., 2015 Gurari et al. 2018 Goyal et al. 2017). The rise of large vision language models (VLMS
has introduced new class of VQA models that can generate long-form answers (Alayrac et al. 2022 Team et al. 2023 Driess et al.) 2023 Chen et al.) 2023c |Achiam et al.) 2023). 3 The progress in VLMs meets a long-standing need from VQA solutions (Chen et al.) 2023b). For instance, some questions necessitate long-form answer, such as "How do make this boxed cake?" Also, some users find answers with nuanced information be helpful, such as answers with explanations and supplementary details (Xu et al. 2022) Naik et al. (023). While long-form visual question answering (LFVQA) has enormous potential, we have limited knowledge about the content and quality of long-form answers.
Our work aims to understand the content and quality of long-form answers to visual questions asked by blind and low vision (BLV) people. LFVQA can be particularly impactful for BLV people who take pictures and ask questions to overcome real visual accessibility barriers. In fact, BLV people already active consumers of long-form VQA applications like Be My AI (BeMyEyes) 2020) powered by GPT-4V. Authentic VQA involves unique challenges not present in artificial settings, such conversational questions and low-quality images due to blur and framing errors. We seek to understand the potential and limitations of LFVQA for BLV users given its societal impact.
'Our data and code available httttp///ggiiabb..m//tttac//////

---

VizWiz
VizWiz-LF
FUNCTIONAL
INFORMATION
Original Image
image blury, includes the word MUSTARD.
label that confrrmaion
image
Image qualiyy
Given the color of the liquid visible through the bottle, it's sauce [-]
answer
explanation
Image content
likely type
Mustard sauces are commonly used as a condiment or dressing for sandwiches, salads, and many other dishes. you have information clearer picture, could attempt to provide a more accurate identificatio..
auxiiaary information
external
suggestion
Original Question
What in
Gemiin
This bottle contains Heinz Distilled White Vinegar.
answer
image content
Original Short Answer
tomato
can't tell what the bottle.
answer fallure
image content
If had to guess, would say it was some sort of cooking oil because has
color and pictures vegetables [.] could an apple juice or another type of juice.
explanation
image content
answer
image content
Figure 1: We collect long-form answers for image-question pairs in the VizWiz dataset (uu- rari et al. 2018). For each visual question, we collect answer from human expert describers and 6 VLMs (GPT-4V, Gemini, LLaVA, InstructBLIP, QWEN, BLIP-2). To understand the content and structure of long-form visual question answers, we create a taxonomy of func- tional roles and annotate answer functional roles and information types at a sentence level.
We introduce Vizwiz-LF, dataset of 4.2k long-form answers to visual questions from BLV people seeking visual assistance (Bigham et al. 2010
and evaluate long-form answers from human experts and Gurari et al. 2018). We collect
tate-ot-theear VQA mod-
els (GPT-VV (Achiam et al. 2023), Gemini (Team et al. 2023), LLaVA (Liu et al. 2024), InstructBLIP Dai et al. 2024), Qwen-VL-Chat (Bai et al. 2023) and BLIP-2 (Li et al. 2023a
To understand the content of LFVQA, we design and annotate the communicative roles (eg., answer, explanation, suggestion) and information sources (e.g,, image content, image quality, or external information) of long-form answer sentences in our dataset. We create classifir for the functional roles and information sources LFVQA that performs on par with humans then annotate our dataset with this classifier (example in Figure IT. While the majority of answers from 5 VLMs (Gemini, LLaVA, InstructBLIP, QWEN, BLIP=2) contained only two functional roles (confirmation, answer), human expert describers and GPT-4V answers frequently included many functional roles (e.g., explanation, suggestion, auxiliary information). Most long-form answers described image content, but human experts and GPT-4V also frequently described image quality.
To assess the performance of VLMs in LFVQA, we conduct automatic evaluation of long-form answers using reference-based metrics (ROUGE (Lin) 2004), METEOR (Eliiott & Keller 2013), BERTScore (Zhang et al. 2019) and LAVE (Mañas et al. 2023 with short- form reference answers from VizWiz and long-form reference answers from VizWiz-LF, Reference-based VQA evaluations typically use short reference answers and thus penalize long answers for including extra information (Krishna et al. 2021 Xu et al. 2023). We show that extracting answer sentences from long answers using our classiher can minigate this.
To understand how humans evaluate long-form answers, we conduct an evaluation study with both sighted and BLV people. We examined the effect of visual priming bias by evaluating with four different conditions of (BLV, sighted x (with image, without image) Our results also reveal that sighted people's 's evaluation without an image is not strong proxy for BLV preferences. For instance, BLV evaluators tend to perceive incorrect answers as more plausible and rate extraneous details as less relevant than sighted evaluators.
As VLMs often hallucinate with answer sentences for unanswerable visual questions, we assess six VLMs' ability to abstain from providing an incorrect answer. To determine whether a model abstains, we use our functional role classifier to check for the Answer
For BLV users, simulate "with image condition" by providing them description of the image from the original VizWiz dataset.
2

---

Failure role in the long-form answer, which expresses model's inability to answer the question. Only GPT-4 achieved high recall for abstention (0.82) followed by QWEN (0.42) with default settings. We further experiment with multiple abstention approaches and reveal that prompting VLMs with abstain instructions can reduce hallucinations.
While this work addresses visual questions posed by BLV people, our findings can inspire broader research in VQA. First, our introduction of long-form answers to supplement the short answers in the original VizWiz dataset results in the first dataset with both short and long answers. Potential downstream benefits include enabling the transfer from short- answer VQA tasks, which current models already excel, to long-answer VQA tasks. Second, our proposed LFVQA functional roles can support improvement and evaluation of LFVQA for our use case and beyond. Finally, our human evaluation highlights the importance of evaluating LFVQA beyond factual accuracy (the focus of short-form VQA) to further consider end users' experience with metrics such as relevance and plausibility.
2 Related Work
VQA Datasets Most existing VQA datasets have brief answers consisting of one to two
words collected from crowd workers (Antol et al. 2015 Zhu et al. 2016 Gurari et al. 2018 Wang et al.) 2017 Krishna et al.) 2017 Wu et al. 2017). This includes the widely-studied dataset with questions asked by BLV individuals, VizWiz-VQA (Gurari et al., 2018), even
though BLV people often explicitly ask for detailed answers (e-g., "Yes llus is a Google Street View image, need detail, detail of the description, as much possible." " user question from VizWiz dataset). To explore the potential of LFVQA in accessibility, we gather and analyze a dataset of detailed, peattenc-to-ppragraape expert describers and 6 modern VLMs.
answers to VizWiz questions collected from
A few VQA datasets also provide longer responses: ContextVQA (avg. 11 words) (Naik et al., 2023), ScienceQA (Saikh et al. 2022) (avg. 4 words), and VQAOnline (Chen et al. 2023b (avg. 173 words). While the answers in these datasets are collected online or from crowd workers, we collect high-quality answers from experts proficient in image description tasks. We also address questions posed by BLV individuals, which cover a different distribution of images (e.g,, quality) and questions (e.g,, describing physical objects).
Long-form Question Answering (LFQA) LFQA systems (Fan et al. 2019 Nakano et al.
2021) generate comprehensive, paragraph-level answers to text-only questions. Prior work etu 2022) analyzes the content types of human- long-form answers from online community forums (Fan al., 2019: Nakano al. 2021), finding that they convey rich information beyond simply answering the questions. They propose six functional roles for answer sentences, such as "example" and "auxiliary information". As we develop functional roles for LFVQA generated by both humans and models, we reveal new roles (eg,, confirmation of photos, answer failure) and different role distributions across experts and models. We also annotate information sources of LFVQA answers to assess how models reference images (egg,, image content, image quality).
Evaluating VOA Models Most traditional automatic VQA evaluation metrics are reference-based (Papineni et al., 2002 Lin 2004: Elliott & Keller 2013 Zhang et al. 2019). Yet, Krishna et al. (2021) and Xu et al. (2023) highlighted the difficulty of evaluating lorg-form
answers using these metrics designed for short-form answers, as they poorly correlate with human judgments. Recent LLM-based metrics (Chan et al. 2023: Liu & Chen 2023
Kim et al. 2023 Huang et al.) 2024) align better with human judgments, yet have not been explored in LFVQA. Recently, Jing et (2023) introduced a reference-fre free metric evaluate hallucinations in VLMs' answers to visual questions. Researchers have also explored VLMs' ability to abstain when having low confidence to reduce hallucinations (Whitehead et al., 2022 Li et al. 2023b: Wang et al. 2023). We build upon research on automatic evaluation of VQA and further conduct a human evaluation with both sighted and BLV individuals to understand their preferences. As VizWiz frequently contains unanswerable questions, we present an evaluation of VLMs' ability to abstain from answering when provided with images of poor quality.
3

---

ViaWiz
VizWiz-LF
# words
A of words
#f words
Category
Question
Answer
Human
GPT-V
Gemini
LLaVA
QWEN
BLIP-2
InstructBUIP
Identification
5.2 (4.3)
(1.2)
42,9 (30.0
720 (25.4)
15.7 (15.6)
28.9 (21.0)
21,2
12.5
(4.9.
Description
(5.0)
(1.4)
(30.)
57.6 (46.5)
(18.0)
(17.9)
12.2 (2.7)
(13.6)
Reading
7.9 (4.4)
(.8)).))
39.8 (36.5)
66.5 (54.9)
16.3 (27.6)
22.2 (30.4)
18.6 (26.8
12.1 (3.0)
20.2 (17.8)
Others
(7.4
(1.7.
(37
96.6 (541)
(29.5)
41.9 (38.4)
34.7
12.6 (3:4)
23.0 (25.0)
Total
(5.9)
(..6)
(3.00)
73.2 (48.9)
16.4 (23.7)
27.0 (29.8)
22.0 (26.8)
12.3 (3.0)
18.9 (17.9)
Table Statistics of the sampled VizWiz data and the long-form answers from human expert describers and 6 VLMs. We collect 150 questions from four categories. Numbers in each cell represent the average with the standard deviation in parentheses.
3 Dataset
Visual Questions Our dataset extends the VizWiz dataset (Gurari et al. 2018)) that contains visual questions from BLV individuals seeking visual assistance. To select diverse range of visual questions, we sample 600 image-question pairs balanced between all four question types (Identification, Description, Reading, and Others) in VizWiz question taxonomy (Brady ltt al. 2013). See A.1.1 for details on sampling and filtering questions.
Long-form Answer Collection To collect long-form answers from expert describers, we hired 20 people experienced in describing images for BLV people using Upwork Unlike the VizWiz dataset [Gurari et al. 2018) that collected crowd workers' short and quick responses, we encouraged expert describers to write detailed responses. Describers spent 3.9 minutes (SD=1.2) per question ($..1.2). To evaluate the performance of modern VLMs in LFVQA, we generated long-form answers with six VLMs: GPT-4V (Achiam et al. 2023), Gemini (Team al. 2023), LLaVA (Liu et al. 2024), InstructBLIP (Dai et al. 2024), Qwen-VIL VL-Chat (Bai et al. 2023 and BLIP-2 (Li et al. 2023a We select these models as they are publicly available, have strong zero-shot VQA capabilities, and represent diverse model architectures.
|
In Table we report statistics comparing the collected long-form answers with the original VizWiz short-form answers Human expert long-form answers are 24x more words than original VizWiz short answers, indicating previous short answers were insufficient. Modern VLMs also generate long responses, but diverge from human expert answers GPT-4V answers are 1.7x longer, while the other VLMs generate answers that are 0.7x or shorter.
4 Functional Role Analysis
Functional Roles We design and annotate functional roles of sentences in LFVQA, to characterize the communicative goal of answer sentences in addressing visual questions. To derive the taxonomy, three authors used open coding (Hashimov) 2015), an iterative process for assigning conceptual labels to segments of data. Then, they met to merge similar functional roles and create a codebook with a name, definition, and example for each functional role. The researchers iteratively coded samples and revised the codebook to achieve final codes, containing eight functional roles (Confirmation, Answer, Answer Failure, Auxiliary Information, Auxiliary Information Failure, Explanation, Suggestion and Miscelleneous). We provide definitions and examples for each role in A.2.1
Information Source Short-form answers contain mostly visual information about the image content (Figure T. We observe that sentences in long-form answers often provide additional information, such as describing the image quality or providing external knowl- edge outside of the image. Thus, create separate taxonomy to categorize sentence-level
'https:/ //ttps://www... upwork.com/
*We generated long-form answers with a zero-shot setting, Configuration details in A.1.3
We used the majority-voted answer among 10 crowd answers (Zeng al. 2020 Chen 2023a
4

---

Answers (%)
Confirmation
Answer ans..
Ans. Failure
Audliary ans.
Aux, Failure
Explanation Suggestion ans.. unans.
Misc.
ins..
unans.
unans.
aMa.
unams.
umans.
aNs.
unans..
wnans.
wnans.
Expert
50
50
96
62
6
56
53
45
2
3
35
58
6
27
3
7
GPT-IV
61
60
72
38
59
52
4
1
84
35
63
13
Gemini
29
15
97
87
2
14
16
13
0
0
11
2
3
LLaVA
53
39
98
87
2
14
29
0
0
24
2
15
2
OWEN
46
30
89
11
15
1
4
BLIP-2
46
30
85
76
8
16
3
1
0
0
7
1
11
11
InstructBLIP
35
96
Table 2: Distribution of answers with each functional role to answerable (ans.) and unan- swerable (unans.) questions. 230 of 600 questions were marked unanswerable in VizWiz.
information sources following the same process. We identify sentences with the following information sources: (1) visual information about the Image Content, (2) descriptions of the lamage Quality, and (3) External Information. We provide definitions and examples for each
information source in A.2.1
Annotation and Classification To annotate the functional roles and information source types of the 4.2k long-form answers collected in 33 we first collect human annotations of 180 long-form answers (522 sentences) as ground-truth. Given the question, the image, and the answer paragraphs, the annotation task is to (1) assign up to three functional roles and (2) assign all applicable information source types for each sentence in the answer paragraph. Five authors participated in three-way annotations and reached substantial agreement with Fleiss Kappa (Fleiss& & Cohen 1973 (x 0.76 for functional roles, = 0.87 for information sources). We annotate the rest of the data using a GPT-4 classifier, which is constructed with few-shot prompt with definitions of each role and eight in-context examples (see 5A.2. for the full prompt). We evaluate the classifier's 's performance against human annotations, achieving a weighted per-role F1 of 0.74 (compared to human-human F1 of 0.79).
4.1 Analysis
Table shows the distribution of functional roles and information sources of long-form answers in our dataset. As 38% of our sampled questions are marked as unanswerable in VizWiz
we report the distribution for answerable and unanswerable questions separately.
Expert and GPT-4V's answers contain diverse functional roles. The most common func- tional role in answers from both experts and six VLMs was a direct Answer to visual questions. LLaVA and Gemini have the highest overall percentage of answers that had Auswer roles. Gemini often generated single-sentence answers, only addressing the question without providing additional information. On the other hand, expert and GPT-4V answers have sentences with diverse functional roles including Confirmation, Explanation, Auxiliary, and Suggestion. GPT--VV answers often contain Confirmation sentence when the visual ques- tion does not refer to a specific object (e.g., The image you 've provided is blurry, but it shows part of a bottle with blue label.) confirm that users shared the right image. Answers from both experts and GPT-4V provide Auxiliary information to complement the answer. For example, when asked about an attribute of an object in the image (e.g., "What is the color of this f-shirt?"), GPT-4V and expert answers additionally describe the same attribute of other objects in the image(egg,, color of the pants), or other attributes of the same object (e.g., text logo the t-shirt). Example Misc. sentences include organization sentences (e.g., "The screen displays the following texf:") or final remarks (e.g,, "Hope this helps, thanks!") In BLIP-2, answers sometimes repeat the input question and are classified Misc.
Most VLMs rarely abstain, even when the question is unanswerable. Four VLMs (Gem- ini, LLaVA, BLIP-2, InstructBLIP) all have a low percentage of answers with Answer Failure sentences for both answerable and unanswerable questions (Table 2). Expert, GPT-4V, and QWEN indicate Answer Failure more often to unanswerable questions, yet GPT-4V over- abstained to answerable questions when the image was not clear. Expert and GPT-4V often
VizWiz(Gurari Following VizWiz Gurari et al. 2018), we consider the questions with majority short-form answers of "unanswerable" and unnsuitable" unanswerable and otherwise answerable.
5

---

have both Answer and Answer Failure sentences in single answer. These answers typically offer a tentative guess (egg, "The color suggests if might be a jar of pasta sauce.") while also expressing the challenges in providing definitive answer (e.g., "However, the text is not completely legible to identify the exact product. ") When GPT-4V abstains from providing an answer to the visual question, the answer often contains an Explanation for why it abstained (e.g,, describing the low image quality) or Suggestion for taking better images.
Long-form answers provide informa- tion beyond image content. T shows the distribution of information source types. Answers from both ex- pert describers and all VLMs have a high percentage of information de- rived from the Image Content. Addition- ally, expert and GPT-4V answers often describe the Image Quality. Compared to other answers, GPT-4V answers pro- vide more External Information. For ex- ample, when asked to identify the type of frozen food in an image, answers from these models further provide non- visual information, such as the origin of the food and popular recipe used.
Image Content unans.
Image Quality Extermal Info.
Answers (%)
wnans
Expert GPT-VV
24
12
25
86
82
56
67
45
55
Gemini
90
84
5
16
24
LLaVA
33
QWEN
86
20
BLIP-2
84
60
17
InstructBLIP
84
68
5
12
35
Table 3: Distribution of answers with each infor- mation sources to answerable (ans.) and unanswer- able (unans.) questions. Among 600 questions, 230 were marked unanswerable in VizWiz.
Automatic Evaluation
To analyze the performance of VLMs in LFVQA, we conduct an automatic evaluation of long-form answers with reference-based metrics. To adapt the traditional reference-based metrics to long-form answers, we consider both short crowd answers (Gurari et al., 2018 and long expert answers (our dataset) as ground truth references and explore how our functional role classifier can be used for sentence-level evaluation. To our knowledge, we are the first to consider long-form reference answers when evaluating VQA.
Data We selected 360 examples in our dataset written by experts, GPT-4V and Gemini (demonstrated state-of-the-ar for image understanding tasks Yue et al. 2024 Team et al. 2023) and used in accessibility apps (BeMyEyes, 2020 TalkBack] (024 given the cost evaluation. The samples were balanced across question categories (Identification, Description, Reading, and Others).
Method We evaluate long-form answers using four reference-based VQA evaluation
metrics: ROUGE Lin 2004), METEOR (Elliott Keller 2013), BERTScore (Zhagg et al., 2019), and LAVE, an LLM-based metric that uses GPT-4 Mañas et al., 2023 to compare a candidate answer against a reference answer (details in (A.3.1). Comparing long-form answers to short-form reference answers penalize long-torm answers for including additional information (e.g,, explanation, suggestion) (Krishna et al. 2021 Mañas et al. 2023). To better evaluate LFVQA, consider two approaches. First, explore the potential tial of long-form references in VQA evaluation by leveraging the expert long-form answers in our VizWiz-LF as ground truths (I,) for evaluating model long answers (l.). However, many existing VQA datasets have only short ground truth answers (Antol et al.) 2015) Gurari al. 2018), and collecting long-form ground truths on a large scale costly. Thus, we also explore the use of functional roles in VQA evaluation with short-form references (s,) as ground truths for evaluating extracted Answer role sentences from long-form answers (1²).
Results Table Jummarrizs the evaluation results with reference-based metrics. In con- ventional reference-based metrics (ROUGE, METEOR, BERTScore), Gemini outperformed GPT-4V and experts. This was often due to the additional information in the long-fon answers of GPT-4V and experts being penalized by reference-based metrics. In LLM-bbsed
6

---

Answer Source
Lex-Overlap (Unigram)
ROUGE
METEOR
BERTScore
GPT-4
s,t1, 5 1,+k 1,11 s,tl, s,tl 1,+1, 1,+1 s,t1, 5,+] 1,tl, 1,+] s,t1, 6,t, 1,41, 1
GPT-4V
0.35
0.04
0.19
0.22
0.32
0.08
0.17
0.29
0.31
0.8
0.83
0.87
0,88
0.76
0,76
0.76
0.73
Gemini
0.22
0.19
0.22
0.22
0.26
0.18
0.20
0.16
0.21
0.84
0.84
0.87
0.87
0.63
0.64
0.45
0.51
Expert
0.36
0.06
0.2
0.09
0.18
0.81
0.84
0.71
0.7
Table 4: Evaluation results with reference-based metrics (reference+candidddl)) For reference answers, we use VizWiz crowd's majority-voted answers (s,), experts' long-form answers (l,), and extracted answer sentences of experts' long-form answers (1²). For candidate answers, we use original long-form answers generated by models and experts (le) and extracted answer sentences (l)).
evaluation (LAVE), the trend reversed, and GPT-4V and experts scored higher than Gemini. Among all metrics, only the LLM-based metric demonstrated a moderate correlation to human ratings (A..3.1), demonstrating the potential of LLMs in LFVQA evaluations. For all metrics, using long-form form answers reference answers increased the score compared to using short reference answers (s,). GPT-4V answers that showed low scores with 6, exhib- ited more substantial improvement in score for ROUGE and METEOR with ong-form references (l,) iFigure/Biin Appendix). While expert long-form answers are difficult to collect at scale, our evaluation highlights the need for long-form references to effectively evaluate LFVQA. Also, future work can consider more diverse functional roles than answer roles to enable more fine-grained evaluation of VQA. For instance, sentences with confirmation of photo roles can be evaluated using image groundings, and external information sentences can be assessed with fact-checking approaches.
6 Human Evaluation
While prior research with BLV people has shown that they want detailed image descrip- tions [MacLeod et al., 2017: Salisbury et al.] 2017), no work has explored BLV people's preferences for answer lengths in VQA. Thus, first conduct a preliminary survey with 8 BLV people who compare short short answers from crowd workers (from VizWiz) vs long answers from expert describers (from VizWiz-LF). Participants preferred long-form answers 75% more often than short-form answers, and this preference was significant (p < .0001, details in $A.3.2). Participants ranked shorter answers higher than long-fon form answers when the question asked about specific attribute of an object (e.g., "What is the color of this t-shirt?").
To understand preferences for LFVQA, we conduct an evaluation study with 20 BLV and 20 sighted people While most prior research in VQA evaluates answers with sighted people (Dua et al. 2021 Gardner et al. 2020), long-form answers should be evaluated beyond factual accuracy to consider their usefulness to end users, specifically BLV people. We conduct evaluations under all conditions of (BLV, sighted > image (description), without image (description)) to account for both scenarios where users have or do not have the context regarding the image settings. For BLV participants, we share brief image description to give them context for evaluating the answer. We obtain the image description from the original VizWiz dataset Gurari et al. 2018) authored by crowd workers
Method We conducted human evaluation on the same data used in Section 5 The
evaluation study involved (1) a preference ranking task and (2) fine-grained answer rating task followed by open-ended interview to understand their rankings and ratings. In the first task, evaluators were provided 12 visual questions, from each question category (ldentification, Description, Reading, and Others). For each visual question, participants read and ranked long-form answers from 3 sources (GPTT4V, Gemini, and expert describers) for a total of 36 long-form answers. In the second task, evaluators provided set of ratings for
7 Approved by our Institutional Review Board (IRB). Demographics of BLV participants in A.3.3 We randomly sample one from the five default captions after manually removing spam captions.
7

---

GFT-4V
Gemiri
Expert
Panking
Relevenccl
Hapfuiess
Flusncy
Pleusiiiity
Comectness
0.75
0.8
0.85
0.6
0.5
B+1
8+
Figure 2: Average rankings and ratings of long-form answers (GPT-4V, Gemini, Expert)
across four groups: BLV (B), BLV with image caption (B+I), Sighted without image (S), Sighted with image (S+I). We measured significance using the Freidman test followed by the pair-wise Wilcoxon test. < 0.05 marked with < 0.01 marked with , < 0.001 marked with *** after Bonferroni correction.
36 long-form answers? Participants provided one of the three labels (1 not, 2 partially, and 3 - very) for each criterion:
Relevance measures how relevant the answer is the question (Levinboim et al. 2019 is important to understand how people rate relevance in oong-form answers as they often contain extra information in addition to the core answer.
Helpfulness measures how helpful the answer is for people who cannot see the image.
Plausibility measures how likely the answer is to be correct (Gardner et al. 2020). Fluency measures how clearly the response conveys information, often related to the answer's 's grammar or consistency (Levinboim et al. 2019).
For the group of sighted evaluators who were provided with images during the evaluation, they are asked to evalaute with Correctness (i.e., how correct the provided information is given the image) instead of Plausibility.
Results Figure 2 displays the normalized rankings and ratings of three long-form answer types across all Tour participant groups. All evaluator groups ranked responses from experts and GPT-4V higher than those from Gemini, often attributing their choice the rich information in expert and GPT-4V responses. For relevance specifically, BLV evaluators preferred shorter answers from Gemini as they found GPT-4V answers often share extra information that is not useful (egg, describing shirt's surroundings when asked for the shirt's color). BLV evaluators also noted that GPT-4V's frequent image quality descriptions (e.g., "You uploaded blurry photo of...") felt repetitive and unnecessary, unless used to explain an answer failure, as they often took low-quality photos.
Sighted evaluators assessing the answer with the image (S+I in Figure 2 rated Gemini answers significantly lower than GPT-4V and expert answers for Helpfulness and Correctness and explained that low ratings were due to incorrect information provided in the answer. In contrast, BLV groups rated Plausibility high (0.84 out of 1.0, SD=0.28) for all types of long-form answers. Our follow-up interviews with BLV evaluators revealed that their trust in long-form answers is influenced by the level of detail and expressions of uncertainty in the answer. For GPT-4V and expert answers, extensive details made BLV evaluators perceive the answers as more correct. However, compared to Gemini, GPT-4V and experts frequently expressed uncertainty due low image quality, thus BLV evaluators trusted Gemini answers despite their brevity.
7 Experiments on VQA Model Abstention
Our human evaluation reveals that the perceived correctness of long-form answers for BLV raters does not align with the answer correctness. Assessing answer correctness is challeng- ing for BLV users, as they cannot visually verify the answer against the image 10 Prior work
We sample one answer from each visual question to avoid bias (eg., giving high ratings if two answers provide the same information.)
As workaround, BLV people check for common object detection errors Huh et al. 2023b or conflicting information among multiple model outputs Huh et al. (2023a).
8

---

Model
abs-acc vanilla prompt vanilla prompt
abs-prec
abs-recall vanilla prompt vanilla prompt vanilla prompt vanilla prompt
abs-f1
abs-%
ans-acc
Random
0.50
0.34
0.34
0.34
38
Majority
0.62
0.0
0.0
Expert
0.73
0.68
0.57
0.62
32
0.60
GPT-VV
0.77
0.73
0.66
0.60
0.82
0.88
0.73
0.71
47
56
0.68
0.68
Gemini
0.66
0.71
0.78
0.65
0.14
0.51
0.24
0.57
7
30
0.46
0.55
LLaVA
0.80
0.14
0.74
0.24
7
0.43
0.61
OWEN
0.71
0.76
0.70
0.64
0.42
0.84
0.52
0.72
23
50
0.52
0.64
BLIP-2
0.63
0.70
0.54
0.39
0.16
0.73
0.24
0.65
11
47
0.27
0.54
InstBLIP
0.63
0.65
0.63
0.53
0.10
0.72
0.17
0.61
6
0.25
0.41
Table 5: Performance of vanilla model and abstention instruction on six VLMs. Full results with other abstention strategies in Table 26 in the Appendix.
shows that BLV people often fill details to resolve incongruent model descriptions rather than suspecting errors (MacLeod al. 2017). Thus, false information hallucinated by VLMs can mislead users. Unanswerable questions are prevalent in questions asked by BLV users due to poor image conditions (egg., blurry images, absence of target object), yet VLMs often hallucinate answers to such questions $5 We quantify the level of hallucinations of VLMs with such images and evaluate their ability to correctly abstain from providing an answer when the question is unanswerable (Xiong et al., 2023 Feng et al. 2024 We investigate suite of prompting methods and evaluate their etfectiveness our dataset.
7.1 Experiment Setting
Data and Answerability Labels We consider the 600 VizWiz questions in our dataset ($3 and use short-form answers from VizWiz as gold answerability labels, for a total of 38% of questions labeled unanswerable. To determine whether löng-form answer abstains from providing an actual answer, we classify its sentence-level functional roles ($4) and check whether at least one sentence labeled as Auswer Failure, which expresses model's inability to answer the question (examples in Table
Evaluation Metrics We report four sets metrics: (1) Abstain Accuracy (abs-accc, which measures whether the abstain decisions correct against the gold answerability label; (2) Abstain Precision (abs-prec), Recall (abs-recall) and F1 (abs-f1), treating abstaining to answer the positive label; (3) Percentage of questions that the model abstains from answering (abs-%) and (4) Answer Accuracy (ans-acc), which compares the answer against the ground truth short answers using LLM--ased metric (Mañas al. 2023), , as it correlates well with human judgment ($6). We report the performance of three baselines: a random where an answerability label is randomly assigned according the distribution and majority baseline where we always predict Answerable. We also report expert performance by evaluating the long-form answers written by experts.
Abstention Strategies We investigate VQA models' abstention capabilities using three prompting techniques. The first approach Abstention Instruction includes a detailed instruc- tion in the prompt and instructs the model determine whether question is unanswerable (eg,, overly blurry images), and if so, abstain from answering. We also used two additional prompting strategies from prior work Cui et al. (2023); Wang et al. (0222) but they do not yield performance gains (details in 6A.42
7.2 Results
presentss the performance of the default (oanilla) and abstention instruction (prompt) approaches. The abstention instruction prompt increases abstention frequency across all models (abs-%), while also improving the ability of all models to correctly abstain from
We do not evaluate the factuality of all the information presented in the long-form answer, which might also contain hallucinations. Future work can leverage our role analysis to identify appropriate sources for the factuality evaluation of the entire answer.

---

Confimation, Arstmait Ansover Falums, Auxilirry Auxilary Fallun, Explination, Suggestion, Micc
GPT-4V
Promet
7%
17%
21N
22%
2856
[4%
Gemini
80%
1456
Prompt
3995
18%
71%
LLaVA
Vanila
18%6
44%
18
1996
10%
59%
Prompt
InstructBLIP 20%
Vanila
10%
10%
29%
OWEN
18%
4136
10%
12%
1395
Prompt
44%
65%
Prompt
4056
47%
Figure 3: Functional role distribution of models' long-form answers before and after applying abstention instruction.
answering unanswerable questions (abs-recall), especially for models with low abstention frequency in the vanilla setting (Gemini, LLaVA, BLIP-2, InstructBLIP). While precision (abs-prec) declines with abstention instructions, the F1 (abs-fl) and answer accuracy (ans-acc) increase for all models except GPT-4V. For low-quality images, opting to abstain rather than providing low-confidence answers may benefit users in the context of accessibility where wrong but confident answers can cause harm.
Prompting VQA models abstain also changes the functional role distributions in their answers compared to the vanilla setting 3 Answers from GPT-4V, Gemini, LLaVA, and InstructBLIP provide more auxiliary information and explanations when instructed to abstain, whereas answers from QWEN and BLIP-2 exhibit patterns of short-form VQA by generating primarily answer or answer failure roles.
Additionally, note discrepancy in abstention between expert answers and crowd- sourced short answers. Expert and crowd answers agreed on answerability in 52% of the 600 questions. Among 230 questions marked unanswerable by the crowd workers, 99 were answered by expert describers. This often occurred when few crowd workers provided answer but the question was labeled "un "unanswerable" using majority voting A.4.3 Among 370 questions marked answerable by VizWiz, 61 were not answered by experts. This occurred when the experts were not familiar with specific objects in Identification questions.
Conclusion
In this work, we presented VizWiz-LF, a dataset of long-form answers to visual questions posed by BLV users. Our work is the first to explore the content and structure of LFVQA from both humans and models, and to compare long-form answers to short-form answers. Our functional role and information source analysis demonstrates the rich information provided in LFVQA by both human experts and models. We also revealed that while BLV evaluators preferred long-form answers to short-form answers, there remain opportunities to improve the relevance and correctness of LFVQA. We found that abstention instructions helped VQA models to better abstain from answering unanswerable questions. Future work may use functional roles to further evaluate and improve the per-role performance of LFVQA and tailor descriptions to context and preference. In this era of VQA, we must go beyond domain-agnostic factual accuracy to consider how to efficiently deliver relevant information. We hope that our work encourages future development and evaluation of VLMs with people who may benefit the most from using them.
Acknowledgments
The study is partially funded by NSF grant I[S-2312948.
10

---

Ethics Statement
Hallucinations in LLMs and VLMs: LFVQA answers in our dataset are collected from expert describers and VLMs. As LLMs and VLMs might generate factual errors and hallucinations, we have examined the data to make sure they do not contain harmful con- tent before the human evaluation study. Our work also uses LLMs to classify functional roles and evaluate VQA. While we have shown that these approaches better align with humans, LLMs are not free from potential hallucinations and can lead to incorrect results.
Human Evaluation Study: Our human evaluation study was approved by our institu- tion's 's International Review Board (IRB) as exempt research. In our user study, we ensured that all participants were compensated fairly for their time and contributions. The pay- ment was determined based the average market rate for such studies, reflecting the complexity and duration of the tasks involved.
Application to other domains: The experience of visual disabilities is individual and can affect people's preferences in image description (Stangl et al. 2021 Huh et al. (2022) With our proposed functional roles of LFVQA, future systems can support VQA of adaptive length and content based on user contexts (Kreiss et al. 2022 Peng et al.
2022: Gubbi Mohanbabu & Pavel. 2024 Srivatsan et al.1(202) and preterences (Jones
et al. 2024 Van Daele et al.] 2024 While we analyzed long-form answers to visual questions asked by BLV people, LFVQA also occurs in other use cases (e.g, asking how to solve math problem with a screenshot of the problem) (Chen et al. 2023b). We expect that our functional roles will cover common roles and information types in other domains (e.g., answer, auxiliary information, suggestion). Other domains are likely to encounter different proportions of our functional roles and information types (e.g,, fewer "image quality" issues and related suggestions) and other domains may have additional functional roles and information types we have not yet observed. Future work will explore extending our functional roles and information types to other domains.
Risk of model evaluation for accessibility use cases: Unlike applications of VQA where people can visually check the answers to their questions, BLV people consume model an- swers without further verification. Our human evaluation study reveals that BLV people have higher trust in model-generaae scores from evaluations may further encourage trust in models leading to potential risks of unnoticed model errors. We hope robust evaluation approaches and an understanding of long-form answers can provide more realistic picture of model performance.
answers than sighted people. Additionally, high
Reproducibility Statement We will release our codes, prompt, and data collected publicly.
Our VizWiz-LF dataset contains long-form answers from expert describers who are skilled in describing images for BLV people. Because what they decide to include in the long-form answers can be subjective, the diversity and representativeness of the human describers can influence the results. Also, compared to hiring crowd workers, it is more challenging to scale up the collection of expert long-form answers due to the difficulty of finding skilled describers and the cost. We collected one expert long-form answer per visual question to prioritize a larger dataset over multiple long-form answers for a smaller dataset which aligns with prior work (Gurari et al. 2019 Zhou et al.) 2017) that collects one annotation per example from expert annotators. Future work can augment our VizWiz-LF dataset with more ground truth references and perform human correlation studies (Gupta et al. 2019).
Collecting Long-form Expert Descriptions:
Use of LLM as an evaluator: We use GPT-4 in classifying functional roles and evaluating answer accuracies. While prior work has demonstrated its improved correlation with human judgment (Mañas et al. 2023 Chan et al. 2023), we acknowledge that results are nondeterministic. Additionally, paper are not open-source, the models are subject to arbitrary change, replacement, or removal. We hope that efforts to increase open-access language models will alleviate these concerns in the future. Finally, our work relies on language models that contain many billions of parameters and involve expensive API calls. We hope that future advances in LLM inference and architecture will enable low-cost use of these models with good performance.
some of the LLMs and VLMs explored in our
11

---

References
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, lige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. Xiv preprint ar Xi::23303.08874 2023.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716-2 1-23776--2773,, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, with
Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language versatile abilities. arXiv preprint arXiv:2308. 12966, 2023.
http://upporr..bbeyysscsooo BeMyEyes. Tips for visual interpretation,, 2020. URL Interpprttttntr
Jeffrey Bigham, Chandrika Jayant, Hanjie Ji, Greg Little, Andrew Miller, Robert Miller, Robin Miller, Aubrey Tatarowicz, Brandyn White, Samual White, et al. Vizwiz: nearly real-time answers to visual questions. In Proceedings of the 23nd annual ACM symposium on User interface software and technology, pp. 333-3432 2010.
Erin Brady, Meredith Ringel Morris, Yu Zhong, Samuel White, and Jeffrey P Bigham. Visual challenges in the everyday lives of blind people. In Proceedings of the SIGCHI conference on human facters in computing systems, pp. 2117-2126, 2013.
David Chan, Suzanne Petryk, Joseph E Gonzalez, Trevor Darrell, and John Canny. Clair: Evaluating image captions with large language models. arXiv preprint ar xrXie:2310.12971, 2023.
Chongyan Chen, Samreen Anjum, and Danna Gurari. Vqa therapy: Exploring answer differences by visually grounding answers. In Proceedings of the TEEE/CVF International Conference on Computer Vision, pp. 15315-15325, 2023a.
Chongyan Chen, Mengchen Liu, Noel Codella, Yunsheng Li, Lu Yuan, and Danna Gurari. Fully authentic visual question answering dataset from online communities. arv preprint ar arXit:2311.15562 2023b.
Xi Chen, Xiao Wang, Lucas Beyer, Alexander Kolesnikov, Jialin Wu, Paul Voigtlaender, Basil Mustafa, Sebastian Goodman, Ibrahim Alabdulmohsin, Piotr Padlewski, et al. Pali-3 vision language models: Smaller, faster, stronger. Xrviv preprint ar xxXi:2231000919, 2023c.
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. Universal self-consistency for large language model generation. arXio preprint ar xrx::23311113111, 2023d.
Eunsol Jennimaria Palomaki, Matthew Lamm, Tom Kwiatkowski, Dipanjan Das, and Michael Collins. Decontextualliztton: Making sentences atand-alone. Transactions of the Association for Computational Linguistics, 9:447-461, 2021.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction- finetuned language models. arviv preprint ar 2rXiv:2210.11416, 2022.
Chenhang Cui, Yiyang Zhou, Xinyu Yang, Shirley Wu, Linjun Zhang, James Zou, and Huaxiu Yao. Holistic analysis of hallucination in gpt-4v (ision): Bias and interference challenges. arXiv preprint ar arXio:2311.0328) 2023.
12

---

Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arviv preprint ar Xrix:23033003776 2023.
Radhika Dua, Sai Srinivas Kancheti, and Vineeth N Balasubramanian. Beyond vqa: Generat- ing multi-word answers and rationales to visual questions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1623-1632, 2021.
Desmond Elliott and Frank Keller. Image description using visual dependency representa- tions. In Proceedings of the 2013 conference on empirical methods in natural language processing, Pp. 1292-1302, 2013.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5: Long form question answering. ar arviv preprint ar Xitx19907.09190 2019.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the ÍEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19358-19369, 2023.
Shangbin Feng, Weijia Shi, Yike Wang, Wenxuan Ding, Vidhisha Balachandran, and Yulia Tsvetkov. Don't hallucinate, abstain: Identifying Iim knowledge gaps via multi-llm collaboration. arviv preprint 2rXio02402.00367, 2024.
Joseph Fleiss and Jacob Cohen. The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability. Educational and Psychological Mensurement, 33(3):613-619, 1973, doi: 10.1177 101177/000316447333330399 URL htttts::/doi.rrg//0000 1177/ 001316447308500309
Samir Yitzhak Gadre, Gabriel Ilharco, Alex Fang, Jonathan Hayase, Georgios Smyrnis, Thao Nguyen, Ryan Marten, Mitchell Wortsman, Dhruba Ghosh, Jieyu Zhang, et al. Datacomp: In search of the next generation of multimodal datasets. Advances in Neural Information Processing Systems, 36, 2024.
Rachel Gardner, Maya Varma, Clare Zhu, and Ranjay Krishna. Determining question- answer plausibility in crowdsourced datasets using multi-task learning. arXiv preprint arXiv:2011.04883, 2020.
Cole Gleason, Amy Pavel, Emma McCamey, Christina Low, Patrick Carrington, Kris M Kitani, and Jeffrey P Bigham. Twitter ally: A browser extension to make twitter images accessible. In Proceedings of the 2020 chi conference human factors in computing systems, PP. 1-12,2001 2020.
Yash Goyal, Tejas Khot, Douglas Summers-Star Stay, Dhruv Batra, and Devi Parikh. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and paftern recognition, pp. 6904-6913, 2017.
Ananya Gubbi Mohanbabu and Amy Pavel. Context-aware image descriptions for web ac- cessibility. In Proceedings of the 26th International ACM SIGACCESS Conference on Computers and Accessibility, 2024.
Prakhar Gupta, Shikib Mehri, Tiancheng Zhao, Amy Pavel, Maxine Eskenazi, and Jef- frey P Bigham. Investigating evaluation of open-domain dialogue systems with human generated multiple references. Xiv preprint 1xXit:1907.10568. 2019.
13

---

Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEÉE conference on computer vision and pattern recognition, pp. 1600--617, 2018
Danna Gurari, Qing Li, Chi Lin, Yinan Zhao, Anhong Guo, Abigale Stangl, and Jeffrey P Bigham. Vizwiz-priv: A dataset for recognizing the presence and purpose of private visual information in images taken by blind people. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2019.
Elmar Hashimov. Qualitative data analysis: A methods sourcebook and the coding manual for qualitative researchers: Matthew b. miles, a. michael huberman, and johnny saldaña. thousand oaks, ca: Sage, 2014. 381 pp. johnny saldaña. thousand oaks, ca: Sage, 2013. 303 Pp., 2015.
Yukun Huang, Yixin Liu, Raghuveer Thirukovalluru, Arman Cohan, and Bhuwan Dhin- gra. Calibrating long-form generations from large language models. ar preprint arXiv:442.00654, 2024.
Mina Huh, Yunjung Lee, Dasom Choi, Haesoo Kim, Uran Oh, and Juho Kim. Cocomix: Utilizing comments to improve non-visual webtoon accessibility, In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems, pp. 1-18, 2022.
Mina Huh, Yi-Hao Peng, and Amy Pavel. Genassist: Making image generation accessible. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pp. 1-17, 2023a.
Mina Huh, Saelyne Yang, Yi-Hao Peng, Xiang' Anthony' Chen, Young-Ho Kim, and Amy Pavel. Avscript: Accessible video editing with audio-visual scripts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1-17, 2023b.
Liqiang Jing, Ruosen Li, Yunmo Chen, Mengzhao Jia, and Xinya Du. Faithscore: Evaluating hallucinations in large vision-language models. arxiv preprint xrXi::2311.01477, 2023.
Shuli Jones, Isabella Pedraza Pineros, Daniel Hajas, Jonathan Zong, and Arvind Satya- narayan. "customization is key": Reconfigurable textual tokens for accessible data visualizations. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-14, 2024.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac Hatfield-Dodds, Nova DasSarma, Eli Tran-Johnson, al. Language models (mostly) know what they know. arXiv preprint ar 0tto:22077052211 2022.
Joonghoon Kim, Saeran Park, Kiyoon Jeong, Sangmin Lee, Seung Hun Han, Jiyoon Lee, and Pilsung Kang. Which is better? exploring prompting strategy for Ilm-based metrics. arxiv preprint arXio:2311.03755 2023.
Elisa Kreiss, Cynthia Bennett, Shayan Hooshmand, Eric Zelikman, Meredith Ringel Morris, and Christopher Potts. Context matters for image descriptions for accessibility: Challenges for referenceless evaluation metrics. Xiv preprint ar XrXi2:220 106664660610666,,, 2022.
Kalpesh Krishna, Aurko Roy, and Mohit lyyer. Hurdles to progress in long-form question answering. arXiv preprint ar Xiv:2103.06332, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Con- necting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Tomer Levinboim, Ashish V Thapliyal, Piyush Sharma, and Radu Soricut. Quality es- timation for image captions based on large-scale human evaluations. arXiv preprint XxXi::1909.00336, 2019.
14

---

Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image image encoders and large language models. International conference machine learning, pp. 19730-19742. PMLR, 2023a.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXio preprint ar Xiv:2305510355, 2023b.
Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summariza- tion branches out, pp. 74-81, 2004.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems,, 36, 2024.
Mengchen Liu and Chongyan Chen. An evaluation of gpt-4v and gemini in online vqa. Xrv preprint ar axXio:2312.10637, 2023.
Haley MacLeod, Cynthia L Bennett, Meredith Ringel Morris, and Edward Cutrell. Under- standing blind people's experiences with computer-generater captions of social media images. In proceedings of the 2017 CHI conference human factors in computing systems, pp- 5988-5999, 2017.
Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. Improving automatic vqa evaluation using large language models. arXiv preprint ar arXio:2310002256,2000 2023.
Jacob Menick, Maja Trebacz, Vladimir Mikulik, John Aslanides, Francis Song, Martin Chadwick, Mia Glaese, Susannah Young, Lucy Campbell--Gllingaann Geoffrey Irving, et al. Teaching language models to support answers with verified quotes. ar ar preprint arXiv:2203.11147, 2022.
Nandita Naik, Christopher Potts, and Elisa Kreiss. Context-vga: Towards context-aware and purposeful visual question answering. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2821-2825, 2023.
Rejichiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang, Christina Kim, Christo- pher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, et al. Webgpt: Browser- assisted question-answering with human feedback. ar ar preprint ar Xf1::22122.09332, 2021.
Theo Olausson, Jeevana Priya Inala, Chenglong Wang, Jianfeng Gao, and Armando Solar- Lezama. Demystifying gpt self-repair for code generation. arXiv preprint drXiv:2306.09886, 2023.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei jing Zhu. Bleu: method for automatic evaluation of machine translation. pp. 311-318, 2002.
Yi-Hao Peng, Jason Wu, Jeffrey Bigham, and Amy Pavel. Diffscriber: Describing visual de- sign changes to support mixed-ability collaborative presentation authoring. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology, pp. 1-13, 2022.
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal Digital Libraries, (3(3):289-301 2022.
Elliot Salisbury, Ece Kamar, and Meredith Morris. Toward scalable social alt text: Conversa- tional crowdsourcing as tool for refining vision-to-language technology for the blind. In Proceedings of the AAAI Conference on Human Computation and Crowdsourcing, volume 5, pp. 147-156, 2017.
Nikita Srivatsan, Sofía Samaniego, Omar Florez, and Taylor Berg- Kirkpatrick. Alt-text with context: Improving accessibility for images on twitter. arviv preprint arXiv:230..1479, 2023.
15

---

Abigale Stangl, Nitin Verma, Kenneth R Fleischmann, Meredith Ringel Morris, and Danna Gurari. Going beyond one-size-fits-al image descriptions to satisfy the information wants of people who are blind or have low vision. In Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility, pp- 1-15, 2021.
TalkBack. Experience google ai in even more ways android. httpe://blog. attp////drrodddeo httptttss//adooidoooooggoo// -iie--a---- audd-oddidd-udaeee io- 2004/
/twpe//ee////----eeeeeeaaaaaaaee #ttp://iirclcee -o--ttrrcsseeeeee 2024.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean Jean-Battiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models. arv preprint arXi::2312.11805, 2023.
Upwork Upwork
com/ 2024. [Accessed on March 27, 2024].
Tess Van Daele, Akhil Iyer, Yuning Zhang, Jalyn C Derry, Mina Huh, and Amy Pavel. Making short-form videos accessible with hierarchical video summaries. In Proceedings of the CHI Conference on Human Factors in Computing Systems, pp. 1-17, 2024.
Junyang Wang, Yiyang Zhou, Guohai Xu, Pengcheng Shi, Chenlin Zhao, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Jihua Zhu, et al. Evaluation and analysis of hallucination in large vision-language models. ar Xiv preprint ar Xiv::233 15126, 2023.
Peng Wang, Qi Wu, Chunhua Shen, Anthony Dick, and Anton Van Den Hengel. Fvqa: Fact-based visual question answering. IEEE transactions on paltern analysis and machine intelligence, 40(10):2413-2477, 2017.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. Xre preprint ar ar.ivv2203.11777 2022.
Sean Welleck, Ximing Lu, Peter West, Faeze Brahman, Tianxiao Shen, Daniel Khashabi, and Yejin Choi. Generating sequences by learning to self-correct. arXiv preprint ar arXiv:2111.00053 2022.
Spencer Whitehead, Suzanne Petryk, Vedaad Shakib, Joseph Gonzalez, Trevor Darrell, Anna Rohrbach, and Marcus Rohrbach. Reliable visual question answering: Abstain rather than answer incorrectly. In European Conference Computer Vision, pp. 148-166. Springer, 2022.
Qi Wu, Damien Teney, Peng Wang, Chunhua Shen, Anthony Dick, and Anton Van Den Hen- gel. Visual question answering: A survey of methods and datasets. Computer Vision and Image Understanding, 163:21-44 40, 2017.
Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. Can Ilms express their uncertainty? an empirical evaluation of confidence elicitation in 1lms. Xrxiv preprint ar XxXi:2306613066 2023.
Fangyuan Xu, Junyi Jessy Li, and Eunsol Choi. How do we answer complex questions: Discourse structure of long-form answers. In Smaranda Muresan, Preslav Nakov, and Aline Villavicencio (eds.), Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 3556-3572, Dublin, Ireland, May 2022. Association for Computational Linguistics. doi: 10.18653 (0.1666//11/202..a--0mgg... URL htatts::/aacaattolllooo org. wetgg/2222.c00 a00.2..cll1100 acl- long..1 249
Fangyuan Xu, Yixiao Song, Mohit and Eunsol Choi. A critical evaluation of evaluations for long-form question answering. ArXiv, abs/ abs/2235.18201, 2023. URL hhtpps//aap
semanticscholar wwwooog//oooo I.torpusID:25996600
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 9556-5556999 9567,9956,, 2024.
16

---

Xiaoyu Zeng, Yanan Wang, Tai-Yin Chiu, Nilavra Bhattacharya, and Danna Gurari. Vision skills needed to answer visual questions. Proceedings of the ACM on Human-Computer Interaction, 4CCSCW2):1-31, 2020.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision trans- formers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, Pp. 12104-12113, 2022.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. ar preprint ar Xix::9044006675 2019.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. In Proczedings of the IEEE conference on computer vision and pattern recognition, pp. 633 641, 2017.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. Context-faithful prompting for large language models. ar Xiv preprint WrXiv:2303.11311 2023.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4995-5004, 2016.
17

---

A Appendix
A.1 Dataset Collection
A.1.1 Sampling VizWiz Questions
We first randomly selected 2,500 unique image-question pairs from VizWiz's training and validation sets initial pool to further sample diverse types of questions. We removed image-question pairs with incomplete questions (e.g., Is this?? and questions about the VizWiz service (e.g., Is there someone answerring questions 24 hours day?), resulting in 2447 total pairs.
To select a diverse range of image-question pairs, we categorized the 2,500 questions into four question types (Identification, Description, Reading, and Others) from an existing VizWiz question taxonomy (Brady et al. 2013), then randomly sampled 150 image-question pairs from each of the four question types to obtain a total of 600 image-question pairs. In Identification questions, users ask to identify an object. In Description questions, users ask about the visual attributes (e.g,, color, count, style) of an object or setting, In Reading questions, users ask for the text to be read. Others category includes examples with multiple questions from different categories, questions that involve further reasoning or knowledge outside the image. We did not remove low-quality images (e.g,, blurry, dark) or image- question pairs labeled "unanswerable" by crowd workers we aimed to capture expert and vision language model responses to these scenarios.
A.1.2 Expert Answer Collection
To collect long-form answers from expert describers, we hired 20 people experienced in describing images for BLV people using Upwork (2024). While VizWiz crowd workers aimed to answer the questions in nearly real-time (36 sec, SD=30), we encouraged expert describers to write ideal and detailed responses (234 sec, SD=1.17). We paid expert described by their hourly rate (28 USD, SD=7.84) High-quality annotations are exemplified in Figure
A.1.3 Model Answer Collection
Table 6 provides architecture configurations of the four benchmarked Vision-Language Models (VLMs), including specifications of their language encoders, vision encoders, and adapters. GPT-4V and Gemini's architectures are undisclosed. We use the default VLM temperatures 1optt--110-vviioon-rrrviii 1.0, semini-1.0-pro-viiion 0.4, Ilava-v1.5-13 blip2-flan-t5-xl:: 1, instructblip-flan----xl 1) and top-p (qwen-vl-chat: 0.3, instructblip- flan-t5-xxl: 1).
Model
Language Encoder
Vision Encoder
Adapter FC Layer Q-Former
LLaVA-15 BLIP-2
Vicuna-13B
VIT-L/14
Zhai et al. 2022
FlanT3-XXL((LB) Chung at. 2022
VII-g/ 14
Fang et al 2023
QWEN-VL-Chat
Qwen-7B
ViT-bigG
Cadre et al.
2024
Cross-Atter Aotss-Attention Module
Position-aware
InstructBLIP
FlanT3-XXL(11B)
Chung et al, 2022
ViT-g/14 /1
Fang et al,
2025
Q-Former
Table Details about the four benchmarked VLMs' model architecture.
18

---

A.1.4 Examples of Long-form Answers in VizWiz-LF dataset.
Question What his?
GFT-VV
The the image is Rilstex Up hater. Bliier a brand that produces products, and this particular inte seems to include on SPE, indicating that it offere son protection as seeil.
we
Blister ex
VizWiz original
Gemini
Mister
This is tube of Blisiex by baim.
Expert
ploto shous tube of dory rolored Other contents soen word: "medicated" und a SPT ladei but the wumber not clear.
Question
What's exptration dats an this non dniry cheese?
bus the image too blurry for accunately rond the exptraatin date on the non-duiry cheese package. If you cmn promide clearer imnge or the printed date, I'á be happy to help you deterriie the expiration ditte.
VieWiz original wammsuerabll
spans eyyan
Gemini
2023
dalya
Expert The expiration date en the cheese not vistille in the image.
Question
CPT-V
What is that onlor?
The imuge shows fabric with a light yollow colors, including pinl: purple, and oringe. The patterns consist of fioral and paislay-lle desigus. Plaase wote that the perception color vury depenilig fhe lighting conditions ami the settings of the device displaying the image.
bee color, adorunl with patterns in curiess
VixWis original
gellow
yellow
Expert
The color of the bedsheet soft orange. It also has different designs and flowers in pink, yeliow and red colors.
Question
GPT4V
What's the make and model of that oar and whhid's 's ficense plate mumber??
The car in the photo appeatr to be Cheerolet Corcefte, libely mosie' from the late '90s or sarly 2000, given design characeriitics. Likenee plates personal data specific intiiiiaai'' 's vchisie registration and shuuld not be onione for privary rossos.
plate, procide information.
VizWiz original
Gemini
The make and model of the car is Clicrraltt Causlier, The license plate 367WMD.
Wistex
Expert
model not viaribee licesse plate of white mombers on blue backgrowed. The livver plate lluury but appeors tu consist of four numbers.
Chryeler
Question What tiis?
GPT-4V
This appeors cloze-up, Nurry photograph of keyboard, with indinidual laycaps shunning characters letters. The photo is taken from such an mngic and with such lighting that there's a fright giare refiecting off lays, shhich mabes
challenging to details ciearly. keyboari looks like it might be used anth a desktop computer or isptap.
VieWiz original
keyhoard
Gemini
laptop keyhoard.
Expert
The imuge is blurry close up of knyhoard witt some reflected light in the misille of the imagee
Figure 4: Examples of Long-form Answers in VizWiz-LF dataset. Images, questions, and short-form answers from VizWiz-VQA dataset
19

---

A.2 Functional Roles and Information Types
A.2.1 Human Annotaions
To derive the taxonomy, three researchers used open coding 100 sample responses to obtain potential functional roles. Then, they met to merge together similar functional roles and create codebook with name, definition, and example for each functional role. The researchers iteratively coded samples and revised the codebook to achieve final codes, containing eight functional roles. During the coding process, additional need for annotating information types emerged. To better understand long-form answers that provide information beyond the image content, the researchers additionally generated the codebook for information source types. The full codebook and examples provided to annotators are shown in Table
Using the codebook with examples, we collected three-way annotations of 180 long-form answers from five researchers. They reached substantial agreement for functional roles (Fleiss Kappa 0.76) and a perfect agreement for information types (Fleiss Kappa 0.81). These high-quality annotations were used as few-shot examples for prompting classifier and to evaluate their performance.
Role
Definition
Example
Confirmation
Confirms what the user uploaded.
The image you uploaded appears to be carlon of chocolate soymilk.
Answer
Addresses the question with answer.
expires int September 2015.
Answer
States the inability to address the question.
The expiration date is not visible in this photo.
(Failure)
Auxiliary
Provides additional information not
The size of the milk container is 8 fluid ounces (240 mL).
directly related to the question.
Auxiliary
States the inability to provide information not directly related to the question.
I cannol provide nutritional information due to the blur.
(Failuree
Explanation
The sentence explains the reasoning for the information it provides.
Gioen the presence of soybeans in the image, it is likely to be soymilk.
Suggestion
Suggests retaking or improving the quality of photo get a better answer.
you can provide a clearer image, might be able to better assist.
Miscelleneous
Does not provide new information. Sentences that do not belong any of the categories above.
I'm happy to assist you, let me know you have further requests!
Table 7: Definitions of functional roles identified in VizWiz-LF dataset
Type
Definition
Example
Image
Provides visual information about the image content
You are holding a pink boftle.
Content
Image
Provides visual information about the image quality.
The image you' 've provided is quite blurry and the details are not clear
Quality
External
Provides general knowledge or facts that are not found in the image.
Expiration date is typically located near the top or side of the bottle
Information
Table 8: Definitions of information sources identified in VizWiz-LF dataset
A.2.2 Classifier
We construct a prompt with definitions and in-context examples for each of the function roles information types. Given question and a list of answer sentences, GPT-4 outputs a list of labels for each of the answer sentences. The prompt (which includes the task instruction and shw-shot examples) we use can be found in Table 12 and Table 13 We
20

---

evaluate GPT-4's performance against the majority label from the three-way annotations, using per-label Fl and a weighted average Fl over all the class.
To contextualize GPT-4'ss performance, provide two estimates for human performance collected in $A.2.1 an upperbound, which we compare each annotator's s annotation with the majority label. This inflates the performance one's annotation is correlated with the majority label; an lowerbound which we compare all pairs of annotation and average over them. We report two baselines: (1) Random: which randomly assigns role combination from all of the annotation and (2) Majority: which always labels the sentence as "Answer" (or "Image Content" for information type).
Results are in Table and and 10 We see that GPT-4 significantly outperforms both the baselines, with a moderate gap compared the human lowerbound. The model performs relatively poorly in identifying "Auxiliary (Failure)" and "Confirmation" for which human also exhibit lower agreements.
Model
Confirmation
Answer Ans. Failure Auxiliary Aux. Failure Explanation Suggestion Misc. Average
Majority
0.0
0.74
0.0
0.0
0.0
0.0
0.0
0.35
Random
0.13
0.57
0.04
0.29
0.0
0.01
0.13
0.0
0.37
GPT-4 (B-shot)
0.40
0.84
0.79
0.65
0.18
0.60
0.86
0.53
0.74
Human (lower)
0.51
0.43
0.80
0.79
0.79
Human (upper)
0.75
0.95
0.90
0.85
0.67
0.86
0,90
0.90
0.89
Table 9: Per-role F-1 for automatic functional role classification.
Model
Image Content
Image Quality
External Info.
Majority
0.91
0.0
0.0
Random
0.82
0.08
0.05
GPT-4 (8-shot)
0.95
0.74
0.77
GPT-4V
0.78
0.72
0.72
Human (lower)
0.97
0.92
0.88
Human (upper)
0.95
0.83
0.77
Table 10: Per-role F-1 for automatic information source classification.
21

---

You are given question to an image and an answer paragraph to the question. You job is assign each of the sentence in the answer paragraph into least one up three functional roles listed below. For each sentence, please assign the roles that are applicable.
# Role:
Confirmation of Photo
A Definition: The sentence confirms what the user uploaded. When user asks identification question (egg, what is this?), this sentence may also be annotated as an answer.
"This usually
the beginning of the sentence, provide the overview of the image. This sentence often looks similar typical image
captions.
# Example: "The image you've provided appears to be of a SodaStream Raspberry flavor syrup bottle."
# Role: Answer
# Definition: The sentence directly addresses the question. If the answer provided multiple sentences, they can all be labeled as "answer"
as in the example below. Incorrect answers are still labeled as answers. # Example: (question: what color is this?) "I would describe the shirt as a reddish doaliiss-bbrwwncom color.
# Role:
Answer Fallure
# Definition: The sentence states the inability to address the question, often accompanied by "Explamation of Reasoning" explaining the resson.
# Example: "I cannot provide information such as the cletails on this globe."
# Role: Auxilirr
# Definition: The sentence provides additional information not directly related to the question but observed in the photo, general knowledge or facts related the query.
# Example: (question: Read this label) "Pectin is a natural thickening agent that's extracted from fruils and used commonly in cooking to gel liquid."
#t Role: Auxiiarr Failure
# Detinition: The sentence states the inability provide auxiliary information directly related question.
& Example: (question: What the color of the pants?) "I However, cannot identify its brand name because the image is bhurry."
# Role: Explanation of Reasoning
# Definition: The sentence explains the reasoming for the information it gives by describing its thought process providing the information source. also explain the reasoning behind failure (eg.. g. low quality the photo). a Example: (question: what this?) "Given the color of the liquid visable through the bottle, it's likely to be type of mustard sauce, possibly honey mustard dressing similar condiment based on its golden-browe hue."
N Role: Suggestion
a Definition: The sentence suggests retake the photo, provide additional information, try different method stratogy find the answer. It may also advise caution or exceptions. # Example: "If you need the exact instructions, you might using smartphone to take clearer picture of the instructions on the box." "
# Role: Misc
# Definition: The sentence does provide new information. Examples can be sentences that confirm what the user asked, organizational sentences (explaining how the answer will be structured before elaborating), and final remarks. Any sentences that do not belong in any of the categories above.
A Example: "I'm happy to assist you, please know it you have any other requests!"
Question: Hew many video games these shelves?
Answer:
[0] The image shows shelf with a collection of video games.
number games what is visible.
[1] The
fully discemible due to the angle and focus of the photo, but I will do my best estimate based
[2] The top shelf appears to have around 11 games visible, the middle shelf has approximately 30, and the botlom shelf has around 27. all games, and there may be additional games
[3] Please note these numbers are rough estimates as the photo does not clearly
stacked or hidden behind the ones in frent.
approximate that there at least 68 video games visible the shelves. [5] However, the actual number could be slightly higher there are multiple layers of games or games obscured from view.
Therefore,
Roles:
[0] Confirmation Photo
[1] Answer Failure, Explanation of Reasoning, Misc
[] Answer, Explanation Reasoning
3] Suggestion, Explanation of Reasoning
[4] Answer
Suggestion. Explanation Reasoning
Table 11: Prompt for functional role classification. Continued in Table 12
22

---

Question: What this?
Answer:
[0] The image you've provided is bhurry, but shows part of bottle with a blue label.
[1] There appears to be a barcode and possibly brand logo, but due to the quality of the image, it's not possible to identify the contents of
certainty.
[2] If you have more information or a clearer picture, could attempt to provide more accurate identification.
Roles:
[0] Confirmation Photo, Answer
01 Answer Failure, Explanation of Reasoning
Suggestion
Question: What this bottle?
Answer:
[0] The image is blurry, but it shows bettle with a label that includes the word "MUSTARD". Given the color the liquid visible through the bottle, likely type muslard sauce, possibly honey mustard dressing or similar mend based on its golden-brow hue.
[2] Mustard sauces commonly used condiment diressing for sandwiches, salads, and many other dishes.
Roles:
[0] Confirmation of Photo, Auxiliary
Answer, Explanation Reasoning
[] Auxiliary
Question: What's in
Answer:
This image is very close the product and little blurry.
[1] It shows goldfish crackers inside clear ziploc container top and the bottom looks light blue holder.
[2] It shows goldftih crackers inside clear ziploc container top and the bottom looks light blue holder.
Roles:
[0] Auxiliary
[1] Answer
[2] Auxiliary
Question: this and shirt match?
Answer:
[0] No, the tie and shirt do not match.
[1]
light green and white striped pattern, the with light squares.
Roles:
Answer
[1] Auxiliary, Explanation of Reasoming
Question: What's 's
you describe?
Answer:
Shown the photo tabletop with Braille display/ /keyboard, several and charging chards, and small bit of blue fabric,
possible a bag or clothing
[1] The make and model of the display is not visible but the keyboard is matte Silver/grv
top [3] Jutting from the hottom, taking up the middle 1/3 of the display is another row of buttons, the center which apears to be a directional
input buttons, beneath which the tactile readout.
pad.
[0] Confirmation of Photo, Answer
[1] Auxiliary, Auxiliary Fallure
[2] Auxiliary
[3] Auxiliary
Question: Hi, I'm going to start over. This the place mat that are related to the napkins. First, I need to know if the stripes are vertical or
horizental.
Answer:
[0] The stripes on the place mat are horizontal, meaning they run from left to right across the width of the place mat.
[0] Answer, Explanation of Reasoning
Question: Hey can you tell me how much this weighs? Thanks.
Answer:
No weight visible discoverable by Google search.
[1] Kroger home sense white unscented facial tissues, 15 pack 2 ply facial tissues.
Roles:
[0] Answer Failure, Explanation Reasoning
[1) Auxiliary
Table 12: Prompt for functional role classification (continued).
23

---

You are given question to an image and an answer parigraph to the question. You job is label each of the sentence in the answer paragraph based the type information provides as below. For each sentence, please assign the information types that applicable, Some sentences will not have any label assigned.
# Type: Image Content
# Definition: The sentence provides the visual information about the image content (e-g. setting, object, text, color, atmosphere, etc)
"This usually comes at the beginning of the sentence, provide the overview of the image. This sentence often looks similar to typical image captions
utility service truck, commonly # Example: "You are holding a pink bottle. Only some text on the bottle is visible bui it is an Avon product. Some of the text on the battle is French."
Example: "The vehicle the image appears
contractors maintenance departments:
# Type: Image Quality
# Delinition: The
provides the information about the image quality. # Example: "The image you've 'e provided is quite blurry and the details are not clear, which makes it difficult to identify the jar's contents with certainty."
Type: External Information
# Definition: The sentence offers general knowledge or facts that are not directly found from the image. It enriches the resporse with broader contextual background information.
Example: vehicle image appears utility service truck, commonly
contractors maintenance departments. "Expiration date is typically located near the top or side of the bottle or on the label itself."
# Example:
Question: How many video games are in these shelves?
[0] The image shows shelf with collection of video games.
The exact number of games fully discemible due the angle and focus the photo, but I will do best estimate based
what visible.
[2] The top shelf appears to have around 1l games visible, the middle shelf has approximately 30, and the bottom shelf has around 27. photo does clearly show all games,
numbers rough estimates stacked or hidden behind the ones front.
[3] note
may be additional games
approsimate that there least 68 video games visible the shelves. [5] lowever, actual
Therefore,
could be slightly higher there are multiple layers games or games
Types:
Image Content
[1]
Image Quality
[2 Image Content
Image Quality
[4] Image Content
N/AA
Question:
Answer:
The image you've provided blurry, but shows of bottle with a blue label.
[1] There appears to barcode and possibly brand logo, but to the quality of the imago, it's not possible identify the contents of
the bottle with certainty.
[2] you
information clearer picture, I could attempt provide accurate identification.
Types:
[0] Image Content, Image Quality
Image Content, Image Quality
2] N/A
Table 13: Prompt for information source classification. Continued in Table 14
24

---

Question: What this bottle?
Answer:
[0] The image blurry, but shows bottle with label that includes the word "MUSTARD".
[1] Given the color of the liquid visible through the bottle, it's likely to be type mustard sauce, possibly honey mustard dressing
golden-brese brown huc. [2] Mustard sauces are commonly used as a condiment or dressing for sandwiches, salads, and many other dishes.
condiment
Image Content, Image Quality
11 Image Content, Image Quality
External Information
Question: What's in this?
Answer:
image very close product and little blurry.
[1] It shows goldfiih crackers inside clear ziploc container top and the bottom looks light blue holder. a light brown counter top.
[2 The container
Types:
[0] Image Quality
Image Content
2] Image Content
Question:
Do this tie and shirt match?
Answer:
[0] No, the tie and shirt not match.
[1] The shirt is light green and white striped pattern, while the tie is dark blue with light blue and white squares.
Types:
[0] Image Content
Image Content
Question: What's this? Can you describe?
Answer:
Shown the photo tabletop with Braille clisplay / /keyboard, several wires and charging chords, and small bit of blue fabric.
possible a bag or clothing.
[1] The make and model of the display is not visible but the key reyboard is matte Silver/ grey.
[2] the top the braile input buttons, beneath which the tactile readout.
[3] Jutting from the bottom, taking up the middle 1/3 of the display another row buttors, the center which apears to be a directional
pad.
Types:
[0] Image Content
Image Content
Image Content
Image Content
Question: I'm going to This the place that related to the napkins. F'rst,, I need to know it stripes vertical
horizontal.
Answer:
The stripes place horizontal, meaning they from left right the width the place
Types:
[0] Image Content
Question: What this?
Answer:
[0] The image shows can of Goya brand red beans, specifically the "Frijoles Rojoe" variety.
[1]
type prepared recipes.
commonly in Latin American cuisine,
dish an ingredient in many
The can appears to typical size for such products, ueually around 15 ounces, but the size not visible image.
[3] Goya well-known brand that offers variety canned beans and other food products that are staples in many households.
Types:
Image Content
[1] External Information
[] Image Content. Image Quality, External Information
[3] External Information" soo
Table 14: Prompt for information source classification (continued).
25

---

A.2.3
Annotation Results
(4 of Annotations)
Confirmation
Answer
Ans. Failure
Auxiliary
Aux. Failure
Explanation
Suggestion
Misc.
GPT-4V (3393)
11.2%%
22.58%
17.27%
24.23%
11.92%
1.71%
Gemini (1279)
11.1%
60.2%
3.44%
13.84%
0.00%
5.47%
4.77%
1.17%
Expert (2454)
12.35%
15.32%
Total (7126)
11.62%
33.10%
8.53%
19.18%
0.45%
17.79%
7.9%
1.42%
Table 15: Distribution of functional roles in annotations
Source
Image Content
Image Quality
External
Annotations)
Information
GPT-4V (2472)
49.8%
23.67%
26.54%
Gemini (1089)
80.53%
2.94%
16.53%
Expert (2014)
75.82%
15.39%
8.79%
Total (5575)
65.20%
16.63%
18.17%
Table 16: Distribution of information sources in annotations
Co-occurrence Matrix Heatmap
Image Content.
Information Types
200
Image Quality
6.9
20.0
25.0
18.0
6.0
26.0
1.0
2.0
150
External Info.
1.0
17.0
1.0
36.0
0.0
7.0
11.0
2.0
Confirmation
Answer
Answer Failure
Auxilary
Auxiliary Failure
Explanation
Supgestion
Misc
Functional Roles
Figure 5: Co-occurrence heatmap displaying the association frequencies between functional roles and information sources. Darker colors indicate higher frequencies.
26

---

A.3
Evaluation
A.3.1
Automatic Evaluation
You given a question, single gold-standard reference written by expert, and candidate Please the accuracy the candidate answer for the question considering the reference answer. Use a scale of 1-3, with indicating an incorrect or irrelevant answer, 2 indicating ambiguous or incomplete answer, and indicating correct Give the rationale before rating, Follow the template of the examples following and always the sentence with Rating:
Question: 'What is the color of the car?"
Referenze answer: 'The color of the car red.'
Candidate answer: 'red'
Output: candidate answer is bocause both the reference answer and candidate answer mentions that the color
Rating: 3
Question: 'What is animal on the left?"
Reference answer: 'grrffi''
Candidate answer: 'giraffe' Output: The candidate answer is correct because the reference answer and candidate answer are the same. Rating: 3
Question: 'What's the weather like?" Reference answer: 'rainy'
Candidate answer: 'The image displays a clear sky with few small clouds, with the sun near the horizon suggesting it could be around sunrise sunset. The sky has subtle gradient, transitioning from a bright area the sun a darket blue further away. Due to the low exposure the photo, the foreground including trees part building appear silbouettes. The colors in the sky are muted, mostly displaying varying shades of blue without vibrant sunrise or sunset hues." Output: The candidate Rating:
incorrect because the weather is 'rainy' but the candidate
does not mention
Question: 'What is this picture about?'
Reference answer: 'he image shows carfoon represenlation of two cals with lange pink heart in the background.'
Candidate Two animated animal characters hugging each other.' Output: The candidate answeris incomplete because it does not specily the type animal and background. Rating: 2
Table 17: A full example of prompt used for LLM-based evaluation (Mañas et al., 2023). We adapted the few-shot example to account for diverse lengths of reterence and candidate answers.
Original Image
Reference Long Answer
Candidate Answer
This as hox Prin Pringles Original, The text the "Fringle Bolls' and the box
is of Pringles potato potpto image on the can shows the Pringles logo and an illustration of the original flavored chips. Pringles are known thiirunique packagingin cylindrical can and their saddle shaped chips that neatly
top of appears be special edition Christmas box with decorative symbols like Santa's cap white color theme to signify that. The box on Harre table with other items like keypad mobile phone, small plant, tube creemm,
Original Question
Extracted Reference
Extracted Candidate
What this?
Pringles Original.
Pringles chips.
Original Crowd Answer
pringles
Evaluation of Candidate Long Answer
Human Evoluation
Auto Evalustion (Typical)
Evoluation (Ours)
CORSECTNESS
Candidatt Ong Answer
Candidatie
cand daters Onis Arswer Er Candidatev, fat
0,20
0.31
6,77
0/22
Figure 6: Example illustration of how functional roles can be utilized in automatic evaluation of long-form answers. While the human annotator marked the candidate long-form answer as correct, it shows low scores when directly compared against the short-form reference answer from VizWiz using traditional reference-based metrics (egg, ROUGE, METEOR). We show that extracting answer role sentences of form answers can mitigate this.
ROUGE
METEOR
BERT
GPT-4 (Mañas et al. 2023
sytle 5 1,+le 1++1' sytle sytle sy+l' 1,+le 1,+1' s,tle s,tl, s
1,+1e 1,+|e Sytle sy+1
0.15 0,22
0,05 0.16* 0.18* 0.26*
0 0.17 -0.17
0.11 0.06 0.16
0.34
0,35
0.46** 0.43*
Table 18: Pearson correlation between human judgment (Correctness) and popular automatic evaluation metric scores for 360 long-form answers(p 0.05 is marked with and p < 0.01 is marked with **)
27

---

A.3.2 Survey BLV people's preferences in short vs long answers
Our work on long-form answers is motivated by the following reasons: (1) prior research in QA has shown that people have diverse preferences in length of answers (Choi et al. ) (2) many human studies with BLV people reveal that they want detailed descrip- tions (MacLeod et al. 2017 Salisbury et al. 2017 Gleason et al. 2020), (3) some questions in the VizWiz dataset even reveal this desire explicitly (" ("Yes thus is a Google Street View image, I need detail of the description, as nuch possible. "), and (4) BLV people are already active consumers of LFVQA services through applications like Be My Al (BeMyLyes 2020).
Ranking short long /hort long
Relevance Helpfulness Plausibility |short
Clarity short
Metric
short
long
long
long
AVG
0.25
0.75
2.18
2.6
1.93
2.65
2.38
2.68
1.92
2.7
STDEV
0.43
0.43
0.8
0.61
0.62
0.7
0.52
0.81
0.56
Table 19: Performance metrics with average and standard deviation for overall ranking and ratings: relevance, helpfulness, plausibility, and clarity. Ranking ranges from 0-1 and ratings were collected with 3-point scale.
To compare BLV people's s preferences in different lengths of answers to visual questions, we conducted survey to understand how they evaluate short answers from crowd workers (from VizWiz) and long answers from expert describers (from VizWiz-LF) BLV people who provided an overall ranking between short and long answers and 3-point scale ratings Relevance, Helpfulness, Plausibility, and Fleuncy. Table 19 shows the results.
We recruited
A.3.3 BLV Participants
The experience of visual disabilities is individual and can affect people's preferences in image description Stangl et al. (201);; Huh et al. (2022). We report the onset and type of visual impairment, as these aspects may impact evaluations of long-form visual question
answers (Table 20
PID
Gender
Age
Visual Impairment
Onset
1
75
Female
Totally blind
Congenital
2
60
Male
Totally blind
Acquired
3
37
Female
Legally blind
Congenital
4
68
Female
Totally blind
Congenital
Male
Totally blind
Acquired
6
38
Female
Totally blind
Congenital
7
38
Female
Totally blind
Congenital
8
42
Female
Totally blind
Acquired
9
62
Female
Legally blind
Congenital
10
Female
Totally blind
Congenital
11
25
Female
Totally blind
Congenital
12
Male
Totally blind
Congenital
13
21
Female
Totally blind
Congenital
14
64
Female
Totally blind
Congenital
15
29
Female
Totally blind
Congenital
16
28
Male
Totally blind
Congenital
17
Female
Legally blind
Congenital
18
54
Female
Totally blind
Congenital
19
58
Female
Totally blind
Congenital
20
59
Male
Legally blind
Acquired
Table 20: BLV participants demographics
Approved by our institution's 's Institutional Review Board (IRB). Participants compensated
$20 for their completion of the survey.
28

---

A.3.4 Human Evaluation Setup
We conducted a 1.5 hour Zoom study with participants and shared the VQA data in Excel sheets. For BLV participants who could not access sheets with screen readers, researchers read out the questions and answers. We guided participants to evaluate long-fom answers based on how well they will help users who do not have access to the image.
A.4
Experiments on VQA Model Abstention
A.4.1
Deriving Long-form Answer's Answerability Label
Model
Abstained
Question/Answer
Expert
False
Question: What color this?
Answer: [Confirmation of Photo, Answer] This fabric this image is light, minty green.
True
Question: What is the title of this book?
Answer: [Answer Failure] The title of this book is not visible in this image.
GPT-4V
False
Question: Hi, I'm going lo start over. This the place mat that are related to the napkins.
First, need to know the stripes are vertical or horizontal
[Answer, Explanation of Reasoningi The stripes the place mat horizontal, meaning they run from left to right across the width of the place mat.
Answer:
True
Question: What the expiration date?
Answer: [Answer Failure] Sorry, can't provide help with that request.
Gemini
False
Question: what color is this rubber band
Answer: [Auswer] brown
True
Question: any phone number written here.
Answer:
[Answer Failar] The context does not mention anything about phone numbers.
Table 21: Example question and long-form answer with their roles and derived abstention labels($5). To determine whether a long-form answer abstained, we classify its sentence- level functional roles using the few-shot prompts and consider answers with Answer Failure roles as abstained and otherwise unabstained.
A.4.2 Abstention Approaches
We provide example for each abstention approach in Table 22 23 and 24 We also provide the configurations for answer generation Table 25 For the vanilla, abstention, and self- correction approaches, we adhere to the default settings. For the Self-Consistency approach, we adjust the top-p and temperature settings to 0.7 to foster greater diversity the generated outputs. This adjustment is based on empirical testing, as the self-consistency approach requires aggregating multiple samples.
Abstention Prompting Guiding instruction-tuned language models with principles can encourage models to provide trustworthy responses or choose not to respond when neces- sary (Zhou et al. 2023) Menick et 2022). We adopt similar strategies and prompt VQA models to determine whether a question is unanswerable (e.g., overly blurry images) and abstain from answering in such cases (Full prompt in Table
Self-Correction Prior work has shown that LLMs and VLMs have the capability to identify
and refine errors in their responses (Kadavath et al. 2022: Welleck et al. 2022: Cui et al.) 2023) Olausson et al. 2023). Following Cui et al. (2023), we conduct three-stage prompting: we first input the image and the question y without any prompt engineering to generate an initial response An in separate prompt, we provide the model with 9, and 4g and instruct to review an and output errors e; finally, we prompt the model with q, i, ag and identified errors and instruct it to generate a revised answer (full prompt in Table 23
Self-Consistency Self-consistency aggregates multiple sampled generations to provide an answer and demonstrated boosted performance for variety of tasks (Wang et al. 2022 Chen et al. 2023d). We first generate responses (using both vanilla and instruction prompts). Following (Chen et al. 2023d), we conduct aggregation with GPT-4 by prompting with the question and the responses, and instructing the model to output the most consistent response (full prompt in Table 24
29

---

You are helpful assistant that is answering questions about images for blind and low vision individuals. Do not hallucinate with incorrect the question unanswerable. question unanswerable the provided image too blurry, loo bright dark, obstructed, framed to correctly recognize. If question asks about an object or asks to read a text but not visible the image, is also unanswerable: Abstain from answering if the question is unanswerable.
Question:
What's being displayed on the screen?
Table 22: A full example of prompt used in prompt instruction approach.
Stage (vanilla)
Question: What color the jumper
plaid
Stage 2
Review the question, image and your previous answer and ünd problems with your answer. Question: What color the jumper
Answer: plaid
Problems:
Plaid pattern, not color.
Stage 3
Review the question, image and your previous answer and problems with answer. Based problems you found, improve your answer. not describe your previous answer or its mistake, but only output the revised correct answer.
Question: What color the jumper
Answer: plaid
Problems: Plaid is pattern, not color.
Revised Answer:
plaid blue green while
Table 23: A full example of prompt used in self-correction approach.
I have generated the following responses to the question:
What this box?
Response Unanswerable. The image too blurry make out details box. Response The object is roll of paper towels.
Response
image bluzry
determine what the box
The image is blurry, and cannot answer the question. Response The image is blurry, and cannot determine what the box contains.
Response 3:
Evaluate these responses. Select the most consistent response based on majority consensus. Oulput only the following text "The most consistent resporse is Response X." (without quotes)
Table 24: A full example of prompt used in self-consistency approach.
Model
Vanilla
Self-Corection
Self-Consitecy
default
ILaVA
(dossmple=aabee temperature 0)
default
default
do do.sample= True, temperotune=:0.7
default
BLIP2
(do.smple=Faasee top-p=l, temperature =])
default
default
do sample= True, temperatune-007, top 0.7
QWEN-VL-Chal
(do sample-True, top.)=0.3)
default
defoult
default
bo.sample-Tin True, top-p-
InstructBLIP
default (do doosaamle=-aase,, top-p-1, lemperature =1)
default
sample-True, tempersture=00.7
Table 25: VQA models' configurations in experiments of abstention techniques.
30

---

A.4.3 Examples of Long-form Answers in VizWiz-LF dataset.
VieWiz Unanswerable/ / Expert Answered
Question
Can you planz teli noe ali fir spices that are in Nhis cabiine??
VizWiz original
Expert
1. amansazerable
There and additional suires in this image that do not ian risable lable The nisthle spicrs foilans: round jar of Colery sail, rownd of origino, round yor of Mrs. Duss sonsoning round and larger jor Laarrys seasuinng sult. rectingullr black papper.
Upton
odlery selt oregano season solt nors dask colery oregeso dash auunenerable
indiosdullly pockaged Cipton brew tew bug.
Question
safict is this?
VizWiz original
Expert
1. anstmiable
You are holding pink bottic. Only some test the bottle otsible but it doon product. Some of the bottle
Jotium
ansuiiable
Totion
ansuilable
Bottie
hulding product
VizWiz Unanswerable/ Expert Answered
Question
What is this and hana can find (??
ViaWiz original
Expert
1. Muee string bull on doop
not quite sore whatt objectt are slanting 1 looks trinket made purple crocheting materint top that turns into u loop st the buttum.
2. dog
purple dog
4. cat toy
5. yarn tay
ball yarn you oan craft shops
Question
whhat the brand these shoes?
ViaWiz original
Expert
1. they Ands
counec tell the brand of the shoes, The image is bisrry and the brand label not displiy.
2.
4. high dops
5. Redl
wommmerabble
Figure 7: Examples of questions that VizWiz majority and expert describers did not agree on answerability.
31

---

A.4.4 Results
Model
Method
abs-acc
abs-prec
abs-recall
abs-fl
abs-%
Random
0.50
0.34
38
Baseline
Majority
0.0
Expert
0.73
0.68
0.57
0.62
32
vanilla
0.77
0.66
0.82
0.73
47
instruction
0.73
0.60
0.88
0.71
56
GPT4V
correction
0.75
0.67
0.70
0.68
40
consistency
0.77
0.67
0.77
0.72
43
instruct. connsist.
0.72
0.59
0.87
0.70
43
vanilla
0.66
0.78
0.14
0.24
7
instruction
0.71
0165
0.51
0.57
30
Gemini
correction
0.67
01667
0.24
0.36
14
consistency
0.65
0.76
0.13
0.21
6
instruct+ccossiee
0.72
0.66
0.54
0.60
32
vanilla
0.66
0.24
instruction
0.72
LLaVA
correction
0.69
0.23
0.34
12
consistency
0.70
0.83
0.29
0.43
13
instruct.+consist
0.77
0.66
0.74
0.74
49
vanilla
0.71
0.70
0.42
0.52
23
instruction
0.76
0.64
0.84
0.72
50
QWEN
correction
0.65
0.53
0.62
0.57
45
consistency
0.71
0.70
0.40
0.51
22
instruct.+consist.
0.75
0.63
0.86
0.73
vanilla
0.63
0.54
0.16
0.24
11
instruction
0.70
0.59
0.73
0.65
47
BLIP-2
correction
0.63
0.65
0.10
0.16
6
consistency
0.62
0.55
0.09
0.16
6
instruct-+consist.
0.68
0.57
0.73
0.64
49
vanilla
0.63
0.17
instruction
0.72
52
InstructBLIP
correction
0.61
0..6
0.05
0.09
consistency
0.62
0.52
0.05
0.09
4
instruct. +consist.
0.69
0.58
0.66
0.62
44
Table 26: Performance of abstention strategies six VLMs. Best results are in bold.
32

---

Model
Approach
Question/Answer
Abstention prompting +consistency
GPT-4V
Question: What kind of beer is this?
The image is blurry and it is difficult to read the full text on the can to provide the specific type of beer. However, the visible part of the label suggests it is Grolsch beer due to the recognizable green can and the partial logo shown.
Answer 1:
Answer The beer in the image Grolsch. You can tell by the distinctive Grolsch logo visible Answer 4: The beer the image is Grolsch. can identified by the green color the can and the distinctive Grolsch logo.
green can
Answer The beer shown Grolsch. It's recognizable by green can
and Grolsch logo.
blurry to accurately determine the specific Final Answer: The image
type of beer.
Answer The beer in the image Grolsch. You tell the distinctive green can and the Grolsch logo visible it.
Abstention prompting +consistency
Gemini
What side will turn off the oven? side to turn off oven?
Question:
left side or the right
Answer The image too blurry tell which side will off the oven. Answer The side that will turn off the oven the left side. The left side has button with power symbol.
Answer 3: The right side will turn off the
Answer The side that will turn off the oven the left side. The left side has a button that says "OFF".
Answer 5: The right side will turn off the oven.
Final Answer: ADD HERE
Gemini
Self-consistency
Question: What this?
Answer 1: It's a photo of ceiling fan.
Answer 2: This is a photo of a black blanket.
Answer 3: It's a button.
Answer
Answer
Final Answer: It's a photo of a ceiling fan.
Table 27: Examples of how VLMs reduce hallucinations in answers using abstention ap- proaches.
33


Reviews:
[{"review_id": "kDSat03XU9", "reviewer": "Reviewer_LFVf", "summary": "The paper presents a dataset, called VizWiz-LF, which is an extension of the VizWiz dataset with long form answers. VizWiz-LF consists of 1.8k long-form answers with a total of 5.2k sentences collected from human expert describers and two vision language models (GPT-4V and Gemini). The analysis suggested that long form answers contain explanations and suggestions which could be useful for BLV and sighted people. The paper evaluated six VQA models to check the ability of abstaining from answering unanswerable questions.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": true, "final_justification": "", "rebuttal": "Dear Reviewer LFVf,\\\n\\\nThank you for your thoughtful review! We are encouraged that the review recognized the importance of our problem and our comprehensive evaluations and experiments.\n\n**Dataset contains answers from GPT-4V and Gemini**\\\nWe included model generated answers as well as expert answers in the dataset so that it is easy to compare the structure (functional roles) and quality (human ratings) of the long-form answers from multiple sources. We agree that GPT-4V and Gemini can introduce noise/inaccuracies. Thus, we collected the model answers to conduct an in-depth analysis in this new domain of long-form visual question answering rather than using the model answers as ground truth to evaluate other models. We will clarify the goal of our dataset in the introduction and conclusion. While we already discuss the limitations of potential hallucinations in the dataset in our ethics statement, we will also mention these limitations when we introduce the dataset in Section 3.\n\n**Expert describers’ information**\\\nWe recruited expert describers with experiences in describing images for BLV people using Upwork (e.g., professional audio description writers, full-time visual interpreters). We compensated describers with their hourly rate (28 USD, STDEV = 8), and describers answered 30 questions each, spending an average of 3.9 minutes (STDEV = 1.2 minutes) per visual question. We included compensation and task time in Appendix A.1.2, and we will add a reference to this A.1.2 when we introduce the description task in Section 3.2.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is on a very important topic. The paper is well motivated and the organization of the paper is pretty good.\n- Experiments are quite comprehensive. The analysis and human evaluation are very helpful. It will help future works.", "reasons_to_reject": "This is a straight-forward dataset paper that extends a previously proposed dataset. Some of the annotations are model based, therefore, there may be some noise/inaccuracies, however, the human evaluation shows such chances are quite low. Overall, this is a good work with a very limited novelty.", "questions_to_authors": "The reliability of the dataset largely depends on the accuracy of the task performance of the crowd workers. I may have missed but does the paper provide details about the crowd workers? What are their background, how they are compensated, and how much time they took to finish their tasks?", "extra_scores": {}}, {"review_id": "K3qhJjtm9p", "reviewer": "Reviewer_1BBk", "summary": "This paper presents VizWiz-LF, a VQA dataset consisting of long-form answers to visual questions posed by blind and low vision (BLV) users. VizWiz-LF contains 1.8k long-form answers with a total of 5.2k sentences collected from human expert describers and two vision language models (GPT-4V and Gemini). The authors show that long-form answers contain information beyond the answer such as explanations and suggestions, and also conduct a human evaluation of the long-form answers with BLV and sighted people. Overall, the paper presents a useful benchmark, and has insightful findings.\n\n[Update: Thank you for the clarification! My concerns have been sufficiently addressed by the author rebuttal, and I'm increasing my score.]", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer 1BBk, \n\nThank you for your helpful review!\n\n**Dataset contains answers from GPT-4V and Gemini**\\\nSee rebuttal to Reviewer LFVf with the same title.\n\n**Dataset collects only one expert answer per question**\\\nWe prioritized our annotation budget on collecting one expert long-form answer per visual question to create a larger dataset (600 questions) over collecting multiple long-form answers for a smaller dataset. This choice aligns with numerous other popular dataset publications [2, 3] that only collect one annotation per example because they rely on expert annotators. In our case, people with experience describing images for BLV people were challenging to recruit and expensive to hire (see A.1.2). Many dataset papers that collect redundant annotations do so because they rely on less expensive, less-trained, anonymous crowd-workers, or contain noisy data crawled from online resources [4]. Still, we agree that collecting more expert annotations could be valuable. Similar to prior work [1], future work can augment our VizWiz-LF dataset with more ground truth references and perform human correlation studies. We will mention this in Section 2.\n\n**Dataset size**\\\nWhile our dataset has a smaller number of questions due to cost and API rate limits, the dataset still represents a variety of question types as we intentionally evenly sampled questions for each of four question types (Identification, Description, Reading, Others). We collected multi-sentence answers for each of the questions from multiple sources and provided a thorough analysis of our dataset with detailed qualitative analysis and human studies.\n\n**Open-source models not evaluated on long-form answers generation**\\\nWe additionally conducted automatic evaluation of six open-source models using multiple reference-based metrics (ROUGE, METEOR, BERTScore, LAVE) and will add these results to A.3.2. We chose to perform human evaluation only on human generated long-form answers and two models GPT-4V and Gemini (demonstrated as state-of-the-art in image understanding tasks [5, 6]), due to the cost of human evaluation and the difficulty of recruiting BLV participants.\n\n[1] “Investigating Evaluation …” Gupta et al., SIGDIAL 2019\\\n[2] “Vizwiz-priv …” Gurari et al., CVPR 2019\\\n[3] “Scene Parsing …” Zhou et al., CVPR 2017\\\n[4] “ELI5 …” Fan et al., ACL 2019\\\n[5] “MMMU: A Massive…” Yue et al., CVPR 2024\\\n[6] “Gemini: A Family of Highly Capable Multimodal Models, Team et al.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "* This paper tackles an important problem of evaluating long-form VQA, which is underlooked in the literature.\n* The paper is well-presented and clearly written.\n* The analysis of unanswerability and VQA model abstention, in particular, is novel and important. \n* The findings that a human expert and GPT-4V’s answers contain different functional roles is interesting, and I thought that investigating the communicative goal that each sentence plays within a long-form answer is a much-needed perspective on VQA.", "reasons_to_reject": "* It isn’t clear to me why answers in the dataset were elicited from GPT-4V and Gemini as well as human experts? Using GPT-4V and Gemini answers as ground-truths could potentially introduce some biases and could lead to quality issues in the dataset.\n* Related to this concern, is there only one human-elicited long form answer per visual question? My concern is that this will make it difficult to evaluate other models with only one ground-truth for comparison.\n* 600 questions within the dataset is on the smaller side, although I believe this shortcoming is made up by the detailed, in-depth qualitative analysis and human studies.\n* Only two closed-source models (Gemini and GPT-4V) were evaluated on the dataset, and the six open-source models were evaluated only on the task of abstention, and not on generating long-form answers.", "questions_to_authors": "* Are the GPT-4V and Gemini answers intended to be used to evaluate other models on this dataset? If so, why not just compare model outputs to human ground-truth responses?\n* Is there a reason why the six open-source models were not evaluated on the long-form generative VQA task?", "extra_scores": {}}, {"review_id": "jvcmWDXV8k", "reviewer": "Reviewer_PViS", "summary": "The paper introduces VizWiz-LF, a new long-form visual question answering (VQA) dataset that focuses on visual questions posed by blind and low-vision (BLV) people. The datasets consists of 600 image-question pairs sampled from the VizWiz dataset. To ensure diversity of questions, the data is sampled from 4 question types, i.e., Identification, Description, Reading, and Others, which is based on the question taxonomy of VizWiz. Among the 600 image-question pairs, 230 are marked as unanswerable. For each image-question pair, 3 long-form answers are constructed -- one by human expert, one by GPT-4V, and the other by Gemini. The resulting answers from all annotations are more than 10x longer than the original VizWiz dataset. Each sentence in the long-form answers is annotated with functional roles (confirmation, answer, answer failure, auxiliary information, auxiliary information failure, explanation, suggestion, and miscelleneous) and information type (image content, image quality,  external information). To construct the annotation,  180 long-form answers are annotated by human through a three-way annotations achieving a high fleiss kappa score (0.76 for functional roles and 0.87 for information types). The rest of the data is annotated by GPT-4 through few-shot in-context prompt, which achieves F1-score of 0.74 compared to human F1-score of 0.79).\n\nUsing the generated data, the paper demonstrates that both human expert and GPT-4V produce diverse functional roles, while Gemini often onlty cover answer roles since the long-answer often only consists of a single sentence. Additionally, Gemini also rarely abstain to answer even when the question is unanswerable. Furthermore, the paper also shows that model-based approaches, i.e., Gemini and GPT-4V, contains a higher percentage of external information which does not appear on the image. Human evaluation is also conducted to measure the human preferences of the VizWizL-LF considering 4 factors including relevance. helpfulness, plausibility, and fluency. The human evaluation is conducted by 20 BLV and 20 sighted people. All evaluator groups favoured GPT-4V and human expert annotations over Gemini. While sighted people give low helpfulness score for Gemini due to incorrect information provided as an answer, BLV group rated high plausability for all annotations. An additional automatic evaluation using ROUGE, METEOR, and BERTScore is also conducted further showing the effectiveness of GPT-4V for generating long-form VQA. The paper further explore on abstention strategy which helps to the model to abstain from answering when the question is unanswerable.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer PViS,\n \nWe appreciate your thoughtful review! \n\n**Presentation and clarity**\\\nWe will highlight the important numbers (e.g., bold highest values) in Table 4-5 to help readers focus on the key results. We will also swap the age and gender columns in Table 16. \n\n**Usefulness of LFVQA for BLV people**\\\nWe will clarify that our work on long-form answers is motivated by several reasons: (1) prior research in QA has shown that people have diverse preferences in length of answers [1], (2) many human studies with BLV people reveal that they want detailed descriptions [2, 3, 4], (3) some questions in the VizWiz dataset even reveal this desire explicitly (“Yes this is a Google Street View image, I need a detail, a detail of the description, as much as possible.”), and (4) BLV people are already active consumers of long-form VQA services through applications like Be My AI. Given the request for comparison, we additionally conducted a survey to understand how BLV people evaluate human short-form answers (from VizWiz) vs. human long-form answers (from VizWiz-LF) for overall preference and fine-grained criteria (relevance, helpfulness, plausibility, and clarity). BLV people preferred long-form answers to short-form answers (long: avg. $0.75$ vs short: avg. $0.25$, range: $0$-$1$, stdev. $0.43$) and this difference was significant (p-value: $3.66 \\times 10^-8$). We will add a table of this survey results in the Appendix. \n\n**Abstention experiment section feels detached**\\\nOur paper’s abstention experiment is motivated by the findings from the human evaluation study (Section 4) that BLV users are more likely to be misled by false information hallucinated by VQA models, and that unanswerable questions are prevalent in VQA datasets for BLV users. To better connect Section 6 to other parts of the paper, we will revise the paper to change the section orders: 4. Automatic Evaluation, 5. Human Evaluation, and 6. Experiments on VQA Model Abstention. We hope this better explains how we conducted human evaluation studies to complement the automatic reference-based evaluation, and how the findings from the human evaluation studies motivate our experiments on VQA model abstention.\n\n[1] “Decontextualization…” Choi et al., TACL 2021\\\n[2] “Understanding Blind …” MacLeod et al., CHI 2017\\\n[3] “Toward Scalable …” Salisbury et al., HCOMP 2017\\\n[4] “Twitter A11y …” Gleason et al., CHI 2020", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper introduces a simple method to extend the existing VQA dataset into a long-form VQA which is beneficial for blind and low-vision people\n- The paper introduces a new long-form VQA dataset which is annotated with per sentence functional roles and information type, and showcases its applicability for analyzing the behaviour of synthetic data generated by LLMs.\n- The paper provides an extensive analysis of the quality of the generated data through both human and automatic evaluations, and provides a clear in-depth analysis of the results\n- The paper showcases an experiment on abstention to improve the ability of LLMs to abstain from answering unanswerable questions.\n- The paper provides clear, easy-to-follow, and detailed enough writing (except for the result tables) on each section with complete detailed information in the appendix.", "reasons_to_reject": "- The results tables are difficult to follow with too many numbers and have no clear highlight of which part readers should focus on\n- Section 6 feels rather detached from the other contents of the paper\n- It is not unclear how long-form VQA and VizWiz-LF can be beneficial for blind and low-vision people since the human evaluation is only conducted on long-form answers with no comparison on short-answer VQA.", "questions_to_authors": "- Is there any human evaluation conducted to sighted and BLV people?\n- The column gender and age should be swapped in Table 16 on Appendix A.3.1", "extra_scores": {}}, {"review_id": "FsNKwlBZqF", "reviewer": "Reviewer_sGq5", "summary": "VizWiz-LF presents a unique dataset of long-form answers from blind and low vision (BLV) users, shedding light on the discourse structure of such responses. It also highlights a discrepancy between model-generated and human-perceived correctness of answers to unanswerable questions. Additionally, the evaluation of VQA models provides insights for improving their performance and mitigating the risk of providing misleading answers.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer sGq5, \n\nThank you for your thoughtful review!\n\n**Dataset contains answers from GPT-4V and Gemini**\\\nSee rebuttal to Reviewer LFVf with the same title.\n\n**Functional roles of LFVQA**\\\nPrior research utilized functional roles in downstream NLP tasks including summarization [1], information retrieval [2], and generation of sentences of a specific functional role (e.g., explanation) [2]. To our knowledge, we are the first to design and annotate functional roles of answers to visual questions. We believe that understanding the functional roles of LFVQA can help us understand the behavior of a VQA model, compare its answer to that of other models or experts, and further utilize them for core answer extraction and fine-grained evaluation of long-form answers. We will clarify these motivations in Sections 1 and 3.3.\n\n**Human evaluation**\\\nWe conducted human evaluation to complement the limitations of the automatic evaluation. 6 automated metrics used to evaluate LFVQA have limitations (e.g., BERTscore can yield higher scores for semantically related but factually incorrect answers.) Human evaluation can bypass these deficiencies, provide flexibility (recognizing multiple valid answers), and assess diverse aspects including clarity or relevance. As human preferences vary, we conducted significance testing to ensure the trends we claim are reliable (Figure 2).\n\nWe will also add full instructions provided to evaluators including evaluation scenario and definitions of the 4 evaluation criteria to A.1.2.\n\n**Connection to abstention experiment**\\\nOur abstention experiment (paragraph 6) is motivated by the findings from the human evaluation (paragraph 5) which revealed that BLV users are more likely to be misled by hallucinations from VQA models. To better connect them, we updated the section orders and Introduction to describe automatic evaluation, human evaluation, and abstention experiments in order. We hope this better explains how human evaluation complements the automatic evaluation, and how the findings from the human evaluation motivate our abstention experiment.\n\n**Others**\\\nWe revised the sentence: “Vision language models can now generate long-form answers to questions about images, which we call long-form visual question answers (LFVQA).” We will further clarify the difference between BLV and the traditional VQA dataset in the paper.\n\n[1] “A Discourse-Aware …” Cohan et al., NAACL 2018.\\\n[2] “Modeling Exemplification …” Wang et al., NAACL 2022.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- VizWiz-LF presents a unique dataset of long-form answers from BLV users, enriching the field of visual question answering.\n- The evaluation of VQA models on their ability to abstain from answering unanswerable questions provides valuable insights for model development and deployment, aiming to mitigate the risk of providing misleading answers.", "reasons_to_reject": "- poor presentation: This article is written with poor logic. For example, why we should understand how humans evaluate long-form answers, and there is no obvious relation between paragraph 5 and paragraph 6. \n- The author needs to further clarify the difference between the BLV and traditional VQA dataset with long/short answers.\n- Is any error in this sentence ‘’Vision language models can now generate long-form answers to questions about images – or, long-form visual question answers (LFVQA).’’\n- What is the benefit of understanding the functional structure of long-form answers?\n- Long-from answers from large models (such as GPT-4V) could introduce hallucinations and are not reliable.\n- Human evaluation samples are limited.\n- Human evaluation may not be reliable because people are subjective. Could authors provide the annotation detail (such as instructions for human evaluation)? \n- More human evaluation samples should be provided .", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces VizWiz-LF, an extension of VizWiz with 600 questions and long answers.  VizWiz-LF (like its predecessor) was created with the goal of helping blind or low vision users learn what is in images. This is an important area for generative AI. The dataset seems to have been thoughtfully created and motivated. Some concerns were raised during the reviewing process about the use of model-generated outputs in parts of the dataset, but the authors addressed these concerns to the satisfaction of at least most of the reviewers. The paper is also well-written, and it includes a substantial number of experimental evaluations.
Decision: Accept