Long-Form Answers to Visual Questions from Blind and Low Vision People

Authors: mina  huh
fangyuan  xu
yi-hao  peng
chongyan  chen
hansika  murugu
danna  gurari
eunsol  choi
amy  pavel
Venue: colmweb_org_COLM_2024_Conference
Published: 2024-03-22 01:52:52

Abstract: Vision language models can now generate long-form answers to questions about images – long-form visual question answers (LFVQA). We contribute VizWiz-LF, a dataset of long-form answers to visual questions posed by blind and low vision (BLV) users. VizWiz-LF contains 4.2k long-form answers to 600 visual questions, collected from human expert describers and six VQA models. We develop and annotate functional roles of sentences of LFVQA and demonstrate that long-form answers contain information beyond the question answer such as explanations and suggestions to retake photos. We further conduct automatic and human evaluations involving BLV and sighted people to evaluate long-form answers. While BLV people perceive both human-written and generated long-form answers as plausible, generated answers often hallucinate incorrect visual details, especially for unanswerable visual questions (e.g., blurry or irrelevant images). To reduce hallucinations, we evaluate VQA models on their ability to abstain from answering unanswerable questions.

Main paper: 
 introduction: 
 Traditional visual question answering (VQA) models respond to visual questions about images with short answers. This is because they were designed for mainstream dataset challenges for which answers typically are brief, containing one or two words (Antol et al., 2015; Gurari et al., 2018; Goyal et al., 2017) . The rise of large vision language models (VLMs) has introduced a new class of VQA models that can generate long-form answers (Alayrac et al., 2022; Team et al., 2023; Driess et al., 2023; Chen et al., 2023c; Achiam et al., 2023) . The progress in VLMs meets a long-standing need from VQA solutions (Chen et al., 2023b) . For instance, some questions necessitate a long-form answer, such as "How do I make this boxed cake?" Also, some users find answers with nuanced information to be helpful, such as answers with explanations and supplementary details (Xu et al., 2022; Naik et al., 2023) . While long-form visual question answering (LFVQA) has enormous potential, we have limited knowledge about the content and quality of long-form answers.Our work aims to understand the content and quality of long-form answers to visual questions asked by blind and low vision (BLV) people. LFVQA can be particularly impactful for BLV people who take pictures and ask questions to overcome real visual accessibility barriers. In fact, BLV people are already active consumers of long-form VQA applications like Be My AI (BeMyEyes, 2020) powered by GPT-4V. Authentic VQA involves unique challenges not present in artificial settings, such as conversational questions and low-quality images due to blur and framing errors. We seek to understand the potential and limitations of LFVQA for BLV users given its societal impact. For each visual question, we collect an answer from human expert describers and 6 VLMs (GPT-4V, Gemini, LLaVA, InstructBLIP, QWEN, BLIP-2). To understand the content and structure of long-form visual question answers, we create a taxonomy of functional roles and annotate answer functional roles and information types at a sentence level.We introduce Vizwiz-LF, a dataset of 4.2k long-form answers to visual questions from BLV people seeking visual assistance (Bigham et al., 2010; Gurari et al., 2018) . We collect and evaluate long-form answers from human experts and six state-of-the-art VQA models (GPT-4V (Achiam et al., 2023) , Gemini (Team et al., 2023) , LLaVA (Liu et al., 2024) , InstructBLIP (Dai et al., 2024) , Qwen-VL-Chat (Bai et al., 2023) , and BLIP-2 (Li et al., 2023a )).To understand the content of LFVQA, we design and annotate the communicative roles (e.g., answer, explanation, suggestion) and information sources (e.g., image content, image quality, or external information) of long-form answer sentences in our dataset. We create a classifier for the functional roles and information sources in LFVQA that performs on par with humans then annotate our dataset with this classifier (example in Figure 1 ). While the majority of answers from 5 VLMs (Gemini, LLaVA, InstructBLIP, QWEN, BLIP-2) contained only two functional roles (confirmation, answer), human expert describers and GPT-4V answers frequently included many functional roles (e.g., explanation, suggestion, auxiliary information). Most long-form answers described image content, but human experts and GPT-4V also frequently described image quality.To assess the performance of VLMs in LFVQA, we conduct an automatic evaluation of long-form answers using reference-based metrics (ROUGE (Lin, 2004), METEOR (Elliott & Keller, 2013) , BERTScore (Zhang et al., 2019) and LAVE (Ma ñas et al., 2023)) with shortform reference answers from VizWiz and long-form reference answers from VizWiz-LF. Reference-based VQA evaluations typically use short reference answers and thus penalize long answers for including extra information (Krishna et al., 2021; Xu et al., 2023) . We show that extracting answer sentences from long answers using our classifier can mitigate this.To understand how humans evaluate long-form answers, we conduct an evaluation study with both sighted and BLV people. We examined the effect of visual priming bias by evaluating with four different conditions of {BLV, sighted}×{with image, without image}. 2Our results also reveal that sighted people's evaluation without an image is not a strong proxy for BLV preferences. For instance, BLV evaluators tend to perceive incorrect answers as more plausible and rate extraneous details as less relevant than sighted evaluators.As VLMs often hallucinate with answer sentences for unanswerable visual questions, we assess six VLMs' ability to abstain from providing an incorrect answer. To determine whether a model abstains, we use our functional role classifier to check for the Answer Failure role in the long-form answer, which expresses a model's inability to answer the question. Only GPT-4 achieved high recall for abstention (0.82) followed by QWEN (0.42) with default settings. We further experiment with multiple abstention approaches and reveal that prompting VLMs with abstain instructions can reduce hallucinations.While this work addresses visual questions posed by BLV people, our findings can inspire broader research in VQA. First, our introduction of long-form answers to supplement the short answers in the original VizWiz dataset results in the first dataset with both short and long answers. Potential downstream benefits include enabling the transfer from shortanswer VQA tasks, at which current models already excel, to long-answer VQA tasks. Second, our proposed LFVQA functional roles can support improvement and evaluation of LFVQA for our use case and beyond. Finally, our human evaluation highlights the importance of evaluating LFVQA beyond factual accuracy (the focus of short-form VQA) to further consider end users' experience with metrics such as relevance and plausibility. 
related work: 
 VQA Datasets Most existing VQA datasets have brief answers consisting of one to two words collected from crowd workers (Antol et al., 2015; Zhu et al., 2016; Gurari et al., 2018; Wang et al., 2017; Krishna et al., 2017; Wu et al., 2017) . This includes the widely-studied dataset with questions asked by BLV individuals, VizWiz-VQA (Gurari et al., 2018) , even though BLV people often explicitly ask for detailed answers (e.g., "Yes this is a Google Street View image, I need a detail, a detail of the description, as much as possible." -user question from VizWiz dataset). To explore the potential of LFVQA in accessibility, we gather and analyze a dataset of detailed, sentence-to-paragraph-long answers to VizWiz questions collected from expert describers and 6 modern VLMs.A few VQA datasets also provide longer responses: ContextVQA (avg. 11 words) (Naik et al., 2023) , ScienceQA (Saikh et al., 2022) (avg. 4 words), and VQAOnline (Chen et al., 2023b) (avg. 173 words). While the answers in these datasets are collected online or from crowd workers, we collect high-quality answers from experts proficient in image description tasks. We also address questions posed by BLV individuals, which cover a different distribution of images (e.g., quality) and questions (e.g., describing physical objects).Long-form Question Answering (LFQA) LFQA systems (Fan et al., 2019; Nakano et al., 2021) generate comprehensive, paragraph-level answers to text-only questions. Prior work (Xu et al., 2022) analyzes the content types of human-written long-form answers from online community forums (Fan et al., 2019; Nakano et al., 2021) , finding that they convey rich information beyond simply answering the questions. They propose six functional roles for answer sentences, such as "example" and "auxiliary information". As we develop functional roles for LFVQA generated by both humans and models, we reveal new roles (e.g., confirmation of photos, answer failure) and different role distributions across experts and models. We also annotate information sources of LFVQA answers to assess how models reference images (e.g., image content, image quality). 2023) introduced a reference-free metric to evaluate hallucinations in VLMs' answers to visual questions. Researchers have also explored VLMs' ability to abstain when having low confidence to reduce hallucinations (Whitehead et al., 2022; Li et al., 2023b; Wang et al., 2023) . We build upon research on automatic evaluation of VQA and further conduct a human evaluation with both sighted and BLV individuals to understand their preferences. As VizWiz frequently contains unanswerable questions, we present an evaluation of VLMs' ability to abstain from answering when provided with images of poor quality. 4 . We select these models as they are publicly available, have strong zero-shot VQA capabilities, and represent diverse model architectures.In Table 1 , we report statistics comparing the collected long-form answers with the original VizWiz short-form answers 5 . Human expert long-form answers are 24x more words than original VizWiz short answers, indicating previous short answers were insufficient. Modern VLMs also generate long responses, but diverge from human expert answers -GPT-4V answers are 1.7x longer, while the other VLMs generate answers that are 0.7x or shorter. 
functional role analysis: 
 Functional Roles We design and annotate functional roles of sentences in LFVQA, to characterize the communicative goal of answer sentences in addressing visual questions.To derive the taxonomy, three authors used open coding (Hashimov, 2015) , an iterative process for assigning conceptual labels to segments of data. Then, they met to merge similar functional roles and create a codebook with a name, definition, and example for each functional role. The researchers iteratively coded samples and revised the codebook to achieve final codes, containing eight functional roles (Confirmation, Answer, Answer Failure, Auxiliary Information, Auxiliary Information Failure, Explanation, Suggestion and Miscelleneous).We provide definitions and examples for each role in §A.2.1.Information Source Short-form answers contain mostly visual information about the image content (Figure 1 ). We observe that sentences in long-form answers often provide additional information, such as describing the image quality or providing external knowledge outside of the image. Thus, we create a separate taxonomy to categorize sentence-level To annotate the functional roles and information source types of the 4.2k long-form answers collected in § 3, we first collect human annotations of 180 long-form answers (522 sentences) as ground-truth. Given the question, the image, and the answer paragraphs, the annotation task is to (1) assign up to three functional roles and (2) assign all applicable information source types for each sentence in the answer paragraph. Five authors participated in three-way annotations and reached substantial agreement with Fleiss Kappa (Fleiss & Cohen, 1973 ) (κ = 0.76 for functional roles, κ = 0.87 for information sources). We annotate the rest of the data using a GPT-4 classifier, which is constructed with a few-shot prompt with definitions of each role and eight in-context examples (see §A.2.2 for the full prompt). We evaluate the classifier's performance against human annotations, achieving a weighted per-role F1 of 0.74 (compared to human-human F1 of 0.79).Table 2 -3 shows the distribution of functional roles and information sources of long-form answers in our dataset. As 38% of our sampled questions are marked as unanswerable in VizWiz6 , we report the distribution for answerable and unanswerable questions separately.Expert and GPT-4V's answers contain diverse functional roles. The most common functional role in answers from both experts and six VLMs was a direct Answer to visual questions. LLaVA and Gemini have the highest overall percentage of answers that had Answer roles. Gemini often generated single-sentence answers, only addressing the question without providing additional information. On the other hand, expert and GPT-4V answers have sentences with diverse functional roles including Confirmation, Explanation, Auxiliary, and Suggestion. GPT-4V answers often contain Confirmation sentence when the visual question does not refer to a specific object (e.g., The image you've provided is blurry, but it shows a part of a bottle with a blue label.) to confirm that users shared the right image. Answers from both experts and GPT-4V provide Auxiliary information to complement the answer. For example, when asked about an attribute of an object in the image (e.g., "What is the color of this t-shirt?"), GPT-4V and expert answers additionally describe the same attribute of other objects in the image(e.g., color of the pants), or other attributes of the same object (e.g., text logo on the t-shirt). Example Misc. sentences include organization sentences (e.g., "The screen displays the following text:") or final remarks (e.g., "Hope this helps, thanks!") In BLIP-2, answers sometimes repeat the input question and are classified as Misc.Most VLMs rarely abstain, even when the question is unanswerable. Four VLMs (Gemini, LLaVA, BLIP-2, InstructBLIP) all have a low percentage of answers with Answer Failure sentences for both answerable and unanswerable questions (Table 2 ). Expert, GPT-4V, and QWEN indicate Answer Failure more often to unanswerable questions, yet GPT-4V overabstained to answerable questions when the image was not clear. Expert and GPT-4V often have both Answer and Answer Failure sentences in a single answer. These answers typically offer a tentative guess (e.g., "The color suggests it might be a jar of pasta sauce.") while also expressing the challenges in providing a definitive answer (e.g., "However, the text is not completely legible to identify the exact product.") When GPT-4V abstains from providing an answer to the visual question, the answer often contains an Explanation for why it abstained (e.g., describing the low image quality) or a Suggestion for taking better images. Table 3 : Distribution of answers with each information sources to answerable (ans.) and unanswerable (unans.) questions. Among 600 questions, 230 were marked unanswerable in VizWiz.Long-form answers provide information beyond image content. Table 3 shows the distribution of information source types. Answers from both expert describers and all VLMs have a high percentage of information derived from the Image Content. Additionally, expert and GPT-4V answers often describe the Image Quality. Compared to other answers, GPT-4V answers provide more External Information. For example, when asked to identify the type of frozen food in an image, answers from these models further provide nonvisual information, such as the origin of the food and popular recipe used. 
automatic evaluation: 
 To analyze the performance of VLMs in LFVQA, we conduct an automatic evaluation of long-form answers with reference-based metrics. To adapt the traditional reference-based metrics to long-form answers, we consider both short crowd answers (Gurari et al., 2018) and long expert answers (our dataset) as ground truth references and explore how our functional role classifier can be used for sentence-level evaluation. To our knowledge, we are the first to consider long-form reference answers when evaluating VQA. To better evaluate LFVQA, we consider two approaches. First, we explore the potential of long-form references in VQA evaluation by leveraging the expert long-form answers in our VizWiz-LF as ground truths (l r ) for evaluating model long answers (l c ). However, many existing VQA datasets have only short ground truth answers (Antol et al., 2015; Gurari et al., 2018) , and collecting long-form ground truths on a large scale is costly. Thus, we also explore the use of functional roles in VQA evaluation with short-form references (s r ) as ground truths for evaluating extracted Answer role sentences from long-form answers (l ′ c ).Results Table 4 summarizes the evaluation results with reference-based metrics. In conventional reference-based metrics (ROUGE, METEOR, BERTScore), Gemini outperformed GPT-4V and experts. This was often due to the additional information in the long-form answers of GPT-4V and experts being penalized by reference-based metrics. In LLM-based , and extracted answer sentences of experts' long-form answers (l ′ r ). For candidate answers, we use original long-form answers generated by models and experts (l c ) and extracted answer sentences (l ′ c ).evaluation (LAVE), the trend reversed, and GPT-4V and experts scored higher than Gemini. Among all metrics, only the LLM-based metric demonstrated a moderate correlation to human ratings ( §A.3.1), demonstrating the potential of LLMs in LFVQA evaluations. For all metrics, using long-form answers as reference answers increased the score compared to using short reference answers (s r ). GPT-4V answers that showed low scores with s r exhibited a more substantial improvement in score for ROUGE and METEOR with long-form references (l r ) (Figure 6 in Appendix). While expert long-form answers are difficult to collect at scale, our evaluation highlights the need for long-form references to effectively evaluate LFVQA. Also, future work can consider more diverse functional roles than answer roles to enable a more fine-grained evaluation of VQA. For instance, sentences with confirmation of photo roles can be evaluated using image groundings, and external information sentences can be assessed with fact-checking approaches. 
human evaluation: 
 While prior research with BLV people has shown that they want detailed image descriptions (MacLeod et al., 2017; Salisbury et al., 2017) , no work has explored BLV people's preferences for answer lengths in VQA. Thus, we first conduct a preliminary survey with 8 BLV people who compare short answers from crowd workers (from VizWiz) vs long answers from expert describers (from VizWiz-LF). Participants preferred long-form answers 75% more often than short-form answers, and this preference was significant (p < .0001, details in §A.3.2). Participants ranked shorter answers higher than long-form answers when the question asked about a specific attribute of an object (e.g., "What is the color of this t-shirt?").To understand preferences for LFVQA, we conduct an evaluation study with 20 BLV and 20 sighted people 7 While most prior research in VQA evaluates answers with sighted people (Dua et al., 2021; Gardner et al., 2020) , long-form answers should be evaluated beyond factual accuracy to consider their usefulness to end users, specifically BLV people. We conduct evaluations under all conditions of {BLV, sighted}×{with image (description), without image (description)} to account for both scenarios where users have or do not have the context regarding the image settings. For BLV participants, we share a brief image description to give them context for evaluating the answer. We obtain the image description from the original VizWiz dataset (Gurari et al., 2018) authored by crowd workers8 .We conducted a human evaluation on the same data used in Section 5. The evaluation study involved (1) a preference ranking task and (2) a fine-grained answer rating task followed by an open-ended interview to understand their rankings and ratings. In the first task, evaluators were provided 12 visual questions, 3 from each question category (Identification, Description, Reading, and Others). For each visual question, participants read and ranked long-form answers from 3 sources (GPT-4V, Gemini, and expert describers) for a total of 36 long-form answers. In the second task, evaluators provided a set of ratings for Sighted with image (S+I). We measured significance using the Freidman test followed by the pair-wise Wilcoxon test. p < 0.05 marked with *, p < 0.01 marked with *, p < 0.001 marked with *** after Bonferroni correction. 36 long-form answers.9 . Participants provided one of the three labels (1 -not, 2 -partially, and 3 -very) for each criterion:• Relevance measures how relevant the answer is to the question (Levinboim et al., 2019) . It is important to understand how people rate relevance in long-form answers as they often contain extra information in addition to the core answer. • Helpfulness measures how helpful the answer is for people who cannot see the image.• Plausibility measures how likely the answer is to be correct (Gardner et al., 2020) .• Fluency measures how clearly the response conveys information, often related to the answer's grammar or consistency (Levinboim et al., 2019) .For the group of sighted evaluators who were provided with images during the evaluation, they are asked to evalaute with Correctness (i.e., how correct the provided information is given the image) instead of Plausibility.Results Figure 2 displays the normalized rankings and ratings of three long-form answer types across all four participant groups. All evaluator groups ranked responses from experts and GPT-4V higher than those from Gemini, often attributing their choice to the rich information in expert and GPT-4V responses. For relevance specifically, BLV evaluators preferred shorter answers from Gemini as they found GPT-4V answers often share extra information that is not useful (e.g., describing a shirt's surroundings when asked for the shirt's color). BLV evaluators also noted that GPT-4V's frequent image quality descriptions (e.g., "You uploaded a blurry photo of...") felt repetitive and unnecessary, unless used to explain an answer failure, as they often took low-quality photos.Sighted evaluators assessing the answer with the image (S+I in Figure 2 ) rated Gemini answers significantly lower than GPT-4V and expert answers for Helpfulness and Correctness and explained that low ratings were due to incorrect information provided in the answer.In contrast, BLV groups rated Plausibility high (0.84 out of 1.0, SD=0.28) for all types of long-form answers. Our follow-up interviews with BLV evaluators revealed that their trust in long-form answers is influenced by the level of detail and expressions of uncertainty in the answer. For GPT-4V and expert answers, extensive details made BLV evaluators perceive the answers as more correct. However, compared to Gemini, GPT-4V and experts frequently expressed uncertainty due to low image quality, thus BLV evaluators trusted Gemini answers despite their brevity. 
experiments on vqa model abstention: 
 Our human evaluation reveals that the perceived correctness of long-form answers for BLV raters does not align with the answer correctness. Assessing answer correctness is challenging for BLV users, as they cannot visually verify the answer against the image10 . Prior work 11 We investigate a suite of prompting methods and evaluate their effectiveness on our dataset.Data and Answerability Labels We consider the 600 VizWiz questions in our dataset ( §3) and use short-form answers from VizWiz as gold answerability labels, for a total of 38% of questions labeled unanswerable. To determine whether a long-form answer abstains from providing an actual answer, we classify its sentence-level functional roles ( §4) and check whether at least one sentence is labeled as Answer Failure, which expresses a model's inability to answer the question (examples in Table 21 ).We report four sets of metrics: (1) Abstain Accuracy (abs-acc), which measures whether the abstain decisions are correct against the gold answerability label;(2) Abstain Precision (abs-prec), Recall (abs-recall) and F1 (abs-f1), treating abstaining to answer as the positive label; (3) Percentage of questions that the model abstains from answering (abs-%) and (4) Answer Accuracy (ans-acc), which compares the answer against the ground truth short answers using LLM-based metric (Ma ñas et al., 2023), as it correlates well with human judgment ( §6). We report the performance of three baselines: a random where an answerability label is randomly assigned according to the distribution and a majority baseline where we always predict Answerable. We also report expert performance by evaluating the long-form answers written by experts.We investigate VQA models' abstention capabilities using three prompting techniques. The first approach Abstention Instruction includes a detailed instruction in the prompt and instructs the model to determine whether a question is unanswerable (e.g., overly blurry images), and if so, abstain from answering. We also used two additional prompting strategies from prior work Table 5 presents the performance of the default (vanilla) and abstention instruction (prompt) approaches. The abstention instruction prompt increases abstention frequency across all models (abs-%), while also improving the ability of all models to correctly abstain from answering unanswerable questions (abs-recall), especially for models with low abstention frequency in the vanilla setting (Gemini, LLaVA, BLIP-2, InstructBLIP). While precision (abs-prec) declines with abstention instructions, the F1 (abs-f1) and answer accuracy (ans-acc) increase for all models except GPT-4V. For low-quality images, opting to abstain rather than providing low-confidence answers may benefit users in the context of accessibility where wrong but confident answers can cause harm.Prompting VQA models to abstain also changes the functional role distributions in their answers compared to the vanilla setting (Figure 3 ). Answers from GPT-4V, Gemini, LLaVA, and InstructBLIP provide more auxiliary information and explanations when instructed to abstain, whereas answers from QWEN and BLIP-2 exhibit patterns of short-form VQA by generating primarily answer or answer failure roles.Additionally, we note a discrepancy in abstention between expert answers and crowdsourced short answers. Expert and crowd answers agreed on answerability in 52% of the 600 questions. Among 230 questions marked unanswerable by the crowd workers, 99 were answered by expert describers. This often occurred when a few crowd workers provided an answer but the question was labeled "unanswerable" using majority voting ( § A.4.3). Among 370 questions marked answerable by VizWiz, 61 were not answered by experts. This occurred when the experts were not familiar with specific objects in Identification questions. 
conclusion: 
 In this work, we presented VizWiz-LF, a dataset of long-form answers to visual questions posed by BLV users. Our work is the first to explore the content and structure of LFVQA from both humans and models, and to compare long-form answers to short-form answers. Our functional role and information source analysis demonstrates the rich information provided in LFVQA by both human experts and models. We also revealed that while BLV evaluators preferred long-form answers to short-form answers, there remain opportunities to improve the relevance and correctness of LFVQA. We found that abstention instructions helped VQA models to better abstain from answering unanswerable questions. Future work may use functional roles to further evaluate and improve the per-role performance of LFVQA and to tailor descriptions to context and preference. In this era of VQA, we must go beyond domain-agnostic factual accuracy to consider how to efficiently deliver relevant information. We hope that our work encourages future development and evaluation of VLMs with people who may benefit the most from using them. 
 
 Appendix: • Hallucinations in LLMs and VLMs: LFVQA answers in our dataset are collected from expert describers and VLMs. As LLMs and VLMs might generate factual errors and hallucinations, we have examined the data to make sure they do not contain harmful content before the human evaluation study. Our work also uses LLMs to classify functional roles and evaluate VQA. While we have shown that these approaches better align with humans, LLMs are not free from potential hallucinations and can lead to incorrect results. • Human Evaluation Study: Our human evaluation study was approved by our institution's International Review Board (IRB) as exempt research. In our user study, we ensured that all participants were compensated fairly for their time and contributions. The payment was determined based on the average market rate for such studies, reflecting the complexity and duration of the tasks involved. people can visually check the answers to their questions, BLV people consume model answers without further verification. Our human evaluation study reveals that BLV people have higher trust in model-generated answers than sighted people. Additionally, high scores from evaluations may further encourage trust in models leading to potential risks of unnoticed model errors. We hope robust evaluation approaches and an understanding of long-form answers can provide a more realistic picture of model performance.We will release our codes, prompt, and data collected publicly.• Collecting Long-form Expert Descriptions: Our VizWiz-LF dataset contains long-form answers from expert describers who are skilled in describing images for BLV people. Because what they decide to include in the long-form answers can be subjective, the diversity and representativeness of the human describers can influence the results. Also, compared to hiring crowd workers, it is more challenging to scale up the collection of expert long-form answers due to the difficulty of finding skilled describers and the cost. We collected one expert long-form answer per visual question to prioritize a larger dataset over multiple long-form answers for a smaller dataset which aligns with prior work (Gurari et To derive the taxonomy, three researchers used open coding on 100 sample responses to obtain potential functional roles. Then, they met to merge together similar functional roles and create a codebook with a name, definition, and example for each functional role. The researchers iteratively coded samples and revised the codebook to achieve final codes, containing eight functional roles. During the coding process, an additional need for annotating information types emerged. To better understand long-form answers that provide information beyond the image content, the researchers additionally generated the codebook for information source types. The full codebook and examples provided to annotators are shown in Table 7 .Using the codebook with examples, we collected three-way annotations of 180 long-form answers from five researchers. They reached a substantial agreement for functional roles (Fleiss Kappa = 0.76) and a perfect agreement for information types (Fleiss Kappa = 0.81). These high-quality annotations were used as few-shot examples for prompting a classifier and to evaluate their performance.Confirms what the user uploaded.The image you uploaded appears to be a carton of chocolate soymilk.Addresses the question with an answer.It expires in September 2015.States the inability to address the question.The expiration date is not visible in this photo.Provides additional information not directly related to the question.The size of the milk container is 8 fluid ounces (240 mL).States the inability to provide information not directly related to the question.I cannot provide nutritional information due to the blur.The sentence explains the reasoning for the information it provides.Given the presence of soybeans in the image, it is likely to be soymilk.Suggests retaking or improving the quality of a photo to get a better answer.If you can provide a clearer image, I might be able to better assist.Miscelleneous Does not provide new information. Sentences that do not belong in any of the categories above.I'm happy to assist you, let me know if you have further requests! You are holding a pink bottle.Provides visual information about the image quality.The image you've provided is quite blurry and the details are not clearProvides general knowledge or facts that are not found in the image.Expiration date is typically located near the top or side of the bottle We construct a prompt with definitions and in-context examples for each of the function roles or information types. Given a question and a list of answer sentences, GPT-4 outputs a list of labels for each of the answer sentences. The prompt (which includes the task instruction and few-shot examples) we use can be found in Table 11 -12 and Table 13 . We evaluate GPT-4's performance against the majority label from the three-way annotations, using per-label F1 and a weighted average F1 over all the class.To contextualize GPT-4's performance, we provide two estimates for human performance collected in §A.2.1: an upperbound, which we compare each annotator's annotation with the majority label. This inflates the performance as one's annotation is correlated with the majority label; an lowerbound which we compare all pairs of annotation and average over them. We report two baselines: (1) Random: which randomly assigns a role combination from all of the annotation and (2) Majority: which always labels the sentence as "Answer" (or "Image Content" for information type).Results are in Table 9 and Table10. We see that GPT-4 significantly outperforms both the baselines, with a moderate gap compared to the human lowerbound. The model performs relatively poorly in identifying "Auxiliary (Failure)" and "Confirmation", for which human also exhibit lower agreements. Question: How many video games are in these shelves?Answer: [0] The image shows a shelf with a collection of video games.[1] The exact number of games is not fully discernible due to the angle and focus of the photo, but I will do my best to estimate based on what is visible.You are given a question to an image and an answer paragraph to the question. You job is to label each of the sentence in the answer paragraph based on the type of information it provides as listed below. For each sentence, please assign all the information types that are applicable. Some sentences will not have any label assigned.# Type: Image Content # Definition: The sentence provides the visual information about the image content (e.g., setting, object, text, color, atmosphere, etc) *This usually comes at the beginning of the sentence, to provide the overview of the image. You are given a question, a single gold-standard reference answer written by expert, and a candidate answer. Please rate the accuracy of the candidate answer for the question considering the reference answer. Use a scale of 1-3, with 1 indicating an incorrect or irrelevant answer, 2 indicating an ambiguous or incomplete answer, and 3 indicating a correct answer. Give the rationale before rating. Follow the template of the examples following and always end the sentence with Rating: X.Question: 'What is the color of the car?' Reference answer: 'The color of the car is red.' Candidate answer: 'red' Output: The candidate answer is correct because both the reference answer and candidate answer mentions that the color is red. Rating: 3 Question: 'What is the animal on the left?' Reference answer: 'giraffe' Candidate answer: 'giraffe' Output: The candidate answer is correct because the reference answer and candidate answer are the same. Rating: 3 Question: 'What's the weather like?' Reference answer: 'rainy' Candidate answer: 'The image displays a clear sky with a few small clouds, with the sun near the horizon suggesting it could be around sunrise or sunset. The sky has a subtle gradient, transitioning from a bright area near the sun to a darker blue further away. Due to the low exposure of the photo, the foreground including trees and a part of a building appear as silhouettes. The colors in the sky are muted, mostly displaying varying shades of blue without vibrant sunrise or sunset hues.' Output: The candidate answer is incorrect because the weather is 'rainy' but the candidate answer does not mention it. Rating: 1 Our data and code are available at https://github.com/utcshci/lfvqa.For BLV users, we simulate "with image condition" by providing them a description of the image from the original VizWiz dataset.https://www.upwork.com/We generated long-form answers with a zero-shot setting. Configuration details in §A.1.3We used the majority-voted answer among 10 crowd answers (Zeng et al., 2020; Chen et al., 2023a).Following VizWiz(Gurari et al., 2018), we consider the questions with majority short-form answers of "unanswerable" and "unsuitable" as unanswerable and otherwise answerable.Approved by our Institutional Review Board (IRB). Demographics of BLV participants in §A.3.3.We randomly sample one from the five default captions after manually removing spam captions.We sample one answer from each visual question to avoid bias (e.g., giving high ratings if two answers provide the same information.)As a workaround, BLV people check for common object detection errors Huh et al. (2023b) or conflicting information among multiple model outputs Huh et al. (2023a).We do not evaluate the factuality of all the information presented in the long-form answer, which might also contain hallucinations. Future work can leverage our role analysis to identify appropriate sources for the factuality evaluation of the entire answer. 


Decision: This paper introduces VizWiz-LF, an extension of VizWiz with 600 questions and long answers.  VizWiz-LF (like its predecessor) was created with the goal of helping blind or low vision users learn what is in images. This is an important area for generative AI. The dataset seems to have been thoughtfully created and motivated. Some concerns were raised during the reviewing process about the use of model-generated outputs in parts of the dataset, but the authors addressed these concerns to the satisfaction of at least most of the reviewers. The paper is also well-written, and it includes a substantial number of experimental evaluations.

Reviews: review: paper_summary: paper_summary: The paper presents a dataset, called VizWiz-LF, which is an extension of the VizWiz dataset with long form answers. VizWiz-LF consists of 1.8k long-form answers with a total of 5.2k sentences collected from human expert describers and two vision language models (GPT-4V and Gemini). The analysis suggested that long form answers contain explanations and suggestions which could be useful for BLV and sighted people. The paper evaluated six VQA models to check the ability of abstaining from answering unanswerable questions.
strength_weakness: strength_weakness: This is a straight-forward dataset paper that extends a previously proposed dataset. Some of the annotations are model based, therefore, there may be some noise/inaccuracies, however, the human evaluation shows such chances are quite low. Overall, this is a good work with a very limited novelty.
questions: questions: The reliability of the dataset largely depends on the accuracy of the task performance of the crowd workers. I may have missed but does the paper provide details about the crowd workers? What are their background, how they are compensated, and how much time they took to finish their tasks?
paper_summary: paper_summary: This paper presents VizWiz-LF, a VQA dataset consisting of long-form answers to visual questions posed by blind and low vision (BLV) users. VizWiz-LF contains 1.8k long-form answers with a total of 5.2k sentences collected from human expert describers and two vision language models (GPT-4V and Gemini). The authors show that long-form answers contain information beyond the answer such as explanations and suggestions, and also conduct a human evaluation of the long-form answers with BLV and sighted people. Overall, the paper presents a useful benchmark, and has insightful findings.

[Update: Thank you for the clarification! My concerns have been sufficiently addressed by the author rebuttal, and I'm increasing my score.]
strength_weakness: strength_weakness: * It isn’t clear to me why answers in the dataset were elicited from GPT-4V and Gemini as well as human experts? Using GPT-4V and Gemini answers as ground-truths could potentially introduce some biases and could lead to quality issues in the dataset.
* Related to this concern, is there only one human-elicited long form answer per visual question? My concern is that this will make it difficult to evaluate other models with only one ground-truth for comparison.
* 600 questions within the dataset is on the smaller side, although I believe this shortcoming is made up by the detailed, in-depth qualitative analysis and human studies.
* Only two closed-source models (Gemini and GPT-4V) were evaluated on the dataset, and the six open-source models were evaluated only on the task of abstention, and not on generating long-form answers.
questions: questions: * Are the GPT-4V and Gemini answers intended to be used to evaluate other models on this dataset? If so, why not just compare model outputs to human ground-truth responses?
* Is there a reason why the six open-source models were not evaluated on the long-form generative VQA task?
paper_summary: paper_summary: The paper introduces VizWiz-LF, a new long-form visual question answering (VQA) dataset that focuses on visual questions posed by blind and low-vision (BLV) people. The datasets consists of 600 image-question pairs sampled from the VizWiz dataset. To ensure diversity of questions, the data is sampled from 4 question types, i.e., Identification, Description, Reading, and Others, which is based on the question taxonomy of VizWiz. Among the 600 image-question pairs, 230 are marked as unanswerable. For each image-question pair, 3 long-form answers are constructed -- one by human expert, one by GPT-4V, and the other by Gemini. The resulting answers from all annotations are more than 10x longer than the original VizWiz dataset. Each sentence in the long-form answers is annotated with functional roles (confirmation, answer, answer failure, auxiliary information, auxiliary information failure, explanation, suggestion, and miscelleneous) and information type (image content, image quality,  external information). To construct the annotation,  180 long-form answers are annotated by human through a three-way annotations achieving a high fleiss kappa score (0.76 for functional roles and 0.87 for information types). The rest of the data is annotated by GPT-4 through few-shot in-context prompt, which achieves F1-score of 0.74 compared to human F1-score of 0.79).

Using the generated data, the paper demonstrates that both human expert and GPT-4V produce diverse functional roles, while Gemini often onlty cover answer roles since the long-answer often only consists of a single sentence. Additionally, Gemini also rarely abstain to answer even when the question is unanswerable. Furthermore, the paper also shows that model-based approaches, i.e., Gemini and GPT-4V, contains a higher percentage of external information which does not appear on the image. Human evaluation is also conducted to measure the human preferences of the VizWizL-LF considering 4 factors including relevance. helpfulness, plausibility, and fluency. The human evaluation is conducted by 20 BLV and 20 sighted people. All evaluator groups favoured GPT-4V and human expert annotations over Gemini. While sighted people give low helpfulness score for Gemini due to incorrect information provided as an answer, BLV group rated high plausability for all annotations. An additional automatic evaluation using ROUGE, METEOR, and BERTScore is also conducted further showing the effectiveness of GPT-4V for generating long-form VQA. The paper further explore on abstention strategy which helps to the model to abstain from answering when the question is unanswerable.
strength_weakness: strength_weakness: - The results tables are difficult to follow with too many numbers and have no clear highlight of which part readers should focus on
- Section 6 feels rather detached from the other contents of the paper
- It is not unclear how long-form VQA and VizWiz-LF can be beneficial for blind and low-vision people since the human evaluation is only conducted on long-form answers with no comparison on short-answer VQA.
questions: questions: - Is there any human evaluation conducted to sighted and BLV people?
- The column gender and age should be swapped in Table 16 on Appendix A.3.1
paper_summary: paper_summary: VizWiz-LF presents a unique dataset of long-form answers from blind and low vision (BLV) users, shedding light on the discourse structure of such responses. It also highlights a discrepancy between model-generated and human-perceived correctness of answers to unanswerable questions. Additionally, the evaluation of VQA models provides insights for improving their performance and mitigating the risk of providing misleading answers.
strength_weakness: strength_weakness: - poor presentation: This article is written with poor logic. For example, why we should understand how humans evaluate long-form answers, and there is no obvious relation between paragraph 5 and paragraph 6. 
- The author needs to further clarify the difference between the BLV and traditional VQA dataset with long/short answers.
- Is any error in this sentence ‘’Vision language models can now generate long-form answers to questions about images – or, long-form visual question answers (LFVQA).’’
- What is the benefit of understanding the functional structure of long-form answers?
- Long-from answers from large models (such as GPT-4V) could introduce hallucinations and are not reliable.
- Human evaluation samples are limited.
- Human evaluation may not be reliable because people are subjective. Could authors provide the annotation detail (such as instructions for human evaluation)? 
- More human evaluation samples should be provided .

Comments: title: Thanks for your response
Reply to Author R
Thank you for the rebuttal
comment: Some of my concerns are addressed, but other concerns still are not addressed (such as hallucinations). Therefore, I tend to improve my score mildly.
Thank you for the clarification! My concerns have been addressed, and I'm increasing my score.
Thank you for addressing the comments. After careful consideration, I have decided to retain my score.
Thank you for addressing all the questions and comments. Upon considering all the reviews and authors' responses, I have decided to keep my current score.