Rephrase, Augment, Reason: Visual Grounding of Questions for Vision-Language Models
Authors: Archiki Prasad
Elias Stengel-Eskin
Mohit Bansal
Venue: ICLR 2024 poster (2024)
Keywords: visual question answering
zero-shot
large vision language models
visual reasoning
underspecification
grounding language to vision
Primary area: representation learning for computer vision, audio, language, and other modalities

Abstract: An increasing number of vision-language tasks can be handled with little to no training, i.e., in a zero and few-shot manner, by marrying large language models (LLMs) to vision encoders, resulting in large vision-language models (LVLMs). While this has huge upsides, such as not requiring training data or custom architectures, how an input is presented to an LVLM can have a major impact on zero-shot model performance. In particular, inputs phrased in an underspecified way can result in incorrect answers due to factors like missing visual information, complex implicit reasoning, or linguistic ambiguity. Therefore, adding visually-grounded information to the input as a preemptive clarification should improve model performance by reducing underspecification, e.g., by localizing objects and disambiguating references. Similarly, in the VQA setting, changing the way questions are framed can make them easier for models to answer. To this end, we present **Rep**hrase, **A**ugment and **Re**ason (RepARe), a gradient-free framework that extracts salient details about the image using the underlying LVLM as a captioner and reasoner, in order to propose modifications to the original question. We then use the LVLM’s confidence over a generated answer as an unsupervised scoring function to select the rephrased question most likely to improve zero-shot performance. Focusing on three visual question answering tasks, we show that RepARe can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%, and 7.94% points increase on A-OKVQA, and VizWiz respectively. Additionally, we find that using gold answers for oracle question candidate selection achieves a substantial gain in VQA accuracy by up to 14.41%. Through extensive analysis, we demonstrate that outputs from RepARe increase syntactic complexity, and effectively utilize vision-language interaction and the frozen LLM.

Published as a conference paper at ICLR 2024
REPHRASE, AUGMENT, REASON: VISUAL GROUND- ING OF QUESTIONS FOR VISION-LANGUAGE MODELS
Archiki Prasad Elias Stengel-Eskin Eskin Department of Computer Science University of North Carolina Chapel Hill farchiki, esteng, mbansal |ecs. enc
Mohit Bansal
ABSTRACT
An increasing number of vision-language tasks can be handled with little to no training, i.e., in zero and few-shot manner, by marrying large language models (LLMMs) to vision encoders, resulting large vision-language models (Ls)..s)). While this has huge upsides, such as not requiring training data or custom architec- tures, how an input is presented to an LVLM can have a major impact on zero-shot model performance. In particular, inputs phrased in an underspecified way can re- sult in incorrect answers due to factors like missing visual information, complex implicit reasoning, linguistic ambiguity. Therefore, adding risually-grounde information to the input as a preemptive clarification should improve model per- formance by reducing underspecification, e.g.. by localizing objects and disam- biguating references. Similarly, in the VQA setting, changing the way questions are framed can make them easier for models to answer. To this end, we present Rephrase, Augment and Reason (REPARE), gradient-free framework that ex- tracts salient details about the image using the underlying LVLM as a captioner and reasoner, in order to propose modifications to the original question. We then use the LVLM's confidence over a generated answer as an unsupervised scoring function to select the rephrased question most likely to improve zero-shot perfor- mance. Focusing on three visual question answering tasks, we show that REPARE Re can result in a 3.85% (absolute) increase in zero-shot accuracy on VQAv2, 6.41%. and 7.94% points increase on VQQ,VQA, and VizWiz respectively. Additionally, we find that using gold answers for oracle question candidate selection achieves a substantial gain in VQA accuracy by up to 14.41% %. Through extensive analy- sis, we demonstrate that outputs from REPARE increase syntactic complexity, and effectively utilize vision-language interaction and the frozen LLM.
INTRODUCTION AND MOTIVATION
Recent advancements in foundational vision-language (VL.) models such as GPT-4 (OpenAI, 2023), BLIP-2 (Li et al., 2023), and Flamingo (Alayrac et al., 2022) have enabled tremendous strides in visual understanding tasks (Gan et al., 2022; Zhang et al., 2023a; Yin et al., 2023). Similar to large language models (LLMs) in the text domain (Ouyang et al., 2022; Chowdhery et al., 2022: Touvron et al., 2023, inter alia), these large vision-language models (LVLMs) can be guided through well- designed input prompts perform tasks without munitun i.e., in zero- and few-shot fashion. This is a powerful capability, allowing models to be applied to rision-language tasks without access to large annotated training datasets. In this setting, the prompt's phrasing becomes crucial model performance (Webson & Pavlick, 2021; Mishra et al., 2022; Prasad et al., 2023). Further contributing to the challenge of zero-shot tasks underspecification, tasks. In this work, we use visual question answering (VQA) as representative VL task and seek to improve zero-shot model performance by addressing underspecifiication In VQA, underspecified questions might provide inadequate information for an interlocutor to understand their intended meanings and answer them correctly (Pezzelle, 2023; Zhu et al., 2023a; Hu al., 2022).
common phenomenon in various VL
Our code is puplicly available:
thwp///tgthhht..c https://github..mmaarchhh// htt..//caaac.i///peeeee..

---

Published as a conference paper at ICLR 2024
LVLM
Orig. What the relationship between these two people?
Orig. A:
two tennis players
VQA Prompt
Players
csption?
position wer. net
Mod. Q': What is the relationship between the two tennis players on
rationale?
RRepARe)
Opponents
RepARe
either side of the net ?
Mod. Q': Where these crowded people the side? A' RRepARe):: Market
Mod. Q': Which type of shorts the person riding the skateboard wearing? A ARRepARe): Khaki
Orig. Q: Where
RepARe
Orig. Q: What kind of shorts is he wearing?
RepARe
at?
Figure I: Top: The original question (in A-OKVQA) lacks information about implicit reasoning, leading to an incorrect answer. REPARE interacts with the LVLM to extract attributes like "tennis players" and "position w.r. net" that are key to answering the question correctly. Adding these modifiers the question elicits the correct response from LVLM. Bottom: Underspecified questions from A-OKVQA (left) and VQAv2 (right) datasets along with REPARE outputs.
Underspecification in VL tasks like VQA can manifest in several ways, leading to incorrect model predictions. Firstly, language lacking in visual details (i.e., questions underspecified w.r.t. image) can make it harder for models align text and visual features (Pezzelle, 2023). For example, in Fig. (bottom-left), "we" is not grounded to the image. Furthermore, abstract questions often require complex reasoning or external world knowledge that may not be present in the model or at least may be hard to access; in other words, the question is underspecified w.z. the world (Marino et al.,, 2019; Schwenk et al., 2022). For instance, in Fig. (top), the spatial arrangement of players on opposite sides suggests they are opponents. Explicitly referencing "the ner" could help the model access this commonsense knowledge. While LVLMS might still rank "opponent" highly in their predictions for the original question, the rephrased question more clearly specifies the intent of the inquiry, leading to "opponent" as the generated response. Finally, some questions are inherently ambiguous, with multiple valid answers. Even if the model is capable of generating all possible responses, it is not clear which one is intended (Bhattacharya et al., 2019; Stengel-Eskin et al., 2023), i.e., the question is underspecified w.r.f. intended meaning. For example in Fig. (bottom- right) there are two men, which results in multiple possible referents for "he". Building upon prior research in textual question reframing (Dong al., 2017; Majumder et al., 2021; Pyatkin et al., 2023), we hypothesize that making some of the details needed to answer the question more explicit could improve model performance.
There are several paths to addressing the challenges posed by underspecificattio. One approach in- volves additional VL pretraining to better align underspecified text to images as well as to enhance the LVLM's s internal world model, enabling it to decifer underspecified questions in human- ways. However, scaling up VL pretraining can be prohibitively expensive (Alayrac et al., 2022; Driess et al., 2023). Note that, in addition to the expense of finetuning, it could be that underspec- ification in the training data leads to continued subpar performance on underspecified questions. Another option is acquiring additional data information from the user, such as clarifications. This strategy is infeasible for most standard VL benchmarks as they are static datasets (Kiela et al., 2021; Sheng et al., 2021). Furthermore. clarification interactions with users are cine-consuming and costly. Thus, our method preemptively incorporates clarifications to reduce ambiguity. emphasize relevant visual details, and suggest reasoning steps, thereby, automatically improving the LVL.M's VQA per- formance without the need for human intervention. Moreover, using preemptive clarifications could also hold value VL dialogue systems, where users prefer concise interactions but often pose vague questions. This approach has several advantages: (i) it allows for flexible, gradient-free framework to improve the performance of existing LVLMs without the need for additional pretraining or man- ual annotations; (ii) text-based edits are human readable i.e., are relevant and consistent with the question's 's intent; and (iii) crucially, our method harnesses the asymmetric strength of most existing LVLMs,, whose L.M components typically have far more
verify that added details
2

---

Published as a conference paper at ICLR 2024
pacity and pre- pre-training data than the vision component," 2 and which often have strong reasoning and planning abilities on multimodal data (Wei et al., 2022b; Brohan et al., 2023; Guo et al., 2023). In other words, by preemptively rephrasing questions, we can align more closely with the strengths of existing LVLMs, making rich visual information from the image easier to access.
In Fig. (top), we illustrate a high level how rephrasing and modifying questions based on the im- age improves model predictions. Note that our method does not have any access to the gold answer, using model confidence select question. While the original question elicits generic response, pinpointing the "temiis players" and emphasizing their positions "relative to the ner" helps the model answer correctly. These modifications are obtained via self-interaction with the LVLM to get more information about the entities in the question as well as other salient objects from -geer-geee rationales and captions. To this end, we introduce Rephrase, Augment and Reason (REPARF), a gradient-free, instance-level language adaptation framework to address underspecification. Broadly, REPARE consists of two stages: question rephrasing and augmentation, followed by question selec- tion. First, we identify salient entities from the question and generate rationales as well as captions. These features help incorporate visually grounded information into the question. Conditioned on this information, we sample modified question candidates including the original question. In the next stage, we utilize a confidence-baid bbased selection function to choose the most promising candidate, assuming that questions leading higher-confidence answers are easier for the model to answer, and thus more likely to be correct. The overall pipeline is illustrated in Fig. 2.
Empirically, we show that REPARE improves zero-shot VQA performance by up to 3.85%, 6.41%, and 7.94% on the VQAv2 (Goyal et al., 2017), A A-OKVQA (Schwenk et al., 2022), and VizWiz (Gu- rari et al., 2018) datasets, respectively using LVLMs including BLIP-2 (Li et al., 2023), MiniGPT- 4 (Zhu et al., 2023b), and VLa (Liu et al., 2023a) models in Sec. 4. Note that all percent- ages we report in this paper are absolute improvements. We further demonstrate the capabilities of REPARE in an oracle setting, establishing an ppuer-bound performance increase of up to 9.84%, 14.41 %, and 20.09% on VQAv2, A-OKVQA, and VizWiz. tasks, respectively. We extensively evalu- ate our design choices in Sec. 4.1 and quantitatively show the importance of incorporating visual in- formation to address underspecification. as done in REPARE, compared to paraphrasing in Sec. 4.2. We analyze REPARE'S outputs using geguisiicalllinnoomee tance (Gibson et al., 2000) and idea density (Boschi et al., 2017). This reveals that the resulting questions are indeed less underspecified, i..., more complex (see Sec. 4.3). Finally, in See. 4.4, verify that questions from REPARE make better use of existing LVLMs by leveraging the strength of the LLM while still benefitting from the image. In summary, our contributions include:
metrics like average dependency dis-
We propose REPA RF, a novel zero-shot pipeline that interacts with LVLMS to modify underspec- ified questions by extracting and fusing information from keywords, rationales, and captions. This grounds questions in the image and commonsense knowledge while also making them less am- biguous, preemptively clarifying them to address underspecification without any human feedback. We empirically demonstrate that REPARE boosts zero-shot performance on three standard VQA benchmarks for a collection of LVLMs varying in model architecture, size and VL pretraining by up to 7. 7.94%. Our oracle results suggest that we can obtain as high as 20.09% increase in zero-shot VQA accuracy solely by modifying the question.
Extensive analysis shows that REPARE enhances question complexity via semantic modifications, outperforms paraphrasing, and harnesses LVLM's strengths with simple yet effective modules.
RELATED WORK
Large Vision-Languee Models. Significant strides have been made in jointly processing lan- guage and images, especially in visual question answering. VQA (Antol et al., 2015; Goyal et al., 2017: Hudson & Manning, 2019; Johnson et al., 2017) has become a benchmark task for VL mod- els. Recent methods address VQA as zero- and few-shot learning task. These approaches can be categorized into two groups: (i) those relying on continuous image representations (Alayrae et al., 2022; Tsimpoukelli et al., ;0 Zhu et al., 2023b; Liu et al., 2023b; Li et al., 2023, inter alia); and (ii) those extracting linguistic information such as captions from images (c.g.. Yang et al., 2022;
'E.g.. BLIP-2ptas xi (Li et al., 2023) consists of a ViT vision encoder, a Q-former fusion module, and a Flan-T5 L.M with 1B, 0.11B, 3B and parameters respectively. i.c., the I.I is ~33 times more powerful. Note that this is not a permanent feature of such models, and future models could have equally-sized components.
3

---

Published as a conference paper at ICLR 2024
G. What period of the day does this photo roflect?
Extract
g): What period of the day does this reflact?
ay Marning P(o L.ee) :0.61
Keywords
Collect Visual Details
Sentence Fusion
Generate Rationales (key objects)
Vown Mig
Ms. What period of the day builting with clack fower
Gy Evening P(l...)) :0.82
choc lal kmn
Cloudy elack click.
Culdkig clook tover...
LVLM
Caption Generation
dy. What period of cloudy;
āg Allemoon (.)
day with some sunlight it?
I. Image
(a), Extracting Visual Details:
K(). Augmenting Questions
II. Scoring
Figure 2: Schematic of REPARE for an image requiring implicit reasoning from A-OKVQA. We
first extract keywords, captions, and rationales from the image conditioned on the question, which are used to identify important objects (e.g.. day and clock). We query an LVLM about these objects to collect visual details in I(a), that are fused into the original question to produce, in this case, 3 candidates (I(b)). Lastly, we score and select from candidates using LVLM's 's answer confidence (II).
Changpinyo al., 2022; Guo et al., 2023; Berrios al., 2023). Given the higher performance of projection-based models on VQA, we focus our efforts on the former class.
LLMs can be used for multimodal chain-of-thought (CoT) reasoning (Zhang et al., 2023c); while we use forms of CoT in REPARE, the overall framework differs from CoT. Firstly, CoT is typi- cally open-ended, whereas we follow a principled set of modules, which we validate individually in Sec. 4.1. Secondly, while CoT is generally useful on large models over 100 billion parameters (Magister et al., 2023; Wei al., 2022a), REPARE can also applied to LLMs like Flan-T5 (which do not generally benefit from CоT) without any modifications to the model (Wei et al., 2022a).
Underspecification and Ambiguity. Underspecification and ambiguity are well-studied within both NLP and linguistics (Schutze, 1995; Futeral et al., 2022; Berzak ct al., 2015; Min et al., 2020; Rasmussen & Schuler, 2020). In the multimodal context, Pezzelle (2023) emphasizes underspeci- fication as a significant source of errors in VL tasks we develop REPARE as a concrete solution to address underspecification by adding visual information. Similarly, Bhattacharya find underspecification to be a factor contributing to annotator disagreement in VQA, while Stengel- Eskin al. (2023) focus on ambiguity in VQA and propose rephrasing method for disambiguation. Unlike REPARE, their method relies on access to gold answers and involves further model training.
al. (2019)
Prompt Editing. Both LLMs and LVLMs suffer from inherent randomness and sensitivity to choice of training examples, instructions, and prompt template (Zhao et al., 2021; Min et al., 2022; Lu et al., 2022; Awal et al., 2023) in zero-shot and few-shot settings. As result, several works aim to search for better prompts via gradient- based (Shin et al., 2020; Gao et al., 2021; Jia et al., 2022; Khattak et al., 2023)
gradient-free methods (Sun et al., 2022: Deng et al., 2022; Prasad et al., 2023; Zhang et al., 2023b). However, existing gradient-bart based methods can be computation- ally expensive (Sung et al., 2022a), are infeasible for gated models accessible only via APIs, and are often uninterpretable (Khashabi et al., 2022). On the other hand, existing gradient-free methods primarily designed for language-only models and select the best prompt based on scores (Liu et al., 2022; Prasad et al., 2023) or learned policy (Deng et al., 2022: Zhang et al., 2023b) using labeled training set. In contrast, REPARE directly addresses underspecifiiation in VL tasks by making targeted edits to the question using gradient-free, instance-level edits without any train set.
METHODOLOGY
In this section, we describe the overall pipeline of our method: Rephrase, Augment and Reason (REPARE). Broadly, REPARE consists of two stages: (I) generating rephrased and augmented question candidates and (II) candidate selection. The first stage yields n modified question can- didates, incorporating visual information, and information from rationales using the underlying LVLM. We then use a selection module to identify the best candidate. Note that in all cases, se- lected questions should preserve the intent of the original question while making it easier for the model to answer. Fig. 2 provides a detailed illustration of our REPA RE pipeline in action.
3.1 GENERATING REPHRASED AND AUGMENTED QUESTION CANDIDATES
Stage I(a): Extracting Visual Details from Captions and Rationales. To augment the question with pertinent visually- grounded details, we focus on extracting all relevant information from the image, conditioned the question.
4

---

Published as a conference paper at ICLR 2024
(i) Salient Question Entities: Intuitively, entities mentioned in the question provide vital infor- mation about the expected answer. To implement extract key entities from the question, we use off-the-shelf keyword extraction system (Rose et al., 2010). For instance, it extracts "day"" from the question in Fig. 2.
(ii) Information from Rationales: Answering complex questions can often require world knowl- edge and implicit reasoning skills (Schwenk et al., 2022). To incorporate this, we sample rationales from the LVLM, which we use identify relevant objects and features in the im- age (Chowdhery et al., 2022; Zhang et al., 2023c). This allows REPARE to identify what features might be worth focusing on.' 3 For instance, in Fig. 2. the model might extract the clock on the top of the building as an important feature in determining the time of day, based on a rationale like "Clocks can tell time, so read the clock to determine the time of day."
(iii) General Information from Image Captions: Questions may be underspecified to the extent that they do not contain any salient entities (c.g.. "where are we at" in Fig. 1). Thus, we also prompt the LVLM to generate detailed caption for the image. This allows us to capitalize on LVLMs' asymmetric abilities: they excel at image captioning (Alayrac et al., 2022; Tsim- poukelli et al., 2021; Zhu et al., 2023b), and can generate detailed captions (Zhu et al., 2023b; Xie et al., 2022). For example, in Fig. 2, the captioning model might generate a caption like "A tall, stone building with a clock tower on top on a cloudy day".
After identifying salient objects and entities from (i) and (ii), we prompt the LVLM to obtain perti- nent details about them based on the image. We add this list to the image captions from (iii) to get the input for for the next stage. We describe the implementation of this stage in detail in Appendix A.3.
Stage I(b): Rephrasing and Augmenting the Question. Drawing on work in sentence fu- sion (Geva al., 2019; Lebanoff et al., 2020), we leverage the frozen LLM component of the LVLM to incorporate fine- fine-graines grained details into the question. We combine all the extracted details into a single prompt, and generate
modified question candidates, yielding a total of candi- dates including the original question (see stage 1(b) in Fig. 2). To prevent significant alteration of the question's 's meaning (especially for yes/no questions), we use an off-the-shelf natural language inference model (Laurer et al., 2022) to discard any candidates that contradict the original question. After generating n question candidates, we prompt the LVLM to answer each question, leading to n question-answer (QA) pairs. We use
5 as default in all our experiments and discuss the impact of increasing n as well as using the full LVLM for sentence fusion in Appendix A.5. Note that all our prompts used within REPARE or for VQA do not contain any annotated examples from any VQA dataset (zero-shot setting). Further details and all prompts can be found in Appendix A.2.
3.2 QUESTION SELECTION
To select the final QA pair from I(b), REPARE requires way of scoring the QA candidates generated using the modules above, in order to choose the QA pair most likely to improve accuracy.
Stage I:: Confidence- Selection. As discussed in Sec. 2, most prompt search methods re- quire a labeled dataset to learn a scoring model or a selection policy. In our setting, we perform instance-leved edits, meaning that such a supervised scoring scheme would require access to addi- tional annotated data. Therefore, consistent with Liu et al. (2021), at inference time we compute an unsupervised score by utilizing the L.LM's ability to self-assess the quality of its generations (Rae al., 2021; Srivastava et al., 2023; Kadavath et al., 2022).1 Following Kadavath et al. (2022), use the LVLM's confidence in generating proposed answer à conditioned on the image and question candidate gi to select candidate o' (and its corresponding answer à') for subsequent evaluation:
score(q,, â,) = ( q:); q' à' argmax(score(qg. a:))
#E[1.n]
Oracle Setting. As an -oundd we also explore an 'oracle' setting in which we have access to the (gold) annotated answer from the dataset. In this setting, we select the candidate that yields the
"Note that in the scope of this work, we do not address the veracity and utility of generated rationales which is relatively harder to judge using the same underlying model (Pruthi et al., 2022; Saha et al., 2023). Moreover, for of the LVLMs (MiniGPT-44 using larger and more powerful LLMs, we prompt the model to generate an explanation in the common zero-shot VQA prompt for generating answers. Refer Appendix A.2 for details.
'While past work has found LLMs to be overconfident on variety of tasks (Mielke et al., 2022; Lin et al., 2022; Zhou et al., 2023; Stengel-Eskin & Van Durme, 2023), this does not impact our results, as we choose q' based on the I.IM's relative confidence. For more details, we refer readers to Appendix A.5.
5

---

Published as a conference paper at ICLR 2024
correct answer (in case of ties, perform random selection). This gives us the maximum possible performance of REPARE for a fixed number of candidate questions n, discussed further in Sec. 4.
3.3 EXPERIMENTAL SETUP
Vision Language Models. We use three recent state-of-the-art LVL.s: BLIP-2 (Li al., 2023), MiniGPT-4 (Zhu et al., 2023b), and LLaVA-1.5 (Liu et al., 2023a). At a high level, the model archi- tecture comprises of an image encoder (Radford et al., 2021; Fang et al., 2023) and an LLM (Chung et al., 2022; Chiang et al., 2023) (both frozen) connected by a relatively small trained transformer model called the Q-former (Li et al., 2023). The Q-former acts as a bridge, facilitating information flow between the image encoder and the LLM, resembling et al., 2022b). Beginning with image-to-text pre-training, the Q-former extracts key visual details and then connects to the LLM using fulll-cconneeee layer to project query embeddings into the em- bedding space of the LLM. Note that while BLIP-2 uses an encoder-decoder beaedd LLM (Flan-TS) TS), MiniGPT-4 and VL. use Vicuna with decoder-only architecture (details in Appendix A.3).
adapter (Houlsby et al., 2019; Sung
VQA Datasets and Metrics. We use the VQAv2 dataset (Goyal et al., 2017) for general visual un- derstanding. To specifically capture underspecification due to lack of reasoning or world-knowledge, we use the A-OKVQA dataset (Schwenk et al., 2022) containing image-question pairs that require broader commonsense and world knowledge to answer. A-OKVQA has two settings: (i) directly generating the answer (direct), and (ii) 4-way multiple choice (MC). Since the test sets of these benchmarks are not publicly available, we report performance on the validation sets (unless men- tioned otherwise). Lastly, we also evaluate on the challenging VizWiz benchmark (Gurari et al., 2018) consisting of life information-sede seeking questions about (often low-quality) images sourced from -imaaiee people. While developing REPARE, we sample a small set of data points from the train set of the datasets to form our dev set. the "direct answer" setting, we use the standard soft VQA evaluation metric for VQAv2, VizWiz, and A-OKVQA (Antol et al., 2015). In A-OKVQA's MC setting, we use accuracy. See Appendix A.I for further dataset details.
RESULTS AND ANALYSIS
In this section, we present the results of our experiments. First, we establish the effectiveness of the REPARE framework in See. 4.1. Then, in See. 4.2, we quantitatively distinguish REPARE from simply paraphrasing the question. Furthermore, we provide quantitative analysis of outputs from REPARE, addressing semantic complexity (in See. 4.3). Lastly, in Sec. 4.4, we show that REPARE leverages the asymmetric strength of the LLM in an LVLM, allowing the LLM to perform more of the task without eliminating the need for the image. Note that all improvements this paper are reported absolute percentage increase.
4.1 OVERALL EFFECTIVENESS OF REPARE
Main Results. Our main results are presented in Table 1. When compared to the original ques- tions, using questions after applying REPARE increases the overall zero-shot accuracy of BLIP-2 by 3.85%. of MiniGPT-4 by up to 3.02%, and that of LLa by 1.14% on VQAv2. On the A- OKVQA dataset, where answering the question may require a combination of world knowledge and reasoning skills, we show that REPARE improves the zero-shot performance of BLIP-2, MiniGPT. 4, and 1.5 models by up to 5.47%, 6.41%, and 3.63% respectively. when directly generating the answer. In the multiple-choice setting with the MiniGPT4viamn 7B model, this improvement can be as high 21.54%. Moreover, on the challenging VizWiz dataset, REPARE improves perfor- mance by 7.94%, 3.46%, and 2.39% points with MiniGPT-4, BLIP-2, and LLa models. Fur- thermore, using gold answers in the oracle setting. we establish empirical upper bounds for REPARE in Table 1. On BLIP-2, 2,, REPARE can yield up to 9.84% across both datasets, while using MiniGPT- 4, we can obtain a maximum oracle improvement of 14.41% and 33.94% the A-OKVOL VQQ dataset in the direct and multiple-choice settings, respectively. Lastly, REPARE in oracle setting yields up to 7.61% accuracy improvements on the VizWiz dataset. This demonstrates REPARE's efficacy on VQA datasets with different LVLM architectures varying in size and underlying LLM.
Design Ablations. In Sec. 3, we described various design choices made to develop REPARE. In Table 2, we evaluate the effectiveness of different components within REPARE RE on our dev splits.
We refer readers Sec. for broader discussion of asymmetric strength and ability.

---

Published as a conference paper at ICLR 2024
Method
VQAv2
A-OKVQA
VizWiz
Overall
Y/N
Num.
Other
Direct
MC
Overall
MiniGPT-4 7B
51.47
71.03
27.13
42.75
27.51
41.66
29.87
-REPARE
54.49 1.44
75.77+ 201
32.58 2.29
43.79 0.72
33.23 2:31
63,20 5.64
37.81 4.26
RREPARE (Oracle)
59.66 293
86.56
6.12
38.86 4.57
44.32 L0S
41.92 4.86
75.60 7.36
50.47 5.32
MiniGPT-4vicsa 138
61.98
82.76
39.83
51.71
41.53
56.41
44.18
REPARR
64.03.+ 1.26
83,54.
1.12
47.50+ 3.17
53.34+
0.98
47.94 2:25
62.18 4.48
51.33 3.79
+ REPPARE (Oracle)
68.35 3.62
89.33 4:28
51.41 5.67
56.30 2.27
54.20 5.18
80.67 8.26
64.27
BIIIP-2Ft T5.at
62.58
83,99
38.63
52.91
41,86
73.89
59.63
+ REEPARE
66,43 1.21
89.94, 1.27
48,56 2.85
52.94 0.61
44.87+ 34
77.20 144
62.38 1,14
REPPARE (Oracle)
72.42
94.26
50.67
61.25
49.20 2.38
81.14
65.20
BLIP-2Fia T5-sd
65.08
85.15
39.91
56.19
41.86
76.59
62.81
REPPARE
68.92. 1.36
90.54 3.18
43.30 2.41
58.96
47.33 56
79.23
66.27 1.87
REPARR (Oracle)
74.05 1.26
94.56+ 4.37
54.40 4.72
63.36 1884
55.67 2.19
82.80 2.36
70.13 2.49
ALaVA-1.5vioma
76.21
91.83
58.27
68.85
62.56
77.38
57.07
REEPARE
77.35 + 0.42
92.64
0.34
59.92
0.51
70.12 0.76
66.19
1.53
78.21 0.37
59.46 0.92
REPARE (Oracle)
79.84
1,03
94.39 1: 094
62.07
1.25
73.28 +
1.31
70.17 2.49
80.75 1.27
62.48 1.61
Table 1: Comparison of baseline shot accuracy (%) and REPARE on VQAv2, A-OKVQA and VizWiz. We run REPARE for n 5 and average performance across 3 random seeds to account for randomness in generating question candidates in Sec. 3.1. We highlight the oracle performance with REPARE using gold answers. The overall best numbers for each dataset are in bold, and the highest numbers for each model are underlined.
Importance of Rationales, Captions, and Question Entities: We measure the utility of details about objects mentioned in the: (i) original question, (ii) image caption, and (iii) rationales, by re-running REPARE with BLIP-2 using all but one type of object descriptions. From Table 2, we observe that excluding rationales, captions, or question entities adversely impacts zero-shot performance, with the largest drop in accuracy occurring when rationales are not utilized.
Impact of Removing Visual Tokens during Fusion: Next, we explore the impact of including visual tokens, projected onto the LM, in augmenting the question with visual details in Stage I(b). This involves performing the same sentence fusion task using the entire LVLM, while retaining the image embedding in the input to the frozen LM (refer to Table 2). Our findings reveal that the image embedding can serve as a distraction the language model when rephrasing the question, resulting in up to 3.1 point drop in overall accuracy (see qualitative examples in Appendix A.5). Design of Scoring Function: Lastly, we examine our scoring function described in Sec. 3.2. To ablate the scoring method, we run REPARE but with candidates based on the likelihood of the question alone, i.c. score(q,) = (L instead of score(q;, â.) = PLYLM (á. I, q:). Table shows that using question likelihood instead of the answer confidence yields small drop in the downstream performance by at least 1.09% (further ablations in Appendix A.5).
4.2 REPARE ADDS SEMANTIC INFORMATION TO ADDRESS UNDERSPECIFICATION
Comparison with Paraphrasing in Oracle Setting. Following past work leveraging para- phrases to improve QA (Dong et al., 2017), we experiment with a paraphrastic bascline, where we simply paraphrase the question using Pegasus, strong off-the-shelf model (Zhang et al., 2020).
Table 3 shows that paraphrasing the question leads to major improvements over the zero-shot setting under oracle selection (described in Sec. 3.2). For VQAv2, BLIP-2's performance increases from 62.58% to 70.99% and for A-OKVQA it improves from 73. 89% to 79.91% % in the multiple choice setting. This indicates that BLIP-2 and its underlying LLM, Flan-T5 are brittle the phrasing of the question, i.c., without altering the information or meaning of the question, a paraphrased question candidate may yield a higher VQA score (Webson & Pavlick, 2021).
Comparison with Paraphrasing during Inference. If the modifications from REPARE were purely cosmetic rewrites, then REPARE and a paraphrastic baseline should have roughly the same performance during inference. Table 3 demonstrates that selecting from paraphrased question didates without access to gold answers (oracle) presents a challenge. In fact, in 2 out of 3 settings, opting for paraphrased questions results in lower performance compared to using the original ques-
7

---

Published as a conference paper at ICLR 2024
Method
VQAv2 A-OKVQA
Method
VQAv2
A-OKVQA
REPARE
67.28
45.01
Overall
Direct
MC
w/o Rationales
64.57
43.15
Baseline (BLIP-2)
62.58
41.86
73.89
w/o Caption
66.04
44.36
Paraphrase Oracle
70,99
46,66
79,91
w/o Question Entity
65.89
44.62
REPARE Oracle
72.42
49.24
81.14
w/ Embeddings in Fusion
63.18
42.38
Paraphrase Selection
62,91
40,23
73.57
w/ score (v
65.47
43.92
REPARE Selection
66.43
44.87
77.20
Table 2: Ablation of design choices REPARE
Table 3: Comparison of REPARE (using BLIP-
using BLIP-2 on our dev splits (direct answers).
2) with paraphrasing questions in the oracle set- ting and unsupervised candidate selection.
tions by up to 1.63%. Therefore, although some paraphrased questions may elicit correct answers, choosing them solely based on the model's confidence yields poor results. In contrast, questions generated by REPARE show distinct pattern: not only do these questions outperform paraphrased questions in the oracle setting, but they are also more easily chosen by the unsupervised scoring function. This indicates that incorporating additional semantic information from both images and rationales in REPARE simultancously makes questions easier to answer as well as easier to select.
4.3 ANALYSIS OF INCREASED COMPLEXITY REPARE'S QUESTIONS
In Sec. 1, we highlight underspecification as source of errors in VL tasks like VQA. Table 1, we empirically show that REPARE enhances VQA accuracy across datasets and LVLM architectures. Here, we analyze the questions generated by REPARE and compare them against original questions to confirm that the rephrased questions are in fact more complex, i.e., less underspecified. We present quantiative results from two complexity metrics; see Appendix A.4 for qualitative examples.
Complexity Metrics. Qualitatively, we find that REPARE questions have increased syntac- tic and semantic complexity (cf. Table 6). We quantify this with two common complex- ity metrics: average dependency distance (ADD) and idea density (ID), implemented using BlaBla toolkit (Shivkumar et al., 2020) and Stanza (Qi et al., 2020). Average Dependency Distance (ADD) measures the syntactic complexity of sentences by calculating the average lin- ear distance between each token and its parent node in a syntactic parse. It is commonly used to measure syntactic complexity (Gibson al., 2000; Oya, 2011; Liu et al., 2017).
ADD ranges on [0, inf) with a higher score indicating more complexity. Idea Density (ID) is the sum of the number of verbs, adjectives, adverbs, prepositions, and conjunctions divided by the total number of words (Boschi al., 2017). It is commonly used as a measure of semantic complex- ity (Chand et al., 2012; Kemper, 1992). ID ranges between [0, 1] and higher scores indicate more complexity.
Dataset
Type
ADD
ID
Original
25.40
0.282
A-OKVQA
REPARE
32.81
0.299
Original
17.87
0.258
VQAv2
REPARE
29.52
0.296
Quantitative Complexity Analysis. Our quantitative re- sults can be seen in Table 4, where we compute ADD and ID for a subset of 100 instances from the official validation set. Here, we use BLIP-2 as the backbone for REPARE. Compared to the original questions, both complexity measures are higher for REPARE across models and datasets. This indicates that RE- PARE adds syntactic complexity and semantic content to the questions; which in turn suggests that the rephrased questions are less underspecified. For example, a REPARE question like "Why would you use this suitcase packed both sides?" from Table 6 has more modifiers than the original, "Why would you use this bug?", leading to a higher ID score. It also has a more complicated syn- tactic structure, with nested modifiers ("suitcase packed on both sides") ") leading higher ADD.
Table 4: Complexity measures for questions before and after REPARE.
4.4 REPARE LEVERAGES VL INTERACTION TO IMPROVE PERFORMANCE
We further explore the asymmetric strength hypothesis (discussed in Sec. 1), which could explain the improvements seen in Table Specifically, we examine how REPARE's addition of visual information to the question allows the LVLM'S LLM component to do more of the heavy lift- ing in the QA task. We test the performance of the original and REPARE questions without the image in the input, i.e., to what extent the constituent L.M alone can answer each question cor-
8

---

Published as a conference paper at ICLR 2024
rectly. If REPARE leverages the strength of the LLM well, we should expect the LVLM's LLM- only performance to increase when using REPARE. In Table 5, we evaluate this hypothesis us- ing BLIP-2 as the underlying model. First, we observe that the image is crucial to good perfor- mance; in all settings, BLIP-2's LLMM-only accuracy is quite low. Furthermore, REPARE questions improve in the LLM-only setting, indicating that modified questions take better advantage of the LLM's QA strength (cf. rows and 4), Note the substantial gap of 255 between settings with and without the image for REPARE (cf. rows 2 and 4), which indicates that the rephrased question is complementary to the image, i.e. that REPARE does not make questions trivial to answer with just an LLM. Finally, when us- ing just the L.M as the QA model, we find that adding the caption or extracted image details from stage I(a) of REPARE along with the orig- inal question improves performance over the original question alone; however, these details do not make up for the lack of the image (c.f. rows 2, 5, and 6 in Table 5). Thus, REPARE improves LVLM performance via both vision- language interaction and leveraging the LLM.
LVLM Setting
VQA22
A-OKVQA
Overall
Direct
MC
[1] Img + Orig. Q
60.29
41.72
72.56
[3] Img+ REPARE Q
67.28
45.01
78.43
[3] Orig. Q Q(I..M-only)
32.84
15.93
45.20
[4] REPARE Q (..L.-
40.53
20.33
54.21
[5] Caption Orig. Q
52.88
27.64
65.80
[6] REPARE I(a) + Orig. Q
54.31
30.27
66.60
Table 5: BLIP-2's LLM-only vs. full model per- formance on original and rephrased questions.
5 DISCUSSION AND CONCLUSION
Asymmetric Strength and Ability. As alluded to in Sec. 1, REPARE is based on two assump- tions about existing LVLMs. First, existing LVLMs have larger LLMs than vision components, i.e. they have asymmetric strength. Thus, moving some of the burden of the VQA task onto the LLM improves performance, as shown in Sec. 4.4. However, we also assume that the image is still helpful in answering the question-thi this is also borne out in Sec. 4.4, where visual information still improves the model. This differentiates our work from recent work like Berrios et al. (2023) and Hu et al. (2022) which translate an image into text descriptions and then apply a language-only model for VL tasks, i.e. also make use of asymmetric strength, but not of the image. Our work also holds greater promise for capturing fine- -rn-ggd visual details, which can be challenging to describe lin- guistically. Second, REPARE RE also relies on the asymmetric zero-shot abilities of individual LVLMs. While there is a large gap in QA between zero-shot LVLMs and fine-tuned, task-specific models, LVLMs are competitive at image captioning. We can use this to advantage. hamnsssing captions to improve the question. Similarly, while LVLMs may not be able to implicitly reason about the image during QA, their LLMs can extract useful rationales and fuse them into the question.
Redundancy and Language Bias. Qualitatively, much of the information in Table 6 may appear redundant to humans who can perceive the entire image and hone in details already given in the image. It is worth noting that past work has found that humans tend to over-specify when describing visual scenes (Ford & Olson, 1975; Sonnenschein, 1985; Pechmann, 1989; Koolen et al., 2011). In other words, redundancy in descriptions the model. VQA datasets can suffer from language bias, where many questions can be answered correctly without access to the image (Goyal al., 2017). The analysis in Sec, 4.4 indicates that REPARE questions have stronger LLM-only (i.c., language-only) performance. However, note that the information that gives REPARE questions their higher performance is extracted from the image using the same underlying LVLM. Thus, a comparison to a language-only bias here is not entirely accurate, since rephrased questions contain information soureed from the image.
questions is not uncommon, and may in fact benefit
Limitations. One limitation of our method is cost: rather than answering a question directly, we generate several question candidates and then select one. We note, however, that other multi-step approaches, including Shain-of--Thuuhh (Wei et al., 2022b), or exploration-guider reinforcement learning, and search methods also increase the number of tokens and inference steps, and our cost scales linearly in the number of candidates. Note also that while strategies like CoT can help with rationale-style reasoning in particular, they are harder to apply in most existing LVLMs (partly due to the size of LLM components in existing LVLMs, which is typically significantly less than 100B parameters). Addressing underspecification alone is not cure-all for solving VQA or broader visual understanding tasks. Underlying dataset issues, such as low-quality images and inaccurate human annotations (Bhattacharya et al., 2019), can still prevent models from achieving high accuracy.
9

---

Published as a conference paper at ICLR 2024
ETHICS STATEMENT
Instructions are a useful tool for conveying extrinsic information to LLMs. However, they can also be misused intentionally or unintentionally (Weidinger al., 2021) in order to alter model outputs to elicit harmful, biased and problematic content. Being based on LLMs, LVLMs are succeptible to similar misuse via targeted instructions or questions. The intended use of REPARE is to obtain modifed questions that work well for LVLMs and help improve model performance for the given in- stance without significanlty altering the intended meaning: this is orthogonal to whether the original question displays malicious intent, which is Additionally, in our work we use images and questions from VQA and A-OKVQA; these datasets have been vetted for quality in the past Lin al. (2014); Goyal et al. (2017); Schwenk et al. (2022) but inappropriate or offensive queries and images could remain since they are quite large. To mit- igate the risk of offensive, malicious, or inappropriate questions being generated by REPARE, we manually examined subsample of 250 generated outputs from REPARE and verified that the gen- erated questions do not display a malicious or offensive intent.
general issue that applies all L.LM//LVI.M
ACKNOWLEDGEMENTS
We thank Jaemin Cho, Peter Hase, Nithin Sivakumaran, David Wan, Jachong Yoon, and Shoubin Yu for their valuable feedback and inputs for the paper. This work was supported by DARPA ECOLE Program No. HR001 12390060, NSF-AI Engage Institute DRL-2112635, DARPA Ma- chine Commonsense (MCS) Grant N66001-19-2-4031, ARO Award W91 10220, and ONR Grant N09014-23-1-2566. The views contained in this article are those of the authors and not of the funding agency.
REFERENCES
Cean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716- 23736, 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zit- nick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2015.
Rabiul Awal, Le Zhang, and Aishwarya Agrawal. Investigating prompting techniques for zero-and few-shot visual question answering. arXiv preprint arXiv:2306. 2023.
William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, and Amanpreet Singh. Towards language models that can see: Computer vision through the lens of natural language. arXiv preprint arXiv:2306. 16410, 2023.
Yevgeni Berzak, Andrei Barbu, Daniel Harari, Boris Katz, and Shimon Ullman. Do you see what I mean? Visual resolution of linguistic ambiguities. In Proceedings of the 2015 Conference on Empirical Methods Natural Language Processing, pp. 1477-1487, 2015.
Nilavra Bhattacharya, Qing Li, and Danna Gurari. Why does a visual question have different swers? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4271-4280, 2019.
Veronica Boschi, Eleonora Catricala, Monica Consonni, Cristiano Chesi, Andrea Moro, and Ste- fano F Cappa. Connected speech in neurodegenerative language disorders: review. Frontiers in psychology, 8:269, 2017.
Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Juliam Ibarz, Alex Irpan, Eric Jang, Ryan Julian, et al. Do as i can, not as i say: Grounding language in robotic affordances. In Conference on Robot Learning, pp. 287-318. PMLR, 2023.
10

---

Published as a conference paper at ICLR 2024
Vineeta Chand, Kathleen Baynes, Lisa M Bonnici, and Sarah Tomaszewski Farias. A rubric for extracting idea density from oral language samples. Current protocols in neuroscience, 58(1): 10-5,
Soravit Changpinyo, Doron Kukliansy, Idan Szpektor, Xi Chen, Nan Ding, and Radu Soricut. All you may need for vqa are image captions. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technolo- gies, pp. 1947-1963, 2022.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Síyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric P. Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023. URL https: / wtlmsys.org/blgg. 2023- 03- vicuna/.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.023111 2022.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, al. Scaling -innttttiooninnuneedd language mod- els. arXiv preprint arXiv:2210. 1416, 2022.
Mingkai Deng, Jianyu Wang, Cheng-Ping Hsich, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. RLPrompt: Optimizing discrete text prompts with rein- forcement learning. In Proceedings of the 2022 Conference on Empirical Methods in Natu- ral Language Processing, pp. 3369-3391, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 96665//1/2222.mmmpmmmmaaaa https://aclanthoooogy orgg 2022. mat.://e....mmmmmmmmmaaaala
URL
222.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers). pp. 4171-4186 Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 1011863//1//11914255 URL https: // //aclanthology 9 9-MM19-142..
Li Dong, Jonathan Mallinson, Siva Reddy, and Mirella Lapata. Learning to paraphrase for question answering. Proceedings of the 2017 Conference on Empirical Methods Natural Language Processing, pp. 875-886, Copenhagen, Denmark, September 2017. Association for Computa- tional Linguistics. doi: 10.18653W1//17--1099. URL / wrt
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong. Tianhe Yu, et al. Palm-e: An embodied multi- modal language model. arXiv preprint 23X3:2230330377, 2023.
Yuxin Fang, Wen Wang, Binhui Xie, Quan Sun, Ledell Wu, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva: Exploring the limits of masked visual representation learning at scale. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp- 19358-19369, 2023.
William Ford and David Olson. The elaboration of the noun phrase in children's description of objects. Journal of Experimental Child Psychology, 19(3)371-3822 1975.
Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoît Sagot, and Rachel Bawden. Tackling am- biguity with images: Improved multimodal machine translation and contrastive evaluation. arXiv preprint arXiv:2212.101442 2022.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trends@ in Computer Graphics and Vision, 14(3-4):163 163-352,, 2022.

---

Published as a conference paper at ICLR 2024
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Lin- guistics and the 1 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 3816-3830, Online, August 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/2022 http:/2022..ccoono2222 URL https: http:s/s///aaaaooooogoooooo org 2021
295.
Mor Geva, Eric Malmi, Idan Szpektor, and Jonathan Berant. DiscoFuse: A large-scale dataset for discourse-based sentence fusion. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, Volume (Long and Short Papers), pp. 3443-3455, Minneapolis, Minnesota, June 2019. Association for Computational Linguisties. doi: 10. 10.18653/ //aclanthology 1///N1911134
URL https:
Edward Gibson et al. The dependency locality theory: A distance-based theory of linguistic com- plexity. Image, language, brain, 2000:95-126, 2000.
Yash Goyal. Tejas Khot, Douglas Summers-Stur Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904-6913, 2017.
Jiaxian Guo, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Boyang Li, Dacheng Tao, and Steven Hoi. From images to textual prompts: Zero-shot visual question answering with frozen large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10867-10877. 2023.
Danna Gurari, Qing Li, Abigale Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3608-3617, 2018.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, 2019.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-officient transfer learning for NLP. In International Conference on Machine Learning, pp. 2790-2799, PMLR, 2019. URL http://proceedingg wtwpaiins..l....es//// wtt.prss/////houlsbyy//or 9a.. html.
J Edward Hu, Huda Khayrallah, Ryan Culkin, Patrick Xia, Tongfei Chen, Matt Post, and Benjamin Van Durme. Improved lexically constrained decoding for translation and monolingual rewriting. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 839-850, 2019.
Yushi Hu, Hang* Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. Promptcap: goiddt task-aware image captioning. arXiv preprint arXiv:22/1.09699 2022.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700-6709, 2019.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie. Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In European Conference on Computer Vision, pp. 709-727. Springer, 2022.
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2901-2910, 2017.
Saurav Kadavath, Tom Conerly, Amanda Askell, Tom Henighan, Dawn Drain, Ethan Perez, Nicholas Schiefer, Zac els (mostly) know what they know. arXiv preprint arXiv:2207.05221 2022.
Nova DasSarma, Eli Johnson, et al. Language mod-
12

---

Published as a conference paper at ICLR 2024
Susan Kemper. Language and aging. The handbook of aging and cognition, 1992.
Daniel Khashabi, Xinxi Lyu, Sewon Min, Lianhui Qin, Kyle Richardson, Sean Welleck, Han- naneh Hajishirzi, Tushar Khot, Ashish Sabharwal, Sameer Singh, and Yejin Choi. Prompt way- wardness; The curious case of discretized interpretation of continuous prompts. In Proceed- ings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 3631-3643, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022. naacl- main.266. URL org/2022 naacl 266.
Muhammad Uzair Khattak, Hanoona Rasheed, Muhammad Maaz, Salman Khan, and Fahad Shah- baz Khan. Maple: Multi-modal prompt learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19113-19122, 2023.
Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, et al. Dynabench: Rethinking bench- marking in nlp. In Proceedings of the 2021 Conference of the North American Chapter of the As- sociation for Computational Linguistics: Human Language Technologies, pp. 4110-4124, 2021.
Ruud Koolen, Martijn Goudbeek, and Emiel Krahmer. Effects of scene variation on referential over- specification. In Proceedings of the Annual Meeting of the Cognitive Science Society volume 33, 2011.
Moritz Laurer, W Atteveldt, Andreu Casas, and Kasper Welbers. Less annotating, more sessifying-aadressiing the data scarcity issue of supervised machine learning with deep trans- fer learning and bert-nli, 2022. URL https://ostt io/wqc86/.
Logan Lebanoff, John Muchovej. Franck Dernoncourt, Doo Soon Kim, Lidan Wang, Walter Chang, and Fei Liu. Understanding points of correspondence between sentences for abstractive sum- marization. In Proceeulings of the 58th Annual Meeting of the Association for Computational Linguistics: Student Research Workshop, pp. 191-198, Online, July 2020, Association for Com- putational Linguistics. doi: 10.18653/v1/2020..cc acl- 21..22.. URL 2020. acl-srw. 26.
org/
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. arXiv preprint arXiv:2301. 12597, 2023.
Stephanie Lin, Jacob Hilton, and Owain Evans. Teaching models to express their uncertainty in words. Transactions on Machine Learning Research, 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haitao Liu, Chunshan Xu, and Junying Liang. Dependency distance: A new perspective on syntactic patterns in natural languages. Physies of life reviews, 21:171-193, 2017.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction 2023a.
tuning. arXiv preprini
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. arXiv preprint arXiv:2304.0845: 2023b.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for gpt-3? arXiv preprint arXiv:2101.06804 2021. URL htthtt:: //arxiv. wrw. abs/ 2101 06804.
Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. What makes good in-context examples for GPT-3? In Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learn ing Architectures, pp. 100-114 Dublin, Ireland and Online, May 2022. Association for Compu- tational Linguistics. doi: 10.11653//1/2022..dee 1.10. URL https: / http///cllaaaa http://aaaaaantoooaao/oao org/ 2022, 2022.deelion dew..i-1..111
13

---

Published as a conference paper at ICLR 2024
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8086-8098, Dublin, Ireland, 2022. Association for Computational Linguistics. doi: 10.18653/v1/2022.aa---2
URL https: http://aaaaaaattlocooooooooocl wwwpoo/g....... sol-long.- 556.
Lucie Charlotte Magister, Jonathan Mallinson, Jakub Adamek, Eric Malmi, and Aliaksei Severyn. Teaching small language models to reason. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). pp. 1773-1781, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 10.186533/1/20223.adl 151. URL httph:tt///t... http:::///caaaathlloooooo wrw.:/23..23333 acl.scort..tt111 151.
Bodhisattwa Prasad Majumder, Sudha Rao, Michel Galley, and Julian McAuley. Ask what's missing and what's useful: Improving clarification question generation using global knowl- edge. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 4300-4312, Online, June 2021. Association for Computational Linguistics. doi: 10.18653/v1/2021 https://aclanthoolog)) 2021 maacl-mainn 340.
URL
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/cvf conference on computer vision and pattern recognition, pp. 3195-3204, 2019.
Sabrina Mielke, Arthur Szlam, Emily Dinan, and Y-Lan Boureau. Reducing conversational agents' overconfidence through linguistic calibration. Transactions of the Association for Computational Linguistics, 10:857-872. 2022.
Sewon Min, Julian Michael, Hannaneh Hajishirzi, Luke Zettlemoyer. AmbigQA: Answering ambiguous open-domain questions. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 7783-5797, 2020.
Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. Rethinking the role of demonstrations: What makes in-context learning work? In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 11048-11064, Abu Dhabi, United Arab Emirates, 2022. Association for Computational Lin- guistics. doi: 10. 10.18663/1122222.mmmg main. 759. URL pettps:/aacantthollee emnlp-main. 759.
2022.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannanch Hajishirzi. Re- framing instructional prompts to gptk's 's language. In Findings of the Association for Com- putational Linguistics: ACI. 2022. Association for Computational Linguisties, 2022. URL
https://arxiv wwwpiii..rg/ab///////ea 07830.
OpenAl. Gpt-4 technical report, 2023.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.021 220.022155..... 2022.
Masanori Oya. Syntactic dependency distance as sentence complexity measure. In Proceedings of the 16th International Conference of Pan-Pacific Association of Applied Linguistics, volume 1, 2011.
Thomas Pechmann. Incremental speech production and referential overspecification. Linguisties, 1989.
Sandro Pezzelle. Dealing with semantic underspeciffcation in multimodal NLP. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pp. 12098-12112, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 18633/ 1011665//1/2223... 1/2023.ack aa222223.al-ooon.66555 URL htttpp://aclannee hwc.tthhollanacoooooooooooooo wrgg:/rr///// 2023.222..22 acl-long.66 675.
John Platt et al. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3):61-74, 1999.
14

---

Published as a conference paper at ICLR 2024
Matt Post and David Vilar. Fast lexically constrained decoding with dynamic beam allocation for neural machine translation. In Proceedings of the 2018 Conference of the North American Chap- of the Association for Computational Linguisties: Human Language Technologies, Volume I (Log Papers), pp. 1314-1324, 2018.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based in- struction search for prompting large language models. Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 3827-3846, 2023.
Danish Pruthi, Rachit Bansal, Bhuwan Dhingra, Livio Baldini Soares, Michael Collins, Zachary Lipton, Graham Neubig, and William W Cohen. Evaluating explanations: How much do ex- planations from the teacher aid students? Transactions of the Association for Computational Linguistics, 10:359-375 2022.
Valentina Pyatkin, Jena D. Hwang, Vivek Srikumar, Ximing Lu, Liwei Jiang, Yejin Choi, and Chan- dra Bhagavatula. ClarifyDelphi: Reinforced clarification questions with defeasibility rewards for social and moral situations. In Proceedings of the 6 Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 11253-11271, Toronto, Canada, July URL
2023. Association for Computational Linguistics. doi: 122.6633///222...aooooo.ggg https: htaps://aclaan http://aaaaaaaooo....oooo wrg.2023. acl- llng. 630.
Peng Qi, Yuhao Zhang, Yuhui Zhang, Jason Bolton, and Christopher Manning. Stanza: python natural language processing toolkit for many human languages. In Proceedings the 58th Anmual Meeting of the Association for Computational Linguistics: System Demonstrations, pp. 101-108, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pp- PMLR, 2021.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:21 2021.
Nathan Ellis Rasmussen and William Schuler. A corpus of encyclopedia articles with logical forms. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020), 2020.
Stuart Rose, Dave Engel, Nick Cramer, and Wendy Cowley. Automatic keyword extraction from individual documents. Text mining: applications and theory. pp. 1-20, 2010.
Swarnadeep Saha, Peter Hase, and Mohit Bansal. Can language models teach weaker agents? teacher explanations improve students via theory mind. arXiv preprint arXiv:2306.09290 092999 2023.
Hinrich Schutze. Ambiguity in language learning: Computational and cognitive models. Stanford University, 1995.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pp. 146-162. Springer, 2022.
Sasha Sheng, Amanpreet Singh, Vedanuj Goswami, Jose Magana, Tristan Thrush, Wojciech Galuba, Devi Parikh, and Douwe Kiela. Neural Information Processing Systems, 34:20346-20333 2021.
visual question answering. Advances in
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. AutoPrompt:
Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 4222 4235, Online, November 2020. Association for Computational Linguistics. doi: 10. 18653/v h/tp://e00000022mma....a 346. URL https: //ap::::///aaa http////aaaaallg/////// .rg. 2220..200220 emnlp-main. 346.
15

---

Published as a conference paper at ICLR 2024
Abhishek Shivkumar, Jack Weston, Raphael Lenain, and Emil Fristed. Blabla: Linguistic feature extraction for clinical analysis in multiple languages. Proc. Interspeech 2020, pp. 2542-222222222 2226, 2020.
Susan Sonnenschein. The development of referential communication skills: Some situations in which speakers give redundant messages. Journal of Psycholinguistic Research, 14(5):48--088 1985.
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonoo, et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. Transac- tions on Machine Learning Research, 2023. ISSN 2835-8856. URL https://openrevieet net/ forum?idFuyTLLBvos))
Elias Stengel-Eskin and Benjamin Van Durme. Calibrated Interpretation: Confidence Estimation in Semantic parsing. Transactions of the Association for Computational Linguistics, 2023. doi:
Elias Stengel-Eskin, Jimena Guallar-Blased Yi Zhou, and Benjamin Van Durme. Why did the chicken cross the road? rephrasing and analyzing ambiguous questions in VQA. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pa- pers). pp. 10220-10233 Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10.18653/v wwt222222...1loo........
URL https: / /aclanthollaancoooooooooooooo org/ 2023. acl-long. 569.
Tianxiang Sun, Yunfan Shao, Hong Qian, Xuanjing Huang, and Xipeng Qiu. Black-box tuning for language-moddll-as--sseees arXiv preprint arXiv:2201.035144 2022.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 35: - 2022a.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. VI-adapter: Parameter-efficient transfer learning for dision-and-languaal tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5227-5237, 2022b.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Niko- lay Bashlykow, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open founda- tion and fine-tuned chat models. arXiv preprint arXiv:2307.009288 2023.
Maria Tsimpoukelli, Jacob Menick, Serkan Cabi, S. M. Ali Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL /ttpss//openreviii
met/forum?id::tmiyyyy
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Albert Webson and Ellie Pavlick. Do prompt-based models really understand the meaning of their http://bb///19.11../.
prompts? arXiv preprint arXiv:2109.01247, 2021, URI
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgcaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large lan- guage models. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856. URL https://openreviee www.rrvviieeeeeeforrreee.. wwwt/oeumt/mene teee//forummddyyykSUSz2ddd Survey Certification.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi. Quoe V Le, Denny Zhou, et al. Chain-of thouhht prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824-24837, 2022b.
16

---

Published as a conference paper at ICLR 2024
Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadch, et al. Ethical and social risks of harm from language models. arXiv preprint arXhv:2112.043359 2021. URL https://arxxv..rr/// 2112, 04359.
Thomas Wolf, Lysandre Dehut, Vietor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, Online, October 2020. Association for Computational Linguistics. URL https:/ /www aclweb. org/ httgg//lloggy/////e/ooooo (etphoollgyy2200...0 2020. emm.pp.eeepppmmmmmmmmmmmmmpmmlmm momlp-ddmos.. 6.
Yujia Xie, Luowei Zhou, Xiyang Dai, Lu Yuan, Nguyen Bach, Ce Liu, and Michael Zeng. Visual clues: Bridging vision and language foundations for image paragraph captioning. Advances in Neural Information Processing Systems, 35: 15:17287-17300 2022.
Zhengyuan Yang. The Gan, Jianfeng Wang, Xiaowei Hu, Yumao Lu, Zicheng Liu, and Lijuan Wang. An empirical study of gpt-3 for few-shot knowledge-based vqa. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp. 308 1081-3 2022.
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. arXiv preprint arXiv: 13549, 2023.
Bianca Zadrozny and Charles Elkan. Transforming classifier scores into accurate multiclass proba- bility estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowl- edge discovery and data mining, pp. 694-699, 2002.
Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference Machine Learning, pp. 11328-11339, PMLR, 2020. URL http://proceedingg wtt.ssmlr..prrrr htt..rprre//1111/aa22202.. zhang20ae. pdf.
Jingyi Zhang, Jiaxing Huang, Sheng Jin, and Shijian Lu. Vision-language models for vision tasks: A survey, arXiv preprint arXiv:2304. 00685, 2023a.
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E. Gonzalez. TEM- PERA: Test-time prompt editing via reinforcement learning. In The Eleventh International Conference on Learning Representations, 2023b. URL http:///pppneeeeeeeeeeeerrreeree JSPo.
Zhuosheng Zhang, Aston Zhang, Mu Li, Hai Zhao, George Karypis, and Alex Smola. Multimodal thouugt reasoning in language models. arXiv preprint arXtv:2302..0993,, 2023c.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. Calibrate before use: Improv- ing few-shot performance of language models. In International Conference on Machine Learn- ing. pp. 12697-12706. PMLR, 2021. URL. http://prooceddings mlr, wttpsm.ess...e1///ee./ess/ press//139/2hho21ee zhao21c.pdff
Kaitlyn Zhou, Dan Jurafsky, and Tatsunori Hashimoto. Navigating the grey Expressions of overconfidence and uncertainty in language models. arXiv preprint arXiv:2302. 13439, 2023.
Deyao Zhu, Jun Chen, Kilichbek Haydarov, Xiaoqian Shen, Wenxuan Zhang, and Mohamed Elho- seiny. Chatgpt asks, blip-2 answers: Automatic questioning towards enriched visual descriptions. arXiv preprint arXiv:2303.06553 2023a.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing rision-languee understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023b.
17

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 DATA
We three datasets, VQAv2 (Goyal al., 2017), A-OKVQA (Schwenk et al., 2022), and VizWiz (Gurari et al., 2018). VQAv2 is an extension of the original VQA dataset (Antol et al., 2015), which incorporates similar images yielding different answers to the same question. This augmenta- tion doubles the number of image-question pairs, emphasizing the reliance on visual information for accurate answers. While VQA questions are open-ended, the answer vocabulary is relatively limited in size (10M), consisting of mostly one-word responses. In VQAv2, each example is associated with 10 ground-truth answer labels provided by different human annotators. On the other hand, the A-OKVQA dataset is smaller (25K questions in total) but is more challenging. Similar to VQAv2, in the direct answer setting. 10 human annotated 1-2 word answers are provided for each question. The multi-choice setting comes with 4 options along with the index of the correct option. Lastly, the VizWiz dataset contains 32.8K information-sseking questions asked by visually-impairee peo- ple based on images clicked mobile devices. This dataset can be challenging the images often blurred, auder/over-exposee or rotated. During the design and analysis REPARE, we use separate development set consisting of 5K, 1K, 500 randomly sampled image-question pairs from the train sets of VQAv2, A-OKVQA, and VizWiz respectively. For testing, we use the entire vali- dation set; this corresponds to 214K examples for VQA, 1.1K examples for A-OKVQA, and 4.3K examples for VizWiz. We use the standard VQA metric for open-ended evaluation. According to this metric, a model-ggnerated answer deemed 100% accurate if least 3 of the 10 annotators provided that exact answer.
# humans that said ans
Accuracy voA min
The predicted answer is also pre-processed by lowercasing, converting numbers to digits, and re- moving punctuation/arttiles. Since LLMMs generate free-form text, we constrain the answers using length-penalty of during generation which encourages shorter answers that align better with hu- man annotations.
A.2 PROMPTS
Table 11 contains an exhaustive list of all prompts used in REPARE for various models. Limited prompt engineering (2-3) trials were done for each prompt on our dev split. For sentence fusion, we use two hypothetical examples (note that these do not include images):
1. Question: What is the man wearing?: Object: man; Detail:
he is standing on the sidewalk; Modified Question: What the man who is standing on the sidewalk wearing?
2. Question: Are there any flowers?; Object: flowers; Detail: There is flowers are in a vase. The vase is blue color and sitting on a table; Modified Question: Are there any flowers in the vase on the table?
A.3 EXPERIMENTAL DETAILS
Model Checkpoints In Sec. 3.3, we use BLIP-2 with ViT-g frozen image encoder (1B param- eters), and Flan-T5 XL with 3B model parameters. The pretrained Q-former is an encoder-only transformer model (Vaswani et al., 2017) that shares similar architecture with BERT (Devlin et al., 2019) comprising of 107M parameters. MiniGPT-4 and LLaVA-115 are based on the BLIP-2 ar- chitecture with addition VL pretraining. One key difference is that it uses the Vieuna family of LLMs. We experiment with the two official checkpoints with 7B and 13B model parameters. In Sec. 4.2, we popular Pegasus-basee based paraphrasing model available on HuggingFace (Wolf et al., 2020). 6 We also use an off-the shelf NLI model that achieves competent performance on a suite of NLI benchmarks Laurer al. (2022). In Sec. 3, we also use the rake_nltk python package.
'Link to Checkpoint:
http://ff.......teaeeeeeeeeeeeee co//uuerr887/pgaaal paraphrase LLink: https: http://aww.......oeeeeeeeeeeeeee Large- wwwllllffee---ee anl i- 1-ling-wanl
18

---

Published as a conference paper at ICLR 2024
VQAv2
Orig. Q: Does the water have ripples?
Orig. Q: What behind boy?
Orig. Q: What time say?
Orig. Q: What color wearing?
A-OKVQA
Orig. Why would you use this bag?
Orig. Q: Which dothing is unique to one person here?
Orig. built here one day?
Orig. man facing camera are ell weering what?
Example images and original questions for Table 6. Some questions (e.g.. "What is behind the boy" are underspecified, while others refer to small objects in the image (c.g.. "What color wetsuit is he wearing?")
Figure 3:
Stage I: Extracting Visual Details and Generating Candidates. As described in Sec. 3.1, extract salient visual details from the image using 3 components described below in further detail.
Salient Question Enfities: Given only the question from data instance, extraction tool from rake_nltt package to identify salient keywords mentioned in the ques- tion. For instance, in the example shown in Fig. 2, "day" is identified the salient entity. (ii) Information from Rationales: For this module, we use both the image and the original question for the data point and adopt two step approach. First, we ask the model to generate an explanation for its answer. Next, based on the explanation and question, we ask the model to identify salient entities mentioned or used in the rationales. Refer to Rationale (ii) prompts listed in Table 11.
the keyword
(iii) Information from Captions: We adopt the straightforward approach of using the caption prompts from Table 11 to generate the image caption using the LVLM.
We extract visual information about entities identified in steps (i) and (ii) using the LVLM by query- ing it using the extraction of details prompt listed in Table 11 for each entity separately. This information concatenated [details]. To this prompt, add the generic details from the image caption via additional line: image: [capt in]] This list of details along with the original question are added to the sentence fusion prompt from Table 11 along with in-context examples mentioned in Appendix A.2 above which generates the modified question candidates by sampling multiple outputs (same LVLM call).
prompt in the form of a bulleted list of format [entity]
Text Generation in REPARE. To ensure the paraphrasing model generates a valid question end- ing with "?' we employ constrained decoding by setting a positive constraint on generating the '?' token (Post & Vilar, 2018; Hu et al., 2019). To ensure diverse samples in the sentence fusion stage (determines the diversity of question candidates) we use top-p sampling Holtzman al. (2019) with p 0.95. To sample rationales, we employ beam search with 5 beams and temperature of 0.7. After generating question candidates, we filter out sentences that are not valid (do not end with question mark) or are a verbatim repetition of the original question. Additionally, we also filter out contradictory generations as described in Sec. 3. sample enough candidates such that we are left with n distinct candidates in the end. If this is not feasible, repeat the original question in the candidate set to make up for the difference.
Candidate Selection via REPARE" RE's Score (Stage II). We employ the VQA prompts mentioned in Table 11 to obtain answers for each question candidate. Typically, the answers (direct answers or option label) correspond to one word or token. In case where we are scoring multiple tokens (as in Sec. 3.2 Sec. 4.1), compute the length (number of tokens) normalized log- bog-prbbabilities that are subsequently exponentiated to obtain probabilites. Note that we are only interested in the relative order, therefore, can alternatively use Iog- proaabbbaslsss to score and select candidates too.
19

---

Published as a conference paper at ICLR 2024
A.4 QUALITATIVE EXAMPLES AND ANALYSIS
REPARE questions exhibit an increased degree of specificity, with additional modifiers and fewer ambiguous references, egg., "the person riding the wave on the surfboard" as opposed to "he" the original. Even when questions are unambiguous, REPARE questions include reasoning and location clues. For example, a rephrased question like "What time on the clock at the top building?" indicates which region of the image is important.
Original
REPARE
Does the water have ripples?
Does the water have the small ripples around the boats?
What time does the clock say?
What time is on the clock at the top of huilding?
A-OKVQA VOA
What is behind the boy?
What is behind the boy doing trick on a skateboard?
What color wetsuit is he wearing?
What color the wetsuit of the person riding the wave surfboard?
Why would you this bag?
Why would you use this suitcase packed on both sides?
Which piece of clothing unique to
What piece of clothing is unique to the man on the right?
one person here?
What will be built here one day? The men facing the camera all wearing the same pair of sunglasses?
What will be built at this construction site?
The men facing the camera are all
wearing what?
Table 6: Qualitative examples of original and REPARE generated questions for both datasets with BLIP-2 as the underlying model. For corresponding images, refer to Fig. 3.
Fig. 3 shows the images corresponding to the examples given in Table 6. Each image is paired with its original question as well as the rephrased question from REPARE.
Answers Questions. In some cases, e.g. in the final A-OKVQA question about sunglasses in Table 6, the correct answer ( ("sunglasses" ") is added the question by REPARE. We first note that this is not an unfair advantage, since REPARE operates on the same information as the QA model (image and question), using the same LVLM. Any additional information in a REPARE RE question is extracted from captions and rationales, which are obtained in a realistic zero-shot test-time setting without any access to the gold answer. Similarly, REPARE'S selection module does not use the gold answer in selection. Nevertheless, we report the percentage of times the correct answer is found in the REPARE question and not in the original question. Here, we use the A-OKVQA open-ended (direct) setting and the BLIP-2 model. In random sample of 100 examples we find that 7% of rewritten questions from REPARE contain gold answer. This indicates that part of REPARE'S advantage likely comes from an ability to extract the correct answer from the caption and rationale information, incorporate into a question candidate, and then select that candidate.
A.5 ADDITIONAL ABLATIONS
Impact of LVLMs on Generating Question Candidates. As mentioned in Sec. 3.1, we only used the underlying LLM to fuse or incorporate the extracted visual details into the given question. In Table 2 of Sec. 4. we quantitatively show that including the visual tokens, i.e., using the entire LVLM negatively impacts the overall REPARE pipeline and decreases downstream performance. To provide additional insights, Table contains qualitative examples of fusion using only the LLM and the entire LVLM using the BLIP model. We observe that the image embeddings serve as a distraction the LLM when performing primarily linguistic task and the resultant question is often ill-formed and heavily dominated by the image caption and/or visual details.
Alternate ways of computing Answer Confidence. Kadavath et al. (2022) demonstrate that the self-evaluation ability model better in multiple-choice settings than settings in which the LM is required to directly generate the answer. Note that a multiple choice setting is also better specified, since the model is conveyed a set of options to choose from. For instance, if multiple plausible answers exist, only one would be mentioned in the options, indirectly communicating the type of intended response. This is reflected by the contrast in A-OKVQA accuracy (cf. Table 1) in direct and MC settings. In the direct answer setting, compare computing model's 's answer confidence in
20

---

Published as a conference paper at ICLR 2024
Original
REPA RE w/ LLM Fusion
REPA RE LVLM Fusion
Does the water have rip- ples?
Does the water have the small ripples around boats?
Yes, there are ripples in the body of water where group of boats are docked?
What time does the
What time is on the clock at the top of building? What behind the boy doing trick skateboard?
What time does the clock tower say in the building?
clock say?
What is behind the boy?
person is doing trick on skate- board behind the boy
VQA
middle of a parking lot in the city at night in front of a large building with skyscraper in the background? What color wetsuit
What color wetsuit wearing?
What color the wetsuit of person riding the wave on the surfboard?
riding in?
Why would you use this bag?
Why would you use this suitcase packed on both sides?
Why would you use this bag with lot items it?
Which piece of clothing
What piece of clothing is unique to the on the right?
What piece of clothing is unique to one person: man and woman posing for picture?
is unique to one person here?
A-OKVON
What will be built here one day?
What will be built this construction site?
truck is driving down a street with construction cones and a construc- tion site in the background, what will be built there one day? The men facing camera are all wearing the same thing. what is it?
The men facing the The men facing camera are camera are all wearing wearing the same pair of sunglasses? what?
Table Qualitative comparison of generated question candidates with and without visual tokens in Stage (sentence fusion) of REPARE. Corresponding images shown in Fig. 3.
two additional ways that Kadavath et al. show to be better calibrated. First, we compute True/False answer confidence by adding the following suffix to the VQA prompt:
Proposed Answer: [d:]- Is the proposed answer true or false? (A) True, (B) False. The proposed answer is:
Then, P we also employ the strategy of showing the model multiple generated answers to estimate answer confidence. For this take advantage of different question candidates that yield different answers ( hee(..)) Hence, we add the following prefix to the VQA prompt:
substitute for answer confidence PLVLM(á 4.). Additionally,
Plausible Answers: [(d+)ve(L.)) Proposed Answer: d;j
Is the proposed answer true or false? (A) True, (B) False.
The proposed answer is:
We denote this setting as PLYLM True (e(t.)) and substitute this probability instead in the score function. The results are shown in Table 8. We find that all the implementations of LVLM's answer confidence yield comparable performance across datasets (<1 point difference). Since we use the same LVLM to rank various question candidates for a given image, the relative ordering of scores (Sec. 3.2) should not be significantly affected by the model's calibration. Post-hoc calibration methods such Platt scaling (Platt et al., 1999) isotonic regression (Zadrozny & Elkan, 2002) preserve relative ordering, so they would have no effect on the selection criterion.
REPARE score
VQA
A-OKVQA
PinM( 1.):) Avtu(True) PruM (a))))))))
67.28
45.01
68.01
44.78
67.56
44.87
Table 8: Comparison of performance of REPARE with BLIP-2 using different score functions for computing answer confidence.
21

---

Published as a conference paper at ICLR 2024
Imapet of increasing the number of candi- dates n2 in REPARE. In Fig. 4, we explore the impact of increasing the number of candi- dates in REPA RE, i.e., n. on its effectiveness enhancing BLIP-2's VQA (direct) perfor- mance. We find that initially increasing from 2 to 5 leads to performance gains in both the inference and oracle settings for VQAv2 and A-OKVQA datasets. However, the gains saturate after = 10, 15 for both datasets. In fact, during inference, wherein we select 1 out of question candidates, we find the VQA accuracy gradually decreases at 15. This is expected, since increasing allows for diverse candidates; however, selection from a very large pool of candidates (like n = 15) more challenging, and REPARE's selection module is more likely to make suboptimal choice hence the growing gap between oracle and REPARE performance.
RepARe (Oracle)
w/ RepARe A A-OKVQA w/ RepARe
A-OKVOA w/ RepARe (Oracle)
VQA Accuracy
70-
60
10
Number of Candidates (n)
Figure 4: Trends in VQA performance of REPARE for different values of
Analysis with MiniGPT-4. In Sec. 4, we use BLIP-2 for conducting our analysis. However, BLIP-2 uses an encoder-decoder LLM while the remaining LVLMs (MiniGPT-4 and LLaVA-1.55 use Vicuna, which decoder-only LLM. In this section, we show that the choice of underlying LLM architecture does not impact the relative trends. In Tables and 10, we repeat the analysis Sec. 4.1 with MiniGPT-4 models corresponding to Tables 2 and respectively. Our ablation study in Table once again highlights the importance of each component of RepARe in improving VQA performance. Similarly, Table 10 reveals that even with MiniGPT-4 as the underlying LVLM, candidates generated by REPARE significantly outperform paraphrased question candiates when selected based on the model's ' answer confidence.
Method
VQAv2 A-OKVQA
Method
VQAv2
A-OKVQA
REPARE
57.74
31.23
Overall
Direct
MC
w/o Rationales
53.29
28.62
Baseline (MiniGPT-4)
51.47
27.51
41.66
w/o Caption
56.49
29.51
Paraphrase Oracle
57.41
39.83
73.68
w/o Question Entity
54.91
29.28
REPARE Oracle
59.66
41.92
75.60
w/ Fmheddings in Fusion
54.49
28.97
Paraphrase Selection
51.39
26.28
40.52
w/ score PLVlM(q.IT)
56.46
29.19
REPARE Selection
54.49
33.23
63.20
Table 9: Ablation design choices in RE- PARE using MiniGPT-4vium 78 on our dev splits (direct answers).
Table 10:
Comparison of REPARE (MiniGPT- 4iiunn 7B) with paraphrasing questions in the or- acle setting and unsupervised candidate selection.
22

---

Published as a conference paper at ICLR 2024
Dataset
Setting
Prompt
VQAv2
Question: [Question] Short Answer:
Question: [Question]
A-OKVQA (MC)
VQA Prompt
Options: A. [Choice 11, B. [Choice 2], C. [Choice
3] D. [Choice 4]
Answer: Option
BLIP-2
Caption
(Default, emply string)
Rationale (i)
[VQA Prompt] Explanation:
Rationale (ii)
[LVLM Response for Rationale (i)]
Question: [Question]
All
Which entities or objects from this image would I need to observe to answer this question?
Extraction
of
Question: What can you tell me about [entity] in this image?
Details
### Human: <Img> <ImageHere> </Img>### Human: Based the image, answer the question below in preferably only word.
VQAv2
VQA Prompt
Question: [Question]
### Human: <Img> <ImageHere> </Img>## Human: Based on the image, answer the question below. Explain your answer.
VQA Prompt (i)
A-OKVQA
Question: [Question]
[VQA Prompt (i)]### Assistant: [LVLM Response]
### Human: Shorten your answer to the question as much possible, preferrably only word.
VQA Prompt (ii)
MiniGPT-4
Human: Based on the image, select the correct answer to the question from the options. You MUST mention option labels, i.e., 'A.', 'B.', 'C. or *D." in your response. Explain your answer.
VQA Prompt (i)
Question: [Question]
Options: [Choice 1], B. [Choice 2], C. [Choice
A-OKVQA (MC)
3] D. [Choice 4]
[VQA Prompt (i)] Assistant: [LVLM Response] ##* Human: So which option your final answer: A. 'B.', 'C. or 'D.'?
VQA Prompt (ii)
All
Caption
#*# Human: <Img> <ImageHere> </Img>### Human:
Describe the image in couple of sentences.
All
Extraction
of
##i Human: What can you tell about [entity] in
Details
this image?
##f Human: are given description image, a question and its response below.
Image Content: [Caption Response] Question: [Question]
Rationale (ii) MiniGPT-4
Response: [Rationale Response from VQA prompt]. List up to objects or from the image were relevant to answering the question? Describe each object ONLY 2-3 words. #a# Assistant: Enumerated list of top-3 relevant objects used:
All
Sentence Fusion
are given a question about an image. Modify question by adding descreptive phrases to entities based on the provided details. Both original and modified questions MUST have similar meaning and answer. [2 Hypothetical Examples] Question: [Question]
Details: [Bulleted list of entities and 1-2 sentences of corresponding details.] Modified Question:
Table 11: All the prompts used in REPARE, '(i' and '(ii)' indicate a sequential conversation. FFor Vicuna model, we add ### Human, and ### Assistant prefixes.
23


Reviews:
[{"review_id": "ndBQp2mfUP", "reviewer": "Reviewer_Jt1P", "summary": "This paper introduces \"Repare\", a gradient-free framework that consists of three phases: Visual Details Extraction, Question Rephrasing and Augmentation, and Question Selection. \"Repare\" rephrases questions with more precise information, drawing from image captions, question keywords, and rationale details. Following rephrasing, Repare produces multiple question variations, which are then filterd through an unsupervised quality score provided by a Language Model (LM). The paper concludes with an assessment of Repare's performance on VQAv2 and A-OKQA datasets, showing enhancements when compared to the BLIP2 and MiniGPT4 baseline models.", "questions": "Please take a look at the weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is easy to read and generally well-written\n- Interesting idea of improving VL models in VQA tasks by just modifying one modality (e.g text).\n- Improvements over baselines (BLIP2, MiniGPT4) looks reasonable.", "weaknesses": "- Evaluation suite should be improved. For example including: TextVQA, VizWiz. Additionally, authors should consider evaluating tasks such as HatefulMemes which might be more challenging to the proposed approach. Also, consider recent evaluation tasks such as MME.\n- Including recent instruct multimodal models (e.g. LLaVa, Qwen-VL) would be an interesting experiment to see if the gain with Repare is still relevant in these models.\n- Minor:\n    - >One approach involves additional VL pretraining ...\n   \n         In this case, LENS does not involve additional multimodal pretraining.\n  - Why not include BLIP2-XXL as one of your main baselines and improve it with Repare?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hcbaENDxKB", "reviewer": "Reviewer_j2wD", "summary": "The paper proposes an pipeline for using LVLMs to solve VQA that modifies the question using visual information from the image and select the answer with the best confidence. Experiment results on VQAc2 and A-OKVQA show improvement in all question categories. The authors use ablation studies and shown that the visual details and question entity do help improve the performance.", "questions": "1. Any explanations on why questions with numeric answers benefit the most from RepARe?\n2. What does it mean using `golden` answers for selection? And also `paraphrase oracle`.\n3. In 4.4, why is that \"In all cases, REPARE increases the gap between LLM-only and BLIP-2\"? As it seems that the gap actually decreases. (The numbers should be negative)\n4. The baseline of adding captions alone to the question, i.e. <caption><question>, should be compared.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The performance improvement from the method seem to be solid.\n2. The paper is mostly clear with extensive experiments.", "weaknesses": "1. Some of the text requires further clarification.\n2. The underlying hypothesis should be stated more clearly, which in my understanding is that distribution of more specified questions are more aligned with the training data of LVLMs and the answers with higher confidence are more likely to be correct.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wKRcZg36pl", "reviewer": "Reviewer_KySW", "summary": "This paper proposed a pipeline framework for improving the accuracy of LVLMs in VQA tasks.\nFirst, details are identified from the question and the image, including entities (using off-the-shelf tools) and rationales (using the LVLM), which are then used to obtain related details from the image (using the LVLM), and descriptions from the image (using the LVLM).\nSecond, the details are fused with the question, generating multiple question candidates (using the LVLM), the results of which are filtered to exclude semantically inconsistent ones (using off-the-shelf tools).\nFinally, the LVLM answers each candidate question; the final answer is selected using the LVLM with the confidence-based method.\n\nOn two benchmark sets, i.e., VQAv2 and A-OKVQA, solid improvements are shown on 3 LVLMs. The with/without kind of ablation is conducted suggesting each element in the pipeline contributes to the improvement. Further analyses are conducted to try to show that the new questions are effective because of less underspecification and ambiguity, i.e., because of the added details that are obvious for humans but not quite so for LVLMs. The asymmetric strength hypothesis seems to suggest that the visual components alone are not quite up to the job of the VQA task.", "questions": "Please see the numbered points in weaknesses and comments.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The motivation is straight-forward, clear, and reasonable.\n- The improvements seem solid and the analyses support the improvements that come from the proposed pipeline.\n- The code and the data are provided and the authors promised public release. This is very important because the proposed pipeline is complicated and not easily reproducible.", "weaknesses": "- The pipeline seems overcomplicated and involves many steps that are indispensable to the overall performance. Unlike plain CoT, which usually conducts inference once, the proposed pipeline conducts inference multiple times using the LVLM and involves off-the-shelf tools twice. The complexity may affect the reproduction of the method and the incurred (computation and time) cost may hinder the adoption of the method in application.\n- The main results (Table 1) need more explanation. (1) For example, the standard deviation considering the oracle implementation is high. I did not expect that using the optimal candidate would lead to higher variance. Are there results regarding the choice of the number of the question candidates? (2) I would love to see more QA datasets (from diverse sources) tested on. (3) I wonder if the asymmetric strength hypothesis holds, is it possible that stage 2 and stage 3 can be changed to using the original question and the extracted details in texts without the image? (4) All analyses are based on BLIP-2, an encoder-decoder model. I don't think I find a discussion on the effect of model architecture (encoder-decoder or decoder-only).\n- The writing and the organization can be improved. Personally, I would like Section 3 Methodology to be more straightforward. From what I understand, Stage I (ii) adopts different post-processing from (i) and (iii). The paragraph before Section 3.2 states \"we prompt the LVLM to answer each question\" and the first paragraph in Section 3.2 states \"To select which question to answer\", which are contradictory. I had to check the appendix and the footnotes multiple times to guess what's going on.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)