VLIS: Unimodal Language Models Guide Multimodal Language Generation
Authors: Jiwan Chung
Youngjae Yu
Venue: EMNLP 2023 Main (2023)
Keywords: visual-language models
image captioning
multimodal understanding
language model decoding
TL;DR: For better multimodal language modeling, we decouple visual conditioning capability of visual-language models and fuse it with language understanding of unimodal language models.

Abstract: Multimodal language generation, which leverages the synergy of language and vision, is a rapidly expanding field. However, existing vision-language models face challenges in tasks that require complex linguistic understanding. To address this issue, we introduce Visual-Language models as Importance Sampling weights (VLIS), a novel framework that combines the visual conditioning capability of vision-language models with the language understanding of unimodal text-only language models without further training. It extracts pointwise mutual information of each image and text from a visual-language model and uses the value as an importance sampling weight to adjust the token likelihood from a text-only model. VLIS improves vision-language models on diverse tasks, including commonsense understanding (WHOOPS, OK-VQA, and ScienceQA) and complex text generation (Concadia, Image Paragraph Captioning, and ROCStories). Our results suggest that VLIS represents a promising new direction for multimodal language generation.

VLIS: Unimodal Language Models Guide Multimodal Language Generation
Jiwan Chung and Youngjae Yu Yonsei University
https://gituub.cmm///wwaatuuuu
Abstract
Named Entities
VLMs
VLIS
Multimodal language generation, which lever- ages the synergy of language and vision, is a rapidly expanding field. However, exist- ing vision-language models face challenges in tasks that require complex linguistic under- standing. address this issue, we introduce models as Importance Sam- pling weights @ VLIS), a novel framework that combines the visual conditioning capa- bility of vision-language models with the lan- guage understanding of unimodal text-only lan- guage models without further training. It tracts pointwise mutual information of each image and text from a visual- language model importance sampling
Who this?
Diego Maradona
soccer player
Yes
Does care for his family?
Not sure the man in sult
Corleone
does.
Distractors
VLMs
GPT
VLIS
Can ostriches fly?
Yes
No
No
X
VLMs
GPT
VLIS
Do chimpanzees
Yes
No
No
have tails?
and uses the value weight to adjust the token likelihood from a text-only model. language models diverse tasks, including commonsense understanding (WHOOPS, OK- VQA. and ScienceQA) and complex text gen- eration (Concadia, Image Paragraph Caption- ing. and ROCStories). Our results suggest that VLIS represents a promising new direction for multimodal language generation.
improves vision-
Figure 1: TOP: VLIS correctly recognizes named enti- ties, unlike the VLMs. Bottom: VLIS not deceived by the distractor images. Note that the images show seagull and monkey, not an ostrich and chimpanzee. VLIS inherits this favorable linguistic capability from a text-only language model (Touvron al., 2023; Zhang et al., 2022), and use VLMs a guide for visual align- ment. The examples are truncated for visualization purposes: we provide the full-text in appendix A.2.
1 Introduction
Visual Language Models (VLMs) extend unimodal text-only language models by conditioning their outputs on image context. Recent VLMs (Li et al., 2022a, 2023b; Wang al., 2022) can perform di- verse multimodal tasks from commonsense VQAs (Marino et al., 2019; Schwenk al., 2022) to in- context learning (Alayrac et al., 2022; Awadalla et al., 2023; Huang et al., 2023). Moreover, instruc- tion tuning with visual inputs (Liu et al., 2023; Li et al., 2023a; Dai et al., 2023) has improved the VLMs' responsiveness to even more extensive variety of tasks (Lu et al., 2022a; Yang et al., 2021). However, most VLMs only partially inherit the linguistic understanding capability of the unimodal models (Iki and Aizawa, 2021). We here illustrate two intriguing failure cases of the recent VLMs,
using both strong image captioning model (BLIP- (Li et al., 2023b)) and instruction-tume model (LLAVA (Liu et al., 2023)). Firstly, VLMs avoid specifying named entities. The upper examples of Figure 1 show the VLM failing describe pub- lic figure (Diego Maradona) or movie character (Don Corleone). The problem not the lack knowledge: after applying our zero-shot method (VLIS), the VLM tells the names. We further inves- tigate this phenomenon in the landmark recognition experiment in appendix A.I.
Secondly, VLMS rely on the image context, even when they should not. The lower examples of the same figure show the VLM being misled by im- age context to deny commonsense knowledge. The

---

Question
Wh cause the color of the vehicle?
Text-Only LM (text)
Pointwise Mutual Information
Importance Sampling
A: because has been painted green.
Ptexe eMM(()
Porl A
VLM (ot)
VLM (wt)
log
Porl A
A: its original paint wom off due old age
(a) Standard VLM Decoding
(b) VLIS
Figure 2: Comparison of VLIS and standard VLM decoding process. Using the VLM, we first obtain the image- conditional Pet (Answeerle (magge) and text-only likelihood Par( Answer) given an image and a prompt or question. Then, we compute the exponentiated pointwise mutual information (PMI) with the likelihoods. Finally, the exponentiated PMI score is used as the importance weights for the text-only model likelihood Preae (reze(Arnn (Answer).
questions not unanswerable: the text-only lan- guage model without the image context answers both correctly. We provide more samples on visual distraction in appendix A.2.
periments consist of various tasks that require both reliable language modeling and strong vi- sual conditioning, including weirdness identifica- tion (WHOOPS (Bitton- Cetton et al., 2023)) and commonsense VQA (OK-VQA (Marino et al., 2019), ScienceQA (Lu et al., 2022a)), extended image captioning (Concadia (Kreiss et al., 2022) and Image Paragraph Captioning (Krause et al., 2017)), and open-ended generation (ROCSto- ries (Mostafazadeh al., 2016)), Compared to the dataset-specific state-of-the-an baselines and the base VLMs, VLIS improves linguistic capabilities such as responsiveness to prompts while maintain- ing visual conditioning according to a comprehen- sive set of evaluation metrics.
Hence, the linguistic capabilities of the VLMs are not optimal yet. On the other hand, the unimodal text-only language models them- selves (Brown et al., 2020; Touvron et al., 2023) show reliable linguistic understanding and known for their knowledge understanding (Petroni al., 2019; Meng et al., 2022) and complex reasoning capabilities (Kojima et al., 2022; Qin al., 2023). Hence, it becomes rcasonable to delegate the bur- den of language modeling to the text-only models.
To this end, we propose Visual-Languee mod- els as Importance Sampling weights ( VLIS) as plug-ande blug-and-play method to enhance the unreliable linguistic understanding of the VLMs. When gen- erating each text token, VLIS follows the token likelihoods of the unimodal text-only language model. Furthermore, VLIS multiplies importance sampling (Tokdar and Kass, 2010) weights derived from a VLM to provide the visual alignment sig- nals. To isolate the visual conditioning capabil- ity of the VLMs from their language modeling preference, incorporate the exponentiated point- wise mutual information (PMI) (Church and Hanks, 1990) of the image context and the current text to- ken as the weights. As result, VLIS can maintain the favorable language modeling capability of the text-only model and control the visual conditioning strength simultaneously.
VLMs as Importance Sampling Weights
We propose Visual-Language models as Impor- tance Sampling weights (VLIS) to harmonize the visual conditioning capability of the VLMs with the linguistic fluency of the text-only language mod- els. We provide the intuition behind our approach in $2.1, describe our token-level visual alignment scores $2.2, and combine the said scores with the text-only model via importance sampling in $2.3.
2.1 Intuition
Many recent Visual Language Models (VLMs) (Li et al., 2023b; Alayrac et al., 2022; Liu et al., 2023) are often built on top of text-only language mod- els (Lyer et al.. 2022: Hoffmann et al., 2022; Tou- vron et al., 2023). At each timestep t, the per-token likelihood of the autoregressive text-only language models is modeled as Ptezz (x, [x))) where a de- notes text token. To build VLM Put. one can finetune the text-only model on data consisting of paired image e and text x with maximum likeli- hood estimation as the objective.
We evaluate VLIS on two VLM backbones to test whether VLIS is effective both when the lan- guage modeling capability of the VLM is weaker than that of the text-only model (BLIP-2 (Li et al., 2023b)) and when the VLM is expected to model language well owing to the visual instruction tun- ing process (LLAVA (Liu et al., 2023)). Our ex-
0ut ~ argming Fi(z,)ssl- (o)c) (1)

---

However, while this objective only maximizes the image-conditional likelihood Pur(z]l)) it may lead to unpredicted artifacts in the marginal likeli- hood (x;) that does not depend on any particular image. For example, image captioning models are known to not only reflect but also amplify the social bias present in the training data (Hendricks et al.. 2018), distort the original language model's com- monsense knowledge described in $1.
practice, we use two images: a black-filled image Cy and white-filled image Cw-
1
E pulassleet (4)
cElebitul
This efficient alternative works reasonably well in practice and is used in all experiments. As result, VLIS runs three forward passes of VLM (one for the conditional likelihood and two for the marginal likelihood) and single pass of the text- only model on each step of the generation process. We explore more choices of selecting the marginal image set later in appendix C, which shows that our specific set of images provides reasonable trade-off between generation quality and inference time.
We henceforth seek to extract the visual condi- tioning capability the VLMs isolated from their dubious language modeling skills.
2.2 Extracting Visual Weights
Here, we aim to find quantity that extracts the visual conditioning strength of VLM stripped of its language modeling preference. We employ Pointwise Mutual Information (PMI) (Church and Hanks, 1990), which measures the association be- tween two events (text and image in our setting). On each step, we want to compute the PMI between the image context and the next token I; given the previous text context Tezt
2.3 Computing VLIS Scores
We start from the token likelihood of text-only lan- guage models preat (at |c, x<(). To control the de- gree of confidence in the text-only models' deci- sions, we introduce a language temperature r smooth or smothh the text-only distributions:
Pur cxe))
PMI( = log
(2)
Prezt Tet) & (rale x<a)
(5)
Pel (Tele, IEE))
Then, we multiply the exponentiated PMI intro- duced in $2.2 with the likelihood for better visual alignment. VLIS decides the next token 2, with the following score f(x)):
log Pot (xt
(3)
eq. (3) reformulates the definition in eq. (2) for better tractability. The numerator is the image- conditional likelihood of the VLM and is easily obtainable. However, the denominator requires marginalization over the image context c. We enu- merate three proxies below that bypass the exces- sive computation required to obtain the expectation over all possible images.
f(z)) Prent Zee1) PMI( (6)
Pet PellFrle,
PPeat(Fele, 2<))
(7)
Pel (Ft
Written as eq. (7), VLIS performs importance sampling of the smoothed text-only model like- lihood Ptext- Importance sampling (Tokdar and Kass, 2010) is Monte-Carlo method of estimat- ing a quantity v(x) from the nominal distribu- tion p(x) with samples from another distribution called importance distribution q(æ). The estimated quantity here is the text-only model likelihood Prest (Tz), the nominal distribution is the VLMs' image-conditioned likelihood (xzlo,)) and the im- portance distribution is the marginal (x+)
Approximating the marginal. The first approx- imation is training separate text-only model with the VLMs* training data S. Considering the mas- sive scale of dataset S, this option requires a consid- erable burden of additional training. Also, there is no guarantee that the newly trained model will ac- curately estimate the marginal likelihood due to the additional complexity of training another model. The second option is using sample mean of the pre-selected image set as proxy to the real mean. Lastly, the score for only one two images might suffice as the sample image
Elf(x:) P] F E p(a1) ((xz)) (8)
v(x)) = Pr
We use the last method with the least computa- tional overhead. Here, the sample set is a tiny set of images with close to no visual information. In
p(x)) = (o((
((x)) = (o((x((21))

---

Implementation--iss,, we replace the expecta- tion with a single sample (current generated text), Thus, VLIS effectively treats the current token candidate as sampled from the VLMs' marginal (xt( and reweigh its importance with the VLMs' conditional (z))c.)))
Models
Pipe
0-shot
Acc (%)
Chance
50
BLIP-2
50
BLIP-2
73
Model Caption
59
GT Caption
74
Fluency masking. The log visual weights (MI(I,,c/Z<i) of VLIS is log-likelihood ratio and is unbounded. Hence, some extreme cases, such as tiny values of the marginal likelihood Pot (o((ae))))) may overrule the language generation process of the text-only model, yiclding degenerate text. To prevent such text degeneracy, we apply a fluency mask to our importance sampling score f ( c): only the tokens with text-only likeli- hood larger than the threshold a are allowed to be selected. We omit the dependency on the contexts Eett, in the equation below for simplicity.
VLM (LLAVA)
59
VLM (Lynx)
V
71
Ours (LLAVA)
V
73
Ours (Lynx)
V 80
Table 1: Results in the identification of weird images task of WHOOPS dataset (Bittonn-Guetta et al., 2023). Pipe represents further pipelining with GPT3 and 0- shot denotes a zero-shot method. The best numbers are bolded and the second best ones are underlined.
1.3B (lyer et al., 2022) as our backbones.' 1 Note that the choices model pairs are intentional: we impose similar computational requirements on both the VLM and the text-only model to limit the ad- ditional computational burden of VLIS. In both cases,
f(m.) f(xt), if E Viop inn, otherwise
(9)
Viop (x))Pten((r)) 2 a}
(10)
the base VLM general baseline to evaluate the gain from VLIS. Also, to verify the contribution of the PMT weights, we implement Naive Ensemble which simply multiplies the token likelihood of the VLM and the text-only model.
Intuitively, this mask filters out any token can- didates the text-only model sees as the next token with a probability lower than a. We fix the fluency threshold to a = 0.001 in all experiments except for alternative architecture (appendix E). Still, VLIS is not overly sensitive to the specific value of the fluency threshold. We conduct a hyperparame- ter search experiment to verify this in appendix D.
Evaluation metries. We evaluate closed-ended questions with binary (WHOOPS) and multi- choice (ScienceQA) accuracy. The open-ended VQAs (OK-VQ VQA and VQAv2) use the task-specific VQA metric (Antol et al., 2015).
The token that maximizes this final score f(ri). c, x is greedily selected as the next token. When VLIS is combined with other decoding meth- ods, such as beam search, the score substitutes the original token likelihood as per-token scores.
3.1 Identification of Weird Images
WHOOPS (Bitton-Guctta et al., 2023) is visual commonsense benchmark to check VLM's capa- bility understand images that defy commonsense. We adopt identification of weird images, subtask of the WHOOPS benchmark, which tasks a model to discriminate potentially weird images.
3 Experiments: Describing Facts
We verify that VLIS can alleviate the factual inac- curacy concern raised in Figure with various mul- timodal benchmarks: weirdness identification $3.1, commonsense understanding $3.2, and scientific reasoning $3.3. VLIS consistently outperforms the backbone VLM and shows comparable factual cor- rectness to the strong baselines.
Approach and Baselines. Following the origi- nal paper (Bitton-Guetta et al., 2023), we employ pipelining to turn the original binary classifica- tion problem into deseription generation prob- lem. Specifically, pipelining means that model first generates oolllnattion-o--volattt (EoV) de- scription of the given two images, which is then
Experimental setups. We explore two experi- mental setups. Our experiments on the WHOOPS dataset incorporate LLAVA (Liu et al., 2023) and Lynx (Zeng et al., 2023) as the VLMs and Vi- cuna 7B (Chiang et al., 2023) as the text-only model. In the VQA experiments, we use BLIP- 2 OPT 2.7B (Li et al., 2023b) and OPT IML Max
WWe assign different tasks for different backbones for fair comparisons: BLIP-2 fails to generate long explanation-of- violation since it is only trained on short captions, while is not trivial to evaluate AVA on short-answer VQAs in zero shot manner due to its tendency to generate long explanations.

---

What is unusual about this image?
Models
V L
OKVQA
VQAv2
VLIS
Few
16.5
47.7
Albert Einstein holding a smuriphone unusual because he lived before the invention of such technology.
Frozen
5.9
29.6
VLKD
13.3
42.6
VLM
BLIP-2
31.7
53.5
not commmn to see a professional in field science using cell phone.
OPT-IML
19.1
36.0
Naive
26.6
34.6
VLIS
Ensemble
Duck swimming alongside baby rubber ducks in the image unusual they not real animal.
Ours
34.2
53.6
Table 2:
Results in the validation set of OK-
VLM
VQA (Marino et al.. 2019) and VQAv2 (Goyal et al.. 2017). V denotes using VLM and I. denotes using unimodal language model.
not very common to see large with three ducklings
Figure 3: Qualitative samples from WHOOPS (Bitton-
Guetta al., 2023) experiments. As marked green, specific descriptions are required to explain weirdness.
Models
IMG
TXT
NO
ALL
UnifiedQA Amall
44.1
50.2
44.5
45.8
UnifiedQA
48.1
53.1
46.7
48.5
processed the off-the-shelf text-only classi- fier GPT3 (Brown et al., 2020) yield bi- nary decision on which image is weird. We use VLIS to generate such EoV descriptions. The pipelined baselines include EoV from the backbone VLM (LLAVA), conventional machine machine-generated captions, and ground-truth captions from the WHOOPS dataset. We also include pipeline-less BLIP-2 (both supervised and zero-shot) as base- line. The same prompt we used for both VLIS and the backbone VLM is illustrated in appendix F. Results. Table and Figure 3 presents results with LLAVA (Liu et al., 2023), an instruction-tuned VLM. VLIS- "LIS-generated weirdness explanations per- form on par with the ground-truth captions, which
GPT-3
65.7
74.2
79.6
74.0
BLIP-2
35.5
34.6
24.2
28.2
OPT-IML
45.4
52.2
49.8
49.0
Naïve Ensemble
45.9
53.6
49.7
49.7
Ours
49.3
53.1
49.1
50.2
Table 3: Zero-shot results on ScienceQA test set (Lu al., 2022a). IMG denotes subset with image context, TXT the text context subset, and NO the subset without any context.
VQA (Marino et al.. 2019) as exam- ple of commonsense commonsens--augmeeedd VQA and VQAv2 (Goyal et al., 2017) visually intensive VQA problem. We compare VLIS with strong VLM models, including FewVLM (Jin et al., 2022), Frozen (Tsimpoukelli et al., 2021), and VLKD (Dai et al., 2022).
are manually annotated to contain details necessary identify the strangeness. Also, method zero-shot method shows comparable performance the supervised baseline BLIP-2. Interestingly, LLAVA alone cannot outperform conventional cap- tions, even with instruction tuning and prompting.
Results: commonsense knowledge. the OK- VQA (Marino al., 2019) experiment in Table 2, we show that VLIS achieves meaningful develop- ment over the backbone VLM (BLIP-2). Also, the text-only backbone (OPT-IML) and Naïve Ensem- ble perform substantially worse, proving that VLIS is not just imitating the text-only model outputs. Instead, VLIS adaptively fuses the commonsense understanding capability of the text-only model with the visual conditioning of the VLM.
3.2 Commonsense Understanding
Unimodal language models embody commonsense knowledge (Petroni al., 2019; Davison et al., 2019; Tamborrino et al., 2020). If VLIS can in- herit this commonsense understanding capability, it would outperform the base VLM tasks requir- ing both commonsense and visual understanding. Here, we examine this possibility with common- sense VQA benchmark of OK-VQA (Marino et al., 2019). Further, VLIS is also shown to maintain visual specificity in VQAv2 (Goyal et al., 2017). Approach and baselines. We use OK-
Results: maintaining visual specificity. When VQAs do not require text-based reasoning, VLIS should focus on visual conditioning only. The right- most column of Table 2 summarizes results on VQAv2 (Goyal et al., 2017) dataset, a VQA dataset that has its textual bias intentionally removed. As

---

Model
Zeroshot
Cap
Desc
4
Experiments: Text Generation
Kreiss et al.
11.3
17.4
In addition to factual knowledge, text-only lan- guage models manifest two critical capabilities: following prompt instructions and generating flu- ent and diverse text. We demonstrate that VLIS extends these qualities the visual domain with contextualized captioning ($4.1), paragraph cap- tioning ($4.2), and visual story generation ($4.3).
Socratie Model
38.9
22.6
BLIP-2
V
20.0
30.6
Naïve Ensemble
V
24.7
18.4
Ours
44.1
28.3
Table 4: Results on Concadia (Kreiss et al., 2022) test set. Cap denotes caption and Desc description annota- tions. We report CIDEr following the literature.
Metries. Both captioning benchmarks use au- tomatic text metrics, including CIDEr (Vedantam et al., 2015), METEOR (Banerjee and Lavie, 2005), and Bleu-4 (Papineni al., 2002). In the open- ended generation problem of visual storytelling, we use various fluency metrics (2- 2-gram repetition, di- versity, coherence, MAUVE (Pillutla al., 2021)) and a visual strength metric (CLIPScore (Hessel et al., 2021)). Refer to (Su et al., 2022a) for details on the fluency metrics.
Model
Shots
M
C
B4
Krause et al.
Full
16.0
13.5
8.7
Liang al.
Full
17.1
16.8
9.0
SCST SCST Rep. Penalty
Full
13.6
13.8
5.9
Full
17.9
30.6
10.6
HSGED
Full
18.3
36.0
11.3
PaG-MEG-SCST
Full
18.2
29.4
11.5
BLIP-2
3
10.8
6.5
4.9
OPT-IML
3
9.5
2.5
2.2
Naïve Ensemble
3
9.8
6.0
3.6
4.1
Contextualized Captioning
Ours
14.6
14.8
6.4
Concadia (Kreiss al., 2022) is an image caption- ing dataset with the additional context of a para- graph from the Wikipedia article. The dataset pro- vides two types of annotations: caption, which takes the article into account and description. which ignores the article context.
Table 5: Results the Paragraph Captioning (Krause et al., 2017) test set. M denotes METEOR, C CIDEr, and B4 Bleu-4 scores.
shown in the VQA score, VLIS (Ours) preserves the VQA capability of the backbone VLM (BLIP-2). Note that Naïve Ensemble falls behind the text-only backbone (OPT-IML), offering a poor trade-off be- tween visual and linguistic understanding.
Approach and Baselines. Following the orig- inal evaluation scheme (Kreiss et al., 2022), we generate single text to compare against both the ground-truth caption and description. We include both supervised (Kreiss et al., 2022) and zero-shot (Socratic Model (Zeng et al., 2022)) baselines. Result. In Table 4, VLIS outperforms the Socratic Model (Zeng al., 2022) implementa- tion based on stronger language model (GPT-3 175B (Brown et al., 2020)). Interestingly. the base VLM (BLIP-2) and VLIS (Ours) show a completely different text style. VLIS captions are better aligned with caption-style, showing that our method re- flects the Wikipedia article better than the baselines.
3.3 Scientific Reasoning
ScienceQA (Lu al., 2022a) evaluates multimodal science reasoning cupability. Here, the goal of VLIS would be to improve the answers the pres- ence of image contexts (IMG) and preserve the answers from the text-only model in the absence of such visual context (TXT and NO).
Baselines. We compare zero-shot VLIS against zero-shot baselines including a VLM (Uni- fiedQA (Khashabi al., 2020)) and text-only language model (GPT-3 (Brown et al., 2020)).
On the other hand, the VLM generates description- style texts better. Still, VLIS captions are similar to the visually intensive caption (description) com- pared to all other baselines except for the VL.M.
Results. Table 3 demonstrates the findings in ScienceQA. On IMG split, VLIS significantly im- proves the text-only OPT-IML and Naïve Ensemble baselines. Also, VLIS maintains the performance of the text-only backbone in TXT and NO split. Finally, the base VLM (BLIP-2) falls behind by wide margin, indicating that solid language under- standing is necessary for scientific reasoning.
4.2 Paragraph Captioning
Image Paragraph Captioning (Krause al., 2017) has paragraph-long captions that describe the im- age in finer detail than sentence-level captions. Approach and baselines. We saw that neither the VLM nor the text-only model could follow the

---

Models
rep-24
div.f
coh.1
Mauvet
CLIP1
(a)
Question: What kind of dog is in this
Cont. Search
2.60
0.34
0.86
0.65
picture?
MAGIC
2.49
0.38
0.85
0.68
rothweiler
BLIP-2
24.26
0.39
0.32
0.47
VLIS: rotlweeler
Naive Ensemble
1.85
0.98
0.27
0.93
BLIP-2: dog
Ours
2.31
0.97
0.38
0.96
0.72
OPT: pug
Naive Ensemble: a dog
Table 6: Results in the ROCStories story generation dataset (Mostafazadeh et al., 2016). rep-2 denotes 2- gram repetition, div. diversity, coh. coherence, and CLIP. CLIPScore. Higher is better except for rep-2.
(b)
Question: W Which one of these animals
native north america?
GT: deer
VLIS: deer
BLIP-2: zebra
OPT: wolf
Naïve Ensemble: zebra
style of the ground-truth annotation in early experi- ments. Hence, provide the model with three in- context examples (3-shot). Note that the setting is still much more challenging compared that of the fully supervised baselines (Krause at el. (Krause et al., 2017). Liang et al. (Liang et al., 2017), SCST with repetition penalty (Melas-Kyriate et al., 2018), HSGED (Yang et al., 2020), and PaG-MEG- SCST (Nguyen and Fernando, 2022)).
(c)
Question: What material is burning?
GT: wax
VLIS: peper
BLIP-2: umbrella
OPT: wood
Naive Ensemble: the material is
burning
(d)
Q Question: What ocean is being surfed?
GT: atlantic, pacific
VLIS: water
BLIP-2: the ocean
Results. As visible in Table 5, VLIS greatly improves the base VLM (BLIP-2) to generate para- graph captions comparable to the supervised base- lines. We provide an interprctation this improve- ment in qualitative samples in appendix G: VLIS shows less text degeneracy than the base VLM, while keeping visual hallucination at minimum unlike Naïve Ensemble.
OP.: ocean
Naive Ensemble: the
Generation results in the OK-VQA dataset (Marino et al., 2019). We color the intention of the question green and answers that defy such intention with red. (c) and (d) are failure cases.
Figure 4:
nally, while the base VLM (BLIP-2) shows high image-text correspondence as reflected in high CLIPScore, it cannot generate an articulate story as its low performance on other scores shows.
4.3 Story Generation
Story generation is an open-ended generation task. To excel at it, VLIS should generate open-ended text without falling into text degeneracy, all the while staying close to the image context.
Qualitative Results
Approach and baselines. Unlike previous ex- periments, here we use a supervised text-only model (Su et al., 2022b) finetuned on text-only ROCStories (Mostafazadeh et al., 2016) dataset. Hence, we can safely assume that this specialist text-only model knows the language "better" than the VLM in story generation. We include both visually-conditioned (MAGIC (Su et al., 2022a)) and text-only (Contrastive search (Su and Collier, 2023)) baselines. Refer to appendix B for more details on the baseline results.
Commonsense Understanding. Figure il- lustrates zero-shot results in the OK-VQA dataset (Marino et al., 2019). In (a) and (b), the baselines including the base VLM and Naïve En- semble fail to understand the intention of the ques- tion (kind of dog and native to north america). While the text-only model understands the question better and suggests plausible answer candidates (pug and wolf), has no access the visual inputs and ultimately outputs an incorrect answer. On the other hand, VLIS sensibly combines commonsense reasoning and visual context.
Results. Table 6 presents the results of open- ended story generation. VLIS outperforms both Contrastive Search and MAGIC in all metrics. While Naïve Ensemble builds more diverse text (rep-2 and div.), its severely low coherence score suggests that its stories are less consistent, as rep- resented qualitative samples of appendix G. Fi-
Results for images (c) and (d) depict the failure cases. In (c), VLIS follows the reasoning process of the text-only language model to deduce that the answer should
type of material. However, as the VLM focuses on the frontal object (umbrella), VLIS wrongly concludes the answer is the mate-

---

Hey, It's
Today went VLIS: to the library to pick up book for my and found out that they have special section for children with special needs. BLIP2 museum with my son.
Here is romattic message. Answer: VLIS: love you and want to spend the rest of my life in front of you smilling and touching BLIP-2 love you mom
VLIS: your son's teucher, and I'm going to show you picture of me and your son, II.P2 person.
Hey, Ir& me
Today went
Here romantic message, Answen VLIS; Dear bearjohn John, I love you. You are the apple of my eye. Yours truly BI HLIP2 you. you.
VLIS: your friend. have an apple computer and I'm going be at the convention weeks,
VLIS: to an Apple store, bought iMac,
keyboard, and a trackped for my wife's computer
BLIP2 eynthla, I'm computer scientist.
BI 1.2 conference.
Figure 5: Open-ended generation results with BLIP-2 (Li et al., 2023b) as the base VLM. We use three text prompts (Hey, It's me, Today went, and Here is romantic message. Answer:) test whether VLIS can actively adjust its response according to the text prompt while maintaining visual alignment.
rial of that object (paper, which is coincidentally flammable material well). In (d), the text- only model produces an incoherent output (ocean). VLIS inherits this misinterpretation and likewise generates an incorrect answer (water). conclu- sion, VLIS induces coordination of the VLM's vi- sual specificity and the text-only model's s common- sense understanding but carries on the modeling insufficiency of the individual modalities.
eration. ZeroCap (Tewel al., 2022) uses the gradient signal from a pretrained image-text align- ment scorer (CLIP (Radford et al., 2021)) up- date the language model's memory. Magic (Su et al., 2022a) also utilizes CLIP. Unlike ZeroCap and Magic, VLIS utilizes autoregressive VLMs (Li et al., 2023b), rather than CLIP.
Language Model Decoding. Language model decoding is the process of generating text from pretrained language model. Traditional decod- ing methods use greedy decoding and beam search to find the most likely sequence of words. The truncated sampling algorithms such as Top K sam- pling (Fan et al., 2018; Holtzman et al., 2018; Rad- ford et al., 2019), Nucleus sampling (Holtzman et al., 2020), and Typical P sampling (Meister al., 2022) have been proposed to avoid text degener- acy. Recent deterministic algorithms, such as Con- trastive decoding (Li et al., 2022b) and contrastive search (Su et al., 2022b; Su and Collier, 2023). provide a better trade-off between text fluency and model likelihood. Neurologic (Lu et al., 2021) and Neurologic A **sqque decoding (Lu et al., 2022b) control the language models to include given words in their outputs. As shown in the experiments, VLIS can be used jointly with any decoding method, in- cluding beam search and contrastive search.
Open-Enn Ended Generation. Lastly, we demon- strate the open-ended generation capability of VLIS in Figure 5. Here, VLIS should condition its output on the diverse text prompt and the image. Unlike the base VLM, it clings tighter to the prompt and produces realistic self-introduction (hey, it's me), personal journal (loday / wenl), and romantic mes- sages (here is romantic message. answer:). Also, VLIS plays pun on the word apple (see apple lap- top in the image and apple of my eye). Refer to appendix G for more baseline samples.
6 Related Work
Combining VLMs with text-only LMs. Early large-scale VLMs (LXMERT (Tan and Bansal, 2019), VisualBERT (Li et al., 2019) and ViL- BERT (Lu et al., 2019)) saw the benefits of text-only pretraining by initializing their text en- coder with a masked language model BERT (Ken- ton and Toutanova, 2019). Later, Frozen (Tsim- poukelli et al., 2021) started a trend of freezing the language model and learning only the vision- language relationship. More recent models such Flamingo (Alayrac et al., 2022) and BLIP-2 (Li et al., 2023b) also freeze the image encoder. ES- PER (Yu et al., 2022) uses reinforcement learning combine image encoders with language models, Better aligned with our approach are decoding- oriented methods for image-conditioned text gen-
7 Conclusion
We propose VLIS, a novel framework to alleviate the language modeling burden of visual- language models (VLMs). VLIS combines the linguistic un- derstanding capability of the text-only language models with the visual conditioning strength of the VLMs by importance sampling. To isolate the VLMs* visual conditioning power, VLIS uses pointwise mutual information to suppress their text-only marginal distribution. Our framework

---

enhances the base VLM in commonsense reason- ing (WHOOPS (Bitton-Guclta Guetta et al., 2023), OK- VQA (Marino et al., 2019), and ScienceQA (Lu et al., 2022a)) and complicated text generation (Concadia (Kreiss et al., 2022), Image Paragraph Captioning (Krause et al., 2017), and ROCSto- ries (Mostafazadeh et al., 2016)) problems. In the future, VLIS can be extended to incorporate other modalities for which the paired multimodal data is even scarcer. We hope that VLIS sparks an inter- est in better utilization of off-the-shelf multimodal pretrained models.
Finally, VLIS can be short-sighted. The method combines the outputs of the VLMs and the text- only models at the very last stage of token likeli- hood. As a result, VLIS score might be misleading when both models assign high probabilities to the same token for different reasons (e.g. homophones). It may help to estimate scores for the future gener- ated text by rolling out few generative steps and aggregating the output (Lu et al., 2022b), which we leave future works.
Acknowledgement
This work was supported by Institute of Informa- tion & communications Technology Planning & Evaluation (ITP)) grant funded by the Korea gov- ermment (MSIT) (No.2020-0-0136)) 361), Institute for Project-Y, Y, and NCSOFT Vision/NLP Center.
8 Ethical Considerations & Limitations
Potential ethical concerns. As an inference time method, VLIS inherits some known problems of both the VLMs and the unimodal text-only lan- guage models as well:
Hallucination: VLMs are known to hal- lucinate information absent in the training data (Rohrbach et al., 2018). While VLIS may strengthen visual conditioning and thereby contribute to reducing the rate of visual hallu- cination, completely eradicating it is beyond the scope of this research.
References
Jean-Baptiss Alayrac, Jeff Donahue, Pauline Luc, An- toine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. 2022. Flamingo: a visual language model for few-shot learning. arXiv preprint arXiv:2204. 14198.
Stanislaw Antol, Aishwarya Agrawal. Jiasen Lu, Mar- garet Mitchell, Dhruv Batra, C. Lawrence Zitnick, and Devi Parikh. 2015. VQA: Visual Question An- swering. In International Conference on Computer Vision (ICCV).
Social bias: It widely known that VLMs reflect or even amplify (Hendricks et al., 201 8; Hirota et al., 2022) social bias (e.g. gender or race) in the training data. We have yet to determine how VLIS affects social bias in the base models. Thus, outputs generated using VLIS may contain social bias.
Anas Awadalla, Irena Gao, Joshua Gardner, Jack Hes- sel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Jenia Jitsev, Simon Kornblith, Pang Wei Koh, Gabriel Ilharco, Mitchell Wortsman, and Ludwig Schmidt. 2023. Open- flamingo.
is a mcaningful direction to combine VLIS with reinforcement learning (Ramamurthy et al., 2023; Yu et al., 2023) or reward- based decoding al- gorithm (Su et al., 2022a) to alleviate the problems above, but leave that to future research.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metrie for MT evaluation with im- proved correlation with human judgments. In Pro- ceedings of the ACL Workshop on Intrinsic and Ex- trinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72.
Limitation of VLIS and future work. Firstly, we acknowledge that this paper only explores a small fraction the possible combinations of text- only models and VLMs. A harge-scale wide search in this regard would reveal 1) the better-performing pairs of text-only LM and VL.M and 2) the required characteristics of good model pair.
Nitzan Bitton- Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici. Gabriel Stanovsky. and Roy Schwartz. 2023. Breaking common sense: Whoops! vision-and rision-and-languaaa benchmark of syn- thetic and compositional images. arXiv preprint arXiv:2303.0727.
Secondly, VLIS could be extended more modalities than the image-to-text generation prob- lem covered here. Other modalities, such as audio and document may also benefit from applying VLIS to their modality-spedd speccifc foundational model.
Tom Brown, Benjamin Mann, Nick Ryder, Melanic Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. 2020. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877-1901.

---

Wei-Lin Chiang. Zhuohan Li. Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric Xing. 2023. Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality.
Yusuke Hirota, Yuta Nakashima, and Noa Garcia. 2022. Quantifying societal bias amplification in image cap- tioning. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 13450-13459.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Men- sch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, al. 2022. Train- ing compute compute-optimal large language models. arXiv preprint arXiv: 15556.
Hyung Won Chung, Le Hou, Shayne Longpre, Bar- ret Zoph. Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022. Scaling instruction- finetuunnd language models. arXiv preprint arXiv: 12..22220...10 11416.
Kenneth Church and Patrick Hanks. 1990. Word associ- ation norms, mutual information, and lexicography. Computational linguistics, 16(1):22-29 29.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text de- generation. In International Conference on Learning Representations (ICLR).
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. 2022. Enabling multimodal generation on clip via vision-language knowledge distillation. In Findings of the Association for Com- putational Linguistics: ACL 2022, pages 2383-2395.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018, Learn- ing to write with cooperative discriminators. In Pro- ceedings of the 56th Annual Meeting of the Associa- tion for Computational Linguistics (Volume Long Popers) (ACL), pages 1638-1649.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. 2023. In- structblip: Towards general-purpose vision-language models with instruction tuning.
Shaohan Huang. Li Dong, Wenhui Wang, Yaru Hao. Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. 2023. Language not all you need: Aligning perception with language models. arXiv preprint arXiv:2302.14045.
Joe Davison, Joshua Feldman, and Alexander M Rush. 2019. Commonsense knowledge mining from pre- trained models. In Proceedings of the 2019 con- ference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages I 1173-1178.
Taichi Iki and Akiko Aizawa. 2021. Effect of visual ex- tensions on natural language understanding in vision- and aud-language models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan- guage Processing, pages 2 2189-2196.
Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. 2022. Llm.int8(): 8-bit matrix multi- plication for transformers at scale. arXiv preprint 23X0:2208.07339.
Srinivasan Iyer. Xi Victoria Lin, Ramakanth Pasunuru, Todor Mihaylov, Dániel Simig, Ping Yu, Kurt Shus- ter, Tianlu Wang, Qing Liu, Punit Singh Koura, al. 2022. Opt-iml: Scaling language model instruc- tion meta learning through the lens of generalization. arXiv preprint arXi::2212.120117
Angela Fan, Mike Lewis. and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (ACL), pages 889-898.
Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. 2022. A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-langual models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 2763-2775.
Yash Goyal, Tejas Khot, Douglas Summers- Dhruv Batra, and Devi Parikh. 2017. Making the in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference compuler vision and patlern recognition (CVPR), pages 6904-6913.
Lisa Anne Hendricks, Kaylee Burns, Kate Saenko, Trevor Darrell, and Anna Rohrbach. 2018. Women also snowboard: Overcoming bias in captioning mod- els. In Proceedings of the European conference on computer vision (ECCV), pages 771 771-787.
Jacob Devlin Ming- Wei Chang Kenton and Lee Kristina Toutanova. 2019. Bert: Pre-training of deep bidirec- tional transformers for language understanding. In Proceedings of NAACL-HIT,, pages 417 177116166
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. 2021. Clipscore: reference-free evaluation metric for image captioning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7514-7528.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sab- harwal, Oyvind Tafjord. Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format bound- aries with a single qa system. In Findings of the Association for Computational Linguisties: EMNLP 2020, pages 1896-1907.

---

Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yu- taka Matsuo, and Yusuke Iwasawa. 2022. Large language models are zero-shot reasoners. ArXiv, abs/2205.11916.
Ximing Lu, Sean Welleck, Peter West, Liwei Jiang. Jungo Kasai, Daniel Khashabi, Ronan Le Bras, Lian- hui Qin, Youngjae Yu, Rowan Zellers, et al. 2022b. Neurologic a* esque decoding: Constrained text gen- eration with lookahead heuristics. In Proceedings of the 2022 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 780-799.
Jonathan Krause, Justin Johnson, Ranjay Krishna, and Li Fei-Fei. 2017. A hierarchical approach for gener- ating descriptive image paragraphs. In Proceedings of the IEEE Conference on Computer Vision and Pat- tern Recognition (CVPR), pages 317-325.
Ximing Lu, Peter West, Rowan Zellers. Ronan Le Bras. Chandra Bhagavatula, and Yejin Choi. 2021. Neuro- logic (ccoding.(un) supervised neural text generation with predicate logic constraints. In Proceedings of the 2021 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, pages 4288 4299..
Elisa Kreiss, Fei Fang, Noah Goodman, and Christopher Potts. 2022. Concadia: Towards image-base based text generation with a purpose. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4667-4684,
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang. Jingkang Yang. and Ziwei Liu. 2023a. Otter: multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi. and Roozbeh Mottaghi. 2019. Ok-vqa: visual ques- tion answering benchmark requiring external knowl- edge. In Conference on Computer Vision and Pattern Recognition (CVPR).
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023b. 2: Bootstrapping language-iman gaguuge--maaeeeae pre- training with frozen image encoders and large lan- guage models. arXiv preprint ar 2iv:2330.11299
Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2022. Locally typical sampling. Transac- tions of the Association for Computational Linguis- tics (TACL), 11:102-121.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. 2022a. Blip: Bootstrapping hanguag--image pre-training for unified vision-language understand- ing and generation. In International Conference on Machine Learning, pages 12888- 12900. PMLR.
Luke Melas SerlasKyyiizzi. Alexander M Rush, and George Han. 2018. Training for diversity in image paragraph captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Process- ing (EMNIP)), pages 757-761.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019. Visualbert: A sim- ple and performant baseline for vision and language. arXiv preprint
Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022. Locating and editing factual associ- ations in gpt. Advances in Neural Information Pro- cessing Systems, 15:17399-17372.
Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2022b. Contrastive de- coding: Open-ended text generation as optimization. arXiv preprint 1..x::2210.1099)
Nasrin Mostafazadeh, Nathanael Chambers, Xiaodong He, Devi Parikh, Dhruv Batra, Lucy Vanderwende, Pushmeet Kohli, and James Allen. 2016. A corpus and cloze evaluation for deeper understanding of commonsense stories. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL), pages 399-899.
Xiaodan Liang. Zhiting Hu, Hao Zhang, Chuang Gan, and Eric P Xing. 2017. Recurrent topic-transition gan for visual paragraph generation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3362-3371.
Thanh-Son Nguyen and Basura Fernando. 2022. Ef- feetive multimodal encoding for image paragraph captioning. IEEF Transactions on Image Processing, 3 116381-6635.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual instruction tuning.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguis- tie representations for aaiion-and-language tasks. Ad- vances in Neural information Processing Systems (NeurlPS), 32.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei- Jing Zhu. 2002. Bleu: a method for automatic evalu- ation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Compu- tational Linguistics, pages 311-318.
Pan Lu. Swaroop Mishra. Tony Xia, Liang Qiu, Kai- Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. 2022a. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Neu- ral Information Processing Systems (NeurIPS).
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowl- edge bases? In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference

---

on Natural Language Processing (EMNLP-UJCNLP). pages 2463-2473.
Yixuan Su, Tian Lan, Yan Wang, Dami Yogatama, Ling- peng Kong, and Nigel Collier. 2022b. A contrastive framework for neural text generation. In Advances in Neural Information Processing Systems.
Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. Mauve: Measuring the gap he- tween neural text and human text using divergence frontiers. Advances in Neural Information Process- ing Systems (NeurIPS), 14:4816-4822
Alexandre Tamborrino, Nicola Pellicand, Baptiste Pan- nier, Pascal Voitot, and Louise Naudin. 2020. Pre- training is (almost) all you need: An application to commonsense reasoning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACI.), pages 3878-3887.
Chengwei Qin, Aston Zhang, Zhuosheng Zhang, Jiaao Chen, Michihiro Yasunaga, and Diyi Yang. 2023. Is chatgpt a general-purpose natural language process- ing task solver? arXiv preprint arXiv:2302.06476
Hao Tan and Mohit Bansal. 2019. Lxmert: Learning cross-modality encoder representations from trans- formers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNILP-ICCNIPP), pages 5100-5111.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sas- try. Amanda Askell, Pamela Mishkin, Jack Clark, et al. 2021. Learning transferable visual models from natural language supervision. In International Con- ference on Machine Learning (ICML), pages 8748- 8763. PMLR.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. 2022. Zerocap: Zero-shot image-to-texx generation for visual-semantic arithmetic. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7998-17928.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. 2019. Language models are unsupervised multitask learners. OpenAl blog. 18):9.
Surya Tokdar and Robert E Kass. 2010. Importance sampling: a review. Wiley Intendisciplinary Reviews: Computational Statistics, 2(1):54-66 60.
Colin Raffel. Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yangi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(4):5485-5551
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur' aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. ArXiv, abs/2302.13971.
Rajkumar Ramamurthy, Prithviraj Ammanabrolu, Kianté Brantley, Jack Hessel, Rafet Sifa, Christian Bauckhage, Hannaneh Hajishirzi, and Yejin Choi. 2023. Is reinforcement learning (not) for natural language processing?: Benchmarks, baselines, and building blocks for natural language policy optimiza- tion. In International Conference on Learning Rep- resentations (I.LR).
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. 2021. Mul- timodal few-shot learning with frozen language mod- els. Advances in Neural Information Processing Sys- tems (NeurIPS), 34:200-212.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. 2018. Object hallu- cination in image captioning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4035-4045.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. CIDEr: Consensus- based image de- scription evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pages 4566-4575.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. 2022. A-okvqa: A benchmark for visual question answering using world knowledge. In Computer Vision-ECCCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part VIII. pages 146-162. Springer.
Jianfeng Wang, Zhengyuan Yang. Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. 2022. Git: A generative image-to-tex transformer for vision and language. arXiv preprint arXiv:2205.14101
Yixuan Su and Nigel Collier. 2023. Contrastive search is what you need for neural text generation. Transac- tions on Machine Learning Research.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du Andrew M Dai, and Quoc V Le. 2022. Finetuned language mod- els are zero-shot learners. International Confer- ence on Learning Representations.
Yixuan Su. Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, and Nigel Collier. 2022a. Language models can see: Plugging visual controls in text generation. arXiv preprint trXiv:2205.06655
Tobias Weyand, Andre Araujo, Bingyi Cao, and Jack Sim. 2020. Google landmarks dataset v2-a large- scale benchmark for instance-level recognition and

---

retrieval. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pages 2575-2584.
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laplev, and Cordelia Schmid. 2021. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the IEFF/CVE International Con- ference on Computer Vision, pages 1686 1697.
Xu Yang. Chongyang Gao, Hanwang Zhang, and Jianfei Cai. 2020. Hierarchical scene graph encoder-decoder for image paragraph captioning. In Proceedings of the 28th ACM International Conference on Multime- dia (MM), pages 418! 1-4189.
Youngjae Yu, Jiwan Chung, Heeseung Yun, Jack Hessel, Jae Sung Park, Ximing Lu, Rowan Zellers, Prithviraj Ammanabrolu, Ronan Bras, Gunhee Kim, and Yejin Choi. 2023. Fusing pre-trained language mod- els with multimodal prompts through reinforcement learning. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition (CVPR), pages 10845-10856.
Youngjae Yu, Jiwan Chung. Heeseung Yun, Jack Hessel, JaeSung Park, Ximing Lu, Prithviraj Am- manabrolu, Rowan Zellers, Ronan Le Bras, Gun- hee Kim, al. 2022. Multimodal knowledge align- ment with reinforcement learning. arXiv preprint arXiv:2205.12666
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vin- cent Vanhoucke, et al. 2022. Socratic models: Com- posing zero-shot multimodal reasoning with lan- guage. arXiv preprint MXXx:2224400099.
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia. Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. 2023. What matters in training gpt4-style language model with multimodal inputs? arXiv preprint arXiv:2307.02469
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher De- wan, Mona Diab, Xian Li, Xi Victoria Lin, et al. 2022. Opt: Open pr-ttrained transformer language models. arXiv preprint 200500000

---

A VLM Failure Cases
Models
GT Caption
Acc
Prec
GPT3
1.00
1.00
A.1 Landmark Recognition Experiment
LLAVA
0.16
0.48
To better understand the named entity recogni- tion problem in VLMs image descriptions, check whether their descriptions for pictures of popular landmarks contain the proper names. We first collect the names of the 100 most popular landmarks 2. Then, we filter the list by remov- ing names of landmarks without proper nouns (e.g. Middle of the Earth), keeping 80 landmarks in total. Finally, we download the corresponding pictures from Wikipedia. Given the prompt What is this?, we task the VLM to generate response long as 100 tokens and check whether the output contains the name of the given landmark. Note that some landmarks have alternative names. Hence, we col- lect alternative names from Wikipedia and count the model-generated answer as correct when it con- tains any of the possible names. Finally, we check whether the model tried answer not by inspect- ing whether the model-generse
Ours
0.41
0.70
Table Results our landmark recognition experi- ment. Ace denotes accuracy and Prec denotes precision.
Ours:
AL nighttime, the Marina Bay Sands building, also known as the hotel tower, is floodlit and its surrounding harbor is bustling with activity.
LLAVA:
The image features stunning view of large building situared the water. The building appears to be a hotel resort, and it is connected nearby island by bridge.
text contains the name of any landmark in our list. We caleulate the precision score by dividing the number of correct predictions by the number of tries.
Ours:
Our landmark dataset 3 is tiny compared to the
The structure pictures the well-known Sagrada Familia busilica Barcelona, Spain.
similar dataset (Weyand et al., 2020) for a purpose: we want to check whether the VLM avoids telling the named entities, not whether the VLM saw them in the training process. Hence, we narrow the scope of evaluation the most popular landmarks, in which we can assume that most of the entity names are found in the VLM training dataset..
LLAVA:
the image featurres a large cathedral with tall tower and steeple, which is likely famous landmark in the city.
Figure 6: Comparison of LLAVA and VLIS in the land- mark recognition experiment.
Table 7 and Figure 6 compare base LLAVA (Liu et al., 2023) and VLIS in our landmark recognition dataset. The result shows that the VLM (LLAVA) knows at least about half the landmarks' names, but does not tell them without applying VLIS. Also, VLIS shows good precision, showing that it does not get more correct answers by guessing more. We further demonstrate that a proper answer to our prompt What is this? should contain the name of the landmarks: when we present GPT3 with the ground-truth alt captions and the prompt, GPT3 always includes the landmark names in its output.
more samples for the failure case 2: the base VLM (LLAVA) is distracted by misleading visuals while VLIS does not.
B Implementation Details
Computational Requirements. Using LLM.int8 approximation (Dettmers et al., 2022). single NVIDIA TITAN RTX GPU (24GB Memory) fits both the BLIP-2 2.7B and OPT 1.3B models. Flan- T5 XL and XXL models need more memory and VLIS using the larger backbones requires NVIDIA A6000 GPU (48GB) for inference. Both LLAVA 13B and Vicuna 7B fit into an A6000 GPU at the same time. Generating 50 tokens takes ~ 20 sec- onds in all settings.
A.2 More Qualitative Results
Figure shows full raw text outputs for the VLM failure cases shown in Figure 1. Figure 8 depicts
Hyperparameters. We fix the fluency thresh- old a = 0.001 all experiments and use beam search with beam size For QA problems, we
'The list landmarks is from www. www. listchallengess. the-world.
com/ 1010000--sstt-me 100--ostt-aauus---- Landus--anndarrss "W will release the dataset to the public.

---

apply length penalty < 0 on the beam score to in- duce suceinct answers following the literature (Li et al., 2023b). The opposite behavior required for longer text generation, so we set the value larger than 0 for open-ended generation problems. The language temperature т is manually selected by ex- amining the text quality of three samples per task. For
Models
Random
# Images
OKVQA
VLM-only
31.7
Ours
False
2
34.2
Ours
True
1
29.0
Ours
True
2
32.2
Ours
True
10
35.3
Table Results in the OK-VQA validation set. Our default option (prefined set with two images) is marked bold.
Task-Specific Hyperparameters. VQAv2 (Goyal et al., 2017), OK-VQA (Marino et al., 2019), and ScienceQA (Lu et al., 2022a) 1.25
datasets, we set the language temperature and length penalty - 1100 induce succinct answers generated with stronger visual conditioning. In Concadia (Kreiss et al., 2022). t = 0.67 and length penalty 2.0 is used for succinct caption-style text with better text conditioning. For Image Paragraph Captioning (Krause et al., 2017) experiments we use 7 = 0.67 and length penalty to induce longer captions. Also, apply contrastive search (Su and Collier, 2023) with a penalty of 0.6 to avoid text degeneracy.
degradation. We employ the technique to jointly run both text-only LM and VLM single GPU. Randomness. As VLIS is deterministic infer- ence time algorithm, no randomness is involved in any of the experiments. A stochastic sampling version of VLIS may require variance analysis, but we leave that to future research.
Evaluating Story Generation. While the offi- cial repository of MAGIC (Su et al., 2022a) shares the inference results, it does not contain the eval- uation scripts. Thus, we consult the repository Contrastive Decoding (Li et al., 2022b) for the eval- uation script for an open-ended generation problem. Due to the difference in the evaluation code, our baseline scores are different from the results re- ported in MAGIC (Su et al., 2022a). However, we still use the public inference results for the base- lines and evaluate each model with publicly avail- able code, making our evaluation pipeline unbiased, transparent, and reproducible.
Flan-T5 Hyperparameters. For the backbone comparison study in appendix E, we the VLM backbone BLIP-2 Flan-T5 (Li et al., 2023b) and text-only model to Flan-T5 (Chung et al., 2022). For Flan-T5 variants, we compensate the overconfi- dence of the model with a large temperature of 1.5 to normalize the logit outputs. For the same reason, 0.0001.
we also relax the fluency threshold Finally, the language temperature is set to 0.9.
Baseline Hyperparameters. We share the same hyperparameters in VLIS for all our imple- mented baselines; LLAVA, BLIP-2, OPT-IMI, and Naïve Ensemble. We do not modify the beam size 5 and fluency threshold o 0.001, and change the length penalty accordingly to the task following the VLIS hyperparameters.
C Marginal Approximation Experiment
In the main paper, we propose using one or two images with minimal visual information (black- filled and white-filled) as a functional candidate with minimum computational overhead. To investi- gate the alternative approaches, we conducted an additional experiment in the OK-VQA dataset. The variables considered here fined (black-filled and white-filled) set of images and 2. The number of images used approximate the expectation. We keep everything else the same as in Table and only adjust the marginal approxi- mation scheme.
Few-Shot Settings. For Image Paragraph Captioning (Krause et al., 2017), we use three ground-truth examples to prime the models for the paragraph-long generation task. However, one cannot provide multiple images as inputs to the backbone VLM model (BLIP-2 (Li et al., 2023b)). Hence, simply insert the few-shot samples in the text domain and provide only the single target image as the visual context.
Random prede-
Our results summarized in Table 8. First, random set of images is inferior our prede- fined set of images for approximating the marginal. Second, random image set offers a better approx- imation than the predefined set two images. Still, the 10 random images option requires passes
uint8 Inference. LLM.int8 (Dettmers et al., 2022) is an approximated inference technique for large language models. It applies vector-wise quan- tization and mixed sixxed-rrecision decomposition to re- duce memory consumption without performance

---

Models
a
OKVQA
VLM
LLM Backbone
Ours
Vanilla
Naïve
Backbone
LLM
VLM
Ensemble
VLM-only
31.7
2PT
34.2
19.1
31.7
26.6
Ours
le-1
13.8
F-15xi.
F-T
29.8
12.5
40.7
34.4
Ours
le-2
30.1
F-T5xi
F-T5x..
43.4
19.3
40.7
39.0
F-T5xL
F-T5xx.
43.9
21.3
40.7
42.0
Ours
le-3
34.2
F-T
R-T5xxL
47.5
21.3
45.9
44.4
Ours
lc-4
34.4
Ours
le-5
33.1
Table 10: Backbone comparison experiments the
validation set the OK- VQQ dataset (Marino et al., 2019). F-T5 denotes T5 trained on FLAN dataset (Wei et al., 2022).
Ours
0
32.3
Table 9: Results in the OK-VQA validation set. Our default fluency threshold value (a = le - 3) is marked bold.
human language distribution better than the VLM, this deterioration of performance further supports our explanation of why VLIS works.
of VLM per token generation, making it largely inefficient for practical usage.
Prompt Templates
D Fluency Threshold Experiment
In the prompt templates below, TLM denotes the prompt presented to the text-only model and VLM denotes that given to the VLM.
Here, examine the effect of fluency threshold value the generation quality of VLIS. This experiment extends the OK-VQA commonsense reasoning experiment in Table and keeps all other variables the same except for a.
OK-VQA & VQAv2
Variables:
Table 9 shows that VLIS consistently outper- forms the VLM-only baseline for all values of a in the range of [le - 3, le 5]. Too large values (1e- 1e- 2)) still harm the performance as they typically leave only for the VLIS Score to choose from.
[QUESTION]
TLM
Question: [QUESTION] Answer:
-LMM
two token candidates
Question: [QUESTION] Answer:
ScienceQA
E Backbone Scale Experiment
Variables:
We conduct comparison study to test whether the improvement offered by VLIS is generalizable to a wider set of architectures and model sizes. Here, we mainly evaluate VLIS with Flan-T5 vari- ants as both the text-only LM and VLM backbones. T5 (Raffel et al., 2020) is an encoder-decoder trans- former unlike the decoder-only autoregressive lan- guage models (e.g. OPT (Zhang al., 2022) and GPT-3 (Brown et al., 2020)). Flan-T5 (Chung al., 2022) further trains T5 for better responsive- ness in instruction prompts. Table 10 summarizes the backbone comparison results on the OK-VQA dataset (Marino et al., 2019). In all combinations of model sizes except for FlanT5Bass VLIS improves the commonsense reasoning capability of the VLM backbone. Also, Naive Ensemble performs unreli- ably depending on the choice of the text-only LM and performs worse than the VLM itself in most of the settings. The plan LM makes VLIS work worse than the VLM. Since VLIS is built on the assumption that the text-only LM knows the
[QUESTION], [CONTEXT], [CHOICES]
TLM
Answer the multi-choice question
given the image. Question:
[QUESTION] [CONTEXT] Choices:
[CHOICES] Answer:
VL.
Answer the multi-choice question given the image. Question:
[QUESTION] [CONTEXT] Choices: [CHOICES] Answer:
Concadia
- -LLM
Write a short caption that describes the image. Article: "[ARTICLE]" Caption:
VLM
The image describes

---

Image Paragraph Captioning
G More Qualitative Samples
Variables:
We include more qualitative samples for image paragraph generation in Figure 9, story generation in Figure 10, and open-ended generation in Fig- ure 11.
[ARTICLE]
TLM
Write a multi-sentence long paragraph describing the image. Each sentence should describe different aspect of the image and should be concise. \n (Image 1) Image Description:
[Description Sample 1]\n (Image 2) Image Description:
[Description Sample 2]
(Image 3) Image Description:
[Description Sample 3]
(Image 4) Image Description:
- VLM
Write a multi-sentence long paragraph describing the image. Each sentence should describe different aspect of the image and should be concise. \n (Image 1) Image Description:
[Description Sample 1]\n
(Image 2) Image Description:
[Description Sample 2]
(Image 3) Image Description:
(Description Sample 3]
(Image 4) Image Description:
ROCStories
Variables:
[TOPIC]
TLMM
[TOPIC] 

- VLL
The image describes
WHOOPS
TLLM
Decide whether the image is strange or natural terms of physics, commonsense, or etc. \n Start with "The image shows" "
VLM
Decide whether the image is strange or natural terms of physics, commonsense, or etc. \n Start with "The image shows" "

---

(a) Describing named entities
BLIP-2
person, argentina's greatest player of all time
L.LAVA
image features soccer player has just won chumpionship, as lifted up by people celebrating the victory. soccer player holding trephy. signifying achievement.
Ours
Sources suggest that the person being described Diego Maradona, professional footballer who won the Golden Boot award the 1982 World Cup. In the image, Maradona is being lifted above a stadium crowd
Who is this?
BLIP-2
yes
I.LAVA
While camnot determine the man's intentions or emotions an AI, the image shows a man in suit, who appears to be smiling and has his fingers crossed. This body language might suggest that he is confident
Ours
Michael Corleone, also known Vito Andolini Corleone, churacter depicted in the Godfather film series, shown to care deeply for his immediate family.
Does he care for his family?
(b) Visual distractors for simple facts
GPT3
[[mage of Sengull]
No, ostriches are not capable of flying. They are the largest living species of bird and are known for their long, powerful legs and neck, which they use to run at high speeds, but they do not have physical characteristics
BLIP-2
but they are not good at it
I.LAWA
Yes, ostriches can fly. Although they are large birds, they are capable taking flight using powerful wings, Ostriches have strong muscles and high lifi-to-drig ratio, which enables them become air Ours
Can ostriches fly?
No. ostriches are not capable of flying. They are ground-dwelling birds that are known for their long.
powerful legs and fast ramning speed. In the wild, ostriches migrate by running long distances, but
[Image af Monkey]
chimpunzees have tuils, They the family Hominidue, along with humans, banobos, gorillas, and orangutuns.
the few primates that lack tail. Chimpanzees belong
BLIP-2
yes.
LLAVA
Yos, chimpanzees do have tails. Their tails short and usually held upright curled over their backs. In the image, the chimpanzoe is walking on tree branch with its tail in
Ours
Chimpanzees
tails. Chimpanzees primates and closely related humans. They are also known apes and characterized their large brains, hands and feet, and the ability
Do chimpanzees have tuik?
Figure 7: Raw text output for samples described in Figure 1.

---

Visual distractors for simple facts
GPT3
While are generally
avension water, some cupable swimming. However, swimming is natural behuvior for most cats, and they usually prefer avoid water whenever possible.
LLAVA
Yes, cats can swim. image, a gray cut swimming body water, demonstrating that cats the ability navigate and efficiently through water.
Ours
Some cats may be able to swim to some extent, but they are not natural swimmers and are not as comfortable in water as dogs are. wild, cats typically avoid water and do not have the same instinctive swimming capability
Do cats swim?
GPT3
No, you cannot withdraw eash from blood bank, Blood banks facilities that collect, process, und domated bloud for medical use.
LLAVA
possible to withdraw from bank, context image suggests that the person actually taking money
(Automated Teller Machine).
Ours
Can withdraw eash from a blood bank?
No, is not possible withdraw cash from blood bank. Blood banks are typically medical facilities that collect and process blood donations, and they do not provide cash withdrawals.
GPT3
No, sharks are not herbivores. Sharks are camivorous predators, meaning they primurily on animals.
LLAVA
Yes, sharks are generally considered herbivores. They primarily food ea a variety of plants, including algac. as well as small animals and fish. In the image, the shark consuming sea vegetation, which Ours
berbivores?
No. sharks are not herbivores, the wild, sharks primarily foed fish, squid. and other marine animals. However, in aquariums and marine parks, it is commen for sharks to be fed
GPT3
ln general, most coins made metals such copper, nickel, or zine alloys will slnk water their higher deusity. However, float momentarily or for short period of time before eventually sinking.
very small size unique shape that allows trap air, may
LLAVA
Yes, coins do flatt on water. In the image, a single penny floating on the surface of a body water, such the coin
a pool. Ours
muterials
than water.
Do coins float water?
Coins do not flout on water. In fuct, they will usually sink to the bottom, This is hecause coins are made of material that is donser than water, so they will tend to sink rather than float. In the case of
Figure 8: More examples in which the visual context deceives the base VLM (LLAVA (Liu al., 2023)) into denying simple facts.

---

GT: Two elephants are standing together outside by concrete wall. One elephant is considerably smaller then the other Right by the elephants there large amount of bushes and trees with green Jeafs. The ground is mostly dirt with patches of green grass throughout. On the gress eating there ahout dozen so large black crows.
Ours: An elephant in an enclosure surrounded by crows and pigeons is looking around for something eat. The elephant has a trunk its mouth and is sniffing ground. The ground around the elephant. is covered in dirt and grass. The elephant is standing on its tusks,
BLIP-2: An elephant stands in the mi ddle of field of gress. The grass is green and lush. The elephant has brown trunk and brown face. The elephant has a brown trunk and brown lace. The elepbant has a brown lrunks and brown face.
OPT: young
is sitting on park bench. He weering black I shirt and blue jeans. He is holding baseball bat in his right hand. He is throwing the ball from pitcher's mound. The pitcher bas uniform on that is gray and hlue. His cap is black with a fork shaped symbol.
(a)
Naive Ensemble: A young man is sitting on a beach in park. He wearing blue shirt and blue jeans. holding haseball bat in his right hand. He looking the hell that being thrown from a pitcher's mound. The pitcher has uniform on that is grey and blue. His cap is black with a fork shaped symbol.
GT: A train is opening it's doors. The train is currently parked at train station. The train is blue, the doors are red, and it has white stripes on it. There a long yellow lime near the train area.
Ours: A passenger train is parked at a train stalion. The train is red in color with blue and while stripes on it. There are power lines overhead that are connected to train. The train pulling into station. man is sitting on the train with bis back the camera.
BLIP-2. 2: The man on the skateboard is Jeuning his body or balance. He is in motion riding along the read. He is wearing a gryy belmet with a print on it. There is an orange cone in the street that be will have to go arvund. O A young man is sitting on a park bench. He weering black t shirt and blue jeans. He is holding a baseball bat that is grey and blue. His Naive E Ensemble: The man on the skateboard Is leening his body or balance. He is in motion riding along the road. He wearing grey helmet with a print on it. There an orange cone in the street that he will have to go amund.
is throwing the ball from pitcher's mound. The pitcher has uniform on right band. black with fork shaped symbol.
(b)
GT: Four almllnes are flying in the sky in order put a show. The planes are dark colored and rather small. Each plane is flying straight up into the sky and large plume of white exhaust is leaving trail from the planes. The trail of exbaust shows that the planes started our llying up close logether, but they are now separaling and flying away from each other. The sky in the background blue with layer of light white clouds.
Ours: Four planes fly through the blue sky with smoke trailing behind them. They are performing for the public. The planes are flying in formation, One of the planes is in the middle of the formation and the other three planes are in front of the plane in the middle of the formation
BLIP-2: An aimlane flying in the sky. The airplane has flying in blue sky. The airplane is flying in clear blue sky. The airplane is flying in clear blue sky.
white, and blue color scheme. The airplane is
OPT: A young man is sitting on a park bench. He is wearing black t shirt and blue jeans. He is holding baseball bat in his right band. He is throwing the ball from pitcher's mound. The pitcher has a uniform on that is grey and blue. His cap black with fork shaped symbol.
Naive Ensemble: A young man is sitting on beach in park. He wearing blue shirt and blue jeuns. He holding basaball bat in his right hand. He wearing baseball cap on his head. He is looking up at the sky with his lell eye. He is looking down at the ground with his right eye.
(c)
Figure 9: Generation results on the Image Paragraph Captioning dataset (Krause al., 2017). The correct visual expressions are marked in green and wrong ones are in red.

---

Title: Joanne Went Sky Diving
GT: Joanne always wanted to go skydiving but besitant. Joanne was besitant to go because she was afraid of hsights. Joanne decided to face her fears bead on and skydiving. On her way in the car, she listened to music and took deep breaths. Joanne went sky diving and was very proud of herself.
Ours: Joanne went skydiving with friends. She had great time on the water at the end the day. W hen was to off parachute was broken. Joanne's friends had to call for help to get it fixed. Naive E Ensemble: Joanne was the beach with her friends. She decided to she saw her friends in the water. Joanne jumped out of the water and ran to the other side of the water. She had no idea w was going on until she was restued by rescue workers.
dive. W hen she got to the water,
MAGIC: Joanne was at the beach with her friends. She decided to jump off sand see what was out there. big that she bad slop and get, help. Joanne able
W hen she landed, she land safely and retur to the beach.
huge wave, The
Contrastive Search: J Joanne was at the beach with her friends. She decided to go on a dive. W hen she got to the water, she saw her friends in the water Joanne jumped out of the water and ran to the other side water: Sbe had idea what going on until she was rescued by rescue workers.
Title: Jobs
GT: Lana liked spend money, but didn't have job, One day, her dad made her find jooo so she could spend her own money. Lana found a job, but didn't work very hard, Eventually, Lana got fured. Her dad stopped giving ber money to spend.
TIME
Ours: Steve wanted job. But he didn't know what to do with his money. So he went to the bank and asked for one. And it turned out to very cheap. Steve started own company that same year.
b
Naive Ensemble: Steve wanted start career his spure time after high school. be went to college and got his barbelor's degree at the of 18. But be didn't know what he wanted to do with his free time. He decided to
become an electrician for the city of New York.
MAGIC: The man went to work. was asked to join new job. He did not like the job. He quit the job. His boss found out that had quit.
Contrastive Search: The man went work. He wanted to be manager. He asked his boss for help. His boss said he had no idea. The man quit job.
Title: Looking for peace
GT: Nina was dreaming of wodd peace her whole live. She knew she couldn't change the world but made small changes. She contributed to her community and then congratulated. She was proud when she received Nobel prize for her work.
country, came up with so many good ideas, she was
Ours: Yesterday went for walk the woods. It was dark and windy so couldn't see anything out of the comer of my eye. Atte walking through the woods for an hour I found peace. I decided to return home and look for peace in the future.
(c)
Naive Ensemble: A girl with long hair walked into my yard. She wanted talk but didn't her in front of me. I asked her if she was looking for someone and she said yes. The next day my neighhor called me to tell me her hait was in bed shape
MAGIC: The man walkod into house: He saw an emply lot. He asked his neighbors to look for peace, The neighhors were skeptical. was not enough solve the problem.
Contrastive Search: The went into a house, He man asked her to Leave. Her mother took her to the police station.
old woman sitting porch. Sbe was crying, The
Figure 10: Generation results on the ROCStories dataset (Mostafazadeh et al., 2016). Expressions showing alignment to the title are colored in green, alignment the image in blue, and misinterpretations in red.

---

Hey. It's
Today went
Here is a romantic message. Answer: Ours: I love you and want to spend the rest my life in front of you smiling and touching
Ourss, your son's teacher, and I'm going to show you a picture of me and your
Ours: to the library to pick up book for my son and found out that they have special section children with special needs. IP-2: tbe museum with my son. OPT: to the gym for the first lime in two weeks. It felt good to be back in the swing of things hnh''''s awesome! hat you do?
IL-P--2, person.
BLIP-2: love you mom
OPT: your brother: Hey it's me, your sister:
OT:: love you and I want
spend the rest of my life with you.
Naive Ensemble: , guy who wrote the article about the guy who wrote the article about the
ou are the best thing that ever happened
Naive E Ensemble: to the doctor's office for the first time in a long time, and the doctor's
Naive E Ensemble: love you and want to spend the rest of my life with you and want you to
guy who wrote the
office is in the same building as the doctor's 's...
Mey, It's me
Today went
Here romantic message. Answer:
Ours: your friend. I have an apple computer and I'm going to be at the convention few weeks,
Ours: to A pple store. bought an iMac,
O urss: Dear John, I love You are the apple of my eye. Yours truly
keyboard, and a trackpad for my wife's computer:
BLIP-2: love you.
IP-2: eynthia, I'm computer scientist.
BL IP-2: conference.
to spend the rest of my life with you. You are the best thing that ever happened to me
OPT: love you and
OPT: to the gym for the first time in two weeks. It felt good to be back in the swing of things InT hat's awesome! hat you do? Naive E nnsemble: to the library check book, but the librarians wouldn't check it out because it was too late in the day. So had to wait until the nexe day
(b)
OPT: your hrother. Hey it's me, your sister.
Naive Ensemble: , your friend. I'm here tall you that you're douchebag and you should feel bad for being such an asshole to your friends and family.
Naive Ensemble: love you and you hope you having great lime with your fumily and friends. love you and miss you and hope you
Hey. It's
Today went
Here romantic message. Answer: Ours: (Romantic) would like to take my girfireadd to eal pizza. W hat do you think is the best, pizzeria in your area
Ours: and my friend. We were wondering if you would sell us a pizza and a coke at the same time. Would you be to that
Ours: pizza place and got large popperoni pizza. It was pretly good. also gol a diet coke
BLIP-2: pizza place.
BL BP-2 again!
OPT: to the gym for the first time
BL-P-2: pizza on table.
OPT: your brother: Hey it's me, your sister
weeks. It felt good to be back in the swing of things. InT hat's awesome! What hat did you da? Naive Ensemble: to the grocery store to get food for dinner. hen I got the deli counter I noticed that there very large amount of food
OPT: I Love you and I want to spend the rest of my life with you. You are the best thing that ever happened to
Naive Ensemble: again. I just wanted to let you all know that will be out of town for the next couple of days. I will be out of town
Naive E Ensemble: how many pints of cocteeu are in the fnige at the the work day on friday the thirteenth and salurday
Figure 11: Open-ended generation results with various text prompt. Here we include more baselines than in Figure 5.


Reviews:
[{"review_id": "uulT2iwYfS", "reviewer": "Reviewer_TDrJ", "summary": "", "questions": "n/a", "limitations": "", "rating": 2, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's acknowledgment that VLIS enhances the VLMs.\nWe will update the paper draft to better describe the points raised by the reviewer, including the additional experiment results.\n\n## 1. Regarding reproducibility\n\nWe regret that we accidentally omitted the code in the supplementary files in the recent revision. However, we hope that the reviewer reconsiders the reproducibility score on the following grounds.\n\n- Our implementation code was included in the supplementary files of our earlier revisions on the OpenReview website, specifically in the first submission version.\n- We have added the main pseudo-code at the end of this response for reference.\n- We promise to release the code publicly as soon as the anonymity period ends. Were it not for the anonymity constraint, we would have shared the code link here.\n- VLIS is really simple to implement. It does not require any training or complex pipelining: rather, one only needs the logit outputs for both the text-only LM and VLM.\n- All datasets we used for the experiments are publicly available.\n- All hyperparameters we used are reported in the main paper and the appendix (also the first submission version). Additionally, VLIS introduces only two hyperparameters: $\\tau$ and $\\alpha$.\n- The GPU requirement is manageable: even VLIS with LLAVA 13B runs on a single NVIDIA A100 (40GB) with num_beams=5. VLIS with BLIP-2 runs on a single 24GB GPU.\n\n## 2. LLAVA & BLIP-2 are not SOTA backbones\n\nFirst of all, the SOTA model the reviewer mentioned (Lynx [1]) was only submitted to ArXiv on 5 July 2023 which was weeks before the EMNLP submission deadline. Hence, it was impossible to include it in our paper.\n\n\nNevertheless, we conducted an additional experiment using Lynx backbone as the reviewer suggested. Following the experiment in Table 1 of our paper, we use the WHOOPS dataset here to measure both linguistic reasoning and visual alignment capability. The Table below shows that VLIS enhances the VLM performance with Lynx backbone as well.\n\nTable: Experiment with Lynx backbone in WHOOPS dataset.\n| model | Decoding | Acc (%) |\n|---|---|---|\n| LLAVA | Standard | 59 |\n| LLAVA | Ours | 73 |\n| | | |\n| Lynx | Standard | 71 |\n| Lynx | Ours | 80 |\n\n## 3. Potential influent responses\n\nWe first note that it is a common practice to modify the token likelihoods at inference time both in NLP [2, 3] and multimodal domains [4, 5], which works without compromising the text fluency if engineered properly.\n\nVLIS maintains linguistic fluency using two mechanisms; 1. Fluency masking and 2. Following language model token likelihood. We show that VLIS indeed maintains text quality in the open-ended generation experiments on the ROCStories dataset (Table 6 on the paper). VLIS outperforms the baselines in Coherence and MAUVE score which are measures of linguistic fluency and language modelling strength, respectively.\n\nWhile we hope that we evaluate the fluency of VLIS on more open-ended multimodal generation benchmarks and metrics, there is a scarcity of publicly available data and suitable metrics in this regard, especially for long-text generation. So we instead include some qualitative samples of long open-ended generation below. We promise to include a comprehensive set of samples in the final drafts.\n\n```\nModel: LLAVA + Vicuna\n<Sample 1>\n**Image Context**: Borobudur Temple\n**Prompt**: What is this?\n—\n**Ours**: The structure in the pictures is a stone temple or fortress complex dated back to the eighth century CE, known locally as Borobudur. It is located on the slopes of a hill and made out of lava rock, giving it a unique and impressive appearance. The complex is surrounded by green grass and has multiple tiers.\n**VM**: This image features a group of people standing in front of a large and impressive stone structure, which appears to be a historical or culturally significant site. The structure is adorned with many ornate statues, and the sky serves as a beautiful backdrop. The people seem to be tourists or visitors, admiring the architecture and taking in the sight. If you were to provide a description of this scene, what would you include?\\n### \\u200b\\u200b\\u200b\\u200bH\n</Sample 1>\n<Sample 2>\n**Image Context**: Petronas Twin Towers\n**Prompt**: What is this?\n—\n**Ours**: The structures in the image are the twin Petronas Towers, which are towering skyscrapers located in the capital city of Malaysia, Kuala Lumpur. They were completed in 1998 and were the tallest buildings in the world at that time, with a total height of 452 meters (1,483 feet). The towers are decorated with intricate motifs inspired by Islamic architecture.\n**VM**: This is a nighttime view of a large building with two tall towers, each adorned with a clock. The building appears to be illuminated, with the clock faces and the top of the towers lit up, creating a visually striking scene. The towers are connected by a bridge, and the entire structure stands under a dark, cloudy sky. The image captures the grandeur and beauty of the architectural design, making it an impressive sight to behold.\n</Sample 2>\n<Sample 3>\n**Image Context**: Hands withdrawing money from an ATM.\n**Prompt**: Can I withdraw cash from a blood bank?\n—\n**Ours**: No, it is not possible to withdraw cash from a blood bank. Blood banks are typically medical facilities that collect and process blood donations, and they do not provide cash withdrawals.\n**VM**: While it is possible to withdraw cash from a blood bank, the context of the image suggests that the person is actually taking money out of an ATM (Automated Teller Machine).\n</Sample 3>\n```\n\n## 4. References\n\n- [1] What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?, https://arxiv.org/abs/2307.02469\n- [2] NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics\n- [3] A Contrastive Framework for Neural Text Generation, https://arxiv.org/abs/2202.06417\n- [4] ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic, https://arxiv.org/abs/2111.14447\n- [5] Language Models Can See: Plugging Visual Controls in Text Generation, https://arxiv.org/abs/2205.02655\n\n## 5. Pseudo-code\n\nHere is the pseudo-code for the token score calculation part of VLIS. Except for this part, the implementation of VLIS follows the standard language generation code which is widely available via public libraries such as Huggingface.\n\n```\n<inputs>\nlist_marginal_vlm_lprobs  # List of VLM log token probability for each of the marginal images and the previously generated text.\nvlm_lprobs  # VLM log token probability for the input image and the previously generated text.\nlm_lprobs # text-only log token probability for the previously generated text.\nfluency_threshold  # scalar hyperparameter \\alpha for fluency masking\nfluency_alpha  # scalar hyperparameter \\tau for language model temperature control\n</inputs>\n<code>\nfluency_mask = math.log(fluency_threshold) <= lm_lprobs\nmarginal_vlm_lprobs = mean(list_marginal_vlm_lprobs)\nvisual_score = vlm_lprobs - marginal_vlm_lprobs\nvlis_score = lm_lprobs * fluency_alpha + visual_score\nvlis_score[!fluency_mask] = -math.inf\n</code>\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "VLIS improves multi-modal LLMs' performances (namely, LLAVA and BLIP-2 OPT 2.7B) on several tasks. ", "reasons_to_reject": "1) The experiments can be improved. The chosen baselines such as LLAVA and BLIP-2 OPT 2.7B are not SoTA multimodal LLMs according to recent study[1]. Concretely, LLAVA generates long texts ignoring the image inputs, while BLIP-2 tends to generate short answers, usually a word. \n\n2) The reviewer concerns that the proposed method to directly adjust the token generation probability could result in influent responses. More evaluation is preferred. ", "questions_to_authors": "n/a", "paper_topic_and_main_contributions": "This paper proposes to combine the visual conditioning capability of vision-language models with the language understanding of unimodal text-only language models without further training. Concretely, VLIS extracts pointwise mutual information of each image and text from a visual-language model and uses the value as an importance sampling weight to adjust the token likelihood from a text-only model. VLIS improves vision-language models on several tasks. \n", "excitement": 2, "reproducibility": 2, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] What Matters in Training a GPT4-Style Language Model with Multimodal Inputs?\nhttps://arxiv.org/abs/2307.02469", "Typos_Grammar_Style_and_Presentation_Improvements": "n/a"}, "extra_text": {}}, {"review_id": "bPhDYNGnfw", "reviewer": "Reviewer_Vw57", "summary": "", "questions": "Question A: Can you provide more insights into the choice of the fluency threshold value (α=0.001) used in the fluency masking step? How sensitive is VLIS performance to different fluency threshold values?\n\nQuestion B: In the experiments, you use either one or two images with almost no visual information (black-filled and white-filled) to estimate the VLM marginal likelihood. Have you explored other alternatives or investigated the impact of different image sets on the VLIS performance?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging that our method *offers a new direction for multimodal language generation* and finding our experiments to be *thorough*.\nWe commit to updating the paper draft to better describe the points raised by the reviewer, including the additional experiment results.\n\n## 1. Regarding reproducibility\n\nWe regret that we accidentally omitted the code in the supplementary files in the recent revision. However, we hope that the reviewer reconsiders the reproducibility score on the following grounds.\n\n- Our implementation code was included in the supplementary files of our earlier revisions on the OpenReview website, specifically in the first submission version.\n- We add the main pseudo-code at the bottom of this response.\n- We promise to release the code publicly as soon as the anonymity period ends. Were it not for the anonymity constraint, we would have shared the code link here.\n- VLIS is really simple to implement. It does not require any training or complex pipelining: rather, one only needs the logit outputs for both the text-only LM and VLM.\n- All datasets we used for the experiments are publicly available.\n- All hyperparameters we used are reported in the main paper and the appendix (also the first submission version). Additionally, VLIS introduces only two hyperparameters: $\\tau$ and $\\alpha$.\n- The GPU requirement is manageable: even VLIS with LLAVA 13B runs on a single NVIDIA A100 (40GB) with num_beams=5. VLIS with BLIP-2 runs on a single 24GB GPU.\n\n## 2. Regarding reasons to reject\n\nWe acknowledge the points raised by the reviewer. However, the main contribution of VLIS is that multimodal inference can benefit from harmonizing existing language models with VLMs. While more backbone combinations, other modalities, and future estimations might strengthen the robustness and generalizability of VLIS, VLIS still works well as-is in the visual-language domain as we show in various experiments.\n\nIn conclusion, we hope that the reviewer considers these points as future research directions that VLIS provides instead of grounds to reject the method altogether, as we described in the future works section in our paper.\n\n## 3. Choice of the fluency threshold\n\nVLIS’s performance is not overly sensitive to the specific value of the fluency threshold, provided it is within a reasonable range. The primary function of Fluency masking is to filter out token candidates that have abnormally high visual weights, which can be attributed to the modeling insufficiency of the VLM.\n\nTo show this, we conducted an additional experiment on OKVQA (with BLIP-2) for varying values of the threshold while keeping everything else the same as the main paper. In the Table below, for all values in the range of  ($\\[1e-3, 1e-5\\]$), VLIS consistently outperforms the VLM-only baseline. Too large values of $\\tau$ still harm the performance ($\\[1e-1, 1e-2\\]$) as they typically leave only one or two token candidates for the VLIS Score to choose from.\n\nTable: Fluency threshold experiment on OKVQA\n| model | $\\tau$ | VQAScore |\n|---|---|---|\n|VLM-only| |31.7|\n| | | |\n|Ours|1e-1|13.8|\n|Ours|1e-2|30.1|\n| | | |\n|Ours|1e-3|34.2|\n| | | |\n|Ours|1e-4|34.4|\n|Ours|1e-5|33.1|\n|Ours|0|32.3|\n\n## 4. Different image sets for the marginal approximation\n\nWe offer using one or two images with minimal visual information (black-filled and white-filled) as a functional candidate with minimum computational overhead. To investigate alternative approaches, we conducted an additional experiment on OKVQA. The variables considered here are 1. Random vs. predefined (black-filled and white-filled) set of images and 2. The number of images used for expectation approximation. We keep everything else the same as in the main paper and only vary the marginal approximation scheme.\n\nThe Table below shows our results. First, using a single or two random images within the validation set suffers from the expected performance drop. Second, using 10 random images offers a better approximation than the standard scheme (Random Images = False). Still, the 10 random images option requires 11 passes of VLM per token generation, making it largely inefficient for practical usage.\n\nTable: Marginal image set experiment on OKVQA\n| model | Random Images | Num Images | VQAScore |\n| --- | --- | --- | --- |\n|VLM-only|||31.7|\n| | | | |\n|Ours|False|2|34.2|\n| | | | |\n|Ours|True|1|29.0|\n|Ours|True|2|32.2|\n|Ours|True|10|35.3|\n\n## 5. Pseudo-code\n\nHere is the pseudo-code for the token score calculation part of VLIS. Except for this part, the implementation of VLIS follows the standard language generation code which is widely available via public libraries such as Huggingface.\n\n```\n<inputs>\nlist_marginal_vlm_lprobs  # List of VLM log token probability for each of the marginal images and the previously generated text.\nvlm_lprobs  # VLM log token probability for the input image and the previously generated text.\nlm_lprobs # text-only log token probability for the previously generated text.\nfluency_threshold  # scalar hyperparameter \\alpha for fluency masking\nfluency_alpha  # scalar hyperparameter \\tau for language model temperature control\n</inputs>\n<code>\nfluency_mask = math.log(fluency_threshold) <= lm_lprobs\nmarginal_vlm_lprobs = mean(list_marginal_vlm_lprobs)\nlog_visual_score = vlm_lprobs - marginal_vlm_lprobs\nlog_vlis_score = lm_lprobs * fluency_alpha + log_visual_score\nlog_vlis_score[!fluency_mask] = -math.inf\n</code>\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper proposes VLIS, an innovative method that effectively combines the visual conditioning capability of vision-language models with the language understanding of unimodal text-only language models without additional training. This offers a new direction for multimodal language generation.\n2. The paper conducts thorough experiments on various tasks that require both reliable language modeling and strong visual conditioning, such as commonsense understanding, scientific reasoning, and text generation. These experiments demonstrate the effectiveness and versatility of the VLIS framework.\n3. The paper compares VLIS against several baselines, including text-only models, VLMs, and naive ensemble approaches, providing valuable insights into the performance of VLIS in different settings and its ability to adaptively fuse commonsense understanding with visual conditioning.\n4. The VLIS framework is not limited to specific VLM architectures and can be potentially applied to other multimodal tasks or extended to other modalities, such as audio and documents.\n5. The paper provides qualitative and quantitative analyses of the generated outputs, highlighting the improved linguistic capabilities, responsiveness to prompts, and visual conditioning achieved by VLIS compared to the base VLMs.", "reasons_to_reject": "1. The paper only explores a small fraction of possible combinations of text-only models and VLMs.\n2. VLIS combines the outputs of VLMs and text-only models at the very last stage of token likelihood, which might be misleading when both models assign high probabilities to the same token for different reasons.\n3. While the paper suggests that VLIS could be extended to other modalities, such as audio and documents, it does not provide experimental results to support this claim.", "questions_to_authors": "Question A: Can you provide more insights into the choice of the fluency threshold value (α=0.001) used in the fluency masking step? How sensitive is VLIS performance to different fluency threshold values?\n\nQuestion B: In the experiments, you use either one or two images with almost no visual information (black-filled and white-filled) to estimate the VLM marginal likelihood. Have you explored other alternatives or investigated the impact of different image sets on the VLIS performance?", "paper_topic_and_main_contributions": "This paper introduces Visual-Language models as Importance Sampling weights (VLIS), a novel framework designed to combine the visual conditioning capability of vision-language models (VLMs) with the language understanding of unimodal text-only language models without further training. The main contributions of the paper are:\n1. Proposing VLIS, a method that enhances the linguistic understanding of VLMs by using the token likelihoods of unimodal text-only language models and importance sampling weights derived from a VLM for visual alignment.\n2. Employing exponentiated pointwise mutual information (PMI) to isolate the visual conditioning capability of the VLMs from their language modeling preferences.\n3. Demonstrating the effectiveness of VLIS in various multimodal tasks, including commonsense understanding, complex text generation, and showing that VLIS improves vision-language models on these tasks.\n4. Comparing VLIS against several baselines on different tasks and showing its promise as a new direction for multimodal language generation.\n\nOverall, the paper addresses the challenge of enhancing the unreliable linguistic understanding of VLMs and suggests VLIS as a promising approach to combine the strengths of both unimodal and multimodal language models.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "NsdyxfbR8A", "reviewer": "Reviewer_z5TT", "summary": "", "questions": "As stated in the paper, a better visual alignment can be achieved, which may help to reduce the hallucination. Could you provide the results on the image captioning task, which is a less complex text generation task, and provide an analysis related to hallucinations, including the CHAIR scores [1]?\n\n[1] Rohrbach, Anna, et al. \"Object hallucination in image captioning.\" arXiv preprint arXiv:1809.02156 (2018).", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging our method as *innovative and intriguing* and our experiment results *promising and impressive*. We commit to updating the paper draft to better describe the points raised by the reviewer, including the additional experiment results.\n\n## 1. Regarding reproducibility\n\nWe regret that we accidentally omitted the code in the supplementary files in the recent revision. However, we hope that the reviewer reconsiders the reproducibility score on the following grounds.\n\n- Our implementation code was included in the supplementary files of our earlier revisions on the OpenReview website, specifically in the first submission version.\n- We add the main pseudo-code at the bottom of this response.\n- We promise to release the code publicly as soon as the anonymity period ends. Were it not for the anonymity constraint, we would have shared the code link here.\n- VLIS is really simple to implement. It does not require any training or complex pipelining: rather, one only needs the logit outputs for both the text-only LM and VLM.\n- All datasets we used for the experiments are publicly available.\n- All hyperparameters we used are reported in the main paper and the appendix (also the first submission version). Additionally, VLIS introduces only two hyperparameters: $\\tau$ and $\\alpha$.\n- The GPU requirement is manageable: even VLIS with LLAVA 13B runs on a single NVIDIA A100 (40GB) with num_beams=5. VLIS with BLIP-2 runs on a single 24GB GPU.\n\n## 2. Running time efficiency\n\nYes, VLIS introduces additional inference time costs. The default setup of VLIS requires three passes of the VLM with a single pass of the text-only LM.\n\nHowever, VLIS is considerably more efficient than previous multimodal inference time methods, which require either expensive backpropagation [1] or numerous (~300) passes of the visual model [2] per token generation.\n\n## 3. Results on the ROCStories dataset\n\nIn Table 6 of the main paper, VLIS secures the best scores in Coherence and MAUVE, while achieving the second-best scores in other metrics. We believe this is enough to show that VLIS is better than all other baselines. The reasons are stated below.\n\nOpen-ended generation tasks such as ROCStories generation are hard to evaluate; thus we include an extensive set of metrics following previous work [2]. A method with only a high CLIPScore (BLIP-2) just generates image captions (and no stories) as shown in its bad score in all other metrics. Also, while Naive Ensemble performs well in Repetition and Diversity scores, it shows the worst Coherence meaning that it fails to generate fluent text and instead achieves diversity by generating random visually relevant words.\n\nIn conclusion, VLIS outperforms the baselines in ROCStories. We saw that the qualitative samples match this conclusion as well; we will include them in the final draft.\n\n## 4. Image captioning hallucination experiment\n\nWe appreciate the reviewer’s insightful suggestion on measuring the image-text alignment strength of VLIS. We thus conducted an additional experiment in the COCO-Captions image captioning benchmark. In the Table below, we show that VLIS reduces object hallucination of both BLIP-2 and LLAVA backbone. Note that this improvement does not come from just mentioning fewer objects overall, VLIS’s better $CHAIR_I$ shows that it sensibly discriminates objects within the image from those that are not.\n\nWe’d like to note two auxiliary details here. First, conventional image captioning is not the strongest suit of VLIS since the task doesn’t necessitate superior language modeling capability beyond what’s already present in the VLM itself. Nevertheless, the Mutual Information score of VLIS helps overcome the VLM language bias. Second, our baseline results outperform those reported in a recent paper [3]. Unlike their approach, we confined the generation to 20 tokens and took only the first sentence for a fair comparison between the backbone that generates short captions (BLIP-2) and that with long text (LLAVA) by default.\n\nTable: Image captioning hallucination experiment on COCO validation set (Karpathy split). Lower is better.\n| model  | Decoding |  CHAIR_I  |  CHAIR_S  | Length |  \n|---|---|---|---|---|\n| BLIP-2 | Standard | 3.7       | 4.3       | 7.1    |  \n| BLIP-2 | Ours     | 2.4       | 3.1       | 7.1    |  \n| | | | | |\n| LLAVA  | Standard | 5.4       | 8.0       | 12.6   |  \n| LLAVA  | Ours     | 5.0       | 7.2       | 13.3   |\n\n## 5. References\n\n- [1] ZeroCap: Zero-Shot Image-to-Text Generation for Visual-Semantic Arithmetic, https://arxiv.org/abs/2111.14447\n- [2] Language Models Can See: Plugging Visual Controls in Text Generation, https://arxiv.org/abs/2205.02655\n- [3] Evaluating Object Hallucination in Large Vision-Language Models, https://arxiv.org/abs/2305.10355\n\n## 6. Pseudo-code\n\nHere is the pseudo-code for the token score calculation part of VLIS. Except for this part, the implementation of VLIS follows the standard language generation code which is widely available via public libraries such as Huggingface.\n\n```\n<inputs>\nlist_marginal_vlm_lprobs  # List of VLM log token probability for each of the marginal images and the previously generated text.\nvlm_lprobs  # VLM log token probability for the input image and the previously generated text.\nlm_lprobs # text-only log token probability for the previously generated text.\nfluency_threshold  # scalar hyperparameter \\alpha for fluency masking\nfluency_alpha  # scalar hyperparameter \\tau for language model temperature control\n</inputs>\n<code>\nfluency_mask = math.log(fluency_threshold) <= lm_lprobs\nmarginal_vlm_lprobs = mean(list_marginal_vlm_lprobs)\nlog_visual_score = vlm_lprobs - marginal_vlm_lprobs\nlog_vlis_score = lm_lprobs * fluency_alpha + log_visual_score\nlog_vlis_score[!fluency_mask] = -math.inf\n</code>\n```", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1) The method proposed in the study is innovative and intriguing, as it rationally leverages the properties and strengths of both the visual language model (VLM) and the large language model (LLM). Furthermore, its training-free nature significantly cuts down on associated training costs.\n\n2) The performance of the proposed methodology across numerous datasets is promising, which is impressive.", "reasons_to_reject": "1) As delineated in Appendix E, the inference time is seen to increase due to the requirement of multiple forward passes, which potentially affects the method's efficiency.\n\n2) Despite exhibiting impressive results across several datasets, the proposed method fails to achieve the best outcomes in some, including the ROCStories dataset.", "questions_to_authors": "As stated in the paper, a better visual alignment can be achieved, which may help to reduce the hallucination. Could you provide the results on the image captioning task, which is a less complex text generation task, and provide an analysis related to hallucinations, including the CHAIR scores [1]?\n\n[1] Rohrbach, Anna, et al. \"Object hallucination in image captioning.\" arXiv preprint arXiv:1809.02156 (2018).", "paper_topic_and_main_contributions": "In this research article, the authors introduce a novel approach, referred to as VLIS, aimed at mitigating two prevalent issues associated with the comprehension capabilities of current visual language models (VLMs). The first problem arises when VLMs fail to identify entities in certain scenarios, while the second problem stems from the over-reliance of VLMs on image context, which subsequently results in numerous unsuccessful cases.\n\n\nThe paper's contribution is two-fold. Firstly, a training-free methodology is proposed to extract importance sampling weights from the VLM, providing visual alignment to adjust the token likelihood emanating from the large language model (LLM). Secondly, the proposed methodology exhibits promising performance in various distinct types of tasks.", "excitement": 4, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agreed that the proposed inference-time method of combining a unimodal LM with a VLM has promising/impressive performance improvements, and generally agreed that the experiments are thorough, using a wide range of datasets. Two reviewers described the method as "innovative". Reviewer TDrJ was less excited about the paper, given that previous conditioned generation methods also modify token likelihoods. However, I agree with the author response that this doesn't limit the novelty of this paper as that framework is so general, and this paper is an effective and novel application to grounding within this general framework. The same reviewer also had some concerns about fluency and robustness -- but in my opinion, the wide range of datasets and the coherence evaluations alleviate this concern.
Decision: Accept-Main