Improving Visual Commonsense in Language Models via Multiple Image Generation

Authors: Guy Yariv, Idan Schwartz, Yossi Adi, Sagie Benaim
Primary area: foundation or frontier models, including LLMs
Keywords: Language Models, Image generation, Machine learning

Abstract:
Commonsense reasoning is fundamentally based on multimodal knowledge. However, large language models (LLMs), trained using textual data only, are limited with their ability to incorporate essential visual information. In contrast, Visual Language Models (VLMs), which excel at visually-oriented tasks, often fail at non-visual tasks such as textual commonsense reasoning. 
This divergence highlights a critical challenge - the integration of robust visual understanding with foundational text-based reasoning. To this end, we introduce a method aimed at enhancing LLMs' visual commonsense while maintaining textual modeling and commonsense reasoning performance. Specifically, our method generates multiple images based on the input text prompt and integrates these into the model's decision-making process by mixing their prediction probabilities. To facilitate multimodal grounded language modeling, we employ a late-fusion layer that combines the projected visual features with the output of a pre-trained LLM conditioned on text only. This late-fusion layer enables predictions based on comprehensive image-text knowledge as well as text only when required. We evaluate our approach using several visual commonsense reasoning tasks together with traditional NLP tasks, including common sense reasoning and reading comprehension. Our experimental results demonstrate significant superiority over existing baselines. When applied to recent state-of-the-art LLMs (e.g., Llama3), we observe improvements not only in visual commonsense but also in NLP benchmarks.

Decision: reject
Reviewer scores: 3, 8, 6, 5

Reviews and discussion:
<thread>
creation date:
1730366901180

thread_type:
official_review

<thread_object>
creator: reviewer_hoax

<content>
summary:
This paper propose a novel late-fusion method incorporating multiple images during training and inference to augment visual commonsense of LLMs. During inference, vLMIG leverage a Text2Image model to generate multiple images based on input text and use features of these image to enhance the KV in fusion layer upon the LLM.

strengths:
1. Performance improvements  on object commonsense tasks are significant compared with baseline VaLMs.
2. The proposed method is simple to implement and ablation studies is comprehensive.

weaknesses:
1. The reproducibility of results are questionable since the papers fails to clearly introduce the exact setting of the experimental results. For example, Sec. 4.1 states VG and Laion-220K are both used while #371 says only VG is used. 
2. The comparison setting might be unfair. How to decide whether an incoming question needs image to augment or not? For questions that are irrelevant to visual knowledge, generated images can possibly harm the performance. Specifically, does the model generated images during evaluation on datasets like SQuAD.

questions:
1. Are some paramaters frozen during training? The performance of vLMIG based on Llama3-8B is even improved on QuAC while only training on Visual Genome which exhibit limited text diversity.
2. Maintaining text understanding capability during training VLMs is studied comprehensively, such as MM1 and Deepseek-VL. I am curious if these VLMs can directly give good performance on tasks requiring visual commonsense knowledge while keeping a nice performance on textual tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730614222852

thread_type:
official_review

<thread_object>
creator: reviewer_pmkl

<content>
summary:
This work proposed vLMIG, a novel visually-augmented language modeling framework for visual commonsense reasoning.

strengths:
1. This work resolved a significant issue in current LLaVA-style MLLMs, in which the lightweight visual SFT always leads to significant catastrophic forgetting and performance degradation on text-only capabilities. VILA (Lin et al., 2023) resolved this via large-scale multimodal pre-training, while the efficient training pipeline for resolving this issue is not explored well. Concluding from the results in Table-2, LLaVA-Next also gets a 2.9 scores degradation compared with its LLM backbone vicuna-7b, while the vLMIG achieved a slightly +0.2 score improvement.

weaknesses:
1. This architecture design only slightly differs from Flamingo, in which Flamingo adopts the cross-attention and this work adopts joint self-attention following VaLM. An ablation study on adopting cross-attention layer design is also appreciated to investigate some potential in architectural design.

2. An ablation study on the number of LFALs can also strengthen the contribution of this work. Flamingo adds cross-attention to each layer, while vLMIG only adds one LFAL. If there is a scaling effect on the number of introduced LFALs, then the users can decide how many LFALs to add to achieve a great tradeoff between performance and efficiency.

3. A significant baseline of LLaVA-llama3-8b is missed. You can consider to try to eval the https://huggingface.co/lmms-lab/llama3-llava-next-8b model and add the results to Table 2.

4. The efficiency is still another concern due to the image generation at inference stage in Table 3. Maybe retrieval is a better way? Or some explorations on efficient image generation can be also added to this work.

questions:
1. How many A100 hours are used for your training?

2. Why an out-of-date Wikitext-103 dataset is adopted as the training corpora, maybe ShreadPajama is another better choice? Or you adopt this dataset for the density of knowledge?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730784937264

thread_type:
official_review

<thread_object>
creator: reviewer_qlne

<content>
summary:
This paper proposes a new method for Visually-augmented Language Models (VaLM). The method proposes using fixed (frozen) pre-trained LLM and visual encoders and trainable visual projection and multimodal fusion layers. The pre-trained components never see inputs from the other modality. The fusion layer feeds into the output (softmax) layer. A second contribution of the paper is to use multiple generated images at inference with weighted-average vectors. The method shows empirical improvement in visual understanding tasks while showing neutral-positive behavior on pure text-understanding tasks.

[Update after author rebuttal] Given authors' clear rebuttals with additional results, I have revised my rating from 6 --> 8.

strengths:
The main strengths of the paper are as follows:

* The empirical studies are conducted on variety of models (BERT, GPT-2, Llama-8b, Gemma-2B) which is commendable and also shows the broader applicability of the method.

* The empirical improvements on the datasets presented in the paper are significant, especially at smaller scale models (BERT, GPT-2, Gemma-2B).

* The proposal method is simple and easy to integrate in any existing LLMs.

weaknesses:
The main weaknesses of the paper are:

* While the proposed method shows significant improvements in visual understanding tasks at small scale, the results at larger scale (namely Llama-8b in Table 2) show very marginal improvements over the baseline which suggests either (a) the late fusion network requires many more parameters (more # of transformer layers) as the base model becomes more capable, or (b) at larger base models, we need a more end-to-end approach where we feed concatenated text+image tokens at the input layer as opposed to the proposed late fusion approach.

* The visual understanding dataset used in Table 1 (object common sense) seems too simple to form a very informed opinion of the proposed method. The authors mention running this method on more complex datasets as future work but we may need to see some results during the discussion phase if possible.

questions:
For the different vLMIG models in Table 1 & 2, how many layers are used in the fusion module?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731384813320

thread_type:
official_review

<thread_object>
creator: reviewer_2969

<content>
summary:
The paper introduces a method called VLMIG, aimed at enhancing visual commonsense reasoning in large language models (LLMs) by incorporating multiple generated images into the inference process. The method is particularly innovative in its use of a late-fusion mechanism, allowing the language model to consider both text and visual information without compromising on text-based tasks. VLMIG integrates various image predictions generated by a pre-trained text-to-image model, weighting them to improve visual commonsense understanding. Experimental results demonstrate VLMIG’s effectiveness across several tasks, such as visual commonsense reasoning and traditional language benchmarks, outperforming current baselines and maintaining strong performance in non-visual tasks.

strengths:
VLMIG’s late-fusion approach, which combines textual and visual data just before final prediction, allows it to perform well on both visual and textual commonsense tasks. This ensures that LLMs benefit from visual grounding while retaining their core language abilities. The model excels across multiple task types, including object commonsense tasks (e.g., color and shape recognition) and reading comprehension. This versatility makes it a well-rounded approach for both visual and language-based tasks. VLMIG consistently outperforms existing models, including VLMs and VaLMs, in visual commonsense benchmarks, showing that multiple image generation and aggregation is a promising direction for enhancing multimodal reasoning capabilities. The authors conduct thorough experiments, including ablation studies, to validate the effects of different components, such as multi-image generation and the late-fusion layer. This helps substantiate the robustness and adaptability of the proposed model.

weaknesses:
1. The method relies heavily on the quality of generated images. Poor text-to-image generation could potentially harm performance, but this potential failure mode isn't thoroughly discussed. The effectiveness of VLMIG depends on the quality and relevance of the generated images. If the text-to-image model generates incorrect or low-quality images, this could lead to suboptimal or incorrect predictions, as the ensemble approach relies on the integration of various visual predictions
2. The approach requires generating and processing multiple images during inference, which likely increases computational costs and latency significantly. Although high-quality images can be generated quickly, this approach still results in considerable computational and time costs, especially when k is large. This could limit the method's practicality in time-sensitive applications
3. The paper doesn't adequately address how the approach scales with larger language models or longer text inputs, particularly given the computational requirements of processing multiple images.
4. There isn't sufficient analysis of the contribution of each component (VTP, LFAL, multiple image generation) to the overall performance.
5. Only test on VLMIG with GPT-2 backbone for table 1 may be really far away from current VLM community.

questions:
As weakness
</content>
</thread_object>

</thread>
<thread>
creation date:
1733249295246

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Rebuttal Summary

comment:
We sincerely thank the reviewers for their valuable feedback, which has greatly enriched our work. 

The reviewers collectively recognized several key strengths in our approach:
- **Novelty**: Reviewers appreciated our novel late-fusion method and the use of multiple image generations to enhance visual - commonsense reasoning in language models (Reviewers 2969, QLne, PmKL, Hoax).
- **Strong performance improvements** on visual commonsense reasoning tasks (Reviewers 2969, QLne, Hoax).
- **Maintaining text understanding capabilities** while improving visual reasoning, thus addressing catastrophic forgetting in multimodal LLMs (Reviewer PmKL).
- **Versatility** of our model, across multiple task types (Reviewers 2969, QLne).
- **Simplicity and ease of integration** into existing LLMs (Reviewers QLne, Hoax).
- **Comprehensive experiments and ablation studies** validating our approach (Reviewers 2969, Hoax).

We have addressed the reviewers' concerns:
- **Efficiency:** Addressed computational costs and proposed alternatives for time-sensitive applications, enabling smaller models to achieve comparable performance to larger LLMs (Reviewers 2969, PmKL, Hoax).
- **Dependency on image quality:** Experiments show our method remains effective even with unrelated images (Reviewer 2969).
- **Scalability with larger models:** Conducted additional experiments demonstrating our method scales effectively with larger models (Reviewer QLne).
- **Additional benchmark:** Evaluated on the more complex VEC benchmark, outperforming all baselines (Reviewer QLne).
- **Architectural design and ablations:** Performed ablation studies confirming our design choices and demonstrating trade-offs between performance and efficiency (Reviewer PmKL).
- **Comparison with additional baselines:** Included comparisons with additional baselines, showing our method's superiority while maintaining or improving text-based performance (Reviewers PmKL, Hoax).
- **Training data:** Experiments with RedPajama corpus showed improved results (Reviewer PmKL).

Once again, we appreciate the reviewers' efforts, which have greatly strengthened our work. This has also led to increased scores (Reviewers QLne, Hoax). We believe we have thoroughly addressed all concerns and validated our contributions.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes vLMIG, a method for enhancing visual commonsense reasoning in language models by generating and incorporating multiple images during inference via a late fusion mechanism. The late-fusion approach allows LLMs to leverage visual information while maintaining text understanding capabilities, addressing the catastrophic forgetting issue common in multimodal models; The use of multiple generated images with weighted averaging improves robustness, which also shows promising improvements over baselines on visual commonsense reasoning tasks.

However, the main weakness is the computational overhead from generating multiple images during inference, though the authors demonstrate ways to mitigate this through parallel processing and using fewer images. In addition, while the method shows strong results on smaller models, the improvements become more marginal with larger backbones, suggesting potential scaling limitations. Most concerns have been addressed during the rebuttal, but some reviewers still remain unconvinced.