If I understand the context, I will act accordingly: Combining Complementary Information with Generative Visual Language Models
Authors: Yuyu Bai, Sandro Pezzelle


Reviewer paper summaries: - The paper evaluates the degree to which VLMs combine complementary visual and textual information using the BD2BB dataset.
- They evaluate 4 VLMs selected to be open-source and comprise frozen pre-trained unimodal models.
- The authors formulate the dataset as 3 zero-shot tasks: multiple-choice answering, answer ranking based on the model's loss, and open-ended generation. The open-ended generation is evaluated by model-based metrics (BERTScore) and humans.
- Out of the examined models, only BLIP-2 is found to achieve good performance (in the multiple-choice and generation task formats). The authors link this to BLIP-2 being the only model using an instruction-tuned LM and being pretrained with in-domain images.
Summary of strengths: - Evaluating the degree to which generative VLMs integrate complementary text and visual information is an important research question.
- The authors evaluate VLMs in both closed and open-ended settings to provide a comprehensive evaluation of performance on the BD2BB dataset.
- Clear and detailed description of the experimental setup.
Summary of weaknesses: 1. Scope of the study: More in-depth exploration is needed to understand how successfully modern generative VLMs combine multimodal information (see below and Suggestions).
2. Fairness of the extrinsic evaluation: Given that the models other than BLIP2 are not trained with multiple-choice tasks, and the base LLM is also not instruction-tuned, it would be unexpected that the models are able to follow the format of the task in a zero-shot manner.
3. Inconsistency of performance in intrinsic vs extrinsic evaluation: IDEFICs and MAPL do considerably better than BLIP-2 in the intrinsic evaluation setup, which does not match the extrinsic and open-ended evaluations. How is the input formatted from the intrinsic evaluation? Could this explain the inconsistency between the performance in intrinsic and extrinsic evaluations?
4. Definition of intrinsic vs. extrinsic evaluation: The distinction between intrinsic vs. extrinsic evaluation is ambiguous, as both evaluations test the same underlying model capability and are intended to provide similar insights.
Comments, suggestions and typos: Suggestions:

I would suggest going into more depth about when / why VLMs fail at integrating complementary multimodal information. For example, 
- By evaluating other instruction-tuned models that are not trained on COCO data (one example could be LLaVA [1]), you could provide more evidence to disentangle the effect of in-domain pretraining data vs using an instruction-tuned LM. Also, since MAPL and IDEFICS were developed for few-shot learning, it could be interesting to see to what extent the performance improves with in-context examples.
- Based on your error analysis did you find any consistent failure patterns? Do models tend to prefer one of the text or image-based decoy answers? 
- You could potentially provide additional insights by analyzing model representations. For instance, are there any different attention patterns for correct vs incorrect predictions?

[1] Visual Instruction Tuning, Liu et al., 2023


Typos:
- Table 1 and lines 161-162: The mapping network of MAPL is a transformer followed by fully-connected layers
- Line 346: "intrinsically and intrinsically"
- Lines 467, 526: Appendix (?)
Overall assessment scores: 2.5