Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion (2024)

Abstract: With the bloom of Large Language Models (LLMs), Multimodal Large Language Models (MLLMs) that incorporate LLMs with pretrained vision models have recently demonstrated impressive performance across diverse vision-language tasks. However, they fall short to comprehend context involving multiple images. A primary reason for this shortcoming is that the visual features for each images are encoded individually by frozen encoders before feeding into the LLM backbone, lacking awareness of other images and the multimodal instructions. We term this issue as prior-LLM modality isolation and propose a two phase paradigm, browse-and-concentrate1, to enable in-depth multimodal context fusion prior to feeding the features into LLMs. This paradigm initially “browses” through the inputs for essential insights, and then revisits the inputs to “concentrate” on crucial details, guided by these insights, to achieve a more comprehensive understanding of the multimodal inputs. Additionally, we develop training strategies specifically to enhance the understanding of multi-image inputs. Our method markedly boosts the performance on 7 multi-image scenarios, contributing to increments on average accuracy by 2.13% and 7.60% against strong MLLMs baselines with 3B and 11B LLMs, respectively.

heading: 1 Introduction
text: Multimodal Large Language Models (MLLMs) have recently garnered attention for their surging popularity and impressive performance across diverse Vision-Language (VL) tasks (Team et al., 2023; OpenAI, 2023; Qi et al., 2023). Among these MLLMs, the paradigm that extending Large Language Models (LLMs) with pre-trained vision encoders has shown remarkable abilities in visual reasoning and visual instruction-following (Wu et al.,
*These authors contribute equally. BCorresponding authors: Peng Li and Fei Huang. 1Code is released at https://github.com/THUNLP-MT/
Brote
2023; Yin et al., 2023). These models also draw attention for their feasibility and flexibility in adapting to varied scenarios and demands (Liu et al., 2023b; Zhu et al., 2023).
Despite its impressive abilities, this paradigm faces challenges that obscure a deeper understanding of multi-image and interleaved inputs (Dai et al., 2023; Luo et al., 2023; Zhao et al., 2024; Li et al., 2023d). The approach of simply gluing up pre-trained vision and language models via intermediate components (Li et al., 2023e; Liu et al., 2023d) potentially neglects essential crossmodality and inter-image interactions, leading to the LLM being presented with isolate visual and textual features without recognition of interleaved multimodal inputs. We refer to this problem as prior-LLM modality isolation, which further divides two issues image-text isolation and interimage isolation. These challenges have received
11229
considerable attention but remain unresolved.
Firstly, image-text isolation happens when frozen vision encoders produce generic visual features, overlooking crucial target-specific information. For instance, in Figure 1 (a), the emphasis should be on the “registration plate”. This plate, occupying only a minor area of the image, is prone to being overshadowed by predominant elements due to inadequate image-text interaction. To tackle this problem, Dai et al. (2023) and Luo et al. (2023) integrate textual instructions into visual feature extraction to enhance the responsiveness of these features to the given instructions. Moreover, some researchers propose to alter the internal structure of LLMs to bridge the gap between visual and linguistic spaces (Wang et al., 2023). While these methods are effective in single-image scenarios, they do not address the concurrent fusion of multiple images.
Secondly, inter-image isolation arises from encoding images separately, disrupting semantic links among images and conveying misinformation of the multi-image context. This issue is particularly prevalent in scenarios involving interleaved and multiple images. As illustrated in Figure 1 (b), the moving direction regarding the other image should be considered. However, such relational information remains isolated and fails to transmit across images. Consequently, the lack of awareness regarding relevant content from other images can lead to the exclusion of essential visual information. To handle this issue, recent studies have developed context schemes that aim to improve image-text correlations and the connections between multiple images (Zhao et al., 2024; Li et al., 2023b). Nevertheless, the prior-LLM fusion of multimodal context are overlooked.
To mitigate the two outlined issues, we utilize a cognitive strategy that mirrors the process through which humans typically understand new content: by first grasping the main ideas during an initial browsing and then revisiting the material to deepen their understanding with the browsing insights (Garner, 1987). Inspired by this approach, we propose a novel paradigm named Browse-andConcentrate (Brote). This paradigm begins with a browsing phase to generate a condition context vector, serving as a collection of browsing insights, encapsulating the main intent and visual information derived from images. Subsequently, a concentrating phase is employed to comprehend multimodal inputs, guided by the condition context
vector. Furthermore, to enhance the effectiveness of the browsing insights, we have developed training strategies that prompt the model to implicitly leverage these insights for more precise extraction of image features, allowing for the possibility of bypassing explicit browsing in some scenarios. Our contributions can be summarized as follows:
• We address the challenge of prior-LLM modality isolation by proposing the browseand-concentrate paradigm, alongside training strategies to encourage the model to leverage and explore the browsing insights. • We explore two method to implement our paradigm, demonstrating that Brote not only learns to concentrate on interleaved inputs via explicit context vectors, but also integrates this ability directly into the model implicitly. • We conduct comprehensive evaluations on 7 multi-image scenarios and exhibits notable advancements, improving the average accuracy by 2.13% and 7.60% against baselines with 3B and 11B LLMs, respectively.

heading: 2 Related Work

heading: 2.1 Empowering LLMs with Visual Abilities via Pre-trained Vision Models
text: With the surging of LLMs, MLLMs that empower LLMs with visual abilities have also witnessed a rapid growth. Following the initial effort (Tsimpoukelli et al., 2021) to convert visual features into readable embeddings for LLMs, researchers have proposed to bridge vision and language modalities via diverse visual prompt generators (VPG), such as Resampler (Alayrac et al., 2022), Q-Former (Li et al., 2023e; Dai et al., 2023), and linear projections (Liu et al., 2023d; Huang et al., 2023). They utilize image features from frozen vision models (Dosovitskiy et al., 2021; Radford et al., 2021), and subsequently integrate these features into pretrained LLMs. These MLLMs inherit cognitive and perceptual abilities from vision models and the emergent ability from LLMs, exhibiting impressive performance without intensive training. However, they bear the modality isolation issue that obscures a deeper understanding multimodal context.

heading: 2.2 Enhancing Visual Features with Textual Instructions
text: Recent studies have concentrated on augmenting the capability for MLLMs to follow visual instructions (Liu et al., 2023d; Dai et al., 2023; Luo et al.,
2023; Wang et al., 2023; Ye et al., 2023). Some researchers have focused on the fine-tuning LLMs to better response to visual instructions (Ye et al., 2023; Liu et al., 2023c), employing techniques such as LoRA (Hu et al., 2022a). While other studies target the issue of image-text isolation by manipulating with the visual features. For instance, Dai et al. (2023) enhance Q-Former with textual instructions to obtain instruction-aware visual features. Luo et al. (2023) integrate learnable instruction features directly into the vision encoders. Despite these innovations, they primarily incorporate instructions into the vision modules pay less attention to the complexity of multi-image senarios.

heading: 2.3 MLLMs Enhanced for Comprehending
text: Multiple Images
The ability to comprehend multiple images simultaneously draws considerable attention (Alayrac et al., 2022; Zhao et al., 2024; Li et al., 2023b; Shukor et al., 2023). Multi-image scenarios can be categorized into interleaved image-text formats and multimodal ICL settings. To improve ICL preformance, Shukor et al. (2023) analyse prompt-based approaches, introducing three templates for multimodal ICL. Meanwhile, some researchers work on methods requiring model-tuning (Alayrac et al., 2022; Shukor et al., 2023; Sun et al., 2023). Ad-
ditionally, some scholars broaden the exploration of multi-image scenarios to include both ICL and interleaved inputs. Li et al. (2023d) insert middlelayer LLM outputs into the VPG as additional guidance for spotting the differences between images. Zhao et al. (2024) and Li et al. (2023b) construct datasets targeting the multi-image issue, and propose context schemes to improve the understanding of interleaved inputs. Despite these advancements, the prior-LLM multimodal context fusion is not sufficiently explored.

heading: 3 Method

heading: 3.1 Overview
text: To stimulate prior-LLM multimodal context fusion and improve the awareness of multimodal context of the LLM, we propose a paradigm, Browse-andconcentrate (Brote). It progressively comprehends images via two phases, browsing and concentrating. As illustrated in Figure 2 (a), in the browsing phase, the MLLM browses the entire input and generates a condition context as the browsing result, denoted as C in the rest of this paper. Then, in the concentrating phase, the model comprehends multimodal inputs under the guidance of C. We refer to the model of browsing phase as MB and the model of concentrating phase as MC .
Our proposed Brote can be further divided into two modes, explicit and implicit, regarding the distinct approaches of incorporating C. The explicit browse-and-concentrate (Figure 2 (a)-1), denoted as Brote-EX, operates with separated parameters (MB ̸= MC). This explicit mode first generates C using MB , followed by MC to infer the final outcomes. In contrast, for the implicit browseand-concentrate (Figure 2 (a)-2), denoted as BroteIM, employs shared parameters for both phases (MB = MC), permitting MC to directly predict the answer without the need to explicitly produce intermediate vectors from the other model. Along with the proposed paradigm, we devise training strategies for the explicit browse-and-concentrate mode. This strategies encourage the model to leverage and explore the generated condition context vectors. The explicit mode serves as a precursor to the implicit mode, preparing the model with fundamental and essential ability to understand C.
We will elaborately describe the workflow of Brote in §3.2, followed by the proposed strategies for pre-training (§3.3) and fine-tuning (§3.4).

heading: 3.2 Browse-and-Concentrate Paradigm
text: We represent the interleaved multimodal input as x, defined as x = [xm0 , x m 1 , . . . , x m n , . . . , x m N−1] for N tokens, with n = 0, 1, . . . , N − 1. Each token is associated with modality m, where m ∈ {image, text}. Images are individually encoded by vision encoder gϕv(·) with parameters ϕv, which provides image features v = gϕv(x m n ), for m = image. Referring to Emb(·) as the embedding mapping, v is subsequently integrated with textual instructions htext = Emb(xmn ), for m = text, via a Q-Former fϕQ(·, ·) parameterized by ϕQ,
h = [hm0 , h m 1 , . . . , h m n , . . . , h m N−1] (1)
hmn = { Emb(xmn ) if m = text fϕQ(v, [Q;h text]) if m = image (2)
where h denotes the multimodal embeddings, and Q is the learnable query tokens in Q-Former.
The LLM component of MB is denoted by fϕL(·) with parameters ϕL. The browsing phase produces C by extracting the last hidden states of the LLM fϕL(·), denoting as follows:
C = f (l)ϕL(h). (3)
where l represents the last layer of fϕL(·).
In the concentrating phase, the images undergo alterations conditioned on C. We add C to query tokens Q, and obtain the altered visual token embeddings h̃image as,
h̃image = fϕQ(v, [Q+ Linear(C);htext]), (4)
where Linear(·) denotes the linear projection, and [·; ·] denotes concatenation. In this phase, QFormer accepts an extra input C compare to the browsing phase. Finally, the prediction y with T tokens is formulated as follows:
y = MC(x, C) = argmax
y p(y|x, C; fϕ′L , fϕ′Q , gϕ′v) (5)
= argmax y
T∏
t=1
p(yt|y<t,x, C; fϕ′L , fϕ′Q , gϕ′v),
where yt is the t-th token of the prediction, y<t = y1, · · · , yt−1, fϕ′L , fϕ′Q , gϕ′v are the LLM, QFormer, and vision encoder components of MC parameterized by ϕ′L, ϕ ′ Q, and ϕ ′ v, respectively.

heading: 3.3 Context-Enhanced Pre-Training
text: Condition context-enhanced pre-training. The pre-training stage aims at adapting the model to utilize C and enhancing visual feature extraction with its conveyed multimodal context. To this end, we propose a training task that challenges the model to generate task-specific descriptions without direct exposure to the question. Initially, we obtain C by feeding the intact inputs into MB . Then, in the concentrating phase, MC is required to generate image descriptions specialised for the questions
that are not explicitly presented but instead implicitly encoded within C. As depicted in Figure 2 (c) “Pre-training”, the model is presented with only the text “Please describe the image” alongside altered visual tokens h̃image. This strategy urges the model to explore C for target information. Additionally, we combine the task-specific training targets together with the general ones, enabling the model to discern between inputs with and without C. The objective for pre-training is as follows:
LMC =− T∑
t=1
ŷt log p(yt|x, C; fϕ′L , fϕ′Q , gϕ′v), (6)
where ŷt is the t-th groundtruth token. Data construction. In alignment with the taskspecific training strategy, we design a data generation method to secure task-specific supervisions as mentioned above. Inspired by PromptCap (Hu et al., 2022b), we leverage LLMs to craft target-aware image descriptions. Our approach is extended from the producing of individual image descriptions to addressing multiple interleaved inputs, enabling a more profound understanding of multi-image and interleaved context. We obtain the image-target related descriptions as demonstrated in Figure 3. The LLM receives a triplet (P, CK , QAJ), comprising task instruction prompt P , general image descriptions CK and question-answer pairs QAJ , where K and J represent the counts of images and targeted questionanswer pairs respectively. Also noticing that K is not necessarily equal to J . For each k-th image (k = 1, 2, . . . ,K), the LLM is required to generate image description Dk that satisfies the target clarified in P . Accordingly, Dk contains specific messages for questions in QAJ and information about the other related images o, k ̸= o.
We construct a total of 56k data for the pretraining stage, and manually assess the quality of the generated captions by randomly sampling 230 generated captions. We detect that 36 (out of 230) captions contain hallucination or minor incorrect information, while the rest 84% are of good quality, containing desired and correct question-aware information. Please refer to Appendix A for details of the generated data.

heading: 3.4 Condition-Aware Task Fine-Tuning
text: To encourage further exploration of information from C for VL tasks, we propose a new training strategy named context-dropping training. The
strategy intentionally omits particular inputs yet requiring the model to infer for answers solely with the assistant of C. It motivates the model to compensate for the missing information from the provided condition context C. We propose different dropping strategies as illustrated in Figure 2 (b):
• Drop images: This involves two approaches, removing certain images (Figure 2 (b), “Context Dropping (IMG-N)”), and replacing original images by blank placeholders (Figure 2 (b), “Context Dropping (IMG-B)”). • Drop text: We remove the text before the last image as shown in Figure 2 (b), “Context Dropping (TXT)”. • Drop ALL: A combination of the above settings denoted as “ALL”, applied with the same probabilities.
To ensure integration with C, we preserve the last image across all dropping strategies. Notice that the “drop images” approaches are not applicable to inputs with only one image. These strategies compel the model to infer indispensable information from C that should have been given in the input.
As mentioned in §3.1, we investigate two modes for incorporating C, Brote-EX and Brote-IM. For Brote-EX, we apply context-dropping strategies to the concentrating phase with C provided by frozen model MB . The training objective for explicit mode is LMC as described in Equation 6. While for Brote-IM, parameters of MB are shared with MC . When optimizing the shared parameters, we also take into account the loss for MB as follows:
LMB = − T∑
t=1
ŷt log p(yt|x; fϕL , fϕQ , gϕv). (7)
For the training of Brote-IM, we sum up the two losses, for MB and MC respectively, as LMB + LMC , denoted by dual-loss. Details of the training process are documented in Appendix C.

heading: 4 Experiments

heading: 4.1 Implementation
text: We implement our method upon InstructBLIP (Dai et al., 2023) with FlanT5 (Chung et al., 2022) as the language backbone. We pre-train our model on the 56k generated data as described in §3.3, and then extract about 490k data from the MIC dataset (Zhao et al., 2024) for model fine-tuning. The fine-tuning data is sampled according to the data-balanced sampling algorithm suggested by
Dai et al. (2023). Please refer to Appendix B for details of the training data and Appendix C for more information of the training process.

heading: 4.2 Evaluation Settings
text: Baselines. We primarily employ models designed for accepting multiple images or interleaved image-text inputs as baselines, such as MMICL (Zhao et al., 2024), Otter (Li et al., 2023b) and VPG-C (Li et al., 2023d). Additionally, MLLMs that are used to develop these baselines are also considered, such as BLIP-2 (Li et al., 2023e) and InstructBLIP (Dai et al., 2023). Please refer to Appendix D for detailed information of the employed baselines. For models whose results are not
officially reported, we utilize the publicly available checkpoints for evaluation.2
Benchmarks and Metrics. We investigate diverse VL benchmarks and focus on multi-image tasks, including visual reasoning (NLVR2 (Suhr et al., 2019)), few-shot ICL for image QA (VQAv2 (Goyal et al., 2017) and A-OKVQA (Schwenk et al., 2022)), video QA (MSVD QA (Xu et al., 2017), MSRVTT QA (Xu et al., 2017), SEED-Bench (Li et al., 2023c)), and
2We use the public checkpoints to obtain the missing results for MMICL (https://huggingface.co/BleachNick), InstructBLIP (https://huggingface.co/Salesforce), and Otter (https://huggingface.co/luodian), together with official scripts and required environments.
multi-image instruction following (DEMON (Li et al., 2023d)). Note that SEED-Bench comprises of both images and videos. For video benchmarks, following Zhao et al. (2024), we uniformly extract eight frames from the given video clips for answering the questions. For few-shot ICL, we employ the widely used four-shot setting. Additionally, we conduct experiments on single-image tasks to fairly compare with models that are not designed for multi-image settings. These tasks include zero-shot setting for VQAv2, A-OKVQA and MME (Fu et al., 2023). ScienceQA (Saikh et al., 2022) (SciQA) is designed for Chain-of-Thought (CoT) (Wei et al., 2022) scenario with accompany hints, and we adopt the zero-shot CoT (ZS-CoT) setting for this dataset. Details of these evaluation benchmarks, including data scale, the type of tasks and evaluation metrics, are listed in Appendix E.

heading: 4.3 Results
text: We report results for multi-image settings in Table 1 and single-image settings in Table 2. Drawing conclusions from these tables, our method presents significant improvement for multi-image settings, while concurrently improves the performance of 3 single-image tasks.
Our models exhibit notable advancements over models in Table 1, showing profound comprehending ability for multi-image and interleaved inputs. We outperform strong baselines, such as InstructBLIP, MMICL and VPG-C, which include shallow prior-LLM instruction-image fusion. Our method goes beyond merely cross-modality integration between image and text to also include intra-modality fusion among images. Impressively, our models show consistent advantage over benchmarks involving videos and multiple images, and for few-shot
ICL of QA tasks as well. For the average scores of models following InstructBLIP paradigm, our models achieve improvements of 2.13% and 7.60% for XL and XXL models respectively, over MMICL.
For single-image tasks reported in Table 2, our models continue to manifest progress, presenting higher average scores. We improve the performance for two zero-shot VQA tasks and one MLLM benchmark, MMBench. However, our models only show modest performance on MME.

heading: 4.4 Ablation Study
text: Impact of condition context vectors. To determine whether condition context vectors C contribute to the improvement, we conduct ablation study by removing C and observe a decline in accuracy across various tasks, evaluation settings, and model scales. In detail, we replace these vectors by zero vectors to simulate the absence of C. Experiments are conducted on zero-shot and fewshot VQA, and multi-image visual reasoning tasks. As shown in Table 3, the models augmented by C (“Ours”) consistently outperform those with zero vectors (“Ours-None”). The most substantial average discrepancy is observed in Brote-EX at the XL scale (2.03%), while the smallest gap is presented by Brote-IM at the XXL scale (0.54%). We notice that Brote-EX tends to gain more directly from C compared to Brote-IM, and conclude that Brote-IM directly integrates additional benefits provided by C into the model through dual-loss training. More sophisticated analysis are documented in §5.1.
Impact of different training strategies. For efficient iteration and validation, we create a subset by sampling one-third of the training data for ablation
studies on different training strategies, denoting the resulting models as Ours-sampled. For fair comparison, we also reproduce MMICL-XL (with InstructBLIP backbone) using this subset3. We evaluate the average scores of training strategies described in §3.4 and §3.3, with a special focus on priorLLM multimodal context fusion for multi-image scenarios. The averaged scores for multi-image tasks and the overall tasks are reported in Table 4, with detailed results provided in Appendix G. The performance of InstructBLIP serves as the baseline and is used to indicate the contribution of each of the designed strategies. Summarised from Table 4, the models equipped with context-dropping strategies yield higher average scores. Notably, the dropping-ALL strategy presents the highest average scores of both multi-image and overall tasks, showing profound multimodal context handling ability. We consequently adopt this strategy for training our Brote models.

heading: 5 Discussions and Analysis

heading: 5.1 Explicit Versus Implicit
text: As discussed in §4.4, Brote-EX exhibits a more significant benefit from C compared to Brote-IM. We propose two potential reasons for this observation:
• Brote-IM gains advantages from extra training steps rather than insights provided by C; • Brote-IM effectively incorporates the capabilities afforded by C into the parameters of LLM and Q-former during the training process.
For further investigation, we extend the training of Brote-EX with the same configurations and objectives as applied to Brote-IM, except that we zero out C for the concentrating phase. Specifically, we replace C in Equation 5 by zero vectors. Brote-IM is trained for two epochs based on Brote-EX as
3We use the published code from https://github.com/ HaozheZhao/MIC
detailed in Appendix C. Hence, we train Brote-EX for additional two epochs, denoting this model as Brote-EX (+2epoch). We report the results in Table 5. The results reveal that the observed improvements of Brote-IM over Brote-EX are not solely attributable to increased training steps. Rather, the improvements stem from the integration with C during training. Although Brote-EX (+2epoch) presents an average increase of 1.54% over BroteEX, Brote-IM exhibits an additional 0.59% average improvement over Brote-EX (+2epoch) with the participant of C during training, culminating in a total increment of 2.13% over Brote-EX.
Furthermore, as detailed in Table 3, the absence of C does not prevent Brote-IM models from outperforming Brote-EX. This finding supports the conclusion that Brote-IM integrates the function of C into the model parameters themselves without explicitly generate C from the other model, facilitating a more profound comprehension of multimodal inputs. In contrast, Brote-EX relies on an extra explicit representation, condition context C, to obtain multimodal comprehension and achieve good performances. The superior performance of Brote-IM affirms the efficacy of the dual-loss training strategy. Brote-IM markedly benefits from C, thereby enabling the development of a more apt parameter set for multimodal context.

heading: 5.2 Case Study
text: In Figure 4, we illustrate a case study on multimodal ICL, highlighting the coherent performance of our model in response to the input. Specifically, our model demonstrates an acute awareness of the target information conveyed through the multimodal inputs, capturing an intra-image connection characterized by “an animal sitting on/in a certain place”. Compared to MMICL, our model produces a response that precisely aligns with the input, showcasing its profound ability to comprehend multimodal contexts.

heading: 5.3 Efficiency of Inference
text: We investigate the efficiency of inference with our models in terms of both time and GPU memory, as our methods involve two forward iterations and an additional C compared to other InstructBLIP-based models. We conduct experiments with XL models using single NVIDIA A100 GPU, with batch size 10 and data type float32. Results indicate that Brote-EX requires almost equal GPU memory (18G) and inference time (around 2.3 second per batch) compared to MMICL. However, Brote-IM exhibits an increase of GPU memory from 18G to 24G for an additional “browsing” iteration, and doubles the time cost to 5 second per batch.

heading: 6 Conclusion
text: In this paper, we address the prior-LLM modality isolation issue for both image-text and inter-image context, which lacks sufficient investigation in previous works. To mitigate this issue, we propose browse-and-concentrate paradigm that leverages the initial browsing insights for the prior-LLM multimodal context fusion to stimulate more profound comprehending of multi-image and interleaved inputs. We present in-depth analysis on our proposed training strategies and the two approaches for implementing our proposed paradigm. The two approaches, explicitly or implicitly browse through and then concentrate on the context, exhibits comprehensive multimodal context understanding. Our method demonstrates remarkable improvements on 7 multi-image tasks against strong baselines that enable prior-LLM image-text fusion.
Limitations
We conlude the limitations of our method as follows: First, although presenting improved results for multi-image scenarios, our method does not achieve equally impressive performances across all single-image tasks evaluated. This discrepancy can be attributed to the employed backbone models (InstructBLIP), which already incorporate the textual instructions into the visual feature extraction process, partially addressing the challenge of prior-LLM modality isolation we aim to overcome. Our future work includes validating the proposed paradigm on broader backbone models. Second, we do not specifically incorporate datasets designed for visual instruction tuning, such as LLaVA (Liu et al., 2023d), which could be a reason for the modest performance on MME benchmark. In this
paper, we primarily focuse on multi-image scenarios, such as question-answering and visual reasoning, without a particular emphasis on following visual instruction. Third, as we introduce a twophase paradigm, the time cost and the required GPU memory for inference with Brote-IM are also increased.

heading: Acknowledgements
text: This work is supported by the National Key R&D Program of China (2022ZD0160502) and the National Natural Science Foundation of China (No. 61925601, 62276152). We appreciate all the reviewers for their insightful suggestions. We thank Siyu Wang for her participation in this work, and Haozhe Zhao for his technical support. We thank Tong Su for providing photos presented in Figure 4, and Alan (the cat) for being the model.

heading: A Details of Our Constructed Data for Pre-training
text: In this section, we provide details of the data generation process mentioned in §3.3. Inspired by PromptCap (Hu et al., 2022b), we employ LLM APIs to generate target-aware image descriptions, but explore broader types of tasks. Extending from single-image descriptions, we require the LLM4 to generate descriptions regarding other images as well, enabling a more profound understanding of multi-image and interleaved content. We utilize
4We use the GPT-4 API with version “gpt-4-1106preview”
datasets targeting difference aspects of visual reasoning, including the maintenance of general world knowledge, the spatial and temporal information, the OCR ability, and the ability to distinguish differences of images. These datasets are as follows:
• VQAv2 (Goyal et al., 2017) is a single-image visual captioning dataset. It is utilized to consolidate the general ability of our model.
• ST-VQA (Biten et al., 2019) and IconQA (Lu et al., 2021) are two single-image VQA datasets. They primarily conrtibute the tasks of OCR, object identification and counting.
• VSR (Liu et al., 2023a) is a VQA dataset addressing the spatial relation between two objects. It is employed to enhance the spatial reasoning ability.
• VCR (Zellers et al., 2019) is a visual reasoning dataset with images extracted from video scenes, focusing on the relations between presenting figures and objects.
• NLVR2 (Suhr et al., 2019) and MIMICIT(CGD) (Li et al., 2023b) both contain two interleaved images with text. They help with the ability to distinguish the differences between two images.
• iVQA (Yang et al., 2021) is a video question answering dataset. We utilize this dataset to promote the ability to deal with the sequential information.
For datasets with naturally interleaved formats, VCR, NLVR2, and MIMIC-IT, we directly employ them to prompt LLMs, aiming to generate taskspecific and multi-image aware descriptions. For other datasets, VQAv2, ST-VQA, IconQA, VSR, and iVQA, we adopt their few-shot versions from the MIC dataset (Zhao et al., 2024). In these versions, single-image instances are reconfigured into a few-shot format, featuring one or multiple images for zero to eight shots. These adapted instances serve as QAJ , as detailed in §3.3, with the corresponding questions designed as targeted tasks for LLM responses. The statistics of our generated data are listed in Table 6. We split instances containing multiple images into single image paired with the corresponding descriptions, and use them for pre-training as described in §3.3.
A.1 Prompt Templates for Pre-training Data Generation
To be consistent with §3.3, we represent our employed prompt including task instruction P , the general image descriptions CK , and the questionanswer pairs QAJ , where K and J are the number of images and targeted question-answer pairs. We provide prompt templates for the included datasets as belows:
Prompt Template for Data Generation
General task instruction P ======== The general descriptions <caption> for each image are as follows: C1, C2, . . . , CK ======== Here are the additional information of <Question>-<Answer> you should focus on! QA1, QA2, . . . , QAJ
Detailed task instruction of P for different datasets are as follows:
• Task Instruction for VQAv2 & VCR & IconQA & ST-VQA & VSR: Generate detailed captions of each image involved in the following text according to the original caption and the given <Question><Answer> pairs. You should pay attention to the information in the <Answer>. Your
output should be in the json format, as {"image0":"", "image1":"", "image2":""}. Your output should also be natural as an original caption and not include words like "answer" or "caption"!
• Task Instruction for NLVR2: Generate detailed captions of each image involved in the following text according to the original caption and the given <Question><Answer> pairs. You should pay attention to the information in the <Answer>. Your output should be in the json format, as {"image0":"", "image1":"", "image2":""}. Your output should also be natural as an original caption and not include words like "answer" or "caption"! You should also notice that <image0> is the left image and <image1> is the right image.
• Task Instruction for iVQA: Generate detailed captions of each image involved in the following text according to the original caption and the given <Question><Answer> pairs. You should pay attention to the information in the <Answer>. Your output should be in the json format, as {"image0":"", "image1":"", "image2":""}. Your output should also be natural as an original caption and not include words like "answer" or "caption"! You should notice that there exists sequential information between images!
• Task Instruction for Task Instruction for MIMIC-IT(CGD): Generate detailed captions of each image involved in the following text according to the original caption and the given <Option><Answer> pairs. You should pay attention to the information in the <Answer>. Your output should be in the json format, as {"image0":"", "image1":"", "image2":""}. Your output should also be natural as an original caption and not include words like "answer" or "caption"! Your output should also not clearly contain comparison while the information in <Option>-<Answer> pair should be presented!
Here is a detailed example from ST-VQA:
An Example for Data Generation
Generate detailed captions of each image involved in the following text according to the original caption and the given <Question>-<Answer> pairs. You should pay attention to the information in the <Answer>. Your output should be in the json format, as {"image0":"", "image1":"", "image2":""}. Your output should also be as natural as an original caption and not include words like "answer" or "caption"! ======== The original caption for each image are as follows. <image0>: a sign with chinese characters on it; <image1>: a man is walking down a hallway with a television above him. ======== Here are the additional information that you should focus on!
The image 0: <image0> is the primary source of information for answering the questions. Please refer to it carefully when answering question: What does the street sign say? Answer: anping jie
Answer each question based on the information presented in image 1: <image1>. Given the picture <image1>, what is the answer to the question: What does the green sign say? Answer: exit
The corresponding outputs for the two images are “The green street sign displays the words ‘anping jie’ in Chinese characters.” and “A man is strolling through a hallway while a television monitor is mounted above him alongside an indication of an ‘exit’ on a green sign.”

heading: B Training data for Model Fine-tuning
text: The select 17 datasets targeting different tasks from MIC dataset. Following Dai et al. (2023) and Zhao et al. (2024), We sampled about 490k instances from MIC according to this equation:
pd = √ Nd∑i=1
D
√ Ni , (8)
where pd refers to the probability to select N instances of dataset d, from a total of D datasets. We list the involved datasets in Table 7.

heading: C Model Training
text: This section describes detailed pre-training and fine-tune setting.
C.1 Pre-training
We initially acquire all the condition context vectors for the data outlined in Appendix A by MMICL models, where MMICL-XL and MMICL-XXL models are employed for Brote-XL and BroteXXL, respectively. Leveraging these vectors, we bypass the forward iteration stage during pretraining and directly proceed the concentrating phase. We set the learning rate for the condition projection at 1× 10−4 and for both the Q-Former and the language projection at 1×10−5, applying a cosine learning rate scheduler. These experiments are conducted on the NVIDIA A100 GPU, with the pre-training configurations detailed in Table 8. As a complement to Figure 2, we provide a detailed models structure in Figure 5.
C.2 Fine-tuning
In the pre-training stage, we adapt the parameters of the Q-Former and the condition projection to effectively integrate C, enhancing the models’ ability to interpret multimodal contexts. Based on this, the subsequent fine-tuning encompasses both browsing and concentrating phases. Following Dai et al. (2023) and Zhao et al. (2024), we fine-tune our model on multiple originated datasets to enable the ability to accomplish practical and diverse tasks. As described in §3.1, we develop two approached for incorporating C, each predicated on differing
objectives: For Brote-EX, condition context vectors are derived from the frozen MMICL model, whereas Brote-IM generates these vectors internally. Training specifics for Brote-EX-XL involve four epochs focused on the objective LMC , while Brote-IM-XL extends this with two epochs under a dual-loss objective, LMB + LMC , starting from the Brote-EX-XL foundation. For the XXL models, the training duration for both explicit and implicit training modes are adjusted to half that of their XL counterparts. Detailed configurations are listed in Table 9, not that the settings of learning rates identical to that of pre-training stage.

heading: D Baselines
text: We compare to MLLMs who also notice the multiimage scenarios, including MMICL (Zhao et al., 2024), Otter (Li et al., 2023b), VPG-C (Li et al., 2023d), KOSMOS-1 (Huang et al., 2023) and EMU (Sun et al., 2023). For models that are not initially designed for accepting multiple images, such as InstructBLIP, we concatenate the visual embeddings for all the input image together to enable the multi-image processing ability. The details of the baselines are listed together with the results in Table 11 and Table 12.

heading: E Benchmarks and Metrics for Evaluation
text: The employed benchmarks and corresponding metrics are listed in Table 10. We investigate diverse conventional VL benchmarks and recently proposed MLLM benchmarks, including VQAv2 (Goyal et al., 2017), A-OKVQA (Schwenk et al., 2022), ScienceQA (Saikh et al., 2022), NLVR2 (Suhr et al., 2019), MSVD QA (Xu
et al., 2017), MSRVTT QA (Xu et al., 2017), SEED-Bench (Li et al., 2023c)), DEMON (Li et al., 2023d)), MME (Fu et al., 2023) and MMBench (Liu et al., 2023e).
For VQAv2 and A-OKVQA, we test the zeroshot question answering ability given a single image, and also evaluate the ability to gain information from other related images under few-shot ICL setting. ScienceQA is proposed for Chainof-Thought (CoT) (Wei et al., 2022) scenario, and we adopt the corresponding zero-shot CoT setting. SEED-Bench (Li et al., 2023c) is a recently proposed benchmark that also aims at question answering, which comprises both images and videos. We evaluate the zero-shot ability on it because no training set is available for extracting few-shot examples. As NLVR2 contains naturally interleaved image-text instances, we only conduct zero-shot evaluation. With recently proposed benchmarks for MLLMs, such as MME (Fu et al., 2023), MMBench (Liu et al., 2023e), and DEMON (Li et al., 2023d), we employ the zero-shot setting.
Following previous works (Antol et al., 2015;
Saikh et al., 2022; Suhr et al., 2019; Huang et al., 2023), we report the soft-accuracy scores (Antol et al., 2015) for A-OKVQA and VQAv2, and calculate the accuracy scores on ScienceQA, NLVR2, MSVD QA and MSRVTT QA. The accuracy+ is employed as the metric for MME, and I4-score (Li et al., 2023d) is used for DEMON-Core.

heading: F Full results of other popular MLLMs
text: As we evalution the performance on a variaty of task, where some results are missing for certain closel related baselines. We use the public checkpoints to obtain the missing results for MMICL, InstructBLIP and Otter 5, together with official scripts and required environments. Apart from the MLLMs that focus on interleaved and instruction-
5The detailed model versions with links are as follows: MMICL (https://huggingface.co/BleachNick/ MMICL-Instructblip-T5-xl and https://huggingface. co/BleachNick/MMICL-Instructblip-T5-xxl); InstructBLIP (https://huggingface.co/Salesforce/ instructblip-flan-t5-xl and https://huggingface. co/Salesforce/instructblip-flan-t5-xxl); Otter: (https://huggingface.co/luodian/ OTTER-Image-MPT7B).
following settings, we also provide a table of results from other popular MLLMs. Table 11 and Table 12 record the results for multi-image and single-image settings respectively. The detailed results of the subtasks from DEMON-core are listed in Table 13.
In Table 11 and Table 13, our models demonstrate better performance over the others of different scales, including models of larger scales. We outperform strong baselines, such as InstructBLIP, MMICL and VPG-C, which include also consider prior-LLM instruction-image fusion. This support our finding that our browse-and-concentrate paradigm contributes to a more in-depth understanding of multimodal context with the assistant of these intermediate browsing insights.
However, for single-image tasks reported in Table 12, we notice a different trend on MME benchmark. For models with LLMs small than 10B, VPG-C-Vicuna and Otter show impressive performance on MME. For models with LLMs larger than 10B, MMICL-XXL (BLIP2) presents the best performance, followed by its variant MMICL-XXL (InstructBLIP). Our models only outperform InstructBLIP models. This is potentially caused by the limitaion of training data, where we exclude the visual instruction tuning dataset such as LLaVA (Liu et al., 2023d) during pre-training and
fine-tuning, because the outputs can vary subjectively. On the contrary, our models continue to manifest progress for single-image QA tasks and the other MLLM benchmark MMBench.

heading: G Details for Ablation Study on the Training Strategies
text: In this section, we provide the detailed results for ablation study of our proposed strategies as an accompany of Table 4. Table 14 lists the results for each tasks, and averaged scores for multi-image tasks (AVG-Multi), single-image tasks (AVG-Single) and overall tasks (AVG). In the settings without context-dropping strategies„ our model with pre-training presents superior multiimage comprehension, as evidenced by its performance on the A-OKVQA 4-shot and SEEDvideo settings, in comparison to its counterpart without pre-training. Nonetheless, without contextdropping strategies, both models exhibit a limitation in achieving a balanced performance across single-image and multi-image scenarios. To address this, we incorporate context-dropping strategies designed to encourage the models to effectively utilize the given condition context vector, as detailed in Section 3.4. We eventually adopt the “Drop-ALL” setting for training our Brote models.

heading: Future Work
text: Our future work includes validating the proposed paradigm on broader backbone models. Second, we do not specifically incorporate datasets designed for visual instruction tuning, such as LLaVA (Liu et al. , 2023d), which could be a reason for the modest performance on MME benchmark. In this
paper, we primarily focuse on multi-image scenarios, such as question-answering and visual reasoning, without a particular emphasis on following visual instruction. Third, as we introduce a twophase paradigm, the time cost and the required GPU memory for inference with Brote-IM are also increased