DualFocus: Integrating Macro and Micro Perspectives in Multi-modal Large Language Models

## Abstract

Abstract We present DualFocus, a novel framework for integrating macro and micro perspectives within multi-modal large language models (MLLMs) to enhance vision-language task performance.
Current MLLMs typically singularly focus on inputs at a predefined resolution, resulting in deficiencies in detailed questions involving local regions.
We introduced a DualFocus mechanism where the model concentrates on the image from a macro perspective, responds to the question, and identifies suitable sub-regions to zoom in for subsequent micro perspective analysis. Via the integration of answers from both macro and micro perspectives, the model is adept at addressing tasks that encompass global, detailed, and combined considerations.
To endow the DualFocus mechanism in MLLMs, we curated a tailored dataset derived from the Visual Genome (VG) and adapted it to align with the training regimen of DualFocus.
Through comparative studies across different model sizes and benchmarks, we demonstrate DualFocus’s superiority in balancing detailed examination with holistic insight, significantly reducing hallucination instances in MLLMs and improving their performance in various vision-language tasks.
Code is available at: https://github.com/InternLM/InternLM-XComposer/blob/main/projects/DualFocus

## 1 Introduction

Large Language Models (LLMs) like ChatGPT (OpenAI, ), GPT-4 (OpenAI, ), and PaLM (Chowdhery et al., ) have revolutionized the field of natural language processing with their astounding ability to follow human instructions and tackle open-ended tasks. These models demonstrate an exceptional understanding of language and can generate text that is often indistinguishable from that produced by humans.
Building upon this foundation, Multi-modal Large Language Models (MLLMs) such as MiniGPT-4 (Zhu et al., ), LLaVA (Liu et al., ), and InstructBLIP (Dai et al., ) have emerged, integrating the linguistic prowess of LLMs with visual understanding capabilities. Drawing on open-source LLMs like LLaMA (Touvron et al., ), Qwen (Qwen, ) and InternLM (Team, ), these MLLMs extend their insight to the visual domain, allowing for a more comprehensive understanding of questions that necessitate both visual and textual processing.

One of the primary challenges in advancing MLLMs resides in effectively incorporating visual information.
Current models such as MiniGPT-4 (Zhu et al., ) and LLaVA (Liu et al., ) often rely on imagery of a fixed, small resolution. This approach simplifies processing but limits the model’s ability to discern micro details crucial for answering specific questions.
Conversely, models like Monkey (Li et al., ) with 896$\times$896 pixel inputs and OtterHD (Li et al., ) with 1024$\times$1024 pixel inputs leverage high-resolution images to tackle fine detailed visual analysis.
However, dramatically increased image resolution introduces overwhelming details, where most of them are irrelevant to a specific question. As a result, it is more challenging to attain a balance between global context and local information, which is crucial to benchmarks that demand a more holistic understanding like MMBench (Liu et al., ) and SEED ().

Drawing inspiration from human cognitive processes where an individual typically scans an image globally before focusing on relevant details to answer a question, we argue for the integration of similar strategies in MLLMs.
Unlike existing approaches that follow a SingleFocus strategy, which looks the image once and answer the question directly.
Our proposed DualFocus strategy imitates human cognition by first analyzing the entire image to grasp the macro context, thereby deducing the important areas. Subsequently, the model zooms into these identified subareas for a detailed examination to accurately answer the provided question.
This method, inspired by the Chain of Thought (CoT) (Wei et al., ) in NLP, is extended by intertwining visual cues into the cogitative sequence through an auto-zoom mechanism.

To endow MLLMs with the DualFocus capacity to tackle both the micro and macro perspectives within an image,
we curate a new dataset that derived from the Visual Genome (VG) (Krishna et al., ), where we meticulously select images and annotations and explicitly align the format with our dual focus protocol.
Throughout the model training phase, the MLLM learns to discern the relevant coordinates defining the important subregion for any given query. The model then leverages a magnified view of this localized area to ascertain the correct answer.
During the inference stage, the model employs macro and micro answer pathways, respectively, thereby yielding two potential answers. To select the optimal response, we employ Perplexity (PPL) (Jelinek, ) as a decision metric, comparing the computed losses from the two answers and opting for the one with the lower loss as the final prediction.

In our experimental evaluation, we utilize LLaVA 1.5 (Liu et al., ) and Qwen-VL-Chat (Bai et al., ) as baseline models, selected for their robust performance in multi-modal tasks.
Comparative experiments were conducted across model sizes of 7B and 13B parameters, and a diverse set of benchmarks that ranged from multi-modal and traditional VQA benchmarks.
Specifically, DualFocus improves LLaVA 1.5 by 2.8, 3.0, 4.2, 4.2 and Qwen-VL-Chat by 1.2, 2.6, 4.0, 2.2, on SEED (), MM-Benchmark (Liu et al., ), TextVQA (Singh et al., ), and GQA (Hudson & Manning, ), respectively.
Additionally, we observed a notable reduction in hallucinatory responses in MLLMs when tested on the POPE benchmark (Li et al., ), highlighting the framework’s potential to curb the generation of spurious detail by maintaining a balanced perspective.
The comparative studies reinforce the versatility of DualFocus across a spectrum of datasets, affirming the effectiveness of the DualFocus mechanism.

## 2 Related Work

### 2.1 Large Language Model (LLM)

The evolution of LLMs has significantly shaped the natural language processing (NLP) landscape, showcasing the extraordinary capabilities of the Transformer architecture. Initiated by encoder-decoder models such as BERT (Devlin et al., ), T5 (Raffel et al., ) and decoder-centric architectures like GPT (OpenAI, ), these models have excelled across various NLP tasks. With GPT3 (Brown et al., ), decoder-only models have become increasingly prevalent due to their effectiveness in few-shot and zero-shot scenarios. Enhancements in model parameterization and dataset breadth are epitomized by Google’s PaLM (Chowdhery et al., ), which pushed the performance boundaries of LLMs even further. To tailor models for natural conversational responses, strategies such as fine-tuning and reinforcement learning derived from human feedback have been adopted in InstructGPT (Ouyang et al., ) and ChatGPT (OpenAI, ). The contribution of the open-source community, exemplified by the release of models like LLaMA (Touvron et al., ), Vicuna (Chiang et al., ), Qwen (Qwen, ), LLaMA2 (Touvron et al., ), Baichuan2 (Baichuan, ), and InternLM (Team, ), has spurred a continuous stream of innovation, setting new benchmarks for NLP research.

### 2.2 Multi-Model Large Language Model (MLLM)

Recent research in MLLM has made significant advances in the visual language learning domain by exploring the integration of visual knowledge into LLMs.
Models such as CLIP (Radford et al., ; Sun et al., ) and BLIP (Li et al., ) have demonstrated the effectiveness of contrastive learning to synchronize image and text modalities, remarkably improving zero-shot learning in tasks like Image Captioning and Image-Text Retrieval.
Models such as MiniGPT-4 (Zhu et al., ), LLaVA (Liu et al., ), InstructBLIP (Dai et al., ), and Otter (Li et al., ) have pushed further, enhancing dialogic interactions and contextual understanding in image-text scenarios by focusing on precise pre-training alignments and fine-tuning processes.
Notably, advanced techniques employing grounding data have been developed to anchor the models’ perceptions more firmly in reality, as demonstrated by mPLUG-Owl (Ye et al., ), Shikra (Chen et al., ), Opera (Huang et al., ), VIGC (Huang et al., ) and KOSMOS-2 (Peng et al., ). Such initiatives mitigates the issue of hallucinations and leads to more reliable performances across visually grounded tasks, together with more rich multi-modality datasets (Zhao et al., ; He et al., ; Wang et al., ) resulting in the development of more advanced MLLMs (Zhang et al., ; Dong et al., ; Hong et al., ; Qi et al., , ).

Figure: Figure 1: Framework. Initial processing involves feature extraction from a visual input ($v$) via Visual Encoder ($V$) and question tokenization ($q_{1}$) via tokenizer $T$ to determine the subregion $\hat{box}$ (for brevity we omit connection layer $W$ here), followed by the generation of a localized, zoomed view ($v^{\prime}$). The MLLM then fuses the macro-contextual representation ($h_{v},h_{q_{1}},h_{box}$) with the micro-focused information ($h_{v^{\prime}},h_{q_{2}}$) to accurately respond to the query ($\hat{ans}$).
Refer to caption: x2.png

### 2.3 High Resolution MLLMs

Recently, MLLMs have primarily utilized fixed, lower-resolution inputs, typically 224 pixels (Liu et al., ; Chen et al., ; Zhu et al., ). Innovations such as LLaVA-1.5 (Liu et al., ) and BLiVA (Hu et al., ) have sought to enhance performance by expanding input resolution to 336 pixels and integrating task-specific with global features, respectively. Moreover, advancements like Qwen-VL (Bai et al., ) and OtterHD (Li et al., ) have pushed resolution boundaries to 448 pixels and preserved original image sizes during inference, leading to more refined detail discernment. Notably, Monkey (Li et al., ) has significantly increased resolution to 896 pixels. Nonetheless, an inherent tension persists within MLLM architectures: models employing lower resolution inputs struggle with detecting finer details, whereas those with higher resolutions often underperform on tasks emphasizing global comprehension.
This paper introduces the DualFocus mechanism, reconciling the conflicting demands of micro-detail reflectivity and macro-contextual understanding, thereby offering a nuanced equilibrium for MLLM designs.

## 3 Our Approach

In this section, we provide an initial overview of the Multi-modal Large Language Model (MLLM) (Sec. 3.1). Following that, we elucidate the methodology, covering aspects such as dataset construction (Sec. 3.2), the training phase (Sec. 3.3), and the inference process (Sec. 3.4).

### 3.1 Preliminaries

The contemporary MLLMs usually adopt a modular architecture, comprising a visual encoder $V$, a series of connection layers $W$, and a large language model $L$. Given an input image $v$ and its corresponding question $q$, the visual encoder $V$ initially processes the image and encodes it into a set of visual tokens $z_{v}=V(v)$. These visual tokens are then transformed to align with the embedding space of the language model through the connection layers, such that $h_{v}=W(z_{v})$. Concurrently, the text query $q$ is tokenized into linguistic tokens $h_{q}$ by the tokenizer $T$, becoming $h_{q}=T(x_{q})$. These visual and text tokens are concatenated into a unified sequence $[h_{v},h_{q}]$, which serves as the input to the decoder component of the large language model $L$. The model then utilizes this combined representation to infer the appropriate answer $ans=L([h_{v},h_{q}])$, demonstrating the capability of these models to perform cross-modal reasoning and answer multimodal queries.

### 3.2 Data Construction

To enhance the MLLM with the DualFocus mechanism, we curated a dataset derived from the extensive Visual Genome (VG) dataset (Krishna et al., ), which provides a diverse array of images coupled with corresponding questions, answers, and annotated bounding boxes. These bounding boxes explicitly demarcate the regions of interest within the image pertinent to the question posed.

Ambiguity Filtration.
Initially, we scrutinize each data entry from VG to ensure its precision and clarity. During this process, we encountered instances where a question such as “What is the color of the person’s shirt?” might correspond to a scene depicting multiple individuals, leading to ambiguity in the dataset. To establish a one-to-one mapping between visual cues and textual queries, we employed a strict filtering criterion to exclude such ambiguous samples. Through this rigorous refinement, we distilled our dataset to 143k unequivocal image-question pairs.

Reformatting.
For enhanced interaction with our MLLM’s training regime, we transmuted the dataset samples into a conversational format that encapsulates both the query and spatial awareness components. The schema of a data sample is as follows,

$$ $\displaystyle\texttt{$Q_{1}$}:\texttt{<img>}\textit{Provide the box coordinates of the region}$ this question is asking about: <question> $\displaystyle\texttt{$A_{1}$}:\texttt{<box>}$ $\displaystyle\texttt{$Q_{2}$}:\texttt{<sub img>}\textit{Combine these two images and}$ answer the question: <question> $\displaystyle\texttt{$A_{2}$}:\texttt{<answer>}\vspace{-2mm}$ $$

In the first round of inquiry ($Q_{1},A_{1}$), we task the MLLM with deducing the important subregion <box> that pertinent to the question <question> in the image <img>, supplying it with micro-level details it needs to focus on.
The subsequent round ($Q_{2},A_{2}$) is constructed to aggregate the augmented view <sub img> of the identified sub-region and the original contextualized image <img> to infer the answer <answer>.

### 3.3 Training

During training, we integrate our curated VG data with standard VQA datasets to enhance the model’s capabilities on both micro and macro levels.
We adhere conventional MLLM training procedures using standard VQA datasets to equip the model with macro capabilities.
Subsequent sections primarily focus on elaborating how we augment the model’s proficiency in identifying micro details through our transformed VG data.
This enhancement is achieved by dividing the training process into two distinctive yet interconnected tasks.

Task I: Identification of the Pertinent Subregion.
Given an image $v$ and the query $q$, we prompt the model with instruction $q_{1}$ to identify the region corresponding to the query $q$. To model this, we tokenize $q_{1}$ into tokens $h_{q_{1}}$ using the tokenizer $T(.)$, and the visual embedding $h_{v}$ is obtained from the input image $v$. The model prediction $\hat{box}$, representing the bounding box coordinates, is then inferred through the language model:

$$ $\hat{box}=L([h_{v},h_{q_{1}}]),$ (1) $$

where $\hat{box}=(\hat{x_{1}},\hat{y_{1}},\hat{x_{2}},\hat{y_{2}})$, representing the coordinates of the two corners of the bounding box within the image. The coordinates are expressed as numeric values embedded in natural language, with no additional formatting or special tokens, to maintain coherence with the LLM’s language processing capabilities.

Task II: In-depth Examination and Answer Generation.
Upon deducing the focused region, we extract and upscale the sub-image $v^{\prime}$ using the corresponding bounding box coordinates to maintain the original resolution:
$v^{\prime}=zoom(crop(v,box))$.
To ensure that the context of the entire image is not lost, both the original image $v$ and the processed sub-image $v^{\prime}$ are encoded separately by the visual encoder, producing two sets of visual tokens $h_{v}$ and $h_{v}^{\prime}$, respectively.
These visual tokens are concurrently concatenated with the text embedding generated from the original question and its corresponding answer from the first task, structured as $[h_{v},h_{q_{1}},h_{box},h_{v}^{\prime},h_{q_{2}}]$. The model then employs this concatenated information to produce the final answer,

$$ $\hat{ans}=L([h_{v},h_{q_{1}},h_{box},h_{v}^{\prime},h_{q_{2}}]).$ (2) $$

Objective Function.
The training loss is partitioned into two distinct segments corresponding to the tasks detailed above. Since both the bounding box and the final answer are enunciated in natural language, we employ a standard cross-entropy loss function $\mathcal{L}_{CE}$ for each task. Formally, the collective loss is the aggregation of these binary components:

$$ $\mathcal{L}_{total}=\mathcal{L}_{1}(\hat{box},box)+\mathcal{L}_{2}(\hat{ans}, ans),$ (3) $$

where $\mathcal{L}_{1}$ computes the discrepancy between the actual ($box$) and predicted ($\hat{box}$) bounding boxes, and $\mathcal{L}_{2}$ quantifies the differential between the true final answer ($ans$) and the inferred one ($\hat{ans}$).

Through systematic training across these two focused tasks, the model gradually develops an adeptness in isolating and scrutinizing specific subregion within an image, thereby refining its capacity for fine-grained detail discernment.

Figure: Figure 2: PPL distribution for MicroFocus compared to baseline on tasks emphasizing different cognitive demands.
Refer to caption: x3.png

### 3.4 Inference

Upon training completion, our model acquires dual capabilities, namely the ability to generate macro-level answers ($\hat{ans}_{\text{macro}}$) directly from the holistic image and the capacity to produce micro-level answers ($\hat{ans}_{\text{micro}}$) using the fine-grained details from the predicted subregion. Thus, we adopt two distinct pathways for interpreting the given data.

Inference Pathways.
Specifically, the macro answer pathway, akin to the traditional method, which maintains the conventional inference process, directly generating an answer,

$$ $\hat{ans}_{\text{macro}}=L([h_{v},h_{q}]),$ (4) $$

without emphasizing localized regions.
Contrarily, the micro answer pathway mimics the training phase,

$$ $\hat{ans}_{\text{micro}}=L([h_{v},h_{q_{1}},h_{\hat{box}},h_{v}^{\prime},h_{q_ {2}}]),$ (5) $$

except during inference we utilize the predicted bounding box $\hat{box}$ instead of the ground truth in Equ. 2. The micro pathway leverages the predicted bounding box $\hat{box}$ to focus on a specific sub-region.

Answer Selection Via Perplexity.
To ascertain the most coherent response, we evaluate both $\hat{ans}_{\text{macro}}$ and $\hat{ans}_{\text{micro}}$ through their respective perplexity (PPL). The PPL serves as an estimate of the likelihood for a given sequence of tokens, with lower values indicating higher probability (better model confidence). This is given by:

$$ $\text{PPL}(\hat{ans})=\exp\left(-\frac{1}{N}\sum_{i=1}^{N}\log{p(w_{i}|w_{<i}) }\right),$ (6) $$

where $N$ is the number of tokens in the answer, $p(w_{i}|w_{<i})$ represents the model’s estimated probability for token $w_{i}$ given the preceding context.
The answer affiliated with the lower PPL is deemed more likely correct and thus selected as the final answer:

$$ $\hat{ans}=\begin{cases}\hat{ans}_{\text{macro}},&\text{if PPL}(\hat{ans}_{ \text{macro}})<\text{PPL}(\hat{ans}_{\text{micro}})\\ \hat{ans}_{\text{micro}},&\text{otherwise}\end{cases}$ (7) $$

The Motivation of Perplexity-guided Selection.
As depicted in Fig. 2 (Left), the micro answer demonstrates superior confidence in scenarios requiring detailed discernment (e.g., text understanding). However, its performance degrades in tasks involving global comprehension (e.g., spatial relationships), as shown in Fig. 2 (Right), despite the micro answer being generated by concatenating the original image and sub-images. We conjecture that this degradation is attributable to the high dependence of the micro answer on the nearest image, i.e., sub-image, akin to the high dependency observed in closely located text tokens.
This suggests selecting the micro and macro answers via perplexity to integrate both perspectives during the inference of MLLM.
By using a perplexity-guided dual-path inference system, the MLLM dynamically switches between a global understanding and a focused comprehension dependent on the nature of the query, ultimately enhancing the efficacy of the model’s responses.

**Table 1: Comparison with baseline methods on various benchmarks. Our DualFocus consistently demonstrates improvements across various baselines and benchmarks.**
| Method | Encoder-V | LLM | SEED${}^{IMG}$ | MMB | GQA${}^{*}$ | VQA${}^{T}$ |
| --- | --- | --- | --- | --- | --- | --- |
| LLaVA-1.5 | ViT-L | Vicuna-7B | 66.2 | 64.3 | 67.2 | 58.2 |
| LLaVA-1.5 + DF | 68.9 (+2.7) | 66.6 (+2.3) | 69.3 (+2.1) | 62.0 (+3.8) |  |  |
| LLaVA-1.5 | ViT-L | Vicuna-13B | 68.2 | 67.7 | 69.3 | 61.3 |
| LLaVA-1.5 + DF | 71.0 (+2.8) | 70.7 (+3.0) | 74.5 (+4.2) | 65.5 (+4.2) |  |  |
| Qwen-VL-Chat | ViT-G | Qwen-7B | 65.4 | 60.6 | 69.0 | 61.5 |
| Qwen-VL-Chat + DF | 66.6 (+1.2) | 63.2 (+2.6) | 73.0 (+4.0) | 63.7 (+2.2) |  |  |

## 4 Experiments

### 4.1 Benchmarks

To thoroughly assess our proposed DualFocus enhanced MLLM, we evaluated its performance across a spectrum of benchmarks, covering traditional academic Visual Question Answering (VQA) tasks (GQA (Hudson & Manning, ), TextVQA (Singh et al., )) and recent benchmarks specifically designed for evaluating large multimodal models, namely MMBench (Liu et al., ) and SEED ().
MMBench is constructed with manually designed questions to critically assess the model’s vision-related reasoning and perceptual abilities.
SEED, leveraging GPT-4 for generation, introduces a dataset of nearly 19,000 questions centered on images and videos. Herein, our emphasis is placed on the image component., referred to as SEED${}^{IMG}$.
GQA and TextVQA represent benchmarks in the domain of traditional Visual Question Answering tasks, with GQA assessing the model’s ability to answer open-ended questions about images accurately, and TextVQA focusing on questions requiring the understanding of text within images.
Notably, GQA’s evaluations revealed considerable variability due to discrepancies in the answer format. To address this, we employed GPT-3.5 to reformat answers into a multiple-choice question format, resulting in an adjusted benchmark referred to as GQA${}^{*}$.

### 4.2 Implementation Details

All experiments were performed using LLaVA-1.5 (Liu et al., ) and Qwen-VL-Chat (Bai et al., ), adhering to their default hyper-parameters and training configurations, unless stated otherwise. Our methodology uniquely altered the fine-tuning stage by incorporating the converted 143k VG data to fortify the MLLM with the DualFocus mechanism.
For LLaVA-1.5, CLIP-ViT-L (Radford et al., ) served as the visual encoder at 336-resolution, and Vicuna 7B(13B) (Chiang et al., ) functioned as the LLM. During training we only freeze the visual encoder but fine-tune the connection layers and LLM.
For Qwen-VL-Chat, CLIP-ViT-G served as the visual encoder at 448-resolution and Qwen-7B (Bai et al., ) functioned as its LLM. During training, given memory constraints, we freeze the visual encoder and LLM, only fine-tune the LoRA (Hu et al., ) and connection layers.
The fine-tuning process for both models was completed within a single epoch.

### 4.3 Main Results

Comparison with Baseline Model.
We first conducted comparisons against baseline MLLMs LLaVA-1.5 and Qwen-VL-Chat across four benchmarks: SEED, MMBench, GQA, and TextVQA. Our DualFocus mechanism notably enhances the performance of both methods, as outlined in Table 1.
Specifically, our DualFocus improves LLaVA-1.5 with Vicuna-7B by 2.7, 2.3, 2.1, and 3.8, respectively.
With the larger LLM, Vicuna-13B, DualFocus secures even more substantial gains: 2.8, 3.0, 4.2, and 4.2, on SEED, MMBench, GQA, and TextVQA, respectively.
This trend is consistent when applying DualFocus to Qwen-VL-Chat, yielding boosts of 1.2, 2.6, 4.0 and 2.2 on the same benchmarks, respectively. These results highlight DualFocus’s versatility and its capability to significantly elevate MLLM performance across diverse benchmarks.

**Table 2: Comparison with SoTA methods on various benchmarks. The best result and the second-best result should be indicated using bold and underline, respectively.**
| Method | Res | Encoder-V | LLM | SEED${}^{IMG}$ | MMB | GQA${}^{*}$ | TQA${}^{T}$ |
| --- | --- | --- | --- | --- | --- | --- | --- |
| InstructBLIP | 224 | ViT-G | Vicuna-7B | 53.4 | 36.0 | - | 50.1 |
| LLaVA | 224 | ViT-L | Vicuna-7B | 25.5 | 34.1 | - | - |
| LLaVA-1.5 | 336 | ViT-L | Vicuna-7B | 66.2 | 64.3 | 67.2 | 58.2 |
| Share4V | 336 | ViT-L | Vicuna-7B | 69.7 | 68.8 | 70.5 | 60.4 |
| Qwen-VL-Chat | 448 | ViT-G | Qwen-7B | 65.4 | 58.2 | 69.0 | 61.5 |
| Monkey | 896 | ViT-G | Qwen-7B | 64.3 | 59.6 | - | 67.6 |
| OtterHD | 1024 | - | Fuyu-8B | - | 58.3 | - | - |
| BLIP-2 | 224 | ViT-L | Vicuna-13B | - | 46.4 | - | 42.5 |
| Shikra | 224 | ViT-L | Vicuna-13B | - | 58.8 | - | - |
| LLaVA-1.5 | 336 | ViT-L | Vicuna-13B | 68.2 | 67.7 | 69.3 | 61.3 |
| Share4V | 336 | ViT-L | Vicuna-13B | 70.8 | 68.5 | 71.1 | 62.2 |
| LLaVA-1.5-DF (ours) | 336 | ViT-L | Vicuna-13B | 71.0 | 70.7 | 74.5 | 65.5 |
| Share4V-DF (ours) | 336 | ViT-L | Vicuna-13B | 72.9 | 70.7 | 75.7 | 66.2 |

Comparison with SoTA Model.
Subsequently, we conduct a comparison of DualFocus with other SoTA MLLMs that vary in their input resolutions (Res), visual encoders (Encoder-V), and language models (LLM) on Table 2.
We incorporate our DualFocus into LLaVA-1.5 Vicuna-13B and ShareGPT4V (Chen et al., ), a derivative of LLaVA, named as LLaVA-1.5-DF and Share4V-DF, exhibit superior performance across four distinct benchmarks.
Specifically, Share4V-DF surpasses its closest competitor by 2 on the SEED benchmark. Similarly, LLaVA-1.5-DF leads the second-best performer by 1.9 on the MMBench.
The results are even more pronounced on the GQA and Text-VQA benchmarks, which demand a higher capacity for detailed perception.
Specifically, DualFocus improved Share4V by 4.7 and 4.0 on these two benchmarks, respectively.
While Monkey (Li et al., ) achieves the highest score of 67.6 on TextVQA using a larger input of 896 x 896, it falls short on more comprehensive benchmarks like SEED and MMBench. In contrast, our Share4V-DF performs similarly on TextVQA with a much smaller input size of 336 x 336 and significantly better on the other two benchmarks, demonstrating DualFocus’s ability to maintain a balance between a micro and macro perspective, making it a versatile and efficient mechanism for improving MLLM performance.

### 4.4 Ablation Study

In this section, we first analyze the impact of each inference pathway, and then we show the effect of each component and why they work. Unless otherwise specified, all ablation results are based on LLaVA-1.5.

**Table 3: Performance comparison on different inference strategy for baseline LLaVA-1.5 and our model. “Macro” and “Micro” refers to employ macro and micro answer pathway, respectively. “N/A” denotes the model fail to follow the instruction.**
| Method | Macro | Micro | SEED${}^{I}$ | VQA${}^{T}$ |
| --- | --- | --- | --- | --- |
| Base | ✓ |  | 66.2 | 58.2 |
|  | ✓ | N/A | N/A |  |
| Ours | ✓ |  | 66.7 | 58.6 |
|  | ✓ | 67.7 | 61.3 |  |
| ✓ | ✓ | 68.9 | 62.0 |  |

Inference Pathway Analysis.
Table 3 illustrates the contributions of the micro and macro inference pathways to our method’s performance. The initial results from the baseline model, LLaVA-1.5, indicate failure to implement the micro pathway due to the absence of training with similar directives. The integration of our custom 143k VG dataset, enabled the model to follow the DF inference guidelines. However, this adaptation led to minor improvements, i.e., increasing by +0.5 on the SEED metric and +0.4 on the TextVQA metric, suggesting that the dataset alone is insufficient to enhance performance.

Furthermore, the micro pathway’s implementation resulted in a significant +1.0 gain on the SEED metric and a notable +2.7 gain on the TextVQA metric, supporting our hypothesis that the micro pathway excels in nuanced tasks. Conversely, global comprehension tasks benefit from the PPL model, as evidenced by a +1.2 gain on the SEED metric and a moderate +0.7 gain on the TextVQA metric. This data underscores the importance of employing the appropriate inference pathway based on the task’s requirements.

**Table 4: Results on POPE. “LLaVA” refers to LLaVA-1.5. DualFocus is beneficial to mitigate Hallucination of MLLM. Here A, P, R denotes adversarial, popular and random split of POPE, respectively. “F1” and “Acc” denotes F1 score and accuracy, respectively.**
|  | F1(A) | Acc(A) | F1(P) | Acc(P) | F1(R) | Acc(R) |
| --- | --- | --- | --- | --- | --- | --- |
| LLaVA | 84.2 | 85.2 | 86.2 | 87.3 | 87.4 | 88.2 |
| LLaVA + DF | 86.0 | 86.2 | 88.6 | 89.1 | 89.7 | 90.0 |

Figure: Figure 3: Accuracy of baseline LLaVA-1.5 and our LLaVA-1.5-DF on SEED Benchmark tasks across various granularities. Our Dual-Focus significantly improves accuracy on fine-grained tasks.
Refer to caption: x4.png

Figure: Figure 4: Comparative visualizations between LLaVA-1.5 and our DualFocus method on SEED (first row) and TextVQA datasets (second row). LLaVA-1.5 is struggle to capture micro details. In contrast, our DualFocus mechanism leverages the zoomed-in sub-region (highlighted with a yellow bounding box) to achieve improved discernment of fine-grained details.
Refer to caption: x5.png

Hallucination Mitigation.
Hallucination within MLLM presents a critical challenge where the model creates imaginary content that is not actually present in the image. The benchmark POPE (Li et al., ), is designed to evaluate such hallucinations in MLLM through three distinct data splits: adversarial (A), popular (P), and random (R). As indicated in Table 4, integrating our DualFocus into MLLM yields substantial improvements in accuracy and the F1 score across these data splits.
Specifically, it improvements baseline by 2.4 and 2.3 on the F1 score of splits “P” and “R”, respectively. Even on the most difficult split “A”, it yields 1.8 gains on the F1 score.
The effectiveness of DF is attributed to that our DualFocus directs the model’s attention towards specific, relevant parts of an image in connection to the posed question, reducing the generation of non-pertinent features and subsequently diminishing the likelihood of hallucinations.

Fine-Grained Perception Enhancement.
In this section, we delve into the effectiveness of the DualFocus mechanism on tasks emphasizing different cognitive demands.
For a clear demonstration, we only involve the micro answer pathway here.
We use SEED as a benchmark because it provides a comprehensive assessment of a model’s capabilities across different dimensions and levels of detail.
Specifically, we examine four key dimensions: Instance Counting, Scene Understanding, Instance Attributes, and Text Understanding. The first two dimensions primarily concern the broader context of a situation, emphasizing a macro perspective. In contrast, the latter two focus on more intricate, micro-level details.
Experiment results are presented in Figure 3, illustrating that while our DualFocus mechanism delivers modest improvements in the domains of Instances Counting and Scene Understanding (-0.4, +0.7), it significantly enhances performance on Instance Attributes and Text Understanding (+4.3, +12.9).
These results underscore the effectiveness of the DualFocus approach, particularly in tasks requiring acute attention to detail, thereby confirming its utility in dissecting and interpreting finer elements within data.

**Table 5: Performance comparison of different methods for implementing the PPL strategy on MMBench. “LLaVA” and “Qwen” refer to LLaVA-1.5 and Qwen-VL-Chat. “LLaVA + PPL” denotes using PPL to choose answers generated by LLaVA with two distinct prompts, a process mirrored in “Qwen + PPL”. “LLaVA + Qwen + PPL” refer to use PPL to select the best answer from LLaVA and Qwen.**
| Method | LLaVA | Qwen | PPL | Acc |
| --- | --- | --- | --- | --- |
| Base | ✓ |  |  | 64.3 |
| ✓ |  | ✓ | 64.1 |  |
|  | ✓ |  | 60.6 |  |
|  | ✓ | ✓ | 60.7 |  |
| ✓ | ✓ | ✓ | 62.5 |  |
| Our | LLaVA-DF | 66.6 |  |  |
| Qwen-DF | 63.2 |  |  |  |

PPL-Guided Answer Specialization.
During inference, we utilized PPL to select answers by combining the micro and macro inference pathways, essentially creating a unique assembly method. We further examine various assembly approaches, detailed in Table 5.
The first strategy involves using PPL to merge answers from same models but varying input formats, labeled as “LLaVA + PPL” and “Qwen + PPL”.
Given that the base model is limited to macro inference pathways, we applied two distinct prompts. Results indicate a minor impact on performance, with changes of -0.2 and +0.1, respectively.
Another assembly strategy involves using PPL to merge answers from different models, tagged as “LLaVA + Qwen + PPL”. This approach significantly improved Qwen by +1.9, yet it reduced LLaVA’s performance by 1.8. We suspect this variance results from differing model architectures and training methodologies.
In contrast, DualFocus integrates micro and macro pathways within each model, applied to LLaVA and Qwen resulted in substantial gains of +2.3 and +2.6, respectively, are both higher than “LLaVA + Qwen + PPL”. This suggests combining micro and macro inferences within a single model outperforms assembling answers across different models.

## 5 Conclusion

In this work, we introduced DualFocus, a novel approach to enhance the performance of Multi-modal Large Language Models (MLLMs) by integrating both macro and micro perspectives for improved visual question answering. Through comparative studies, DualFocus demonstrated superior capability in handling detailed features and mitigating hallucination, thereby outperforming existing methods. This method not only advances MLLM efficacy but also paves the way for more human-like visual reasoning in AI.

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. Dataset Diversity: Given the potential limitations of the Visual Genome dataset, did the authors consider constructing a generalized sub-region grounding dataset to improve the model's robustness across diverse visual contexts?
2. Computational Efficiency: Since DualFocus introduces additional computational layers, have the authors explored lightweight optimizations or alternative methods to achieve similar performance enhancements with reduced resource requirements?
3. Failure Case Analysis: Could the authors provide further insights into cases where DualFocus may not perform as expected? A more detailed analysis of such scenarios would be valuable for refining the model and understanding its limitations in practical applications.
4. Lack of Analysis on Visual Encoder: The ViT visual encoder is only pretrained on holistic images, whereas in this paper, ViT is used for both holistic images and sub-region zoomed images. This may create an input domain gap, making it challenging for ViT to accurately represent sub-region image features. Could the authors address this issue?
soundness: 3
strengths: 1. Innovative Approach: DualFocus presents a novel approach by combining macro and micro perspectives, offering a unique method for MLLMs to enhance their visual question answering capabilities.
2. Comprehensive Evaluation: The paper thoroughly evaluates DualFocus across multiple benchmarks, including SEED, MMBench, and TextVQA, showing its versatility and effectiveness in improving MLLM performance.
3. Practical Impact: By reducing hallucinations and improving the model’s ability to capture fine-grained details, DualFocus addresses practical challenges in MLLMs, making it potentially impactful for real-world applications that require accurate visual interpretation.
summary: This paper presents DualFocus, an innovative approach designed to improve multi-modal large language models (MLLMs) in visual question answering by integrating both macro (global) and micro (focused) perspectives. Inspired by human perceptual behaviors, DualFocus enables models to examine an entire image first and then zoom in on question-relevant regions for finer detail. This dual-path approach allows for accurate responses to questions requiring both broad and detailed visual understanding. Through extensive experiments on benchmarks like SEED and GQA, the authors demonstrate that DualFocus enhances model accuracy and reduces hallucinations, addressing significant limitations in current MLLMs that struggle with fine-grained details.
weaknesses: 1. Dataset Limitation: The reliance on the Visual Genome dataset for subregion identification may introduce limitations in generalizability, as VG may not represent general real-world visual contexts.
2. Computational Cost: DualFocus introduces additional computational overhead with its two-step zoom and evaluation process, which may limit further research and its deployment in resource-constrained environments.
3. Limited Analysis of Failure Cases: While the paper demonstrates the model's improvements, there is limited discussion on scenarios where DualFocus might fail, especially for highly ambiguous or complex images where even zoomed views could mislead the model.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: The paper is well-written and easy to understand.

The DualFocus strategy is an interesting and novel approach, integrating macro and micro perspectives to improve visual comprehension. 

The effectiveness of this approach, especially when combined with the perplexity (PPL) metric for answer selection, is well-supported by the experimental results provided in the paper.
summary: The paper introduces DualFocus, a novel approach for multi-modal large language models (MLLMs) that integrates macro (global) and micro (detailed) perspectives to enhance performance on vision-language tasks. Inspired by human perceptual behavior, DualFocus allows the model to examine an entire image globally and then focus on specific sub-regions for detailed analysis, emulating a zoom-in function. This dual analysis is achieved by generating both a macro-level answer and a micro-level answer, with a perplexity (PPL)-guided selection process choosing the more confident response as the final answer.
weaknesses: 1. **Differentiation from Prior Work**: The paper lacks a clear discussion on how DualFocus fundamentally differs from similar prior approaches, such as the V* model. Clarifying the unique contributions of DualFocus in comparison to existing methods would strengthen the case for its novelty.

2. **Performance Gains from Ppl-Guided Answer Selection:** Perplexity-guided answer selection is a commonly used strategy to boost performance during inference. It is unclear whether the observed performance gains can be attributed specifically to DualFocus’s capabilities in enhancing perceptual understanding, as claimed by the authors, or if they are largely due to the PPL metric. Testing this strategy with a baseline model (e.g., LLaVA-1.5) under similar dual inference conditions could provide a clearer understanding of where these gains originate. **This also makes me question the contribution of this paper.**

3. **Further Comparison**: The comparisons in the paper could be more comprehensive. If we use an external detection model to generate bounding boxes as prompts for baseline models, can we also yield performance improvements?

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: See weaknesses
soundness: 2
strengths: This paper:
1. proposes a feasible (and potentially applicable) solution for high-resolution QA problems. It utilizes the model’s region perception ability to dynamically sample image crops and provide further answers, enhancing detail perception without significantly increasing image tokens.
2. explicitly supervises region prediction, effectively improving performance.
3. uses PPL (Perplexity) to dynamically select answers, effectively balancing responses from both perspectives.
summary: The paper introduces DualFocus, which addresses questions related to images (primarily high-resolution) from both global and regional perspectives. It systematically selects responses through PPL. Improvements have been achieved on several perception benchmarks.
weaknesses: 1. Regarding motivation, there are two core issues:
- While cropping out regions as independent contexts can help the model better perceive these areas, it may weaken the model’s perception of other local areas at that moment. This seems to conflict with human perceptual mechanisms.
- Methods like Qwen2VL, which use a sufficient number of tokens to encode each image, are becoming mainstream and are engineering-wise feasible. Given this trend, is it still necessary to employ such dynamic region sampling methods?
2. Technically, this paper employs some trivial techniques to improve upon V*, achieving certain effects but lacking in novelty and insight.
3. This paper leverages the model’s own capability to perform region search, replacing the heatmap-based search algorithm used by V*. In fact, grounding methods based on MLLM are already quite mature. For instance, Shikra/QwenVL directly model bounding boxes, and GPT-4ROI incorporates regions into context. While this article claims to model this capability, it lacks a reasonable evaluation and an assessment of how this modeling approach impacts general performance.