Reconstructive Visual Instruction Tuning

## Abstract

Abstract This paper introduces r ec o nstructive vi s ual in s truction tuning ( Ross ), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals .
In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, Ross prompts LMMs to supervise visual outputs via reconstructing input images.
By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision.
However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals.
To address this issue, Ross employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values.
This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations.
Empirically, Ross consistently brings significant improvements across different visual encoders and language models.
In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, Ross delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs .

## 1 Introduction

The success of GPT-style Large Language Models (LLMs) (Radford et al., ; ; Brown et al., ; OpenAI, ; Yang et al., ; Touvron et al., ; Chiang et al., ; Dubey et al., ) has motivated researchers to adapt LLMs to understand multimodal inputs (Liu et al., ; ; Dai et al., ; Bai et al., ).
Notably, visual instruction tuning approaches (Liu et al., ) demonstrate superior performance with cost-efficient training recipes.
Some approaches (Chen et al., ; Li et al., ) even surpass GPT-4V(ision) (OpenAI, ) on benchmark evaluations.

Typically, these Large Multimodal Models (LMMs) based on visual instruction tuning adopt a plug-in architecture, as depicted in Figure 1a, where pre-trained vision-language foundation models such as CLIP (Radford et al., ) are responsible for projecting images into visual tokens.
They serve as prefix tokens for multimodal comprehension.
However, this type of design, i.e., visual encoder $\to$ connector $\to$ LLM $\Leftarrow$ language instructions, where “$\Leftarrow$” indicates supervision, is primarily LLM-centric: (i) visual comprehension largely depends on vision-to-text alignment and the selected vision models, and (ii) supervision derives exclusively from text data.
As a result, they exhibit systematic visual shortcomings such as recognizing specific visual patterns (Tong et al., ).

Until very recently, some concurrent works proposed vision-centric solutions (Tong et al., ; ).
Illustrated in Figure 1b, their solutions leverage extrinsic assistance via aggregating several different visual experts.
Inspired by the evolution in image recognition, from manually designed visual features (Sánchez & Perronnin, ) to learnable deep convolutional models (Krizhevsky et al., ), we suggest that intrinsic activation offers a more viable path forward.
Just as deep models automatically learn hierarchical and abstract features from raw data, we believe intrinsic activation methods are similarly more adaptable for multimodal comprehension, reducing reliance on hand-crafted engineering, thereby enhancing both generalization and performance.
Therefore, we aim to explore intrinsic activation solutions based on the following principles:

- 1.
Supervise Visual Outputs.
Current LMMs solely supervise text outputs, neglecting a significant amount of visual outputs unused.
For instance, LLaVA-v1.5 (Liu et al., ) utilizes $576$ visual tokens to represent a single $336\times 336$ image, yet their corresponding outputs remain unsupervised.
Intuitively, since input images themselves inherently provide rich and detailed information, we regard LMMs reconstructing input images as the supervision of those visual outputs.
This approach encourages LMMs to maintain low-level details, thereby enhancing their fine-grained comprehension abilities and reducing hallucinations.
- 2.
Explore the Optimal Formulation.
Designing this self-supervised task effectively is not straightforward.
Motivated by the success of masked autoencoder (He et al., ) compared to its basic version denoising autoencoder (Vincent et al., ), we identify handling heavy spatial redundancy of visual signals as the underlying key factor.
To this end, we formulate our approach as follows: (i) for reconstruction targets, instead of raw RGB pixels, we make LMMs reconstruct latent visual tokens, and (ii) for reconstruction objectives, to avoid directly regressing exact token values, we adopt per-token denoising.

To this end, we propose Ross, termed of reconstructive visual instruction tuning, which utilizes input images as direct supervision signals illustrated in Figure 1c.
Technically, to address the spatial redundancy inherent in natural visual signals (He et al., ), we train a small denoising network, which takes high-level visual outputs $x$ as conditions to recover low-level fine-grained visual tokens $z$, representing an underlying distribution $p(z|x)$.
These latent tokens $z$ are derived from a frozen teacher tokenizer such as continuous VAE (Kingma, ) and discrete VQGAN (Esser et al., ).
Unlike extrinsic assistance solutions (Tong et al., ; ), our intrinsic activation solution naturally maintains a lightweight inference procedure.
More importantly, when adapting to new visual domains, our solution avoids a careful choice of new domain-specific experts, e.g., MiDaS-3.0 (Birkl et al., ) for understanding depth maps, which is more efficient and easier to implement.

Figure: Figure 1: Conceptual comparison between different pipelines. (a) Typical visual instruction tuning approaches (Liu et al., ; ) follow a LLM-centric design that solely leverage text supervision. (b) Aggregated visual instruction tuning alternatives (Tong et al., ; ) leverages extrinsic assistance via combining several visual experts, requiring a careful selection of visual experts. (c) Our Ross, with a single visual encoder, e.g., CLIP (Radford et al., ) and SigLIP (Zhai et al., ), designs extra vision-centric reconstructive supervision as intrinsic activation. In this way, LMMs are required to preserve every detail of input images, thereby enhancing multimodal comprehension capabilities and reducing hallucinations.
Refer to caption: x1.png

Empirically, Ross achieves top performance across a wide range of multimodal comprehension benchmarks.
Notably, our Ross excels in fine-grained vision-centric benchmarks (Tong et al., ; Masry et al., ) and hallucination benchmarks (Guan et al., ; Li et al., ).
To be specific, with a single SigLIP (Zhai et al., ) as the visual encoder, Ross-7B achieves 57.3 on HallusionBench (Guan et al., ) and 54.7 on MMVP (Tong et al., ), significantly outperforms state-of-the-art alternatives with similar model sizes which aggregate several visual experts as extrinsic assistance, e.g., Cambrian-1-8B (Tong et al., ).
In-depth analysis demonstrates the effectiveness of Ross for directing focus towards visual elements and understanding depth maps.
We hope our research will inspire future work in designing supervision signals for large multimodal models.

## 2 Related Work

Visual Instruction Tuning.
Most visual instruction tuning-based LMMs adopt a plug-in architecture (Liu et al., ; ; Bai et al., ), where a language-supervised visual encoder (Radford et al., ; Zhai et al., ) is responsible for extracting visual tokens.
A connector is used to map those visual representations into the LLM space, e.g., Resamplers (Alayrac et al., ), Q-Formers (Li et al., ; Dai et al., ; Bai et al., ; Ge et al., ), and MLPs (Liu et al., ; ; Li et al., ; Liu et al., ; Li et al., ).
These LMMs usually follow a two-stage training recipe.
During the alignment stage, the connector is trained on high-quality caption data.
Next, the full model is trained on single-image visual instruction tuning data.
However, only text outputs are supervised.
Ross, on the other hand, introduces novel vision-centric supervision via reconstructing fine-grained visual tokens conditioned on visual outputs.

Visual Encoders for LMMs.
As the original CLIP (Radford et al., ) adopted by conventional visual instruction tuning approaches is trained on noisy image-text pairs, it exhibits specific visual shortcomings, and thus stronger backbones (Fang et al., ; Zhai et al., ; Chen et al., ) have been introduced to LMMs.
Some concurrent works (Tong et al., ; ) leverage extrinsic assistance, which further utilizes vision-only self-supervised models (Oquab et al., ; Wang et al., ; ; ; He et al., ; Caron et al., ) and domain experts (Kirillov et al., ; Birkl et al., ; Rombach et al., ).
Ross, from a new intrinsic activation perspective, aims to catalyze enhanced comprehension through reconstructing input images with no extra visual experts.

Generative Objectives for LMMs.
Another line of work introduces pre-trained text-to-image diffusion models (Rombach et al., ) to make LMMs capable of both comprehension and generation (Dong et al., ; Ge et al., ; Sun et al., ; Ge et al., ; Sun et al., ).
Our Ross, with a totally different motivation, targets to catalyze multimodal comprehension via reconstruction.
Specifically, conditions are different, where Dong et al. () and Sun et al. () take outputs corresponding to learnable queries as conditions, while our Ross takes outputs corresponding to visual inputs.
Those methods are generative while Ross is reconstructive.
The detailed pipeline comparison can be found in Appendix C.

## 3 Preliminaries

Large Multimodal Models.
In the literature (Radford et al., ; ), a $\theta$-parameterized LLM models the canonical causal distribution of each text token $\bm{x}_{i}$ as $p_{\theta}(\bm{x})=\prod_{i=1}^{T}p_{\theta}(\bm{x}_{i}|\bm{x}_{<i})$, where $\{\bm{x}_{i}\}_{i=1}^{T}$ represents a sequence of text tokens.
To make LLMs understand visual contents, typical plug-in style LMMs (Liu et al., ; ) regard a sequence of visual tokens as prefix tokens.
Specifically, an input image $\bm{I}\in\mathbb{R}^{H\times W\times 3}$ is first projected into a sequence of visual tokens by a $\xi$-parameterized visual encoder $\mathcal{G}_{\xi}$ such as CLIP (Radford et al., ) and SigLIP (Zhai et al., ), where $(H,W)$ indicates the spatial resolution.
Then, a $\phi$-parameterized multimodal projector $\mathcal{H}_{\phi}$ is utilized to project these visual tokens into the feature space of LLMs.
As a result, the canonical causal distribution in a multimodal sentence containing an image $\bm{I}$ becomes

$$ $p_{\Theta}(\bm{x})=\prod_{i=1}^{T}p_{\Theta}(\bm{x}_{i}|\bm{x}_{<i},\bm{v}), \quad\bm{v}=\mathcal{H}_{\phi}\circ\mathcal{G}_{\xi}(\bm{I}),$ (1) $$

where $\Theta=\{\theta,\xi,\phi\}$ is the parameters and $\bm{v}\in\mathbb{R}^{N\times D}$ indicates the projected visual tokens.
$N$ is the number of visual tokens and $D$ indicates the feature channel.
The visual encoder $\mathcal{G}_{\xi}$ could be either frozen (Liu et al., ; ; Tong et al., ) or fine-tuned (Liu et al., ; Bai et al., ; Li et al., ; Wang et al., ).

Training Recipes for LMMs.
LMMs almost follow a two-stage training recipe (Liu et al., ), i.e., the pre-training stage (or the alignment stage) and the supervised fine-tuning stage (or the instruction tuning stage).
The instruction (supervision) comes from languages such as the answers to VQA tasks, maximizing the log-likelihood of text outputs:

$$ $\mathcal{L}_{\mathrm{LMM}}^{\mathrm{text}}(\Theta=\{\theta,\xi,\phi\},\bm{x}, \bm{I})=\frac{-1}{T-N}\sum_{i=N+1}^{T}\log p_{\Theta}(\bm{x}_{i}|\bm{x}_{<i}, \bm{v}),$ (2) $$

where $N$ represents the number of visual tokens and visual outputs (one input token corresponds to one visual output).
From Equation 2, we can tell that only text outputs $\bm{x}_{i>N}$ are supervised.

## 4 Ross : Reconstructive Visual Instruction Tuning

In this section, we first provide an overview of our reconstructive visual instruction tuning (Ross).
Then, we discuss our explorations towards the optimal formulation in the following subsections, with the ultimate goal of handling spatial redundancy of visual signals to provide meaningful visual supervision.
Our explorations mainly include reconstruction targets and the training objective.

Figure: Figure 2: Overview of Ross. Given an input image and the corresponding text to this image, Ross aims to supervise visual outputs by reconstruction.
Refer to caption: extracted/5922035/images/method_overview.png

Variants of Ross.
Evidently, different choices of $\mathcal{F}$ and $\mathcal{M}$ contribute to different variants.
$\mathcal{F}$ controls the reconstruction target while $\mathcal{M}$ defines the objective:

- 1.
Towards the target, $\mathcal{F}$ can be the pachify operation (Dosovitskiy et al., ), resulting in pixel-level reconstruction, or pre-trained fine-grained visual tokenizers such as VAE (Kingma, ) and VQGAN (Esser et al., ), leading to latent-level reconstruction.
$\mathcal{F}$ could even be vision-only models such as DINOv2 (Oquab et al., ), making LMMs learn specific visual patterns from $\mathcal{F}$, which is also a type of latent-level reconstruction.
- 2.
Towards the objective, the most straightforward choice of $\mathcal{M}$ is MSE or cosine similarity for regressing raw pixel values or latent features, respectively.
We also explore the denoising objective (Ho et al., ) to avoid being overwhelmed by fitting exact values.

We introduce our explorations step by step in the following sections.
The ultimate goal of our exploration is to design an appropriate self-supervised reconstructive pre-text task that provides meaningful vision-centric supervision signals to LMMs, where handling the spatial redundancy of visual signals (He et al., ) becomes the crux.

### 4.1 Ross R R {}^{\text{R}} start_FLOATSUPERSCRIPT R end_FLOATSUPERSCRIPT : Regressing as Reconstructive Visual Instruction

In this section, we introduce straightforward variants, i.e., regressing as reconstructive visual instruction.
As shown in Figure 3, depending on the choice of $\mathcal{F}$, it mainly has three variants: (a) Ross${}^{\text{R}}$-Pixel, (b) Ross${}^{\text{R}}$-Latent, and (c) Ross${}^{\text{R}}$-Latent2Pixel.

Directly Regressing Raw RGB Values.
The most straightforward variant is to directly regress raw RGB values illustrated in Figure 3a, called “Ross${}^{\text{R}}$-Pixel”.
Under such a setting, $\mathcal{F}$ is the patchify operation (Dosovitskiy et al., ), reshaping the image $\bm{I}\in\mathbb{R}^{H\times W\times 3}$ into a sequence of flattened 2D patches $\bm{I}_{p}\in\mathbb{R}^{N\times(3P^{2})}$, where $(P,P)$ is the resolution of each image patch and $N=HW/P^{2}$ indicates the resulting number of patches.
$\mathcal{J}_{\pi}$ can be a simple MLP, mapping the dimension of visual outputs $\bm{x}_{i\leq N}$ from $D$ to $3P^{2}$.
The measurement $\mathcal{M}$ is MSE.
However, as visual signals suffer from heavy spatial redundancy (He et al., ), such a design may not provide meaningful supervision to LMMs.
An intuitive alternative to avoid directly regressing raw RGB values while still reconstructing the image is to urge LMMs to reconstruct latent tokens, introduced as follows.

Regressing Latent Tokens.
Illustrated in Figure 3b, Ross${}^{\text{R}}$-Latent aims to regress fine-grained latent tokens extracted by the teacher tokenizer.
$\mathcal{F}$ can be models trained with discriminative tasks such as DINOv2 (Oquab et al., ) and DEIT-III (Touvron et al., ).
The encoder part of models trained with reconstruction tasks such as VQGAN (Esser et al., ) and VAE (Kingma, ) are also capable.
$\mathcal{M}$ here is the consine-similarity.
Intuitively, the decoder part of the latter is able to remap latent tokens into the pixel space.
Therefore, supervising in the pixel space via decoding becomes another valid variant introduced as follows.

Regressing RGB Values via Decoding.
Shown in Figure 3c, Ross${}^{\text{R}}$-Latent2Pixel requires a decoder to project predicted latent tokens $\hat{\bm{z}}$ into the RGB pixel space, resulting in predicted image $\hat{\bm{I}}$.
Let $\mathcal{F}^{-1}$ be the decoder part of VQGAN (Esser et al., ) or VAE (Kingma, ), and the regressive MSE objective $\mathcal{M}$ is performed on pixel-space.
Note that we simply use $\mathcal{F}^{-1}$ to represent the decoding process, which is actually not the inverse function of $\mathcal{F}$ mathematically.

Figure: Figure 3: Variants of Ross${}^{\text{R}}$, where regression objectives are either computed on raw RGB values in (a) and (c), or specific latent space determined by $\mathcal{F}$ in (b). We adopt MSE as $\mathcal{M}$ for pixel regression in (a) and (c), and cosine-similarity for latent regression in (b), respectively.
Refer to caption: extracted/5922035/images/RossR-Pixel.png

Discussion.
Recall that we need to find the optimal solution to address the spatial redundancy of natural visual signals, the target-level exploration above achieves this goal partially, as the objective is limited to vanilla regression.
To this end, inspired by Ho et al. () and Li et al. (), we further incorporate a novel denoising objective in the following section.

### 4.2 Ross D D {}^{\text{D}} start_FLOATSUPERSCRIPT D end_FLOATSUPERSCRIPT : Denoising as Reconstructive Visual Instruction

As an objective for handling heavy spatial redundancy to provide meaningful vision-centric supervision signals, denoising is better than vanilla regressing, since the introduction of noise into the training data acts as an implicit form of data augmentation and regularization.
The denoising process encourages the model to focus on the underlying data manifold rather than memorizing specific instance values (Chen et al., ; Song & Ermon, ; Karras et al., ; Yang et al., ).

Techinically, as illustrated in Figure 4a,
our final Ross${}^{\text{D}}$ takes high-level visual outputs $\bm{x}_{i\leq N}$ as conditions to recover clean fine-grained tokens $\bm{z}_{0}$ from noisy tokens $\bm{z}_{t}$.
Specifically, clean tokens $\bm{z}_{0}=\mathcal{F}(\bm{I})$ are obtained from the teacher tokenizer $\mathcal{F}$.
By default, we utilize a continuous VAE (Kingma, ) regularized by Kullback–Leibler (KL) divergence provided by Rombach et al. (), since it is believed to capture sufficient image details.
The training procedure of the denoiser $\mathcal{J}_{\pi}$ follows a diffusion process (Ho et al., ):

$$ $\mathcal{L}_{\mathrm{LMM}}^{\mathrm{visual}}(\Theta=\{\theta,\xi,\phi,\pi\}, \bm{x},\bm{I})=\mathbb{E}_{t,\bm{\epsilon}}\left[||\mathcal{J}_{\pi}(\bm{z}_{t };\mathbf{x}_{i\leq N},t)-\bm{\epsilon}||^{2}\right].$ (4) $$

The denoiser $\mathcal{J}_{\pi}$ actually estimates the conditional expectation $\mathbb{E}[\bm{\epsilon}\sim\mathcal{N}(\bm{0},\mathbf{I})|\bm{z}_{t}]$.
More details about the background knowledge of diffusion models can be found in Appendix A.

Figure: Figure 4: Illustration of (a) the training procedure of Ross${}^{\text{D}}$ and (b) the detailed architecture of the denoiser $\mathcal{J}_{\pi}$. (a) Ross${}^{\text{D}}$ introduces visual guidance via denoising fine-grained visual tokens $\bm{z}_{0}$ conditioning on visual outputs $\bm{x}_{i\leq N}$. (b) The denoiser takes noisy tokens $\bm{z}_{t}$, current timesteps $t$, and conditions $\bm{x}_{i\leq N}$ as inputs and outputs the predicted noise $\hat{\epsilon}_{t}$. Each denoiser block consists of three linear projection layers and a standard self-attention block (Vaswani et al., ).
Refer to caption: extracted/5922035/images/method.png

Architecture of the Denoiser.
As conditions $\bm{x}_{i\leq N}$ are causal, we introduce a self-attention module to model the inter-token dependencies illustrated in Figure 4b.
Specifically, the architecture of the denoiser $\mathcal{J}_{\pi}$ is a stack of Transformer Encoder blocks (Vaswani et al., ) and each block contains three extra projections for conditions $\bm{x}_{i\leq N}$, inputs $\bm{z}_{t}$, and timesteps $t$, respectively.

Choices of the Teacher Tokenizer.
By default, we adopt latent denoising and we take a continuous tokenizer provided by Rombach et al. () as $\mathcal{F}$, since it manages to reconstruct input images with a low rFID (Heusel et al., ) and thus it is expected to preserve many low-level details of input images.
This extra reconstructive objective, however, is not limited to any certain tokenizer $\mathcal{F}$.
Discrete tokenizers such as VQGAN (Esser et al., ), and vision self-supervised models such as DINOv2 (Oquab et al., ), are also qualified to be the tokenizer.
Even the patchify operation (Dosovitskiy et al., ) is capable, resulting in pixel denoising.

## 5 Experiments

### 5.1 Ablation Study

Implementation Details.
All ablation studies are implemented based on LLaVA-v1.5 (Liu et al., ).
The visual encoder $\mathcal{G}_{\xi}$ is CLIP-ViT-L/14@336 (Radford et al., ) and the base LLM is Qwen2-7B-Instruct (Yang et al., ).
The training data is LLaVA-558K (Liu et al., ) and Cambrian-737K (Tong et al., ) for the pre-training stage and the instruction tuning stage, respectively.
We evaluate our each variant of Ross mainly on (i) hallucination: POPE (Li et al., ) and HallusionBench (Guan et al., ), (ii) fine-grained comprehension: MMVP (Tong et al., ) and ChartQA (Masry et al., ), and (iii) general comprehension: MMBench (Liu et al., ) English dev split.
All evaluations are conducted with VLMEvalKit (Duan et al., ).
Evaluation prompts can be found in Appendix B.

Pixel Regression v.s. Latent Regression.
Starting from the visual instruction tuning baseline (Liu et al., ; ), we first explore the effectiveness of using regression as the objective for our reconstructive visual instruction tuning.
We utilize a continuous VAE (Kingma, ) with an encoder-decoder architecture provided by Rombach et al. (), where the encoder part serves as $\mathcal{F}$ for Ross${}^{\text{R}}$-Latent while the decoder part is $\mathcal{F}^{-1}$ for Ross${}^{\text{R}}$-Latent2Pixel.
As illustrated in Figure 8, our vision-centric regression supervision outperforms the visual instruction tuning baseline in most cases.
Moreover, latent regression performs the best since regressing raw RGB pixels fails to provide meaningful supervision signals, regardless of whether utilizing a decoder or not.

Figure: Figure 5: Pixel Regression v.s. Latent Regression. The teacher tokenizer $\mathcal{F}$ for Ross${}^{\text{R}}$-Latent is the encoder of a continuous VAE (Kingma, ) provided by Rombach et al. (), while its decoder serves as $\mathcal{F}^{-1}$ for Ross${}^{\text{R}}$-Latent2Pixel. Our vision-centric reconstructive supervision surpasses the visual instruction tuning baseline in most cases. Among three regression variants, Ross${}^{\text{R}}$-Latent performs the best, as it avoids explicitly regressing redundant raw RGB values.
Refer to caption: x2.png

Choices of $\mathcal{F}$.
We study the effectiveness across different latent teacher tokenizers $\mathcal{F}$ in Figure 8, including KL-16 provided by Rombach et al. (), which is a continuous VAE (Kingma, ) with Kullback–Leibler (KL) divergence, self-supervised DINOv2 (Oquab et al., ), fully-supervised DEiT-III (Touvron et al., ), and language-supervised EVA02CLIP (Fang et al., ).
Among them, KL-16 is the best choice.
One intuitive explanation is that it is expected to preserve the most image details, since it was originally designed to accurately reconstruct input images.

Regression v.s. Denoising.
In Figure 8, we study the effectiveness of the denoising objective over vanilla regression across different tokenizers, i.e., KL-16 (Rombach et al., ) and DINOv2 (Oquab et al., ).
Notably, even if Ross${}^{\text{R}}$-Latent (KL-16) has already outperformed the visual instruction tuning baseline by a large margin, Ross${}^{\text{D}}$ manages to bring significant improvements by replacing regression with denoising.
Therefore, denoising is better at handling visual spatial redundancy.

Finally, we leverage the insights and conclusions from all our previous studies to train our Ross.
Specifically, we regard the optimal formulation Ross${}^{\text{D}}$ (KL-16), i.e., denoising with the KL-16 (Rombach et al., ) tokenizer, as our final Ross.

### 5.2 In-Depth Analysis

**Table 1: Quantitative comparison on attention values. We conduct a T-test (Student, ) to compare the means and a Mann-Whitney U test (Mann & Whitney, ) to compare the medians of the two distributions. The mean and median of Ross are both significantly higher than those of LLaVA.**
| Statistic ($\times$10${}^{-\text{4}}$) | LLaVA | Ross | P-value |
| --- | --- | --- | --- |
| Mean | 2.03 | 2.36 | 1.27$\times$10${}^{-\text{7}}$ |
| 25th Percentile | 1.50 | 1.81 | – |
| Median | 1.90 | 2.26 | 4.39$\times$10${}^{-\text{9}}$ |
| 75th Percentile | 2.42 | 2.76 | – |
| 95th Percentile | 3.51 | 3.69 | – |

**Table 2: Generative v.s. Reconstructive. Following Sun et al. () and Dong et al. (), we adopt 576 learnable latent tokens to query the LMM and utilize the corresponding outputs as conditions to the denoiser for generative cases. Extra 102K caption data from ShareGPT4V (Chen et al., ) is introduced to the original SFT data, facilitating text-to-image creation. Reconstructive objectives boost comprehension while generative alternatives cannot.**
| Method | SFT Data | w/ $\mathcal{L}_{\mathrm{LMM}}^{\mathrm{visual}}$ | Hallucination | Fine-Grained | General |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| 737K | 102K | POPE | Hallu. | MMVP | ChartQA | MMB${}^{\text{EN}}$ |  |  |
| Baseline | 737K + Caption 102K | – | – | 86.2 | 55.1 | 32.0 | 30.9 | 74.4 |
| Reconstructive | 737K + Caption 102K | ✓ | ✓ | 87.6 | 58.0 | 38.7 | 40.4 | 75.2 |
| Reconstructive | 737K + Caption 102K | – | ✓ | 87.6 | 56.3 | 37.3 | 39.7 | 74.9 |
| Generative | 737K + Creation 102K | – | ✓ | 85.4 | 52.0 | 30.0 | 31.2 | 73.9 |

Generative v.s. Reconstructive.
We ablate the effectiveness of reconstruction over generation in Table 3.
Similar to Sun et al. () and Dong et al. (), for generative cases, we adopt 576 learnable latent tokens to query the LMM and utilize the corresponding outputs as conditions to the denoiser.
The detailed pipeline of these two methods can be found at Figure 11 in Appendix C.
However, generative methods require specific creation data and can not naively implemented on the original SFT data, i.e., Cambrian-737K (Tong et al., ).
To build creation data, we utilize GPT-4o to transfer 102K caption into text-to-image creation data from ShareGPT4V (Chen et al., ) and combine them with the original SFT data.
From Table 3, we can tell that reconstructive objectives boost comprehension while generative alternatives cannot.

Ross with Different LLMs and Visual Encoders.
To demonstrate the effectiveness of our vision-centric supervision $\mathcal{L}_{\mathrm{LMM}}^{\mathrm{visual}}$ adopted by our Ross, we conduct systematic experiments across different base LLMs and visual encoders.
From Table 3, Ross contributes to significant improvements consistently, especially on fine-grained comprehension benchmarks, i.e., MMVP (Tong et al., ) and ChartQA (Masry et al., ).

**Table 4: Comparison to state-of-the-art LMMs. A mixture of 2M caption data and 1.2M instruction tuning data are utilized for pre-training and fine-tuning, respectively. Our model outperforms them in most of the settings. We evaluate these models on benchmarks: POPE (Li et al., ): the averaged accuracy of three categories (random, popular and adversarial) on the validation set of MSCOCO (Lin et al., ), Hallu.: HallusionBench (Guan et al., ) average accuracy, MMB${}^{\text{EN}}$: MMBench (Liu et al., ) English dev split, MMB${}^{\text{CN}}$: MMBench (Liu et al., ) Chinese dev split, SEED${}^{\text{I}}$: SEED-Bench-1 (Li et al., ) with image accuracy, MMMU (Yue et al., ) validation split, MMVP (Tong et al., ), GQA (Hudson & Manning, ) test-dev-balanced split, and AI2D (Hiippala et al., ) test split. ^‡We evaluate the official checkpoint/api using VLMEvalKit (Duan et al., ) since the original paper did not report the performance.**
| Model | POPE | Hallu. | MMB${}^{\text{EN}}$ | MMB${}^{\text{CN}}$ | SEED${}^{\text{I}}$ | MMMU | MMVP | GQA | AI2D |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| GPT-4V-1106 (OpenAI, ) | 75.4 | 65.8^‡ | 75.8 | 75.1^‡ | 71.6 | 53.8 | 50.0 | 36.8 | 78.2 |
| Gemini-1.5 Pro (Team et al., ) | – | – | 73.6 | – | 70.7 | 47.9 | – | – | – |
| MM-1-8B (McKinzie et al., ) | 86.6 | – | 72.3 | – | 69.9 | 37.0 | – | 72.6 | – |
| LLaVA-v1.5-7B^‡ (Liu et al., ) | 86.2 | 47.5 | 65.5 | 58.5 | 66.0 | 34.4 | 20.0 | 62.0 | 55.4 |
| LLaVA-v1.6-7B^‡ (Liu et al., ) | 86.5 | 35.8 | 67.4 | 60.1 | 70.2 | 35.8 | 37.8 | 64.2 | 67.1 |
| Mini-Gemini-7B (Li et al., ) | – | – | 72.7 | – | 73.2 | 37.3 | 18.7 | 64.5 | 73.5 |
| DeepSeek-VL-7B (Lu et al., ) | 85.8^‡ | 44.1^‡ | 73.2 | 72.8 | 70.4 | 36.6 | – | – | 64.9^‡ |
| Cambrian-1-8B (Tong et al., ) | 87.4^‡ | 48.7^‡ | 75.9 | 68.9^‡ | 74.7 | 42.7 | 51.3 | 64.6 | 73.0 |
| Ross-7B | 89.2 | 57.3 | 79.0 | 76.1 | 73.0 | 43.4 | 54.7 | 65.5 | 76.8 |
| Base LLM: Vicuna-13B-v1.5 |  |  |  |  |  |  |  |  |  |
| LLaVA-v1.5-13B^‡ (Liu et al., ) | 82.5 | 44.9 | 68.8 | 63.6 | 68.2 | 36.6 | 31.9 | 63.3 | 60.8 |
| LLaVA-v1.6-13B^‡ (Liu et al., ) | 86.2 | 36.7 | 70.0 | 64.1 | 71.9 | 36.2 | 35.6 | 65.4 | 72.4 |
| Mini-Gemini-13B (Li et al., ) | – | – | 68.6 | – | 73.2 | 37.3 | 19.3 | 63.7 | 70.1 |
| Cambrian-1-13B (Tong et al., ) | 85.7^‡ | 54.0^‡ | 75.7 | 65.9^‡ | 74.4 | 40.0 | 41.3 | 64.3 | 73.6 |
| Ross-13B | 88.7 | 56.4 | 73.6 | 67.4 | 71.1 | 41.3 | 44.7 | 65.2 | 73.8 |

### 5.3 Comparison with State-of-the-Arts

Ross utlizes a single SigLIP-ViT-SO400M/14@384 (Zhai et al., ) as the visual encoder.
Ross-7B utilizes Qwen2-7B-Instruct (Yang et al., ) and Ross-13B adopts Vicuna-13B-v1.5 (Chiang et al., ) as the base LLM.
The implementation almost follows LLaVA-v1.5 (Liu et al., ) without the high-resolution image-slicing technique (Liu et al., ).
Thus, our primary comparison of Ross with alternative methods focuses on benchmarks that do not require exceptionally high-resolution inputs.
When comparing with other methods, we incorporate more training data.
Specifically, we use a mixture of 2M caption data for the pre-training stage, which consists of 1246K from ShareGPT4V (Chen et al., ) and 707K from ALLaVA (Chen et al., ).
The instruction tuning data is a mixture of Cambrian-737K (Tong et al., ) and SMR-473K (Zhang et al., ).

Illustrated in Table 4, we compare our Ross with both private models (OpenAI, ; Team et al., ; McKinzie et al., ) and open-sourced alternatives (Liu et al., ; ; Tong et al., ; Li et al., ; Lu et al., ).
The previous open-source state-of-the-art Cambrian-1 (Tong et al., ) leverages extrinsic assistance that aggregates CLIP (Radford et al., ), SigLIP (Zhai et al., ), DINOv2 (Oquab et al., ), and ConvNext (Liu et al., ).
On the other hand, our Ross stands for intrinsic activation.
With only a single SigLIP (Zhai et al., ) model as the visual encoder, our Ross surpasses Cambrian-1 (Tong et al., ), under most cases, without a careful choice of the visual experts and naturally maintains a lightweight inference procedure.
Ross is also data-efficient compared with Cambrian-1 (Tong et al., ), since it requires 7M instruction tuning data.
Notably, Ross-7B even surpasses GPT-4V-1106 and Gemini-1.5 Pro on several benchmarks such as POPE (Li et al., ), MMBench (Liu et al., ), and MMVP (Tong et al., ).

**Table 5: Transfer learning on SpatialBench (Cai et al., ). “RGB” indicates using only RGB images for testing, while “RGB + D” represents taking depth maps as extra inputs. The performance of GPT-4o is obtained from Cai et al. (). LMMs can better comprehend depth maps with $\mathcal{L}_{\mathrm{LMM}}^{\mathrm{visual}}$.**
| Method | Test Inputs | $\mathcal{L}_{\mathrm{LMM}}^{\mathrm{visual}}$ | MiDaS | Size | Reaching | Position | Existence | Counting | Average |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| LLaVA | RGB | – | – | 20.0 | 51.7 | 58.8 | 70.0 | 74.6 | 55.0 |
| RGB + D | – | – | 21.7 $\uparrow$ 1.7 | 45.0 $\downarrow$ 6.7 | 58.8 – 0.0 | 65.0 $\downarrow$ 5.0 | 77.7 $\uparrow$ 3.1 | 53.6 $\downarrow$ 1.4 |  |
| RGB | – | ✓ | 21.7 | 60.0 | 64.7 | 80.0 | 84.1 | 62.1 |  |
| RGB + D | – | ✓ | 21.7 – 0.0 | 51.7 $\downarrow$ 8.3 | 70.6 $\uparrow$ 5.9 | 65.0 $\downarrow$ 15.0 | 91.1 $\uparrow$ 7.0 | 60.0 $\downarrow$ 2.1 |  |
| Ross | RGB | ✓ | – | 25.0 | 53.3 | 64.7 | 70.0 | 75.3 | 57.7 |
| RGB + D | ✓ | – | 28.3 $\uparrow$ 3.3 | 65.0 $\uparrow$ 11.7 | 67.6 $\uparrow$ 2.9 | 85.0 $\uparrow$ 15.0 | 84.6 $\uparrow$ 8.7 | 66.1 $\uparrow$ 8.4 |  |
| GPT-4o | RGB | – | – | 43.3 | 51.7 | 70.6 | 85.0 | 84.5 | 67.0 |
| RGB + D | – | – | 40.0 $\downarrow$ 3.3 | 51.7 – 0.0 | 61.8 $\downarrow$ 8.8 | 90.0 $\uparrow$ 5.0 | 85.2 $\uparrow$ 0.7 | 65.7 $\downarrow$ 1.3 |  |

### 5.4 Transfer Learning on Understanding Depth Maps

We further evaluate the transfer learning capability of our Ross on SpatialBench (Cai et al., ), which requires the model to understand depth maps.
We compare our Ross with the visual instruction tuning baseline, with the same training data and model architecture.
Also, we compare the effectiveness of the extrinsic assistance solution, i.e., combining a depth expert MiDaS-3.0 (Birkl et al., ) to visual instruction tuning, with our intrinsic activation solution.
Specifically, the pre-training data is LLaVA-558K (Liu et al., ) and the fine-tuning data is SpatialQA-853K (Cai et al., ), where each conversation contains the RGB image and the depth maps extracted by ZoeDepth (Bhat et al., ).
The visual encoder is CLIP-ViT-L/14@336 (Radford et al., ) and the base LLM is Qwen2-7B-Instruct (Yang et al., ).

As demonstrated in Table 5, our Ross manages to make use of the extra depth map, as consistent and significant improvements are observed when taking “RGB + D” inputs for testing.
Extrinsic assistance approaches cannot take advantage of extra depth maps when testing.
Even GPT-4o cannot fully understand depth maps.
Figure 9 provides qualitative results.
The extra depth understanding visual expert, i.e., MiDaS (Birkl et al., ), fails to help LMMs understand depth maps both quantitatively in Table 5 and qualitatively in Figure 9.

Figure: Figure 9: Qualitative comparisons on SpatialBench (Cai et al., ). We take RGB + D inputs when testing. Note that “LLaVA” here indicates the visual instruction baseline with exactly the same model architecture and training data. Notably, the extra depth expert MiDaS-3.0 (Birkl et al., ) sometimes harms comprehension (see the second example).
Refer to caption: x6.png

## 6 Conclusion

This paper introduces reconstructive visual instruction tuning (Ross).
Different from conventional LMMs that solely utilize text supervision, Ross leverages a vision-centric reconstructive objective to supervise visual outputs.
To avoid being overwhelmed by heavily redundant raw RGB values, we train a denoiser to recover clean latent visual representations conditioning on visual outputs.
Experimentally, the proposed objective indeed brings enhanced comprehension capabilities and reduced hallucinations.
Ross outperforms the state-of-the-art under most cases with only a single SigLIP (Zhai et al., ) as the visual encoder.
In-depth analysis demonstrates the effectiveness of Ross for directing focus towards visual elements and understanding depth maps.

Discussion.
One limitation is that Ross does not have generation capabilities, since Ross is designed for enhanced multimodal comprehension, without the need to generate photorealistic aesthetic images.
Furthermore, the gap in training data between comprehension and generation methods also matters.
For instance, PixArt-$\alpha$ (Chen et al., ), which is one of the most efficient text-to-image models, was trained on nearly 400M images to model the pixel discrepancy just in the first training stage.
By contrast, our Ross is only trained on nearly 3M images for one epoch.
Future topics include achieving photorealistic text-to-image generation via incorporating more training samples.

## Acknowledgements

The work was supported by the National Science and Technology Major Project of China (No. 2023ZD0121300), the National Natural Science Foundation of China (No. U21B2042, No. 62320106010), the 2035 Innovation Program of CAS, and the InnoHK program.

## Appendix

## Appendix A Latent Diffusion Models

Given a set of clean latent tokens $\bm{z}_{0}$ drawn from $p(\bm{z})$, the forward diffusion process is a Markov chain that gradually adds random Gaussian noise to the original sample:

$$ $q(\bm{z}_{t}|\bm{z}_{t-1})=\mathcal{N}(\sqrt{1-\beta_{t}}\bm{z}_{t-1},\beta_{t }\mathbf{I}),$ (5) $$

where $\mathcal{N}(\bm{\mu},\bm{\Sigma})$ denotes the Gaussian distribution, and $t$ indicates discrete timesteps.
$\beta_{t}\in(0,1)$ indicates a pre-defined time-dependent variance schedule.
According to Ho et al. (), to admit sampling $\bm{z}_{t}$ at an arbitrary timestep $t$ directly from $\bm{z}_{0}$, this transition can be reformulated as

$$ $\displaystyle q(\bm{z}_{t}|\bm{z}_{0})=\mathcal{N}(\sqrt{\bar{\alpha}_{t}}\bm{ z}_{0},(1-\bar{\alpha}_{t})\mathbf{I}),$ (6) $\displaystyle\bm{z}_{t}=\sqrt{\bar{\alpha}_{t}}\bm{z}_{0}+\sqrt{1-\bar{\alpha} _{t}}\bm{\epsilon},\quad\bm{\epsilon}\sim\mathcal{N}(\bm{0},\mathbf{I}),$ $$

where $\alpha_{t}=1-\beta_{t}$ and $\bar{\alpha}_{t}=\prod_{i=1}^{t}\alpha_{t}$.
A latent diffusion model learns to reverse this progressive noise addition process for latent tokens.
Specifically, to iteratively generate clean tokens $\bm{z}_{0}$ from pure noise $\bm{z}_{T}$ conditioned on $\mathcal{C}$, we need to reverse the forward process by

$$ $\bm{z}_{t-1}=\frac{1}{\sqrt{\alpha_{t}}}\left(\bm{z}_{t}-\frac{1-\alpha_{t}}{ \sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\pi}(\bm{z}_{t};\mathcal{C},t)\right) +\sigma_{t}\bm{\epsilon},$ (7) $$

where a $\pi$-parameterized neural network $\bm{\epsilon}_{\pi}$ is trained to predict the added noise during the forward process.
$\sigma_{t}$ indicates the posterior noise variance.
The training objective of $\bm{\epsilon}_{\pi}$ is

$$ $\mathcal{L}(\pi,\bm{z}_{0})=\mathbb{E}_{t,\bm{\epsilon}}\left[||\bm{\epsilon}_ {\pi}(\sqrt{\bar{\alpha}_{t}}\bm{z}_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon} ;\mathcal{C},t)-\bm{\epsilon}||^{2}\right].$ (8) $$

## Appendix B Implementation Details

**Table 6: Hyperparameters of Ross. We obtain most of the configurations from Liu et al. ().**
| Config | Stage I | Stage II |
| --- | --- | --- |
| Trainable parts | projector + denoiser | projector + LLM + denoiser |
| Frozen parts | visual encoder + LLM + teacher tokenizer | visual encoder + teacher tokenizer |
| Global batch size | 256 | 128 |
| Batch size per GPU | 16 | 4 |
| Accumulated steps | 2 | 4 |
| DeepSpeed zero stage | 2 | 3 |
| Learning rate | 1$\times$10${}^{-\text{3}}$ | 2$\times$10${}^{-\text{5}}$ |
| Learning rate schedule | warmup + cosine decay |  |
| Warmup ratio | 0.03 |  |
| Weight decay | 0 |  |
| Epoch | 1 |  |
| Optimizer | AdamW |  |
| Precision | bf16 |  |

**Table 7: Details of the instruction tuning dataset provided by Tong et al. ().**
| Dataset | # Samples |
| --- | --- |
| LLaVA (Liu et al., ) | 158K |
| ShareGPT (Team, ) | 40K |
| VQAv2 (Goyal et al., ) | 83K |
| GQA (Hudson & Manning, ) | 72.1K |
| OKVQA (Marino et al., ) | 9K |
| OCRVQA (Mishra et al., ) | 80K |
| A-OKVQA (Schwenk et al., ) | 50K |
| TextVQA (Singh et al., ) | 21.9K |
| RefCOCO (Kazemzadeh et al., ) | 30K |
| VG (Krishna et al., ) | 86.4K |
| DVQA (Kafle et al., ) | 13K |
| DocVQA (Mathew et al., ) | 15K |
| ChartQA (Masry et al., ) | 28.1K |
| AI2 Diagrams (Kembhavi et al., ) | 15.5K |

**Table 9: Summary of the evaluation benchmarks. Prompts are mostly borrowed from VLMEvalKit (Duan et al., ) and lmms-eval (Li et al., ).**
| Benchmark | Response formatting prompts |
| --- | --- |
| POPE (Li et al., ) | – |
| HallusionBench (Guan et al., ) | Answer the question using a single word or phrase. |
| MMBench (Liu et al., ) | Answer with the option’s letter from the given choices directly. |
| SEED-Bench (Li et al., ) | Answer with the option’s letter from the given choices directly. |
| MMMU (Yue et al., ) | Answer with the option’s letter from the given choices directly. |
| MMVP (Tong et al., ) | Answer with the option’s letter from the given choices directly. |
| AI2D (Hiippala et al., ) | Answer with the option’s letter from the given choices directly. |
| RealWorldQA (xAI, ) | Answer with the option’s letter from the given choices directly. |
| GQA (Hudson & Manning, ) | Answer the question using a single word or phrase. |
| ChartQA (Masry et al., ) | Answer the question using a single word or phrase. |
| OCRBench (Liu et al., ) | Answer the question using a single word or phrase. |
| DocVQA (Mathew et al., ) | Answer the question using a single word or phrase. |
| InfoVQA (Biten et al., ) | Answer the question using a single word or phrase. |
| TextVQA (Singh et al., ) | Answer the question using a single word or phrase. |

Figure: Figure 10: The architecture of the denoiser and the choice of fine-grained tokenizer. The self-attention module illustrated in Figure 4b is crucial since orange bars consistently outperform others on hallucination and fine-grained comprehension benchmarks, while maintaining similar performances on the general understanding benchmark. KL-16 provided by Rombach et al. () is better than VQ-16 provided by Sun et al. (), as quantization may lead to information loss.
Refer to caption: x7.png

Hyperparameters.
The hyperparameters of Ross are provided in Table 6.
We simply borrow most configurations from LLaVA-v1.5 (Liu et al., ) without further tuning, as we find it works well with our Ross, even if we adopt SigLIP (Zhai et al., ) and Qwen2 (Yang et al., ) while the original LLaVA-v1.5 (Liu et al., ) utilized CLIP (Radford et al., ) and Vicuna-v1.5 (Chiang et al., ).
As SigLIP represents a single $384\times 384$ image with $729$ tokens and the downsampling ratio of the teacher tokenizer KL-16 (Rombach et al., ) is $16$, we set the input resolution of the teacher tokenizer as $432=\sqrt{729}\times 16$ to produce $729$ fine-grained tokens as denoising targets.

Instruction Tuning Data.
When comparing with state-of-art LMMs in Table 4, our Ross is trained on approximately 1.2M instruction tuning data, which is a mixture of Cambrian-737K (Tong et al., ) and SMR-473K (Zhang et al., ).
Details of these two instruction tuning datasets are listed in Table 8 and Table 8, respectively.
There might be some overlap but we simply concat these two datasets as it is already empirically effective.

Evaluation Prompts.
We provide a thorough examination of all evaluation benchmarks utilized in this paper in Table 9.
Notably, for MMVP (Tong et al., ), which is not officially supported by VLMEvalKit (Duan et al., ), we follow Cambrian-1 (Tong et al., ) to reformat the original question into a multiple-choice format and compute the accuracy using exact matching.

Figure: Figure 12: Qualitative comparisons on HallusionBench (Guan et al., ).
Refer to caption: x9.png

## Appendix C More Experiments

KL-16 v.s. VQ-16.
Our default tokenizer is a continuous VAE (Kingma, ) with Kullback-Leibler (KL) divergence trained by Rombach et al. ().
We further conduct experiments with a discrete tokenizer provided by Sun et al. (), which is a VQGAN (Esser et al., ), i.e., VQVAE (Oord et al., ) with additional perceptual loss (Zhang et al., ) and adversarial loss (Goodfellow et al., ).
KL-16 outperforms VQ-16.
One intuitive explanation is that KL-16 preserves more low-level details than VQ-16 since quantization may lead to information loss.
Moreover, quantitatively, on ImageNet (Deng et al., ) 256$\times$256 validation set, KL-16 achieves 0.87 rFID (Heusel et al., ) while the rFID (Heusel et al., ) of VQ-16 is 2.19.

Architecture of the Denoiser.
Illustrated in Figure 11, the self-attention module is crucial, as original visual outputs $\bm{x}_{i\leq N}$ are actually causal and we need to model inter-token discrepancy via self-attention.
The number of trainable parameters is not the crux.

Generative v.s. Reconstructive.
We offer a detailed pipeline comparison in Figure 11.
Experimental results have already been provided in Table 3.
The implementation of generative methods is similar to Sun et al. () and Dong et al. (), where we adopt 576 learnable queries as inputs and take the corresponding outputs as conditions for the denoiser.

Figure: Figure 13: Qualitative comparisons on MMbench (Guan et al., ) English dev split.
Refer to caption: x10.png

Figure: Figure 14: Qualitative comparisons on MMVP (Tong et al., ).
Refer to caption: x11.png

Qualitative Comparisons.
We provide sufficient qualitative comparisons in Figure 12, Figure 13, Figure 14, and Figure 9 on HallusionBench (Guan et al., ), MMBench (Liu et al., ) English dev split, MMVP (Tong et al., ), and SpatialBench (Cai et al., ), respectively.
In Figure 12, Figure 13, and Figure 14, we compare our Ross-7B with the instruction tuning baseline LLaVA-v1.5-7B (Liu et al., ), the state-of-the-art open-source method using extrinsic assistance Cambrian-1-8B (Tong et al., ), and GPT-4V (OpenAI, ).

As demonstrated in Figure 12, where we highlight the wrong parts of each prediction in red, our Ross manages to correctly answer the question with reduced hallucinations even when GPT-4V fails.
Cambrian-1 (Tong et al., ) even fails to follow the instructions in the second example.
This could be because a super huge SFT data (7M) may harm the instruction-following abilities of LMMs.
Qualitative results shown in Figure 13 demonstrate both enhanced reasoning abilities (the first example), low-level comprehension capabilities (the second example), and spatial understanding skills (the third example).
Figure 14 illustrates that our Ross is good at recognizing various visual patterns, implying that the introduced reconstructive vision-centric objective indeed makes up the visual shortcomings of the original visual encoder.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: It would be cool if authors could address the concerns in weakness.
soundness: 3
strengths: **Novel Image-Based Supervision**: ROSS leverages image reconstruction as a supervisory signal, enabling the model to capture fine-grained visual features and semantics that significantly reduce hallucination artifacts compared to text-supervised approaches. The idea conceptually makes sense and is proven in the experiments.

**Comprehensive Analysis of Model Variants**: The paper provides a thorough study of various architectural choices and configurations within the ROSS framework, offering insight into optimal setups.

**Empirical Validation**: Extensive ablation studies and benchmark evaluations demonstrate ROSS's superior performance metrics, particularly in tasks requiring high-fidelity visual understanding, with statistically significant improvements over state-of-the-art baselines.
summary: The paper proposes a new LMM training approach with an additional branch for input image reconstruction. The results show the model enhances fine-grained comprehension and reduces hallucinations. ROSS employs a denoising objective to address spatial redundancy that reconstructs latent visual tokens rather than raw RGB values. Empirical evaluations demonstrate that ROSS consistently outperforms conventional LMMs using single or multiple visual encoders on visual understanding benchmarks.
weaknesses: - Potential Unfairness in Comparisons: While the paper includes an ablation study where variables like training data are controlled for fair comparison with other models, its main results table appears to use different datasets compared to competing methods. This inconsistency in data setup might lead to an unfair advantage for ROSS, making it difficult to assess the true comparative effectiveness of the approach against state-of-the-art methods.

- Computational Overhead: The denoising process introduces extra computational overhead during training, but the paper does not quantify or discuss this cost, leaving readers uncertain about the practical trade-offs of using this approach.

- Limited Analysis of Generation vs. Reconstruction Performance: The paper compares ROSS’s reconstructive approach to generative methods, noting that the generative approach underperforms in comprehension tasks. However, it lacks a thorough exploration of why the generative method yields lower performance. A more in-depth discussion of the limitations and differences between the two approaches would enhance understanding and help identify when reconstruction might be preferable to generation in multimodal tasks.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: How will the results change at higher resolution and what would the comparison look like when compared against tile-based methods like LLava-1.6 or any of the more recent VLM work like Qwen2-VL etc ...?
soundness: 3
strengths: Whereas text-based LLMs have achieved amazing results only with next-token prediction, when we have image + text VLMs, it has always seemed that only doing next-token prediction for text could be improved upon.  In that regard, the technique proposed in this paper, to use image denoising as a pretext task, seems like step forward, as a way to add more supervision to the VLM and to improve results. 

The benefits to the metrics are actually significant in some cases, not just epsilon levels, which is great to see.

It seems to me like the method is described clearly and the results are presented clearly.
summary: The paper uses image denoising as an auxiliary training task to improve VLMs abilities.
Denoising encourages the VLM to preserve image detail.
The work is motivated by the MAE (masked auto-encoder) line of work for training foundational vision encoders.
The auxiliary task helps VLMs achieve higher benchmark numbers.
weaknesses: 1. I wish the benchmarks cited in the paper to measure the benefits of their method, i wish those benchmarks more closely matched recent popular work such as "The Llama 3 Herd of Models" or "Qwen2-VL", which include benchmarks like TextVQA, DocVQA, etc ... It may not change the conclusion but when we compare methods, it's important to look at a representative distribution of benchmarks. Table 4 has some of these common benchmarks, but not all of them. Furthermore, i wish Table 4 (or perhaps Table 3) included the same benchmarks but also had a very clean A/B experiment that was a baseline without the method vs. using the method. Table 3 has this but Table 3 has a different set of benchmarks! So it's rather confusing what conclusion to draw.

2. The fact that the work was done at lower image resolution also limits the impact of the work. While it may be a perfectly reasonably thing to study the problem initially at lower resolution, I believe that most people care about the models with the best metrics, and all of those methods today use image tiling to handle high resolution images.

---

confidence: 4
contribution: 4
initial_rating: 6
presentation: 3
questions: This not be a major issue, but it's not very natural to see that using vision encoder to encode image pixels into LLM's embeddings and use another denosing module to reconstruct it back to image pixels. 

This may introduce improvements since the model better maintains the information of original images, but it may be more natural to see it's used in a quantized tokenizer based LMM like EMU-3 and Chamelon.

I was wondering how the authors feel about this and have insights on this question?
soundness: 4
strengths: This paper is very novel and address the very important topic on vision-centric learning in LMM.

Specifically, the paper introduces an innovative vision-centric supervision method that leverages the inherent richness of input images, addressing a clear gap in existing LMM training approaches. The use of denoising objectives for latent representation reconstruction is particularly clever as it handles the spatial redundancy problem.

The authors conduct extensive experiments across multiple benchmarks, including thorough ablation studies that systematically evaluate different components of their approach (regression vs. denoising, different tokenizers, architecture choices). The comparison with state-of-the-art methods is particularly thorough.

ROSS achieves competitive or superior performance using only a single visual encoder, making it more efficient than existing approaches that require multiple visual experts. This has significant practical implications for deployment and scalability.

The methodology is well-grounded in existing literature and builds thoughtfully on previous work in both vision and language domains. The authors clearly explain how their approach differs from both traditional visual instruction tuning and newer aggregated visual instruction tuning methods.
summary: This paper introduces ROSS, a novel approach to enhance Large Multimodal Models (LMMs) through vision-centric supervision signals. Unlike conventional visual instruction tuning that only supervises text outputs, ROSS introduces a reconstructive objective where LMMs must reconstruct input images' latent representations. 

The authors address the challenge of spatial redundancy in visual signals by employing a denoising objective to reconstruct latent representations rather than raw RGB values. The approach demonstrates significant improvements across various benchmarks, particularly in fine-grained visual comprehension and hallucination reduction, while maintaining a lightweight architecture compared to existing methods that rely on multiple visual experts.
weaknesses: I think the major weakness is about the Computational Costs. While the paper emphasizes the efficiency of using a single visual encoder, it lacks detailed analysis of training time, memory requirements, and computational costs compared to baseline methods.

Besides, the paper doesn't thoroughly discuss the sensitivity of ROSS to various hyperparameters, such as the denoising schedule or architecture choices. It would be benefitical to add this part analysis and show ROSS's denoising part is robust and easy to train.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. The results in Table 4 appear counter-intuitive, as ROSS-13B performs consistently worse than ROSS-7B. This raises concerns about whether the proposed method is well-suited to larger-scale models. Clarification on this disparity and potential scalability issues would strengthen the paper.
2. The analysis in Section 5.2 is unclear. My understanding is that the authors aim to demonstrate that additional visual supervision enables the model to better focus on relevant areas of the image during VQA tasks. However, the reasoning behind this effect is not well-explained. Further elaboration on the mechanisms or evidence supporting this claim would enhance interpretability.
soundness: 3
strengths: 1. The paper is well-organized and easy to follow, with a clear presentation of the key ideas. It is always inspiring to see that a simple auxiliary loss can improve the performance of large multimodal models.
2. The authors have put a lot of effort into designing experiments that thoroughly ablate the proposed training methods. This rigor in experimentation is highly appreciated and adds to the paper’s credibility.
summary: This paper introduces Reconstructive Visual Instruction Tuning (ROSS), a novel approach that leverages input images as additional supervision signals to enhance fine-grained visual perception capabilities. Through extensive empirical studies, the authors explore optimal training settings, such as auxiliary module design and the nature of supervision signals. Experimental results indicate that ROSS consistently improves the performance of existing vision-language models.
weaknesses: 1. The proposed method lacks novelty and insight. As noted in Section 2, previous work, such as [1], has explored using images as additional supervision in the form of generative objectives. Consequently, the contribution of this paper is limited, focusing mainly on a variation of established methods.
2. The proposed method lack interpretability. While using pixel-level reconstruction as an auxiliary task may enhance fine-grained image recognition, this approach risks impairing the language capabilities of the multimodal model, as the final task output is high-level semantic language. The authors provide insufficient experiments and explanations regarding this trade-off, leaving questions about potential impacts on language performance.
3. The scalability of the empirical findings is uncertain. It remains unclear whether the optimal settings identified would hold in different scenarios or with variations in model scale, training data, and other factors. Although the authors attempt to address this concern with results in Table 3, these efforts are insufficient, as many relevant variables remain unexplored.

[1] Sun Q, Cui Y, Zhang X, et al. Generative multimodal models are in-context learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14398-14409.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. The novelty is somewhat incremental. Utilizing image supervision is a very straightforward idea, and can be implemented by various ways. Denoising-based reconstruction is a very well-developed strategy in the field of image diffusion/generation. There is no specific in-depth design in the proposed architecture.

2. The authors claim that the proposed method capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. In which cases, the information within the image is important? The authors should provide a more detailed analysis of this aspect. Is this information crucial for all common cases?

3. No experiments on complexity and efficiency. Of course, utilizing more self-supervision loss can definitely improve the model's performance. However, this image-aware training may cost more time and GPU memories compared to previous text-only supervision. The authors should provide an in-depth analysis of complexity and efficiency for fair comparison.

4. Given an image-text pair input, just some of the image-based contents are aligned with the text. I do not see any text-guided image reconstruction design in the architecture. This may help reduce the redundancy contexts.
soundness: 2
strengths: 1. The paper is well-organized and easy to read.
2. The motivation is straightforward.
summary: This paper proposes a new variant of visual instruction tuning. Different from previous works that only utilize textual supervision, the proposed method additionally exploits visual supervision by reconstructing the contexts of input images. In particular, a denoising structure is introduced to better learn the latent representations. Experiments are conducted on several benchmarks.
weaknesses: 1. The novelty is somewhat incremental. Utilizing image supervision is a very straightforward idea, and can be implemented by various ways. Denoising-based reconstruction is a very well-developed strategy in the field of image diffusion/generation. There is no specific in-depth design in the proposed architecture.

2. The authors claim that the proposed method capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. In which cases, the information within the image is important? The authors should provide a more detailed analysis of this aspect. Is this information crucial for all common cases?

3. No experiments on complexity and efficiency. Of course, utilizing more self-supervision loss can definitely improve the model's performance. However, this image-aware training may cost more time and GPU memories compared to previous text-only supervision. The authors should provide an in-depth analysis of complexity and efficiency for fair comparison.

4. Given an image-text pair input, just some of the image-based contents are aligned with the text. I do not see any text-guided image reconstruction design in the architecture. This may help reduce the redundancy contexts.

Meta-review: This paper introduces a new approach to visual instruction tuning by incorporating an additional reconstruction objective. The proposed method is interesting and demonstrates clear improvements. Reviewers provided overall positive feedback, and the authors submitted strong rebuttals addressing their comments. Taking all reviewers' feedback into consideration, the AC has recommended the paper for acceptance.

Decision: Accept (Poster)