Reconstructive Visual Instruction Tuning
Authors: Haochen Wang
Anlin Zheng
Yucheng Zhao
Tiancai Wang
Zheng Ge
Xiangyu Zhang
Zhaoxiang Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Large Multimodal Models
Multimodal Comprehension
Primary area: foundation or frontier models, including LLMs
TL;DR: We design a vision-centric reconstructive supervision that boosts multimodal comprehension capabilities.

Abstract: This paper introduces reconstructive visual instruction tuning (ROSS), a family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, ROSS prompts LMMs to supervise visual outputs via reconstructing input images. By doing so, it capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, ROSS employs a denoising objective to reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs to maintain image detail, thereby enhancing their fine-grained comprehension capabilities and reducing hallucinations. Empirically, ROSS consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance state-of-the-art alternatives that aggregate multiple visual experts, ROSS delivers competitive performance with a single SigLIP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs. The code will be made publicly available upon acceptance.

Published as a conference paper at ICLR 2025
RECONSTRUCTIVE VISUAL INSTRUCTION TUNING
Haochen Wang Anlin Zheng
Yucheng Zhao Zhaoxiang Zhang
Tiancai Wang**
Zheng Ge Xiangyu Zhang
1 New Laboratory of Pattern Recognition, State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences
2 University of Chinese Academy of Sciences
3 University of Hong Kong MEGVII Technology
StepFun
wwanhhacchn2022,, thaoxiang aaaeiiaggg.zaz.gg zhang) aia.ac.cr.
wangriaaceaiemegrii.oo
Project Page: https: hectps:/hhacche-eeeee http:/hhhhhhhhanaaaae -ig.gengh www.ggittthhiiogoo ic/ross
ABSTRACT
This paper introduces reconstructive visual instruction tuning (Ross), family of Large Multimodal Models (LMMs) that exploit vision-centric supervision signals. In contrast to conventional visual instruction tuning approaches that exclusively supervise text outputs, Ross prompts LMMs to supervise visual outputs via recon- structing input images. By doing it capitalizes on the inherent richness and detail present within imput images themselves, which are often lost in pure text supervi- sion. However, producing meaningful feedback from natural images is challenging due to the heavy spatial redundancy of visual signals. To address this issue, Ross employs denoising objective reconstruct latent representations of input images, avoiding directly regressing exact raw RGB values. This intrinsic activation design inherently encourages LMMs maintain image detail, thereby enhancing their fine-graines comprehension capabilities and reducing hallucinations. Empirically, Ross consistently brings significant improvements across different visual encoders and language models. In comparison with extrinsic assistance bate-offthe--as alternatives that aggregate multiple visual experts, Ross delivers competitive per- formance with single SigL.PP visual encoder, demonstrating the efficacy of our vision-centric supervision tailored for visual outputs.
INTRODUCTION
The success of GPT-style Large Language Models (LLMs) (Radford al., 2018; 2019; Brown et al., 2020; OpenAL, 2023b; Yang et al., 2024a; Touvron et al., 2023; Chiang et al., 2023; Dubey et al., 2024) has motivated researchers to adapt LLMs to understand multimodal inputs (Liu et al., 2023a; 2024a; Dai et al., 2023; Bai et al., 2023). Notably, visual instruction tuning approaches (Liu et al., 2023a) demonstrate superior performance with cost-efficient training recipes. Some approaches (Chen et al., 2024b; Li et al., 2024c) even surpass GPT-4V(ision) (OpenAI, 2023a) on benchmark evaluations.
Typically, these Large Multimodal Models (LMMs) based on visual instruction tuning adopt a plug-in architecture, depicted in Figure Ia, where pre-trained vision-language foundation models such as CLIP (Radford et al., 2021) are responsible for projecting images into visual tokens. They serve as prefix tokens for multimodal comprehension. However, this type of design, i.e., visual encoder connector -> L.M = language instructions, where "=" indicates supervision, is primarily LLM-- centric: (i) visual comprehension largely depends vision-to-texx alignment and the selected vision models, and (ii) supervision derives exclusively from text data. As a result, they exhibit systematic visual shortcomings such as recognizing specific visual patterns (Tong et al., 2024b).
Until very recently, some concurrent works proposed vision-centric solutions (Tong et al., 2024a;b). Illustrated in Figure 1b, their solutions leverage extrinsie assistance via aggregating several different visual experts. Inspired by the evolution in image recognition, from manually designed visual
"Corresooniigg authors. Project lead.

---

Published as a conference paper at ICLR 2025
(b) Aggregated Visual Intruction Tuning with Extrinsic Assistance
(c) Reconstructive Visual Instruction Tuning with Intrinsic Activation
(a) Visual Instruction Tuning (e.g,, LLa)A)
Input
Image
Text
Image
Text
Image
Text
Visual Encoder
Single Encoder
Multiple Encoder
Single Encoder
Language Model
LLM
LLM
LLM
Supervision
Text
Text
Image
Figure 1: Conceptual comparison between different pipelines. (a) Typical visual instruction tuning approaches (Liu et al., 2023a; 2024a) follow a LLM-centric design that solely leverage text supervision. (b) Aggregated visual instruction tuming alternatives (Tong et al., 2024a;b) leverages extrinsic assistance via combining several visual experts, requiring a careful selection of visual experts. (c) Our Ross, with single visual encoder, e.g., CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023), designs extra vision-centric reconstructive supervision as intrinsic activation. In this way, LMMs required preserve every detail of input images, thereby enhancing multimodal comprehension capabilities and reducing hallucinations.
features (Sánchez & Perronnin, 2011) to learnable deep convolutional models (Krizhevsky al., 2012), suggest that intrinsic activation offers more viable path forward. Just deep models automatically learn hierarchical and abstract features from raw data, we believe intrinsic activation methods similarly more adaptable for multimodal comprehension, reducing reliance on hand- crafted engineering, thereby enhancing both generalization and performance. Therefore, we aim to explore intrinsic activation solutions based on the following principles:
1. Supervise Visual Outputs. Current LMMs solely supervise text outputs, neglecting signif- icant amount of visual outputs unased. For instance, I.L.VA-vI.5 (Liu et al., 2024a) utilizes 576 visual tokens to represent single 336 x 336 image, yet their corresponding outputs remain unsupervised. Intuitively, since input images themselves inherently provide rich and detailed information, those visual outputs. This approach encourages LMMs to maintain low-level details, thereby enhancing their fine-grained comprehension abilities and reducing hallucinations.
regard LMMs reconstructing input images the supervision of
2. Explore the Optimal Formulation. Designing this self supervised task effectively is not straightforward. Motivated by the success of masked autoencoder (He et al., 2022) compared its basic version denoising autoencoder (Vincent al., 2008), we identify handling heavy spatial redundancy of visual signals as the underlying key factor. To this end, we formulate our approach as follows: (i) for reconstruction targets, instead of raw RGB pixels, we make LMMs reconstruct latent visual tokens, and (ii) for reconstruction objectives, to avoid directly regressing exact token values, we adopt per-token denoising.
To this end, we propose Ross, termed of reconstructive visual instruction tuning, which utilizes input images as direct supervision signals illustrated in Figure Ic. Technically, to address the spatial redundancy inherent in natural visual signals (He al., 2022), which takes high-level visual outputs as conditions to recover low-level fine ine-grained visual tokens 2, representing an underlying distribution p(2 (z) These latent tokens are derived from frozen teacher tokenizer such as continuous VAE (Kingma, 2013) and discrete VQGAN (Esser et al., 2021). Unlike extrinsic assistance solutions (Tong et al., 2024a;b), our intrinsic activation solution naturally maintains a lightweight inference procedure. More importantly, when adapting our solution avoids a careful choice new domain-specifie experts, e.g., MiDaS-3.0 (Birkl et al., 2023) for understanding depth maps, which is more efficient and easier to implement.
train small denoising network,
visual domains,
Empirically, Ross achieves top performance across a wide range of multimodal comprehension benchmarks. Notably, our RoSS excels in fine-grained vision-centrie benchmarks (Tong al., 2024b; Masry et al., 2022) and hallucination benchmarks (Guan et al., 2024; et al., 2023c). To be specific, with single LIP (Zhai et al., 2023) the visual encoder, Ross-7B achieves 57.3
2

---

Published as a conference paper at ICLR 2025
HallusionBench (Guan et al., 2024) and 54.7 on MMVP (Tong et al., 2024b), significantly outperforms state-of-the the-art alternatives with similar model sizes which aggregate several visual experts as extrinsic assistance, e.g., Cambrian- 1-BB (Tong et al., 2024a). In In-depth analysis demonstrates the effectiveness of Ross for directing focus towards visual elements and understanding depth maps. We hope our research will inspire future work in designing supervision signals for large multimodal models.
2 RELATED WORK
Visual Instruction Tuning. Most visual instruction tuning-based LMMs adopt plug-in architec- ture (Liu et al., 2023a; 2024a; Bai et al., 2023; Wang et al., 2024c), where a language-supervised visual encoder (Radford al., 2021; Zhai et al., 2023) is responsible for extracting visual tokens. A connector is used to map those visual representations into the LLM space, e.g., Resamplers (A (Alayrac et al., 2022), Q-Formers (Li et al., 2023b; Dai et al., 2023; Bai et al., 2023; Ge et al., 2024a), and MLPs (Liu et al., 2023a; 2024a; Li et al., 2024c; Liu et al., 2024b; Li et al., 2024a). These LMMs usually follow a two-stage training recipe. During the alignment stage, the connector is trained on high-quality caption data. Next, the full model is trained tuning data. However, only text outputs are supervised. Ross, on the other hand, introduces novel vision-centric supervision via reconstructing fine-grained visual tokens conditioned on visual outputs.
single-image visual instruction
Visual Encoders for LMMs. As the original CLIP (Radford et al., 2021) adopted by conventional visual instruction tuning approaches is trained on noisy image-text pairs, it exhibits specific visual shortcomings, and thus stronger backbones (Fang et al., 2024; Zhai et al., 2023; Chen et al., 2024c) have been introduced LMMs. Some concurrent works (Tong et al., 2024b;a) leverage extrinsic assistance, which further utilizes vision-oni -oll self-supervised models (Oquab et al., 2023; Wang al., 2023a;bre; He al., 2022; Caron et al., 2021) and domain experts (Kirillov et al., 2023; Birkl et al., 2023; Shen et al., 2023; Gu et al., 2024; Zhang et al., 2024a; Wang et al., 2022; 2024a;b). Ross, from a new intrinsic activation perspective, aims to catalyze enhanced comprehension through reconstructing input images with no extra visual experts.
Generative Objectives for LMMs. Another line of work introduces pre-trained text-to-image diffusion models (Rombach et al., 2022) to make LMMs capable of both comprehension and gen- eration (Dong et al., 2024; Ge et al., 2024a; Sun et al., 2024b; Ge et al., 2024b; Sun et al., 2023; Ren et al., 2025). Our Ross, with a totally different motivation, targets to catalyze multimodal comprehension via reconstruction. Specifically, conditions are diffcrent, where Dong et al. (2024) and Sun et al. (2024b) take outputs corresponding to learnable queries conditions, while Ross takes outputs corresponding to visual inputs. Those methods are generative while Ross is reconstructive. The detailed pipeline comparison can be found in Appendix C.
3 PRELIMINARIES
Large Multimodal Models. In the literature (Radford et al., 2018; 2019), a 0-parameterized LLM models the canonical causal distribution of each text token æ; as po( = I Po (ax; (a:xs)) where {ax) represents a sequence of text tokens. To make LLMs understand visual contents, typical plug-in style LMMs (Liu et al., 2023a; 2024a) regard sequence of visual tokens as prefix tokens. Specifically, an input image I E # x x3 is first projected into a sequence of visual tokens by a É-parameterized visual encoder Gg such CLIP (Radford et al., 2021) and SigLIP (Zhai et al., 2023), where (H, W) indicates the spatial resolution. Then, a o-parameterized multimodal projector H.g is utilized project these visual tokens into the feature space of LLMss. As result, the canonical causal distribution in a multimodal sentence containing an image I becomes
(σ II po (m v), v = Hs o (T(T),
where = E, )) is the parameters and v E NxD indicates the projected visual tokens. N is the number of visual tokens and D indicates the feature channel. The visual encoder G3 could be either frozen (Liu et al., 2023a; 2024a; Tong et al., 2024a) or fine-tuned (Liu et al., 2024b; Bai et al., 2023; Li et al., 2024c; Wang et al., 2024e).
Training Recipes for LMMs. LMMs almost follow a two-stage training recipe (Liu et al., 2023a), Le., the pre-training stage (or the alignment stage) and the supervised fine-tuning stage (or the
3

---

Published as a conference paper at ICLR 2025
instruction tuning stage). The instruction (supervision) comes from languages such as the answers to VQA tasks, maximizing the log ligeliicl text outputs:
L.MM (6 (0,
-1
Σ N+1
log log
where N represents the number of visual tokens and visual outputs (one input token corresponds to one visual output). From Equation (2), we can tell that only text outputs Ni> are supervised.
4 Ross: RECONSTRUCTIVE VISUAL INSTRUCTION TUNING
In this section, we first provide an overview our reconstructive visual instruction tuning (Ross). Then, discuss explorations towards the optimal formulation in the following subsections, with the ultimate goal handling spatial redundancy of visual signals to provide meaningful visual supervision. Our explorations mainly include reconstruction targets and the training objective.
Overview. Illustrated in Figure 2, the over- all philosophy of our Ross is to construct reconstructive visual supervision signals on visual outputs NigN- The training ob- jective includes (i) the original next-token prediction on Tty N shown in the right part of Figure 2, and (ii) another reconstruc- tive term in the left part of Figure 2, i.e., Lvisnal Specifically, this visual term could be any custom mea- surements M between DiSN and specific reconstruction targets of image I:
Reconstructive Visual Supervision L
Text Supervision LIM
Large Language Model
LRsss Ltext
Visual Encoder Projector
Text Tokenizer
What type of aircraft shown the image?
USER:
ASSISTANT: The image shows
LLMM (O (0,5,0, v),
large passenger jet belonging
(3)
= ([T((M)gN)
Text (Prompt and Instructions)
where Jr indicates the n-parameterized post projection that maps the dimensions of visual tokens xı<N to be consistent with the teacher tokenizer F.
Figure 2: Overview Ross. Given an input image and the corresponding text to this image, ROSS aims to supervise visual outputs by reconstruction.
Variants of Ross. Evidently, different choices of F and M contribute to different variants. F controls the reconstruction targer while M defines the objective:
1. Towards the target, F be the pachify operation (Dosovitskiy al., 2021). resulting in pixel-level reconstruction, or pre-trained fine-grained visual tokenizers such as VAE (Kingma, 2013) and VQGAN (Esser al., 2021), leading latent-level reconstruction. F could even be vision-only models such DINOv2 (Oquab et al., 2023), making LMMs learn specific visual patterns from F, which also a type of latent-level reconstruction.
2. Towards the objective, the most straightforward choice of M is MSE or cosine similarity for regressing raw pixel values latent features, respectively. We also explore the denoising objective (Ho et al., 2020) to avoid being overwhelmed by fitting exact values.
We introduce our explorations step by step the following sections. The ultimate goal of our exploration to design an appropriate self-supervised reconstructive pre-text task that provides meaningful vision-centric supervision signals to LMMs, where handling the spatial redundancy of visual signals (He al., 2022) becomes the crux.
4.1 R REGRESSING AS RECONSTRUCTIVE VISUAL INSTRUCTION
In this section, introduce straightforward variants, i.e., regressing reconstructive visual in- struction. As shown in Figure 3, depending on the choice of F, mainly has three variants: (a) Ross&-Pixel, (b) Loset, and (c)
4

---

Published as a conference paper at ICLR 2025
Supervise M.S, 8)
Supervise MM3,83)
Supervise M TT)
Predicted Image i
Patchified lokens
Predicted Tokens
fine-Grained Tokens
Predicted Tokens å
Detokentzer
Patchify F
Projector
Tokenizer
Projector J.
Projector J.
Visual Outputs N
Image 2 Mawws
Visual Outputs sigh
Image MM
Visual Dutputs Sigh
(a) RoSs
Raseet
(c)
Rasen-Laten
Figure 3: Variants of Ross*, where regression objectives are either computed on raw RGB values in
(a) and (c), or specific latent space determined by JF in (b). We adopt MSE as M for pixel regression in (a) and (c), and cosine-sinee simil--imilarity for latent regression in (b), respectively.
Directly Regressing Raw RGB Values. The most straightforward variant is to directly regress raw RGB values illustrated in Figure 3a, called "rosss-Pixel"" Under such a setting, F is the patchify operation (Dosovitskiy et al., 2021), reshaping the image I E IR Hg-xxx33 into a sequence of flattened 2D patches I E (3P=)) where (P, P) is the resolution of each image patch and N HW/ p2 indicates the resulting number of patches. Jx can be simple MI.P, mapping the dimension of visual outputs Ei<N from D 3P2. The measurement M MSE. However, as visual signals suffer from heavy spatial redundancy (He et al., 2022), such a design may not provide meaningful supervision to LMMs. An intuitive alternative to avoid directly regressing raw RGB values while still reconstructing the image is to urge LMMs to reconstruct latent tokens, introduced as follows.
Regressing Latent Tokens. Illustrated in Figure 3b, Ross -Ltenn aims to regress fine-grained latent tokens extracted by the teacher tokenizer. F can be models trained with discriminative tasks such as DINOv2 (Oquab et al., 2023) and DEIT-III (Touvron et al., 2022). The encoder part of models trained with reconstruction tasks such as VQGAN (Esser et al., 2021) and VAE (Kingma, 2013) are also capable. .M here is the consine-similarin is able to remap latent tokens into the pixel space. Therefore, supervising in the pixel space via decoding becomes another valid variant introduced as follows.
Intuitively, the decoder part of the latter
Regressing RGB Values via Decoding. Shown in Figure 3c, Ross Mosss.Latent2Pixde requires decoder project predicted latent tokens & into the RGB pixel space, resulting in predicted image f. Let F-1 be the decoder part of VQGAN (Esser et al., 2021) or VAE (Kingma, 2013), and the regressive MSE objective M is performed on pixel-space. Note that we simply use F represent the decoding process, which is actually not the inverse function of F mathematically.
Discussion. Recall that need find the optimal solution address the spatial redundancy of natural visual signals, the target-level exploration above achieves this goal partially, as the objective limited to vanilla regression. To this end, inspired by Ho al. (2020) and Li al. (2024c), we further incorporate novel denoising objective in the following section.
4.2 Ro DENOISING AS RECONSTRUCTIVE VISUAL INSTRUCTION
As an objective for handling heavy spatial redundancy to provide meaningful vision-centro super- vision signals, denoising is better than vanilla regressing, since the introduction of noise into the training data acts as an implicit form of data augmentation and regularization. The denoising process encourages the model to focus on the underlying data manifold rather than memorizing specific instance values (Chen al., 2023c; Song & Ermon, 2019; Karras et al., 2022; Yang et al., 2024b).
Techinically, as illustrated in Figure 4a, our final Ross takes high-level visual outputs æisNN as conditions recover clean fine-grained tokens 2o from noisy tokens zt. Specifically, clean tokens žu F(I) are obtained from the teacher tokenizer F. By default, we utilize a continuous VAE (Kingma, 2013) regularized by Bellbackk-eibble (KL) divergence provided by Rombach et al. (2022), since it is believed to capture sufficient image details. The training procedure of the denoiser Jn follows diffusion process (Ho al., 2020):
ovisnal ( { Φ, m. x. I) = Er,, [[]T, (isN =|| LLMM
(4)
The denoiser Js actually estimates the conditional expectation Ele ~ No, I)|x.. More details about the background knowledge of diffusion models can be found in Appendix A.
5

---

Published as a conference paper at ICLR 2025
Reconstructive Visual Instruction Tuning
XN
Denolsing
MIPP
Predicted Noise 4
Text instructions
Norm
Supervise
Cond. Visual Outputs #
Denoiser Ve
Text Outputs diis
Notay 2>
Large Language Model
A
Self foff-tteention
Fine Grained #g
R
Norm
Tokenizer
Projector 7i,
Text Tokenizer
Uieer
Linear
Linear
Image & PooWa
Visual Encoder Gt
Text
(a) The framework of Ross
(b) The denoiscr.
Figure 4: Illustration of (a) the training procedure of Ross and (b) the detailed architecture of the denoiser Jx. (a) Ross introduces visual guidance via denoising fine-grained visual tokens zo conditioning on visual output N, (b) The denoiser takes noisy tokens 2t, current timesteps t, and conditions Ni as inputs and outputs the predicted noise & Each denoiser block consists of three linear projection layers and a standard self-attention block (Vaswani et al., 2017).
Architecture of the Denoiser. As conditions I are causal, we introduce self-attention module to model the inter-token dependencies illustrated in Figure 4b. Specifically, the architecture of the denoiser Jn is stack of Transformer Encoder blocks (Vaswani et al., 2017) and each block contains three extra projections for conditions NiSN, inputs žt. and timesteps t, respectively.
Choices of the Teacher Tokenizer. By default, we adopt latent denoising and we take a continuous tokenizer provided by Rombach et al. (2022) F. since it manages reconstruct input images with a low rFID (Heusel et al., 2017) and thus is expected to preserve many low-level details of input images. This extra reconstructive objective, however, is not limited to any certain tokenizer F. Discrete tokenizers such as VQGAN (Esser et al., 2021), and vision self-supervised models such as DINOv2 (Oquab et al., 2023), are also qualified be the tokenizer. Even the patchify operation (Dosovitskiy et al., 2021) is capable, resulting in pixel denoising.
5 EXPERIMENTS
5.1 ABLATION STUDY
Implementation Details. All ablation studies are implemented based on LLaVA-v1.5 (Liu et al., 2024a). The visual encoder G5 is CLLP-VIT-L114@333 (Radford et al., 2021) and the base LLM is Qwen2-7B IIIstructt (Yang et al., 2024a). The training data is LLaVA-558K (Liu et al., 2023a) and Cambrian-7377 (Tong et al., 2024a) for the pre-training stage and the instruction tuning stage, respectively. We evaluate each variant of Ross mainly (i) hallucination: POPE (Li al., 2023c) and HallusionBench (Guan et al., 2024), (ii) fine-grained comprehension: MMVP (Tong al., 2024b) and ChartQA (Masry et al., 2022), and (iii) general comprehension: MMBench (Liu et al., 2023b) English dev split. All evaluations are conducted with VLMEvalKit (Duan et al., 2024). Evaluation prompts can be found in Appendix B.
Pixel Regression v.s. Latent Regression. Starting from the visual instruction tuning baseline (Liu al., 2023a: 2024a), we first explore the effectiveness of using regression as the objective for our reconstructive visual instruction tuning. We utilize continuous VAE (Kingma, 2013) with an encoder-decoder architecture provided by Rombach et al. (2022), where the encoder part serves as F for Ross -Ltent while the decoder part is F-1 for - As illustrated in Figure 5, our vision-centric regression supervision outperforms the visual instruction tuning baseline in most cases. Moreover, latent regression performs the best since regressing raw RGB pixels fails to provide meaningful supervision signals, regardless of whether utilizing a decoder or not.
Choices of F. We study the effectiveness across different latent teacher tokenizers F in Figure including KL-16 provided by Rombach et al. (2022), which is a continuous VAE (Kingma, 2013) with Kullback-Leibler (KI.) divergence, self-supervised DINOv2 (Oquab al., 2023), fully-supervised

---

Published as a conference paper at ICLR 2025
363
.944
342
Thatin
Riceet-Pooo
26.2
ROsS Lakent
Ross LakentiPock
HallusoonBacch
MMPP
ChanQA
MMBanch
Figure 5: Pixel Regression Latent Regression. The teacher tokenizer F for Ross*-Latent is the encoder of a continuous VAE (Kingma, 2013) provided by Rombach et al. (2022), while its decoder F=1 for Ross Lassn-LlttntPPixcl.. Our vision-centric reconstructive supervision surpasses the visual instruction tuning baseline in most cases, Among three regression variants, Ross*-Lateni performs the best, as it avoids explicitly regressing redundant raw RGB values.
T12
Lakest
212
ROsS Laket (DCNDN2)
20.5
ROSS Latent (DCTF)
Rosel (EVAII))
POPE
HillasionBeehh
MMVP
Cheng
MD-Bendh
Figure Choices the latent teacher tokenizer F. KL-16 (Rombach et al., 2022) is the best tokenizer it is originally used for reconstruction, and it is expected preserve the most image details. Other alternatives are utilized for classification (Touvron et al., 2022), instance-level representation learning (Oquab al., 2023), and language alignment (Fang al., 2024), respectively.
Visua Irstruction Thring
(DINDN2)
25L
(DINOVZ)
203
300
Rossi-L (XL,
Rooss (KL-18)
POPR
HällusionReerch
MMVP
ChaiQAA
MMBkrch
Figure Regression v.s. Denoising. With KL-16 the tokenizer, the denoising objective introduced in Equation (4) brings significant improvements over vanilla regression using MSE it avoids overfitting exact latent token values, even Rosse-Latent (KL-16) has already outperformed visual instruction tuning baseline by large margin.
Is there lemon inside the drink in the cup or are all the lemons outside? (a) There is one inside.
From which angle is this image taken? (a) Front. (b) Side.
Are all easter eggs placed in container? (a) Yes. (b) No.
(b) All are outside.
LLaVA: (b)
ROSS (a)
LLaVA: (a)
ROSS (b)
LLaVA: (al
ROSS (b)
Figure 8: Qualitative comparison on attention maps on MMVP (Tong et al., 2024b), where we keep the same LLM and training data. With extra vision-centrie supervision signals, Ross urges the model to focus on specific image contents corresponding to the question with higher attention values.
DEIT-III (Touvron al., 2022), and gegguage-supervisee EVA02CLIP (Fang al., 2024). Among them, KL-16 is the best choice. One intuitive explanation is that it is expected to preserve the most image details, since
originally designed to accurately reconstruct input images.
Regression v.s. Denoising, Figure 7, we study the effectiveness of the denoising objective over vanilla regression across different tokenizers, i.e., KI.-16 (Rombach al., 2022) and DINOv2 (Oquab
7

---

Published as a conference paper at ICLR 2025
Table 2: Generative v.s. Reconstructive. Following Sun et al. (2024b) and Dong et al. (2024), we adopt 576 learnable latent tokens to query the LMM and utilize the corresponding outputs as conditions to the denoiser for generative cases. Extra 102K caption data from ShTv (Chen et al., 2023b) is introduced to the original SFT data, facilitating text-to-image creation. Reconstructive objectives boost comprehension while generative alternatives cannot.
w/ LMM
Hallucination
Fine-Grained
General
Method
SFT Data
737K
102K
POPE
Hallu.
MMVP
ChartQA
MMB
Baseline
737K Caption 102K
86.2
55.1
32.0
30.9
74.4
Reconstructive
737K + Caption 102K
87.6
58.0
38.7
40.4
75.2
Reconstructive 737K + Caption 102K
87.6
56.3
37.3
39.7
74.9
Generative
737K + Creation 102K
85,4
52.0
30.0
31.2
73.9
Table 3: The effectiveness of the vision-centric supervision among various LLMs and visual encoders, where Cvisual manages to bring significant improvements consistently.
LMM
Language Model Cyisnal
POPE
Hallu.
MMVP
ChartQA
OCRBench
MMB
Visual Encoder: CLIP-VIT VIT-L/14
86.3
52.5
28.0
32.9
339
67.0
Vicuna-7B-vl.5
V
87.2 1 0.9
55.8 f 3.3
36.0
8.0
39.8 1 6.9
350 f 11
67.6
0.6
87.9
55.0
29.3
34.0
363
73.8
Wwen2-7B-Instruee
88.4 1 0.5
56.7
1.7
42.0 f 12.7
37.1 1 3.1
381 f 18
75.2 f 1.4
Visual Encoder: SigLIP-ViT-SO-A V000-00000000444400
384
86.0
50.4
27.3
36.2
354
64.5
Vicuna-7B-vl.55
86.8 1 0.8
53.2 1 2.8
38.0 1 10.7
41.6 1 5.4
365 f 11
65.7 f 1.2
88.5
57.3
40.7
44.4
432
76.3
Qwen2-7B-Instruet
88.7 1 0.2
58.2
0.9
49.3
8.6
46.3
1.9
448
1 16
76.9 0.6
et al., 2023). Notably, even RoseetLatent (KL-16) has already outperformed the visual instruction tuning baseline by a large margin, Ross manages to bring significant improvements by replacing regression with denoising. Therefore, denoising is better at handling visual spatial redundancy.
Finally, we leverage the insights and conclusions from all our previous studies to train our Ross. Specifically, we regard the optimal formulation Ross (KL-16), i.e., denoising with the KL-16 tokenizer, as our final Ross. Please refer to Appendix C.2 for ablations on the architecture of the denoiser, continuous tokenizer discrete tokenizer, and the denoising schedule.
5.2 IN-DEPTH ANALYSIS
Attention Analysis. We compute the attention scores of the last token with respect to all visual tokens on MMVP (Tong et al., 2024b). Quantitative and qualita- tive comparisons between the visual instruction tuning baseline (LLaVA) (Liu et al., 2024a) and our Ross provided in Table 1 and Figure 8, respectively. Table reveals that the attention scores achieved by RoSS are significantly higher than those of LLaVA, indicating that the inclusion of vision-centric recon- structive objective LLMM effectively directs focus towards input images, thereby enhancing the compre- hending visual signals. Similarly, Figure 8 demon- strate that the implementation of LLMM enables the alignment of attention closely with the relevant visual elements corresponding to the text query.
Table 1: Quantitative comparison on atten- tion values. We conduct a T-test (Student, 1908) to compare the means and Mann- Whitney U test (Mann & Whitney, 1947) to compare the medians of the two distributions. The mean and median Ross are both sig- nificantly higher than those of LLaVA.
Statistic (x10 LLaVV
Ross
P-value
Mean
2.03
2.36
1.27x10-?
25th Percentile
1.50
1.81
Median
1.90
2.26
10
75th Percentile
2.42
2,76
95th Percentile
3.51
3.69
Generative v.s. Reconstructive. We ablate the effectiveness of reconstruction over generation in Table 2. Similar to Sun et al. (2024b) and Dong et al. (2024), for generative cases, we adopt 576 learnalle latent tokens query the LMM and utilize the corresponding outputs conditions the denoiser. The detailed pipeline of these two methods can be found at Figure 11 in Appendix C.2. However, generative methods require specific creation data and not be naively implemented on
8

---

Published as a conference paper at ICLR 2025
Table 4: Comparison to state-of-the-at LMMs. mixture of 2M caption data and 1.2M instruction tuning data are utilized for pre-training and fine-tuning, respectively. Our model outperforms them in most of the settings. We evaluate these models on: POPE (Li et al., 2023c) averaged accuracy,
Hallu.: HallusionBennhh (Guan et al., 2024) average accuracy, MMB ;N; MMBench (Liu et al., 2023b) English dev split, MMBCN; MMBench (Liu et al., 2023b) Chinese dev split, SEED': SEED-Bench- (Li et al., 2023a) with image accuracy, MMMU (Yue et al., 2024) validation split, MMVP (Tong et al., 2024b), GQA (Hudson Manning, 2019) test-dev-balancee split, and A12D (Hippala et al.,
2021) test split. WWe evaluate the official checkpoint/api using VLMEvalKit (Duan et al., 2024).
Model POPE
Hallu. MMB EN
MMB
SEED
MMMU
MMVP GQA
A12D
GPT-4V-1106 (OpenAI, 2023a)
75.4
65.8*
75.8
75.1
71.6
53.8
50.0
36.8
78.2
Gemini-1.5 1.5 Pro (Team et al., 2023)
73.6
70.7
47.9
MM-1-8B (McKinzie et al., 2024)
86.6
72.3
69.9
37.0
72.6
Mini-Gemini--B (Li al., 2024f)
72.7
37.3
18.7
64.5
73.5
DeepSeck-VL: VL-7B (Lu et al., 2024)
85.81
44.1
73.2
72.8
70.4
36.6
64.9
Cambrian-1-88 1-8B (Tong et al., 2024a)
87.4*
48.7*
75.9
68.9F
74.7
42.7
51.3
64.6
73.0
Ross-7B 88.3
57.1
79.1
77.1
73.6
46.6
56.7
65.5
79.3
LLM: -B-v.--
LLaVA-vI.5-7B (Liu et al., 2024a)
86.2
47.5
65.5
58.5
66.0
34.4
20.0
62.0
55.4
LLaVA-v1.6-7B' (Liu al., 2024b)
86.5
35.8
67.4
60.1
70.2
35.8
37.3
64.2
67.1
Ross-7Baiue
88.2
55.2
67.7
61.3
67.6
36.9
39.3
63.7
69.3
Base LLM: Vicuma-138-v1.5
LLaVA-v1.5-13BB (Liu et al., 2024a)
82.5
44,9
68.8
63.6
68.2
36.6
32.0
63.3
60.8
LL 6-133 (1.iu et al., 2024b)
86.2
36.7
70.0
64.1
71.9
36.2
35.3
65,4
72.4
Mini-Geminii-33B (Ii et al., 2024f)
68.6
73.2
37.3
19.3
63.7
70.1
Cambrian-1-13B (Tong et al., 2024a)
85.71
54.0
75.7
65.97
74.4
40.0
41.3
73.6
Ross-13B
88.7
56.4
73.6
67.4
71.1
41.3
44.7
65.2
73.8
the original SFT data. To build creation data, we utilize GPT-4o transfer 102K caption into text-to- image creation data from ShareGPT4V (Chen et al., 2023b) and combine them with the original SFT data. From Table 2, we can tell that reconstructive objectives boost comprehension while generative alternatives cannol. An intuitive explanation of this evidence can be found in Appendix C.2.
Ross with Different LLMs and Visual Encoders. To demonstrate the effectiveness of our vision- centric supervision LMM adopted by Ross, conduct systematic experiments across different base LLMs and visual encoders. From Table 3, Ross contributes to significant improvements consistently, especially on fine-gra aineigrieed comprehension benchmarks, i.e., MMVP (Tong et al., 2024b) and ChartQA (Masry et al., 2022). Extended experiments on more representative benchmarks can be found at Table 12 in Appendix C.2.
Reconstruction Results. We fine-tune the denoiser to recover latent tokens from a frozen KI-16 provided by Rombach et al. (2022) conditioned Ross-7B features on ImageNet-1K 1K (Deng et al., 2009) for only five epochs, where the denoiser manages to produce reasonable reconstruction results as illustrated Figure 9 in Appendix C.I. This interesting finding demonstrates that high-level Ross-7B features actually contain image details. We hope this finding will inspire future work.
Computational Overhead. The denoising process introduces a negligible increase in training time (~10% compared to the baseline), while the benefits outweigh the minor additional costs. Please refer Table 10 in Appendix for details.
5.3 COMPARISON WITH STATE-OF-TIE--RTTS
Ross utlizes a single SigL.P- LLL.--VIT-SS4MM//44000 384 (Zhai et al., 2023) as the visual encoder. Ross- 7B utilizes Swen2-7B-Insteen
(Yang et al., 2024a) and Ross-13B, vicuma adopts Secuma-1BB-vll (Chi- ang et al., 2023) as the base LLM. The implementation almost follows LLaVA-v1.5 (Liu et al., 2024a) without the high-resolution image-slicing technique (Liu et al., 2024b). Thus, our primary compar- ison of Ross with alternative methods focuses on benchmarks that do not require exceptionally high-reson nigh-resolution inputs. We use mixture of 2M caption data for the pre-training stage, which consists of 1246K from ShareGPT4V (Chen et al., 2023b) and 707K from ALLaVA (Chen et al., 2024a). The instruction tuning data is mixture of Cambrian-737K (Tong al., 2024a) and SMR-473K (Zhang

---

Published as a conference paper at ICLR 2025
Table 5: Transfer learning on SpatialBench (Cai al., 2024). "RGB" indicates using only RGB images for testing, while "RGB + D" represents taking depth maps as extra inputs. The performance of GP is obtained from Cai al. (2024). LMMs can better comprehend depth maps with Lyisnal
Method Test Inputs
LMM
MiDuS
Size
Reaching
Position
Existence
Counting
Average
RGB
20.0
51.7
58.8
70.0
74.6
55.0
RGB + D
21.7 f 1.7
45.0 L 6.7
58.8 0.0
65.0 5.0
77.7 3.1
53.6 + 1.4
LLavA
RGB
21.7
60.0
64.7
80.0
84.1
62.1
RGB +
21.7
51.7 4. 8.3
70.6 1
65.0
15.0
91.1
7.0
60.0 4
RGB
V
25.0
53.3
64.7
70.0
75.3
57.7
Ross
RGB + D
V
28.3 f 3.3
65.0 f 11.7
67.6 2.9
85.0
15.0
84.6 x 8.7
66.1 1.1.884 8.4
RGB
43.3
51.7
70.6
85.0
84.5
67.0
GPT-4o
RGB +
40.0 3.3
40.0
51.7 0.0
61.8 8.8
90.0 1 5.0
85.2
0.7
65.7
(,.74..1 1.3
et al., 2024b). We further incorporate our ROSS with the "anyres" technique (Liu et al., 2024b) and compare with others on high-resolution benchmarks at Table 13 in Appendix C.3.
Illustrated in Table 4, we compare our Ross with both private models (OpenAI, 2023a; Team et al., 2023; McKinzie et al., 2024) and open open-sourced alternatives (Liu et al., 2024a;b; Tong et al., 2024a; Li et al., 2024f; Lu et al., 2024). The previous open-source state-of-the-art Cambrian-1 (Tong et al., 2024a) leverages extrinsic assistance that aggregates CLIP (Radford et al., 202 SigLIP (Zhai et al., 2023), DINOv2 (Oquab et al., 2023), and ConvNext (Liu et al., 2022). On the other hand, our Ross stands for intrinsic activation. With only a single SigLIP (Zhai et al.. 2023) model as the visual encoder, Ross surpasses Cambrian-1 1 (Tong et al., 2024a), under most cases, without careful choice of the visual experts and naturally maintains a lightweight inference procedure. Ross is also data-efficient compared with Cambrian-i (Tong et al., 2024a), since it requires 7M instruction tuning data. Notably. Ross-7B even surpasses GPT-4V-1 106 and Gemini- 1.5 Pro on several benchmarks such POPE (Li et al., 2023c), MMBench (Liu et al., 2023b), and MMVP (Tong et al., 2024b).
5.4 APPLICATIONS
Transfer Learning on Understanding Depth Maps. We further evaluate the transfer learning capability of our Ross on SpatialBench (Cai et al., 2024), which requires the model to understand depth maps. We compare our Ross with the visual instruction tuning baseline, with the same training data and model architecture. Also, we compare the effectiveness of the extrinsic assistance solution, i.e., combining a depth expert MiDaS-3.0 (Birkl et al., 2023) to visual instruction tuning, with our intrinsic activation solution. Specifically, the pre- treiirann data is LLa VA-558K (Liu et al., 2023a) and the fine-tur to-intnnngg data is SpatialQA- 853K (Cai et al., 2024), where each conversation contains the RGB image and the depth maps extracted by ZeeDepth (Bhat et al., 2023). The visual encoder is CLIP- ViT-I/1 11@@ 336 (Radford et al., 2021) 1) and the base L.MM is Qwen2-7B-Instruee (Yang et al., 2024a). As demonstrated in Table 5, our ROSS manages make and significant improvements are observed when taking "RGB + D" inputs for testing. Extrinsic assistance approaches cannot take advantage of extra depth maps when testing. Even GPT-4o cannot fully understand depth maps. Qualitative results can be found at Figure 16 in Appendix C.5.
of the extra depth map, as consistent
CONCLUSION
This paper introduces reconstructive visual instruction tuning (Ross), leveraging a vision-centric reconstructive objective to supervise visual outputs. To avoid being overwhelmed by heavily redun- dant raw RGB values, we train denoiser to recover clean latent visual representations conditioning on visual outputs. Experimentally, the proposed objective indeed brings enhanced comprehension capabilities and reduced hallucinations. Ross outperforms the state-of-the-art under most cases with only a single SigLIP (Zhai et al., 2023) as the visual encoder. The in-depth analysis demonstrates that high-level features from Ross-7B actually contain sufficient details for low-level image reconstruc- tion. This finding reveals the possibility of equipping comprehension LMMs with the ability of naive generation without the help of generation experts such as Stable Diffusion (Rombach et al., 2022).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
The work was supported by the National Science and Technology Major Project of China (No. 20232D012130),, the National Natural Science Foundation of China (No. U21B2042, No. 62320106010), the 2035 Innovation Program of CAS, and the InnoHK program.
REFERENCES
-aptiBat Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Flamingo: visual language model for fow-shot learning. Advances in neural information processing systems, 0123716--37735 2022.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: frontier large vision-language model with versatile abilities. arXiv preprint ar3iv:2308. S.di::200.//966, 12966, 2023.
Shariq Faronq Bhat, Reiner Birkl, Diana Wofk, Peter Wonka, and Matthias Müller. Zoedepth: Zero-shot transfer by combining relative and metric depth. arXiv preprint arXiv:2302. /2288, 2023.
Reiner Birkl, Diana Wofk, and Matthias Müller. Midas v3. 1-a model zoo for robust monocular relative depth estimation, arXiv preprint arXiv:2307.14460 2023.
Ali Furkan Biten, Ron Litman, Yusheng Xie, Srikar Appalaraju, and R Manmatha. Latr: Layout-aware transformer for scene-lext vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are leameess Advances in Neural Information Processing Systems (NeurIPS), 2020.
Iearners.
Wenxiao Cai, Yaroslav Ponomarenko, Jianhao Yuan, Xiaoqi Li, Wankou Yang, Hao Dong, and Bo Zhao. Spatialbot: Precise spatial understanding with vision language models, arXiv preprint arXiv:2406. 223i:22000..622. 13642, 2024.
Jie Cao and Jing Xiao, An augmented benchmark dataset for genmetric question answering through dual parallel text encoding. In Proceedings of the 29th International Conference on Computational Linguistics (COLING), 2022.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties supe-supervised vision transformers. In Proceedings the IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang. Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Hamesssin sptth-synthesiiedd data for a vision- Lasion-language model. arXiv preprint arXiv:2402.11684, 2024a.
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang. Lingbo Liu, Eric P Xing, and Liang Lin. Geoqa: geometric question answering benchmark towards multimodal numerical reasoning. arXiv preprint arXiv:2 14517, 2021.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xic, Yue Wu, Zhongdao Wang. James Kwok, Ping Luo, Huchuan Lu, al. Pixart-o: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12793, 2023b.
Minshuo Chen, Kaixuan Huang, Tuo Zhao, and Mengdi Wang. Score approximation, estimation and distribution recovery of diffusion models on low-dimensional data. In International Conference on Machine Learning (ICML), 2023c.
Zhe Chen, Weiyun Wang. Hao Tian, Shenglong Ye, Zhangwei Gao, Erfei Cui, Wenwen Tong, Kongzhi Hu, Jiapeng Luo, Zheng Ma, et al. How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.1692 16821, 2024b.
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Interavl: Scaling up vision foundation models and aligning for generic visual- -inguuiic tasks. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2024c.
11

---

Published as a conference paper at ICLR 2025
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng. Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, lon Stoica, and Eric Xing. Vicuna: An open-source chatbot impressing gpt-4 with 90%" chatgpt quality, March 2023, URL htttps://mmys.. wwg.yy...rggbooo. /
2023-03- 1000300-vicuu
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li. Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint urXiv:2305,0m 06500, 2023.
Jia Deng, Wei Dong, Richard Socher. Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hierarchical image database. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition
(CVPR), 2009.
Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi. Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. In International Conference on Learning Representations (ICLR), 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenbom, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Marthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition scale. International Conference on Learning Representations (ICLR), 2021.
Haodong Duan, Jumming Yang, Yuxuan Qiao. Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang. Pan Zhang, Jiaqi Wang, Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. arXiv preprint arXiv:2407. 1691,, 2024.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2 2783, 2024.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Yuxin Fang, Quan Sun, Xinggang Wang. Tiejun Huang, Xinlong Wang, and Yuc Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing, 2024.
Yuying Ge, Sijie Zhao, Ziyun Zeng, Yixiao Ge, Chen Li, Xintao Wang. and Ying Shan. Making Ilama see and draw with seed tokenizer. International Conference on Learning Representations (ICLR), 2024a.
Yuying Ge. Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi- qulti-granularity comprehension and generation. arXiv preprint 1rX1v22404.14396 2024b.
Ian Goodfellow, Jean Touget-Abadic, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances Neural Information Processing Systems (NeurIPS), 2014.
Yash Goyal, Tejas Khot, Douglas Summers-Star Staan Dhruv Batra, and Devi Parikh, Making the v vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Xin Gu, Heng Fan, Yan Huang, Tiejian Luo, and Libo Zhang. Context-guided spatio-temporal video grounding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 18330-183339 2024.
Tianrui Guan, Fuxiao Liu, Xiyang Wu. Ruiqi Xian, Zongxia Li, Xiaoyu Liu, Xijun Wang. Lichang Chen, Furong Huang. Yaser Yacoob, et al. Hallusionbench: an advanced diagnostic suite for entangled language hallucination and visual illusion in large vision- language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patter Recognition (CVPR), 2022.
Dan Hendrycks. Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawa Song, and Jacob Steinhardt. Mcasuring massive multitask language understanding, arXiv preprint 0tt:::00000030000 2020.
Jack Hessel, Ari Holtzman, Maxwell Forhes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprint arXiv:2104.08718 2021.
12

---

Published as a conference paper at ICLR 2025
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Tuomo Hiippala, Malihe Alikhani, Jonas Haverinen, Timo Kalliokoski, Evanfiya Logacheva, Serafina Orekhova, Aino Tuomainen, Matthew Stone, and John A Bateman. Ai2d-rst: A multimodal corpus of 1000 primary school science diagrams. Language Rexources and Evaluation, 55:661-683 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion-based generative models. Advances in Neural Information Processing Systems (NeurIPS), 2022.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP), 2014.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi. A diagram worth dozen images. European Conference on Computer Vision (ECCV), 2016.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannanch Hajishirzi. Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Diederik P Kingma. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg. Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma. et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), 2017.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems (NeurIPS), 2012.
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Ilms supercharge multimodal capabilities in the wild, May 2024a. URL.
gitub.. http///oggg//222// -00--000----00 10- llava next iitt--ste ssrssssrrrrrr-ree wwwoongrrrmmmm.....
Bo Li, Peiyuan Zhang, Kaichen Zhang, Fanyi Pu, Xinrun Du, Yuhao Dong, Haotian Liu, Yuanhan Zhang, Ge Zhang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Accelerating the development of large multimodal models, 2024b. URL https: / 11 thub. com/ www.liiiinnnseeeii Lab/ Lnms-eval.
Bo Li, Yuanhan Zhang, Dong Guo. Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Yanwei Li. Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408. 03326, 2024c.
Bohao Li, Rui Wang. Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed- Benchmarking multimodal Iims with generative comprehension. arXiv preprint arXiv:2307.16125 2023a.
Junnan Li, Dongxu Li, Silvio Savarese, Steven Hoi. BLIP-2: bootstrapping language-iman pre-training with frozen image encoders and large language models. In International Conference on Machine Learning (ICML), 2023b.
Lei Li. Yuqi Wang. Runxin Xu, Peiyi Wang, Xiachong Feng, Lingpeng Kong, and Qi Liu. Multimodal arxiv: A dataset for improving scientific comprehension of large vision-language models. arXiv preprint arXiv:2403.000 0023), 2024d.
13

---

Published as a conference paper at ICLR 2025
Tianhong Li, Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization, arXiv preprint arXiv:2406. /1..000........ 1838, 2024c.
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 2024f.
Yifan Li. Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Evaluating object hallucina- tion in large
models. arXiv preprint 23t:::3005.13355 2023c.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. In Advances Neural Information Processing Systems (NeurIPS), 2023a.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jac Lec. Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jac Lec. Lexta-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL httptt//LLLaaaa-LL wwwlllaaaaeeee next/.
1o/blog/2000 00000/20000000----2-0
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? arXiv preprini arXiv:2307.06220 07.20,,,,66,..111 2023b.
Yuliang Liu, Zhang Li, Biao Yang, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. On the hidden mystery of ocr in large multimodal models. arXiv preprint arXiv:2305.07899 2023c.
Zhuang Liu, Hanzi Mao, Chao-Yuaa Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepseek-vl: towards real-world vision-language understanding. arXiv preprint SMl::2403.0552. 2024.
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan I Liang, and Song-Chun Zhu. Inter-gps: Interpretable geometry problem solving with formal language and symbolic reasoning. arXiv preprint arXiv:2105.0m 04165.046...2220 2021.
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209. 14610, 2022.
Henry B Mann and Donald R Whitney. On a test of whether one of two random variables is stochastically larger than the other. The annals of mathematical statistics, pp. 50-60, 1947.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottgghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shatiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244420...044 2022.
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Doevqa: dataset for vqa on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision ( (WACV), 2021.
Brandon McKinzie, 7he Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, et al. Mm1: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint arXiv:2403. 00444 2024.
Anand Mishra, Shashank Shekhar, Aject Kumar Singh, and Anirban Chakraborty, Ocr-vqa: Visual question answering by reading text in images. In International Conference on Document Analysis and Recognition (ICDAR), 2019.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference on Machine Learning (ICMI.). pp. 16784-16804. PMLR, 2022.
14

---

Published as a conference paper at ICLR 2025
Aaron den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. arXiv preprint arX0v:/711.00937, 2017.
OpenAL. GPT-4V(ision) System Card, 2023a. URL http:::tttpt.pcppeeeeeeeeeeeeece http:ll..cpppppppppeee
System_ pard
OpenAL Gpt-4 technical report. arXiv, pp. 2303 08774, 2023b.
Maxime Oquab, Timothée Darcet. Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. DINOv2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.071 2023.
Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, Ilya Sutskever, al. Language models unsupervised multitask learners. OpenAl blog, 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Adity Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML). 2021.
Zhongwei Ren, Yunchao Wei, Xun Guo, Yao Zhao, Bingyi Kang, Jiashi Feng, and Xiaojie Jin. Videoworld: Exploring knowledge learning from unlabeled videos. arXiv preprint arXiv:2501.097) 2025.
Robin Rombach, Andreas Blatmmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2022.
Tanik Saikh, Tirthankar Ghosal, Amish Mittal, Asif Ekbal, and Pushpak Bhattacharyya. Scienceqa: A novel resource for question answering on scholarly articles. International Journal on Digital Libraries, 23(3): 289-301.20 2022.
Jorge Sánchez and Florent Perronnin. High figh-dimensional signature compression for large-scale image classifi- cation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2011.
Christoph Schuhmann, Romain Beaumont, Richard Veneu, Cade Gordon, Ross Wightman, Mchdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale datasct for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS), 35:25278-25294 2022.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: benchmark for visual question answering using world knowledge. In European Conference on Computer Vision (ECCV), 2022.
Ramprasnath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: visual explanations from deep networks via gradient-based localization. International Journal of Computer Vision (IJCV), 128:336-359, 2020.
Yaojie Shen, Xin Gu, Kai Xu, Heng Fan, Longyin Wen, and Libo Zhang. Accurate and fast compressed video captioning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 15558 15567, 2023.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that read. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances Neural Information Processing Systems (NeurIPS), 2019.
Student. The probable error of mean. Biometrika, pp. 1-25, 1908.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint arXiv:2406.06555, 2024a.
Quan Sun. Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang. Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint ar 20770220.. 2023.
15

---

Published as a conference paper at ICLR 2025
Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang. Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024b.
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean- Cean-Baptiste Alayrac, Jiahui Yu, Radu Soricut,
Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: a family of highly capable multimodal models.
arXiv preprint arXiv:331.. 1805, 2023.
Qwen Team. Qwen2.5: party of foundation models, September 2024. URL WtRLhtttss////ennlne http:////wwlllg.ghh......
io/blog/ wowllo/g/w/....... qweeg. 5/.
ShareGPT Team. Sharegpt, 2023. URL htttp:://ssaaaeeee
com.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu. Sai Charitha Akula, Jihan Yang. Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-I: A fully open, vision-centric exploration of multimodal llms. arXiv preprint 2rov 16860, 2024a.
Shengbang Tong. Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCum, and Saining Xie. Eyes wide shut? exploring the visual shortcomings multimodal llms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024h.
Hugo Touvron, Matthieu Cord, and Hervé Jégou. Deit III: Revenge the vit. European Conference on Computer Vision (ECCV), 2022.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. LLaMA 2: Open foundation and fine-tuned chat models. arXiv preprint urXiv:2307.0922 0098889888.......88 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, Advances in Neural Information Processing Systems (NeurIPS), 2017.
Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol. Extracting and composing robust features with denoising autoencoders. In International Conference on Machine Learning (ICML), 2008.
Haochen Wang, Junsong Fan, Yuxi Wang, Kaiyou Song, Tiancai Wang. Xiangyu Zhang, and Zhaoxiang Zhang. Bootstrap masked visual modeling via hard patches mining. arXiv preprint arXiv:23/2.13714, 2023a.
Haochen Wang. Junsong Fan, Yuxi Wang, Kaiyou Song, Tong Wang, and Zhaoxiang Zhang. Droppos: Pre- training vision transformers by reconstructing dropped positions. Advances in Neural Information Processing Systems (NeurIPS), 2023b.
Haochen Wang, Kaiyou Song, Junsong Fan, Yuxi Wang. Jin Xie, and Zhaoxiang Zhang, Hard patches mining for masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023c.
Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li, Liwei Wu, Yuxi Wang, and Zhaoxiang Zhang. Pulling target to source: A new perspective on domain adaptive semantic segmentation. International Journal of Computer Vision (ICCV), pp. 1-24, 2024a.
Haochen Wang. Yuchao Wang. Yujun Shen, Junsong Fan, Yuxi Wang, and Zhaoxiang Zhang. Using unreliable pscudo-labels for label-efficient semantic segmentation. International Journal of Computer Vision (IJC)), pp- 1-23, 2024b.
Jiacong Wang, Bohong Wu, Haiyong Jiang. Zhou Xun, Xin Xiao, Haoyuan Guo, and Jun Xiao. World to code: Multi-modal data generation via self-instructed compositional captioning and filtering. In Proceedings of the Conference on Empirical Methods in Natural Lanuagge Processing (EMNLP), pp. 4608-4623, 2024c.
Ke Wang. Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402. /4804, 2024d.
Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision- laiiua- model's perception of the world at any resolution. arXiv preprint 2409 12191, 2024c.
Yuchao Wang, Haochen Wang, Yujun Shen, Jingjing Fei, Wei Li. Guoqiang Jin, Liwei Wu, Rui Zhao, and Xinyi Lee Semi-supervised semantic segmentation using unreliable pseudo-labels. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 4248-4257, 2022.
16

---

Published as a conference paper at ICLR 2025
XAI Grok. 2024.
Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. In International Conference Learning Representations (ICLR), 2022.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXiv:2407.1067 10671, 2024a.
Ruofeng Yang, Zhijie Wang, Bo Jiang, and Shuai Li. The convergence of variance exploding diffusion models under the manifold hypothesis. 2024b. URL
wwt.riiwwwweeeeeeeeeeeeeeeeeeeee forum? Desesrumind-DDD- DNNOXYY
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren. Yuxuan Sun, al. Mmmu: massive distiliie multimodal understanding and reasoning benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint (rXiv:1905.0788) 2019.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023.
Libo Zhang. Xin Gu, Congcong Li, Tiejian Luo, and Heng Fan. Local compressed video stream learning for generic event boundary detection. International Journal of Computer Vision (IJCV), 132(4):1187- 1204, 2024a.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), 2018.
Yi-Fan Zhang, Qingsong Wen, Chaoyou Fu, Xue Wang. Zhang Zhang, Liang Wang, and Rong Jin. Beyond Ilava-hd: Diving into high-resolution large multimodal models. arXiv preprint arXiv:2406.08449 2024b.
Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou. Instruction followwing evaluation large language models. arXiv preprint arXiv:2311.07911, 2023.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Youngjae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion-scale corpus of images interlcaved with text. Advances in Neural Information Processing Systems, 36, 2023.
17

---

Published as a conference paper at ICLR 2025
APPENDIX
A LATENT DIFFUSION MODELS
Given set of clean latent tokens 20 drawn from p(z), the forward diffusion process is Markov chain that gradually adds random Gaussian noise to the original sample:
q<<||=<1) = -V(VI- 3,
(5)
where N(p Σ) denotes the Gaussian distribution, and indicates discrete timesteps. 3r E (0. 1) indicates pre-defined ameedeeendnnt variance schedule. According to Ho al. (2020), admit sampling at an arbitrary timestep / directly from Zo. this transition can be reformulated as
( 2u) NN(āā (1 Tx)
Ze = 20 + arE E E~
where a = - 3r and ā = -IT 0g. latent diffusion model learns to reverse this progressive noise addition process for latent tokens. Specifically, iteratively generate clean tokens zg from pure noise 2y conditioned on C, we need to reverse the forward process by
1 (z,: vT-ar 0t
2e
OLE,
Va
where a r7-parameterized neural network en is trained to predict the added noise during the forward process. d, indicates the posterior noise variance. The training objective of Ex is
[(T, z0) Er, [||ea (0 vI C,E;C, 8) E <E|31
(8)
B
IMPLEMENTATION DETAILS
Table 6: Hyperparameters of Ross. We obtain most of the configurations from Liu et al. (2024a).
Config
Stage
Stage II
Trainable parts
projector + denoiser
projector + LLM denoiser
Frozen parts
visual encoder LLM + teacher tokenizer
visual encoder teacher tokenizer
Global batch size
256
Batch size per GPU Accumulated steps
16
4
2
4
DeepSpeed zero stage Learning rate Learning rate schedule
2
3
x 10 33
2x 10 -
warmup cosine decay
Warmup ratio
0.03
Weight decay
Epoch
Optimizer Precision
AdamW bf16
Table 7: Details of the instruction tuning dataset provided by Tong et al. (2024a).
Table 8: Details of the instruction tuning dataset provided by Zhang et al. (2024b).
Dataset
# Samples
Dataset
Samples
LLaVA (Liu et al., 2023a)
158K
ScienccQA (Saikh et al., 2022)
9K
ShareGPT (Team, 2023)
40K
TextbookQA (Kembhavi et al., 2017)
9.5K
VQAv2 (Goyal al., 2017)
83K
A12 Diagrams (Kembhavi et al., 2016)
12.4K
GQA (Hudson & Manning, 2019)
72.1K
ChartQA (Masry et al., 2022)
28.3K
OKVQA (Marino et al., 2019)
9K
DVQA (Kafic et al., 2018)
200K
OCRVQA (Mishra et al., 2019)
80K
ArxivQA (Li al., 2024d)
100K
A-OKVQA (Schwenk al., 2022)
50K
GeoQA3 (Chen et al., 2021)
5K
TextVQA (Singh al., 2019)
21.9K
Geometry3K (Lu et al., 2021)
2.1K
RefCOCO (Kazemzadeh et al., 2014)
30K
GeoQA+ (Cao & Xiao, 2022)
72.3K
VG (Krishna et al., 2017)
86.4K
Math Vision (Wang al., 2024d)
2.7K
DVQA (Kafle et al., 2018)
13K
TabMWP (Lu al., 2022)
30.7K
DocVQA (Mathew et al., 2021)
15K
ChartQA (Masry al., 2022)
28.1K
A12 Diagrams (Kembhavi et al., 2016)
15.5K
18

---

Published as a conference paper at ICLR 2025
Table 9: Summary of the evaluation benchmarks. Prompts are mostly borrowed from VLMEvalKit (Duan et al., 2024) and Imms-eval (Li et al., 2024b).
Benchmark
Response formatting prompts
POPE (Li et al., 2023c)
HallusionBench (Guan al., 2024)
Answer the question using a single word or phrase. Answer with the option's letter from the given choices directly. Answer with the option's 's letter from the given choices directly. Answer with the option's 's letter from the given choices directly. Answer with the option's 's letter from the given choices directly. Answer with the option's 's letter from the given choices directly.
MMBench (Liu et al., 2023b)
SEED-Bench (Li et al., 2023a)
MMMU (Yue et al., 2024)
MMVP (Tong al., 2024b)
A12D (Hiippala et al., 2021)
RealWorIdQA (xAI, 2024)
Answer with the option's 's letter from the given choices directly.
GQA (Hudson & Manning, 2019)
Answer the question using a single word or phrase.
ChartQA (Masry al., 2022)
Answer the question using single word or phrase.
OCRBench (Liu et al., 2023c)
Answer the question using a single word or phrase.
Doc Doc (Mathew et al., 2021)
Answer the question using a single word or phrase.
InfoVQA (Biten et al., 2022)
Answer the question using a single word or phrase. Answer the question using single word or phrase.
TextVQA (Singh al., 2019)
Table 10: Comparisons on computational costs during the instruction tuning stage with Cambrian- 737K (Tong et al., 2024a), where evaluations are conducted using 8 A100 GPUs with a global batch size of 128. Due to the limited GPU memory, accumulate 4 gradient steps and the batch size per GPU is 4. The whole stage requires 5757 training steps. GPU memories are averaged over 8 GPUs
with DeepSpeed Zero 3.
Trainable Parameters
GPU Memory
Vision
Base LLM
LMM
Speed (s/iter)
Time
CLJP-L//36
Qwen2-7B- Tn-strucct
7.63 B
8.31
13h 17min
45.34 G
CLIP-L/336
Qwen2-7B-Inee Instructt
7.68 B
9.02 (1.09x)
14h 25min
46.62 G (1.03x)
CLIP-L/336
Vicuna-I3B-v
13.05 B
13.33
21h 19min
48.62 G
CL3P-L/336
Vicuna-13B-v1..
13.11
14.69 (1.10x)
23h 30min
49.07 G (1.01x)
SigLIP-L/384
Bren2-BB-Instruce
7.63 B
8.77
14h Imin
47.08 G
SigLIP-L/384
Owen2-77-Innttuun
7.68 B
9.48 (1.08x)
15h 9min
52.07 G (1.1Ix)
SigLIP-LL/84
Vicuna-13B-vIl
13.05 B
14.22
22h 44min
48.80 G
SigLIP-L/384
Vicuna-13B-vll
13.11 B
15.32 (1.08x)
24h 30min
52.68 G (1.08x)
Hyperparameters. The hyperparameters of Ross are provided in Table We simply borrow most contigurations from LLaVA-v1.5 (Liu et al., 2024a) without further tuning, as we find it works well with our Ross, even if we adopt SigLIP (Zhai et al., 2023) and Qwen2 (Yang et al., 2024a) while the original LLaVA-v1.5 (Liu et al., 2024a) utilized CLIP (Radford et al., 2021) and Vicuna-v1.5 1.5 (Chiang al., 2023). As SigLIP represents a single 384 384 image with 729 tokens and the downsampling ratio of the teacher tokenizer KL-16 16 (Rombach et al., 2022) is 16, we set the input resolution of the teacher tokenizer as 432- = 729 x 16 to produce 729 fine-grained tokens as denoising targets.
Instruction Tuning Data. When comparing with ofate-of-art LMMs in Table 4, our Ross is trained on approximately 1 1.2M instruction tuning data, which is a mixture of 7ambrian-737K (Tong et al., 2024a) and SMR-473K (Zhang et al., 2024b). Details of these two instruction tuning datasets are listed in Table 7 and Table 8, respectively. There might be some overlap but we simply concat these two datasets as it is already empirically effective.
We provide thorough examination of all evaluation benchmarks utilized in this paper in Table 9. Notably, for MMVP (Tong et al., 2024b), which is not officially supported by VLMEvalKit (Duan et al., 2024). follow Cambrian-1 (Tong al., 2024a) to reformat the original question into a multiple-choice format and compute the accuracy using exact matching.
Evaluation Prompts.
Computational Costs. As demonstrated in Table 10, the denoising process introduces a negligible increase in training time (==00% compared to the baseline), while the benefits outweigh the minor additional costs.
19

---

Published as a conference paper at ICLR 2025
Figure 9: Reconstruction results on ImageNet-Ik 1K (Deng et al., 2009) validation set. For each tuple, we show the input image (left) and the reconstructed image (right). Reasonable reconstruction results demonstrate that high-level features of Ross-7B can projected back into the pixel space.
542
3k SalfArt (SBM)
3x wo Salivtta (1784)
12x we (881
POPE
HallustonBarch
MMVP
ChastQA
Figure 10: The architecture of the denoiser and the choice of fine-grained tokenizer. The self- attention module illustrated in Figure 4b is crucial since orange hars consistently outperform others hallucination and fine-grained comprehension benchmarks, while maintaining similar performances on the general understanding benchmark. KL-16 provided by Rombach et (2022) better than VQ- 16 provided by Sun et al. (2024a), as quantization may lead to information loss.
C MORE EXPERIMENTS
C.1 RECONSTRUCTION RESULTS
We fine-tune the denoiser recover latent tokens from a frozen KL-16 provided by Rombach et al. (2022) conditioned on frozen Ross-7B features on ImageNet- IK (Deng et al., 2009) for only five epochs, where the denoiser manages produce reasonable reconstruction results illustrated in Figure 9. This interesting finding demonstrates that high-level Ross-7B features actually contain image details.
C.2 MORE ABLATIONS
KL-16 v.s. VQ-16. Our default tokenizer is a continuous VAE (Kingma, 2013) with Bullback-Leible
(KL) divergence trained by Rombach et al. (2022). We further conduct experiments with a discrete tokenizer provided by Sun et al. (2024a), which a VQGAN (Esser al., 2021). i.e., VQVAE (Oord et al., 2017) with additional perceptual loss (Zhang et al., 2018 8) and adversarial loss (Goodfellow et al., 2014). As demonstrated in Figure 10, KL-16 outperforms VQ-16. One intuitive explanation that KL-16 preserves more low-level details than VQ-16 since quantization may lead to information loss. Moreover, guantitatively, on ImageNet (Deng et al., 2009) 256x validation set, KL-16 achieves 0.87 rFID (Heusel et al., 2017) while the rFID (Heusel al., 2017) of VQ-16 is 2.19.
Architecture of the Denoiser. Illustrated in Figure 10, the self-attention module is crucial, as original visual outputs VęSN actually causal and need to model inter-token discrepancy via self-attention. The number of trainable parameters is not the crux.
Schedule of 3. We study the effectiveness of different schedules of in Table 11. From the table, we can tell that even with different schedules of B, Ross consistently improves the baseline, demonstrating its robustness to the denoising schedule.
20

---

Published as a conference paper at ICLR 2025
Table 11: Ablations on different schedules of 3. Ross consistently improves the baseline, demon-
strating its robustmess to the denoising schedule.
Schedule of B
POPE
Hallu.
MMVP
ChartQA
MMB
87.9
29.3
34.0
Linear (Ho et al., 2020)
88.1
f 0.2
57.3
42.0 f 12.4
39.2
75.1
f 1.3
Scaled Lincar (Rombach et al., 2022)
88.4
1 0.5
1.3
40.0
f 10.4
40.7
f 6.7
75.3
f 1,5
GLIDE Softmax (Nichol et al., 2022)
88.4 0.5
59.1
42.7
1 13.4
40.4
6.4
75.2
1.4
GeoDiff Sigmoid (Xu al., 2022)
88.2 0.3
1..7
41.3
11.7
38.9
4.9
75.5
1.7
Reconstructive Vaual Supervision
Test Supervision
Generative Visual Supervtion
Large Language Model
Large Language Model
Projector
Tosanizer
type
show in the image?
logg passengor jot belonging
ADS STANT: The shown passenger helonging
China Airinas.
oquary> Fromet and
China Airlines.
(homph instructionss)
Pipeline comparison between reconstructive and generative. The reconstructive objective (left) does nor require specific data formulations and can be easily combined with current visual instruction tuning data. However, the generative objective (right) needs specific cext-to-image creation data, which could be converted by image-to-text caption data.
Figure 11:
Generative v.s. Reconstructive. We offer detailed pipeline comparison in Figure 11. Experimental results have already been provided in Table 2. The implementation of generative methods similar to Sun al. (2024b) and Dong al. (2024), where adopt 576 learnable queries as inputs and take the corresponding outputs as conditions for the denoiser.
We hypothesize that the underlying reason for the lower performance of generative methods in comprehension tasks is the weak correspondence between inputs and supervision under generative settings, which typically arises from both the (I) data and the (2) design of these methods.
(I) Typical generative methods that explore the synergy of comprehension and generation, usually leverage image generation conditioned text instructions (i) text-to-image datasets (ii) interleaved datasets as extra supervision. However, (i) text-to-image datasets are typically designed generate high-aesthetic samples rather than text-aligned ones, and (ii) interleaved datasets aim to enable few-shot learning via interleaving independent supervised examples, where reasoning becomes more important than alignment. Therefore, there exists clear disconnect where the supervision (image) has little to do with the input (text instruction). For example, the CLIP-Scor (Hessel et al., 2021), which measures the similarity between text and images, only 0.3043 for the LAION-Art dataset (Schuhmann et al., 2022) and 0.28422 for the MMC4 dataset (Zhu et al., 2023), indicating that these datasets are not well-suited for tasks requiring strong text-image alignment.
(2) Even when we attempt to ensure image-text alignment by converting aligned caption data into creation data for supervision, the results demonstrated Table 2 remain unsatisfactory. This suggests that the design of generative objectives useef does not inherently require strong correspondence between inputs and supervision targets.
In contrast, reconstructive methods like Ross leverage the original input images auxiliary super- vision, ensuring a strong and direct correspondence, which is crucial for tasks requiring accurate comprehension and interpretation of multimodal data, leading to significantly improved performance.
Extended Ablations on Diffferntt LLMs and Visual Encoders. We extend the ablation Table 3 by incorporating more benchmarks, providing more balanced and representative distribution of tasks. Empirical results in Table 12 demonstrate that our proposed iisionccentric supervision utilized by ROSS leads significant improvements in most cases. Moreover, found Ross contributes more
21

---

Published as a conference paper at ICLR 2025
Table 12: Extended ablations on The effectiveness of the vision-centric supervision Cvisual among
LLMM
various LLMs and visual encoders. Pre-training data is LLaVA-558K (Liu al., 2023a) and instruction tuning data Cambrian-7377K (Tong et al., 2024a). Evaluations of POPE (Li et al., 2023c), HallusionBench (Guan al., 2024), MMBench (Liu et al.. 2023b), SEED-Bench-I (Li al., 2023a), MMMU (Yue et al., 2024), MMVP (Tong et al., 2024b), AI2D (Hiipplaa et al., 2021), OCRBench (Liu al.. 2023c), and RealWorldQA (xAI, 2024) are conducted with VLMEvalKit (Duan al., 2024), while evaluations of ChartQA (Masry et al., 2022), Doc (Mathew et al., 2021), InfoVQA (Biten al., 2022), and Tex(VQA (Singh et al., 2019) are conducted with Imms-eval (Li al., 2024b).
CLIP-VIT-L/14023
SigLIP- ST--I--VTTTSS000/14eee0
Vicuna-7B-v1.5
Bwen2-7B-Instrue
7B-v1.5
Qwen2- 7B-Instruct
Benchmark
LLaVA
Ross
LLaVA
Ross
LLaVA
Ross
LLaVA
Ross
POPE
86.3 87.2 f 0.9
87.9
88.4
0.5
86.0
87.7
f
1.7
88.5
88.7
0.2
HallusionBenchane
52.5 55,8 1 3.3
55.0
59.1
f 4.1
50.4
53.8
3.4
57.3
58.2
0.9
MMBennchENN.
67.0 67.6 1 0.6
73.8
75.2
1.4
64.5
69.2
4.7
76.3
76.9
0.6
MMBench-CNNL
60.0 59.8 4. 0.2
72.9
73.7
1
63.1
63.4
0.3
75.7
76.3
SEEDing
66.7 66.4 0.3
0.3
70.3
70.7
1 0.4
68.2
69.0
0.8
72.3
72.1
0.2
MMMU
30.0 34.0 1 4.0
44.0
45.3
1.3
33.3
38.0
4.7
38.7
41.3
2.6
MMMUval
35.3 36.0 0.7
0.7
41.9
0.77
1.2
43.8
2.0
MMVP
28.0 36.0 8.0
29.3
42.7
1..7113.4
27.3
38.0 t 10.7
40.7
49.3
8.6
AT2Diest
61.2 61.4 0.2
71.9
73.3
1.4
62.6
6..4
74.0
74.5
0.5
ChartQAsex
32.9
39.8
6.9
36.2
41.6
15.4
34.0
48.2
14.2
44.4
46.9
2.5
VQA
33.4
41.6
8.2
31.1
44.7
13.6
40.4
40.7
0.3
39.2
39.3
IQAvl
21.2
26.4
1
5.2
22.1
39.3
1.31160
16.2
22.8
23.3
1.5
0.5
24.0
25.1
1.1
TextVQAuul
55.7
58.7
3.0
52.0
54.1
1.11
60.5
62.6
1.1 2.1
56.3
57.5
1.2
OCRBench
339
350
11
381
18
354
365
11
432
448
16
RealWorI
52.7
53.2
1 0.5
56.7
57.4 1 0.7
55.0
57.1 1 2.1
57.9
59.1
1.2
Average
47.8
50.6 1 2.8
52.1
56.4
1 4.3
49.2
52.4
3.2
55.4
56.9
1.5
Comparison to statc-of-the-art LMMs on benchmarks requires high- high-resolution inputs. We evaluate models on: ChartQA (Masry al., 2022), Doc (Mathew al., 2021) val set, IQA (Biten et al., 2022) val set, TexIVQA (Singh et al., 2019) val set, OCRBench (Liu et al., 2023c), and RealWorldQA (xAI, 2024). We evaluate the official checkpoint.
Table 13:
Model
ChartQ
DoeVQA
InfoVQA TexiVQA OCKBench RealWorldQA
GPT-4V-1 106 (OpenAI, 2023a)
78.5
78.0
645
Gemini-1.5 1.5 Pro (Team et al., 2023)
81.3
86.5
78.1
67.5
Grok-1.5 1.5 (xA1, 2024)
76.1
85.6
78.1
68.7
LLaVA-vI.5-7D (Liu et al., 2024a)
18.2
28.1
25.7
58.2
317
54.9
LLaVA-v1.6-7BB (Liu et al., 2024b)
65.5
74.4
37.1
64.8
S32
57.6
Cambrian- (Tong al., 2024a)
73.3
77.8
71.7
624
64.2
Ross-7Bamre
76.9
50.5
72.2
607
66.2
significant improvements over fine-graines comprehension datasets, such HallusionBench (Guan al., 2024), MMVP (Tong et al., 2024b), and ChartQA (Masry et al., 2022).
C.3 COMPARISON ON HIGH-RESOLUTION BENCHMARKS
We incorporate the "anyres" technique proposed by LLLVA-v (Liu et al., 2024b) into our Ross. Specifically, for each image, we employ grid configuration of 384x (2x2, x (2.3,4), (2,3,4) 1) to identify the input resolution, resulting in a maximum 384x380 crop is required to reconstruct the original input via the denoising objective proposed by Ross. In Table 13, our ROSS-7Bayyre surpasses LLa VA-vv.6-7767 (Liu et al., 2024b) and Cambrian-1- 8B (Tong al., 2024a) under most cases. These results indicate that Ross not only performs well at lower resolutions but also maintains its competitive edge at higher resolutions, making a robust and versatile method.
5x729 = 3,645 visual tokens. Each
22

---

Published as a conference paper at ICLR 2025
Table 14: Evaluations language performance. We evaluate multi-modal benchmarks that mainly require general knowledge following Tong et al. (2024a). Furthermore, we incorporate representative language benchmarks, including general understanding on MMLU (Hendrycks et al., 2020) and HellaSwag (Zellers et al., 2019), and instraction-followingg on IFEval (Zhou et al., 2023). ROSS does not harm language capabilities as it brings improvements in most cases.
CLJP-ViT-L/14033
Sig.JIP-VTTSSO400OOM
Vicuna-7B-v1.5
Swen2-7B-Instruu
7B-vl.-
Qwen2-7B-Insse 7B-Instruct
Benchmark
LLaVA
Ross
LLaVA
Ross
LLaVA
Ross
LLaVA
Ross
Vision Language Benchmarks on Knowledge
ScienceQAsen
68.5
69.0 ? 0.5
76.5
77.4
f 0.9
69.6
71.3
1.7
78.3
78.5
0.2
MMMU
30.0
34.0 f 4.0
44.0
45.3
1.3
33.3
38.0
4.7
38.7
41.3
2.6
MMMU
35.3
36.0 f 0.7
41.9
42.6
0.7
0.7
34.2
35.4
1.2
41.8
43.8
2.0
A12Dues
61.2
61.4
0.2
71.9
73.3
1.4
62.6
62.4.00.2
74.0
74.5
0.5
Language Benchmarks
MMLU
26.5
27.4 + 0.9
57.1
60.7
3.6
26.0
25.9
60.9
61.0
f 0.1
HellaSwagae-mrm
27.0
26.9 L. 0.1
46.4
46,2
0.2
27.1
27.0
0.1
46.6
IFEyalariceins
41.2
44.6 f 3.4
47.1
49.2
2.1
43.6
43.8
1 0.2
47.8
48.1
0.3
IFEvalarss.pongt
28.7
35.3 0 6.7
35.1
37.0
1.9
32.5
33.1
1 0.6
35.3
36.2
0.9
Average
39.8
41.8 1 2.0
52.5
54.0
1 1.5
41.1
42.1
1 1.0
52.8
53.8
1.0
Table 15: Model scaling of Ross. We take Qwen2.5 series (Team, 2024) as the base language model and LLLP--VTTL/1443331 (Radford et al., 2021) as the visual encoder. Pre-training data VA-55K (Liu et al., 2023a) and the instruction tuning data is VA-V6-665K (Liu et al., 2024a). Ross brings improvements over the baseline across different model sizes in most cases.
0.5B
1.5B
3
7B
Benchmark
LLaVA
ROSS
LLaVA
Ross
LLaVA
Ross
LLaVA
Ross
POPE
50.0
60.4
10.4
85.3
87.9
2.4
87.3
88.1
f 0.8
87.9
88.4
f 0.5
HallusionBemchaake
45.8
48.0
2.2
48.7
49.6
0.9
52.2
52.2
0.0
48.7
53.7
1 5.0
MMBench-FNNa
55.2
60.4
5.2
67.5
68.2
1.7
70.6
71.4
0.8
75.0
75.7
1 0.7
MMBench-CN..
45.6
48.9
1.3
62.4
63.9
1.5
68.0
69.1
1.1
73.6
73.5
4.1.
SEED mngg
55.8
55.6 1. 0.2
66.3
0.5
68.2
+
70.6
71.0
1 0.4
OCRBench
229
248
19
291
298 7
313
308 1
334
358
f 24
MM
35.2
36.0
0.8
44.7
45.0
1..0
0.3
48.7
49.0
0.3
48.0
48.0
0.0
MMMU
38.0
40.3
1.7
41.8
1.8
41.6
42.7
111
47.3
48.0 1 0.7
A12D
45.3
46.0
1 0.7
59.0
59.5
0.5
62.9
63.2
0.3
0.3
68.3
68.5 1 0.2
RealWorIdQA
45.1
46.4
1.3
50.5
53.5
3.0
55.7
57.9
2.2
59.5
59.9 1 0.4
Average
43.9
46.7
2.8
55.3
56.8 1.5
58.9
59.3
1 0.4
61.2
62.3 1.1
Table 16: Data scaling of Ross. We take Qwen2-7B Instrruct (Yang et al., 2024a) as the base language model and CLIP-ViT-L/140333 (Radford et al., 2021) as the visual encoder. Ross consistently brings significant improvements the training data scale increases.
PT
SFT
Orianal
POPE
Hallu.
ChariQ
OCRBench
FMB
A12D
87.9
55.0
34.0
363
73.8
72,4
558K
737K
88.4
0.5
59.1
4.1
40.4 6.4
380 17
75.2
1.4
73.3
0.9
88.5
57.3
37.0
389
75.7
74.5
558K
1.2M
88.8 0.3
57.8 0.5
42.0 1 5.0
392 f
76.8
1.1
74.7
0.2
88.1
55.6
37.3
384
76,2
72.3
2M
737K
88.3
0.2
56.2 0.6
41.9
4.5
398
14
77.0
0.8
73.4
1.1
88.5
53.8
41.2
388
76.5
73.9
2M
1.2M
88.9
0.4
57.3
43.2
2.0
405
17
78.0
1.5
74.1
0.2
C.4 MORE ANALYSIS
Language Capabilities. One possible concern of Ross that this type of low-level reconstruction may harm the high-level language capabilities. To investigate this issue, evaluate multi-modal
23

---

Published as a conference paper at ICLR 2025
From the camera's perspective, the spider web very dense or relatively sparse? (a) Very dense
(b) Relatively sparse
1,000 Steps
3,000 Stops
5,000 Steps
7,000 Stops
9,000 Steps
Training Evolution
Figure 12: Qualitative comparison using GradCAM (Selvaraju et al., 2020) MMVP (Tong et al., 2024b). We visualize the gradient of the second-to-last block of the LMM using the option of the ground-truth answer the target class. Equipped with our proposed vision-centric supervision signals, ROSS provides more reasonable gradients and urges LMMs to focus on relevant regions (the spider web) the training evolves.
Wis consin
Lake Michigan
USA
New York
Mirhigan
Jowa
Lakee
Accurding to image, is New & only state that shares border with I. Lake Huron?
the right orange dircle smaller than left orange circle?
No. New Yock is not only U. stat that dhors borter with Labe Ranos. The image deaws the the Crui Laioe region indicie hhe staae of Mishigan, Obis, Indiana, Illineis iecemin and Minnessta,
LLavt v1.570 70
Yes, the ocnne ctrile is smaller than the left orange ctrile
Llakk-r1.5 -.5.--
Combrtn
Yos, orang
omage
which suare bortles
The orange cimtes beth the left and rggtt tmages am setually the same dine Thes is visuel illyenn where the aemrnnting circlee unciuaace the peraaved suze of the carnal oragge cince. the left azzape, the smaller the circies make the oraae cirtle appozz Jarprr whenes right image. the large Niw make onrge apparranalle les
Cambriaa 1-00
Usved on the magge New York is not the only U.S. state that doors bortar will Lakia Baanaa
GPT-W
does not Now York bertentng Loke suppest: that New York booden lakes Octato and Eiie but Lake Huran does not share border with New York.
No.
GPT-4X 4V
Yes, New York the only U.S. state that shares barder with
ROSS-78
No,
dircle than the circle.
Figure 13: Qualitative comparisons on HallusionBench (Guan et al., 2024).
benchmarks that mainly require general knowledge following (Tong al., 2024a), including Sci- enceQA (Saikh et al., 2022), MMMU (Yue et al., 2024), and A12D (Hiippala et al., 2021). Fur- thermore, incorporate representative language benchmarks, including general understanding MMLU (Hendrycks et al., 2020) and HellaSwag (Zellers et al., 2019), and instruction- follloiig on IFEval (Zhou et al., 2023). Empirical results Table 14 demonstrate that Ross does not harm language capabilities as it brings improvements in most cases.
Model Scaling Properties. To study the stability and scalability of RoSs across different model sizes, we use the Qwen2.5 series (Team, 2024) with varying sizes as the base language model while 336 (Radford et al., 2021) as the visual encoder. The pre-training data is LLaVA-558K (Liu et al., 2023a), and the instruction tuning data is LLaVA-666 (Liu et al., 2024a). The results, shown in Table 15, demonstrate that Ross brings improvements over the baseline (LLaVA) across different model sizes in most cases.
keeping the
Data Scaling Properties. To study the impact of the training data scale, we used Qwen2-7B Instruct (Yang et al., 2024a) as the base language model and CLIP-ViT-L/140:30 336 (Radford et al., 2021) as the visual encoder. We compared the performance of Ross and the baseline under different scales of training data. Table 16 demonstrates that Ross consistently brings significant improvements as the training data scale increases.
Gradient Analysis. To better explain the reasoning behind how the vison-centric supervision enables the model to focus on relevant areas of the image during VQA tasks, we provide qualitative comparison using GradCAM (Selvaraju et al., 2020) on MMVP (Tong et al., 2024b) in Figure 12, since GradCAM helps in understanding which parts of the image the model is focusing on, making the model's decision-ni makinn process more transparent. In our analysis, visualize the gradient of
24

---

Published as a conference paper at ICLR 2025
hat the name place shown?
Image highest comtrast?
Italy Mediterranran Sna?
New Hampshire
Upper
East
Connectiout
B. Upper right
B. South
C. New York
C. Down left
C. W est
D. Rhude Island
D. Down right
D. North
A
LLaVA-vi. v1.57B
3
LLaVA v15-7B
Cembeian-1-88 1-8B
Cambeian-1-88
D
Cambrian- 1-BB
GPT-4V
New York
GPT-4V
Down right
GPT-4V
C. West
Figure 14: Qualitative comparisons on MMbench (Guan et al., 2024) English dev split.
From which angle this image taken?
Is the butterily's s abdomen visible in the image?
Is the lock locked or unlocked?
A. Front
A. Yes
A. Locked
B. Side
B. No
B. Unlocked
L1aVA-71.5- --BBBB B
LLaVA-v1.5-78
A
LLaVA-vi
A
Cambrian-
B
Cambrian 1-8B
A
Cambrian: I-BB
A
ROsS-7B
Ross-7B
B
Ross-7B
Are the ears of the dog erect or drooping?
In this image, how eyes can you the animal?
Can you see the side windows of the vehicles? A. Yes
Erect B. Drooping
A. One
B. Two
B. No
LLaVA-v1.5 --1.57B
A
LLaVA-v1.5-78
B
LLaVA-vi 5-78
A
Cambrian: -BB
Cambrian
Cambrian 1-BB
Ross-7B
B
ROSS-7B
A
Ross-7B
B
Figure 15: Qualitative comparisons on MMVP (Tong et al., 2024b).
the second-to-last block of the LMM, regarding the option of the ground-truth answer as the target class. Specifically in this case, where the providing question is about the spider web, proposed vision-centric supervision signals provide more reasonable gradients and urge LMMs to focus on relevant regions, i.e., the spider web, the training evolves.
C.5 QUALITATIVE COMPARISONS
We provide sufficient qualitative comparisons in Figure 13, Figure 14, Figure 15, and Figure 16 on HallusionBench (Guan et al., 2024), MMBench (Liu al., 2023b) English dev split, MMVP (Tong et al., 2024b), and SpatialBench (Cai et al., 2024), respectively. In Figure 13, Figure 14, and Figure 15, we compare our Ross-7B with the instruction tuning baseline LLaVA-v1..57BB (Liu et al., 2024a), the state-of-the-art open-source method using extrinsic assistance Cambrian-1-88 (Tong et al., 2024a), and GPT-4V (OpenAI, 2023a).
As demonstrated in Figure 13, where we highlight the wrong parts of each prediction in red, our Ross manages to correctly answer the question with reduced hallucinations even when GPT-4V fails. Cambrian-1 (Tong al., 2024a) even fails to follow the instructions in the second example. This could be because a super huge SFT data (7M) may harm the iestructiionfollowwee
abilities of LMMs. Qualitative results shown in Figure 14 demonstrate both enhanced reasoning abilities (the first example), low-level comprehension capabilities (the second example), and spatial understanding skills (the third example). Figure 15 illustrates that Ross is good recognizing various visual
25

---

Published as a conference paper at ICLR 2025
In real world, which smaller size?
W hat the positional relationship between the group of people with the flag and the black car?
A. The dog closer to the camera.
B. The dog further to the camera.
A. Behind the black car.
C. They srem to be equally large.
B. ell the black car.
D. can not
image becamse
Right
information given enough.
In front black
elephant?
LLMA
LLAW
LLaVA
Yes.
11AM (we MITXE)
1TAA ow MITUS)
1V owt MIDeS)
Yos
ROSS
ROSS
ROSS
No.
Figure 16: Qualitative comparisons on SpatialBench (Cai al., 2024), We take RGB inputs when testing. Notably, the extra depth expert MiDaS-3.0 (Birkl et al., 2023) sometimes harms comprehension (see the second example).
patterns, implying that the introduced reconstructive vision-centric objective indeed makes up the visual shortcomings of the original visual encoder.
Figure 16 provides qualitative results SpatialBench (Cai al., 2024). The extra depth understand- ing visual expert, i.e., MiDaS (Birkl et al., 2023), fails to help LMMs understand depth maps both quantitatively in Table 5 and qualitatively in Figure 16.
D DISCUSSION
One limitation is that Ross does not have generation capabilities, since Ross designed for enhanced multimodal comprehension, without the need generate photorealistic aesthetic images. Furthermore, the gap training data between comprehension and generation methods also matters. For instance, PixArt-a (Chen et al., 2023a), which is one of the most efficient text-to-image models, was trained on nearly 400M images to model the pixel discrepancy just in the first training stage. By contrast, our Ross only trained on nearly 3M images for one epoch. Future topics include achieving photorealistic text-to- tomee generation via incorporating more training samples.
26


Reviews:
[{"review_id": "trVbgnTC6k", "reviewer": "Reviewer_kvNb", "summary": "The paper proposes a new LMM training approach with an additional branch for input image reconstruction. The results show the model enhances fine-grained comprehension and reduces hallucinations. ROSS employs a denoising objective to address spatial redundancy that reconstructs latent visual tokens rather than raw RGB values. Empirical evaluations demonstrate that ROSS consistently outperforms conventional LMMs using single or multiple visual encoders on visual understanding benchmarks.", "questions": "It would be cool if authors could address the concerns in weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Novel Image-Based Supervision**: ROSS leverages image reconstruction as a supervisory signal, enabling the model to capture fine-grained visual features and semantics that significantly reduce hallucination artifacts compared to text-supervised approaches. The idea conceptually makes sense and is proven in the experiments.\n\n**Comprehensive Analysis of Model Variants**: The paper provides a thorough study of various architectural choices and configurations within the ROSS framework, offering insight into optimal setups.\n\n**Empirical Validation**: Extensive ablation studies and benchmark evaluations demonstrate ROSS's superior performance metrics, particularly in tasks requiring high-fidelity visual understanding, with statistically significant improvements over state-of-the-art baselines.", "weaknesses": "- Potential Unfairness in Comparisons: While the paper includes an ablation study where variables like training data are controlled for fair comparison with other models, its main results table appears to use different datasets compared to competing methods. This inconsistency in data setup might lead to an unfair advantage for ROSS, making it difficult to assess the true comparative effectiveness of the approach against state-of-the-art methods.\n\n- Computational Overhead: The denoising process introduces extra computational overhead during training, but the paper does not quantify or discuss this cost, leaving readers uncertain about the practical trade-offs of using this approach.\n\n- Limited Analysis of Generation vs. Reconstruction Performance: The paper compares ROSS’s reconstructive approach to generative methods, noting that the generative approach underperforms in comprehension tasks. However, it lacks a thorough exploration of why the generative method yields lower performance. A more in-depth discussion of the limitations and differences between the two approaches would enhance understanding and help identify when reconstruction might be preferable to generation in multimodal tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1gcKuUeVxy", "reviewer": "Reviewer_MCFE", "summary": "The paper uses image denoising as an auxiliary training task to improve VLMs abilities.\nDenoising encourages the VLM to preserve image detail.\nThe work is motivated by the MAE (masked auto-encoder) line of work for training foundational vision encoders.\nThe auxiliary task helps VLMs achieve higher benchmark numbers.", "questions": "How will the results change at higher resolution and what would the comparison look like when compared against tile-based methods like LLava-1.6 or any of the more recent VLM work like Qwen2-VL etc ...?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Whereas text-based LLMs have achieved amazing results only with next-token prediction, when we have image + text VLMs, it has always seemed that only doing next-token prediction for text could be improved upon.  In that regard, the technique proposed in this paper, to use image denoising as a pretext task, seems like step forward, as a way to add more supervision to the VLM and to improve results. \n\nThe benefits to the metrics are actually significant in some cases, not just epsilon levels, which is great to see.\n\nIt seems to me like the method is described clearly and the results are presented clearly.", "weaknesses": "1. I wish the benchmarks cited in the paper to measure the benefits of their method, i wish those benchmarks more closely matched recent popular work such as \"The Llama 3 Herd of Models\" or \"Qwen2-VL\", which include benchmarks like TextVQA, DocVQA, etc ... It may not change the conclusion but when we compare methods, it's important to look at a representative distribution of benchmarks. Table 4 has some of these common benchmarks, but not all of them. Furthermore, i wish Table 4 (or perhaps Table 3) included the same benchmarks but also had a very clean A/B experiment that was a baseline without the method vs. using the method. Table 3 has this but Table 3 has a different set of benchmarks! So it's rather confusing what conclusion to draw.\n\n2. The fact that the work was done at lower image resolution also limits the impact of the work. While it may be a perfectly reasonably thing to study the problem initially at lower resolution, I believe that most people care about the models with the best metrics, and all of those methods today use image tiling to handle high resolution images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NBjii7n2yW", "reviewer": "Reviewer_SsLG", "summary": "This paper introduces ROSS, a novel approach to enhance Large Multimodal Models (LMMs) through vision-centric supervision signals. Unlike conventional visual instruction tuning that only supervises text outputs, ROSS introduces a reconstructive objective where LMMs must reconstruct input images' latent representations. \n\nThe authors address the challenge of spatial redundancy in visual signals by employing a denoising objective to reconstruct latent representations rather than raw RGB values. The approach demonstrates significant improvements across various benchmarks, particularly in fine-grained visual comprehension and hallucination reduction, while maintaining a lightweight architecture compared to existing methods that rely on multiple visual experts.", "questions": "This not be a major issue, but it's not very natural to see that using vision encoder to encode image pixels into LLM's embeddings and use another denosing module to reconstruct it back to image pixels. \n\nThis may introduce improvements since the model better maintains the information of original images, but it may be more natural to see it's used in a quantized tokenizer based LMM like EMU-3 and Chamelon.\n\nI was wondering how the authors feel about this and have insights on this question?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "This paper is very novel and address the very important topic on vision-centric learning in LMM.\n\nSpecifically, the paper introduces an innovative vision-centric supervision method that leverages the inherent richness of input images, addressing a clear gap in existing LMM training approaches. The use of denoising objectives for latent representation reconstruction is particularly clever as it handles the spatial redundancy problem.\n\nThe authors conduct extensive experiments across multiple benchmarks, including thorough ablation studies that systematically evaluate different components of their approach (regression vs. denoising, different tokenizers, architecture choices). The comparison with state-of-the-art methods is particularly thorough.\n\nROSS achieves competitive or superior performance using only a single visual encoder, making it more efficient than existing approaches that require multiple visual experts. This has significant practical implications for deployment and scalability.\n\nThe methodology is well-grounded in existing literature and builds thoughtfully on previous work in both vision and language domains. The authors clearly explain how their approach differs from both traditional visual instruction tuning and newer aggregated visual instruction tuning methods.", "weaknesses": "I think the major weakness is about the Computational Costs. While the paper emphasizes the efficiency of using a single visual encoder, it lacks detailed analysis of training time, memory requirements, and computational costs compared to baseline methods.\n\nBesides, the paper doesn't thoroughly discuss the sensitivity of ROSS to various hyperparameters, such as the denoising schedule or architecture choices. It would be benefitical to add this part analysis and show ROSS's denoising part is robust and easy to train.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ysBYbQ73cH", "reviewer": "Reviewer_tGMa", "summary": "This paper introduces Reconstructive Visual Instruction Tuning (ROSS), a novel approach that leverages input images as additional supervision signals to enhance fine-grained visual perception capabilities. Through extensive empirical studies, the authors explore optimal training settings, such as auxiliary module design and the nature of supervision signals. Experimental results indicate that ROSS consistently improves the performance of existing vision-language models.", "questions": "1. The results in Table 4 appear counter-intuitive, as ROSS-13B performs consistently worse than ROSS-7B. This raises concerns about whether the proposed method is well-suited to larger-scale models. Clarification on this disparity and potential scalability issues would strengthen the paper.\n2. The analysis in Section 5.2 is unclear. My understanding is that the authors aim to demonstrate that additional visual supervision enables the model to better focus on relevant areas of the image during VQA tasks. However, the reasoning behind this effect is not well-explained. Further elaboration on the mechanisms or evidence supporting this claim would enhance interpretability.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is well-organized and easy to follow, with a clear presentation of the key ideas. It is always inspiring to see that a simple auxiliary loss can improve the performance of large multimodal models.\n2. The authors have put a lot of effort into designing experiments that thoroughly ablate the proposed training methods. This rigor in experimentation is highly appreciated and adds to the paper’s credibility.", "weaknesses": "1. The proposed method lacks novelty and insight. As noted in Section 2, previous work, such as [1], has explored using images as additional supervision in the form of generative objectives. Consequently, the contribution of this paper is limited, focusing mainly on a variation of established methods.\n2. The proposed method lack interpretability. While using pixel-level reconstruction as an auxiliary task may enhance fine-grained image recognition, this approach risks impairing the language capabilities of the multimodal model, as the final task output is high-level semantic language. The authors provide insufficient experiments and explanations regarding this trade-off, leaving questions about potential impacts on language performance.\n3. The scalability of the empirical findings is uncertain. It remains unclear whether the optimal settings identified would hold in different scenarios or with variations in model scale, training data, and other factors. Although the authors attempt to address this concern with results in Table 3, these efforts are insufficient, as many relevant variables remain unexplored.\n\n[1] Sun Q, Cui Y, Zhang X, et al. Generative multimodal models are in-context learners[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024: 14398-14409.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WbGxfhxJIq", "reviewer": "Reviewer_rXNr", "summary": "This paper proposes a new variant of visual instruction tuning. Different from previous works that only utilize textual supervision, the proposed method additionally exploits visual supervision by reconstructing the contexts of input images. In particular, a denoising structure is introduced to better learn the latent representations. Experiments are conducted on several benchmarks.", "questions": "1. The novelty is somewhat incremental. Utilizing image supervision is a very straightforward idea, and can be implemented by various ways. Denoising-based reconstruction is a very well-developed strategy in the field of image diffusion/generation. There is no specific in-depth design in the proposed architecture.\n\n2. The authors claim that the proposed method capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. In which cases, the information within the image is important? The authors should provide a more detailed analysis of this aspect. Is this information crucial for all common cases?\n\n3. No experiments on complexity and efficiency. Of course, utilizing more self-supervision loss can definitely improve the model's performance. However, this image-aware training may cost more time and GPU memories compared to previous text-only supervision. The authors should provide an in-depth analysis of complexity and efficiency for fair comparison.\n\n4. Given an image-text pair input, just some of the image-based contents are aligned with the text. I do not see any text-guided image reconstruction design in the architecture. This may help reduce the redundancy contexts.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-organized and easy to read.\n2. The motivation is straightforward.", "weaknesses": "1. The novelty is somewhat incremental. Utilizing image supervision is a very straightforward idea, and can be implemented by various ways. Denoising-based reconstruction is a very well-developed strategy in the field of image diffusion/generation. There is no specific in-depth design in the proposed architecture.\n\n2. The authors claim that the proposed method capitalizes on the inherent richness and detail present within input images themselves, which are often lost in pure text supervision. In which cases, the information within the image is important? The authors should provide a more detailed analysis of this aspect. Is this information crucial for all common cases?\n\n3. No experiments on complexity and efficiency. Of course, utilizing more self-supervision loss can definitely improve the model's performance. However, this image-aware training may cost more time and GPU memories compared to previous text-only supervision. The authors should provide an in-depth analysis of complexity and efficiency for fair comparison.\n\n4. Given an image-text pair input, just some of the image-based contents are aligned with the text. I do not see any text-guided image reconstruction design in the architecture. This may help reduce the redundancy contexts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)