LLaVA-OneVision: Easy Visual Task Transfer
Authors: Bo Li
Yuanhan Zhang
Dong Guo
Renrui Zhang
Feng Li
Hao Zhang
Kaichen Zhang
Peiyuan Zhang
Yanwei Li
Ziwei Liu
Chunyuan Li
Venue: Accepted by TMLR ()

Abstract: We present LLaVA-OneVision, a family of open large multimodal models (LMMs) developed by consolidating our insights into data, models, and visual representations in the LLaVA-NeXT blog series. Our experimental results demonstrate that LLaVA-OneVision is the first single model that can simultaneously push the performance boundaries of open LMMs in three important computer vision scenarios: single-image, multi-image, and video scenarios. Importantly, the design of LLaVA-OneVision allows strong transfer learning across different modalities/scenarios, yielding new emerging capabilities. In particular, strong video understanding and cross-scenario capabilities are demonstrated through task transfer from images to videos.

Hao Zhang Published in Transactions on Machine Learning Research (02/2025)
LLavA-OneVision: Easy Visual Task Transfer
Bo Li
libo0013@e.nbu.ccm edu.sg
S-Lab, Nanyang Technological University
Yuanhan Zhang
jumnhan//288.nntu.ccck
S-Lab, Nanyang Technological University
Dong Guo
Tnngguo.ddnua@@gllddae.o com
Byte
Renrui Zhang
wwwrushannyBBikk..ul..
Chinese University of Hong Kong
Feng Li
flaayfBcorneed.us....kk
Hong Kong University of Science and Technology
heshangez&cconnccc.uss.u
Hong Kong University of Science and Technology
Kaichen Zhang
zhan0564 Ge.nbu. sg
S-Lab, Nanyong Technological University
Peiyuan Zhang S-Leb, Nanyang Technological University
peiguan.zhange Onthannngnnnnn.mm edu..ssddu.s
Yanwei Li ByleDance
liganwei@linkkcomm culik. edu.hk
Ziwei Liu S-Lab, Nanyang Technological University
Liveitlu@@ttu.ccm www.inntneedu.... edu.sg
Chunyuan Li Bylce
lichurguun224 @graail. com
Reviewed on OpenReview: https: // openreview. net/ forun? id= ZNu8qULVV6
Abstract
We present LLaVA-OneVisson,, family of open large multimodal models (LMMs) devel- oped by consolidating our insights into data, models, and visual representations in the V.ANN-T blog series. Our experimental results demonstrate that L.LVA-OOeVisioon the first single model that can simultancously push the performance boundarics of open LMMs in three important computer vision scenarios: single-image, multi-image, and video scenarios. Importantly, the design of LLaVA-OneVision allows strong transfer learning across different modalities/sccnarte yiclding new emerging capabilities. In particular, strong vidco understanding and cross-scenario capabilities are demonstrated through task transfer from images to videos.
1

---

Published in Transactions on Machine Learning Research (02/2025)
1 Introduction
It is a core aspiration in AT to build general-purpose assistants with Large Multimodal Models (LMM) (Li et al., 2024c). LLaVA-OneVision is an open model, contimuing to advance the line of research in building large vision-and-lann -aiin-ann-languugge assistant (LLaVA) (Liu et al., 2023c) that can follow diverse instructions to complete a variety of computer vision tasks in the wild. As a cost-efficient recipe, it is typically developed by connecting vision encoders with large language models (T.LM) using simple connection module.
The first LLaVA model (Liu et al., 2023c) demonstrates impressive multimodal chat abilities, sometines exhibiting the behaviors similar to GPT-4V on previously unseen images and instructions for the first time. LLaVA-1.5 (Liu et al., 2024a) significantly expands and improves the capabilities by incorporating more academic-related instruction data, achieving SoTA performance on a dozens of benchmarks with a data-efficient recipe. LLaVA-NeXT (Liu et al., 2024b) inherits this properly, further pushing performance boundaries through three key techniques: AnyRes for handling high-resolution images, expanding tigh-quality instruction data, and utilizing the best open LLM available at the time.
LLa VA-NeTT provides an extendable and scalable protolype, which facilitates several parallel explorations, reported in the LLaVA-NeXT blog series (Liu et al., 2024b; Zhang et al., 2024h; Li et al., 2024b;a;d):
The Video blog (Zhang et al., 2024h) shows that the image-only-traanee LLaVA-NeXT model is surprisingly strong on video tasks with zero-shot modality transfer, due to the design of AnyRes digest any vision signals as sequence of images.
The Stronger blog (Li et al., 2024b) demonstrates the LLM model scaling succuss of this cost-efficient strategy. By simply scaling up the LLM, it achieves performance comparable to GPT-4V on selected benchmarks.
The Ablation blog (Li et al., 2024a) summarizes our empirical exploration except the visual instruction data itself, including the choice of architectures (scaling of LLM & vision encoder), visual representations (resolution & #tokens), as well as training strategies (trainable modules & high- data) in the pursuit of data scaling success.
The Interleave blog (Li et al., 2024d) describes the strategies to extend and improve the capability in new scenarios including multi-image, multi-frame (video) and multi-view (3D), while maintaining the singlo-image performance.
Motivated by these explorations, we aim to offer useful insights and develop the next state-of-the-art level model to handle versatile visual inputs. During the process, we have also been accumulating and curating a large collection of the high-quality datasets. By consolidating these insights, we introduce LLLVA-OneVisioe implement the new model with the available compute, without extensively de-risking individual components. This leaves room for further improvements in capabilities through additional data and model scaling following our recipe. In particular, our paper makes the following contributions:
We
Large multimodal models. We develop LLaVA-OneVision family of open large multimodal models (LMMs) that improves the performance boundarics of open LMMs in three important vision settings, including single-image, multi-image, and video scenarios.
Emerging Capabilities with Task Transfer. Our design in modeling and data representations allow task transfer across different scenarios, suggesting simple approach to yield new emgerging capabilities. In particular, LLLVV--OeVisse demonstrate strong video understanding through task transfer from images.
Open-source. To pave the way towards building general-purpose visual assistant, release the following assets to the public: the generated multimodal instruction data, the codebase, the model checkpoints, and a visual chat demo.
2 Related Work
The SoTA proprictary LMMs, such as GPT-4V (OpenAI, 2023), GPT-4o (OpenAI, 2024), Gemini (Team et al., 2023) and Claude-3.5 (Anthropic, 2024), exhibit excellent performance in versertile vision scenarios, including single-image, multi-image and video settings. Tn the open research community, existing works
2

---

Published in Transactions on Machine Learning Research (02/2025)
typically develop models tailored to each individual scenario separately. Specifically, most focus pushing the performance limits in single-image scenarios (Dai et al., 2024; Liu et al., 2023c; Zhu et al., 2023; Li et al., 2024g; Zhang et al., 2023a; Guo et al., 2023), only a few recent papers have begun to explore multi-image scenarios (Li et al., 2023a; Jiang ct al., 2024). While video LMMs excel in video understanding, they often do so at the expense of image performance (Li et al., 2024f; Lin et al., 2023). is rare to have a single open model that reports excellent performance in all three scenarios. L.LVA-Onne VA-OneVision aims to fill this gap by demonstrating state-of-the-art performance across a broad range of tasks, and showcasing interesting emerging capabilities through cross-scenario task transfer and composition.
To the best of our knowledge, LLaVA-NeTT- performance in all three scenarios, LLa Vl-OOneViisio inherits its training recipe and data for improved performance. Other versatial open LMMs with potentials to excel include VILA (Lin et al., 2024), InternLM- XComposer-2.55 (Zhang et al., 2024c). Unfortunately, their results are not fully evaluated and reported; we compare with them in the experiments. In addition to building systems with versatial capabilities, LLaVA- OneVision is benefited from large-scale high-quality data training, including model-synthesine and the new collection of diverse instruction tuning data. For the former, we inherit all the knowledge learning data in (Li et al., 2024a). For the latter, our are motivated by FLAN (Wei et al., 2021; Longpre et al., 2023; Xu et al., 2024c). The data collection process is con-current with Idefics2 (Laurençon et al., 2024) and Cambrian-1 (Tong et al., 2024), but we focus on a smaller but more carefully curated collection of datasets. A similar conclusion is observed: a large amount of visual instruction tuning data can significantly improve performance. For comprchensive investigations on design choices of LMMs, we refer to several recent studies (Karamcheti et al., 2024; Laurengon et al., 2024; Li et al., 2024a; McKinzie et al., 2024; Tong et al., 2024; Beyer et al., 2024).
(Li et al., 2024d) is the first attempt to report good
knowledge
3 Modeling
3.1
Network Architecture
The model architecture inherits the minimalism design of LLaVA scries, whose primary goals are (i) effectively leverage the pre-trained capabilities of both the LLM and visual model, as well as (ti) facilitate strong scaling behavior in terms of both data and model. The network archtecture is illustrated in Figure 1.
LLM. We choose Qwen-2 (Yang et al., 2024) as our LLM fa(-) parameterized by Φ, as it offers various model size and exhibits strong language capabilities to date among publicly available checkpoints.
Vision Encoder. We consider the SigLIP (Zhai et al., 2023) as the visual encoder (-)()) parameterized by bb, encoding an input image X, into its visual feature Z, = 9(X.). The grid features before and after the last Transformer layer are considered in our experiments.
Projector. We consider a 2-layer MLP (Liu et al., 2024a) pe(.) parameterized by 0, to project image features into the word embedding space, yielding sequence of visual tokens H, = p(Z.).
The model choice is based on our empirical insights in (Li et al., 2024b;a) that stronger LLM typically supercharge stronger multimodal capabilities in the wild, while SigLIP yields higher LMM performance among open vision encoders.
For a sequence of length L, we compute the probability of the target answers X, by:
XX. X,)
II p(. Xr, Xv, Xa Xz,ci);
where Xg and Xa are the instruction and answer tokens in all turns before the current prediction token Vi, respectively. For the conditionals in equation 1, we explicitly add X, to emphasize the fact that the visual signal is grounded for all answers. As explained in Section 3.2, the form of visual signal X, is general. The visual input fed into the vision encoder depends on the corresponding scenarios: the invidiual image crop in the single-image sequence, the invidiual image in a multi-image sequence and the invidiual frame in the video sequence, respectively.
3

---

Published in Transactions on Machine Learning Research (02/2025)
Language Response
X
Qwen-2 2
Language Model f
2-Layer MLP
Projection Po
Hy
A
AHq
Zy
SigLIP
Vision Encoder I
X
X
q
Visual Signal
Language Instruction
Single Image
Multi-Image
Video
Figure 1: LLa AAaVA-OneVision network architecture. Left: The current model instantiation; Right: the general
form of LLaVA architecture in (Liu et al., 2023c), but is extended to support more visual signals.
3.2 Visual Representations
The representation of visual signals is key the success of the visual encoding. relates to two factors, the resolution the raw pizel space and the number of tokens the feature space, leading to the visual input representation configuration (resolution, #token). The scaling of both factors leads improved performance, especially on tasks that require visual details. To strike balance of performance and cost, we observe that the scaling of resolution is more effective than that of token numbers, and recommend an AnyRes strategy with pooling. The comparison is illustrated in Figure 2.
For AnyRes with a configuration width a, height b, it divides the image into a X crops, each with the shape ba, Each crop has the same resolution suitable for the vision encoder. Assuming there are T tokens per crop, the total number of visual tokens is L (a + 1) x T, where the base image is resized before being fed into the vision encoder. We consider threshold т, and reduce the #token per crop, using bilinear interpolation if needed:
Lf> TT>
Tne
(axb+1)
T
LST
A of spatial configurations (a, is defined specify various methods for cropping images, thereby accommodating images of different resolutions and aspect ratios. Among them, the configuration that requires minimum number of crops is selected. Please see our detailed ablations of visual representation in (Li et al., 2024a).
The proposed Higher AnyRes strategy can serve as a flexible visual representation framework, adaptable for multi-image and video representation. The optimal configuration for performance and cost can be adjusted accordingly. We illustratic the configuration Figure 3, describe the detailed in Section A.1 and provide high-level encoding strategies below:
Single-image. We consider large maximum spatial configuration (a, for single-image representation to maintain the original image resolution without resizing. Additionally, we purposcfully allocate large number of visual lokens per image, resulting in a long sequence to effectively represent the visual signal. This is based on the observation that there is a larger number of high-quality training samples with diverse instructions for images compared vidcos. By representing an image with a long soquence that mimics video representation, we facilitate a smoother capability transfer from image video understanding (Zhang et al., 2024h; Li et al., 2024a).
4

---

Published in Transactions on Machine Learning Research (02/2025)
Bilinear Interpolation
split
encode
flatten
LLM
resize
encode
flatten
(a) Higher AnyRes with Bilinear Interpolation
resize & split
encode
flatten
LLM
resize
encode
flatten
(b) The original AnyRes
Figure 2: The visual representations. Top:
The new Higher AnyRes scheme with Bilinear Interpolation to
deal with images of higher resolution; Bottom: the original AnyRes in (Liu et al., 2024b).
Single-Image
N Crops
+ 91 7290 Tokens
729 729 Tokens
Multi Image
N Images
12 729 = 8748 Tokens
N 729 Tokens
Video
N Frames
32 196 6272 Tokens
196 Tokens
Example on Token Strategy
Max Tokens
Figure 3: The visual representation strategy to allocate tokens for each scenario in L.LaVA-OneVision. The maximum number of visual tokens across different scenarios is designed to similar, ensuring balanced visual representations to accommodate cross-scenario capability transfer. Note that 729 the #tokens for SigLIP to encode a visual input of resolustion 384x384.
Multi-image. Only the base image resolution considered and fed into the vision encoder obtain feature maps, eliminating the need for multi-crop high resolution image and thus saving computational resources (Ti et al., 2024d).

---

Published in Transactions on Machine Learning Research (02/2025)
Video. Each frame of the video is resized to the base image resolution and processed by the vision encoder to generate feature maps. Bilinear interpolation is employed to reduce the number of tokens, allowing the consideration of larger number of frames by reducing tokens per frame. Empirical evidence suggests this provides a better tradc-off between performance and computational cost (Zhang et al., 2024h).
These representation configurations are designed for capability transfer with a fixed compute budget in our experiments. With increased computational resources, the number of tokens per image frame can be increased during both training and inference stages to boost performance.
4
Data
In the realm of multimodal training from LLM, the axiom "quality over quantity" especially true. This principle paramount due to the extensive knowledge stored within pre-trained LLMs and Vision Transformers (ViTs). While it is essential accumulate balanced, diverse, and high-quality instruction data by the end of the LMM's training lifecycle, high-quality data for further knowledge acquisition whenever it is available. In this section, discuss the data sources and strategics for high-quality knowledge learning and visual instruction tuning.
often-overlooked aspect is the continuous exposure of the model to new,
4.1 High-Qualin Knowledge
The web-scale public image-text data is often of low-quality, rendering the data scaling of multimodal pre-training less efficient. Instead, we recommend to focus on high-quality knowledge learning, given a limited compute budget. This approach acknowledges that the pre-trained LLMs and ViIs already possess a substantial knowledge base, and the goal to refine and enhance this knowledge with carcfully curated data. By prioritizing the quality of data, we can maximize compute efficiency.
We consider data from three major categories for high-quality knowledge learning:
Re- Captioned Detailed Description Data. LLaVA-NeXT-3al (Liu et al., 2024b) is known for its strong detailed caption ability among open-source LMMs. We used the model to generate new captions for the images from the following datasets: COCO118K, BLIP558K, and CC3M. We combined them to form the Ce-Caatio Detailed Description Data, totaling 3.5M samples. This can be viewed as an simple attempt of self improvemnt AI, where the training data is generated by an early version of the model itself.
Document / OCR Dala. We utilized the Text Reading subset from the UReader dataset, totaling 100K, which is easily accessible through PDF rendering. We used this text reading data along with the SynDOG EN/CN, to form the Document / OCR Data, totaling 1.1M samples.
Chinese and Language Data. We used the original ShareGPT4V (Chen et al., 2023a) images and utilized GPT-4V provided by the Azure API to generate 92K detailed Chinese caption data, aiming to improve the model's capability in Chinese. Since we used a large portion of detailed caption data, we also aim lo balance the model's language understanding ability. We collected 143K samples from the Evo-Instruct dataset (Chen ct al., 2024a).
It is interesting to note that almost all (accounting for 99.8%) of the high-quality knowledge data is synthetic. This is due to the high cost and copyright constraints associated with collecting large-scale, high-quality data in the wild. In contrast, synthetic data can be casily scaled. We believe that learning from large-scale synthetic data is becoming trend as AI models continue to grow more powerful.
4.2 Visual Instruction Tuning Data
Visual instruction tuning (Liu et al., 2023c) refers to the capability of an LMM to understand and act upon visual instructions. These instructions can be the form of language, combined with visual media such as images and vidcos, which the LMM processes and follows to perform a task or provide a response. This involves integrating visual understanding with natural language processing to interpret the instructions and execute the required responses.
6

---

Published in Transactions on Machine Learning Research (02/2025)
Data Collection and Curation. As demosntrated in previous works (Liu al., 2024a; Tong et al., 2024; Laurengon et al., 2024), visual instruction tuning data. is crutial for LMM capaiblity. Therefore, maintaining a high-quality dataset collection is crucial and beneficial to the community. We started to collect large pool of instruction tuning datasets from various original sources, with an unbalanced data ratio among categorics. Additionally, we utilize a few new subsets from the Cauldron (Laurengon el al., 2024) and Cambrian (Tong et al., 2024) dataset collections.
We categorize the data based on threc-level hierachy: vision, instruction, and response.
Vision Input. Three vision scenarios are considered, depding which visual input is considered in the multimodal sequence, including single-image, multi-image, video.
Language Instruction. The instructions, which often appears as questions, define the tasks to perform to deal with the visual imput. We classify the data into five major categorics: General QA, General OCR, Pee/Chart/Scrennt Math Reasoning, and Language. These instructions define the skill sets that trained LMM could cover. We use task categorization to help maintain and balance the skill distribution.
Language Response. The answer not only responds the user request, but also specifies the model behavior. It can be broadly categorized into free-form and fixed-form.
Free-form data is typically annotated by advanced models like GPT-4V/o /o and Gemini, while fixed-form data is derived from academic datasels, e.g. VQAv2, GQA, Visual Genome. For free-form data, we keep the original answers. However, for fixed-form data, we manually review the content and make necessary corrections to the question and answer formats. Wc adhore to the LLaVA-1.5 prompting strategy for multiplo-choice data, short answer data, and specific task data (e.g., OCR). This step is crucial for guiding the model's behavior to correctly balance QA performance, conversational ability, and reasoning skills in more complicated tasks, as well as preventing potential conflicts from different data sources. We list the full details about each dataset. in our collection, and their categorization and formatting prompt in Appendix C.3.
We divide the instruction data into two separate groups: one for single-image scenario and the other for all vision scenarios. This division is based on insights from our earlier studies (Li et al., 2024d; Zhang et al., 2024h), which highlight the relationship between image and video models: a stronger image model can better transfer to multi-image and video tasks. Additionally, the quantity and quality of training datasets available for single images are significantly higher than those for vidcos and multi-image tasks.
Single-Image Data. Since single-image data is crucial for multimodal capabilities, we explicitly compile large single-image data collection for model learning. We select from collected data sources to form a balanced collection, resulting in a total of 3.2 million samples. The overall distribution of single-image data is shown in Figure 4, with detailed information and the roadmap of data collection presented in Appendix C.I.
KEAAWW lack (PSOK)
AOKVVa (06.9 X)
Cambrian (Mloned) (8%) K)
CLIVII (1.7 K)
CuCO Capetos (20.0 K)
Hatehul Stemes (6.5 K)
IccaGA (2.5 K)
Eunge Tistcaliation 196 6 X]
1L (164 X)
Will (scaia) (64.5 XI
LLAVAR (20.0 R)
OXVQA (8.0
Recoodo (50.6 X)
SciesosiAA (6.0
SbazeGPP44 (57.3
Sha..PPTa 4V
97 (14.2
Tallura (9-0
Visios PLAN K)
Genoral
The Remeen
XPD (CPPTV) (4.9 K)
AISDD (IoteroMl)) (19.4 I;:
XPD (Crigine))) (6.9
Chart2/lat (2T.6 K)
ChartiA (15.3 K]
Disgraiz Emagse (0.3 K)
Dec-yQA (36.2 (0)
DVQA (30.0 X)
PiuurQAA [1.0 R)
RETTh (2.5 K)
Enlographic VIJA (4.1 X)
Single-image 322
LRV Chart 21.5
RaBl'T SQA (9.5 KG)
RaDUT WikEiIU (F5.0
RabLTT WTQ (38.2
SreenddWWarrh (10.7
TGA (1.4 K)
URosodr Caption (01.4
URoedlo FP K)
(37.4
MAVIC MCollen
MAVIS Data Begine (100.0 K)
Gec17OK QA (67 K)
Langoage Moth Brasernin mon
mkykk (K.1 K)
GR06 20. K)
Gremekyydk (Mook) (6-7 K)
GoiMMoo (Ma Waack)) (9.
GeoigA+ (baacay2000)) (17.2 X]
MapilA (MMx12V300K) (5.2 K)
CLaVER-Aath (5.3 K)
GecI7ok Allinn (60.5 X)
MailiqA (20.6 10)
Super CLEVE (9.7 1G)
TabMR (46.2 R)
Uniiieo (13.0 X)
GQA (72.1 Kg
LaV Normal (30.5
RAYEN (2.1 K)
Vieual Genone (96.43)
(10.0
SyathDog XEN (40.)
CastCape (21.9 X)
TaxsOR (20.1 K)
Langgagg (24.5%
Maaple Pro (13 MT) (130.0 K
Maggle Pre (6.3 5T) (130.0 RN
Maggle Pro (Quanz ST) (150.0 X)
Figure 4: Single-Image 3.2M. A High- Quality Stugle-Image Dataset Collection. Left: Data Distribution within Each Category. The outer circle shows the distribution of all data categories and the inner circle shows the distribution of data subsets. Right: The detailed quantities of datasets.
OneVision Data. In addition to the single-image stage training, we further fine-tune the model using a mixture of video, image, and multi-image data. We introduce a total of 1.6 million mixed data samples,
7

---

Published in Transactions on Machine Learning Research (02/2025)
comprising 560K multi-image data from Li et al. (2024d), 350K videos collected in this project, and 800K single-image samples. Notably, in this stage, we do not introduce new single-image data but instead sample high-quality and balanced portions from the previous single-image data, as described in Li et al. (2024d). The data distribution and details are presented Figure 5, with additional information available in Appendix C.2.
Mappie (90:8)
Essage
ARC
Cheidzon (60.3)
BheeGaraay (21.0K)
AldaaA (21 (K)
Cambrria (fibared GPT4)) (28,8))
LLAVA-WWSd [taal]] (10-50)
EAJOX-GPEAY (8,00)
LLAVA-156R (7.0K)
Gec170X--QA (5.4K)
GeolTi8--lless (5.9K)
ShaceGa"744 (5,7K)
TabarWE (4.5K)
Single Image
LLAVAR GPT4 (L.0K)
MayAQA (L)R))
Mustia (5:8))
OCR (CIPTVV) (2.3R)
CapeChpe (3.2K)
SctenceigA (.9K)
Piguret2A (1.8K)
GeotZA4 (1 7K)
A12D (lateraNC)) (1.22)
Unitieo 203)
ScoociA (L.)))
ERV-Normal (ftieeed)
Geometty33
Supen-CLEEVI (9.4K)
Vilter
(0.8K)
(0:28)
OneVision
VQA-RAD (0.20)
Lott-Chart
1.6M Mull--mage
Malte Toong (4.00)
NLVR (46 (K)
Cur Ensteret (50.
SemslKoi (40 983
RAVEN (35.08)
IoonUA (34-08)
VIST (20
HeandaA (25.0K)
Costraativeaptiie (25
ALPRED (22.49)
PilciatoowSY (22,3K))
EsaageCluds (18 (I)
Spon-tae-DiE
ReccpeQA--n Innnng-e TK)
TQA (8-2K)
ASSOP DK)
HQg-Biir-DDit (7.0K)
MagicBeushbotet (6.75)
CoMMttS-Disloge (5 (3-96C)
MutriVQA (3.0K)
ViaWis (3D)) (4J9K)
ClaVVa-Uaagge (3.K)
Nestiqa (3.50)
LEdtt (3.5K]
Staee (3.0K)
VQLA (M4 (L.93)
MFF CoppettyClllnnnn Colienacoo (1.9K)
MTT Caacaaaameoo (1 1.99
OCTR-VIQA (MT) (1.9K)
Activity S80
Charnder (23.63))
Eqpet (O.nK))
Figure 5: OneVision 1.6M. A high-quality single-image, multi-image and video dataset collection. Left: Data Distribution within each category. The outer circle shows the distribution of all data categories and the inner circle shows the distribution of data subsets. Right: The detailed quantities of datasets. "MI" means the multi-image version dataset proposed by DEMON (Li et al., 2021e).
Stage-1
Stage-1.5
Stage-2
High-Quality Knowledge Learning
: Language-lmage Alignment
E
T
Visual Instruction Tuning
Stage-1
Stage-1.5
Stage-2
Single-Image
OneVision
Resolution
384
(2Kx (x(2,)) (2.3)x1)
384x((1x1); -.. [6x6])
381x(( (6x6))
# ToToeess
729
Max 729x5
Max 729x10
Max 729x10 (See Fig. 3)
Dataset
LCS
Image (Sec. 4.1)
Image (Sec. 4.2)
(Multi)-Image Video (Sec. 4.2)
#Samples
558K
4M
3.2M
166
Trainable
Projector
Full Model
Full Model
Full Model
Model
U.5B LLM
1.8M
0.8B
0.8B
0.8B
7.6B LLM
20.0M
8.0B
8.0B
8.0B
72.7B LLM
72.0M
73.2B
73.2B
73.2B
Batch Size
512
256/512
256/512
256/512
Troining
1x1u-8
x10-6
x10-6
x10-6
LR: (pproj: ØLLM)
1x10-3
x10-5
1 x10-5
x10-5
Epoch
Table 1: Detailed configuration for each training stage of the LLaVA-OnoVision model. The table outlines the progression of vision parameters, dataset characteristics, model specifications, and training hyperparameters across different stages of the curriculum learning process, We use global batch size of 512 for the 0.5B model, and 256 for the 7B and 72B models.
5 Training Strategies
To enable LLM for multimodal capabilities, we identify three critical functionalities, and systematically divide them into three distinct learning stages for the purpose of ablation studies. As with most existing
8

---

Published in Transactions on Machine Learning Research (02/2025)
research, prior LLaVA models mainly explore the single-image instruction tuning. However, other parts are less frequently investigated and therefore constitute the primary focus of this section.
We train the model via a curriculum learning principle, where training objectives and examples of increasing difficulty are observed in a stage-wise manner. With a fixed compute budget, this strategy helps decompose the training process and produces immediate checkpoints that can be re-used in more experiment trails.
Stage-1: Longuoge-Image Alignment. The goal is to well align the visual features into the word embedding space of LLMs.
Stage-1.5: High- Quality Knowledge Learning. To strike a balance between compute--fiicinncy and injecting new knowledge into LMMs, we recommend to consider the high-quality knowledge for LMM learning. The training configuration mirrors the settings used in Stage-2, ensuring consistency and allowing the model to integrate new information scamlessly.
Stage-2: Visual Instruction Tuning. To teach LMM to solve a diverse set of visual task with preferred responces, we organize the instruction data into different groups, described in Section 4.2. The model is scheduled to train on these groups order.
Specifically, the visual instruction tuning process consists of two phases: (i) Single-Image Training: The model is first trained on 3.2 million single-image instructions, resulting in model with strong performance in following a diverse set of instructions to complete visual tasks using a single image. (ii) The model is then trained on a mixture of video, single-image, and multi-image image data. In this phase, the model expands its capabilitics from single-image sconarios to diverse scenarios. It learns to follow instructions to complete lasks in each new scenario and transfer the learned knowledge across different scenarios, resulting in new emergent capabilities. Note that the proposed OneVision training the post-training stage is probably the simplest and most cost-efficient way to empower the LMMs with the multi-image and video understanding capabilities.
Training:
The training strategy is summarized in Table 1. We progressively train the model to deal with long sequence training. The maximum image resolution and the number of visual tokens gradually increase as training progresses. In Stage-1, the base image representation is considered with 729 tokens. In Stages 1.5 and 2, AnyRes is considered with up to 5 times and 10 times more visual tokens, respectively. Regarding trainable modules, Stago-1 updates only the projector, while the subsequent stages update the full model. It also noted that the learning rate for the vision encoder is 5 times smaller than that for the LLM.
6 Experimental Results
We conduct standardized and reproducible evaluations for LLAVA-OneVision models on all benchmarks using LMMs-Eval (Zhang et al., 2024b). For fair comparison with other leading LMMs, we primarily report results from original papers. When results are unavailable, we onboard the models in LMMs- and evaluate them using consistent settings. All our results are reported with greedy decoding and 0-shot settings unless otherwise specified.
To reveal the generality and effectiveness of the designed paradigm, we comprehensively evaluate our LLaVA- OneVision models across different modalities in Table 2, including single-image, multi-image, and video benchmarks. Detailed results for each modality are presented in Table 3, Table 4, and Table 5, respectively. We denote the the model checkpoint trained after the single-image stage and one-vision stage as LLa VA-OV (SI) or LLa VA-OV, respectively
Three model sizes are provided (0.5B, 7B and 72B), to accomodate applications with different performance- throughput trade-off, ranging from edge device to cloud serving. The GPT-4V and GPT-4o results are presented as references. Our largest model LLaVA-OneVisioon-225 GPT-4V and GPT-4o on most benchmarks. It suggests that the proposed recipe is effecitve, revealing a promising path for further scaling. However, a relatively larger gap remains in complex tasks such as visual chat scenarios, we leave it as future research in stronger LLMs, larger training data and better preference learning.
yiclds superior performance between
9

---

Published in Transactions on Machine Learning Research (02/2025)
6.1 Minle-Image Benchmarks
To validate the performance for single-image tasks in real-world scenories, we consider a comprehensive set of image benchmarks in Table 3. It can be categorized into three classes:
(I) Chart, Diagram, and Document Understanding. As the main visual formats for structured OCR data, we evaluate the results on AT2D (Kembhavi et al., 2016a), ChartQA (Masry et al., 2022), Doc DocVQA (Mathew et al., 2021), and InfoVQA (Mathew et al., 2022) benchmarks. Though current open-source models such as InternVL (Chen et al., 2023b) and Cambrian (Tong ct al., 2024) achieve performance comparable to commercial models, LLaVA-OneVise Visin goes a step further, surpassing GPT-4V (OpenAI, 2023) and approaching the performance level of GPT-4o (OpenAT, 2024).
(2) Perception and Multii-isciiple
Reasoning. Including visual perception scenarios, we reveal the potentials of our model for more complex and challenging reasoning tasks. Specifically, we adopt the perception benchmarks including MME (Yin et al., 2023), MMBench (Liu et al., 2023e), and MMVet (Yu et al., 2023), and reasoning benchmarks such as MathVerse (Zhang et al., 2024e), MathVista (Lu et al., 2023a), and MMMU (Yue et al., 2024). The results of LLaVA-OneVision significantly outperforms GPT-4V on various benchmarks, and comparable to GPT-40 on MathVista. This further confirms the superiority of our framework in visual perception and reasoning tasks.
(3) Real-world Understanding and Visual Chat. We consider the evaluation of LMMs as general-purpose assistant in the wild as the most important metrics, beyond the lab environments. To validate the capabilities in real-world scenarios, Eval (Padlewski et al., 2024b), MM-LiveBench (Zhang et. al., 2024b), and HlaVA-Bench-Wilde (Li et al., 2024b). While our model still has room for improvement compared to GPT-4V and GPT--0,, it achieves competitive performance with open-source models of similar parameter size. Notably, our model performs well on MM-LiveBennch (Zhang et al., 2024b), a benchmark for real-world internet content with constantly updated content, demonstrating the model's broad world knowledge and strong generalization abilities.
utilize several widely-adopted benchmarks, including RealworldQA (x.ai), Vibe-
6.2 Multi-Image Benchmarks
in multi-image interleaved settings, where users may ask questions between multiples images. In particular, we perform comprehensive assessment on the diverse subtasks of LLa VAatteeneeee Bench (Li et al., 2024d), such as Spot the Difference (Jhamtani & Berg- Kirkpatrick, 2018a), Image Edit Instruction (IEI) (Li et al., 2024d), Visual Storytelling (VST) (Huang el al., 2016), Text-rich VQA (TR-VQA) (Liu et al., 2023d), Multi-image VQA (MI-VQA) (Raj et al., 2021), Raven Puzzle (Chia et al., 2024), Q-Bench (QB) (Wu et al., 2023), and NLVR2 (Suhr et al., 2017)). We also utilize several multi-view benchmarks for evaluation, which depict 3D environments with multiple viewpoints, including 3D Dialogue (3D-Chat) and Task Decomposition (3D-TD) from 3D-LLM (Hong et al., 2023), ScanQA (Azuma et al., 2022a), ALFRED (Shridhar et al., 2020), and nuScenes VQA (Bansal et al., 2020). We refer to these datascts as in-domain evaluations, since our training data includes the training split of them.
We further evaluate
Moreover, we conduct evaluations on different out-domain tasks, which reveals the generalization capability of our approach. They include the multi-image split of math QA benchmark MathVerse (Zhang et al., 2024e) and science QA benchmark SeiVerse (Guo et al., 2024), multi-image perception benchmark BLINK (Fu et al., 2024c), MMMU-(mult--nmgge) (Yue et al., 2024) that contains all multi-image QA in MMMU, and MuirBench (Wang et al., 2024) spanning 12 diverse multi-inage tasks.
As shown in Table 4, benchmarks. After additional tuning on multi-image and video data, improvement over GPT-4V in specific areas, with significant margins. This highlights its strong performance in complex tasks such as multi-image reasoning, identifying differences, and understanding 3D envirouments. In addition, we observe consistent performance enhancement on after the one-vision training stage, which is more evident on multi-view benchmarks that are absent in single-image data. This demonstrates the significance of our one-vision paradigm for empowering LMMs with comprehensive visual capbalitics.
(SI) consistently outperforms existing multi-image LMMs in all
shows marked
10

---

Published in Transactions on Machine Learning Research (02/2025)
6.3 Video Benchmarks
Video is also a common modality to build world model, capluring the dynamic nature of the real world over time. We conduct experiments on several open-ended and multi-choice video benchmarks. These include ActivityNet-QA (Yu ct al., 2019) that contains human-annotated action-rolated QA pairs derived from ActivityNet dataset, EgoSchema (Mangalam et al., 2024) and MLVU (Zhou et al., 2024a) focusing on long video understanding, Perception (Päträucean et al.; 2023) designed to evaluate the perception skills, VideoMME (Fu et al., 2024b) and NeXTQQ (Xiao et al., 2021) containing diverse video domains and durations (from minutes to hours), VideoDetailCaption (LMMs-Lab, 2024) and 2023) for video detailed description and visua chat, respectively.
(Maaz et al.,
As shown in Table 5, LLaVA-OneVisioe achieves comparable or better results than previous open source models with much larger LLMs. The superiority of LLaVA-OneVision is particularly evident in complex benchmarks such as EgoSchema and VideoMME. Even compared to the advanced commercial model GPT-4V, LLaVA-OneVision performs competitively on the ActivityNet-QA, MLVU, and VidcoMME benchmarks.
Within the LLaVA-OV split, the smallest performance difference occurs in Perception Test, with a minimal improvement of 0.5 points when scaling the LLM from 0.5B to 7B. This contrasts with at least a 5-point improvement in other datasets. The modest gain at Perception Test suggests that LLaVA-OV's perception capabilities may mainly depend on its vision module, supporting findings from recent studies such as those by Qiao et al. (Qiao et al., 2024), which separate the roles of the image encoder and the LLM in perception and reasoning tasks. Notably, for datasets like EgoSchema that demand significant reasoning, a larger LLM substantially enhances performance.
Moreover, in comparing LLAVA-OV-BB (ST) with LLAVA-OV-7B, the smallest improvement is seen with ActivityNet-QAA This suggests that VA-OV-BB (SI), which is trained only images, can already perform well on this dataset. Delving into ActivityNet-QA QA, 4 it becomes apparent that many questions can be answered by observing just a single frame from the video. For instance, the question "What's the color of the ball?' can be answered throughout the video as the ball is visible from start to finish. This scenario does not require the model to understand the video sequence, allowing LLaVA-OV-7B (SI) to perform well.
7 Emerging Capabilities with Task Transfer
In addition to reporting the LLaVA-OneVisioon' capabilities across varions benchmarks, we also observe the emerging behaviors of the proposed model with task transfer and composition, paving promising way to generalize to tackle woaldworld computer vision tasks in the wild. We illustrate several emerging capabilities using examples as below.
S1: Joint understanding of diagram and chart (Transfer from single-image to multi-image) The capability to understand tables and charts are seperately learned from single image diagram and single-image chart understanding data, and the joint understanding task of table and chart do not appear in multi-image data. As shown in Table 6, ALaVA-OOeVision is capable of understanding and reasoning over the joint of diagram and chart.
S2: GUI for multi-modaa agent (Transfer from single-image and multi-image). GUIs and applying multinodal models to agentic tasks is of great value. In Table 7, recognizes the graphical user interface (GUT) screenshots of an iPhone and provides operational instructions to search for and open the Tik app. This task requires strong OCR capabilities learned from single-image scenarios and relational reasoning skills developed from multi-image scenarios. The example highlights LLaVA-OneVisioon's proficiency in GUI understanding and task exccution.
Understanding
S3: Set-of-mark Prompting (Transfer from single-image task composition). Different from existing open LLMs, LLaVA-OneVision demonstrates excellent set-of-marks (SoM) reasoning (Yang et al., 2023), an emerging capability shown in Table 8. To the best of our knowledge, this is the first time that open LMMs report good emerged SoM ability, as we observe that for many examples in (Yang et al., 2023). This task is not explicitly included in our training data, it is hypothsized that the ability is composed by visual referring and OCR.
is able to produce SoM reasoning
11

---

Published in Transactions on Machine Learning Research (02/2025)
S4: Image-to-Video Editing Instruction (Transfer from single-image and video). LLaVA- OneVision could generate detailed video creation prompis based static image in Table 9. Given an image and a target video, the model constructs a coherent and vivid narrative for the video, detailing elements such as characters, actions, background settings, and scene specifics. This task leverages both single-image analysis and video comprehension. It is hypothesized that this ability is generalized from the composition of single-image editing instruction task and video detailed description task.
S5: Video-to-Vides Difference (Transfer from multi-image and video). Understanding differences in images is common ability in recent. large multimodal models (LMMs), but our models extend this capability to videos. Table 10 showcases LLaVA-OneVisioe sequences with the same beginning frame but different endings. The model provides detailed comparison, describing characters, actions, and scene changes. In Table 11, LLa VL-OOOneeissoo describe the differences one by one between videos with a similar background but different main object in the foreground. This task leverages spot the difference in the multi-image analysis to generalize to video scenarios.
ability to analyze differences between two video
Sclf-driving (Transfer from single-image and multi- image to video). Understanding videos in a normal aspect ratio is straightforward, what about the videos with multi-views? In Table 12, observe that VA-Oneeeniioon could analyze and interprets multi-camera video footage from self-driving cars. Given video showing four camera views, the model describes each view in detail and plans the ego car's next move. This task combines multi-pand comprehension, video detailed description, and spatial-temporal reasoning.
S6: Multi-camera Vidco Understanding
S7: Composed Sub-video Understanding (Transfer from multi-image to video). Besides multi- view vidco, we see our model generalize to vertical videos with two sub-scenes. Table 13 demonstrates LLa VA-OneVissoon's 's ability to understand and describe the content and layout of composed sub-video. Given a vertical video with a series of frames featuring a consistent background and a person in the foreground, the model provides detailed analysis of visual elements, their arrangement, and the narrative context. This task requires single-image analysis, multi-image sequence comprehension, and contextual reasoning.
S8: Visual prompting in video (Task transfer from single-image to video). In Table 14, LLaVA- OneVision is able to understand the highlighed area with a semi-transparent circle in the video, and clearly see the number "10" on the back of the player. The capability of understanding visual prompts and OCR is a capablity of single-image LMMs. Our model displays the capablity of understanding visual prompts in videos, without training on video data with visual prompts.
S9: Visual Referring in Image in Video Understanding. The ability to refer to image query when answering questions about a video as shown in Table 15. This capbility is not seen in LLaV VA-NNTT or LLa VLaVV--Iterrleww this is proabably because strong base single-image training is required for such capabilty to appear.
8 Conclusions
LLa VlaV-nneiisionn is new, open LMM that shines when transferred to broad range of tasks in the scenarios of single-image, multi-image and videos. Our design allows new capabilities to emerge, through training multiple scenarios together and task transfer, eg, strong visual understanding ability from image to video. Our results demonstrate that LMMs trained with this open recipe and resources achieve statc-of-tho-art performance across various benchmarks. We also hope that LLnVA-OneVision serves as valuable starting point for the community to build specific applications, and develop stronger LMMs for diverse vision scenarios through further scaling.
12

---

Published in Transactions on Machine Learning Research (02/2025)
Capability
Benchmark
VLaVA OneVision- 0.5B
LLaVA OneVision- 7B
LLaVA OneVision- 72B
GPT-4V (V- Preview)
GPT-4o
FAI2D Science Diagrams
57.1%
81.4%
85.6%
78.2%
94.2%
Single-Image
[ChartQA
61.4%
80.0%
83.7%
78.5% %
85.7%
Chart Underetanding
tDoeVQA (test)
70.0%
87.5%
91.3%
88.1%
92.8%
Document Understanding
fInfoVQA (test)
41.8%
68.8%
74.9%
Infographic Understanding
MathVerse (vislon-mini)
17.9%
26.2%
39.1%
32.8%
50.2%
Professional Math Reasoning
MathVista (testmini)
34.8%
63.2%
67.5%
49.9%
63.8%
General Math Understanding
MMBench (en-dev)
52.1%
80.8%
85.9%
75.0%
Multi-discip
MME (cog-/perp.)
240/1 238
418/1580
579/1682
517/1409
Multi-discip
MMStar
37.3%
61.7%
66.1%
57.1%
Multi-discip
MMMU (val)
31.4%
48.8%
56.8%
56.8%
69.1%
College-level Multi-disp
MMVet Multi-discip
29.1%
57.5%
63.7%
49.9%
76.2%
SeedBench (image)
65.3%
73.4%
78.0%
49.9%
76.2%
Multi-discip; Large-seale
+Science
67.2%
96.0%
90.3%
75.7%
High-school Science
RealworldQA
55.6%
66.3%
71.9%
61.4%
Realworld Scenes
Vibe-Eval
33.8%
51.7%
50.7%
57.9%
63.1%
Chsallenging Cases
Benee (2406)
49.9%
77.1%
81.3%
92.4%
Internet Content
V.-VWd-ileer (small)
55.0%
67.8%
72.0%
81.0%
85.9%
Realworld Chat
LLa VL-Veenntereaave
33.3%
64.2% %%
79.9%
60.3%
Out-domain
MuirBench
25.3%
41.8%
54.8%
Comprehensive Multi-image
Multi-Image
Mantis
39.6%
64..2%
77.6%
62.7%
Multi-Image
Wild
BLINK
52.1%
48.2%
55.4%
Unusual Visual Scenarios
(Text-rich VQA OCR, Webpage, Ducument
65.0%
80.1%
83.7%
54.5%
Activity Ne
50.3%
56.6%
62.3%
57.0%
Spatio-Tempora Reasoning
Video
EgoSchema
26.8%
60.1%
62.0%
Egocentric Video
PercoptionTest
49.2%
57.1%
66.9%
Perception and Reasoning
SeedBench (vidca)
44.2%
56.9%
62.1%
60.5%
Multi-discip; Video
(val)
45.8%
56.3%
63.2%
60.7%
66.7%
Long Video
MLVU
50.3%
64.7%
68.0%
49.2%
64.6%
Long Video
45.5%
56.7%
59.4%
Multi-discip
VideoChatGPT
3.12
3.49
3.62
4.06
Video Conversation
VME
44.0%
58.2%
66.2%
59.9%
71.9%
Multi-discip
Table 2: Performance comparison to state-of-the-art commercial models with our LLaVA-OneVision models (0.5B to 72B parameters) across diverse evaluation benchmarks spanning multiple modalities. indicates that the training has been observed in our data mixture.
13

---

Published in Transactions on Machine Learning Research (02/2025)
A12D ChartQA DoeVQA InfoVQA
MathVerse
MathVista
MMBench
MME
MMMU
Model
test
teat
val/tese
valltestt
mini-vision
testmini
en-dev
test
val
Qwen VL. (Bai et al, 20236)
/93.1
Gemini-1.5- 1.5Prr (Team, 2024)
91.4
$7.2
-/93.1
-/81.0
63.9
62.2
Claude 3.5 Somnet (Authropie, 2024)
94.7
90.8
/95.2
49.7
67.7
68.3
GPT-1V (OpenAI, 2023)
78.2
- /88.1
19.9
75.0
517/1109
56.8
GPT 4o (OpenAI, 2024)
94.2
85.7
/92.8
50.2
63.8
69.1
Cambrian 34B (Tong al. 2024)
79.7
73.8
/75.5
53.2
81.4
49.7
VILA-34B (Lin al., 2024)
1762
51.9
IXC-2.5-78 (Zhang el al., 2024c)
81.5
82.2
/90.9
-/70.0
20.0
59.6
82.2
2229
42.9
InterVL-2--8I (Chen al, 2023b)
83.8
$3.3
-/91.6
-/71.8
27.5
58.3
81.7
49.3
InterVVL-2-268 (Chen el al., 20236)
84.5
84.9
-/92.9
-/7.99
31.3
59.4
83.4
2280
48.3
LLoVA-OV-0.JB (ST)
54.2
61.0
75.0/71.2
44.8/41.3
17.3
84.0
48.8
278/1217
91.2
LLVA--V-0.55 OV-0.5B
57.1
61.4
73.7/70.0
46.3/4188
17.9
34.8
52.1
240 /1238
31.4
LIa LLAVA-OV-2B (SI)
81.6
18.8
89.3/86.9
69. 89.9/65.3
26.9
56.1
81.7
488/ 188/1626
47.3
LLAVA-OV-RS OV-BB
81.4
90.2/87.5
70.7/68.8
418/1580
48.8
LL (SI)
85.1
84.9
08.3/91.8
77. 7/746
37.7
66.3
86.6
563/1700 1706
57.4
LLVVA-OV77BB
85.6
83.7
93.1/91.3
79.2/74.9
39.1
67.5
85.9
579/1682
56.8
MMVet MMStar S-Bench S-QA ImageDC MMLBench RealWorldQA Vibe-Eval LLaVA-W L-ilde
Model
test
test
image
test
2024-06
test
tess
small
Qwen-VI-Mar VL-Max [Bai et al,, 2023b)
(canini-1.5-Ppg (Team, 2024)
85.9
70.4
60.4
Clande 3.5 Sonnet (Anthropic, 2024)
75.4
02.3
59.9
66.2
102.9
83.1
GPT-1V (OpenAl, 2023)
19.9
57.1
49.9
75.7
91.5
61.4
57.9
98.0
$1.0
GPT-40 (OpeAAT, 2024)
76.2
76.2
92.5
58.6
63.1
106.
85.9
Cambrian-31B (Tong al, 2024)
85.6
67.8
VIIAA-34B (Tin al., 2024)
53.0
75.8
$1.3
LXC-2.5-7B (2hang ol., 2024c)
45.2
78.1
InteraVL-2-SU [Chen al, 20236)
60.0
59.4
76.0
97.0
87.1
73.4
64.4
46.7
81.5
62.5
IntemVL--2226B (Chen et al, 2023b)
65.4
60.4
76.8
97.5
91.0
77.2
66.8
51.5
99.6
70.2
LLVVA-OV-0.5 (91)
26.9
36.9
63.4
67.8
83.0
43.2
58.7
34.9
71.8
51.5
ILAVA-OV-0.5B
20.1
37.5
65.5
67.2
63.3
49.9
55.6
33.8
74.2
55.0
LLaVA-OV-28 (S1)
58.8
63.5
47.2
MaVA-OV-7R
57.5
61.7
75.4
51.7
90.7
ELY11-0V-28B (Sf)
60.0
65.9
77.6
91.9
91.3
84-4
78.8
$6.7
08.7
78.9
LLaVA-OV-72B
63.7
66.1
78.0
90.3
91.2
81.5
71.9
50.7
98..5
72.0
Table 3:
performance on single-image benchmarks. *GPT-4V reports 4-shot results on ChartQA. All results are reported as 0-shot accuracy.
14

---

Published in Transactions on Machine Learning Research (02/2025)
MathVerse
MuirBench
MI-VQA
NIVR2
Puzzle
Q-Bench
Spot. Diff
TR-VQA
3D-Chat
3D-TD
ScanQA
ALFRED
SuSeenes
BLINK
Mantie
SciVerse
VST
Model
in-domain mulii-mage
in-domain multi-view
out-domain
GPT-4V (OpenAl, 2023]
76.5
12.5
54.5
10.9
31.2
35.4
32.6
10.3
63.7
51.1
62.7
62.3
06.9
MlaVA-N-Image-7B (Läu al,, 2024b)
12.9
59.6
10.1
41.8
46.1
13.5
VPG-C-7B (Li et al,, 20123a)
15.2
46.8
73.2
2.4
57.6
27.8
38.9
21.5
43.1
52.4
24.3
23.1
Mantis7B (Jiang al. 2024)
11.2
52.5
87.4
25.7
69.9
17.6
45.2
12.5
2.60
14.7
16.1
14.0
46.2
46.4
59.5
27.2
36.1
29.3
LLaVA-N- (Li al., 2024d)
LLaVA-N- 14B (L. et al., 2024d)
24.5
95.0
91.1
59.9
76.7
40.5
78.6
33.3
70.6
52.2
34.5
62.0
76.7
52.1
66.4
33.4
40.7
32.7
MLVA--VV- (SI)
15.6
44.8
56.1
50.0
45.8
8.5
36.7
7.6
22.1
29.1
16.9
25.5
8.2
37.9
08.2
20.9
22.7
86.7
LLVVA-OV-.5BB
65.0
29.8
60.0
48.0
39.6
MaVA-OV-72 (81)
20.5
60.3
75.0
24.6
56.0
7.9
52.8
8.4
24.5
20.9
22.1
32.0
70.8
45.6
54-2
26.3
02.7
30.0
LLAVA-OV-7B
22.2
90.2
60.4
53.3
74.5
39.2
80.1
31.7
62.8
52.6
30.1
61.0
79.8
48.2
64.2
67.6
41.8
79.1
LLa LLAVA-OV-78B (SI)
44.3
95.4
21.9
48.5
46.0
56.8
LLAVA-OV-72B
22.5
95.3
93.8
63.4
83.2
43.3
83.7
31.5
63.2
53.3
35.8
66.3
78.8
53.4
77.6
91.6
518
91.9
Table 4: LLa VA- innoonnin performance multi-inage benchmarks with all results reported in accuracy. denotes the LLaVA-NeXTR M..VA-NeXT-Vieuaa--7 (2024-01). We use IEI for Image Edit Instruction, MI-VQA for Multi-image VQA, NLVR2 for Natural Language for Visual Reasoning, SDiff for Spot the Difference, VST for Visual Story Telling, TR-VQA for Text-rich VQA. For MathVerse and SciVerse, we report the accuracy on their multi-image splits.
VideoBench
VideoChattPP
ActNet-QA
EgoScheens
MVBench
Bonch
VideoMME
NexQQ
Parcep
VideoDC
MIVU
Model
test
test
m-avg
test
me
val
video
test.
test
wo/w-subs
val
GPT-4V (OpenAI, 2023)
57.0
49.2
60.5
4.06
50.9/63.3
GPT-4o (OpenAI, 2024)
64.6
71.9/77.2
66.7
Gemini-1.8 1.5-Flash (Team et al., 2023)
55.3
65.7
70.3/ 75.0
61.6
Gemini-1.5 (Team et al., 2023)
57.5
72.2
75.0/81.3
64.0
VILA-40B (Lán al., 2024)
58.0
58.0
67.9
54.0
3.36
3.37
60.1/61.1
PLLaVA-34B (Xu al,, 2024s)
60.9
58.1
3.48
-LaVA-N-Video- (Zhang et al., 2024h)
58.8
49.3
70.2
51.6
3.34
3.48
52.0/54.9
50.5
LongVA-7B (Zhang et ul., 2024d)
50.0
56.3
68.3
3.20
52.6/5 554.3
IXC-2.5-BB (Zhang al., 2024c)
52.8
37.3
69.1
71.0
34.4
3.46
3.73
55.8/58.8
MaVA-N-Vido-32h (Zhang et ol., 2024h)
54.3
60.9
63.5
77.3
39.4
3.59
3.84
60.2/63.0
MaVA-OV-ue (8T)
$9.0
47.9
43.3
6
48.6
43.4
$ 4147/4
$1.9
LLaVA-OV-0.5B
50.5
26.8
50.3
45.5
57.2
49.2
44.2
3.12
3.55
44.0/43.55
45.8
LLVAA-OV-BB (SI)
55.1
52.9
60.2
51.2
61.6
54.9
51.1
3.54
3.51
55.0/59.1
54.9
LLaVA-OV-7B
56.6
60.1
64.7
56.7
79.4
57.1
56.9
3.51
3.75
58.2/61.5
56.4
LLaVA- LLVA-OV-72B (SI)
62.1
58.6
60.9
57.1
67.9
62.9
60.9
3.55
8.60
64. 04.8/602
58.9
MaVA-OV-72B
62.3
68.0
80.2
62.1
3.62
66.2 /6.55
Table 5: LLaVA-OneVis performance on video benchmarks. We report the score out of for VideoDC, VideoChatGPT while other results are reported in accuracy. All results are reported as 0-shot accuracy.
15

---

Published in Transactions on Machine Learning Research (02/2025)
S1: Joint Understanding of Diagram and Chart from Multi-Image
Insurance Company
Price
State Farm
68
Allstate Liberty Mutual USAA
63
59
90
User
Ross owns house similar to the brown sector following image They want to get the insurance from Allstate. The price per unit area is given in the following figure What would be the cost of insuring the whole house? All the computations are rounded to two places of decimal.
MLVA-OV
To lind the cost of insuring the whole house, need culculate the of the sector and then multiply the formula (0/360) ++rr, where r Pluggiug these values into the formula, we get A (40/360) 38.01. The price per unit. area given the table as $63. Multiplying per unit area, we get the cost insuring the whole house as 38.01 $63 ~ $2 386.03.
the price per unit The of the sector central angle and the radius. In this case, 112 area by the price
calculated using
Table 6: LLaVA-OneVision transfers its ability to understand diagram and table to mallii-iaage scenarios, interpreting multiple images in a coherent manner.
16

---

Published in Transactions on Machine Learning Research (02/2025)
S2: Multi-image GUI Understanding for Agents
0000R
00-1
00:14
06O18
Search
Cowel
Charta
1ash dode
TRTN
Mak'
Seggested
TikTok
Mlas
Evorys
Dembaa For fends Vest
h
There four sequential mobile UI screens, and transformed by throe steps of tap operations.
Please respectively describe the three tap operations.
LLVA-OV
The image shows a sequence of four mobile application interface screens. Here's a breakulown of the three tap operations based on the visible changes:
1. First. Tap Operation:
The user starts by tapping on the search bar at the top of the screen.
The keyboard is shown at the bottom of the screen, indicating that the user now type search query.
The user types 'TikTok" in the search bar and taps the search icon the keyboard to initiate search.
Second Tap Operation:
Affter tapping the search icon or pressing 'Enter', the screen displays list search results.
'The user taps on first result, which the TikTok application.
This action opens the TikTok's app store page, which includes the app loon, developer information
3. Third Tap Operation:
The user taps the "Open" button on the upp store page.
This action opens the TikTok application, and the screen transitions to the app's main interface.
Table 7: LLaVA-OneVision plays the role of agent. It recognizes multiple screenshots the iPhone and take action to interact with the iPhone, providing operation instructions for automating tasks.
17

---

Published in Transactions on Machine Learning Research (02/2025)
S3: Set-of-mark Prompting
Here's image with objects labeled numbers. Please describee the objects that marked 5 and
LLaVA-OV
The image shows room with various itemus marked with numbers. Here's the description for the items marked as 4, 5, and 7:
The item marked as 4 is framed picture hanging on the wall. Tt features black and white illustration of dandelion with seeds blown away, creating a sense of movement. The item marked storage unit that is part of the room's furniture. It is located between the two desks. The item marked 7 black chair with modern design. It has metal frame and seat, suggesting contemporary style. This chair positioned in front of the desks.
is white bookshelf with multiple shelves. It appears to be
Table 8: LLAVA-OneVision exhibits excellent set-of-marks prompting capabilities, ie, referring to marks when answering questions. This example demonstrates that describing specific objects based on numerical labels within an image highlights its comprehension skills in handling fine-grained visual content.
S4: Image-to- Video Editing Instruction
User
Please provide the viden generation instruction that can change the source image to the largel video.
LLAVA OV
To generate video from the provided image, you would need to create sequence of images that depict the progression of the mushrooms being transformed into penguins. Here's step-by-step guide on how to create this video:
Start with the initial image the mushrooms on a plate.
In the next image, begin to change the appearance the mushrooms to resemble penguins.
with the smallest mushrooms gradually move the larger ones. Continue transform the mushrooms into penguins, adding details such eyes, beaks, and flippers.
mushrooms should be transformed, and the penguins should appear interact with each other and the mushrooms.
As the video progressss,
of penguins interacting with the mushrooms, creating playful and whimsical scene. Remember to maintain consistent style and color palette throughout the video to ensure cohesive look. The trausformation should be smooth and realistic, making it seen though the mushrooms are naturally turaing into penguins.
5. The final images should show
Table 9: LLa LLaVA-OneVision learns to generate detailed video creation prompls based on static image. This capability is generalized to videos from the image-to- -oaige- language editing generation.
18

---

Published in Transactions on Machine Learning Research (02/2025)
S5: Video-to-Video Difference
User
Here are two videos, please provide analysis on their differences.
LLaVA-OV
The first video shows soquence of images where man in blue plaid shirt approaches woman in red cress, engages in conversation, and eventually gives her kiss on the chrek. The woman in the red dress looking at the man with slight smile, and another woman in light blue top observing the scene. The setting appears be street with buildings in the background.
second video shows the same man the blue plaid shirt walking away woman in the dress, who now looking in the opposite dircction. The woman in the light blue top no longer visible in this sequence. The man is walking down the street, passing by the buildings and other pedestrians. The overall mood the video changes from the intimate and personal interaction in the first video to more solitary and neutral scene in the second video.
Table 10: HaVAA-OneVssion learns to analyze differences between videos with the same starting frame but different endings.
19

---

Published in Transactions on Machine Learning Research (02/2025)
S5: Video-to-Video Difference
What the differences between the two vidoos??
LLAVA-OV
video features a caterpillar with green and black stripes on its body, resting on leaf. The caterpillar has white, bristly hair-like structures along its back. The background is blurred, but it appears to a garden or natural setting with green foliage and some sunlight filtering through. The second video shows a butterfly with blue and black wings, resting on a leaf. The butterfly has white spots on its wings and an orange patch near the base of the wings. The background is also blurred but suggests a similar garden or natural setting with greenery. The differences between the two videos are:
1. The first video shows eaterpillar, while the second video shows butterffy.
2. The caterpillar has green and black stripes, while the butterfly has blue and black wings with white spots.
3. The caterpillar has white, bristly hair-like structures along its back, which are not present
butterfly.
4. butterfly has more delicate and transparent appearance compared to the fuzzy
texture of the caterpillar.
5. The wings of the butterfly are spread out, while the caterpillar is in curled position.
6. The butterfly has long anternae, which are not visible in the caterpillar's frame.
Table 11: LLaVA-OneVision learns to analyze differences between videos with similar backgrounds but different foreground objects.
20

---

Published in Transactions on Machine Learning Research (02/2025)
S6: Multi-camera Video Understanding in Self-driving
User
Here is a vidco showing four camera views a driving car. The top two show front views, and the bottom two show back views. First describe the video of each view in detail, and then plan the next move of the ego car.
LLaVA-OV
The video consists six different camera views driving car, arranged with three and back perspectives. the front views:
columns Each view captures the from different angle, showing both front
1. The first image shows the car approaching an intersection with pedestrian crossing.
There are cars ahead, and the pedestrian is visible on the sidewalk.
2. The second image shows the car at the intersection, with the pedestrian crossing and cars ahead. The pedestrian is now closer to the car.
3. The third image shows a pedestrian visible the sidewalk.
In the back views:
1. The first image shows the approaching the intersection with pedestrians crossing and black aside.
The second image shows the car the intersection, with several cars bebind. The third image shows the car passing through the intersoction, with the podestrion visible on sidewalk.
The next move of the ego car would to continue driving forward, following the road and traffic rules, and ensuring the pedestrian's 's safety as they cross the street. next move of the ego car would be to continue driving forward, following the road and traffic rules, and ensuring the pedestrian's 's sufety as they cross the street.
Table 12: LLa VA-OneVision analyzes and interprels mulli-camera video foolage in self-driving conterts.
21

---

Published in Transactions on Machine Learning Research (02/2025)
ST: Composed Sub-video Understanding
HOUSE s DRAGON
HOUSE x DRAGON
HousE DRAGON
DRAGON HOUSE DRAGON
User
Please describe the video's content and layout in detail.
LLaVA-OV
The video showeases series of fromes featuring ocntral, obscured foreground view of room, presumably game
office setup. The room is adorned with several posters the wall, a model of humanoid figure wearing armor, and various other items that suggest space dedicated to hobbies gaming. the foreground, there is person, likely woman, who wearing headphones and expression, indicating that she may be reacting something off-screen.
neutral slightly surprised facial
The backdrop throughout the video is dynamui, digital illustration from the television show "Game of Thrones," specifically from the series 'House the Dragon." This series is a prequel to 'Game of Throues' and centers around the House Targaryen, a prominent family within the show's fictional world. The images in question feature characters from the series, including members of House Targaryen. Some characters are seen in battle attire, riding dragons, and engaged in combat scenarios. The scenes are detailed and vivid, depicting intense moments from the show, such the White Walkers and dragon battles.
The video starts with the person the foreground wearing headphones and black shirt. Behind her, the consistent backdrop from 'House of the Dragon' unfolds, focusing characters from the series. sense of drams and tension that is central the narrative of "House the Dragon." the video progresses, the scenes transition through various moments from the series. The charncters are shown in different scenarios, such as riding dragons, engaging in combat, and experiencing intense momeuts. The visunls are rich with color and detail, highlighting the epic scale and dramatic tone of the series. The consistent presence of the game room setup in the foreground provides sense of continuity throughout the video.. The video captures the essence of 'House of the Dragon' by using dramatic and action- packed scenes from the series. The person the foreground remains coustant, adding clement of personal reaction the intense moments presented in the background. summary, the video juxtaposes calm, possibly surprised individual game room setup with dynamic and intense from "House of the Dragon," creating contrast dramatic background
evident that the series elements meant evoke
between the peaceful foroground
Table 13: Vaav-OeVneiiion learns understand and describe composed sub-videos in detail.
22

---

Published in Transactions on Machine Learning Research (02/2025)
S8: Visual Prompting in Video
User
Describe the player highlighted in the video.
LLaVA-OV
The player highlighted in the video is wearing white kit, and the number *10" visible their shirt. This player positioned the of the pitch and is actively involved in the play, demonstrating control and dribhling skills as they navigate around the field. The player is surrounded by opponents red kits, indicating that they are likely a key player for their team, possibly a playmaker or attacking midfielder. Throughout the video, the player maintains possession of the ball and creates opportunities for their team, showcasing their importance in the game's 's dynamics.
Table 14: LLaVA-OneVision learns to provide detoiled descriptions of highlighted subjects video content.
23

---

Published in Transactions on Machine Learning Research (02/2025)
S9: Referring Image in Video Understanding
MESS
User
There are 2 images. Dors the person in image appear in image 2? If yes, what is he doing
MLVA-OV
Yes, the person from image appears in image 2. He is playing soncer with two other individuals in a residential outdoor area.
User
Who is the man in image 1?
LLaVA-OV
The man Lionel Messi.
MESSI
Here image and video. the shown in the image appear the video? If yes, deseribe his behavior detail.
LaVA-OV
the man activity in grassy outdoor shoes. Throughout the video, he and at various points, he is either preparing to kick the ball or has just kicked it. His actions suggest he is actively participating in the game, and be apprars to be in motion, indicating movement and activity.
the image appears the viden.
engaging soncer He dressed white shirt, dark pants, and white yellow and black soccer ball,
standing
Here video and an inage. Does the man shown in the image appear in vidlo?? Explain
The shown the image does appear the video. The image shows person in red sports jersey with the number 7, while the video features individuals playing with ball in grasey outdoor between the image and the video, indicating that they are separste and not related.
The clothing, background, and activity different
User
Who is the man in the image?
The Cristiano Ronaldo.
Table 15: Vaaei-Oneissson''ss capability in referring image and video understanding. Tt accurately identifies the same individual in two images in the first instance. It identifies the same individual in both the image and the video in the second instance and correctly concludes the absence of the individual in the third instance, indicating its understanding capability to relate visual query in both image and video understanding.
24

---

Published in Transactions on Machine Learning Research (02/2025)
References
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In AAAI, 2019. 40
Aida Amini, Saadia Gabriel, Peter Lin, Rik Koncel-Kedziorsk Yejin Choi, and Hannaneh Hajishirzi. Mathqa: Towards interprotable math word problem solving with operation-based formalisms, 2019. URL www.ai.oor//aa///00/0/. 13319. 40
Anthropic. Claude-3.5.
http://pp....ceeeeeee/////aaaaaa 2024. 2, 14
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In ICCV, 2015. 40
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding, In proceedings of the IEEE/CVF conference on compuler vision and pallern recognition, pp. 29129-191330 2022a. 10
Daichi Azuma, Taiki Miyanishi, Shuhei Kurita, and Motoaki Kawanabe. Scanqa: 3d question answering for spatial scene understanding. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), 2022b. 41
Haoping Bai, Shancong Mou, Tatiana Likhomanenko, Ramazan Gokberk Cinbis, Oncel Tuzel, Ping Huang, Jiulong Shan, Jianjun Shi, and Meng Cao. Vision datascts: A benchmark for vision-based industrial inspection, 2023a. URL http:///aaxvvv..aaaabaee
07890.. 41
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile vision-language model for understanding, localization, text reading, and beyond. Technical Report, 2023b. 14, 38
Ankan Bansal, Yuting Zhang, and Rama Chellappa. Visual question answering on image sets. In Compuler Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 28, 2020, Proceedings, Port XXI 16, pp. 51-67. Springer, 2020. 10
Lucas Beyer, Andreas Steiner, André Susano Pinto, Alexander Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for transfer. arXiv preprint arX:v:2477.07116, 2024. 3
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In ICCV, 2019. 40
Holger Caesar, Varun Bankiti, Alex H. Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. muscenes: A multimodal dataset for autonomous driving, 2020. URI. https://arxiz: /tt.:/ hwwp:/wwwwwwrrg//////././// 11027 11027. 41
Jimmy Carter. Textocr-gpi4v. https:/ http::///gggggggaceooo/aaaaa/tta 40
2024.
Shuaichen Chang, David Palzer, Jialin Li, Eric Fosler-Lussier, and Ningchuan Xiao. Mapqa: dataset for question answering choropleth maps, 2022. URL https://axxiv.. http:///aaiw...orra/////////// 08545. 40
Yingshan Chang, Mridu Narang, Hisami Suzuki, Guihong Cao, Jianfeng Gao, and Yonatan Bisk. Webqa: Multihop and multimodal qa. arXie preprint arXiw:2109.00553 2021. 41
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gpt4v-synthesized data for lite vision-language model. Xiv preprint 2xx::2240.116666 2024a. 6, 40
Jiaqi Chen, Tong Li, Jinghui Qin, Pan Lu, Liang Lin, Chongyu Chen, and Xiaodan Liang. Unigco: Unifying geometry logical reasoning via reformulating mathematical expression, 2022a. URL https:
//arxiv. wttp://wrwr/aaa/2...///2// 02746 40
25

---

Published in Transactions on Machine Learning Research (02/2025)
Jiaqi Chen, Jianheng Tang, Jinghui Qin, Xiaodan Liang, Lingbo Liu, Eric P. Xing, and Liang Lin. Geoqa: A geometric question answering benchmark towards multimodal numerical reasoning, 2022b. URL https:
//arxiv.org/abs/2211.... 14411. 40
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXio:2811.12990
2023a. 6
Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Bin Lin, Zhenyu Tang, Li Yuan, Yu Qiao, Dahua Lin, Feng Zhao, and Jiaqi Wang. Sharegpi4video. Improving video understanding and generation with better captions. arXiv preprint or arXiv:246.. (.000000.2,, 2024b. 39, 41
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifong Dai. Internvl: Scaling up vision foundation models and aligning for generic risual-linguistic tasks. arXiv preprint arXiv:2312.14255 14238, 2023b. 10, 14, 38, 40
Zhoujun Cheng, Haoyu Dong, Zhiruo Wang, Ran Jia, Jiaqi Guo, Yan Gao, Shi Han, Jian Guang Lou, and Dongmei Zhang. Hitab: A hierarchical table dataset for question answering and natural language generation. In ACI, 2022. 40
Yew Ken Chia, Vernon Toh Yan Han, Deepanway Ghosal, Lidong Bing, and Soujanya Poria. Puzzlevqa: Diagnosing multimodal reasoning challenges of language models with abstract visual patterns. arXiv preprint 24v:24409. 13315, 2024.
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: anccly-annotated 3d reconstructions of indoor scenes, In Proc. Computer Vision and Pattern Recognition (CVPR), JEEE, 2017. 41
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instruetblip: Towards general-purpose vision-language models with instruction tuning. In NewrIPS, 2024. 3
Maxwell Forbes, Christine Kaeser- Chen, Piyush Sharma, and Serge Belongie. Neural naturalist: Generating fine-grained image comparisons, 2019. URL https://arxiv..o org/ abs/ 1909 04101. 41
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024a. URL attps://arxivy hww. htsg//bs/2200 /20/////2....2... 13394. 36, 38
Chaoyou Fu, Yuhan Dai, Yondong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Vidco-mme: The first-ever comprehensive evaluation benchmark of multi-modal 1lms video analysis. ar preprint arX 21075, 2024b.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data, 2023. URL
attps://arzse www.iv.og/.abaaaa..o.ooaa htrggabb//3000 09344. 41
Xingyu Fu, Yushi Hu, Bangzheng Li, Yu Feng, Haoyu Wang, Xudong Lin, Dan Roth, Noah A Smith, Wei-Chiu Ma, and Ranjay Krishna. Blink: Multimodal large language models can see but not perceive. arXiv preprint arXie:2404. 12390, 2024c. 10
Jiahui Gao, Renjie Pi, Jipeng Zhang, Jiacheng Ye, Wanjun Zhong, Yufei Wang. Lanqing Hong, Jianhua Han, Hang Xu, Zhenguo Li, and Lingpeng Kong. G-llava: Solving geometric problem with multi-modal large language model, 2023. URL https://arxiv.orge wttp:o///////....a/a/e/
40
Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, Miguel Martin, Tushar Nagarajan, Ilija Radosavovic, Santhosh Kumar Ramakrishnan, Fiona Ryan, Jayant Sharma, Michael Wray, Mengmeng Xu, Eric Zhongcong
26

---

Published in Transactions on Machine Learning Research (02/2025)
Xu, Chen Zhao, Siddhant Bansal, Dhruv Batra, Vincent Cartillier, Scan Crane, Tien Do, Morrie Doulaty, Akshay Erapalli, Christoph Feichtenhofer, Adriano Fragomeni, Qichen Fu, Abrham Gebreselasie, Cristina Gonzalez, James Hillis, Xuhua Huang, Yifei Huang, Wenqi Jia, Weslie Khoo, Jachym Kolar, Satwik Kottur, Anurag Kumar, Federico Landini, Chao Li, Yanghao Li, Zhenqiang Li, Karttikeya Mangalam, Raghava Modhugu, Jonathan Munro, Tullie Murrell, Takumi Nishiyasu, Will Price, Paola Ruiz Puentes, Merey Ramazanova, Leda Sari, Kiran Somasundaram, Audrey Southerland, Yusuke Sugano, Ruijie Tao, Minh Vo, Yuchen Wang, Xindi Wu, Takuma Yagi, Ziwei Zhao, Yunyi Zhu, Pablo Arbelaez, David Crandall, Dima Damen, Giovanni Maria Farinella, Christian Fuegen, Bernard Ghanem, Vamsi Krishna Ithapu, C. V. Jawahar, Hanbyul Joo, Kris Kitani, Haizhou Li, Richard Newcombe, Aude Oliva, Hyun Soo Park, James M. Rebg, Yoichi Sato, Jianbo Shi, Mike Zheng Shou, Antonio Torralba, Lorenzo Torresani, Mingfei Yan, and Jitendra Malik. Ego4d: Around the world in 3,000 hours of egocentric video, 2022. URL attpp://axt /wwpiv. wwwpziiiorg///baao hts//bb//2110 27050.7058.....111 39, 41
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309. 00615, 2023. 3
Ziyu Guo, Renrui Zhang, Hao Chen, Jialin Gao, Peng Gao, Hongsheng Li, and Pheng- Ann Heng. Sciverse. io, 2024. URL https: http:://///vveeeeeeeeeieee gi github..... bhuub.io/. io/. 10
Tammay Gupta, Dustin Schwenk, Ali Farhadi, Derek Hoiem, and Aniruddha Kembhavi. Imagine this! scripts to compositions to videos, 2018. URL https://arxiv org/ ataaaaas/1800 180.00668.8.8.... 41
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018. 40, 41
Yining Hong, Haoyu Zhen, Peihao Chen, Shuhong Zheng, Yilun Du, Zhenfang Chen, and Chuang Gan. 3d-lim: Injecting the 3d world into large language models. Advances in Newral Information Processing Systems, 96:20482-20445 2023. 10
Mehrdad Hosseinzadeh and Yang Wang. Image change captioning by learning from an auxiliary task. Tn 2021 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2724 2733, 2021. doi: 10.1109 D0.109/CVPR4643777 CVVPR4113..0011.00777 41
Ting- Hao K. Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Aishwarya Agrawal, Ross Girshick, Xiaodong He, Pushmeet Kohli, Dhruv Batra, et al. Visual storytelling In 15th Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL 2016), 2016. 10
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In CVPR, 2019. 40
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-odit: A high-quality dataset for instruction-based image editing, 2024. URL https://arxie http:://wwwwwo////a//a//e/ arg/abs/ 2404.09990. 41
Phillip Isola, Josoph J. Lim, and Edward H. Adelson. Discovering states and transformations in image collections. In CVPR, 2015. 41
Mohit Iyyer, Varun Manjunatha, Anupam Guha, Yogarshi Vyas, Jordan Boyd-Graber, Hal Dauné III au2, and Larry Davis. The amazing mysteries of the gutter: Drawing inferences between panels in comic book narratives, 2017. URL https://arxiv wwwpniii.orr//aaee
org./oog//.o..o.o..o attpa//s/..........////. 41
Harsh Jhamtani and Taylor Kirk-aiikaatrick. Learning to describe differences between pairs of similar images. XrXiv preprint ar Xiv: 108.1888.00 2018a. 10
Harsh Jhamtani and Taylor httppp///aardv..ogaaa///000
Learning to describe differences between pairs of similar images,
2018b. URL https: /ttp://aaavvvv//aa/eaa/eeaeeaiaa www. 10584. 41
27

---

Published in Transactions on Machine Learning Research (02/2025)
Dongfu Jiang, Xuan He, Huaye Zeng. Cong Wei, Max Ku, Qian Liu, and Wenhu Chen. Mantis: Interleaved multi-image instruction tuning. arXiv preprint or urXiv:205.0014 2024. 3, 15, 41
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017. 40
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In CVPR, 2018. 38, 40
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Akos Kadar, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning, 2018. URL https: /awxivrxvvvvvv/////////////v/i org/ org/abs/ 1710.07300 40
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vims: Investigating the design space of visually-conditioned language models. Technical Report, 2024. 3
Mchran Kazemi, Hamidreza Alvari, Ankit Anand, Jialin Wu, Xi Chen, and Radu Soricut. Geomverse: A systematic evaluation of large models for geometric reasoning. ar preprint arXiv:2312. 12241, 2023.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11 14, 2016, Proceedings, Part IV 14, pp. 235 251. Springer, 2016a. 10, 36, 38
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Seo, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In ECCV, 2016b. 38, 40
Aniruddha Kembhavi, Minjoon Sco, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannanch Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5376-5384, 2017a. doi: D0.109/CVPR.2011
41
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hammaneh Hajishirzi. Are you smarter than a sixth grader? textbook question answering for multimodal machine comprehension. Proceedings of the IEEE Conference on Computer Vision and Pattern recognition, pp. 4999 9009, 2017b.
Douwe Kicla, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreot Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. In NewrIPS, 2020. 40
Geewook Kim, Teakgyu Hong, Moonbin Yim, Jeong Yeon Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document understanding transformer. In European Conference on Computer Vision (ECCV), 2022. 38, 40
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanic Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Fei-Fei Li. Visual genome: Connecting language and vision using crowdsourced dense image annotations, 2016. URI https://arxiv: org/abs/1600 07332. 40
Benno Krojer, Vaibhav Adlakha, Vibhav Vineet, Yash Goyal, Edoardo Ponti, and Siva Reddy. Image retrieval from contextual descriptions. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguisties, Online, May 2022. Association for Computational Linguisties. 41
Shanghai AI Laboratory. Sharegpt-4o: Comprehensive multimodal annotations with gpt-4o, 2023. 39
Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushmant A dataset of clinically generated visual questions and answers about radiology images. Scientific dala, 5(1):1-10, 2018. 40
28

---

Published in Transactions on Machine Learning Research (02/2025)
Hugo Laurengon, Léo Tronchon, Matthieu Cord, and Victor Sanh. What matters when building vision-language models? Technical Report, 2024. 3, 7, 38
Bo Li, Hao Zhang, Kaichen Zhang, Dong Guo, Yuanhan Zhang, Renrui Zhang, Feng Li, Ziwei Liu, and Chunyuan Li. Llava-next: What else influences visual instruction tuning beyond data?, May 2024a. URL
https://llava-v//
11tpb//////222005255------ 1als--llaaneet-aaaaaaaaa nee-- mext-ablations/ 2, 3, 4, 36
Bo Li, Kaichen Zhang, Hao Zhang, Dong Guo, Renrui Zhang, Feng Li, Yuanhan Zhang, Ziwei Liu, and Chunyuan Li. Llava-next: Stronger Iims supercharge multimodal capabilities in the wild, May 2024b. URL next- stronger-1lme/ 2, 3, 10, 37, 39
https://1lava- we.l..ittie wtt.gg.....oooo///2224/4////11/1 000/0lo//004400-10-1111110
Chunyuan Li, Zhe Gan, Zhengyuan Yang, Jianwei Yang, Linjie Li, Lijuan Wang, Jianfeng Gao, et al. Multimodal foundation models: From specialists to general-purpose assistants. Foundations and Trends@ in Computer Graphics and Vision, 2024c.
Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li, Wei Li, Zejun Ma, and Chunyuan Li. Llava-next: Tackling multi-image, vidco, and 3d in large multimodal models, June 2024d. URL https://llava-vl. github. botbub..o//llol o0w.ll///2222166--20 1aa------aa---e---nnee---e---
2, 3, 5, 7, 8, 10, 15, 37, 39
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Hanwang Zhang, Wei Ji, Wenqiao Zhang, Tat-Song Chua, Siliang Tang, and Yueting Zhuang. Empowering vision-language models to follow interleaved vision-language instructions. arXi preprint arXi::2308. 041000 2023a. 3, 15
Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, and Yueting Zhuang. Fine-tuning multimodal llms to follow zero-shot demonstrative instructions, 2024e. URL
org/ abs/aa/8.../////////////222aaa 2310.0415.. 8, 41
Yanwei Li, Chengyao Wang, and Jiaya Jia. Llama-vid: An image is worth 2 tokens in large language models. In European Conference on Compuler Vision, 2024f. 3
Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jia. Mini-gemini: Mining the polential of multi-modality vision language models. Technical Report, 2024g. 3
Yitong Li, Zhe Gan, Yelong Shen, Jingjing Liu, Yu Cheng, Yuexin Wu, Lawrence Carin, David Carlson, and Jianfeng Gao. Storygan: A sequential conditional gan for story visualization, 2019. URL https: //arxiv. wtt.oor////aa///////////////
41
Zhuowan Li, Xingrui Wang, Elias Stengel-Eskin, Adam Kortylewski, Wufei Ma, Benjamin Van Durme, and Alan Yuille. Super-clevr: A virtual benchmark to diagnose domain robustness in visual reasoning, 2023b. URL https: http://aaxvvvvo//ee orgg/vv.orrraoo wtgg/bs////a/ https//222000... 40
Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXim:231 1. 10122, 2023. 3
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
Recognition, pp. 26689 26699, 2024. 3, 14, 15
Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015.
URL https://arxiv org/abs//tteeeooooooggooeee 14////1000000 0312. 38, 40
Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the
Association for Computational Linguistics, 2023a. 40
Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning. arXiv preprint arXiv:2306.14565 2023b. 40
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023c. 2, 3, 4, 40
29

---

Published in Transactions on Machine Learning Research (02/2025)
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In CVPR, 2024a. 2, 3, 7, 38
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, oct, and world knowledge, January 2024b. URL https///llavarr..gittbbb. 2, 5, 6, 15, 38
(tog/2204-01-00-1aaaa-----
Xuejing Liu, Wei Tang, Xinzhe Ni, Jinghui Lu, Rui Zhao, Zechao Li, and Fei Tan. What large language models bring to text-rich vqa? arXiv preprint 0rXiu:2311.03300 2023d. 10
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songvang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, and Ziwei Liu. Mmbench: Is your multi-modal model an all-around player? Technical Report, 2023e. 10, 36
Lab. Video detail caption, 2024. VideoDetailCaption.. 11
URL https://nuggingface http:/efaaa...cacaaaeeeeeeeeeeee datdaatsssstssmmsssss /ab/
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoe V Le, Barret Zoph, Jason Wei, et al. The flan collection: Designing data and methods for effective instruction luning, In International Conference on Machine Learning, pp. 22631 22648. PMLR, 2023. 3
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, and Yaofeng Sun. Deepscek-vl: towards worl-world vision-language understanding. Technical Report, 2024. 38
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song- Chun Zhu. Inter-gps: Interpretable geomelry problem solving with formal language and symbolic reasoning. In ACI, 2021a. 40
Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, and Song-Chun Zhu. Inter- gps: Interpretable geometry problem solving with formal language and symbolic reasoning, 2021b. URL 04165. 40
https://arxte http:///aaivi.oo//a//////0./ org/aba/2105.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song-Chin Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. In NewrIPS, c021c. 40, 41
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai- Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In The 36th Conference on Newral Information Processing Systems (NeurIPS), 2022. 40
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai-Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating math reasoning in visual contexts with gpt-4v, bard, and other large multimodal models. Xi preprint ar Xix:23100002255 2023a. 10, 37, 39
Pan Lu, Liang Qiu, Kai-Wei Chang, Ying Nian Wu, Song- Chun Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt. learning via policy gradient for semi-structured mathematical reasoning. In International Conference on Learning Representations (ICLR), 2023b. 40
Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan, Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint or arX1o:2306.05/2, 2023.
11
Karttikeya Mangalam, Raiymbek Akshulakov, and Jitendra Malik. Egoschema: A diagnostic benchmark for very long-form vidco language understanding. Advances in Neural Information Processing Systems, 36, 2024.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019. 40
30

---

Published in Transactions on Machine Learning Research (02/2025)
U-V Marti and Horst Bunke. The iam-database: english sentence database for offline handwriting recognition. International journal on document analysis and recognation, 5:39-46, 2002.
40
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque Chartqa: A benchmark for
question answering about charts with visual and logical reasoning. In ACL, 2022. 10, 36, 38, 40
Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. In WACV, 2021. 10, 36, 38, 40, 41
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Info- graphicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022. 10, 36, 40
Brandon McKinzie, Zhe Gan, Jean- Phan-ppilippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Wocrs, et al. Mml: Mothods, analysis & insights from multimodal Ilm pre-training. arXiv preprinl arXiv:2403.0000 09611, 2024. 3
A. Mishra, K. Alahari, and C. V. Jawahar. Scene text recognition using higher order language priors. In BMVC, 2012. 40
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Oer-vqa: Visual question answering by reading text in images. In 2019 International Conference on Document Analysis ond Recognition (ICDAR), pp. 947-952, 2019a. doi: (0011999TCDAR22000000
41
Anand Mishra, Shashank Shekhar, Ajeet Kumar Singh, and Anirban Chakraborty. Oer-vqa: Visual question answering by reading text in images. In ICDAR, 2019b. 40
Jason Obeid and Enamul Hoque Chart-to-texl: Generating natural language descriptions for charts by adapting the transformer model, 2020. URL http::///wwwwww.//waaa//o
09142. 40
OpenAl. Gpt-4v.
2023. 2, 10, 14, 15
OpenAL. Hello gpt-4o.
2024. 2, 10, 14, 15
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, Ethan Yeo, Eugenie Lamprecht, Qi Liu, Yuqi Wang, Eric Chen, Deyu Fu, Lei Li, Che Zheng, Cyprien de Masson d' d'Autume, Dani Yogatama, Mikel Artetxe, and Yi Tay. Vibe-eval: hard evaluation suite for measuring progress of multimodal language models, 2024a. URL pettp://aaxxer org/abt abs/2405 02287. 37, 39
Piotr Padlewski, Max Bain, Matthew Henderson, Zhongkai Zhu, Nishant Relan, Hai Pham, Donovan Ong, Kaloyan Aleksiev, Aitor Ormazabal, Samuel Phua, et al. Vibe-eval: A hard evaluation suite for measuring progress of multimodal language models. arXiv preprint 0Xiu:2405.02287 2024b. 10
Dong Huk Park, Trevor Darrell, and Anna Rohrbach. Robust change captioning, 2019. URL https: 41
Taarxiv.rg//abe........ wttp://////..../////// //arxiv.
Renjie Pi, Jianshu Zhang, Jipeng Zhang, Rui Pan, Zhekai Chen, and Tong Zhang. Image textualization:
An automatic framework for creating accurate and detailed image descriptions, 2024. URL https:
//arxiv. httggaa///40..0rr/r/
40
Viorica Päträuccan, Lucas Smaira, Ankush Gupta, Adrià Recasens Continente, Larisa Markoeva, Dylan Banarse, Skanda Koppula, Joseph Heyward, Mateusz Malinowski, Yi Yang, Carl Doersch, Tatiana Matejovicova, Yury Sulsky, Antoine Misch, Alex Frechette, Hanna Klimezak, Raphael Koster, Junlin Zhang, Stephanic Winkler, Yusuf Aytar, Simon Osindero, Dima Damen, Andrew Zissorman, and João Carreira. Perception test: A diagnostic benchmark for multimodal video models. In Advances in Neural Information Processing Systems, 2023. URL https: wtwperrriieeeee//oummmmmmm hi-.rmmEPP.-.-EYYEPPPPP... 11
31

---

Published in Transactions on Machine Learning Research (02/2025)
Yuxuan Qiao, Haodong Duan, Xinyu Fang, Junming Yang, Lin Chen, Songyang Zhang, Jiaqi Wang. Dahua Lin, and Kai Chen. Prism: A framework for decoupling and assessing the capabilities of vlms, 2024. URL
bitts://arziie http//aai...raar////////aaa//e/e org/abs/2400 a/0/aa//////..//////////////ee
14544.. 11
Harsh Raj, Janhavi Dadhania, Akhilesh Bhardwaj, and Prabuchandran KJ. Multi-image visual question answering, arXiv preprint wrXi:2111.137000 2021. 10
Hareesh Ravi, Kushal Kafle, Scott Cohen, Jonathan Brandt, and Mubbasir Kapadia. Acsop: Abstract
encoding of stories, objects, and pictures. In 2021 IEEE/CVF International Conference on Compuler Vision (ICCV), pp. 2032 2043, 2021. doi: 10.110991101 (AA.0///CCC4492..222...2..
41
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In ECCV, 2022. 40
Minjoon Seo, Hannanch Hajishirzi, Ali Farhadi, Oren Etzioni, and Clint Malcolm. Solving geometry problems: Combining text and diagram interpretation. In Proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1466 1466-1476, 2015. 40
ShareGPT. https://sharegpi com/, 2023. 38, 40
Mohit Shridhar, Jesse Thomason, Daniel Gordon, Yonatan Bisk, Winson Han, Roozbeh Mottaghi, Luke Zettlemoyer, and Dieter Fox. Alfred: benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10740 10749, 2020. 10, 41
Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh. Textcaps: a dataset for image captioning with reading comprehension, 2020. 40
Gunnar A. Sigurdsson, Gül Varol, Xiaolong Wang, Ivan Laptev, Ali Farhadi, and Abhinav Gupta, Hollywood in homes: Crowdsourcing data collection for activity understanding. ArXiv e-prints, 2016. URL http: /ttpxi..wrg/aas/////. wwtp./orrrbbbb/....rrr 01753. 39, 41
Alane Suhr, Mike Lewis, James Ych, and Yoav Artzi corpus of natural language for visual reasoning. Proceedings of the 55th Annual Meeling of the Association for Computational Linguislies (Volume 2: Short Popers), pp. 217 223, 2017. 10
Alane Suhr, Stephanie Zhou, Ally Zhang, Tris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs, 2019. URL https: htt/////wwi.rrr/////1/111......0 41
Hao Tan, Franck Dernoncourt, Zhe Lin, Trung Bui, and Mohit Bansal. Exprossing visual relationships via language, 2019. URL https://arxa: wow.oogg//ao....e. httpg//bs/190..000// 07689. 41
Ryota Tanaka, Kyosuke Nishida, and Sen Yoshida. Visualmre: Machine reading comprehension on document images. In AAAI, 2021. 40
Benny J. Tang, Angie Boggust, and Arvind Satyanarayan. Vistext: A benchmark for semantically rich chart captioning, 2023. URL https://arxi:: org/ abs/abb/// 05356.. 40
Gemini Team. Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024. URL attps://arxdv.org/abbb//
05530. 14
Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. XrXiv preprint Xrxi223322 11805, 2023. 2, 15
Ting-Hao Huang, Francis Ferraro, Nasrin Mostafazadeh, Ishan Misra, Aishwarya Agrawal, Jacob Devlin, Ross Girshick, Xiaodong He, Pushmect Kohli, Dhruv Batra, C. Lawrence Zitnick, Devi Parikh, Lucy Vanderwende, Michel Galley, and Margaret Mitchell. Visual storytelling, 2016. URL 0tt//1604003938 41
32

---

Published in Transactions on Machine Learning Research (02/2025)
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Tyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal llms. arXiv preprint arXiv:2/06.16860, 2024. 3, 7, 10, 14, 40
Bryan Wang, Gang Li, Xin Zhou, Zhourong Chen, Tovi Grossman, and Yang Li, Scrcen2words: Automatic httpps///arxi..orr/a////21//
mobile ui summarization with multimodal learning, 2021. URL https://arxti: wtgg 03353. 36, 40
Fei Wang, Xingyu Fu, James Y Huang, Zekun Li, Qin Liu, Xiaogeng Liu, Mingyu Derek Ma, Nan Xu, Wenxuan Zhou, Kai Zhang, et al. Muirbench: A comprehensive benchmark for robust imate understanding. arxiv preprint arXiv:2406. 000600941,,1. 2024. 10
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. arXiv preprint ar 2rXi::2109.0000 2021. 3
Chris Wendler, wendlere/renderedteer 2023. 40
Haoning Wu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Annan Wang, Chunyi Li, Wenxiu Sun, Qiong Yan, Guangtao Zhai, et al. Q-bench: A benchmark for general-purpont foundation models on low-level vision. Xix preprint arXiv:2309.1442 2023. 10
Haoning Wu, Hanwei Zhu, Zicheng Zhang, Erli Zhang, Chaofeng Chen, Liang Liao, Chunyi Li, Annan Wang, Wenxiu Sun, Qiong Yan, Xiaohong Liu, Guanglao Zhai, Shiqi Wang, and Weisi Lin. Towards open-ended visual quality comparison, 2024. 41
x.ai. Grok-1.5 vision preview. URL attps://xx htttps:/xx.a/blloo/. http:/aaaa.lloggooo...... 1.5v 10
Junbin Xiao, Xindi Shang, Angela Yao, and Tat-Seng Chua. Next-qa: Next phase of -anseiiinn to explaining temporal actions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pollern Recognition (CVPR), pp. 9777 9786, June 2021. 11, 39, 41
Lin Xu, Yilin Zhao, Daquan Zhou, Zhijic Lin, See Kiong Ng, and Jiashi Feng. Pllava: Parameter-fie frre llava extension from images to videos for video dense captioning. ar preprint or urxin:2444. 1e.0004..6690 16994, 2024a. 15
Zhangehen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpic: Alignment data synthesis from scratch by prompting aligned llms with nothing. ArXie, abs/2406.08846, 2024b. URL http:://ppp..eeaatiiiieeeeeeeeee
semantccscholar org/Corpus :I: 270391444 36, 38, 40
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled lasks in visual instruction tuning. arXiv preprinl 2xXvv2202.116000 2024c. 3
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flam: Scaling human-labeled tasks in visual instruction tuning, 2024d. URL https: 11690. 38, 40
Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis. Recipeqa: A challenge dataset for multimodal comprehension of cooking recipes, 2018. URL https: htrg/abs/8009.0000 00812. 41
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. Qwen2 technical report. arXiv preprint arXi::2407.1007 2024. 3, 36
Jianwei Yang, Hao Zhang, Feng Li, Xueyan Zou, Chunyuan Li, and Jianfeng Gao. Set-of-mark prompting 2023. 11
unleashes extraordinary visual grounding in gpt-4v. arXiv preprint
Jiabo Ye, Anwen Hu, Haiyang Xu, Qinghao Ye, Ming Yan, Guohai Xu, Chenliang Li, Junfong Tian, Qi Qian, Ji Zhang, Qin Jin, Liang He, Xin Alex Lin, and Fei Huang. Ureader: Universal ocr-free visually-situated language understanding with multinodal large language model, 2023. 38, 40
33

---

Published in Transactions on Machine Learning Research (02/2025)
Shukang Yin, Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey multimodal large language models. Xiv preprint arXiv:2306. 13549, 2023. 10
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C. Berg, and Tamara L. Berg. Modeling context in referring expressions, 2016. URL https: www.aa..w....aaaaa...... 0027.
00272. 40
Weihao Yu, Zhengyuan Yang, Linjic Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprinl 2308 02400, 2023. 10
Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yucting Zhuang, and Dacheng Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 9127-9134, 2019. 11, 39, 41
Ye Yuan, Xiao Liu, Wondimu Dikubab, Hui Liu, Zhilong Ji, Zhongqin Wu, and Xiang Bai. Syntax-aware network for handwritten mathematical expression recognition. arXiv preprint Xxic:2203.01011, 2022. 40
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, and Yuxuan Sun. Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi. In CVPR, 2024. 10, 36, 39
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre- training. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 11975-11980 2023. 3, 36
Chi Zhang, Feng Gao, Baoxiong Jia, Yixin Zhu, and Song-Chun Chun Zhu. Raven: A dataset for relational and analogical visual reasoning. In CVPR, 2019. 40, 41
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guider image editing, 2024a. URL https //tp://aaavvvvii http///axvvioorabba/a///// 1011. 41
Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, and Ziwei Liu. Lmms-eval: Reality check on the evaluation of large multmmoall models. arXiv preprint ar 2rXi:2440..27772 2024b. 9, 10, 36
Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, et al. dhernlm-xxooposee long-contextual input and output. arXie preprint (xXi:2407.03322 2024c. 3, 14, 15
A versatile large vision language model supporting
Peiyuan Zhang, Kaichen Zhang, Bo Li, Guangtao Zeng, Jingkang Yang, Yuanhan Zhang, Ziyue Wang, Haoran Tan, Chunyuan Li, and Ziwei Liu. Long context transfer from language to vision. arXiv preprinl ar (rXii:2406.66822 2024d. 15
Renrui Zhang, Jiaming Han, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, Peng Gao, and Yu Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint 230: 16199, 2023a.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai- Wei Chang, Peng Gao, al. Mathverse: Does your multi-modal Ilm truly see the diagrams in visual math problems? arXiv preprint 140444440,.4664, 2024e. 10
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, and Hongsheng Li. Mavis: Mathematical visual instruction tuning, 2024f. URL https:/ /ttsss//axxvv..e/ wrw. httpaa////.000//
40
Ruohong Zhang, Liangke Gui, Zhiqing Sun, Yihao Feng, Keyang Xu, Yuanhan Zhang, Di Fu, Chunyuan Li, Alexander Hauptmann, Yonatan Bisk, et al. Direct preference optimization of video large multimodal models from language model reward. Xix preprint orxin:2404 01258, 2024g. 39
34

---

Published in Transactions on Machine Learning Research (02/2025)
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding. aiv preprint arXin:2306.177 2023b. 40
Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. Llava-next: A strong zero-shot video understanding model, April 2024h. URL https: //llava-vl. www.gttub........ -aata///////00---------aa-a----a wexllvva-eet-iidoo/o... 2, 4, 6, 7, 15, 36, 37
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zheng Liu. Mlvu: A comprehensive benchmark for multi-lask long video understanding. are preprinl arXiv:2406.44200 2024a. 11
Junjie Zhou, Yan Shu, Bo Zhao, Boya Wu, Shitao Xiao, Xi Yang, Yongping Xiong, Bo Zhang, Tiejun Huang, and Zhong Liu. Mlvu: comprchensive benchmark for multi-task long video understanding, 2024b. URL httppp//aaaxioorr//aa////////// 04266 37
Luowei Zhou, Chenliang Xu, and Jason J. Corso. Towards automatic learning of procedures from web
instructional videos, 2017. URL https://arxis http:/axxiv.or../aeee// org//os///. /ss//bb//773..... 09788. 39, 41
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. arXiv preprint arXiv:2304,1059, 2023.
3
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In CVPR, 2016. 40
35

---

Published in Transactions on Machine Learning Research (02/2025)
A Implmenetation Details
A.1 Token Strategy for Mixed-Modality Data
We provide a detailed explanation of our token strategy for handling mixed-modality data within VLaVA- OneVision's architecture, which is illustrated in Figure 3.
For Linll--mage data, we employ the AngResMor-9 strategy, as previously outlined in blog Li al. (2024a). Using SO400M Zhai et al. (2023) as the Vision Encoder, each input image (or grid) is processed into 729 visual tokens. Consequently, the maximum number of visual tokens for single image is 729 (1 + 9), where 1 X 729 represents the base tokens and 9 x 729 accounts for the grid tokens.
For malti-image data, we utilize a simple padding strategy. Each image is first resized to fit within a 384x384 frame by zero-padding, as required by S0400M, while maintaining the aspect ratio. After processing through the vision encoder, the zero-padding is removed from the tokens. Our training data includes up to 12 images per instance, resulting in maximum of 12 729 multi-image tokens.
For video data, adopt a strategy similar to LLaVA-NeXT- Video Zhang et al. (2024h). Each frame is processed through the vision encoder and then subjected to 2 x 2 bilinear interpolation, resulting in 196 tokens per frame. We sample up to 32 frames per video, leading to a maximum of 32 x 196 video tokens.
As shown in Figure 3, the maximum number of tokens across different modalities is approximately equal. This design strategy aims to balance the data from various modalities, ensuring more equitable represent that is transferable from the perspective of the language model. For instance, a high-resolution image can be interpreted as a composition of multiple images, and multiple images can be understood as a shorter video.
A.2 Language Templates and Special Tokens
We utilize the Qwen-2 series Yang et al. (2024) language models with the template as OpenAl's ChatML During training, we adopt <image> as the marker for image lokens, following previous LLaVA models. This image special token is represented as -200 in the input index after tokenization. For multi-image scenarios, multiple <image> interleaved with text to denote the positions of the images. For video scenarios, we place single <image> at the beginning to indicate the inclusion of video.
One more aspect related to the handling of image tokens is ensuring that there are no extra <image> in the data. For instance, in some code writing tasks, there could be <image>...</iin To avoid potential misunderstandings, we mamually removed around 10 such samples from the Magpie Xu et al. (2024b) and Screen2Words Wang et al. (2021) datasets.
related to HTML code.
B Evaluation Steers Development
B.1 Post-Evaluation as a Development Tool
With the help of our comprchensive evaluation toolkit, LMMs-Eval Zhang et al. (2024b), conduct post-evaluations a selected set of benchmarks after each training experiment concludes.
Our preference for selecting benchmarks is based on whether the targeted scenarios are sufficiently important and specific. These ovaluations should not be too resource-intensive mcaning the benchmarks should not contain too many items, take too long to evaluate, or consume large number of GPT-4V tokens (when using it as the judge model).
In our development, we evaluate on A12D Kembhavi et al. (2016a), ChartQA Masry ct al. (2022), DocVQA Mathew et al. (2021), and InfoVQA Mathew et al. (2022) to examine the model's fine-grained understanding of tables, charts, and diagrams, well as MME Fu et al. (2024a) for formatting control, since it requires only Yes or No answers. We also include MMBench-Dev Liu et al. (2023e) and MMMU-Val Yue
OpeeAA Release v0.28.0/ 228.0/chatml.mm
36

---

Published in Transactions on Machine Learning Research (02/2025)
al. (2024) for multi-discipline evaluation. Quickly obtaining evaluation results these benchmarks will guide our next steps in model development and data curation.
B.2 Improving Model Performance on Key Scenarios
During development process, we gradually recognized the significance of using static evaluation benchmarks perfprmance indicators. Our primary goal at this stage is not to overfit the model to certain datascts to achieve exceptionally high performance. Instead, we benchmark our models against GPT-4V's performance to set our target thresholds (e.g,, initially 80%, gradually increasing to 95%%.. Once the model meets the score roquirements in static evaluations, it indicates that the model has sufficient capabilities in the selected scenarios. Furthermore, we cannot blindly pursue results on benchmarks, as even the test data for A12D may have certain issues
Ultimately, our focus is on optimizing the model's visual chat and reasoning capabilities. In this stage, we monitored the model's performance benchmarks such as MathVista Lu et al. (2023a), LLa Li et al. (2024b), MM-LiveBench Zhou et al. (2024b), and Vibe-Fxal Padlewski et al. (2024a). These benchmarks require the model to engage in visual dialogue with challenging questions, and demand a diverse skill set with extensive world knowledge. This helps us create a model with strong generalization capabilities in real-world scenarios.
B.3 Evaluation Task Information
In this section, we provide information on all the tasks used during the evaluation. Specifically, we use the default post_prompt and pre_prompt from the LMMs-Eval framework. These prompts are consistent with the evaluation of our previous LLaVA-NeXT Li et al. (2024b); Zhang et al. (2024h); Li et al. (2024d). The table below details the specific tasks used LMMs-Eval and their corresponding task names.
Tasks Information
Single-image:
ai2d, chartqa, docvqa_ val, infovqa val, mme, realworldqa, mathvista testmini, llava in the wild, mivet, mmbench en _ev, ocrbench, mmmn, llava Ilava_wilder: wilder ssmass,lmlll__ vibe__e eval, wildvision_ testmini vision, seedbench, scienceqa. img, mmstar, dc100
live live_bende bench 2406, mastvense_testminii
Videos:
activitynetqa, videochatgpt, nextqa mc _est, egoschema, video dc499, videmme, videomme_ __obumme_wssuttttle perceptiontest_wee val me, mlvu, mvbench, longvideobench, val_v
Multi-image:
llava interleave bench, muirbench
By referring to the task names listed here, the audience can directly retrieve the generation arguments and specific prompt information. For instance, the details for tasks=ai2d are available Imms-eval/ ai2d. By following these settings, researchers can easily reproduce results.
C Data Curation Roadmap of LLaVA-NeXT Series
In this section, we provide the in-depth experience and roadmap of data curation in the LLaVA-NeXT scries. To achieve strong multimodal performance, we need collect and curate high-quality data from various sources, which is crucial for the model's generalization capabilities.
C.1 Single-Image Data Curation
As the primary data source, our principle for single-image data has always been that quality outweighs quantity. Given limited resources, we strive use high-quality data maximize the performance.
Discussion AI2D Evaluation
37

---

Published in Transactions on Machine Learning Research (02/2025)
The first version of the LLaVA-NeXT models comprising 760K data samples Liu et al. (2024b), includes 665K samples from VA-1.5 Liu et al. (2024a),
Mistral-7B, Hermes- Yi-34)))
3,247 samples from A12D Kembhavi et al. (2016b), 18,317 samples from ChartQA Masry et al. (2022), 10,194 samples from DocVQA Mathow ct al. (2021), 20,000 samples from DVQA Kafle et al. (2018), 40,093 samples from SynthDGGEN Kim et al. (2022), and 15,131 samples from user requests on LLaVA's 's demo, re-annotated with GPT-4V. In the subsequent iteration, we added 20,000 samples from COCO Caption Lin et al. (2015), forming a new 790K version. This 790K dataset supported the second release of LLaVA-NeXT models
LLaVA-NeNT-Qwen-
-weeee---Q-e-e--e -10Bb.
In subsoquent collections, we accumulated open-sourced datasets from the Internet and referred to the dataset collection processes of other advanced LMMs, such as Qwen-VL Bai et al. (2023b), DeepSeek-V VL Lu et al. (2024), Intern-VI. Chen et al. (2023b), Vision-Flan Xu et al. (2024d), UReader Ye et al. (2023), Idefics-2 (Cauldron) Laurengon et al. (2024), and Cambrian. During the data iteration process, we strictly adhered to the initial LLa VA-1.5 strategy. For each datasot, we manually inspected and ensured its quality and QA format. We also designed specific formalting prompts to make data from different sources compatible with each other, thus avoiding conflicts.
Some data sources, such as A12D and ChartQA, appear in different dataset collections and may be duplicated. Since Cauldron includes special formatting prompts, its data is not straightforward to re-format. Therefore, we prioritize using data from other collections that are closer to the raw format. For the Cambrian dataset, we only selected a subset of the GPT-40 re-annotated data. We also collected math-related data from the MathV and MAVIS datasets.
For the pure language data, replaced the ShareGPT ShareGPT (2023) text data that LLaVA has been using since version 1.5. Given that our largest Qwen2-72B model has achieved performance levels close to latest GPT-4 model in language tasks, we need to use higher quality language data to maintain or further enhance its language capabilitics. To achieve this, we sourced the highest quality language SFT data available, the Magpie-Pro dataset Xu et al. (2024b).
After undergoing the aforementioned process, we have obtained approximately 4 million raw SFT data samples, ensuring their quality and accuracy. Additionally, we utilized Azure's OpenAl GPT-4V and GPT-4o services to re-annotate data, focusing on scenarios that were not adequately covered by the original data but are crucial. These scenarios include:
(1) Detailed Descriptions on Charts and Diagrams: For this scenario, we used images from the A12D and InfoVQA training sets and employed GPT-4V to provide detailed descriptions of the images, resulting in 4,874 detailed descriptions for AI2D and 1,992 samples for TnfoVQA.
(2) Chinese Language: We used images from the LLa LLa VA-158K dataset and employed GPT-4o to provide detailed descriptions in Chinese, resulting in a total of 91,466 samples.
(3) Multi-turn Dialogue: Also with the LLaVA-158K dataset, we employed GPT-4o to create long dialogues with an average of more than 3 turns per conversation, obtaining a total of 26,048 samples.
When resources permit, we recommend a data validation process we used in early stage data sourcing. We extract approximately 100K samples from cach newly added data source or collection (if the selected data source can form collection) and add them to the 790K version of the dataset. We validate newly added data under the 80400M-Qwen-11. 1.5-0.B experimental setting. If the addition of new data results in a performance decline compared to the baseline, we conduct further manual inspections of the data and adjust the formatting prompt accordingly. This step requires abundant resources and must be carried out by highly professional researchers, as it cannot be substituted with average human annotators.
During the collection process, we manually labeled the datasets with two tags: (General, Language, Math- /Reasoning, General OCR, Doc/ heee/hhaat//eeeenn and (Fixed-form, Free-form). Based on these tags, we formed the final distribution of 3.2 million single-image data samples.
Starting with the initial distribution, we gradually increased the amount of free-form (most of them are GPT-4V/o annotated) data and observed the model's performance on various benchmarks and try to balance among them. These benchmarks include academic datasets, such as AI2D Kembhavi et al. (2016a), MME Fu
38

---

Published in Transactions on Machine Learning Research (02/2025)
et al. (2024a), MMMU Yue et al. (2024), MathVista Lu et al. (2023a), and visual chat datasets, such M. Li et al. (2024b), and Vibe-Fval Padlewski et al. (2024a). Ultimately, we gradually established an optimal data distribution for single-image tasks under the 7B setting.
C.2 OneVision Data Curation
In addition to single-image data, incorporate multi-image and video datasets to support wider scope of visual scenarios. We aim to balance the capability among different data modalities, and achieve an overall superior performance with one framework as LLaVA- VA--Oneissio.
For multi-image data, we adopt the diverse interleaved multimodal tasks within M4-Instruct dataset from LLaVA-NeXT-Intee
Li et al. (2024d). This dataset mainly comprises general multi-inage tasks, such spotting the difference, visual story telling, image editing instruction generation, interleaved multi-image dialogue, multi-image puzzle, low-level multi-image assessment, etc. Besides, also utilize the multi-view datasets in M4-Instruct to indicale spatial information in the 3D world, including embodied VQA (dialogue and planning) and 3D scene VQA (captioning and grounding).
For video data, we first integrate the multi-frame data from M4-Instruct, including NExT-QA Xiao et al. (2021) and ShareGPT4Video Chen et al. (2024b). Then, to enable more detailed temporal cues, we select. several datasets commonly used in recent academic research for re-annotation, including Charades Sigurdsson et al. (2016), ActivityNet Yu ct al. (2019), YouCook2 Zhou et al. (2017), and Ego4D Grauman et al. (2022). Initially, annotated captions. Following ShareGPT-40 Laboratory (2023), sampled video frames at frame per second (FPS) and used the pre-defined instructions to prompt GPT-4o for generating video captions. Additionally, following LLa VA-Hound Zhang et al. (2024g), we developed open-ended question-ansseringg pairs and their corresponding multiple-choice versions using the captions created by GPT-4o. We also employed GPT-4o to generate question-answer pairs, obtaining high-quality video data for OneVision training.
C.3 Detailed Dataset Statistics
We primarily use tables to present the statistical information of all datasets utilized in both the Single-Image and OneVision stages. The information includes the dataset category, dataset name, number of samples, and prompt type. The dataset statistics are summarized in Table 16.
39

---

Published in Transactions on Machine Learning Research (02/2025)
Dataset
# Samples
Prompt ID
Dataset
# Samples
Prompt ID
General (1.14M, 36. 1%))
AOKVQA Sehwenk et al. (2022)
66100
1
Cambrian (iltered) Tong et al. (2024)
83131
CLEVR Johnson ct al. (2017)
700
COCO Caption et al. (2015)
20000
Hateful Memes Kiela et al. (2022)
8500
1
IcanQA La al. (2021e)
2494
Image Textualization Pj et al (2024)
90583
11
VLaVA-1 Liu et al. (2023))
158000
M.VA--Wild (train) Láu
(2023c)
54517
MaVAR Zhang ot nl, (2023h)
20000
OKVQA Mariuo et al. [2019)
8998
1
ReCCOCO Yu ct al. (2016)
50586
7,8
StienceQA La ot al. (2022)
4976
5
ShareGPTOO ShareCPT (2021)
57280
11
ShareGPTTV ShareGPT (2023)
92//5
11
ST-VQA Biten et al. (2019)
17217
TallyQA Acharya al. (2019)
9868
1
Vision FLAN Xu al. (2024d)
186070
VisualWW Zhu et al. (2016)
14366
5
VieText Tang ot al. (2023)
9960
15
ViaWiz Gurari et al. (2018)
6614
2
VQARAD Lau et al. (2018)
313
VQAv2 Antol et al. (2015)
82783
1
VSR Liu et al. (2023s)
2157
WebSightt
10000
IntrGPS Lu et al (2021b)
1280
ALLaVA Instroct Chen et al. (2024a)
70000
Doc/
/be/Coun Chaart, /Sereen (20.6%, 641K)
M12D (GPT4V Detailed Caption)
AI2D (InterVLL Chen al. (2023b))
12413
AI2D (Original) Kemblavi et al. (2016b)
3247
5
Chart2 Text Obeid & Hoque (2020)
26961
13
ChartQA Maary et sl. (2022)
18317 1
Diagram Image2Text
300
17
DocVQA Mathew al. (2021)
10194
DVQA Kidle et al. (2018)
20000
FigureQA Kahou al. (2018)
1000
3
HilTab Clacug ct al. (2022)
2500
Infographic VQA Mathew et al. (2022)
1101
1
LRV Chart Liu es al. (2023b)
1787
RoBUT SQA
8514
RoBUT W33SQL
74989
RoBUT WTQ
38246
Screen2Words Wang
(2021)
15730
TQA Kembhhvi et al. (20171)
1365
5
UReader Caption Ye et al. (2023)
91430
9
UReader Ye et al. (2023)
17327
1
UReader KG Ye et al. (2028)
37550)
14
UReader QA Ye (2023)
252954
VisuslMRCTaanae al. (2021)
3027
Math Resasoning (20. 13.632K)
MAVIS Manual Collection Zhang et al. (2024f)
87358
19
MAVIS Data Engine Zhang et al. (2024))
100000 19
CLEVR-Math Johnson et al. (2017)
5290
2
Geo170K Aligu Gao et al. (2023)
60252
Geo170K QA Gao al. (20124
67833
Geometzy3K
(20211)
2101
GEOS Seo et al (2015)
508
6
Geometry'lK (Math)3600)) Lu et al. (2021a)
9734
GeoMVerse (60K))360K) Kazend et al (2023)
9303
20
GeoQA+ (MathV360K) Chen et al. (20226)
17172
6
MapQA (MathW360k) Chang et al. (2022)
5235
MathiQA Amini et al. (2019)
29837
CLEes Li ot al. (2023b)
8652
2
TabMWP La et al. (20255)
45184
2
UniGeo Chen et al. (2022s)
11959
6
GQA Hudson & Manning (2019)
72110)
LRV Normal Liu et al. (2023b)
10500
RAVEN Zhang et al. (2019)
2100
Visual Genome Krishna et al (2016)
86117
7,8
General OCR 281K)
Chrome ChromeWWiting Wendler (2023)
8835
21
HME100K Yuau et al, (2022)
74502
21
HIT15K Mishra et ul. (2012)
20000
22
IAM Marti Bunke (2012)
5663
22
K12 Printing
12832
OCR-VQA Mishra et al. (2019b)
80000
Rondered Text Wondler (2023)
10000
22
Synth EN Kim et al. (3022)
40003
16
Taxt Sidoroe et al. (2020)
21952
9
TextOCT- Carter (2024)
25114
11
Pure Language (450K) (14.3%, 647K)
Magpie Pro Xu (2024b) (L3 MT) Magple Pro (Qwen2 ST)
1/19999
Magpie Pro (L3 ST)
150000
1449996
Table 16: The detailed statistics of Single-Image datasets used in LLaVA-OneVision Prompt ID denotes the ID of Formatting Prompt which is corresponding to the ID in Table 18. denotes no fromatting prompt is used.
40

---

Published in Transactions on Machine Learning Research (02/2025)
Dataset
Samplee
Promut
Dataset
Sumples Prompt ID
Milti-imago Sconarios
Spot-cho-Dif Jhautee Berg-Kirkpartikk (20188)
10.8K
Birdsst- Words Forbes al. (2019)
14.3K
CLEVR-Chagg Park al. (2019); Hosseinndee & Wang (2021)
HQ-Edit-Dit
(2024)
7.0K
MagieBrush Did Zhang et al. (2024a)
6.7K
4
lEdit Taa et al. (2019)
3.5K
19
AESOP Raci
(2021)
6,5K
FlintstonsSVV Cupto
(2018)
22.3KK
PeroroSV
(2019)
VIST Ting-Hoo
(2016)
WebQA Chong et al. (2021)
9.3K
TQA (MI) Kecabhard al. [2017s)
8.2K
OCR-VQA (MJ) Mishan al. (2009s
1.5K
DocVQA (MI) Mathow
(2021)
1.9K
Zhann
(2010)
MIT-SSate Coherroon
(2016)
HHT-PPooeetyCoheeeeee Isola al (2015)
1O9
12
RecipeQA Cobermmee Yageiogla al. (2018)
8:
14
VISION Bai al. (2023a)
9.52
Multi VQA Li (20246)
looQA al. (20216)
Co-lustruet
(2024)
DreammSim Fu et al. (2023)
15.9K
ImapeCoDe Krojen et al. (2022)
16.60 66.6K
muScemes Coesar
Shridhaz (2020)
9.5K
ScanQA Azuma et (2022h)
25.6K
(2020)
ContristCaptiin Tiang (2004)
VieWis (MJ) Gonari al. (2018)
4.K
Sown Dai et (2017)
49.9K
COMICS Dialogue
(2017)
6.9K
NLVR2 Saht (2015)
86K
(Viden)
NExT QA Xiso all (2021)
9.5K
2
ActivityNet Ya et al. (2019)
6.5k
Ego-4D Grouman (2322)
D.KK
Charodes Sigurdsson (2011)
23.6K
YouCook2 Zhon al. (2017)
SharoGP
(2024))
Table 17: The detailed statistics of Multi-Image and Video datasets used in LLaVA-OneVisse Prompt ID denotes the ID of Formatting Prompt corresponding to the ID in Table 19. denotes no fromatting prompt is used. "MI" means it is the multi-image version dataset from DEMON Li et al. (2024e).
41

---

Published in Transactions on Machine Learning Research (02/2025)
ID
Type
Postion
Prompt
1
VQA
Tail
Answer the question with single word (or phrase)..
2
VQA
Head
Hint:
Please answer the question and provide the final answer at the end. Answer the question with Yes or No./ Yes or No?/...
3
VQA (Yes/Nn)
Tail
Choice
Answer the given letter directly
Choice (Option Letter)
Tail
Answer with the option letter from the given choices directly. Please answer.
respond with only the letter the
6
Choice (Option Letter)
Head
Hint: Please answer the question and provide the correct option letter, A, B, C, D, at the end.
7
Region Caption
All
Provide a short description for this region. Provide the hounding hox coordinate the region this sentence describes. Provide a one-sentence caption the provided image./ a compact narrative representing the image presented./.. Summarize the main components in this picture./Provide detailed count of this scroenshot./..
8
Grounding
All
Breif Caption
All
10
Screen Summarization
All
11
Detailed Caption
All
Describe this image in detail. /Explain the visual content of the image in great detail.
12
Science Books
All
Here is diagram figure extracted from some Grade 1 science books. JPPleaee first describe the content this figure detail, including how knowledge visually displayed in the Biagram-\nThee with section title \rrelated knowledge:) briefly and concisely highlight the related domain knowledge and theories that underly this diagram. Note that you do not need to provide much detail. Simply cover the most important concepts. Provide the requested information directly.
13
Information Extraction
Head All
14
Graph Sumerization
Please clarify the meaning conveyed by this graph./Explan what this graph is communicatingg//
Photo Sumarization
All
significant elements this photo./ Mention couple crucial points in this suapshot./...
Highlight.
16
Chart Sumarization
All
What insights can be drawn from this chart? bhart?/Explain the trends shown this chart./...
17
OCR.
Head
OCR this image section by section, from top to hottom, and left to right. Do not insert line breaks in the output text. If a word is split
due to line break in the image, use a space instead Dissect diagram, highlighting the interaction between ele- ments./Interpee the system depicted the diagram, detailing component functions./..
Diagram Linkage
All
19
Code Generation
All
Compose the HTML code to achieve the same design this screenshot. First perform reasoning, then finally select the question from the choicrss in the following format: Answer: xXx.
20
Choice (with Reasoning)
Head
21
Math Computing
Tail
Round computations to 2 decimal places.
LaTeX OCR
All
Please write out the expression of the formula the image using LaTeX format.
Text Reading
All
What is written the image? Auswer this question using the text in the image directly./ /Read and list the text in this image. Please provide your answer stating the letter followed by the full option.
24 Choice (Full Option)
Tail
Table 18: The information of formatting prompts for Single-Image data. The "Position" means the position of the formatting prompt in the prompt where "All' means the formatting prompt is the prompt. Some- times, there are multiple prompts of the same meaning. In this case, the prompt column is fomatted "Prompul/Promp12/...
42

---

Published in Transactions on Machine Learning Research (02/2025)
ID
Type
Postion
Prompt
Video
Choice (Option Letter)
Tail
Auswer with the option letter from the given choices dizecsly. / Please respond with only the lester the carrect answer.
2
Choice (Full Optionn
Tail
Please provide your answer by stating the letter followed by the full option.
Multi Image
Open-Radedl
Head
What's & the differense botwoan 2 imagys?
Open-Euuded
Head
Given the stories paired with the first several images, can you finish the story based on the lust inag??/WWn /With the narratives paired with the initial imagis, how would you conelado the story using the lass picture?//..
5
Multi Choice
Head
Here is a Raven's is Progresaive Matrice in three- byree-b--hhree form. You are provided with the first eighat elements in eight imagee, please select the last one from four choioss following the structural and analogical relations.
6
Multi-Chaire
All
There are sem presibbe explarations irms for the sen different answers a VQAc ... will give you two sets of pictures, questions, and answers detertaine they belong the same QQession-Auwwe Differences You must choose your answer from Choice This 3D scenario.
T
Open- Baud-E
Head
Open-Eaded
Head
will give you several images and question, your job is seek information the slide and question cornert ly. (aa/e an images, pless unewer the following question. ... Provided with a series of diagrams from textbook, your responsibility is to correctly answer the following
the
9
Multi-Chaice
Head
queotion. You must choree your anower from the Choice Lise. /Using selection of textbook disgramns, your task is to provide an accurate respouse to the subeequeni query. You must chouse your answer from the Choice List./-.
Open, Ended
Given six images taken the depicted socne. You must choose your answer from the Choice Last. /Upon receiving six photograpla captared from varions camerns on stred--view our, to questions about
different cameras on street view car, your
to anewer questions about
reponaibilityy is to provide accorate responees socne. You must choose your answer from the Choice List. f...
11
Multi-Chaice
Head
will providle you with two sets of picturte, each of which shows an object in the ogpasite state. Can you tell me if the states of these Choice List. /Il have two sets pictures that show an object in opposite states. Can you tell states of these two sets of pictures are the same? You must choose your answer from the Choice List. /... Are the fallowing four the fillowing four images belong, to the same estepory? You must chocee your unswer from the Choice List. /...
sete of pictures
the same?? You must chouse your
from the the
12
Multi-Chaice
Head
of the cless? You
choose your
from the Chelen List.
13
Multi-Chaice
Head
Are these two workpieees same typei!/ /Are these workpieces of the some kindf Presented with a textual recipe tutorial, your task is to scrutinize it carefully and select the imnage that is incohereat in the provided sequence of linages. You must choose your answer from the Choice List. /Given text based recipe guide, your responsiility to meticulously review it ideatify the image that docsn't fit in the following sequenoe of images. You must chocse your answer from the Choice List. for
14
Multi Choice
Head
Multi Choice
Head
will give you series comnc panels. The dialogue box of the last panel is masked, the must releveant one from the candidates? You must chocee your answer from the Choice List, /Given previous full pancls and masked panel. your job four candidates. Give you a main goal, your job is to figure out what to do now by looking at current onvirments. Your post views well decisions also provireed /Given primary objective and your current suroonddnngs, use your previous decisions and perppectives to determine your next move, f...
you choose
select the most appropriate cialogue among must choose your ansuce from the Choice List. f...
16
Open-Endded Eruded
Head
17
Muiti Choice
Head
will give you two pictures of the book cover. Please look at the pictures and answer question You must choose your answer
the Choice Liss, /t will provide you with two imagess the book cover. Please examine the images and answer question. You must choose your answer from the Choice List. f... will give you some pictures, and each group of pictures will correspond to question. Please answer brielly. You most choose your answer question. Please give a short answer to it. You must chooee your answer from the Chaicn List. /... Please give edising Request describe the trausformstion from the image./W /What image?/...
18
Multi-Choice
Head
the Choice Liet.
each group pictures, there
19
Open- Eauded
Head
image to the target correct image odit instruction that can transfrom the source image to target
20
Eade-
Head
What's is the difference between 2 images? /Identify the alterations between these images. f... What's the differenoo betworn birds? /lidnnify the alterations between these two birds. /... What's la the difference between 2 innges? /Identify the alterations between there two images. f... Glivnn the stories pairod with the image?/ /With the marratives paired with the initial images, how would you conclude the story using the last picture!/ f... Cliven the stories pairod with the first several images, you finish the story based the last image?? ?/ /With the nazratives paired with the initial images, how would you conclude the story using the lost, picturvi!!...
Open-Randol
Head
22
Open- Enided
Head
Open-Raded
Head
images, you finish the story based
Open-Eadedl
Head
Oper- Budee
Head
Given the stories paired with the first several images, can you finish the story based on image?? /With the nazratives paired with the initial images, how would you conclude the story using the last picture?/...
Culsi-Chaice
Answer the following maitiple-chotce question: Here statement deserthing 2 images: ... true
false?
Table 19:
The information of formatting prompts for Oicionsion data. The "Position" means the position of the formatting prompt in the prompt where "All' means the formatting prompt is the prompt. Sometimes, there are multiple prompts of the same meaning. In this case, the prompt column is fomatted as Prompt1/Promptt:/
43

---

Published in Transactions on Machine Learning Research (02/2025)
C.4 Policy Information and Reproducibility
We will open-source most of the public datasets we used. These images and data are already publicly available for academic research; we incorporated them and converted the format for our use. However, a small portion of our data sources related to user data and those obtained using the Azure OpenAI Service cannot be diroctly released due to company policy. We will provide the exact data YAML files used in the final reproduction scripts and will offer reproducible experimental scripts, training logs, and final version checkpoints using fully public data as our compute resources allow.
44


Reviews:
[{"review_id": "mh9cjWH3xI", "reviewer": "Reviewer_xVoN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. A family of LLaVA-OneVision models scaling from 7B to 72B size. \n2. A comprehensive curation of 3.2M single-image and 1.6M multi-modal training samples, with careful consideration of data quality and task distribution\n3. Comprehensive experiments on all the 3 major vision scenarios, single-image, multi-image, and video understanding. And LLaVA-OneVision shows strong results\n4. Insights about the transfer learning in different settings.\n\n**Weaknesses**\n1. The paper lacks statistical details of the curated datasets. For example, the token length distributions for single-image datasets, the number of multiple images in the multi-image datasets, and the video statistics time ranges, etc. The existing detailed information in Table 16 only contains the number of samples and their prompt template ID but missing these important information to help people better understand the datasets's advantages and disadvantages\n2. Lack of training details of LLaVA-OneVision models, such as GPU hours, and number of GPUs, etc.\n3. While section 7 indeed talks about some transfer ability of LLaVA-OneVision in different scenarios, some of them might lack comprehensive evaluation to benchmarking the abilities. For example, Video-to-Video Difference, set-of-mark prompting and visual prompting might lack specific benchmark for the ability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. This paper presents LLaVA-OneVision, a family of large multimodal models that aims to achieve strong performance across single-image, multi-image, and video understanding tasks through a unified architecture. The key contributions include:\n2. The design of higher AnyRes with Bilinear Interpolation is a good technique that enables more vision information being processed by the LLM. This streamlined architecture design enables effective transfer learning between modalities, particularly leveraging strong image understanding to benefit video tasks.\n3. A comprehensive curation of 3.2M single-image and 1.6M multi-modal training samples, with careful consideration of data quality and task distribution.\n4. Detailed transfer learning analysis of LLaVA-OneVision in all kinds of settings.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please provide more details of the curated training datasets as stated in the weaknesses.\n2. Please provide the training details like GPU hours, and other information, if allowed.\n3. In the second paragraph of the related works, \"For the latter, our are motivated by FLAN ...\" seems to a grammar error.\n4. It seems number of tokens per frame is 196, which is lower than the number of tokens per image in the multi-image scenarios. Are there any explanations of this design and the experiments supports the explanations?", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "cATf8FQxYE", "reviewer": "Reviewer_UqMZ", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n1. The work and newly collected data will be open-sourced which are big contributions to the community.\n2. The experimental results show that LLaVA-OV perform competitively against existing open-sourced LMMs and even GPT-4V/-4o. Its performance on video benchmarks is particularly strong. \n3. The transfer task capabilities show great potentials of LLaVA-OV and also provide inspirations for future research.\n4. The paper is clearly written and well-organized.\n\n\n**Weaknesses**\n1. There are lack of ablation studies to verify the contributions of the newly proposed components, such as Higher AnyRes strategy, newly generated data, more use of data, and the curriculum learning during pretrainnig. However, it is also understandable that such ablation studies are too resource-wise intensive.\n2. On Table 3, though LLaVA-OV-72B can outperform all the open-sourced models, but the comparisons are not fair because of the large size of LLM (72B). Actually, InternVL-8b can outperform LLaVA-OV-7B on lots of single-image benchmarks. Can any patterns be found from there? One speculation is that Higher AnyRes strategy leads to long sequences for single image inputs (i.e., a video is 6272 tokens, but a single image is 7290 tokens), thus may lead to lower density of information for each token. Thus, LLaVA-OV needs to aggregate information from a long sequence for single-image problems, which makes the problems possibly harder.\n3. Though LLaVA-OV has improved visual processing ability, how about the language processing/understanding after the training with multimodal data? This is a meaningful and important question for LMMs.\n4. The task transfer capabilities are generally discussed in Section 7. It will be better if a small set of data points can be collected in order to evaluate LLaVA-OV with other LMMs to further demonstrate the emergent capabilities of LLaVA-OV.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "Large multimodal models (LMMs) have advanced multimodal learning research. The recent work, LLaVA-Next, builds prototypes of LMMs and brings insights about data, models, and visual representations for LMMs. Inspired by these insights, this work introduces LLaVA-OneVision (LLaVA-OV) which can handle more variations of visual inputs (e.g., table/chart/multiple videos, etc.). LLaVA-OV processes different visual inputs (images/multi-images/videos) via the proposed Higher AnyRes strategy. The model is pretrained on a mixture of existing datasets and newly generated datasets with a curriculum learning approach. LLaVA-OV is evaluated on LMMs-Eval. The experimental results show that LLaVA-OV-72B can outperform the existing open-sourced LMMs and reach the performance between GPT-4V and GPT-4o on majority of the benchmarks. At last, the work discusses that LLaVA-OV emerges several task transfer capabilities on more versatile visual inputs.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. [Minor] There is only one line of content on Page 4. It will be better if this page only shows the two existing figures.\n2. [Minor] Missing citations for VQAv2, GQA, VG on Page 6 (in Section 4.2).\n3. [Minor] What is “LCS” in Table 1?\n4. [Minor] There are many digits to read into on the main result tables. Is it possible to summarize some of the results with average scores?\n5. [Typo] “responces” -> “responses” on Page 8\n6. [Typo] “the the” -> “the” on Page 9", "broader_impact_concerns": "There are no concerns on the ethical implications.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "a35A2d10E3", "reviewer": "Reviewer_YF8X", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n\n* The model demonstrates strong improvements over existing models and across various scales.\n* The paper is well-structured and clearly presents the stages involved in building such a model.\n* Extensive evaluation across many benchmarks provides a comprehensive view of the model's capabilities.\n\n**Weaknesses**:\n\n1. My main concerns about this paper is that it focus only on showing the final model scores, without providing sufficient ablation studies or analyses on main parts of the training pipeline, for example:\n\n- How does the data mixture used here compare to other datasets, such as Cambrain or Cauldron? Is there a strategic assembly of the training mixture, and how does each dataset contribute to the final model’s performance?\n- The model is trained in four stages. Are all stages essential, or could the first or second stages be omitted without significant loss in performance?\n- Is training the vision encoder necessary, and if so, to what extent does it impact the results?\n- The authors selected Qwen as the LLM —are the results sensitive to this choice? It would be valuable to understand the impact of different LLMs on final scores.\n- Each stage is trained for only one epoch—is this due to performance saturation after the first epoch?\n- Performance comparisons between the Anyres and Higher Anyres models are not addressed. \n\n2. A primary contribution is the model's task transfer abilities. However, this has been a topic in several prior works, which are not cited here. For instance, [1] examines task transfer from image to video, incorporating curriculum learning similar to the approach here. Citing and discussing relevant literature is important to properly position the paper within existing research.\n\n[1] Shukor, Mustafa, et al. \"UnIVAL: Unified Model for Image, Video, Audio and Language Tasks.\" TMLR 2023.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces a strong family of large multimodal models (LMMs) up to 72B parameters, outperforming most current baselines. The core contribution is in training a single model that excels across single-image, multi-image, and video tasks. The paper provides details on training processes, data curation, and extensive evaluation results.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please check the weaknesses section. The work missing ablation study to understand how the model can attain such performance.", "broader_impact_concerns": "The model is a text generation model and based on LLMs, it can generate harmful or toxic text if there is no explicit mechanism to prevent it from doing so.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In this paper, the authors presented a family of large multimodal models (LMMs), termed LLaVA-OneVision, achieving strong performance across single-image, multi-image, and video tasks. A higher AnyRes strategy was proposed. Experimental results show the effectiveness of LLaVA-OV, outperforming most existing open-sourced LMMs. Transfer learning analysis was also presented.

This paper received detailed review comments from three expert reviewers. There were some concerns about the lack of details (e.g. the curated datasets, training details of the models), lack of a detailed ablation study on the key contributing components, experiments (e.g. fair comparison, small set of data,  comparison to other datasets, sensitivity, etc.), and missing discussion/comparison to related works. After the rebuttal, most of these concerns were well addressed and acknowledged by the reviewers. During the discussion phase, all reviewers agreed on the strengths of this paper, including the introduced family of LMMs, the open-source curated data, comprehensive experimental evaluation, and the strong performance. Finally, this paper received a consistent positive recommendation of 2 Leaning Accept and 1 Accept.
Considering the above, the AE is pleased to inform that this paper is accepted to TMLR, subject to the revision of including the additionally added supporting evidence (e.g. the ablation studies, experiments, discussions, etc.) during the rebuttal phase.
Decision: Accept (poster)