A Single Transformer for Scalable Vision-Language Modeling
Authors: Yangyi Chen
Xingyao Wang
Hao Peng
Heng Ji
Venue: Accepted by TMLR ()

Abstract: We present SOLO, a single transformer for Scalable visiOn-Language mOdeling. Current large vision-language models (LVLMs) such as LLaVA mostly employ heterogeneous architectures that connect pre-trained visual encoders with large language models (LLMs) to facilitate visual recognition and complex reasoning. Although achieving remarkable performance with relatively lightweight training, we identify four primary scalability limitations: (1) The visual capacity is constrained by pre-trained visual encoders, which are typically an order of magnitude smaller than LLMs. (2) The heterogeneous architecture complicates the use of established hardware and software infrastructure. (3) Study of scaling laws on such architecture must consider three separate components — visual encoder, connector, and LLMs, which complicates the analysis. (4) The use of existing visual encoders typically requires following a pre-defined specification of image inputs pre-processing, for example, by reshaping inputs to fixed-resolution square images. This inflexibility can create bottlenecks and impede scalability. A unified single Transformer architecture, like \approach, effectively addresses these scalability concerns in LVLMs; however, its limited adoption in the modern context likely stems from the absence of reliable training recipes that balance both modalities and ensure stable training for billion-scale models. In this paper, we introduce the first open-source training recipe for developing SOLO, an open-source 7B LVLM with the single Transformer architecture using moderate academic resources (8 x A100 80GB GPUs). The training recipe involves initializing from LLMs, sequential pre-training on ImageNet and web-scale data, and instruction fine-tuning on our curated high-quality datasets. On extensive evaluation, SOLO demonstrates performance comparable to LLaVA-v1.5-7B, particularly excelling in visual mathematical reasoning.

Published in Transactions on Machine Learning Research (11/2024)
SOLO: A Single Transformer for Scalable Vision-Language Modeling
Yangyi Chen, Xingyao Wang", Hao Peng, Heng
University of Illinois Jrbana-Champaige
Suingic3,xxigyaaee haapyea,,,,aaaan....
haopeng, heopejn,,ennjiie httpgn..innnnnniiiiiie.eee iiiliiniisii
Reviewed OpenReview: http::///ppppeoe..e..eee////ettt
Abstract
We present SOLO, a single transformer for Scalable LisiOn-Language mOdeling. Current large vision-language models (LVLMs) such as LLaVA mostly employ heterogencous architectures that connect pre-trained visual encoders with large language models (LLMs) facilitate visual recognition and complex reasoning. Although achieving remarkable performance with relatively lightweight training, we identify four primary scalability limitations: (1) The visual capacity is constrained by pre- trained visual encoders, which are typically order of magnitude smaller than LLMss. (2) The heterogeneous architecture complicates the use of established hardware and software infrastructure. such architecture must consider three separate components and LLMs, which complicates the analysis.
(3) Study of scaling laws on visual encoder, connector, (4) The use of existing visual encoders typically requires following pre-defined specification of image inputs pre-processing, for example, by reshaping inputs to fixed-resolution square images. This inflexibilityy can create bottlenecks and impede scalability. A unified single Transformer architecture, like SOLD, effectively addresses these scalability concerns in LVLMs; however, its limited adoption in the modern context likely stems from the absence of reliable training recipes that balance both modalities and ensure stable training for billion-scale models. In this paper, we introduce the first open-source training recipe for developing SOLO, an open-source 7B LVLM with the single Transformer architecture using moderate academic resources (8 x A100 80GB GPUs). The training recipe involves initializing from LLMs, sequential pre-training on ImageNet and web- scale data, and instruction fine-tuning evaluation, SOLO demonstrates performance comparable to LLaVA-v1.5-7BB, particularly excelling in visual mathematical reasoning'.
curated high-quality datasets. On extensive
1 Introduction
Large vision-language models (LVLMs) demonstrate remarkable performance downstream tasks (Li et al., 2023c; Zhu et al., 2023; Liu et al., 2023c; Chen et al., 2023c; Kim & Ji, 2024). They can effectively extract. visual information (Wang et al., 2023b) and follow human instructions to generate insightful responses (Li et al., 2023b; Chen et al., 2024d). Two established approaches for vision-language modeling include: (1) Connecting pre-trained visual encoders (Dosovitskiy et al., 2021b; Radford et al., 2021) and large language modcls (LLMs) (Touvron et al., 2023; Jiang et al., 2023) via a Icarned projection module that maps the visual embeddings the embedding space of LLMs (Dai et al., 2023; Gao et al., 2023; Liu et al., 2023c), intermediate symbolic layer Wang al. (2024c). (2) Leveraging pre-trained visual encoder to extract features and aligning feature embeddings with a pre-defined codebook (Esser et al., 2021) to convert each image into sequence of discrete visual tokens, thus enabling LVLMs to process both images and language tokens (Wang et al., 2022b; Peng el al., 2022; Anil et al., 2023; Team, 2024; Diao et al., 2023).
* Eqqual contribution.
The code made public at https://github: Saam/Yaagy--aa Chen/SOLD.

---

Published in Transactions on Machine Learning Research (11/2024)
OUR WORK
Large Language Model
Large Language Model
Large Vision Language Model
Image Tokens
Instruction Embeddingg
Image Embeddings
Instruction Embeddlings
Embeddlings
KSOUK seps
Instruction
Image Embedcling
Embecdirg
Codebook
Multi-Layer Perceptron
Transform to Embeddings
Linear Projector
I2
Discrete Tokens
Transfor
Image: Fmhherddingg
Split
Resize
Visual Encoder
PREVIOUS WORK
Figure (Previous work) The reanstream approaches for vision-language modeling rely on pre-trained visual encoders for visual feature extraction, which exhibits scalability limitations. (Our work) We advocate for unified transformer architecture that processes both images and text, employing simple lincar projection to directly handle raw image pixels. <vision>, </vision>, and <vrow_sepi _ep__ep>>o> are special tokens designed explicitly for visual modality encoding.
However, despite their effectiveness, these approaches have limitations that make them hard to scale. We define an architeclure as scalable when it exhibits consistent performance improvements as compulational resources and training data increase, maintaining a positive scaling law relationship up to practical limits. This is in contrast to architectures that show diminishing returns or performance plateaus at larger scales due to fundamental architectural limitations such as information bottlenecks (í.e., structural limitations in information transmission) in visual perception. The scalability limitations in prevalent LVLMs evident. across four dimensions ($2.1), primarily due to their reliance on pre-trained visual encoder:
(1) Constrained visual capabilities: The visual capacities of pre-trained vision encoder are largely pre-determined and limited by the data distribution and volume used during pre-training. Due to the significantly smaller size of visual encoders- approximately over ten times smaller than LLMs-they can be a performance bottleneck in solving complex vision-language tasks.
(2) Challenges in efficient training and deployment: The heterogeneous architecture of LVLMs with vision encoders complicates adaptation standard frameworks and hardware optimized for unified Trans- former architectures, resulting in reduced computational efficiency in terms of training and inference speed.
(3) Multiple components complicate the scaling analysis: The analysis of scaling laws, which are crucial for the development of foundation models, is complicated by the necessity to consider the size of several distinct components independently: the visual encoder, the connector, and the LLMs.
(4) Limited image pre-processing flexibility: Most vision encoders pre-define a specification on how image inputs should be pre-processed. For example, the widely used visual backbones, such as CLIP--VTT-36 (Radford et al., 2021), require a square image input with a resolution of 336 336. The inflexibility of image pre-processing can cause boutlenecks that hinder scalability.
To address these limitations, present SOLO, which employs single Transformer architecture for unified and end-to-end vision-language modcling. SOLO accepts both raw image patchos (in pizels) and lexts as inputs, without using a separale pre-trained vision encoder (Fig. 1). This simplifies the model design and enhances the scalability and adaptability the LVLM architecture. By simplifying from multi- component LVLM to single Transformer model, this architecture unconstrained on the capabilitics of visual encoders, casier to train and deploy using existing hardware and software, allows more straightforward scaling law analysis, and can easily scale to image data with diverse resolutions and aspect ratio. SOLO, with
2

---

Published in Transactions on Machine Learning Research (11/2024)
7-billion parameter count, is initialized from Mistral LLM v0.1 (Jiang al., 2023) and leverages its extensive pre-trained knowledge.
This modeling strategy is inspired by the foundational modeling framework of VisualBERT (Li et al., 2019) and industry efforts to scale unified LVLMs to the billion-scale (Bavishi et al., 2023). Despite the simplicity and scalability, its limited contemporary adoption can be attributed to the lack of reliable training recipes, as balancing vision and language modalities in unified LVLMs often leads to training divergence. This paper details the first open-source recipe for developing scalable unified LVLMs, using modest academic computational resources, specifically 8 NVIDIA A100 80GB GPUs ($3). Our training recipe involves initializing with pre-trained LLMs, sequential pre-training on ImageNet and web-scale datasots, and instruction fine-tuning on our curated high-quality data mixture.
While still lags behind recent state-of-the LVLMs on evaluation benchmarks, SOLO exhibits performance on par with LLaVA-v1.5-7BB (5-) and the variant LLaVA-7B' (86), which is created following our training recipe in the controlled setting. In particular, SOLO distinguishes itself in the domain of visual mathematical reasoning. Further scalability analysis reveals SOLO's better scaling behaviors, inference speed advantages, easier scaling laws analysis, and the scalability and benefits of our flexible image preprocessing pipeline ($6.2). In addition, through comprehensive ablation studies, we validate the design choices of our training recipe. Our empirical results confirm that the sequential pre-training on ImageNet and web-scale datasets and instruction inne-tunng on our carefully curated data mixture are both essential for the training of such single Transformer LVLMs (85). Interestingly, we find that after removing the first stage of pre-training on ImageNet, the LVLM will produce outputs of drastically different quality while exhibiting similar image-conditioned language modeling loss (55.1, Fig. 3).
2
Tackling Scalability Limitations via Integrated Architectures
2.1 Scalability Limitations in Existing LVLMs
The scalability constraints of existing LVLMs are currently articulated from four critical perspectives that limit their efficiency in utilizing expanded computational resources and larger datasets due to the bottlenecks in the system design:
Fixed and Constrained Visual Capabilities The fixed nature of visual encoders severely limits the adaptability of LVLMs to novel visual data distribution and more complex vision-language tasks since these encoders are trained on specific distributions and training objectives. Current approaches address this issue by contimuing the training of visual encoders (Bai et al., 2023) or by integrating features derived from various visual encoders (Lin et al., 2023). Nonetheloss, the scope of data used for continued pre-training is substantially less than that used initially, which only marginally enhances encoder adaptability, and employing multiple encoders complicates the process of image feature extraction, thereby impeding the scalability of LVLMs. Moreover, the smaller scale of visual encoders compared to LLMs frequently results in the visual understanding component becoming bottleneck. Consequently, visual representation lcarning is limited to the smaller visual encoders, hindering the full utilization of LLM capabilities in existing LVLMs.
Challenges in Efficient Training and Deployment The heterogeneous architecture with multiple components complicates the implementation of machine learning systems for efficient training and deployments in terms of speed. (1) Training challenge: At large training scale (i.e., multi-node clusters), it is necessary to distribute not only the Transformer-based LLMs but also the vision model and the MLP connector across multiple devices, employing techniques such as tensor and pipeline parallelism. Thus, prevalent LVLMs cannot directly use existing industry-grade training frameworks optimized for the Transformer architecture (Shoeybi bi et al., 2019; Cano et al., 2023), thus necessitating the development of new tensor-sharding mechanisms. In ion, Al aligument typically employs algorithmss such as Proxinal Policy Optimization (PPO) (Schulman et al., 2017), which necessitate simultaneously maintaining multiple models (e.g., reward and critic models) in GPU memory and cause difficulty in the algorithm implementations for heterogencous architectures. (2) Deployment: The heterogeneous architecture complicates the deployment process due to similar model and tensor sharding challenges described above. Consequently, this hampers the large-scale services of existing
3

---

Published in Transactions on Machine Learning Research (11/2024)
LVLMs. Moreover, existing specialized AI chips (Techerunch) and inference libraries, such vLLM (Kwon et al., 2023) and MT.C-I.LM (team, 2023), are mostly optimized for Transformer architectures, presenting significant challenges in the deployment of these models on end devices.
Multiple Components Complicate the Scaling Analysis The complexity introduced by the multiple components of LVLMs a significant barrier to understanding and improving these systems. Each component the visual encoder, the connector, and the language models operates with its own parameters and training strategies (Radford et al., 2021; 2019; Brown et al., 2020a), which can lead to lack of cohesion in the overall model behavior. Scaling laws are crucial for guiding the development of large foundational models by forccasting the performance of a target model using data from several sampled models that are significantly smaller in sizes (Kaplan et al., 2020; Bahri et al., 2021). However, applying these approaches to existing LVLMs requires simultaneous consideration and scaling of various components, hereby increasing complexity.
Limited Image Pre-Processing Flexibility The strict requirements for image pre-processing imposed by the specifications of visual encoders may create bottlenecks that hinder the scalability. For instance, the requirement of consistent input resolution can make it difficult to process images that are naturally high-resolution or have non-standard aspect ralios without compromising on the quality or representational fidelity of the input. Current mitigation strategies involve splitting the original image into multiple sub- images, independently extracting visual features from each sub-image using pre-trained visual encoders and subsequently aggregating the representation embeddings (Xu et al., 2024a; Liu et al., 2024a; Dong et al., 2024). However, these approaches seem ad-hoc and can be suboptimal as the visual backbone is not pre-trained to handle these inputs, potentially impacting the effective handling of these high-resolution images.
2.2 Unified Vision-Language Modeling with Integrated Architectures
We revisit the foundational modeling framework of VisualBERT (Li et al., 2019), initially proposed in the early stages of research on pre-trained vision-language models. The key idea to use one single Transformer, initialized from BERT (Devlin et al., 2018) in VisualBERT, to uniformly process the image patches and language tokens. Fuyu-8B exemplifies the industry's 's effort to scale this modeling approach (Li et al., 2019) to billion-scale models (Bavishi et al., 2023). However, the limited widespread implementation of this unified architecture may be due to the lack of an established training recipe, as only the pre-trained model is released by Bavishi et al. (2023) without training details. Training such unified LVLMs presents significant challenges in balancing the two modalities and maintaining stable training, for which clear solutions are currently lacking. In this paper, we present SOLO with full details of its unified and integrated architecture design and training recipe.
3 SOLO: Scalable Vision-Language Modeling
SOLO consolidates image and language capabilities into single model, enables data-driven determination of visual representations and parameter allocation across visual and language modalities, simplifies the scaling laws analysis, and allows it to handle high-resolution images and those with uncommon aspect ratios flexibly. For large-scale training ($3.2), SOLO also seamlessly integrates with established software frameworks for large-scale Transformer pre-training (Shoeybi et al., 2019).
3.1 Model Architecture
The architecture of SOLO is shown in Fig. 1, which diverges from earlier models primarily in the extraction of visual features. Instead of resizing the image into a fixed resolution adapted to the pre-trained image encoders, SOLO keeps their original resolutions and aspect ratios. The feature extraction involves splitting the image into patches with a pre-defined size. Through a trainable linear projection, these raw image patches (in pinels) are transformed to obtain continuous embeddings that represent the visual features of the images. Thus, integrate image and language processing within single model. We maintain a list of special tokens designed explicitly for visual modality encoding: <vision> and </vision> tokens mark the begimning and end of a span of image patches respectively; <vrow_ saopssseppp__ acts as a row soparator within the image patches and helps the model distinguish between different rows of image patches, aiding in structured visual understanding.
4

---

Published in Transactions on Machine Learning Research (11/2024)
Table 1: Summary of datasets used in the three stages of pre-training. Each image patch counts as a vision token. Number of instances is calculated after packing them into sequences with 32K length.
Training Stage
Dataset
(llustances
Linnge
#'Token
# 'ookezs
Vistion
InageNer21KK (Ridmik et al., 2021b)
74, 263
13, 151, 276
2,425, 203, 108 4,340, 877,887
212, 745, 573
2,210, 2,210,657, (6,657,,55 535
Stage-1
SllmPajamm, subset (Soboleva al., 2023)
120,
4,340, 577.587
Total
122
6,704,
(07.32%)
4,553, 623,
(32.085)) 2,210,455 13100,47,540
Caprasion (oulwrt) (Yu st sl, 2024)
204. 978
23. 681 864
064 361, 863
122, 725, 506
1,272, 726, 505
Welnight (Launagou ot sl., 2024)
71, 579
1. 02222, 671
20300, 945, 215
1,067 060, 511
1,215, 884, 7044
CCSM (Sharms al., 2018k)
$0, 760
3,331,
477,311
055,
Detailed Captions (Ix)
225
285, 767
202, 016, 770
44. 758, 200
157. 228. 570
Stage-2
LLaVAR (Zhong et al,, 2025b)
3,002
422, 315
117, 488, 786
310, 556
86, 058, 228
DVQA (Kalle
2013)
2.917
010
853,
650,
200,
OCR-VQQ (Mishins
2010)
1,588
165, 746
51, 920,
21, 161,
90, 789, 087
VigureQA (Kalson ot al., 2017)
1,526
100), 000
491, 686, 3015
24. 4000 256
24, T8S, 1449
Pajaman, different aobret (Seinoleon al, 2023)
120,355
4,300, 998, 161
4,500 968, 161
465,365
14,846, 208,9131
(43.9095)
6,814,674.
(56.1115)
9244
ALLAVA-LAIO (Clse et al, 2024a)
13, 725
4118, 1912
4412, 509, 4201
170, 000, 8088
2100, 8688 5:12
ALLLVA-YLFLAN (Nu ei 202(1))
4,469
207, 549
1444, 317
77. 07.835, 9119
66, 741, 4558
LLVAR (Zhong
2025b)
Stage-3
DVQA (Kalle et ol., 2018)
2,917
200, 000
94, 863, 706
65, 66-653,
50, 200, 000
PigureQA (Kahom et al., 2017;
1,526
100, 000
39, 566, 305
24, 800, 256
24, 783, 019
Slluaajamm, different auboct (Soboleva
12,085
688,
Ge8,
Total
388, 824
1,279, 664, 196
(02.2896) 797, 0012
(97. (87.7256) 482, (183)
Formally, we define the patch size P and the maximal resolu- tion M. For an image of dimension size (W, H), it is resized to (W", H") to ensure divisibility by P. Fig. details the resizing process, which adjusts the W and H to the nearest multiples of P while preserving the original aspect ratio to the extent possible and complying with the constraints imposed by M. Subsequently, the image is divided into N patches, where N = =WW/P) x (H'/P), each with dimensions P x P x 3. A trainable lincar projector then maps each patch from flattened Px P vector to an output dimension compatible with the embedding space of LLMs,, extracting N embeddings as the image's feature representation. These visual embeddings, along with special visual modality tokens and embeddings of the text tokens, are concatenated and processed through a single Trans- former, facilitating unified vision-language modeling. Notably, compared lo prevalent LVLMs, this modeling strategy facilitates much earlier fusion of visual and language modalities, allowing LVLMs to extract relevant information conditioned on the given instructions. In our implementation, we initialize SOLO from the Mistral-7B-v0.1 base LLM. The max resolution M of processed images is set as 1024. The patch size P is set 32.
PATCH_SEEE 52
MAK, RASOLUTTION 1994
dxf get. rasiza, cotpprt twwgo, hwwp.ette(ttaggrttaao 11, image_aise
requested naw, ata
inon PATCH, BEZE 13 PAZOR_ MAY AA2_0R900TTION
nsv_ recuested peapusted__rv___oo Loog usv_sbect Lsl Long abort long) new, short ton short PATOE_
PATDI_St
long,
Dev_long
The input image resize algorithm to maintain the aspect ratio.
Figure 2:
3.2 Training Recipe
We describe our approach for training unified billion-scale LVLMs, including pre-training ($3.2.1) and instruction fine-tuning ($3.2.2). For both stages, we optimize exclusively the language modeling loss on natural language tokens, without optimizing loss on image patches and special image tokens (e.g., <vision>). We substantiate the essential ingredients in our recipe in $5.
3.2.1 Pre- Training
We introduce a threc-stage pre-training curriculum that progressively enhances the visual capabilities of LVLMs while preserving their fundamental language capabilities. We present datasets and their statistics for each stage in Tab. 1.
Stage-1 ImageNet Pro-Training for Initialization We leverage ImageNet21K (Ridnik et al., 2021a), encompassing a broad spectrum of fine- fine-grained visual categories, for the initial pre-training stage. In this process, we train SOLO to predict only fine-grained labels in natural language tokens (class name of images,
5

---

Published in Transactions on Machine Learning Research (11/2024)
e.g., "golden retriever") conditioned on the image patches, thereby developing visual representations that initialize subsequent pre-training runs. In $5, we demonstrate the critical role of this stage in training unified LVLMs: when this stage is removed, the LVLM pre-trained on web-scale data from stage 2 (e.g., captioning) failed to generate meaningful captions (Fig. 4).
Stage-2 Pre-Training on Web-Scale Data ImageNet2IK, composed chiefly of visual concept data annotated by humans, faces scalability constraints in both knowledge breadth and data volume. In Stage-2, we scale up the pre-training data to encompass web-scale data, primarily consisting of image-caption pairs from sources like Capfusion (Yu et al., 2024) and CC3M (Sharma al., 2018a). Additionally, we include synthetically generated web pages with associated HTML code from Websight (Laurengon et al., 2024) to improve OCR. performance, and we also include small set of supervised datasets to improve the data diversity. In this stage, the language modeling loss is applicd uniformly across all language tokens, encompassing captions, HTML code, and questions and responses within the supervised datasets.
Stage-3 Anncaling Following MinICPM (Hu et al., 2024a), we perform a final anncaling stage to conclude the pre-training. In this stage, we incorporate a limited selection of supervised datasets either down-sampled or omitted from the instruction fine-tuming dataset mixture (e.g., ALLaVA, Chen et al. 2024a) to prime SOLO for the subsequent instruction fine-tuning stage. The primary purpose of this stage to transition SOLO from a noisy web data to being trained on high-quality data mixtures.
Balancing Text and Vision Capability Through Language Corpus Blending Initiating with a base LLM and performing full-parameters training necessitates carefully preserving its inherent language comprehension abilities while performing image representation learning since most real-world vision-language tasks require text-only capabilities such as instruction comprehension and complex reasoning. At each stage of SOLO's pre-training, we mix in non-trivial proportion of text-only pre-training data (SlimPajama, Soboleva et al. 2023) to maintain the text capability. We present more empirical results on how data mixture affects image and text loss trade-offs in $7.2.
Pre-Training Infrastructure We modify the standard Megatron-LLM (Cano et al., 2023) to support arbitrary image patch inputs. We use one node with 8 NVIDIA A100 80G GPU for pre-training. We use 2- 2-way tensor parallelism (Shoeybi et al., 2019) and 4-way data parallelism for training. Following Shoeybi et al. (2019), we adopt distributed optimizer shard optimizer states across different GPUs for memory efficiency. In our test on 8xA100 server with identical image-caption pro-training, Megatron achieves 20K tokens per second, 67% higher than DeepSpeed, making it more suitable for our pre-training requirements.
Training Hyperparameter We use global batch size of 128 examples (f.e., 4M tokens) and each pre-training example is packed to 32, 768 tokens. We adopt a learning rate of 5e-5 with cosine decay to a minimum learning rate of 5e-6 and warm up for 200 steps. We use weight decay of 0.1. For training efficiency, we pack shorter sequences into one longer sequence and re-adjust the attention mask to make sure tokens from different examples cannot attend to each other. The training process consists of 1525 steps in Stage 1, 3480 steps Stage 2, and 300 steps in Stage 3.
3.2.2 Instruction Fine- Tunnnn
Dataset Curation We meticulously select diverse range of supervised datasets to perform instruction fine-tuning, aiming to enhance their performance across various domains of vision-language tasks. Our dataset selection strategy is based on the empirical analysis derived in Laurençon et al. (2024); Lin et al. (2024); Lu et al. (2024), and is mainly driven by the objective to cover a comprehensive range of data types, including language-only data, detailed image captions, scientific documents, tables, documents, charts, OCR (VQA) tasks. In $A, we present datasets and
and text-rich images, and general visual their statistics in Tab. 7 and more details regarding data curation.
Implementation Details We utilize DeepSpeed (Rasley et al., 2020), as implemented in Accelerate (Gugger et al., 2022), for instruction fine-tuning. The choice to use Accelerate over Megatron for fine-tuning is
6

---

Published in Transactions on Machine Learning Research (11/2024)
based on practical efficiency. Accelerate enables quick experimentation with data mixtures by modifying a few lines of code, while Megatron demands extensive preprocessing for each run, complicating ablation studies. Additionally, Megatron's complex, multi-layered implementation hinders customization, presenting challenges in potential further extension, such as introducing advanced methods like RLHF for alignment. For hyperparaneters, the global batch size is configured at 640, with a weight decay parameter of 0.1. We train for 1 epoch with a maximum learning rate of 1o-5, which follows a linear warm-up phase and transitions to cosine decay schedule.
4 Comparison to Existing LVLMs
4.1 Model
We select various open-source LVLMs for comparison to better understand the capabilities of SOLO. Based on the release time and capabilities of IVLMs, we select 3 groups of LVL.ss to better understand the current development phase of SOLO. Level-1 LVLMs represent the pioneering generation, which initiate the integration of visual encoders with pro-trained LLMs, with releascs prior to October 2023. Level-2 LVLMs, released before early 2024, typically feature a more refined selection of instruction fine-tuning data to enhance performance. Level-3 marks the state-of-the-art (SoTA) LVLMs, released within the last five months, incorporating advanced training recipes, superior LLM backbones, and support for high-resolution images.
Level-1: (1) OpenFlaming, v2 (Awadalla et al., 2023), (2) MiniGPT4 v2 (Chen et al., 2023a), (3) VisualGLM (Du et al., 2022), (4) InstructBIII (Dai et al., 2023), (5) LLaVA v1 (Liu et al., 2023a).
Level-2: (6) LLaVA v1.5 (Liu et al., 2024a), (7) mPLUG-OWl v2 (Ye et al., 2024), (8) InternLM- XComposer (Zhang et al., 2023a), (9) MiniCPM-vl (Hu et al., 2023),
Level-3: (10) Monkey (Li et al., 2024b). (11) LLaVA-NEXT (Liu et al., 2024b), (12) MiniCPM-12 (Hu et al., 2024b), (13) DeepSeek-Vi VL (Lu et al., 2024).
Each LVLM may have multiple variants based on different LLM sizes and architectures. If possible, we opt for the variant equipped with a 7B Mistral LLM. For the remaining LVLMs, we select the variant whose configuration most closely aligns with our specifications Miistral-7B-LLM, We directly present the evaluation results of existing LVLMs from the leaderboard (OpenCompass) when available, to ensure fair comparison.
4.2 Benchmarks
We select a wide range of benchmarks, encompassing both general vision-language tasks and specific task- oriented datasets, for evaluation and analysis. For general languaee capability evaluation, we choose MMStar (Chen et al., 2024b), MME (Fu et al., 2024), and SEED-Bench (Li et al., 2024a). Specifically, MMStar measures elite vision-indissensablle capabilities, MME measures both the perception and cognition capabilities, and SEED-Bench covers 12 evaluation dimensions covering various aspects of LVLMs capabilities. For scientific document understanding, we choose A12D (Kembhavi et al., 2016) and ScienceQA (Lu et al., 2022a). For visual mathematical reasoning, we choose MathVista (Lu et al., 2023). We adopt VLMEvalKit (Contributors, 2023; Duan et al., 2024) to perform the unified evaluation.
4.3 Results
The experimental results are shown Tab. 2. We find that SOLO significantly outperforms Level-1 LVLMs and also performs comparably to Level-2 LVLMs, despite slightly underperforming Level-3 LVLMs. Furthermore, SOLO excels in task-oriented benchmarks, especially in areas requiring scientific knowledge and mathematical reasoning, due to its successful integration of image representation and complex reasoning within a single unified model. Overall, while SOLO does not yet meet the SoTA performance of the Icading LVLMs (Level-3) within the prevalent multi-component LVLM framework, it marks a substantial progression in unified lasion-language modeling. It is important to consider that SOLO is trained using limited academic resources, specifically 8 A100 GPUs. This in sharp contrast with the resources used to produce SoTA LVLMs. For example, the technical report of DeepSeek- VL (Lu et al., 2024) mentions that DeepSeek VL-7B is trained on cluster of 64 nodes, each comprising 8 Nvidia A A100 GPUs, totaling 512 GPUs this represents resource
7

---

Published in Transactions on Machine Learning Research (11/2024)
Table The main experimental results of SOLO. We compare with LVLMs with diverse capabilities, released different times and categorized into three levels. SOLO aligns with the second level of prior LVLMs advancements focusing on LLa VA-Style modeling, and distinguishes itself in visual mathematical reasoning. (C) denotes the visual encoders from CLIP.
Level
Model
Visual
Language
MMStta
MME
SEED
ScienceQA
MathVista
A12D
OpenPlamnngu v2
(C) ViT-L/14
MPT-7B
26.9
607.2
28.8
44.8
18.6
31.7
MiniGPTT-4-22
EVA-G
LJama2 1315
21.3
968.4
29.4
54.7
23.1
30.5
Level-1
VisuaGLLM
CLIP
ChatGGMM6R
738.1
47.0
56.1
21.9
BLIP
EVA-G
Vieuma-7B
32.7
1391.4
44.5
54.1
24.4
40.6
LlaVA-v1-7b
(C) VTT-1./14
Llama-71
27.1
1075.5
50.4
61.8
25.2
48.3
LLaVA-v1.5 7b
(C) VIT-L/14
Vieuna-V Sitmaa-V1.5-1
V1.5-BB
33.1
1808.4 4
65.8
69.2
25.6
55.5
Level-2
mPLUG-OWL v2
XCompoeer
EVA-G
(C) VTT-1.14
Llama2-713
M-77
34.8
6.9
1874.2
1786.4
64.5
66.1
69.5
89.8
29.8
25.4
56.9
55.7
MiniCPM-V
SigLIP-4000 100M
MinCCMM-2.00
38.6
1650.2
65.6
77.0
30.6
56.3
Monkey
ViT- Vuge
Qwen-BB
64.3
72.1
33.5
Level-3
MiniCPM-ve
LLxVA-Next
SigLIP S0gLIP-400M
(C) ViT-L/14
Mistral-B
MiniPPM-2.4 2.4BB
38.4
39.1
1808.2
1821.2
72.4
67.1
73.0
80.7
31.6
39.8
69.0
62.9
DeepSook-V
Hybrid
DeepSeok-715
70.1
80.9
36.9
Ours
SOLO
Misual-TB
35.5
1260.0
61.4
73.3
34.4
61.4
scale 64 times greater than that used for SOLO. SOLO serves a pivotal model, showcasing the scientific value of training LVLMs with unified architectures. This establishes SOLO viable candidate for future developments aimed at closing the performance gap with SoTA IVLMs, with more flexibility and scalability by avoiding issues in prior architectures ($2.1).
5
Validating Key Ingredients in Our Recipe
5.1
LVLMs Generate Meaningless Captions without Stage-1 Pre-training
We assess the necessity of Stage-1 pro-training by comparing the Stage-2 LVLM checkpoints with and without undergoing Stage-1 ImageNet pre-training. In Fig. 3, we observe that these two variants overall achieve similar pre-training loss curves on vision-language modeling and (text) language modeling.
Select Checkpoints for Comparison We select two checkpoints for comparison: one using caption-only pre-training (Stage-2 only) and the other utilizing SOLD's two-stage pre-training, both of which achieve an equivalent vision-language modeling loss of 2.1.
Qualitative Comparison We randomly select one example for qualitative analysis (in Fig. 4). Despite the equivalent loss of the selected checkpoints Fig. 3, we find that without ImageNet pre-training (Stage-2 only), the model generates irrelevant and meaningless image captions, indicates training divergence.
Quantitative Comparison We perform quantitative comparison the same two checkpoints by training them on the instruction fine-tuning data mixture for 800 steps (see Fig. 5a). Compared to the two-stage pre-trained SOLO, we observe a performance degradation across multiple benchmarks on the checkpoint without Stage-1 pre-training, further validating the importance of the first stage.
Discussion We hypothesize that discrepancies between a model's vision and language capabilities can lead the observed behaviors. Specifically, when there is a significant imbalance- as with the Mistral 7B model, which possesses advanced language abilities but lacks vision understanding the model may reduce loss by replicating caption patterns, inchuding redundant text tokens irrelevant to the visual content. For instance, in caption like "This is dog", the essential element is "dog". Focusing solely on minimizing language modeling loss without a robust initialized vision representation may lead the model to favor generic phrases like "This is a' over the more discriminative "dog". This is because the former includes more tokens, disproportionately influencing the overall language modeling loss. Pre-training on ImageNet at Stage 1, which emphasizes predicting only the "dog" token, helps the model develop a solid visual representation, effectively narrowing the gap between vision and language capabilities and mitigating this issue. In addition, the results indicate that pre-training loss vision-language data does not reliably indicate the performance of
8

---

Published in Transactions on Machine Learning Research (11/2024)
Overall LM Loss
Vision LM Loss
Text LM Loss
4
3x 100
100
x 10
x 10"
M
50
100
150
200
50
100
1.50
200
50
100
150
200
Step
Step
Caption Only Pro tra-training
TWo-Stage Pro re-training UmageNet + Caption)
Figure 3: Image captioning loss using two differently initialized checkpoints: (1) caption-only pre-training (green) initialized from the LLM; (2) two-stage pre- trainiiinng (blue) initialized from the Stage-1 ImageNet pre-trained IVLMM
Input
Output
The 1920x 1200 1920x1200
Task1:
Image Captioning
A puppy in the grass.
the one who is currently holding the office of the
Task2:
Who the president the United States? He is
Language Modeling
the head of state and head of government of the
Caption-Only Pre-Training
S: Two-Stage Pre-Training
Figure 4: Qualitative analysis of caption-only pro-training and SOLO's two-stage pro-training. Comparisons
are made on two checkpoints with comparable visiou-language modeling loss (i.e., 2.1). Specifically, we select the caption-only checkpoint at pre-training step 150, and SOLO at step 100.
IVLMs. Detailed analysis are provided in $7.1, which also demonstrates that training loss the instruction fine-tuning data mixture is similarly unreliable indicator.
5.2 Stage-2 Pre- Training on Web-Scale Data
Stage-2 Pre-Training Improves Performance on Top of Stage-1 We verify the effectiveness of Stage-2 pre-training on web-scale data by comparing the performance of two IVL.Ms. Fach model is fine-tuned for 800 steps using the same instruction mue-tuning data mixture but initialized differently one from the end of Stage 1 and the other from Stage In Fig. 5b, we observe significant improvement on all evaluation datasets after pre-training on web-scale data, showing the substantial advantages of Stage 2 pre-training compared to solely using ImageNet data (Stage only).
Combining ImageNet and Captioning Stage-2 Hurts Performance addition, is pertinent to ask whether ImageNet21K data can be combined with web-scale data for Stage 2. We include an ablation with SOLO trained on ImageNet21K and the web-scale data included in the second stage. Fig. 6 illustrates the training curves for comparison. The results suggest that while ImageNet pre-training effectively establishes initial visual representation, it may not be optimal for subsequent Stage-2 pre-training on web-scale data, it potentially impedes the optimization of vision-language modeling on image captions (i.e., vision language modeling loss stop improving). This discrepancy may arise from the divergence in image classification and captioning capabilities; the former is emphasized the first stage. This two-stage approach aligns with the principles of continual curriculum learning, where the model must maintain proficiency in familiar tasks while integrating new ones. This conclusion is also supported by our evaluation of downstream tasks (see

---

Published in Transactions on Machine Learning Research (11/2024)
Effectiveness Stage-1 Pre-Training on ImageNet Data
Effectiveness of Stage 2 Pre Training on Web Scale Data
Stage3
Sage1 Sage Stgge
Stages
60
50
Performnce
50
Perfrmmncc
40
30
30
20
MMsttr
MME
SEED
ScienccQA MathVista
A12D
MMSttr
MME
SEED
ScienceQA Mathiiista
A1DD
(a) The effectiveness of Stage-1 pre-training on ImageNet data for initialization.
(b) The effectiveness of Stage-2 pre-training on web-scale
data for knowledge breadth and data volume.
Ineffectiveness of Stage-2 Pre-Training on ImageNet Data
70
Stage Imageiae
Performance across Different Training Steps
Stago-3 Web Inageliet
80
70
Performance
Perferance
60
30
50
1000
Misstar
MME
SEED
ScienceQA MathVista
A220
Training Steps
(c) The ineffectiveness inchuding ImageNet data in Stage-2 pre-training.
(d) The performance across training steps on the fine- tuning dataset.
Figure 5: The evaluation performance of various ablations to validate key ingredients of our recipe. The MME scores are normalized for better illustration.
Fig. 5c). We train two different checkpoints with and without ImageNet21K data in the second stage on the instruction fine-tuning data mixture ($3.2.2) for 800 steps. Note that we select two checkpoints with comparable vision-language modeling losses for analysis. The results indicate that incorporating ImageNet21K data in the second stage may detrimentally impact overall performance by inhibiting adaptation to and learning from web-scale data.
5.3 Performance Boost via Instruction Fine- Munnnn
We evaluate the performance of SOLO on different training steps throughout the instruction fine-tuning stage (see Fig. 5d). The results indicate consistent improvement in SOLD's performance with prolonged training on the fine-tuning dataset, although the MME scores exhibit some fluctuations. This outcome contrasts with the findings of Liu et al. (2024a), where the performance quickly plateaus upon training with a limited subset of the fine-tuning dataset. This illustrates the increased scalability of SOLO during the instruction fine-tuning stage, suggesting that acquiring additional high-quality supervised datasets for fin-tuning could consistently enhance performance.
5.4 Additional Validation Experiments
We present the results demonstrate the effectiveness of Stage-3 anncaling $B and to validate the curated data mixture for instruction fine-tuning in &C.
10

---

Published in Transactions on Machine Learning Research (11/2024)
Overall LM Loss
Vision LM Loss
Text LM Loss
3x000
x 10
3
2 xx000
10
10
100
100
200
300
400
500
600
100
200
300
400
5.00
600
100
200
300
400
600
Step
Step
Step
Stage Image Captioning
State Image Captioning ImageNet Classification
Figure 6: Stage-2 language modeling loss when trained on (1) Image captioning objective only (blue); (2) Image captioning objective and image classification objective used Stage-1 (orange).
6 Further Analysis
6.1 Controlled Analysis of Fuyu and LLaVA
We conduct a controlled analysis compare SOLO with LLaVA and Fuyu (see Fig. 7). SOLO with training recipe consistently outperforms Fuyu- 8B, which adopts the same unified modeling strat- egy, across all evaluation benchmarks. To facilitate a controlled comparison with LLaVA, we develop LLAVA-7B', which integrates CLIP-ViT-336 and Baaurall-B--ase---i utilizing our specific training procedure and data. The results reveal that LLaVA- 7B" achieves performance similar with L.-v1.11.5 7B (Liu et al., 2024a), indicating that our training recipe, which utilizes large-scale datasets and exten- sive training, may not significantly impact LLaVA style LVLMs. Notably, while LLaVA-7B* excels in general visual-language lasks, SOLO demonstrates su- perior capabilities in visual mathematical reasoning, with overall performance being similar.
Controlled Analysis Fuyu
70-
Peformance
40
30
10
MMSLar
MMC
SEED
ScienceQA MathViste
A12D
The controlled analysis of Fuyu-8B and LL-7B-
Figure
6.2 Scalability Analysis
Superior Scaling Properties of SOLO We demonstrate that existing LVLMs with heterogeneous ar- chitectures exhibit diminishing returns despite increases in high-quality instruction fine-tuning data while SOLO shows better scaling behaviors. We perform instruction fine-tuning on pro-trained LLaVA obtained in $6.1 and compare its scaling behaviors with SOLO by measuring the performance improvement per training token. For evaluation, we fine-tune both models for steps as their initial checkpoints, as their pre-trained versions are not effective following instructions. Given that both models are fine-tuned on the same dataset, can directly compare their average improvement in benchmark performance per training token during training. In the implementation, we measure performance improvement every 500 steps, normalized by the number of training tokens encountered during those steps, and average this to calculate the final metric. Fig. 8 shows that SOLO outperforms mLLaVA on the "performance improvement per token" metric across all evaluation benchmarks. This suggests SOLO benefits more from high-quality instruction fine- muninuning data and demonstrates better scalability, indicating that its performance could further be improved more compared mLLa VA with more data.
11

---

Published in Transactions on Machine Learning Research (11/2024)
Scalability Comparison
Ablation of the Resolution
70
hg226
Rg672
2.5
Rg000
Rur1024
Perfrmance Gains Per Token
SCLO
2.0
50
Performance
40
1.5
30-
1.0
20
0.5
10
0.0
MMStar
MME
SEED
ScienceOA Mathiiista
A20
MMStar
MME
SEED
ScienceOA MathiVista
A12D
Figure 8: We compare the scaling behaviors of SOLO and LLaVA by measuring the improvement on bench- mark performance per token.
Figure 9: The performance of SOLO when trained and tested on different resolutions of images. R@X denotes the resolution of X.
SOLO Exhibits Training and Inference Speed Advantage We measure the training speed of SOLO compared to LVLMs with heterogeneous archilectures by measuring throughput. Us- ing the same 8xA100 server, we compare the number of tokens processed per second during SOLO training and the official LLaVA implementation (Liu et al., 2023a). Our SOLO imple- mentation achieves a throughput of 20K tokens per second, while LLaVA reaches 10.5K tokens per second, highlighting SOLO's significant advantage in training speed. We also mcasure the inference speed of SDLO compared several LVLMs with heterogeneous archilectures, including VLaVA-Next LLa VA- and Qwen-VI. VL. The evaluation is conducted on single A100 GPU using 10,000 COCO images and consistent prompt: "Generate the Caption for the <image>", ", All inference code is implemented using HuggingFace, and the inference latency is measured from the input being provided to the model and the output of the first token for fair comparison among all models since they may generate free-form responses in different lengths. The results, shown in Tab. 3, indicate that SOLO consistently outperforms other LVLMs with heterogeneous architectures by a significant margin, demonstrating its clear advantage in inference speed and large-scale development.
Table 3: Inference latency comparison.
Model
Inference Latency (seconds)
V.aVA-Next
0.0641
LLaVA- -aaeeleweeee
0.0091
Qwen2-VL.
0.0588
SOLO Facilitates Easier Scaling Laws Anal- ysis We conduct a simplified scaling laws ex- periment to show that the performance gains from scaling up data for SOLD is more pre- dictable compared to LLaVA-Style LVLMs. Specifically, we follow Kaplan et al. (2020) to fit the analytical function that IVLMs trained with a limited dataset (instruction fine-tuning in our case):
Table 4: Predictability the performance gains comparison between SOLO and mLLAVA on various benchmarks.
Rº
MMStar
MME
SEEDBENCH
SeiencrQA
MachiVista
A12D
BLLAVA
40.32
92.89
80.25
62.73
83.29
SOLO
88.75
92.36
98.59
97.75
81.70
96.55
D.
op
L(D) =
D
where Dr and On are constants to be estimated, D is the number of training tokens, L(D) is the benchmark performance in our case. We use data points from the instruction fine-tuning of SOLO and mLLaVA to fit this function and report the coefficient of determination R?, which reflects the quality of the fit and is oquivalent to the predictability of performance. The results below demonstrate that SOLO 's performance is more predictable compared to mL,aVA, indicating that SOLO facilitates easier scaling laws analysis. We further provide more discussion about the better scalability of SOLO in $7.3.
12

---

Published in Transactions on Machine Learning Research (11/2024)
loss
Performance Comparison
4
3
63
reformance
50
2
40
30
20
Step
10
0
500
1k
1.5k
2k
2.5k
3k
MMStar
MME
SEED
Science(A MathVista
AI20
SOLO (I=11e:4
SOLO
(b) The downstream performance of variants.
(a) The training curves of two variants.
Figure 10: The training curves and downstream performance evaluation of two variants of SOLO. We find that they show significant performance differences although achicving a similar loss on the instruction finc-tuning data mixture.
SOLO Demonstrates Improved Performance when Scaling up Image Resolution We train SOLO different image resolutions during the instruction fine-tuning stage for 1,000 steps due to the compute limits (see Fig. 9). The image resolution during inference matches that used in the instruction fine-tuning stage. We find that SOLO continues to improve the performance with increasing image resolution, especially for the visual mathematical reasoning task. In addition, there is no significant difference in performance between the 1024-square resolution and the adapted resolution with 1024 as the maximum used in SOLO. This demonstrates the efficiency and scalability of our flexible image pre- pre-processing pipeline.
SOLO Benefits from its Dynamic High-Rosolution Capability We eval- uate LVLMs on high-resolution images and images with extreme aspect ratios that diverge from those found in natu- ral images. For a controlled analysis, we select two benchmark datasets (Sci- enceQA, MathVista) where SOLO and LLaVN-t achieve comparable performance (within 1 absolute point). In addition, we implement mLLaVA, which follows LLaVA's 's modeling framework but is trained using SOLO 's recipe for further comparison. We select subsets from the original benchmarks that include images with either width height exceeding 800 pixels. Also, we select those subsets with ratio wwidth/height) greater than 3 or less than 1/3 for extreme aspect ratio analysis. We evaluate LVLMs' performance on these subsets. Our results, as shown in Tab. 5, verify SOLO 's advantages regarding the performance on high-resolution images and images with extreme aspect ratios. The poorer performance mLLaVA and LLa LLaVA-Next is likely due to their heuristic resizing rules, which constrain images predefined resolution settings. In contrast, SOLO retains the original aspect ratios, which appears to be a more optimal approach.
Table 5: Comparison of performance on ScienceQA and MathVista under conditions of extreme image resolutions.
Resolution > 800 ScieocrQA
MathVista
Aspect Ratio
ScienceQA
MathVVsta
LLaVA-Nost
60.53
30.18
LLaVA-Next
43.28
25.00
67.09
28.97
mLLAVA
34.45
SOLD
71.19
32.02
3000
50.42
28.13
7 Discussion
7.1
(Pre-)training Loss on Vision-Language Data is Not Reliable Indicator of the Actual Performance
We find that both the pre-training loss and the instruction fine-tuning loss on the vision-language data are not reliable for the estimation of LVLMs' actual performance. References to support this claim regarding the pre-training loss include observations detailed in Fig. 3, Fig. 4, and Fig. 5a. Despite achieving similar
13

---

Published in Transactions on Machine Learning Research (11/2024)
Overall LM Loss
Vision LM Loss
Text LM Loss
1.7x100
x 10
1.6x 10"
4 10'
1.4x 10
3 10
1.3 10
10"
1.2 x 10
10
10p
2 10"
10a
9 10
100
200
300
400
500
600
100
200
300
400
500
600
1.00
200
300
400
500
600
Step
Step
Step
1x Text Data
Text Data
3x Text Data
Figure 11: Stage language modeling loss when trained on a mixture with different quantities of text data. reflects the data mixture in Tab. 1, 2x and 3x represent mixtures with 2 times more text data compared to Ix while keeping the amount of vision data unchanged.
language modcling losses when conditioned on visual inputs, LVLMs exhibit markedly different behaviors and performance across various downstream lasks. This contrasts with findings from pure language modeling, where pre-training loss strongly correlates with various downstream task performance (Du al., 2024).
We also demonstrate that the loss associated with the instruction fine-tuning data mixture does not reliably indicate task performance. We train variant of SOLD with a learning rate of 1e-4, deviating from the prescribed rate of le-5 in our recipe. We show the training curves (Fig. 10a) and the downstream performance evaluation (Fig. 10b) these two variants. The two variants exhibit similar training behaviors and losses on the instruction fine-tuning data mixture, yet they display significant performance disparities in downstream evaluation benchmarks.
Overall, our analysis highlights the need to identify dependable metric for evaluating LVLMs with the unified architecture in pre-training, particularly for establishing scaling laws in future research.
7.2 Balancing Vision and Language Capabilities during Pre- Traininin is Challenging
We find that 7B scale, balancing vision and text capabilities can be challenging. Specifically, observe that during Stage-2 pre-training, despite the inclusion of text-only pre-training data ($3.2.1) to maintain the language capability the original LLM, the language modeling loss on the language-only pre- training subset still steadily increases as training continues. In Fig. 11, we introduce setting where gradually increase the proportion of text-only data per batch (Tab. 1) and monitor the language modeling loss for text. The results suggest that aug- menting text data proportions does not alleviate the rise in language modeling loss, indicating challenges in achicving balanced vision and text capabilitics in 7B-scale model.
Evaluation of the Language Capability
SOLO
Netin (hwo))
Performance
40
30
MMLU
GSMBK
HellaSwag
RACE
Figure 12: The evaluation of language capability.
To further understand the degradation in language ability of SOLO, evaluate SOLD standard LLM evalu- ation benchmarks, including MMLU (Hendrycks et al., 2020), GSM8k (Cobbe et al., 2021), HellaSwag (Zellers et al., 2019), and RACE (Lai et al., 2017). Our analysis includes comparisons with the backbone LLM of SOLO, specifically Distral-7B-vO.--baar as well as Mistral- -B-.-177--v.....tttt. (see Fig. 12). We observe a decline in language capabilitics, particularly knowledge-intensive benchmarks such as MMLU. There are two potential reasons: (1) Integrating vision capabilities may compromise language performance. (2) The quality of Mistral's pre-training corpus is better than the open-source Slimpajama employ. Overall, the
14

---

Published in Transactions on Machine Learning Research (11/2024)
Table 6: Comparison between SOLO and LVLMs with heterogeneous architectures regarding the complexity of scaling laws analysis.
Comparison
BOLO
LVLMs Heterogencous Architecture
Architecture
Honnogeneous
Beterogrneons (vienal encocer, connector, LLM;
Scaling laws analysis
Standard approach (Hotfroonn et al., 2022)
Roçuires extra "parameter allocation law" (an addinionall fuctor in the scaling law formonlation)
Key chailenge
N/A (widely explored in Hoffmaam et al. (2022) Optinal patameter alocction and consiseency issues
Impact computation
Madumtee
High, due to mnllipple aggatiios and auklitional experiments parmuster alkoestion law
Risk of seror propagation
Low
Highor due to adkitiinn complexitios
current results indicate a limitation the current version of SOLO, as effective performance in real-world vision-language tasks often necessitates strong foundational language capabilities, including knowledge and reasoning. Thus, we plan to maintain the language capabilities of SOLO in the upcoming version by enriching the pre-training dataset with higher-quality text corpus and increasing the proportion of text data.
7.3 Scaling Laws Analysis with SOLO
We demonstrate how SOLO 's unified architecture facilitates easier analysis by comparing its experimental wawrrroennous-arrccicee LVLMs. To simplify the analysis process,
plan for deriving scaling laws to those we follow the standard experimental setup in Hoffmann et al. (2022) to proportionally scale up the model size and training tokens by a factor of 20.
For SOLO, we can follow the standard scaling laws approach by selecting sampling models of varying sizes and assigning the appropriate number of training tokens based on the predefined scaling factor (e.g., 20 times). After training each model on its designated tokens, the pre-training loss can thus be measured to obtain the (expended FLOPs, performance) data point. The obtained data points can be used to fit the power law curve that prediets the performance of the target model (e.g., a model with 70B parameters) given the FLOPs expected to expend.
In contrast, for LVLMs with hetcrogencous architcctures, such as LLaVA, scaling laws analysis introduces additional complexity. Two key issues arise: (1) Parameter allocation: How should the model's parameters be distributed among the three components (visual encoder, connector, LLM)? (2) Consistency during scaling: Will this parameter allocation (e.g., a 0.05:0.01:0.94 ratio) remain fixed model size, data size, and compute scale up? To address these, an additional "parameter allocation law" must be derived to predict the optimal distribution of parameters based on model size and training tokens. In implementation, for each size of the sampling model, multiple configurations regarding paramcter allocation must be tested to determine the optimal pre-training loss and the corresponding best allocation, siguificantly increasing computational demands. Moreover, it's challenging to figure out the best analytical form to map the model size and training tokens to the parameter allocation ratios. Consequently, the introduction of the parameter allocation law complicates analysis and increases the risk of error propagation.
8 Related Work
Model Architecture Existing research advances the development of LVLMs capable of addressing di- verse tasks via a unified interface that can directly generate natural language, thus avoiding task-specific modifications (Wang et al., 2021; 2022a; Li et al., 2023c). Utilizing pre-trained LLMs (Brown et al., 2020b; Bubeck et al., 2023) as the language component paired with pre-trained visual encoders (Radford et al., 2021; Dosovitskiy et al., 2021a), recent approaches further enhance the instruction-foolloinng user-friendly responses generation, and complex reasoning ability of LVLMs (Liu et al., 2023c; Zhu et al., 2023; Dai et al., 2023; Alayrac et al., 2022; Li et al., 2023a; Ye et al., 2024). Coneurrently, Wang et al. (2022b); Peng et al. (2022); Anil et al. (2023); Team (2024); Ge et al. (2023) propose to further learn a codebook in the initial stage to discretize the inntus embeddings extracted by visual encoders into a sequence of image tokens. These approaches enable a uniform rision-languse modeling strategy for image and language tokens. However, the dependence on pre-trained visual encoders restricts the scalability of LVLMs. In this study, we address this challenge by readopting the conventional vasion-language modeling approach that utilizes single Transformer for both image and text processing (Li et al., 2019). Furthermore, while Bavishi et al. (2023)
15

---

Published in Transactions on Machine Learning Research (11/2024)
extend this approach to billion-scale models, they do not disclose the specifics of their training processes. We address this gap by offering reproducible training recipes, complete with publicly released code, for scalable vision-language modeling on a 7-billion LVLM.
Training Data Typically, IVLMs leverage extensive image-caption pair datasets (Lin et al., 2014a; Schuhmann et al., 2021; 2022; Yu et al., 2024; Chen et al., 2023b) to train a projector or a codebook that map continuous image features into the embedding space of LLMs, thereby aligning the two modalitics (Li et al., 2023c; Gong et al., 2023; Zeng et al., 2023; Sun et al., 2023a). Furthermore, large-scale vision-language instruction tuning datasets (Su et al., 2023; Wei al., 2023; Liu et al., 2023b; Gong et al., 2023; Gao et al., 2023; Li et al., 2023a) and feedback datasets (Li et al., 2023d; Sun al., 2023b; Chen al., 2024c; Zhang et al., 2024b) are utilized to further boost the fundamental capabilities of LVLMs and align LVLMs with human preferences, ensuring their ability to comprehend instructions and generate responses that are user-friendly. In this work, we propose recipe that encompasses the selection of pro-training and instruction fine-tuning datasets, along with corresponding multi-stage paradigms, to facilitate the training of billion-scale LVLMs of a single Transformer architecture.
Evaluation Benchmarks The progress of LVLMs is guided and measured by the continuous development of evaluation benchmarks (Ferraro et al., 2015; Kafle et al., 2019; Gan et al., 2022; Chen et al., 2024d). Initially, evaluation primarily concentrates on fundamental visual-language skills, such as image captioning (Lin et al., 2014b; Plummer et al., 2015), basic visual information recognition (Antol al., 2015; Goyal et al., 2017), compositional visual understanding (Hudson & Manning, 2019), and knowledge reasoning based on visual information (Marino et al., 2019; Schwenk et al., 2022). Current benchmarks advancing encompass more intricate capabilities, requiring LVLMs to perform detailed visual analysis and complex reasoning ( (Uppal et al., 2022; Zhang et al., 2024a). These benchmarks range from general assessments across various domains and skills (Li et al., 2024a; Fu et al., 2024; Chen et al., 2024b; Yu et al., 2023) to specific tests targeting particular abilities, such as scientific document understanding (Kembhavi et al., 2016; Lu et al., 2022a), mathematical reasoning (Lu et al., 2023; Wang et al., 2024a), musti-discipline understanding and reasoning (Yue al., 2023; Wu et al., 2024), hallucination (Li et al., 2023e), and OCR ability (Liu et al., 2023d). In this work, we select the advanced general and skill-spocific benchmarks for evaluation.
9 Conclusion
This work revisits the simple vision-language modeling framework with a single Transformer. We argue that this approach effectively mitigates the scalability limitations inherent in prevailing models. With academic resourcos, we build SOLO, 7B LVLM initialized from the Mistral LLM. We detail the training recipe and conduct extensive analysis and evaluation to validate the ingredients in our recipe. Experimental results show that SOLO demonstrates performance comparable to LLaVA-v1.5, supporting the contimued investigation into this unified vision-language modeling approach for improved scalability.
Limitations and Broader Impact Statement
The investigation into large-scale vision-language modeling using unified transformer architecture remains nascent, with our model not yet reaching optimal performance across diverse benchmarks. Continued advancements in the direction of unified LVLMs for scalable vision-language modcling are anticipated. However, although developing LVLMs with strong capabilities brings siguificant advancements in AT, it also poses potential negative impacts. One concern is the risk of misuse, where the model could be employed for malicious purposes, such as generating mislcading content that could manipulate public opinion or deceive individuals. Additionally, the model may inadvertently exacerbate biases present in the training data, leading to unfair discriminatory outcomes in decision-making processes.
16

---

Published in Transactions on Machine Learning Research (11/2024)
Acknowledgement
We thank the reviewers and the action editor for their valuable suggestions and comments. This research is based upon work supported by U.S. DARPA ECOLE Program No. HR00112390060, The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, cither expressed or implied, of DARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribule reprints for goverumental purposes notwithstanding any copyright annotation therein.
References
Manoj Acharya, Kushal Kafle, and Christopher Kanan. Tallyqa: Answering complex counting questions. In Proceedings of the AAAT conference on artificial intelligene, volume 33, pp.
2019.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lene, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. arviv preprint arXiv:2204. 14198, 2022.
Rohan Anil, Sebastian Borgeaud, Yonghui Wu,
Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, loamnis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael Isard, Paul Ronald Barham, Tom Hennigan, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, Ryan Doherty, Eli Collins, Clemens Meyer, Eliza Rutherford, Erica Morcira, Karcem Ayoub, Megha Gocl, George Tucker, Enrique Piqueras, Maxim Krikun, lain Barr, Nikolay Savinov, Ivo Danihelka, Becca Roelofs, Anaïs White, Anders Andreassen, Tamara von Glehn, Lakshman Yagati, Mehran Kazemi, Lucas Gonzalez, Misha Khalman, Jakub Sygnowski, and et al. Gemini: A family of highly capable multimodal models. CoRR, abs/2311 2312. 11805, 2023. doi: 10.48550/ 10.48550/ARXIV.10 AAI.//XXIIV..2 2882.11800. URL https: http://dooorgggeoo o0g/10. 48550/arXiv. 201.. 11805.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference computer vision, pp. 2425 2433, 2015.
Anas Awadalla, Trena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, et al. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint wxXiv:2308.01300, 2023.
Yasaman Bahri, Ethan Dyer, Jared Kaplan, Jachoon Lee, and Utkarsh Sharma. Explaining neural scaling laws. arXiv preprint wxXiv:2102.06011 2021.
Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: versatile vision-language model for understanding, localization, text reading, and beyond. 2023.
Rohan Bavishi, Erich Elsen, Curtis Hawthorne, Maxwell Nye, Augustus Odena, Arushi Somani, and Sagnak Tagılaar Introducing our multimodal models, 2023. URL https: : //www adept. ai ai/blog/fuyul
Ali Furkan Biten, Ruben Tito, Andres Mafla, Lhuis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas. Scene text visual question answering. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 4291-4301, 2019.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901, 2020a.
Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020b.
17

---

Published in Transactions on Machine Learning Research (11/2024)
Sébastien Bubeck, Varun Chandrasckaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat. Lee, Yuanzhi Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Túlio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with GPT-4. CoRR, 2023.
Alejandro Hernández Cano, Matteo Pagliardini, Andreas Köpf, Kyle Matoba, Amirkeivan Mohtashami, Olivia Simin Fan, Axel Marmet, Deniz Bayazit, Igor Krawezuk, Zeming Chen, Francesco Salvi, Antoine Bosselut, and Martin Jaggi. epflim megatron-lm, 2023. URL https: /ttp://///hhtbt// htwpchhuu.oommeaaaeoeemmeaeeeeee hetpLLL//MMMaaaooo-LAl
Guiming Hardy Chen, Shunian Chen, Ruifei Zhang, Junying Chen, Xiangbo Wu, Zhiyi Zhang, Zhihong Chen, Jianquan Li, Xiang Wan, and Benyou Wang. Allava: Harnessing gptav-synthesized data for lite vision-language model. arXiv preprint arXiv:2402.1 11664, 2024a.
Jun Chen, Deyao Zhu, Xiaoqian Shen, Xiang Li, Zechun Liu, Pengchuan Zhang, Raghuraman Krishnamoorthi, Vikas Chandra, Yunyang Xiong, and Mohamed Elhosciny. Minigpt-v2: large language model as unified interface for vision-language multi-task learning. arXiv preprint ar wrXiv22310009478, 2023a.
Lin Chen, Jisong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:231..12793, 2023b.
Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Jiaqi Wang, Yu Qiao, Dahua Lin, et al. Are we on the right way for evaluating large vision- lailonla models? arXiv preprint 2XXi:2443.20333 2024b.
Yangyi Chen, Xingyao Wang, Manling Li, Derck Hoiem, and Heng Ji. Vistruct: Visual structural knowlodge extraction via curriculun guided code-vision representation. In Proc. The 2023 Conference Empirical Methods in Natural Language Processing (EMNIP20023, 2023c.
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Dress: Instructing large vision-language models to align and interact with humans via natural language feedback. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 11239-14250, 2024c.
Yangyi Chen, Karan Sikka, Michael Cogswell, Heng Ji, and Ajay Divakaran. Measuring and improving chain-of-thought reasoning in vision-language models. In Proc. 2024 Annual Conference of the North American Chapter the Association for Computational Linguistics (NAACL2024), 2024d.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint 1rX.v:2110.14168,, 2021.
OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models. https: //github.com////////// cowpass/
2023.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven C. H. Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. CoRR, 2023.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805, 2018.
Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, and Jiawei Wang. Write and paint: Generative vision- language models are unified modal learners. In The Eleventh International Conference on Learning Representations, 2023.
Ning Ding, Yulin Chen, Bokai Xu, Yujia Qin, Zhi Zheng, Shengding Hu, Zhiyuan Liu, Maosong Sun, and Bowen Zhou Enhancing chat language models by scaling high-quality instructional conversations. arXiv preprint 1432333 2023.
18

---

Published in Transactions on Machine Learning Research (11/2024)
Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Songyang Zhang, Haodong Duan, Wenwei Zhang, Yining Li, et al. nitenlnnxcomposer--hiie A pioneering large vision-language model handling resolutions from 336 pixels to 4k hd. arXiv preprint arXiv:2404.0551, 2024.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Golly, Jakob Uszkorcit, and Neil Houlsby. Words: Transformers for Image Recognition al Scale. arXiv:2010. 11929 [es].
An Image is Worth 16x16 2021a.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In Iclr, 2021b.
Zhengxiao Du, Yujic Qian, Xiao Liu, Ming Ding, Jiczhong Qiu, Zhilin Yang, and Jie Tang. Glm: General language model pretraining with autoregressive blank infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers), pp. 320 335, 2022.
Zhengxiao Du, Aohan Zeng, Yuxiao Dong, and Jie Tang. Understanding emergent abilities of language models from the loss perspective. arXiv preprint arXiv:2403.15700 2024.
Haodong Duan, Junming Yang, Yuxuan Qiao, Xinyu Fang, Lin Chen, Yuan Liu, Xiaoyi Dong, Yuhang Zang, Pan Zhang, Jiaqi Wang, et al. Vlmevalkit: An open-source toolkit for evaluating large multi-modality models. Xiviv preprint arXiv:2407.11691 2024.
Patrick Esser, Robin Rombach, and Björn Ommer. Taming transformers for high-resolution im- age synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19. 2021, pp. 12873-12888 Computer Vision Foundation / IEEE, 2021. doi: 10. 1109/CV 0099/CVPR4443377 200.P.4.437.221.000 202.202221..22 URL https: httpt::ttpppsaeaeee /ttp:///cppp/.cs.//// thecvf wwwecvc.ccooo//ooooooooooo/ooooo wwtputeent//pn22////tttt
Taming_ Transformers_ for Hiw.HHiggghhhhiee Carrrsootttoooonnnnoeeeee Sesttsststttttseeeeeeeee
html.
Francis Ferraro, Nasrin Mostafazadeh, Lucy Vanderwende, Jacob Devlin, Michel Galley, Margaret Mitchell, et al. A survey of current datasets for vision and language research. arXiv preprint arXiv:1506 100.:135...0833,, 2015.
Chaoyou Fu, Peixian Chen, Yunhang Shen, Yulei Qin, Mengdan Zhang, Xu Lin, Jinrui Yang, Xiawu Zheng, Ke Li, Xing Sun, Yunsheng Wu, and Rongrong Ji. Mme: A comprehensive evaluation benchmark for multimodal large language models, 2024.
Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao, et al. Vision-language pre-training: Basics, recent advances, and future trends. Foundations and Trendsö in Computer Graphics and Vision, 14(3 4):163 352, 2022.
Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model. ar preprint. arXiv:2304110110 2023.
Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.0804: 2023.
Tao Gong, Chengqi Lyu, Shilong Zhang, Yudong Wang, Miao Zheng, Qian Zhao, Kuikun Liu, Wenwei Zhang, Ping Luo, and Kai Chen. Multinodal-gpt: A vision and language model for dialogue with humans. arXiv preprint arXiv:2305.0477
2023.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6904 6913, 2017.
Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://gitthni com//hugginfaccc/cccceeese 2022.
19

---

Published in Transactions on Machine Learning Research (11/2024)
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.0333 2020.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint 2ttii:2223.115550 2022.
Jinyi Hu, Yuan Yao, Chongyi Wang, Shan Wang, Yinxu Pan, Qianyu Chen, Tianyu Yu, Hanghao Wu, Yue Zhao, Haoye Zhang, et al. Large multilingual models pivot zero-shot multimodal learning across languages. arXiv preprint ar arXiv:2308.1203 12038, 2023.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zhen Leng Thai, Kai Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Miniepm: Unveiling the potential of small language models with scalable training strategies. CoRR, abs/2404.00 arXiv 00//22444.03999 2024a. doi: 10.48550/ ARRII/ARRIIVI... 2404.06395. URL https: //doi. org/10 10. 48550/
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, et al. Miniepm: Unveiling the potential of small language models with scalable training strategies. arXiv preprint arXiv:2404.06390 2024b.
Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6700 6709, 2019.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825, 2023.
Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 0644-5656, 2018
Kushal Kafle, Robik Shrestha, and Christopher Kanan. Challenges and prospects in vision and language research. Frontiers in Artificial Intelligence, 2:28, 2019.
Samira Ebrahimi Kahou, Vincent Michalski, Adam Atkinson, Ákos Kádár, Adam Trischler, and Yoshua Bengio. Figureqa: An annotated figure dataset for visual reasoning. arXiv preprint arXiv:1710.07300, 2017.
Yasindu Kamizuru. Diagram image to text. URL http:://hgggggggaaa..c.aaaatt//s
sattrramimmaee_eee textext.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint 1oXiv:200110836: 2020.
Aniruddha Kembhavi, Mike Salvato, Eric Kolve, Minjoon Sco, Hannanch Hajishirzi, and Ali Farhadi. A diagram is worth dozen images. In Computer Vision ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14 2016, Proceedings, Part IV 14, pp. 235-202200 251. Springer, 2016.
Aniruddha Kembhavi, Minjoon Seo, Dustin Schwenk, Jonghyun Choi, Ali Farhadi, and Hannaneh Hajishirzi. Are you smarter than sixth grader? textbook question answering for multimodal machine comprehension. In Proceedings of the IEEE Conference Computer Vision and Pattern recognition, pp. 4999 5007, 2017.
Douwe Kiela, Hamed Firooz, Aravind Mohan, Vodanuj Goswami, Amanproct Singh, Pratik Ringshia, and Davide Testuggine. The hateful memes challenge: Detecting hate speech in multimodal memes. Advances in neural information processing systems, 33:2611 1--22,,,,
2624, 2020.
20

---

Published in Transactions on Machine Learning Research (11/2024)
Jeonghwan Kim and Heng Ji. Finer: Investigating and enhancing fine-grained visual concept recognition in large vision language models. In arxiv, 2024.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gon- zalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehen- sion dataset from examinations. arXiv preprint arXiv:1704.04688, 2017.
LAION. Laion-gpt-4v. URL httpg////bugggggaac...eaeaatttee http///ttatttttaaaao//////ttttt.
Hugo Laurengon, Léo Tronchon, Matthicu Cord, and Victor Sanh. What matters when building lasiun- models? arXiv preprint arXiv:2405.02246, 2024.
Hugo Laurençon, Léo Tronchon, and Victor Sanh. Unlocking the conversion of web screenshots into html code with the websight dataset, 2024.
Bo Li, Yuanhan Zhang, Liangyu Chen, Jinghao Wang, Jingkang Yang, and Ziwei Liu. Otter: A multi-modal model with in-context instruction tuning. arXiv preprint arXiv:2305.03726, 2023a.
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed-bench-2: Benchmarking multimodal large language models. CoRR, abs/2311. 235/2211,77092, 2023b. doi: 10.48550/ARXTY 2311.17092. URL https: //doi. org/10. 485111....888881 arXiv 1311.17092
Bohao Li, Yuying Ge, Yixiao Ge, Guangzhi Wang, Rui Wang, Ruimao Zhang, and Ying Shan. Seed- bench: Benchmarking multimodal large language models. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13299 -3308,, 2024a.
Ju Junnan Li, Dongxu Li, Silvio Savarese, and Steven C. H. Hoi. BLIP-2: bootstrapping language-inage pre pre-ting with frozen image encoders and large language models. CoRR, 2023c.
Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, and Lingpeng Kong. Silkie: Preference distillation for large visual language models. arXiv preprint arXiv:2312.1066) 2023d.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wel Chang. Visualbert: A simple and performant bascline for vision and language. arXiv preprint arXiv: 03557. 2019.
Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models. arXiv preprint arXiv:2305.10355 2023e.
Zhang Li, Biao Yang, Qiang Liu, Zhiyin Ma, Shuo Zhang, Jingxu Yang, Yabo Sun, Yuliang Liu, and Xiang Bai. Monkey: Image resolution and text label are important things for large modal models. In Proceedings of the TEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 26763-2677 2024b.
Ji Lin, Hongxu Yin, Wei Ping, Pavlo Molchanov, Mohammad Shocybi, and Song Han. Vila: On pre-training for visual language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26689-26699, 2024.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, Jamess Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, 2014a.
Tsung Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In Computer Vision ECCV 2014 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V. Springer, 2014b.
21

---

Published in Transactions on Machine Learning Research (11/2024)
Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, et al. Sphinx: The joint mixing of weights, tasks, and visual embeddings for multi-modal large language models. arXiv preprint arXiv:2311.07575, 2023.
Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 11:635 651, 2023a.
Fuxiao Liu, Kevin Lin, Linjie Lí, Jianfeng Wang, Yaser Yacoob, and Lijuan Wang. Aligning large multi-modal model with robust instruction tuning, arXiv preprint arXiv:2306. 1466,,,, 2023b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lec. Visual instruction tuning. CoRR, 2023c.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. In Proccodings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26296 26306, 2024a.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024b. URL https://llaaaaaa www.hb.ittbbbooooo llava-next/.
blog/2024-01-300 1
Yuliang Tuu, Zhang Li, Hongliang Li, Wenwen Yu, Mingxin Huang, Dezhi Peng, Mingyu Liu, Mingrui Chen, Chunyuan Li, Lianwen Jin, et al. On the hidden mystery of ocr in large multimodal models, arXiv preprint 20X.022305.07855, 2023d.
Haoyu Lu, Wen Liu, Bo Zhang, Bingxuan Wang, Kai Dong, Bo Liu, Jingxiang Sun, Tongzheng Ren, Zhuoshu Li, Yaofeng Sun, et al. Deepscek-vl: towards real-world viston-language understanding. arXiv preprint arXiv:2403.055222 2024.
Pan Lu, Liang Qiu, Jiaqi Chen, Tony Xia, Yizhou Zhao, Wei Zhang, Zhou Yu, Xiaodan Liang, and Song- Chun Zhu. Iconqa: A new benchmark for abstract diagram understanding and visual language reasoning. arXiv preprint arXiv:2110. 10.1100..33 132144 2021.
Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NeurIPS, 2022a.
Pan Lu, Liang Qiu, Kai- Wei Chang, Ying Nian Wu, Song-Chul Zhu, Tanmay Rajpurohit, Peter Clark, and Ashwin Kalyan. Dynamic prompt learning via policy gradient for semi-structured mathematical reasoning. arXiv preprint arXiv:2209.148 14610, 2022b.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannaneh Hajishirzi, Hao Cheng, Kai- Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.02255. 2023.
lz. Detailed caption. URL
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In Proccedings of the IEEE/cvt conference on computer vision and pattern recognition, pp.
2019.
Ahmed Masry, Do Xuan Long, Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 2.Xi:22033.0044,, 2022.
Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. In- fographicvqa. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 1697-1706, 2022.
Anand Mishra, Shashank Shekhar, Ajoct Kumar Singh, and Anirban Chakraborty. Ocr-vqa: Visual question answering by reading text in images. In 2019 international conference on document analysis and recognition (ICDAR), pp. 947-952. IEFE, 2019.
22

---

Published in Transactions on Machine Learning Research (11/2024)
Jason Obeid and Enamul Hoque Chart-to-text: Generating natural language descriptions for charts by adapting the transformer model. arXiv preprint arXiv:2010.09142, 2020.
OpenCompass. leaderboard.
Opencompass.
URL. https://hugghhfface.co http:///ww....ppepccppp/cccc///a vlm
Zhiliang Peng, Li Dong, Hangbo Bao, Qixiang Ye, and Furu Wei. Beit v2: Masked image modeling with veclor-quantized visual tokenizers. CoRR, (ae//2208063360 2022. doi: 10.48550/ ARXTV 2208.06366. URL
https://doi.. org/ 10. 48550/ 2208. 220..06366.. 06366.
Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image- -eeatenc models. In Proceedings of the IEEE international conference on computer vision, pp. 2641-2649, 2015.
Jordi Pont- Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari. Connecting vision and language with localized narratives. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceodings, Part V 16, pp. 647-664. Springer, 2020.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodci, Ilya Sutskever, et al. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, 2021.
Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong Hc. Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 3505 1506-,55,,-0 2020.
Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. Advances in neural information processing systems, 28, 2015.
Tal Ridnik, Emanuel Ben- Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 2021a.
Tal Ridnik, Emanuel Ben- Barrca, Asaf Noy, and Lihi Zelnik-Manor Imagenet-21k pretraining for the masses, 2021b.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open datasot of clip-filterdd 400 million image-text pairs. arXiv preprint arXiv:2111.02114 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large- scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278- 0175227722294,, 2022.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1707.06347, 2017. URL http:// www.wiiwwwwww....../..raaaa org/abs//////ooooooo/..o 0770.06347.
Dustin Schwenk, Apoorv Khandelwal, Christopher Clark, Kenneth Marino, and Roozbeh Mottaghi. A-okvqa: A benchmark for visual question answering using world knowledge. In European Conference on Computer Vision, pp. 146-162. Springer, 2022.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Procoodings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Association for Computational Linguistics, 2018a.
23

---

Published in Transactions on Machine Learning Research (11/2024)
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2556 2565, 2018b.
Mohammad Shocybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. Megatron-Im: Training multi-billion parameter language models using model parallelism. arXiv preprint. arXiv:190.006053, 2019.
Oleksüi Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanprect Singh. Textcaps: a dataset for image captioning with reading comprehension. In Computer Vision ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28,8,,22 2020, Proceedings, Part II 16, pp. 742-758. Springer, 2020.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8317-8326, 2019.
Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. Slim SlimPajama: A 627B token cleaned and deduplicated version of RedPajama. https://www. cerebras net/blog/ sl slimpajama ama- a- -lepbtoooe--ceeeeeee--eee -nd-aaddddddddddddddddddd ot - / //huggingface ace. http//aaaeettseeeebrr/////////// -m2-66
2023. URL https:
Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. Pandagpt: One model to instruction- follow them all. arXiv preprint arXiv:2305. 16355, 2023.
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative pretraining in multimodality. arXiv preprint arXiv:2307.05222. 2023a.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu- Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually augmented rlhf. arXiv preprint 45Xiv.2300.14525, 2023b.
Benny Tang, Angie Boggust, and Arvind Satyanarayan. Vistext: A benchmark for semantically rich chart captioning. arXiv preprint arXiv:2307. 053356 2023.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. CoRR, bbs/2405.09818, 2024. doi: 10.4855/AARIIV.24055
URL
MLC team. MLC-LLM, 2023. URL htttp:///gthabb..oommm///aaaaaaa
Techerunch. Etched is building AI chip that only runs type of model. httpss//techhrucch. com/2024/ 1e/25/etchhe-is-bblle wuidiiissbuuiiinn-n-e an---an-as 06.
aawldiinnn-----iiiipaaae ai-ccii--eeee tet-tthtttt----tteee only-runs- toaa--trraaoomme--ddell..... Accessed: 2024-07-
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaci, Nikolay Bashlykov, Sounya Batra, Prajjwal Bhargava, Shruti Bhosale, el al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.0925: 092888, 2023.
Shagun Uppal, Sarthak Bhagat, Devamanyu Hazarika, Navonil Majumder, Soujanya Poria, Roger Zimmer- mann, and Amir Zadeh. Multimodal research in vision and language: review of current and emerging trends. Information Fusion, 77:149 17:149-171, 2022.
Junke Wang, Lingchen Meng, Zejia Weng, Bo He, Zuxuan Wu, and Yu-Gang Jiang. To see to believe: Prompting gpt-4v for better visual instruction tuning. arXiv preprint arX1v:2311.0777,, 2023a.
Ke Wang, Junting Pan, Weikang Shi, Zimu Lu, Mingjie Zhan, and Hongsheng Li. Measuring multimodal mathematical reasoning with math-vision dataset. arXiv preprint arXiv:2402.1488 14804, 2024a.
24

---

Published in Transactions on Machine Learning Research (11/2024)
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. OFA: unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework. In International Conference on Machine Learning, 1CML 2022, 17-23 July 2022, Baltimore, Maryland, USA. Pmlr, 2022a.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as foreign language: Beit pretraining for all vision and vision-language tasks. CoRR, abs/ 2////220.14444 2022b. doi: 10.48550/ ARXIV.2208.J.. URL httpp://doi. org/10 10. 48550/ 2208. 10442.
Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better Ilm agents. arXiv preprint arXiv:2402.01030, 2024b.
Zhenhailong Wang, Ansel Blume, Sha Li, Genglin Liu, Jaemin Cho, Zineng Tang, Mohit Bansal, and Heng Ji. Paxion: Patching video-language foundation models with action knowledge. In Proc. 2023 Conference on Neural Information Processing Systems (NeurIPS2023) [Spotlight Paper], 2023b.
Zhenhailong Wang, Joy Hsu, Xingyao Wang, Kuan-Hao Huang, Manling Li, Jiajun Wu, and Heng Ji. Text-based reasoning about vector graphics. In arxiv, 2024c.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108. 2021.
Lai Wei, Zihao Jiang, Weiran Huang, and Lichao Sun. Instructiongpt-4: A 200-instruction paradigm for fine-tuning minigpt-4. arXiv preprint arXiv:2308.2067, 2023.
Chris Wendler. Renderedtext. URL https::/
Shujin Wu, Yi R Fung, Sha Li, Yixin Wan, Kai-Wei Chang, and Heng Ji. Macaroon: Training vision-language models to be your engaged partners. arXiv preprint MXii:22406.1133) 2024.
Ruyi Xu, Yuan Yao, Zonghao Guo, Junbo Cui, Zanlin Ni, Chunjiang Ge, Tat-Seng Chua, Zhiyuan Liu, Maosong Sun, and Gao Huang. Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images. arXiv preprint wrXiv:2403.11703 2024a.
Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, and Lifu Huang. Vision-flan: Scaling human-labeled tasks in visual instruction tuning. arXiv preprint arXiv:2402. 11690, 2024b.
11690,
Qinghao Ye, Haiyang Xu, Jiabo Ye, Ming Yan, Anwen Hu, Haowei Liu, Qi Qian, Ji Zhang, and Fei Huang. mplug-owl2: Revolutionizing multi-modal large language model with modality collaboration. In Proceedings of the ICEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13040-13051, 2024.
Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfusion: Rethinking image-text data at scale. In Proceedings of the IEEE/CVT Conference on Computer Vision and Pattern Recognition, pp. 14022-14032 14032, 2024.
Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang, Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.022 024990 2023.
Lifan Yuan, Ganqu Cui, Hanbin Wang, Ning Ding, Xingyao Wang, Jia Deng, Boji Shan, Huimin Chen, Ruobing Xie, Yankai Lin, et al. Advancing Ilm reasoning generalists with preference trees. arXiv preprint arXiv:2404.02078, 2024.
Xiang Yuc, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-disciplin multimodal understanding and reasoning benchmark for expert agi. arXiv preprint arXiv:2311.1655 2023.
25

---

Published in Transactions on Machine Learning Research (11/2024)
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? arXiv preprint 1XXv:1905.078300 2019.
Yan Zeng, Hanbo Zhang, Jiani Zheng, Jiangnan Xia, Guoqiang Wei, Yang Wei, Yuchen Zhang, and Tao Kong. What matters in training gpt4-style language model with multimodal inputs? arXiv preprint
arXiv:2307.0248 2023.
Duzhen Zhang, Yahan Yu, Chenxing Li, Jiahua Dong, Dan Su, Chenhui Chu, and Dong Yu. Mm-llms: Recent advances in multimodal large language models. arXiv preprint 1rX1v:2401.13601 2024a.
Pan Zhang, Xiaoyi Dong, Bin Wang, Yuhang Cao, Chao Xu, Linke Ouyang, Zhiyuan Zhao, Shuangrui Ding, Songyang Zhang, Haodong Duan, Wenwei Zhang, Hang Yan, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, and Jiaqi Wang. Internim Internlm-xcomposer xcomposer: A vision-language large model for advancod text-image comprehension and composition. arXiv preprint arXiv:2309.15112 2023a.
Yanzhe Zhang, Ruiyi Zhang, Jiuxiang Gu, Yufan Zhou, Nedim Lipka, Diyi Yang, and Tong Sun. Llavar: Enhanced visual instruction tuning for text-rich image understanding, arXiv preprint arXiv:230..17107, 2023b.
Yongting Zhang, Lu Chen, Guodong Zheng, Yifeng Gao, Rui Zheng, Jinlan Fu, Zhenfei Yin, Senjie Jin, Yu Qiao, Xuanjing Huang, Feng Zhao, Tao Gui, and Jing Shao. Spa-vl: A comprehensive safety preference alignment dataset for vision language model, 2024b.
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models. CoRR, 2023.
Yuke Zhu, Oliver Groth, Michael Bernstein, and Li Fei-Fei. Visual7w: Grounded question answering in images. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4995-5004, 2016.
26

---

Published in Transactions on Machine Learning Research (11/2024)
Table 7: Summary of datasets used the supervised fine-tuning stage.
Category
Dataset
#Sample
General (Wang et al., 2024b) Uttralnteract-SFT (Yuan al., 2024)
71K
Language-Only
288K
UltraChat (Ding et al., 2023)
207K
IVIS-Instructa (Wang et al., 2023a)
223K
ShareGPTTV (Chen et al., 2023b)
102K
Detailed Image Caption
LAION-GPTAY (LAION)
12K
Localized Narratives (Pont-Tuset et al., 2020)
200K
VSR (Liu al., 2023a)
2K
TQA (Kembhavi al., 2017) ScienceQA (Lu et al., 2022a)
2K
Scientific Document
5K
IconQA (Lu al., 2021)
27K
TabMWP (Lu et al., 2022b)
23K
ChartQA (Masry et al., 2022)
18K
Table, Document, and Chart
VisText (Tang et al., 2023)
7K
Chart2Text (Obeid & Hoque, 2020)
27K
DVQA (Kafle et al., 2018)
20K
FigureQA (Kahou al., 2017)
20K
Diagram Image-to- Text (Kamizuru)
300
Infographic VQA (Mathew et al., 2022)
2K
ST-VQA (Biten et al., 2019)
17K
OCR and Text-Ries Rich Images
TextCaps (Sidorov et al., 2020)
22K
VQA (Singh et al., 2019)
22K
OCR-VQA (Mishra et al., 2019)
17K
Rendered-Tes (Wendler)
10K
HatefulMemes (Kiela et al., 2020)
8.5K
OK-VQA (Marino al., 2019)
9K
AOK-VQA (Schwenk et al., 2022)
16.5K
TallyQA (Acharya et al., 2019)
100K
General VQA
Visual7W (Zhu et al., 2016)
14K
COCO-QA (Ren et al., 2015)
46K
VQAV2 (Goyal al., 2017)
82K
GQA (Hudson Manning, 2019)
72K
Appendix
A Details of Instruction-tuning Data Curation
The curated instruction fine-tuning data mixture is shown in Tab. 7. Each category is chosen to address specific challenges and capabilities of SOLO. For instance, datasets like Ultralnteract-SFT (Yuan et al., 2024) and CodeAct-General (Wang et al., 2024b) enable the refinement. of language processing and reasoning abilities, while visually rich datasets such as IIS-Innstt (Wang et al., 2023a) and Localized Narratives (Pont- Tuset et al., 2020) enhance the model's basic image understanding and recognition abilities. Scientific document datasets like TQA (Kembhavi et al., 2017) are included to bolster the model's ability parse and reason with academic visual information. Furthermore, OCR and text-heavy. image datasets such as TextCaps (Sidorov et al., 2020) and OCR- VQA (Mishra et al., 2019) provide crucial source for the model's ability to interpret text within complex images. By selecting datascts with a broad range of complexities, sizes, and focuses, we ensure a robust. fine-tuning process that prepares SOLO to handle real-world applications elfectively, reflecting a deep and detailed understanding of both vision and language data. Additionally, we conduct a thorough manual inspection and comparisons of the fine-tuning datasets, employing random sampling techniques on some datasets such as DVQA (Kafle et al., 2018) and FigureQA (Kahou et al., 2017) to guarantee diversity and prevent data imbalance.
27

---

Published in Transactions on Machine Learning Research (11/2024)
Effectiveness of Stage-3 Pre-Training
Ablation of the Fine-Tuning Data Mixture
Stece-3
SOLO web
Singe
70
Morc GTT Dote
Extendeed Thuining
60
60
Perfrrmncc
50
Performance
.50
40-
30
30
20
10
10
MMStar
MME
SEED
ScienceQA Mathvista
A1D0
MMStar
MME
SEED
ScienceQA MethVista
A/20
(a) The effectiveness of Stage-3 pre-training in priming SOLO for the instruction fine-tuning stage.
(b) The ablation study the fine-tuning data mixture.
Figure 13: The evaluation performance of various ablations to validate the effectiveness of Stage-3 pre-traiiee pre-training and the fine-tuning data mixture.
B Stage-3 Annealing
We directly perform instruction fine-tuning the pre-trained SOLO finished at Stage-2 to understand the effect of the annealing stage. The results shown in Fig. 13a. indicate that introducing annealing stage to conclude pre-training can slightly promote the performance across all evaluation benchmarks.
C Effectiveness of Curated Data Mixture for Instruction Fine- Tuning
We conduct an ablation study to validate the curated data mixture for instruction fine-tuning. The ablations included are: (1) Without GPT-4V Data: All data generated by GPT-4V, including detailed captions and instructional fine-tuning samples, excluded from the fine-tuning mixture. (2) With Additional OCR Data: Additional OCR data from LLaVAR is incorporated into the fino-tuning mixture to enhance OCR capabilities, which are crucial for tasks like require extract text information from charts. (3) With More GPT-4V Data:
Data from GPT-4V used in the third stage of pre-training is added to the fine-tuning mixture. (4) Extended Training Duration: SOLO is trained for additional epoch to investigate the effects of prolonged training.
The results are presented in Fig. 13b. Our analysis indicates that incorporating additional OCR data does not significantly enhance performance in scientific document comprehension or visual mathematical reasoning tasks, which rely extensively on visual text understanding. This lack of improvement can be attributed to the discrepancy between general OCR data and the specific demands of scientific charts. Identifying effective methods for collecting OCR data pertinent scientific chart comprehension remains critical area for future rescarch. Furthermore, incorporating this OCR data scems to adversely affect overall visual-language capabilities, demonstrated by general benchunrrks Regarding the use of GPT-4V data, our findings suggest. that measured inclusion during the pre-training annealing stage enhances performance (see ŞB), whereas excessive incorporation during finc-tuning can hurt the overall performance. We also find that SOLO exhibits minimal performance decline when trained solely on existing supervised datasets, excluding all data generated by GPT-4V. This demonstrates that GPT-4V data not essential for enhancing the core capabilities of SOLO. The results overall justify the choice of datasets included in the supervised fine-tuning data mixture. However, although we observe continual performance improvement across training steps within epoch (see Fig. 5d), prolonged training on repetitive samples could lead to overfitting and decreased performance. This suggests that while extended exposure to diverse training data generally enhances model performance, overfitting remains critical challenge when models are exposed repeatedly to a limited data subset. Overall, the ablation study proves the effectiveness of our curated data mixture.
28


Reviews:
[{"review_id": "2HEjZL3T52", "reviewer": "Reviewer_Fr5d", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n1. **Strong Empirical Results with Unified VLM Architecture**: the proposed model architecture (similar to early models like VisualBERT and PixelBERT) shows strong performance, particularly outperforming in Level-1 and Level-2 categories.\n\n2. **Thorough Experimental Analysis**: The authors conduct comprehensive experiments to validate SOLO's effectiveness across various benchmark datasets. They also provide detailed insights into the training process, highlighting key factors such as the importance of different stages in sequential pre-training, performance improvements with increased token length, and gains from higher image resolution.\n\n\n### Weaknesses\n\n1. **Ambiguity in the Notion of \"Scalability\"**: The paper’s central claim is the \"scalability\" of LVLM, but the definition of scalability is somewhat vague. Typically, scalability refers to expanding the model architecture or dataset size. However, in this paper, scalability focuses on visual capacity, token length, and image resolution. The scope of scalability should be clarified to ensure readers fully understand which aspects are being scaled.\n   -  Based on my understanding, the authors also argue that the unified architecture could offer greater benefits if it becomes scalable in terms of model size and dataset in the future. However, to support this claim, additional experiments and analyses focused on scaling the model size and dataset should be actually included. Without such experiments, the claim of future scalability remains speculative.\n\n2. **Unclear Motivation and Goal (Position)**:  While the authors identify four limitations, the need for a unified single transformer remains unclear. \n\n  -  For  1) Fixed and Constrained Visual Capabilities and 4) Limited Image Pre-Processing Flexibility:  the limitations seem to stem from using pre-trained visual encoders (e.g., CLIP ViT). The difference between previous approaches and this work appears to be based on two factors: 1) the use of a pre-trained visual encoder or training from scratch and 2) the depth of the visual extractor (deep vs. shallow). A more detailed analysis of these two factors is needed.\n     - Moreover,  the authors suggest that increasing visual capacity can be achieved by training a shallow visual encoder from scratch in stage-1, without initializing from a pre-trained encoder. However, it’s unclear whether this represents a genuine contribution, or if a shallow encoder is truly better than a deep one.\n     - Couldn’t a deep visual encoder (with or without initialization) also be trained in stage-1? What is the specific advantage of the unified model over these alternatives?\n     - Additional analyses on performance improvements relative to computational costs, particularly regarding these two factors, are necessary.\n\n - For 3) Multiple Components Complicate the Scaling Analysis: the rationale for why a unified model simplifies scaling analysis is unclear. In fact, with previous models, using multiple components allows for a more isolated evaluation of each modality’s encoders. In contrast, the unified model appears more difficult to analyze since the effects of all components are intertwined, making it harder to assess each modality’s individual contribution.\n - For 2) Challenges in Large-Scale Training and Deployment: If possible, more discrete and real examples would be helpful as to why the model with multiple components is hard to train and deploy. \n\nGiven the above points and the potentially heavy computational overhead (my guess), it is uncertain whether the proposed model and training recipe offer significant practical benefits.\n\n\n3. **Insufficient explanation in FIgure 8**: The performance improvements per token depicted in Figure 8 require a more detailed explanation of the setting and experiments. It’s unclear why SOLO shows greater improvement than LLaVA, and what this result means in terms of the model's overall capability. \n\n4. **Limitations should be described**: the paper lacks a discussion on the limitations of the proposed model. From my understanding, SOLO likely incurs higher computational costs compared to previous methods. Additionally, a comparison of model parameter size with LLaVA would be useful for understanding the trade-offs between the two models.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The authors present the Unified End-to-End Transformer, dubbed SOLO, mitigating four scalability issues they identified: limited visual capabilities, difficulties in large-scale training and deployment, complexity in scaling analysis due to multiple components, and restricted flexibility in image pre-processing. Their model is fully open-source, accompanied by a detailed training recipe that includes initialization, sequential pre-training on diverse datasets, and instruction fine-tuning. Through extensive evaluation, SOLO demonstrates performance comparable to LLaVA-v1.5-7B, with notable strengths in visual mathematical reasoning.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "All requested changes are described in the Weaknesses section", "broader_impact_concerns": "No additional concerns on the ethical implications of this work", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "TNMsrJJ1tC", "reviewer": "Reviewer_4ucY", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- Novel unified architecture: The single Transformer approach for both vision and language processing is innovative and addresses several scalability issues in current LVLMs.\n- Comprehensive training recipe: The detailed three-stage pre-training process and instruction fine-tuning provide a valuable blueprint for future open research.\n- Extensive ablation studies: The paper includes thorough analyses that validate the key components of their approach.\n- Strong performance: SOLO achieves comparable or superior performance to existing LVLMs, particularly in visual mathematical reasoning tasks.\n- Scalability analysis: The paper provides some insights into the scaling behavior of SOLO. However, this analysis could be more comprehensive and comparative with other LVLM architectures.\n\nWeaknesses:\n\n- Insufficient exploration of dynamic high-resolution capabilities: While SOLO supports flexible image resolutions, the paper lacks a thorough analysis of its performance with dynamic aspect ratios and very high-resolution images. This is particularly notable when compared to models like LLaVA-Next, which uses the same Mistral-7B base but achieves better performance, largely attributed to its dynamic high-resolution technique. The paper doesn't adequately address whether SOLO fully leverages its potential to process images with dynamic aspect ratios and high resolutions to boost performance. A more in-depth analysis and comparison in this aspect would have been valuable, especially given that this capability is a key factor in the superior performance of some competing models.\n- Incomplete analysis of scaling laws: The paper mentions the importance of reliable scaling laws early on, but does not sufficiently demonstrate how SOLO's unified architecture facilitates easier derivation of scaling laws compared to other LVLMs. Additionally, the paper lacks discussion on how scaling laws might differ when considering data modalities separately (image-only, image-text, text-only) versus collectively.\n- Lack of inference speed comparison: Despite the potential speed advantages of the unified architecture, the paper doesn't provide comparisons of inference speed with other models, particularly more recent LVLMs.\n- Insufficient explanation of framework choices: The paper doesn't adequately explain the rationale behind using different training frameworks (Megatron and Accelerate) for different stages, nor does it quantify the benefits of these choices.\n- Limited exploration of intermediate datasets: The transition from single-label datasets like ImageNet to noisy web-crawled captions isn't explored in depth, missing potential insights from intermediate datasets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper introduces SOLO, a novel approach to vision-language modeling that employs a single Transformer architecture for both image and text processing. The key contributions are:\n- A unified architecture that potentially addresses scalability limitations in current large vision-language models (LVLMs) that use separate visual encoders and language models.\n- A comprehensive training recipe, including a three-stage pre-training process and instruction-supervised fine-tuning, implemented using moderate academic resources (8 x A100 80GB GPUs).\n- Empirical evidence showing SOLO's performance is comparable to state-of-the-art LVLMs across various benchmarks, with particular excellence in visual mathematical reasoning tasks.\n- Detailed ablation studies validating the effectiveness of each component in the training recipe.\n- Analysis of SOLO's scaling behavior, demonstrating improved performance with increasing image resolution.\n\nThese contributions are likely to be of significant interest to TMLR's audience, particularly those working in the fields of vision-language modeling, multimodal AI, and large language models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Critical changes:\n\n1. Provide a more comprehensive analysis of scaling laws. This should include:\na) A clearer demonstration of how SOLO's architecture facilitates easier derivation of scaling laws compared to other LVLMs.\nb) Discussion on how scaling laws might differ when considering data modalities separately (image-only, image-text, text-only) versus collectively.\n2. Include inference speed comparisons with other LVLMs to substantiate the potential advantages of the unified architecture.\n3. Expand on the rationale behind using different training frameworks (Megatron and Accelerate) for different stages. This should include:\na) A discussion of the potential difficulties practitioners might face when using Megatron.\nb) Quantitative data on the speed advantages of Megatron over Accelerate during pre-training.\n\nSuggested improvements:\n\n1. Provide a more comprehensive analysis of SOLO's performance with dynamic aspect ratios and very high-resolution images. This should include a direct comparison with models like LLaVA-Next, which have shown significant performance gains through advanced high-resolution techniques. Such an analysis would help clarify whether SOLO is fully leveraging its flexible resolution capabilities to maximize performance.\n2. Explore the impact of intermediate datasets (e.g., OpenImages and its comprehensive label set) for stage 1 pre-training as a bridge between single-label datasets like ImageNet and noisy web-crawled captions. Analyze how incorporating such datasets might affect model performance and understanding of complex visual concepts.\n3. Investigate and report on how the ratio of image to text tokens during pre-training affects the model's performance. This analysis would provide valuable insights into the optimal balance between visual and linguistic inputs for large-scale vision-language models, particularly at the 7B parameter scale.", "broader_impact_concerns": "The authors have adequately addressed potential negative impacts of LVLM development in their Limitations and Broader Impact Statement. They demonstrate awareness of the risks of misuse and the potential for bias amplification. This section concisely covers key ethical considerations, which appears appropriate for the scope of the paper.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "rk6uUxiNNm", "reviewer": "Reviewer_Wo6i", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strength: The proposed method is simple and demonstrates performance that, while somewhat inferior, is comparable to existing LVLMs.\n\n- Weakness: The main concern is the lack of novelty. The architectural contribution is limited, as the method closely resembles the VisualBert work. The authors themselves have stated that the proposed SOLO work is inspired by VisualBert. Additionally, I am concerned about the performance, as methods involving other vision encoders generally outperform the SOLO approach. Furthermore, the authors claim that heterogeneous architectures have limitations, such as difficulties with hardware infrastructure, but there is no empirical evidence or investigation of this issue presented in the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a training recipe called Scalable Vision-Language Modeling (SOLO) for training a Large Vision-Language Model (LVLM), from ImageNet-scale pretraining to instructional tuning (fine-tuning). Unlike the LLaVA method, SOLO does not utilize an auxiliary vision encoder to project images into the LLM input embedding space. Instead, it directly maps image patches with a linear projection layer to allow image patches to be inputs of the LLM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "Please address the weaknesses to improve the paper. It would be beneficial to provide more analysis on the advantages of the proposed VisualBert-style architecture compared to LLaVA or other LVLM architectures.", "broader_impact_concerns": "Nothing special to be noticed in this paper. This paper proposes general LVLM training recipe, and would be good to consider impact of generative AI.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: I think the overall contribution of this paper is sufficient to be published as a TMLR paper. However, there are some minor issues that need a minor revision.

1. Clarity: One main issue of the current version is clarity. I don't mean clarity in terms of grammar or fluency, but I mean clarity in terms of terminology. For example, as many reviewers pointed out, this paper uses "scalability" in various meanings. I don't think "an architecture can demonstrate sustained performance improvement through scaling (e.g., more computational resources and/or data) without hitting any inherent bottleneck in the machine learning system" is a rigorous definition. What is "inherent bottleneck in the system"? How is the bottleneck defined? Why do the existing methods have the bottleneck? If they have the bottleneck, can we actually observe the performance degeneration by scaling up? I encourage the authors to polish the manuscript to enhance readability and clarity.
2. The claims may need additional refinement. For example:
    - Constrained visual capabilities: It is not well-supported. If this claim is true, then one might observe any flaw of the existing heterogenous VLMs in terms of "performance". However, this paper does not show the performance limitation of heterogenous VLMs. It it is impossible to support this empirically, I'd like to recommend removing this from the main claim. It could be described in "discussion".
    - Challenging large-scale training and deployment: Inference speed is not the only bottleneck in training and deployment. This argument might be clarified by "better inference speed". If inference speed is not the only reason, then it should be empirically supported as the other claims.
3. I don't think the "Appendix" of this paper is not optional. Since TMLR does not have a page limitation (it is only used for deciding the review period), I strongly recommend moving Appendix contents to the main paper, especially for Section G.
Decision: Accept (poster)