AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding
Authors: Ahmed Masry
Juan A. Rodriguez
Tianyu Zhang
Suyuchen Wang
Chao Wang
Aarash Feizi
Akshay Kalkunte Suresh
Abhay Puri
Xiangru Jian
Pierre-Andre Noel
Sathwik Tejaswi Madhusudhan
Marco Pedersoli
Bang Liu
Nicolas Chapados
Yoshua Bengio
Enamul Hoque
Christopher Pal
Issam H. Laradji
David Vazquez
Perouz Taslakian
Spandana Gella
Sai Rajeswar
Venue: NeurIPS 2025 poster ()
Keywords: vision language models
multimodal
large language models
Primary area: deep_learning

Abstract: Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), lack inductive bias to constrain visual features within the linguistic structure of the LLM’s embedding space, making them data-hungry and prone to cross-modal misalignment. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where visual and textual modalities are highly correlated. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods, with larger gains on document understanding and under low-resource setups. We provide further analysis demonstrating its efficiency and robustness to noise.

ALIGNVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding
Ahmed Masry" 1.2, Juan A. Rodriguez!: 13.4, Tianyu Zhang 133,5 Suyuchen Wang Chao Wang'. Aarash Feizi Akshay Kalkunte Suresh', Abhay Puri Xiangru Jian Pierre-André Noël Sathwik Tejaswi Madhusudhan' Marco Pedersolil.1, Bang L,, Nicolas Chapados', Yoshua Bengio Enamul Hoque?, Christopher Pal .3,893 Issam H. Laradji David Vazquez',
Perouz Taslakian', Spandana Gella*, Sai Rajeswar
ServiceNow York University Mila Quebec AI Institute
*École de Technologie Supérieure UUniversité de Montréal
McGill University 'University of Waterloo
$CIFAR AI Chair
Polytechhique Montréal
Universiiy of British Columbia
Abstract
Aligning visual features with language embeddings key challenge in vision- language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by vision encoder a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), lack inductive bias to constrain visual features within the linguistic structure of the LLM's embedding space, making them data-hungry and prone to cross-modal misalignment. In this work, we propose novel vision-text alignment method, ALIGNVLMM that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. ALIGN V VLM is particularly effective for document understanding tasks, where visual and textual modalities are highly correlated. Our extensive experiments show that VLM achieves state-of-the of-the-art performance compared to prior alignment methods, with larger gains on document understanding tasks and under lew-rese setups. We provide further analysis demonstrating its efficiency and robustness to noise.
Introduction
Vision-Language Models (VLMs) have gained significant traction in recent years as a powerful framework for multimodal document understanding tasks that involve interpreting both the visual
and textual contents of scanned documents (Kim et al. [002 Lee et al.) 2003 Liu et al. 2023a 2024
2024b). Such tasks are common in real world Hu al. 2024 Wang al., 2023a] Rodriguez al.]
commercial applications, including invoice parsing Park et al.) 2019]. form reading (Jaume et al. 2019), and document question answering [Mathew et al. 2021b). VL.M architectures typically consist of three components: (i) vision encoder to process raw images, (ūi) a Large Language Model (LLM) pre-trained on text, and (iil) a connector module that maps the visual features from the vision encoder into the LLM's semantic space.
central challenge in this pipeline is effectively map the continuous feature embeddings of the vision encoder into the latent space of the LLM while preserving the semantic properties of visual concepts. Existing approaches can be broadly categorized into deep fusion and shallow fusion methods. fusion methods, such as NVLM [Dai et al., 2024], Flamingo [Alayrac et al. 2022
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

CogVLM Wang et al. 2023b), and LLama 3.2- Vision [Grattation et al. 2024 integrate visual and textual features by introducing additional cross-attention and feed-forward layers each layer of the L.M. While effective at enhancing cross-modal interaction, these methods substantially increase the parameter count of the VI.M compared the base L.LM, resulting high computational overhead and reduced efficiency.
In contrast, shallow fusionn methods project vi- sual features from the vision encoder into the LLM input embedding space using either mul- tilayer perceptrons (MLPs) [Liu et al. 2023b 2024). convolution mappings such as Honey- Bee Cha et al. [024] and H- H-Reducer [Hu et al.)
DeepForm
KLC
63.49
52.02
InfoVQA
30.8
57
08
1244
2024j. ,, or attention-bann based mechanisms such as
WTQ 38.59
52.56
the Perceiver Resampler (Li et al.
2023b Lau- 2022]. This
23:13
rençon et al., 2024 Alayrac et al.
28.04
20/53
79
63
DocVQA
56
approach is more parameter-cfficient and com- putationally lighter than deep fusion method However, these connectors lack inductive bias to ensure that the projected features remain within the region spanned by the LLM's pre- trained text embeddings. Consequently, the projected visual features may fall outside the distribution the L.LM was trained on, leading to noisy misaligned representations. More- over, these mappings are typically learned from scratch, making them data-inefficient and less effective under low-resource conditions.
78.51
21.83
43.95
55.96
TabFact
53.93
62:10
60/1
67.92
05:43
52.
53.56
TableVQA
71.88
ChartQA
57.38
TextVQA
Ulama-3.2 2.-....----Pereeeeerr
Llama-3.2 30mma-3.2-3----
Liama-3.2-8 MLP
Liama-3.2 3amma-3.-38-AAlpp (ours)
Figure 1: Performance of Different VLM Connec-
tors. The proposed ALIGN connector outperforms other methods across benchmarks using the same training configuration. Radial distance proportion
Recent methods like Ovis [Lu et al. 2024] at- tempt to alleviate these issues by introducing separate visual embeddings indexed from the vision encoder outputs and combined together construct the visual inputs the LLM. However, this approach significantly increases parameter count due to the massive embedding matrix and requires extensive training to learn a new embedding space without guaranteeing alignment with the LLM's input latent space.
of maximal score, truncated at 0.7 (black dot).
To address these limitations, this paper introduces ALIGNV LLM, novel framework that sidesteps direct projection of visual features into the LLM embedding space. Instead, our proposed connector, ALIGN, maps visual features into probability distributions over the LLM's existing pretrained vocabu- lary embeddings, which then combined into weighted representation of the text embeddings. By constraining each visual feature as a convex combination of the LLM text embeddings, our approach leverages the linguistic priors already encoded in the LLM's text space. This ensures that the resulting visual features lie within the convex hull the LLM's embedding space, reducing the risk of noisy or -iitt--distiitttion inputs and improving alignment between modalities. The connector thus enables faster convergence and stronger performance, particularly in low-resource scenarios.
Our experimental results show that ALIGN improves performance on various document understanding tasks, outperforming prior connector methods, with especially large gains in low-data regimes. We summarize our main contributions as follows:
We propose novel connector, ALIGN, to bridge the representation gap between vision and text modalities.
We introduce family of Vision-LLaguse Models, ALIGNVLM,, that achieves state-of-the- art performance on multimodal document understanding tasks by leveraging ALIGN.
We conduct extensive experiments demonstrating the robustness and effectiveness of ALIGN across different LLM sizes and training data setups.
We release our code and research artifacts at alignylm biignvl..ithub... github.io

---

2 Related Work
2.1 Vision-Language Models
Over the past few years, Vision-Language Models (VLMs) have achieved remarkable progress,
largely due to advances in Large Language Models (LLMs). Initially demonstrating breakthroughs
in text understanding and generation [Brown et al. 2020 Raffel et al., 2023 Achiam et al. 2023
Grattafiori et al.] 2024 [wen et al.
2025 Team [024]] LLMs are now mcreasingly used to effectively
interpret visual inputs [Liu et al.
2023b Li et [0224, Wang et al. 2024 Chen et al. 2024b Dai
et al., 2024, Drouin et al. 2024 Rodriguez et al. [022]] This progress has enabled real-world
applications across diverse domains, particularly in multimodal document understanding for tasks like
form reading [Svetlichnaya 2020]. document question answering [Mathew et al. 2021b), and chart question answering |Masry et al. 2022]. VLMs commonly adopt a componett architecture: a
pretrained vision encoder [Zhai et al.] [202]] Radford al. [022]] LLM, and connector module.
A key challenge for VLMs is effectively aligning visual features with the LLM's semantic space to
enable accurate and meaningful multimodal interpretation.
2.2 Vision-Language Alignment for Multimodal Models
Existing vision-language alignment approaches can be classified into deep fusion and shallow fusion. Deep fusion methods integrate visual and textual features by modifying the LLM's architecture, adding cross-attention and feed-forward layers. For example, Flamingo [Alayrac et al. [2022] employs
the Perceiver Resampler, which uses fixed latent embeddings to attend to vision features and fuses them into the LLM via gated cross-attention layers. Similarly, NVLM (Dai et al.) 2024 adopts cross-gated attention while replacing the Perceiver Resampler with simpler MLP. Cog VLM (Wang] eet al. 2023b) extends this approach by incorporating new feed-forward (PFN) and QKV layers for the vision modality within every layer of the LLM. While these methods improve cross- modall alignment, they significantly increase parameter counts and computational overhead, making them less efficient.
On the other hand, shallow fusion methods are more computationally efficient, mapping visual features into the LLM's embedding space without altering its architecture. These methods can be categorized into three main types: (I)
mapping, such as LLa VA (Liu et al., 2023b and PaliGemma [eeyer et al. 2024]. which use multilayer perceptrons (MLP) to project visual features but often produce misaligned or noisy features due to lack of constraints and inductive bias (Rodriguez et al., 2024b); (2) cross-attention mechanisms, BLIP-2 [Li et al. 20236] uses Q-Former, which utilizes fixed set of latent embeddings
artss- to visual features, but that may still produce noisy or OOD visual features; (3) convolution- based mechanisms, such as HoneyBee [Cha et al. 2024] and H-Reducer (Hu et al. 20244, ), which leverage convolutional ResNet (He et 2015 layers to preserve spatial locality while reducing dimensionality; and (4) visual embeddings, such as those introduced by Ovis [Lu et al. 2024), which embeddings indexed by the vision encoder's outputs to produce the visual inputs. While this regularizes feature mapping, it adds substantial parameter overhead and creates a new vision embedding space, risking misalignment with the LLM's text embedding space. fincode-ffree VLMs, like Fuyu-8B and EVE Diao et al. 2024), climinate
dedicated vision encoders but show degraded performance Beyer et al. 2024
In contrast, ALIGN maps visual features from the vision encoder into probability distributions over the LLM's 's text embeddings, using them to compute convex combination. By leveraging the linguistic priors encoded in the LLM's vocabulary, ALIGNVLM. ensures that visual features remain within the convex hull of the text embedding. This design mitigates noisy or out-of-distribution projections and achieves stronger multimodal alignment, particularly in tasks that require joint modalities representation like multimodal document understanding and in low-resource settings.
3 Methodology
3.1 Model Architecture
The overall model architecture, shown in Figure consists of three main components:
3

---

Module
caliy sulet
6 1h IN
wing ailr
Vision Encoder
Weighted Averago
Full Emdondting MANTE
Visio Tapuss
LLM
Text Tokenizer
Selected Text Embeldinng
Question: What percentuge of
online?
1LM Embedding Monix
Text Iepurs
Response: 90%
Figure 2: ALIGNVLM Model Architecture. The vision encoder extracts image features, which are processed to produce probabilities the LLM embeddings. A weighted average combines these probabilities with embeddings to generate vision input vectors. Text inputs are tokenized, and the corresponding embeddings are selected from the embedding matrix, which is then used as input to the LLM. We display the vision layers in blue and the text layers purple
(1) Vision Encoder. handle high-resolution images of different aspect ratios, we divide each input
image into multiple tiles according to one of the predefined aspect ratios (e.g.. 1:1, 1:2, ..... 9:1) chosen via coverage ratio [Lu et al. 2024| Chen et al. 2024a]. Due to limited computational resources, set the maximum number of tiles to 9. Each tile is further partitioned into 14 x 14 patches, projected into vectors, and processed by SigLip-400M vision encoder [Zhai et al. 2023 to extract contextual visual features.
Each tile {1..... , divided into No patches
P. = (Pr.,... .....,PP.M.. p.,N,,,
where Pt, is the i-th patch of tile t. The vision encoder maps these patches to a set of visual feature vectors
F, VisionEnccoeer(P.). F, {fr,,." N. fe, e Ra.
Finally, we concatenate the feature sets across all tiles into single output
= concat (Fi, F2,, Fr).
(2) ALIGN Module. This module aligns the visual features with the LLM. A linear layer W E RDxd first projects the visual features F T to the LLM'' token embedding space: one R
vector per token. A second linear layer W2 E R VxD (initialized from the LLM's language-model head) followed by softmax, produces probability simplex P vocab over the LLM's vocabulary (V
tokens)
P vocab =
Doftmax((ayerrNort) (W2 LayerNorm((W (W,
(1)
We then use the LLM text embeddings
Exxt E VV to compute weighted sum
F' Falign
P
Vocab
E
(2)
Finally, we concatenate F'
align with the tokenized text emheddings to form the LLM input
H Hinput concat (F' Erext (x));
where (x) obtained by tokenizing the input text (011--. xTM) and selecting the corre- sponding embeddings from E such that
(x) [ (xI1) Exc((
(3)

---

(3) Large Language Model. We feed the concatenated vision and text vectors, Hput into the LLM, which then generates output text auto-regressively. To demonstrate the effectiveness of our alignment technique, we experiment with the Ilama 3.1 model family [Grattafiori et al. [024]. These models offer state-of-the-an performance and permissive licenses, making them suitable for commercial applications. In particular, we utilize Llama 3.2- 1B, Llama 3.2-3B, and Llama 3. 3.B.
3.2 Motivation and relation with existing methods
By construction, each RD representation in F Falign is constrained to the convex hull of the points E thus concentrating the visual features in the part of latent space that the L.M effectively interpret. Moreover, we argue that our initialization of W2 to the language model head is an inductive bias toward recycling some of the semantics of these text tokens into visual tokens. This contrasts with past methods that have been proposed to adapt the vision encoder outputs F E R xd to an F' R N to be fed the LLM. Here, consider examples in more detail, highlighting these contrasts.
(I) MLP Connector Liu et al. [023b) applies linear projection with parameters W Rxd and DMLP E RP, followed by an activation function a (e.g., ReLU)
F MIP o (MMLLPP + byLP).
These parameters are all learned from scratch, without any bias aligning them to text embeddings.
(2) Visual Embedding Table Lu et al. [2024] introduces an entire new set of visual embeddings EvEr RKXD which, together with the weights W VVE RKxdd, specifies
F VET softmax W F)
EEETT
When < d, our W₂ W amounts to a low-rank version of WET. There thus much more to learn to obtain F and there is again no explicit pressure to align it with the text embeddings.
3.3 Training Datasets & Stages
We train our model in three stages:
Stage 1. This stage focuses on training the ALIGN Module to map visual features to the LLM's text embeddings effectively. We use the CC-12M dataset Changpinyo al. [202]] a large-scale web dataset commonly used for VLM pretraining Liu et al. [2023b], which contains 12M image-text pairs. However, due to broken or unavailable links, we retrieved 8. 8.IM pairs. This dataset facilitates the alignment of visual features with the text embedding space of the LLM. During this stage, we train the full model, as this approach improves performance and stabilizes the ALIGN Module training.
Stage 2.
The goal is to enhance the model's document understanding capabilities, such as OCR, document structure comprehension, in-depth reasoning, and instruction-folloiing We leverage the BigDocs-7.5M dataset [Rodriguez et al. [2024a] curated collection of license- license-permistt permissive datasets for multimodal document understanding. This dataset aligns with the Accountability, Responsibility. and Transparency (ART) principles Bommasani et al. (223), Vogus and Llansóe [2021]. ensuring compliance for commercial applications. As Stage 1, we train the full model during this stage.
Stage 3. To enhance the model's instruction-tuie tuning capabilities, particularly for downstream tasks like question answering, we further train it on the DocDownstream Rodriguez et al. [2024a] Hu et al. [2024] instruction tuning dataset. In this stage, the vision encoder is frozen, focusing training exclusively on the LLM and ALIGN module.
Experimental Setup
Setup. We conduct all experiments using 8 nodes of H100 GPUs, totaling 64 GPUs. For model training, we leverage the MS-Swift framework (Zhao et al. 2024 for its flexibility. Additionally, we utilize the DeepSpeed framework (Aminabadi et al. 2022 specifically the ZeRO-3 configuration. to optimize efficient parallel training across multiple nodes. Detailed hyperparameters are outlined in Appendix A.
5

---

Table I: Main Results on General Document Benchmarks. We compare ALIGNVLM (ours) with sfate-of-the-ar (SOTA) open and closed-source instructed models, and with base mod- els that we trained using the process described in Section 333 ALIGNVLM models outper- form all Base VLM models trained in the same data regime. Our models also perform com- petitively across document benchmarks even compared with SOTA models, in which the data regime is more targeted and optimized. Color coding for comparison: closed-source models
open- open-souree models below 7B parameters
open-source models between 7-12B parameters
Doe Inc.
4
Info vaa
DespForm naar
KLC
WTO
TahFaet
ChariQA
TexYV
A
TableVQA uoor
Arg. Scure
Model
Closed-Source VLMs
(Opagee Truining Data)
Claude-3.5 Sonnet
88.48
59.05
31,41
24.82
47.13
53.48
51.84
71.42
81.27
$6.54
GeminiPro-I.5
91.23
73.94
32.16
24.07
50.29
71.22
34.68
68.16
80.43
58.46
GPT-4o 20240806
92.80
66.37
38.39
29.92
46,63
81.10
85.70
70.46
72.87
64.91
Source Instruct VLMs
(Somi Opeque Troining Date)
Janus 1.33
Wa 20124
30.15
17.09
0.62
15.06
9.30
51.34
57.20
51.97
18.67
27.93
Qwen2.V1.
Wany ala
2014
89.16
64.11
32.38
25.18
38.20
57.21
73.40
79.90
43.07
55.84
Qwen2.5.VL.
31
2024
93,00
75,83
32.84
24.82
53,46
71.16
83.91
79.29
71.66
65.10
IntemVL-22.5-22
Chvi el al.
20246
87.70
61.85
13.14
16.58
36.33
57.26
74.96
76.85
42.20
51.87
IntemVL-3-2BB
Ziu et
20,5
87.33
66.99
37.90
29.79
39.44
59.91
75.32
78.66
43. 46
57.64
DeepSeak-Vi.2-11
WW
2024b
88.57
25.11
19.04
07
52.15
80.92
80,48
56.30
55.72
Phi3.5-Visioo-Bt
Abbin al P0R0
86.00
56.20
10.47
7,49
17.18
30.43
82.16
73.12
70,70
48.19
Qwen2-VL-77B Waing 21 20241
93.83
76.12
34.55
23.37
52.52
74.68
83.16
84.48
53.97
64.08
Qwen2,5-VL-7BB [B3)
94.88
82.49
42,21
24.26
61.9%
78.56
86.00
85,35
76.10
70.20
LLaVA-NeXT-7R Xil 2004
63.51
30.90
5.35
20.06
52.83
65.10
32.87
36.00
DocOw11.5 8B
el ai 2024
80.73
49.94
68.84
37.99
38.87
79.67
68.56
68.91
$2.60
60.68
IntemmVL22.5888 [Chen el 2024b
91.98
22.31
50.33
74.75
79.00
52.10
62.58
IntemVLL 8.B 7hi mi 20.25
91.99
73.90
51.24
36.41
$3.60
72.27
85.60
82.41
$3.26
66.74
Fuyu-8B Bavishi
48.97
14.55
22.97
Ovis-1.6-C 6-00--6-mm
ei al 2024
88. 84
73.97
45.16
23.91
50.72
76.66
81.40
77.73
48.33
62.96
Llama 3.mma.2-111B 2-11B Grattafoor
82.71
36.62
1.78
3.47
23.03
58.33
23,80
54.28
22.40
34.04
Pixtral- 12B
Agrawal et al. 2024
87.67
49.45
27.37
24.07
48.18
73.53
71.80
76.09
67.13
58.03
Document Understanding Instructed Models
Ulstriction Tinee on BigDecs-7.MM + DecDownSrrem (Rodrignes et al. 2024a Hu et af. 2024)
Dwenn-VLL-2B (basc+) [Wagg al., 2024
2024
57.23
31.88
49.31
61.01
47:53
49.59
ALIGNVL.M M.cNNV.M--llaa-3 3.2ma-3.2-133 (ours)
72.42
38.16
60.47
33.71
28.66
71.31
65.44
48.81
50.29
52.
ALIGNVLM LM Llama- 3,2-3B (ours)
79.63
44.53
63.49
36.25
38.59
78.51
71.88
57.38
60.10
58.81
DocChel1. I.5 SB (haset) Ho et al 2024
78.70
47.62
64.19
36.93
15,09
12.65
65.50
67.30
49.03
37.56
Llama3.2 1-B (buset) Graliatio
2024
78.99
44.27
67.05
37.22
40.18
78.04
71.40
68.46
56.73
60.26
ALIGNVLM Llama-3. 8B5 (aurs)
$1.18
53.75
63.25
35.50
45.31
83.04
75.00
64.60
64.33
62.88
Baselines. Our work focuses on architectural innovations, so we ensure that all baselines are trained on the same datasets. To enable fair comparisons, evaluate our models against set of Base VLMS fine-tuned on the same instruction-tuning tasks (Stages 2 and 3) as our models, using the BigDocs-7.55M and SigDoos-DocDowneeee data, avoiding biases introduced by the Instruct versions of VLMs, which are often trained on undisclosed instruction-tuniin
datasets. This approach ensures consistent training
datasets. Due to the scarcity of recently released publicly available Base VLMs, we primarily compare our model against the following Base VLMs of varying sizes: Qwen2-VL-2B [Wang al. 2024), DocOwl1.5-88
(Hu et al. 2024], and LLama 3.2- 1 Grratafiioii
et al. 2024].
For additional context, we also include results from the Instruct versions of recent VLMs of different
sizes: -iiio--V [Abdin et al.] 2024), DDwe2--VL-22 and 7B [Wang et al. 2024 Qwen2.5-
VL-7B [Qwen et al., 2025 LLa NeX-TTB [Lu et al.)
2024 Intern VL V22.5-2522 and 8B (Chen]
ct al. 2024b), Intern VL3-2B and 8B [Zhu et al. 2025) Janus- 1.3B [Wu et al. 2024a]. DeepSeek-
VL2 VL2-Tiny [Wu et al. 2024b). , Ovisl.6 6-Gemma-99
[Lu et al. 2024]. Llama3.2- IB |Grattafiori et al.)
2024
DocOwI1.5-8B
Hu et al. 2024]. and Pixtral-12h Agrawal et al. 2024
Evaluation Benchmarks. We evaluate our models on a diverse range of document understanding benchmarks that assess the model's capabilities in OCR, chart reasoning, table processing, or form comprehension. In particular, we employ the VLMEvalKit [Duan et al. 2024] framework and report
the results on the following popular benchmarks: VQA [Mathew et al. 202 InfoVQA (Mathew
et al. 2021a DeepForm [Svetlichnaya] 2020). KLC [Stanislawek et al. 2021 J. WTQ Pasupat and
Liang 2015), TabFact [Chen et al.) 2020]. ChartQA [Masry et al. 2022
TextVQA Singh et al.
2019 and Table VQA [Kim et al. 2024

---

Table 2: Impact of Connector Designs VLM Performance: We present the results of experi- ments evaluating different connector designs for conditioning LLMs on visual features. Our proposed ALIGN connector is compared against basic Multi-Layer Perceptron (MLP), the Perceiver Re- sampler, and Ovis. The results demonstrate that ALIGN consistently outperforms these alternatives across all benchmarks.
DocVQ
InfoVQA
Deep
TabFact
CharlQA TRIm
Tex/VQA
Table VQA TEST
Avg. Score
Test
KLC
WTQ
TEST
Model
Llama-3.2-3B 3.233--LLB
71.46
37.56
62.07
33.36
94
73.22
66.48
53.56
50,96
$3.06
IJama-J 3.2-3B- Perciever R.
69.08
34.13
57.08
31.75
27.95
71.93
65.16
51.33
47.76
50.68
Llama-3.2 3.2--- 12-3B-Ovis Ovis
74.68
42.11
58.02
33,50
33.13
76.67
67.92
$2.60
53.93
54.72
1lama-3.2--85 28- ALIGN (ours)
79.63
44.53
63.49
35,25
38.59
78.51
57.38
5%.81
5
Results
5.1 Main Results
Table
presents the performance of ALIGNVLM compared state-of-the-art (SOTA) open- and closed-source instructed models, as well as baseline Base VLMs fine-tuned in the same instruction- tuning setup. The results demonstrate that ALIGNVLM consistently outperforms all Base VLMs within the same size category and achieves competitive performance against SOTA Instruct VLMs limited data regime. Below, we provide a detailed analysis.
despite being trained
ALIGNVLM vs. Base VLMs. Our ALIGN models, based on Llama 3.2-1B and Llama 3.2- 3B, significantly outperform the corresponding Base VLM, Qwen2-VL-2B, by up to 9.22%. Notably, ALIGN LIGNVYLM-Llaaa. L.a-3- surpasses DocOwll.55 1,.-8B,, which has 4B more parameters, demonstrating the effectiveness of ALIGN in enhancing multimodal capabilities compared traditional shallow fusion methods (e.g., MLPs). Furthermore, our 8B model achieves a 2.62% improvement over Llama3.2-11 despite sharing the same Base LLM, Llama3.1-8B Since all models in this comparison were trained on the same instruction-tuning setup, this experiment provides controlled evaluation, isolating the impact of architectural differences rather than dataset biases. Consequently, these results suggest that ALIGNVLM outperforms VLMs with shallow fusion techniques and surpasses parameter-heary deep fusion VLMs, such as Llama3.2-11BB while maintaining a more efficient architecture.
ALIGNVLM vs. Instruct VLMs. Even as open-source Instruct models are trained on significantly larger, often undisclosed instruction-tuninn tunnngg datasets, ALIGNVLM achieves competitive performance. For example, VLIGNVLM-Llammn 3.2-3BB (58.81%) outperforms other strong instruction-tun VLMs in its size class, such as Qwen2-VL-29 and InternVL-3-1 by considerable margins (2.97% and 1.17%, respectively). While it falls slightly behind Qwen2.5-VL-33 a direct comparison is not entirely fair, the latter was trained a proprietary instruction-tuning dataset.
Additionally, our SB model outperforms significantly larger models such as Llama 3.2-11B and PixTral- 12B by substantial margins. It also surpasses Intern VL-2.5-8B and performs competitively with Qwen2.5-VL-7BB. though a direct comparison may not be entirely fair since Qwen2.5-Vi VLL-7 was trained undisclosed intructiin-tuningg dataset. Finally, ALIGNVLM also exhibits comparable performance to closed-source models like GeminiPro-I.5 and GPT4o.
Overall, these results validate the effectiveness of ALIGN and establish ALIGN VLM as a state-of-the- art model for multimodal document understanding.
5.2 Impact of Connector Designs on VLM Performance
5.2.1
High-Resourg Training Regime
To assess the effectiveness of our ALIGN module, we compare it against three different and widely used shallow fusion VLM connectors: MLP, Perceiver Resampler, and Ovis. These experiments were carefully conducted under precisely identical training conditions (datasets, hyperparameters, training stages) as outlined in Appendix A. ensuring a fair and rigorous comparison. The results in Table 2 show that ALIGN consistently outperforms all alternatives, demonstrating its superiority
7

---

Table 3: Connector Performance under a Low-Resource Training Regime: We evaluate the effectiveness of more shallow-fusin fusion connectors when trained on limited data. The ALIGN connector achieves the highest performance, with notably larger gains on document understanding tasks, demonstrating its data efficiency and strong inductive bias.
Model
Document Understanding Tasks
General Vision Tasks
DovVQA
IntoVGA
Charga
TesVQA
Avg.
MMMU
SeeBBench
MMVa
POPE
GQA
Llama-3.2-3B 2-38-MLP
42.11
19.93
48.44
51,97
40.61
33.33
58.54
31.14
87.35
57.62
Llama-3 Llama-3.2-38-Percee P-8-Perrceee
32.18
18.10
40.00
44.31
33.64
35.22
63.70
26.
84.92
55.86
53.17
L.ama-3.2.3B-Ovis
57.73
26.39
54.52
55.60
48.56
31,89
60.97
30.41
88.26
56.23
53.55
1lama-3.2330 Hrrruee
34.59
17.57
45.64
47.13
36.23
35.00
61.82
28.39
87.48
58.24
54.18
Liama-3,2-3B-Houp. Bee
55.86
19.36
55,32
58.13
47.16
32.11
61.18
34.31
89.28
54.79
54.33
L.ama332..B--AIIc. (ours)
71.43
30.50
69.72
63.63
59.32
35.33
63.27
35.32
88.85
61.67
56.88
both in aligning visual and textual modalities in multimodal document understanding. MLP and Perceiver Resampler achieve the lowest performance, 53.06% and 50.68%, respectively, due to their direct feature projection, which lacks an explicit mechanism align visual features with the LLM's text space, leading to misalignment. Ovis introduces separate visual embedding table, but this additional complexity does not significantly improve alignment, yielding only 54.72% accuracy. In contrast, ALIGN ensures that visual features remain within the convex hull of the LLM's text latent space, leveraging the linguistic priors of the LLM to enhance alignment and mitigate noisy embeddings. This design leads to the highest performance (58.81%) %),, establishing ALIGN as the most effective connector for integrating vision and language in multimodal document understanding. We provide some example outputs of the Llama-3. 3.B-BB models with different connector designs in Appendix A..4, Furthermore, we include an analysis the runtime efficiency and memory usage of different connectors in Appendix A.2
5.2.2 Low-Resource Training Regime
The previous section focused on large-scale training setups involving millions of data samples (BigDocs-7.5MD) which require significant compute resources and limit the number of baselines that we were able to compare against. Here, we examine whether ALIGN remains effective a low-resource setting.
We conduct additional experiments using SigLIP-400M the vision encoder and Llama-3.2 3.2--BB
as the language model, fine-tuned on the LLaVA-NeXX dataset Liu et al. [2024]. which contains 779K samples. We follow the official VA-Ne-Ne configuration for both training stages. (i) Pretraining: the model is trained on the LLaVA-558K image-caption dataset Liu et al. [0024] freezing both the LLM and vision encoder while fine-tuning the connector (learning rate 1e-3, batch size 32, epoch on x H100 GPUs). To handle high-resolution document images, adopt the "anyres max_ 9" strategy with grid weaving from Ixl to 6x6, supporting resolutions up to 2304x2304 with 729 tokens per grid; (ii) Instruction tuning: the model is further fine-tuned on the LLaVA-NeXT-7799 instruction dataset with learning rates of le-5 for the LLM and connector, 2 for the vision encoder, batch size for epoch.
This lightweight setup allows direct comparison across more connector architectures including MLP et al. 2023a]. Perceiver Resampler, Ovis Lu et al. [2024) H-Reducer (1x4) Hu et al. [024], and Honey Bee (C-Abstractor) Cha et al. [002], all trained under identical conditions for fairness. Since the V.V-eeee dataset is general purpose and not exclusively seccument-focusee -parpooe
like BigDocs-7.5M [Rodriguez et al.) 2024a). allows us to evaluate whether the ALIGN connector generalizes beyond document understanding broader visual reasoning. Accordingly, we assess all models on a comprehensive suite of benchmarks spanning both document understanding and general vasion-language tasks. The document understanding benchmarks include Doc Mathew et al.
20211b. Info VQA Mathew et al. 2021a] ChartQA Masry et al. [2022], and TextVQA Singh et al. 2019]. J. For general langonel evaluation, report results MMMU-dev Yue al. [2224] SeedBench Li et al. 2023a], and MM Yu et al. (202), Pope [Li et al. 2023c and GQA Hudson
and Manning 2019
As summarized in Table AL.GN consistently outperforms other connectors under this low-data regime, with stronger gains document understanding tasks. The wider performance margin
8

---

Table 4: Performance comparison when evaluating ALIGN with the full text embedding vocabulary (128K) versus the reduced subset of 3.4K high-probability embeddings. The results show negligible performance degradation, indicating that ALIGN relies primarily small subset of embeddings.
DoeVOA
Iafo VAA Iill
DeepFarm Mai
TobFact
ChartQ4
TeuvQs vel
Table QA mar
Arg. Scure
KLC
WTO
Model
Llama-3 3. 3.23B-ALIGN (Fuil Embeddings)
79.63
44.53
63,49
35,25
38.59
78.51
71.88
$7.38
60. 10
58.81
Llama-3.2 3.2:3B-AL1OG (3.4K Embeddings)
79.40
44.13
63.64
35.02
38.26
78.83
71.72
57.48
59.80
58.69
between ALIGN and others connectors under limited data (Table B compared to the high-resource setting (Table 2 underscores the benefit of its inductive bias. By grounding visual features within the LLM's s text embedding space, ALIGN learns more efficiently from fewer samples, unlike direct- projection connectors that rely heavily on large datasets. This makes ALIGN especially valuable for resource-constrained environments such as academic labs or small-scale industrial research setups, where both data and compute are limited.
5.3 Probability Distribution over Text Tokens Analysis
To better understand the behavior of ALIGN, we examine the probability distribution, Pvocab in Eq ( over the LLM's text vocabulary generated from visual features. Specifically, we process 100 document images through the vision encoder and ALIGN, then average the resulting probability
distributions across all image patches. The final distribution shown in Figure 3 As illustrated, the distribution is dense (rather than sparse). with the highest probability assigned to a single token being 0.0118. This can be explained by the vision feature space being continuous and of much higher cardinality than the discrete text space. Indeed, while the LLM has 128K distinct vocabulary tokens, an image patch (e.g., 14x14 pixels) contains continuous, high-dimensional information that cannot be effectively mapped to single or a few discrete tokens.
We conducted deeper analysis of the token probability distributions produced by the ALIGN connector. Our observations show that ALIGN consistently assigns high probabilities to approximately 3.4K tokens from the entire vocabulary, while the remaining tokens receive negligible probabilities (below 10 9) To better understand this behavior, we applied Principal Component Analysis (PCA) to reduce the dimensionality of the embeddings and visualized them in a two-dimensional space, as shown in Figure 4 The visualization reveals that these 3.4K tokens densely and comprchensively span the latent space of the LLM's text embeddings. To validate this finding, we conducted additional evaluation experiments in which we retained only these 3,4K gigh--robbbbilty embeddings in the ALIGN connector, entirely removing the rest during evaluation. As shown in Table the performance difference compared to using the full embedding set (128K) was negligible. This confirms that ALIGN effectively leverages and combines compact subset of embeddings to map visual features into semantically meaningful regions within the LLM's latent text space. Moreover, this suggests that ALIGN can be further optimized through targeted embedding pruning to improve computational efficiency without sacrificing performance.
5.4 Robustness to Noise Analysis
To evaluate the robustness of our ALIGN connector to noisy visual features, we conduct an experiment where random Gaussian noise is added to the visual features produced by the vision encoder before passing them into the connector. Specifically, given the visual features F E IR Nxd output by the vision encoder (where N is the number of feature vectors and d is their dimensionality), we perturbed them as
F=F+N, N~N
Table 5: Robustness to Noise. Comparison of Avg. Scores with and without Gaussian noise (a = 3). including performance drop (A).
Model
Without Noise
With Noise
Drop (A)
I. 1...lm..2221 3.2-3B-MI.P
53.06
27.52
4 25.54
Llama-3.2 3.2-3B-ALB -B-ALIGN (ours)
58.81
57.14
$ 1.67

---

20 PCA Embbedings (AligNLLM)
Probability Distribution LLM's
0.012
Mesiodaniall tranns Aagrrtt)
0.010
0.008
Pobabiiy
0.000
043
0.004
0.002
0.000
-039
20000
40000
60000
80000
200000
120000
Figure 3: Probability distribution over LLM tokens, highlighting dense probabilities for whitespace tokens.
Figure 4: PCA of ALIGN Embeddings: The principal components of the most influential em- beddings in the Align Connector span most of the feature space represented by all embeddings.
As shown in Table our ALIGN connector demonstrates high robustness noise, with only a 7.67% average drop in performance. In contrast, the widely adopted MLP connector suffers a significant performance degradation of 25,54,, highlighting its vulnerability to noisy inputs. Furthermore, we measured the average cosine distance between the original and noise- noise-perturbed visual embeddings using both the AL.GG and MI.P connectors. ALIGN showed significantly lower distances (0.0036) than MIP (0.3938), further validating its robustness to noise. These empirical results support our hypothesis that leveraging the knowledge encoded in the LLM's text embeddings and constraining the visual features within the convex hull of the text latent space act as a regularization mechanism, reducing the model's sensitivity to noisy visual features.
Conclusion
We introduce ALIGN, novel connector designed to align vision and language latent spaces in Iision-language models (VLMs), specifically enhancing multimodal document understanding. By improving cross-modal alignment and minimizing noisy embeddings, our models, ALIGNVLM, which leverage ALIGN, achieve state-of-the-art performance across diverse document understanding tasks. This includes outperforming base VLMM trained on the same datasets and achieving competitive performance with open-source instruct models trained on undisclosed data. Extensive experiments and ablations validate the robustness and effectiveness of ALIGN compared to existing connector designs, establishing it as a significant contribution to vision-language modeling. Future work will explore training on more diverse instruction- instruction-tuiing datasets to generalize broader domains.
References
M. Ahdin, J. Aneja, H. Awadalla, A. Awadallah, A. A. Awan, N. Bach, Bahree, A. Bakhtiari, Bao, H. Behl, A. Benhaim, M. Bilenko. J. Bjorck, S. Bubeck, M. Cai, Q. Cai, Chaudhary, D. Chen, D. Chen, Chen. Y.-C. Chen, Y.-L. Chen, H. Cheng, P. Chopra, X. Dai, M. Dixon, R. Eldan, V. Fragoso, J. Gao, M. Gao, M. Gao, A. Garg, A. D. Giomo, A. Goswami, S. Gunasekar, E. Haider. J. Hao, R. J. Hewett, W. Hu, J. Huynh, D. Iter, S. A. Jacobs, M. Javaheripi, X. Jin, N. Karampatziakis, P. Kauffmann, M. Khademi, D. Kim, Y. Kim, L. Kurilenko, J. R. Lee, Y. T. Lee, Li, Li, Liang, L. Liden, X. Lin, Z. Lin, C. Liu, L. Liu, M. Liu, W. Liu, X. Liu, C. Luo, P. Madan, A. Mahmoudzadch, D. Majercak, M. Mazzola, C. C. T. Mendes, A. Mitra, H. Modi, A. Nguyen, B. Norick, B. Patra, D. Perez-Becker, T. Portet, R. Pryzant, H. Qin, M. Radmilac, I. Ren, G. de Rosa, C. Rosset, S. Roy, O. Ruwase, O. Saarikivi, A. Saied, A. Salim, M. Santacroce, S. Shah, N. Shang, H. Sharma, Y. Shen, S. Shukla, X. Song. M. Tanaka, A. Tupini, P. Vaddamanu, C. Wang. G. Wang, L. Wang. S. Wang. X. Wang. Y. Wang. R. Ward, W. Wen, P. Witte, H. Wu, X. Wu, M. Wyatt, B. Xiao, Xu.
J. Xu, W. Xu, J. Xue, S. Yadav, F. Yang, J. Yang. Y. Yang. Z. Yang. D. Yu, L. Yuan, C. Zhang, C. Zhang, J. Zhang, L. L. Zhang, Y. Zhang, Y. Zhang. Y. Zhang, and X. Zhou. Phi-3 technical report: A highly capable language model locally on your phone, 2024. URL httpstts:://////m //atxpp /arxiv. org/ ab/////4.... 14219
10

---

J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXtv:2303.08774, 2023.
Agrawal, Antoniak, E. B. Hanna, B. Bout, D. Chaplot, J. Chudnovsky, D. Costa, D. Monicault, Garg. T. Gervet, S. Ghosh, A. Héliou, P. Jacob, A. Q. Jiang. K. Khandelwal, T. Lacroix, G. Lample, D. L. Casas, T. Lavril, T. L. Scao, A. Lo. W. Marshall, L. Martin, A. Mensch, P. Muddireddy, V. Nemychnikova, M. Pellat, P. V. Platen, N. Raghuraman. B. Rozière, A. Sablayrolles, L. Saulnier, R. Sauvestre, W. Shang. R. Soletskyi, L.. Stewart, Stock, Studnia, S. Subramanian, S. Vaze, T. Wang, and S. Yang. Pixtral 12h, 2024. URL. httpp////aaaivoor/aaaa/////s///e
J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lone, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi, T. Han, Z. Gong. S. Samangooei, M. Monteiro, J. Menick, S. Borgeaud, A. Brack, A. Nematzadeh, S. Sharifzadeh, M. Binkowski, R. Barreira, O. Vinyals. A. Zisserman, and K. Simonyan. Flamingo: a visual language model few-shot learning, 2022. URL WtRp::tts://arzii org/
abs/2204. 14198
R. Y. Aminabadi, S. Rajbhandari. M. Zhang. A. A. Awan, C. D. Li, and Y. He. Deepspeed inference: Enabling efficient inference of transformer models at unprecedented scale, 2022. htt../ttpp://rxii.orr/////// org/ ab0/ 2207. 00032
E. Zheng. J. Rasley, S. Smith, O. Ruwase,
Anthropic. The claude 3 model family: Opus, sonnet, haiku. 2024.
S. Bai, K. Chen, X. Liu, J. Wang, W. Gc, S. Song, K. Dang, P. Wang. S. Wang. J. Tang. H. Zhong, Y. Zhu, M. Yang, Z. Li, J. Wan, P. Wang. W. Ding, Z. Fu, Y. Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang. Z. Yang. H. Xu, and J. Lin. Qwen2.5-vl technical report, 2025. URL https:/: /atpiv/aivvvv..a wrgg absp//02a2222.... 2502. 13923
R. Bavishi, F. Elsen, C. Hawthome, M. Nye, A. Odena, A. Somani, and S. Taşırlar. Introducing our multimodal models, 2023. URL httpps////w adept. ai wttbl////f/uue foyoog//uuuubbb 8b
L. Beyer. A. Steiner, A. S. Pinto, A. Kolesnikov, X. Wang. D. Salz. M. Neumann, I. Alabdulmohsin. M. Tschan- nen, E. Bugliarello, T. Unterthiner, D. Keysers. S. Koppula, F. Liu, A. Grycner, A. Gritsenko, N. Houlsby. M. Kumar, K. Rong. J. Eisenschlos,
Kabra, M. Bauer, M. Bošnjak, X. Chen, M. Minderer, P. Voigtlaender, P. Papalampidi, O. Henaff, X. Xiong, R. Soricut, J. Harmsen, and X. Zhai. org/
Bica, Balazevic, Puigeerver, Paligemma: A versatile 3b vim for transfer, 2024. URL httthhttt/s/aaaet
07726
R. Bommasani, K. Klyman, S. Longpre, S. Kapoor. N. Maslej B. Xiong, D. Zhang, and P. Liang. The foundation model transparency index, 2023. URL. http::tttp///rtttiie org/ hat///b///// wbs//310. 12941
Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, Shyam, G. Sastry, A. Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 18:1 2020.
J. Cha. W. Kang. Mun, and B. Roh. Honeybee: Locality-enhanced projector for multimodal Ilm, 2024. URL https://arxiv wwwpxii.oogg//bae.. 2b3/2
S. Changpinyo, P. Sharma, N. Ding. and R. Soricut. Conceptual 12m: Pushing web- image-text pre- wre-trinning to recognize long-tail visual concepts, 2021. URL httpt://////xi org/ 2202 08981
W. Chen, H. Wang. J. Chen, Y. Zhang, H. Wang. S. Li. X. Zhou, and W. Y. Wang. Tabfact: A large-scale dataset for table-based fact verification. In International Conference Learning Representations, 2020.
Z. Chen, W. Wang. H. Tian, S. Ye, Z. Gao, E. Cui, W. Tong, K. Hu, J. Luo, Z. Ma, J. Ma. J. Wang. X. Dong, H. Yan, H. Guo, C. Hc, B. Shi, Z. Jin, C. Xu, B. Wang, X. Wei, W. Li, W. Zhang, B. Zhang, P. Cai, L. Wen, X. Yan, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, and W. Wang- How far are to gpt-4v? closing the gap to commercial multimodal models with open-source suites, 2024a. URL attpa://aate arxiv. org/ abs/ 22/4224444..19
Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic lisual-lingustic tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Paltern Recognition, pages 24185-24198. 2024b.
W. Dai, N. Lec, Wang, Z. Yang, Z. Liu, J. Barker, T. Rintamaki, M. Shocybi, B. Catanzaro, and W. Ping. Nvlm: Open frontier-class multimodal lmss. arXiv preprint arxiv: 2409. 11402, 2024.
H. Diao, Y. Cui. X. Li, Y. Wang. H. Lu, and X. Wang. Unveiling encoder- free vision- language models. arXiv preprint 2406 11832, 2024.
11

---

A. Drouin, M. Gasse, M. Caccia, I. H. Laradji, M. D. Verme. T. Marty, L. Boisvert, M. Thakkar, Q. Cappart. D. Vazquez, N. Chapados, and A. Lacoste. Workarena: Hlw capable are web agents solving common knowledge work tasks?, 2024. http:/tss//aaav.rrr//aaaaa/
07718
H. Duan, J. Yang. Y. Qiao, X. Fang, L. Chen, Y. Liu, X. Dong, Y. Zang. P. Zhang, J. Wang, et al. VImevalkit: An open-source toolkit for evaluating large Conference on Multimedia, pages 11198 11201, 2024.
modcls. In Proceedings of the 32nd ACM International
A. Duhey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan, A. Goyal, A. Hartshom, A. Yang. A. Mitra, A. Sravankumar, and et al. The llaaa 3 herd of models. arXiv preprint 27X12:240.211783, 2024.
A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughan, A. Yang, A. Fan, A. Goyal, A. Hartshorn, A. Yang. A. Mitra, A. Sravankumar, A. Korenev, A. Hinsvark, A. Rao, A. Zhang, A. Rodriguez, A. Gregerson, A. Spataru, B. Roziere, B. Biron, B. Tang. B. Chem, C. Caucheteux, C. Nayak, C. Bi, C. Marra, C. McConnell, C. Keller, C. Touret, C. Wu, C. Wong.
C. C. Ferrer, C. Nikolaidis, D. Allonsius, D. Song, D. Pintz, D. Livshits, D. Wyatt, D. Esiobu, D. Choudhary, D. Mahajan, D. Garcia-Olanoo D. Perino, D. Hupkes, E. Lakomkin, E. AlBadawy, E. Lobanova, E. Dinan, E. M. Smith, F. Radenovic, F. Guzmán, F. Zhang. G. Synnaeve, G. Lee, G. L. Anderson, G. Thattai, G. Nail, G. Mialon, G. Pang. G. Cucurell, H. Nguyen, H. Korevaar, H. Xu, H. Touvron, Zarov, I. A. Ibarra, Kloumann, I. Misra, I. Evtimov, Zhang. J. Copet, Lee, J. Geffert, J. Vranes, J. Park, Mahadeokar, J. Shah, J. van der Linde, J. Billock, J. Hong, J. Lee, J. Fu, J. Chi, J. Huang, J. Liu, J. Wang. J. Yu, J. Bitton, J. Spisak, J. Park, J. Rocca, J. Johnstun, J. Saxe, J. Jia. K. V. Alwala, K. Prasad. K. Upasani, K. Plawiak, K. Li, K. Heafield, K. Stone, K. El-Arini, K. Iyer, K. Malik, K. Chiu, K. Bhalla, K. Lakhotia, L. Rantala- Yeary. Martin, Madaan, L. Malo, L. Blecher. L. Landzaat, L. de Oliveira, M. Muzzi, M. Pasupuleti, M. Singh, M. Paluri, M. Kardas, M. Tsimpoukelli, M. Oldham, M. Rita, M. Paylova, M. Kambadur, M. Lewis, M. Si, M. K. Singh, M. Hassan, N. Goyal, N. Torabi, N. Bashlykov, N. Bogoychev, N. Chatterji, N. Zhang, O. Duchenne, O. Çelebi, P. Alrassy, P. Zhang, P. Li, P. Vasic, P. Weng. P. Bhargava, Dubal, P. Krishnan, S. Koura, Xu, Q. He. Q. Dong. R. Srinivasan, R. Ganapathy. R. Calderer. R. Cabral, R. Stojnic, R. Raileanu. R. Maheswari, R. Girdhar, R. Patel. R. Sauvestre, R. Polidoro, R. Sumbaly, R. Taylor. R. Silva, R. Hou, R. Wang. S. Hosseini, S. Chennabasappa,
L. van der Maaten, L. Chen, L. Tan, L. Jenkins,
S. Singh, S. Bell, S. S. Kim, S. Edunov, S. Nic, S. Narang, S. Raparthy, S. Shen, S. Wan, S. Bhosale, S. Zhang, S. Vandenhende. S. Batra, S. Whitman, S. Sootla, S. Collot, S. Gururangan, S. Borodinsky, Herman, Fowler, T. Sheasha, T. Georgiou. Scialom. Speekbacher. T. Mihaylov, Xiao. U. Karn, V. Goswami, V. Gupta, V. Ramanathan, V. Kerkez, V. Gonguet, V. Do, V. Vageti, V. Albiero, V. Petrovic, W. Chu, W. Xiong, W. Fu, W. Meers, X. Martinet, X. Wang, X. Wang. X. E. Tan, X. Xia, X. Xie, X. Jia, X. Wang. Y. Goldschlag, Y. Gaur, Y. Babaei, Y. Wen, Y. Song, Y. Zhang, Y. Li, Y. Mao, 7. D. Coudert, Z. Yan, Z. Chen, Z. Papakipos, A. Singh, A. Srivastava, A. Jain, A. Kelsey, A. Shajnfeld, A. Gangidi, A. Victoria, A. Goldstand, A. Menon, A. Sharma, A. Boesenberg. A. Baevski, A. Feinstein, A. Kallet, A. Sangani. A. Teo, A. Yunus, A. Lupu, A. Alvarado, A. Caples, A. Gu, A. Ho, A. Poulton, A. Ryan, A. Ramchandani, A. Dong, A. Franco, A. Goyal, A. Saraf, A. Chowdhury, A. Gabriel, A. Bharambe, A. Eisenman, A. Yazdan, B. James, B. Maurer, B. Leonhardi, B. Huang, B. Loyd, B. D. Paola, B. Paranjape, B. Liu, B. Wu, B. Ni, Wasti, B. Spenee, B. Stojkovic, B. Gamido, B. Montalvo, C. Parker, C. Burton, C. Mcjia, C. Liu, C. Wang. C. Kim, C. Zhou, C. Hu, -.H Chu, C. Cai. C. Tindal, C. Feichtenhofer. C. Gao, Civin. D. Beaty, D. Kreymer. D. Li, D. Adkins, D. Xu, D. Testuggine, D. David, D. Parikh, D. Liskovich, D. Foss, D. Wang, D. Le. D. Holland, E. Dowling, E. Jamil, E. Montgomery, E. Presani, E. Hahn, E. Wood, E.-T. Le, E. Brinkman, E. Arcaute, E. Dunbar, E. Smothers, F. Sun, F. Kreuk, F. Tian, F. Kokkinos, F. Ozgenel, F. Caggioni, F. Kanayet, F. Seide, G. M. Florez, G. Schwarz, G. Badeer, G. Swee, G. Halpern, G. Herman, H. Inan, H. Shojamazeri, H. Zou, H. Wang. H. Zha, H. Habech, H. Rudolph, H. Suk, H. Aspegren, H. Goldman, H. Zhan, I. Damlaj, I. Molyhog, I. Tufanow, I. Leontiadis, I.-F. Veliche, I. Gat, J. Weissman, J. Geboski, J. Kohli, J. Lam, J. Asher, J.-B. Gaya, J. Marcus, J. Tang. J. Chan, J. J. Reizenstein, J. Teboul, Zhong, Jin, J. Yang. Cummings, Carvill, Sheepard, McPhie, J. Torres. Ginsburg. J. Wang. K. Wu. K. H. U, K. Saxena, K. Khandelwal. K. Zand, K. Matosich K. Veeraraghavan, K. Michelena, K. Li, K. Jagadeesh, K. Huang, K. Chawla, K. Huang, L. Chen, L. Garg. L. A. L. Silva, L. Bell. L. Zhang, L. Guo, L. Yu, L. Moshkovich, L. Wehrstedt, M. Khabsa, M. Avalani, M. Bhatt, M. Mankus, M. Hasson, M. Lennie, M. Reso, M. Groshev, M. Naumov, M. Lathi, M. Kencally, M. Liu, M. Seltzer, M. Valko, M. Restrepo, M. Patel, M. Vyatskov, M. Samvelyan, M. Clark, M. Macey, M. Wang, M. J. Hermoso, M. Metanat, M. Rastegari, M. Bansal, N. Santhanam, N. Parks, N. White, N. Bawa, N. Singhal, N. Egebo, N. Usunier, N. Mehta, N. P. Laptev, N. Dong, N. Cheng, O. Chernoguz, O. Hart, O. Salpekar, O. Kalinli, P. Kent, P. Parekh, P. Saab, P. Balaji, P. Rittner, P. Bontrager, P. Roux, P. Dollar, P. Zvyagina, P. Ratanchandani, P. Yuvraj. Q. Liang, R. Alao, R. Rodriguez. R. Ayub, R. Murthy, R. Nayani, R. Mitra, R. Parthasarathy, R. Li, R. Hogan, R. Battey, R. Wang. R. Howes, R. Rinott, S. Mehta, Siby, S. J. Bondu, S. Datta, S. Chugh, S. Hunt, S. Dhillon, S. Sidorov, S. Pan, S. Mahajan. S. Verma, S. Yamamoto, S. Ramaswamy. S. Lindsay, S. Lindsay. S. Feng. S. Lin, S. C. Zha. S. Patil, S. Shankar. S. Zhang, S. Zhang, S. Wang. S. Agarwal, S. Sajuyigbe. S. Chintala, S. Max. S. Chen, S. Kehoe, S. Satterfield, S. Govindaprasad,
B. Hancock,
G. Sizov, Guangyi. Zhang, G. Lakshminarayanan, A.
12

---

S. Gupta, S. Deng, S. Cho. S. Virk, S. Subramanian. S. Choudhury, S. Goldman, T. Remez, T. Glaser,
T. Best, T. Koehler, T. Robinson, T. Li, T. Zhang. T. Matthews, T. Chou, T. Shaked, V. Vontimitta, V. Ajayi, V. Montanez, V. Mohan. V. S. Kumar, V. Mangla. Ionescu, V. Poenaru, V. T. Mihailescu, Ivanov, W. Li, W. Wang, W. Jiang, W. Bouaziz, W. Constable, X. Tang. X. Wu, X. Wang, X. Wu, X. Gao, Y. Kleinman, Y. Chen, Y. Hu, Y. Jia, Y. Qi, Y. Li, Y. Zhang. Y. Zhang. Y. Adi, Y. Nam, Yu, Wang, Y. Zhao, Y. Hao, Y. Qian, Y. Li, Y. He. Z. Rait, Z. De Z. Rosnbrick. Z. Wen, Z. Yang. Z. Zhao, and Z. Ma. The llama herd of models, 2024. httpssttt://// //wpiv .org abaaa/aaa2224 240. 21783
K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition, 2015. URL https:
//arxiv. , org/abs/ abs/
1512.11..000 aba/a1aa...1/.//////11//11/11111 033855
A. Hu, H. Xu, J. Ye, M. Yan, L. Zhang. B. Zhang, C. Li, J. Zhang, Q. Jin, F. Huang, and Zhou. mplug-docowl docowl 1.5: Unified structure learning for ocr ocr-free document understanding, 2024. URL Uttps///: org/abaa 2403. 12895
D. A. Hudson and C. D. Manning. Gqa: A new datasct for real-world visual reasoning and compositional question answering, 2019. URL /tttp:://axiii org/ abs/ 10tp/99900000
G. Jaume, H. K. Ekenel, and J.PP. Thiran. Funsd: A dataset for form understanding in noisy scanned documents, 2019. URL httthhtt///arttti wwwp http//bb//99900 13538
G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang. S. Yun, D. Han, and S. Park. Ocr-free document understanding transformer, 2022. URL wwwpaawwwwww......a/r/aaaaeetaee argg/bbs/211111 15664
Kim, M. Yim, and K. Song. Tablevqa-benee A visual question answering benchmark on multiple table domains. arXiv preprint arXiv:2404. 19205, 2024.
H. Laurençon, L. Tronchon, M. Cord, and V. Sanh. What matters when building vision-language models?, 2024. URL hattt:://aaxiii:/ org/ abs//bba///// abs/a///// 2//.//440..000 02246
K. Lee, M. Joshi, I. Ture, H. Hu, F. Liu, J. Eisenschlos, U. Khandelwal, P. Shaw, M.-W. Chang, and K. Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding, 2023. URL /ttps://
arxiv.org/aat wwtp//////a....o..//////// 23344 03347
B. Li, R. Wang. G. Wang. Y. Ge, Y. Ge, and Y. Shan. Seed-bench: Benchmarking multimodal liss with generative comprehension, 2023a. URL https::/aaxx...org//aattteeee/ee
16125
B. Li, Y. Zhang, D. Guo, R. Zhang, F. Li, H. Zhang, K. Zhang, P. Zhang. Y. Li, Liu, and C. Li. Llava-onevision: Fasy visual task transfer, 2024. URI. https:ttts://arre http://arivvvorraaa/a/e/a///// 2tt.//408...000 033266
Li, D. Li, Savarese, and S. Hoi. Blip-2: Bootstrapping languggg-gg pre-training with frozen image 12597
encoders and large language models, 2023b. /RRp:ttts://ariii.rrrr
Li, Y. Du, K. Zhou, J. Wang, W. X. Zhao, and J. Wen. Evaluating object hallucination in large vision- language models, 2023c. URL hattps://arxivv org/ abs/ab/s///// 22050 10355
H. Liu, C. Li, Y. Li, and Y. J. Lee. Improved basclines with visual instruction tuning, 2023a.
H. Liu, C. Li, Q. Wu, and Y. J. Lee. Visual instruction tuning, 2023b.
H. Liu, C. Li, Y. Li, B. Li, Y. Zhang, S. Shen, and Y. J. Loe. Llava-next: Icproved knowledge, January 2024. lttts://://l v1. github..ubbb.... 100010//002444000330------- nextt
reasoning, ocr, and world
S. Lu. Li, Q.G. Chen, Z. Xu, W. Luo, K. Zhang, and H.-J. Ye. Ovis: Structural embedding alignment for multimodal large language model, 2024. URL http::///xxxiiiiii..///a wrg. /405. 20797
A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint 22Xi::2203.102444 2022.
M. Mathew, V. Bagal, R. P. Tito, D. Karatzas, E. Valveny, and C. V. Jawahar. Infographicvqa, 2021a. URL atttp://arxiv, wwtpsivvvorg//eo 2b1/2104.12756
M. Mathew, D. Karatzas, and C. V. Jawahar. Doevqa: A dataset for vqa on document images, 2021b. URL actps://arxivv http:://aaaii...oaaa///////a/ .00998
OpenAl, J. Achiam, S. Adler. S. Agarwal, L. Ahmad, L Akkaya, E. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Balaji. V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, et al. G technical report. arXiv preprint arxiv: 2303.08774, 2023. 2023.
13

---

S. Park. S. Shin, B. Lee, J. Lee, J. Surh, M. Seo, and H. Lee. Cord: A consolidated receipt dataset for post-ocr parsing. Document Intelligence Workshop Neural Information Processing Systems, 2019.
P. Pasupat and P. Liang. Compositional semantic parsing on semi-structured tables. In Annual Meeting of the Association for Computational Linguistics, 2015.
Qwen, :, A. Yang, B. Yang. B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin,
J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang. J. Zhou, J. Lin, K. Dang. K. Lu, K. Bao, K. Yang. L. Yu, M. Li, M. Xue, P. Zhang, Q Zhu, R. Men, R. Lin, T. Li, T. Tang. T. Xia, X. Ren, X. Ren, Y. Fan, Y. Su,
Y. Zhang, Wan. Y. Liu. Z. Cui, Z. Zhang, and Z. Qiu. Qwen2.5 technical report, 2025. URL https
//arxiv. abs//a12 15115
A. Radford, J. W. Kim, C. Hallacy A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021. BtRLhttts:/// http:///axvvvv..o org/ hase//a//22/ 221./2111..000020
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer, 2023. URL https://arxiii org/abs/ 1910. 10683
J. Rodriguez, X. Jian, S. S. Panigrahi, T. Zhang, A. Feizi, A. Puri, A. Kalkunte, F. Savard, A. Masry, S. Nayak, R. Awal, M. Massoud, A. Abaskohi, Z. Li, S. Wang, P--. Noël, M. L.. Richter, S. Vadacchino, S. Agarwal, S. Biswas, S. Shanian, Y. Zhang, N. Bolger, K. MacDonald, S. Fauvel, S. Tejaswi, S. Sunkara, J. Monteiro, K. D. Dvijotham, T. Scholak. N. Chapados, S. Kharagani, S. Hughes, M. Özsu, S. Reddy, M. Pedersoli, Y. Bengio, C. Pal, Laradji. Gella, Taslakian, D. Vazquez, and S. Rajeswar. Bigdoes: An open and permissivelylicenseed dataset for training multimodal models on document and code lasks, 2024a. URL
https://arxiv wwtpaaawwi.o..ggaaa. abgg//bbs/24222. 2412.212222222 04626
J. A. Rodriguez, D. Vazquez, Laradji, M. Pedersoli, and P. Rodriguez. Ocr-vggan: Taming withn--tie generation, 2022.
org/ httg/abb/221..... 2210. 11248
A. Rodriguez, A. Puri, Agarwal, H. Laradji, P. Rodriguez, S. Rajeswar, D. Vazquez, C. Pal, and M. Pedersoli. Starvector: Generating scalable vector graphics code from images and text, 2024b. URL 11556
https://arxiv.out wwtp:///////aaaaa/ea
A. Singh, V. Natarajan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach. Towards vqa models that can read. In IEEE Conference Computer Vision Pattern Recognition, 2019.
Stanistawek, F. Graliáski, A. Wróblewska, D. Lipinskii A. Kaliska, P. Rosalska, B. Topolski, and Biecek. Kleister: key information extraction datasets involving long documents with complex layouts. In International Conference on Document Analysis and Recognition, 2021.
S. Svetlichnaya. Deepform: Understand structured documents at scale, 2020.
G. Team. Gemini: family of highly capable multimodal models, 2024, URL https::/arxxi..rrr /bs// 2312. 11805
C. Vogus and F.. Llansóe. Making transparency mcaningful: A framework for policymakers. Center for Democracy and Technology. 2021.
D. Wang, N. Raman, M. Sibue, Z. Ma. P. Babkin, S. Kaur, Y. Pei. A. Nourbakhsh, and X. Liu. Docllm: A layre generative language model for multimodal document understanding. 2023a. URL https //arxiv. org/ http//aa//2/2000//
P. Wang. S. Bai, S. Tan, S. Wang, Z. Fan, J. Bai, K. Chen, X. Liu, J. Wang, W. Ge, Y. Fan, K. Dang, M. Du, X. Ren, R. Men, D. Liu, C. Zhou, J. Zhou, and J. Lin. Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution, 2024. URL httpt::/ :///r/////xxii www.eorggggaoe aa/aa/ba/2299 12191
W. Wang, Lv. W. Yu. W. Hong, J. Qi, Y. Wang, J. Ji, Z. Yang, L. Zhao, Song, et al. Cogylm: Visual expert for pretrained language models. arXiv preprint arXiv:231 1.0309,, 2023b.
Lv.
C. Wu, X. Chen, 7. Wu, Y. Ma, X. Liu, Z. Pan, W. Liu, 7. Xie, X. Yu, C. Ruan, and P. Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024a. URI, htt.::tttt:///a// org/ abs/2410. 13848
Wu. X. Chen, Z. Pan, X. Liu, W. Liu, D. Dai. H. Gao, Ma. C. Wu, Wang Xie, Wu, K. Hu. J. Wang. Y. Sun, Y. Li, Y. Piao, K. Guan, A. Liu, X. Xie, Y. You, K. Dong, X. Yu, H. Zhang, L. Zhao, Y. Wang, and C. Ruan. Deepseek-v12: Mixture-of expertss vision- language models for advanced multimodal understanding. 2024b. https:tss:/// //arxivv// atg/abss/412..n3 10302
14

---

R. Xu, Y. Yao. Z. Guo. J. Cui. Z. Ni. C. Ge. T.-S. Chua, Z. Liu, M. Sun, and G. Huang. Llava-uhd: an Imm perceiving any aspect ratio and high-resolution images. European Conference Computer Vision, 2024. doi: 10.48550/axx...243..777
W. Yu, Z. Yang. L. Li, J. Wang. K. Lin, Z. Liu, X. Wang. and Wang. Metvet: Evaluating large multimodal models integrated capabilities. 2024. /ttts://://arra
wtgg/aa/////// 02490
X. Yue. Ni, K. Zhang. T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang. W. Ren, Y. Sun, C. Wei, Yu, R. Yuan, R. Sun, M. Yin. Zheng. Z. Yang. Liu, W. Huang, H. Sun, Su, and W. Chen, Mmmu: massive multi discipiei multimodal understanding and reasoning benchmark for expert agi, 2024, URL https://arxiv org/ /....//3....111...
X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training, 2023. URL
https://arxiv. www.orgggaaoo hbs///b/////// 2200. 15343
Zhang. S. Wang, L. Li, G. Zhang, P. Taslakian, S. Rajeswar, J. Fu, B. Liu, and Y. Bengio. Ver: Visual caption restoration. arXiv preprint arXiv: 2406. 064622 2024.
Y. Zhao, J. Huang, J. Hu, X. Wang. Y. Mao, D. Zhang, Z. Jiang, Z. Wu, B. Ai, A. Wang, W. Zhou, and Y. Chen. Swift:a scalable lightweight infrastruuturre for fine-tuning, 2024. URL https://arxiv org/ abs/ 2408 05517
J. Zhu, W. Wang, Z. Chen, Z. Liu. S. Ye, L. Gu, H. Tian, Y. Duan. W. Su, J. Shao, Z. Gao, E. Cui. X. Wang. Y. Cao, Y. Liu, X. Wei, H. Zhang. H. Wang. W. Xu, H. Li, J. Wang, N. Deng, S. Li, Y. Hc, T. Jiang, Luo, Shao, Hc, Xiong, W. Qu, P. Sun, Jiao, H. Lv, Wu, Zhang, H. Deng, J. Ge, K. Chen, L. Wang, M. Dou, L. Lu, X. Zhu, Lu, D. Lin, Y. Qiao, Dai, and W. Wang. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models, 2025. URL https: // arxii.iv... org/ abs/ 10479
Y. Wang, C. Hc, B. Shi, X. Zhang,
15

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All the claims and contributions presented in the abstract and introduction are discussed in Section 3 (Methodology) and supported by results in Section
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations of our approach are discussed in the results section of the paper (Section S
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- speciflcciiii asymptotic approximations only holding locally). The authors should reflect how these assumptions might he violated practice what the implications would
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions
online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [NA]
Justification: The paper does not discuss theoretical results. The main contributions and claims are supported empirical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and refs-eeeferenced.
16

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the details necessary to reproduce our results are provided in the Methodology (Section B). Experimental Sctup (Section 4 and Appendix A.1 A. Additionally, our code will be made available upon acceptance.
Guidelines:
The answer means that the paper does not include experiments.
the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice,
example, if the contribution if the contribution is specific model and empirical evaluation, may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to hosted model (e.g., in the of large language model), releasing of model checkpoint, or other means that are appropriate to research performed.
While NeurIPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution a new model (e.g.. large language model), there should either be a way to access this model for reproducing the results a way to reproduce model (e.g.. with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may he that access to the model is limited in some way (e.g.. registered users), hut it should be possible for other rescarchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
We will provide full access to code upon the acceptance of the paper.
Guidelincs:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurTPS code and data submission guidelines httpee:///iis..cp/ub/lee for more details.
guides/Codee Sttleess/oooSSuuusiiiiioooooo
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
17

---

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines htttps://nipa.cc/ppun
Buidee//DodeSubmiissioooPPP
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: All the details necessary to reproduce our results provided in the Methodology (Section 3). Experimental Setup (Section 4) and Appendix A Additionally, our code will be made available upon acceptance.
Guidelines:
answer means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that necessary to appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical sigmificance of the experiments?
Answer: [No]
Justification: Error bars are not reported due to the expensive computational requirements to produce them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
96% CL, if the hypothesis of Normality errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates). error bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Answer: [Yes]
Justification: Information regrading the computing resources are provided in the Experimental Setup (Section 4.
18

---

Guidelines:
answer means the paper does not include experiments.
paper should indicate the type compute workers CPU or GPU, internal cluster. or cloud
provider, including relevant memory and storage.
The paper should provide the amount compute required for each the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't I make it into the paper).
9. Code of ethics
Question:
Does the research conducted in the paper conform, in every respect, with the NeurIPS Code Ethics http:p///eaarrse..ccc////l///aaa
Answer: [Yes]
Justification:
The work conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification:
The work this paper no societal impact.
Guidelines:
The answer means there is no societal impact of work performed.
the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address socictal impact.
Examples negative socictal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance). fairness considerations (e.g.. deploy- ment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of MI...
11. Safeguards
Does the paper describe safeguards that have been put in place responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Our work does not pose any high-level risks.
Guidelines:
The answer NA means that the paper poses such risks.
19

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards allow for controlled use of the model, for example by requiring that adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cite the corresponding papers of the models and datascts that we use in our experi- ments. In addition, work adhere their terms of use and licenses.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms of use the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out to the asset's ' creators.
13. New assets
Are introduced in the paper well documented and is the documentation provided alongside assets?
Question:
Answer: [NA]
Justification:
Our codes and assets will be released upon paper acceptance.
Guidelines:
The answer means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
The work in this paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that paper does not involve crowdsourcing nor research with human subjects.
Including this information in supplemental material is fine, but main contribution of the paper involves human subjects, then as much detail as possible should he included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
20

---

15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: This work does not involve crowdsourcing or research involving human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere to the NeurlPS Code of Ethics the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- ble), such as the institution conducting the review,
Declaration LLM usage
Does the paper describe the usage of LLMSs if it an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: I.LMs were not used in capacity that could be considered important, original, or as non-standard component of the work.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs any important, original, or non-standard components.
Please refer to our LLM policy httt:e///aarrppp.cc////ereeeeerr ccc.Confeeeeccee/2222555. MLMM) for what should or should not be described.
21

---

A Appendix
A.1 Experimental Setup
We provide detailed hyperparameters of experiments in Table
Table 6: Detailed hyperparameters for each training stage across different LLM backbones.
L.M Backhone
Jamm 3.2-18
Thama 3.2-38
Lamma 3.1-8B5
Stage-1
Stage-2
Stage33
Stage1
Stuge-2
Stage-3
Stage-1
Stage-2
Stage3
Tronable Parameters
Full Model
Full Model
LLM & Connector
Full Model
Full Model
LLM Connector
Full Model
Full Madel
LLM Connector
Banch Saze
512
512
312
512
256
512
256
255
Teut Max Length
1024
2048
2048
1024
2048
2003
1024
2048
2003
Epoctts
1
Leuning Rate
x 10
6 10
6x 10
1 xx300
8 10
5 10
x 30
1
x 10
A.2 Runtime Comparison Between Connectors
One caveat the ALIGN connector that includes an additional LM head layer, which slightly increases the total number of parameters. However, this addition has a negligible impact on runtime efficiency due to its simple structure. It only introduces few matrix multiplication operations (as shown in Equations I and instead of stacking many complex layers that require sequential processing, in deep fusion methods.
To empirically validate this claim, we benchmarked the runtime and memory usage of models equipped with different connector types (MLP, Align. Ovis, and Perceiver), following the same experimental setup as in Table As shown in Table [ the results demonstrate that although the ALIGN connector delivers notably superior performance (see Table 2). the variations in inference speed and GPU memory usage among the connectors remain minimal.
Table 7: Runtime and memory comparison between different connector designs. The results show that ALIGN introduces negligible computational overhead compared other connectors.
Model
Samples Avg Time (s)
Tokens/sec
GPU Memory (GB)
Llama-3.2-33 3.2-3B-MLP
2500
0.161
118.3
10.9
2500
0.140
135.1
10.9
1.mma-3.2-3B-Om
2500
0.155
122.5
10.8
2500
0.165
115.4
10.9
Overall, the empirical evidence confirms that the ALIGN connector achieves an effective balance between computational efficiency and performance. It introduces only negligible increase in runtime and memory usage while providing substantial gains in overall accuracy.
A.3 Pixel-Level Tasks Analysis
To rigorously evaluate the ability of rision-languaal models to integrate fine- aneegrraied visual and textual pixel- level cues, we test our model on the VCR benchmark (Zhang et al. 2024), which requires the model to recover partially occluded texts with pixel-level hints from the revealed parts of the text. This task challenges VLM's alignment of text and image in extreme situations. Current state-of-the-ar models like GPT-4V GPT-4V[OpenAA [penAA al. [2023,, Claude 3.5 Sonnet Anthroopic (2024). and 1lama-3.2 Dubey al. (2024) significantly underperform humans on hard VCR task due to their inaiility to process subtle pixel-level cues in occluded text regions. These models frequently discard critical visual tokens during image tokenization on semantic priors, overlooking the interplay between partial character strokes and contextual visual scenes. To evaluate performance VCR, we modify our Stage SFT datasct composition by replacing the exclusive use Downnsstmra with 5:1 blended ratio of DowDowsstream and VCR training data. This adjustment enables direct evaluation of our architecture ALIGN's ability to leverage pixel-level character cues.
From the experimental outcomes, it is evident that ALIGNVLM LM consistently outperforms the MLP Connector Model across both easy and hard settings of the pixel-level VCR task (see Figure S). with improvements ranging from 10.18% the hard setting 14.41% 41% the easy setting.
We provide case study VCR Figure 6 featuring four representative examples. In Figure 6a it is evident that the MLP connector model fails capture semantic consistency as effectively as ALIGNVLM. The phrase "The commune first census in written history in" (where the words in italics are generated by the model while the rest are in the image) is not as semantically coherent as the phrase generated by ALIGN "The commune first appears in written history in".
22

---

65.84
VCR EN Easy
51.43
48.07
VCR EN Hard
37.89
0
20
40
60
Exact Match (%)
(Ours)
Llama-3.2-3B-MLL
Figure 5: Comparison of LLlama-3.2-3b-Auu and 3.2----L.2 on the Easy and Hard VCR tasks.
Beyond the issue of semantic fluency, Figure we also observe that ALIGN successfully identifies the uncovered portion the letter "g" in "accounting" and uses as pixel-level hint infer the correct word. In contrast, the MLP model fails to effectively attend to this crucial detail.
Figures 6c and 6d show examples where ALIGNVLM fails on the VCR task. These carefully picked instances show that method mistakes names of landmarks with common words when the two are very similar. As in the examples, ALIGNVLM mistakes "Llanengan" for "Llanongan" and "Gorden" for "Garden". In both instances, the pairs differ by one character, indicating perhaps that ALIGNVLM tends align vision representations to more common tokens in the vocabulary. One approach that would potentially mitigate such errors would be to train ALIGNVLM with more rontextuallly-elovant data.
A.4
Case Studies
this section, we provide case studies the experiments Section 5.1 Specifically, we provide examples of our Saama-3.23BBAALIom and its counterpart model with alternative connectors Llama-3.2-3B-MM and
Llama-3322-3B-00 on three different datasets: KLC (Stanislawek et al. 2021 Doc (Mathew et al. 2021b and TextVQA (Singh et al. 20199. The examples are shown in Figure
and
23

---

colection
Geo United States cantes
encompasses Panitor, roshoton, Westem and
New Echota,
GT:
(mines situated
GT:
(Gorden County
Apal
The Ghana telophone numbering plan systom 22d
near Llanengan on) MLP: (mines situated near Llanengan on) / (mines situated near Llanongan on) X
Courty, Trarsyhaatia, Ramana a composed vllages, teat
Dupus
telephone Grann H regubited by National Communcatore
home to) (Gorden County home to)
MLP:
GT:
(appears writ- ten history in)
GT:
(the system used for assigning)
MLP:
(census written history in)
MLP:
(the system used
ALIGN
ALIGN
(Garden County home to) X
for accounting) (the system used assigning) /
ALIGN
(appears in writ- history in) /
ALIGN
(c) Negative Example
(d) Negative Example
(a) Positive Example
(b) Positive Example
Figure Case Study for Pixel-Level Tasks. We provide examples of our proposed ALIGN connector compared with the Multi-Lav Layer Perceptron (MLP) connector. The ALIGN connector tends to better map visual elements to common words. GT is the ground truth.
24

---

IS/INALL
ARDINGLY
Ardingly College Limited
REPORT
t UMBIG AUGUUS 2017
Therly Pigilaioon 10me1 Sag legerdiess 6077/eT end
Question: What is the value for the charity name?
Question: What is the value for the address postcode?
GT:
(Ardingly College Ltd.)
GT:
(SW2 2QP)
MLP:
(Ardington College Ltd.) x
MLP: (SW22 (PQ)
Ovis:
(Ardington College Ltd.) X
Ovis:
(SW2 20P) X
ALIGN:
(Andingly College Ltd.) /
ALIGN:
(SW2 2QP) /
(a) Positive Example
(b) Positive Example #2
Bishog's a Stortford Bapiist Church
ANNUAL REPORT
STATEMENTS
Question:
What is the value the charity name?
Question: What is the value for the post town address? (Bishop's Storfford) (Stortford) X
GT:
(Human Appeal) (Humanitarian Agenda)
MIP:
GT:
MLP:
Ovis:
(Human Appeal) /
Ovis: (Bishop's Stortford)
ALIGN:
(Human Rightsappeal) X
ALIGN: (Stortford) x
(c) Negative Example #1
(d) Negative Example #2
Figure 7: Case Study for Connector Comparison on the KLC dataset (Stanislawek et al., 2021].
We show four qualitative examples (including two correct and two incorrect examples) comparing Llama-3.2-2 3.2-3B-ALICC to the same architecture with different connectors, Llama-3.2-3B. and Llama-3.2-3B-Ovis "GT" denotes the ground truth.
25

---

5.15
Question:
What levels does the second ta- ble indicate?
Question: What does the afternoon
GT:
(hematocrit data
Mas-
session begin on June 29?
sachusetts)
GT:
(1:00)
MLP:
(SATISFACTORY) X
MLP:
(2:45) x
Ovis:
(Females) (hematocrit data
Ovis:
(3:30) X
ALIGN:
Mas-
ALIGN:
(1:00) /
sachusetts) /
(a) Positive Example #1
(b) Positive Example #2
Question:
What type of policy is de- scrihed in this document? (Policy on Document Con- trol)
GT:
Question: What was the dict fed to the #1 group?
MIP: (Policy on Document Con- trol) /
(basal diet)
MLP:
(basel diet) /
Ovis:
(General Provisions) X
Ovis:
(Whole blood) X
ALIGN:
(Document Control) X
ALIGN: (control diet) X
Negative Example #1
(d) Negative Example #2
Figure 8: Case Study for Connector Comparison on the DocVQA dataset [Mathew et al. 2021b
We show four qualitative examples (including two correct and two incorrect examples) comparing 1.ama-3.2-3B-AL.. to the same architecture with different connectors, 1aam--32-33--MB and Llama-3.2 1lama-3.2-3B--0 3.vi..-Ovs.. "GT" denotes the ground truth.
26

---

SISONN DY ATOMIC CLOCK
12:04
DATE
Question:
What greeting is written the letter?
Question:
What indoor temperature is
shown?
GT:
(good bye)
GT:
(68.4)
MLP:
(good) x
MLP:
(68 F) X
Ovis:
(good buy) X
Ovis:
(40.0) X
ALIGN:
(good bye) /
ALIGN:
(68.4) /
(a) Positive Example
(h) Positive Example #2
Proffesional Passionate Personal
Hadiah Plus
MORE REW ARDS WITH EVERY SWIPE
We're re the 408
LOYALTY
You can now coewert your Hadish Poims BBC Poiitts free flahts.
topnotch healch CIUE
dur to
Question:
What type of club is advertised? (health club)
Question:
What credit card is this? (hadiah plus)
GT:
GT:
MLP:
(topnote health club) X (health club) / (professional passionate per- sonal)
MLP:
(hadiah plus) (american big loyalty program)
Ovis:
Ovis:
ALIGN:
X
ALIGN:
(hadia plus) X
(c) Negative Example #1
(d) Negative Example
Figure 9: Case Study for Connector Comparison on the TextVQA dataset (Singh et al. 2019 show four qualitative examples (including two correct and two incorrect examples) comparing 3.amm--3.-3B-ALul the same architecture with different connectors, Llama-3.2-3 3.2-3BBMM- and "GT" denotes the ground truth.
27


Reviews:
[{"review_id": "vFITY6BwNF", "reviewer": "Reviewer_T3uT", "summary": "The paper introduces ALIGNVLM, a novel vision-language model architecture designed to improve multimodal document understanding. At the core of ALIGNVLM is a new connector module, ALIGN, which aligns visual features with a large language model (LLM) by mapping them to weighted combinations of pretrained LLM text embeddings. This approach constrains visual features to remain within the LLM’s latent space, addressing issues of out-of-distribution and noisy inputs common in previous shallow fusion methods like MLPs. Extensive experiments across multiple document understanding benchmarks (e.g., DocVQA, TableVQA, ChartQA) show that ALIGNVLM achieves state-of-the-art performance, outperforming both parameter-heavy deep fusion models and strong open-source instruction-tuned baselines, even when trained under the same data regime. The paper also presents robustness and generalization analyses, demonstrating the method’s effectiveness beyond document tasks and under noisy input conditions.", "questions": "1. How sensitive is ALIGN to the choice of the vision encoder? Could the authors provide ablations or analysis on whether stronger or weaker vision backbones (e.g., CLIP-ViT or smaller SigLIP variants) affect ALIGN’s performance or alignment robustness?\n\n2. Can the authors offer a more detailed analysis or visualization of how ALIGN distributes visual features over text token embeddings? For instance, are the top tokens consistently interpretable or class-specific? How do these change across domains?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strength**\n\n1. The proposed ALIGN connector innovatively maps visual features to a convex combination of LLM text embeddings, ensuring the features remain in-distribution with respect to the LLM. This idea is simple, elegant, and grounded in strong inductive bias, avoiding ad hoc projection layers like MLPs.\n\n2. ALIGNVLM consistently outperforms state-of-the-art shallow and deep fusion methods, including larger models like LLaMA 3.2-11B and DocOwl-8B, across a suite of diverse document understanding benchmarks (DocVQA, ChartQA, TableVQA, etc.).\n\n3. The paper includes carefully controlled ablations comparing ALIGN with MLP, Perceiver Resampler, and Ovis under the same training regimes.\n\n4. Although focused on document understanding, the method is tested on general V+L tasks (MMMU, POPE, GQA), showing ALIGN can generalize beyond the document domain.\n\n**Weakness**\n\n1. The paper uses SigLIP-400M, but does not analyze sensitivity to the vision encoder or explore whether ALIGN benefits from stronger backbones (e.g., CLIP ViT-G, SAM). This limits understanding of its scalability.\n\n2. While the paper mentions encoder-free VLMs (like Fuyu-8B, EVE), it does not empirically compare ALIGNVLM against them, nor analyze trade-offs in removing the vision encoder altogether. This limits its positioning within the current landscape of lightweight VLM architectures.\n\n3. The paper claims ALIGN is more efficient than deep fusion methods, but provides no concrete runtime, FLOPs, memory, or inference latency comparisons, which are critical in practical deployments.", "was_revised": true, "final_justification": "I will maintain my score since my concerns are addressed.", "rebuttal": "Thank you for your thoughtful comments and acknowledging that our Align idea is elegant and grounded in strong inductive bias. We address your comments below. \n\n\n# Concern 1: Experimenting with Other Vision Encoders\n\nIn the paper, we focused on SigLip-400m due to its superior performance over CLIP [1]. We agree that experimenting with various vision encoders can further demonstrate ALIGN's generalizability. To address this, we conducted additional experiments using SigLIP-400M and CLIP vision encoders under a low-data regime setup for computational efficiency. Specifically, we finetune on the LLava-Next dataaset [2] and follow their recommended training configurations for both pretraining (freezing llm and vision encoder, learning rate: 1e-3, batch size: 64, epochs: 1) and instruction tuning (LLM and connector learning rate: 1e-5, vision encoder: 2e-6, batch size: 8, epochs: 1).\n\nWe evaluated the models on both document understanding benchmarks (DocVQA, InfoVQA, ChartQA, TextVQA) and general vision-language tasks (MMMU-dev, MMVet, GQA). Our findings indicate that Align still produces strong and comparable performance despite using a weaker vision backbone. This proves that the Align connector generalizes well across different vision encoders. \n\n\n| Model                         | DocVQA | InfoVQA | ChartQA | TextVQA |\n|------------------------------|--------|---------|---------|----------|\n| LLama-3.2-3B–SigLIP-Align    | 71.43  | 30.50   | 69.72   | 65.63    |\n| LLama-3.2-3B–CLIP-Align      | 68.01  | 29.06   | 66.84   | 62.15    |\n\n\n| Model                         | MMMU  | MMVet | GQA   |\n|------------------------------|-------|-------|-------|\n| LLama-3.2-3B–SigLIP-Align    | 35.33 | 35.32 | 61.67 |\n| LLama-3.2-3B–CLIP-Align      | 33.67 | 33.99 | 59.95 |\n\n\n# Concern 2: Comparisons with Lightweight encoder-free VLMs (Fuyu-8B)\n\nWe evaluated Fuyu-8B on the same document, understanding benchmarks and observed significantly limited performance compared to AlignVLM as shown in the table below. Although encoder-free models simplify architecture, they typically lack performance due to not leveraging pretrained visual components [3]. We will include these results in Table 1 and revise the discussions in our revised manuscript.\n\n| Model       | DocVQA | InfoVQA | Deepform | KLC   | WTQ   | TabFact | ChartQA | TextVQA | TableVQA |\n|-------------|--------|---------|----------|-------|-------|---------|---------|----------|-----------|\n| Fuyu-8B     | 48.97  | 23.09   | 4.78     | 6.63  | 14.55 | 47.91   | 44.36   | 46.02    | 15.49     |\n| AlignVLM-8B | 79.63  | 44.53   | 63.49    | 35.25 | 38.59 | 78.51   | 71.88   | 57.38    | 60.10     |\n\n\n\n# Concern 3: Runtime and Memory Comparisons with Deep Fusion Methods \nWe compared Align against the deep fusion mechanism used in LLaMA-3.2 Vision [4]. We use the same backbone vision encoder and llm (SigLIP-400M, LLaMA-3.2-8B) to isolate the impact of the connector itself on inference runtime and memory usage. As shown in the table below, our Align connector reduces runtime and memory consumption, confirming our claim regarding its computational efficiency compared to deep fusion methods. We will include these details in the revised manuscript.\n\n\n\n| Connector           | # Params | Avg. Inference Time (s) | Tokens/s | GPU Memory (GB) |\n|---------------------|----------|--------------------------|----------|-----------------|\n| AlignVLM Connector  | 9B       | 34.12                    | 30.3     | 17.4            |\n| Llama 3.2 Connector | 10.3B    | 35.08                    | 29.4     | 20.0            |\n\n# Concern 4: Further Analysis of the Mapping of Visual Features over Embeddings. \n\nWe conducted a deeper analysis of the token distributions described in Section 5.3. We observed that Align does not directly map visual patches to individual semantic tokens in a one-to-one manner. Instead, it computes a dense convex combination of the LLM text embeddings for each patch as we illustrated in Section 5.3. In addition, we noticed that Align consistently assigns high probabilities to approximately 3.4K tokens from the entire vocabulary, while assigning negligible probabilities (below 1e-6) to the remaining tokens. When we reduced the dimensionality using PCA and plotted the embeddings in a 2D graph, we noticed that these 3.4K tokens densely and comprehensively span the latent space of all the LLM's text embeddings.\n\nTo further validate this observation, we conducted additional evaluation experiments using only these 3.4K high-probability embeddings in the Align connector, removing the remaining embeddings entirely during evaluation. The results presented in the table below demonstrate negligible performance differences compared to using the complete set of embeddings (128K). This finding confirms that Align effectively leverages these select embeddings to guide visual features into meaningful regions within the LLM's latent text space that the LLM can effectively interpret. It also shows that the Align connector can benefit from further pruning to improve its efficiency. \n\n| Model                         | DocVQA | InfoVQA | Deepform | KLC   | WTQ   | TabFact | ChartQA | TextVQA | TableVQA |\n|------------------------------|--------|---------|----------|-------|-------|---------|---------|----------|-----------|\n| AlignVLM-3B (3.4K tokens)    | 79.40  | 44.13   | 63.64    | 35.02 | 38.26 | 78.83   | 71.72   | 57.48    | 59.80     |\n| AlignVLM-3B (full)           | 79.60  | 44.53   | 63.49    | 35.25 | 38.59 | 78.51   | 71.88   | 57.38    | 60.10     |\n\nUnfortunately, we are not allowed to provide links to images in the rebuttal, but we will add these very interesting findings and figures in the revised manuscript. \n\nThank you for your thoughtful review. We hope our response has addressed your concerns and kindly ask you to **consider raising your score** to support the dissemination of our work within the ML community.\n\n\n[1] Sigmoid Loss for Language Image Pre-Training (arxiv)\n[2] Llava-NeXT: Improved reasoning, OCR, and world knowledge\n[3] PaliGemma: A versatile 3B VLM for transfer\n[4] The Llama 3 Herd of Models"}, {"review_id": "B1JQWwIi54", "reviewer": "Reviewer_fkBd", "summary": "The paper describes a new method to align visual and textual features in the context of multimodal vision-language models. The proposed method projects visual features as a weighted combination of textual embeddings of vocabulary tokens. This ensures that the projected visual features lie withing the convex hull of the textual embedding space. The proposed visual alignment is integrated into an state-of-the art Language Model and evaluated on Document Understanding tasks , showing improvement over other alignment methods, although the final performance is lower than current SoA.", "questions": "See above in strengths and weaknesses", "limitations": "Yes, limitations are addressed", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "STRENGTHS\n- The paper proposes a novel method for visual-text alignment, that tries to leverage prior semantics of the textual embedding space. Results show that this alignment can perform better than other existing alignment techniques.\n- The paper is clear and well written,\n\nWEAKNESSES\n- I have some concerns about the design of the alignment technique. One of the problems of the existing methods (as claimed in the introduction) is that current methods can produce out-distribution and noisy representations. It is supposed that the proposed method would alleviate this problem. However, the first step in the proposed alignment method is a linear layer that projects the visual features into the textual space without any prior. Thus, this initial point that is used to get the final representation could also be noisy and out-of-distribution. \n- One of the claims of the paper is that the representation obtained after the alignment can leverage semantic linguistic priors. However, the visualization of the token probability distribution in figure 3 and the predicted tokens in figure 4 raises several concerns about this claim. The dense probability distribution in figure 3 suggests that there is not a semantic differentiation according to the semantics of the visual token, but all textual tokens contribute similarly to the representation of visual tokens. In the same way, figure 4 seems to show that relevant textual tokens tend to concentrate on the most common vocabulary words, without any distinction according to the semantics of the visual token. \n- Experimental results show a lower performance of the proposed alignment method with respect to SoA multimodal models, that also integrate some kind of alignment. When compared with models using the same data regime, all models decrease their performance with respect to the baseline version, which raises concerns about the data used for training the model. In table 1 I miss the comparison of the proposed method with exactly the same version of the baseline Llama model, to see exactly the effect of the alignment method.", "was_revised": true, "final_justification": "The rebuttal have resolved some of my concerns, particularly those related to the design of the alignment method. I still have some concerns about the semantic interpretation of the aligned representation, but in light of the discussion with other reviewers, I agree with some of the positive comments made by other reviewers and I will raise my rating to borderline accept", "rebuttal": "Thank you for your thoughtful comments and acknowledging the novelty of our Align idea. We address your comments below. \n\n\n# Concern 1: Design of Alignment Method\n\nThe initial linear projection layer in Align is essential to match the dimensionality of the visual features with the LLM’s embedding space. This projection is effectively learned during training. The core innovation of the Align connector lies in the subsequent step, where visual features are mapped to a convex combination of the LLM’s text embeddings. This ensures that all visual inputs remain strictly within the convex hull of the LLM’s embedding latent space. This design naturally reduces the risk of noisy or out-of-distribution (OOD) representations. This is also supported by empirical results in Section 5.5; When Gaussian noise was added to the visual features, the Align connector exhibited only a 1.67% drop in performance, in contrast to a 25.54% drop observed with an MLP connector, highlighting the regularization benefits of our approach.\n\n# Concern 2: Align Connector Representation Learning\n\nWe conducted a deeper analysis of the token distributions described in Section 5.3. We observed that Align does not directly map visual patches to individual semantic tokens in a one-to-one manner. Instead, it computes a dense convex combination of the LLM text embeddings for each patch as we illustrated in Section 5.3. In addition, we noticed that Align consistently assigns high probabilities to approximately 3.4K tokens from the entire vocabulary, while assigning negligible probabilities (below 1e-6) to the remaining tokens. When we reduced the dimensionality using PCA and plotted the embeddings in a 2D graph, we noticed that these 3.4K tokens densely and comprehensively span the latent space of all the LLM's text embeddings.\n\nTo further validate this observation, we conducted additional evaluation experiments using only these 3.4K high-probability embeddings in the Align connector, removing the remaining embeddings entirely during evaluation. The results presented in the table below demonstrate negligible performance differences compared to using the complete set of embeddings (128K). This finding confirms that Align effectively leverages and combines these select embeddings to guide visual features into meaningful semantic regions within the LLM's latent text space that the LLM can effectively interpret. It also shows that the Align connector can benefit from further pruning to improve its efficiency. \n\n| Model                         | DocVQA | InfoVQA | Deepform | KLC   | WTQ   | TabFact | ChartQA | TextVQA | TableVQA |\n|------------------------------|--------|---------|----------|-------|-------|---------|---------|----------|-----------|\n| AlignVLM-3B (3.4K tokens)    | 79.40  | 44.13   | 63.64    | 35.02 | 38.26 | 78.83   | 71.72   | 57.48    | 59.80     |\n| AlignVLM-3B (full)           | 79.60  | 44.53   | 63.49    | 35.25 | 38.59 | 78.51   | 71.88   | 57.38    | 60.10     |\n\n\nUnfortunately, we are not allowed to provide links to images in the rebuttal, but we will add these very interesting findings/figures and revise our manuscript to reflect them. .  \n\n\n# Concern 3: Slightly Lower Performance Compared to SOTA Models like Qwen2.5-VL-Instruct\n\nDetailed datasets used by models like Qwen2-VL and InternVL remain undisclosed, making a completely unbiased comparison difficult. However, our deliberate selection of the BigDocs-7.5M and DocDownstream datasets was guided by their focus on document understanding which is the main focus and scope of our paper. \n\nFurthermore, open models (qwen2-vl, Intern-vl2) are anticipated to have been trained on significantly larger datasets (trillions of tokens and billions of images) and benefit from far more extensive GPU resources. We expect that this already places AlignVLM at a disadvantage, as larger-scale training typically leads to stronger generalization and performance improvements. Despite these constraints, AlignVLM remains competitive within the scope of multimodal document understanding. Our paper focuses explicitly on architectural improvements rather than extensive dataset curation.\n\nTo further justify our dataset choice and evaluate Align’s performance under low-data regime, we experimented with an alternative popular dataset, LLava-NeXT [1] which consist of 779K data points. We observed two critical findings: (1) Models trained on BigDocs-7.5M significantly outperform those trained on LLava-NeXT, and (2) Align significantly outperforms the MLP connector in a low-data regime, underscoring the strong inductive bias added through leveraging the LLM’s text embeddings for representing visual features. This makes the Align connector very valuable in resource-constrained environments such as Academic labs.\n\n\n\n| Model                           | DocVQA | InfoVQA | ChartQA | TextVQA | Average |\n|--------------------------------|--------|---------|---------|----------|---------|\n| LLama-3.2-3B-MLP (Llava Next)  | 42.11  | 19.93   | 48.44   | 51.97    | 40.61   |\n| LLama-3.2-3B-Align (Llava Next)| 71.43  | 30.50   | 69.72   | 65.63    | 59.32   |\n| LLama-3.2-3B-MLP (BigDocs)     | 71.46  | 37.56   | 66.48   | 53.56    | 57.26   |\n| LLama-3.2-3B-Align (BigDocs)   | 79.63  | 44.53   | 71.88   | 57.38    | 63.35   |\n\n\n\n\n\n# Concern 4: Performance of Comparable Llama Vision Models in Table 1\n\nOfficial Llama Vision models are limited to **two versions only**: Llama-3.2-90B (outside our scope) and Llama-3.2-11B which is reported in our manuscript (Table 1). Notably, the Llama-3.2-11B uses the exact LLM backbone (Llama-3.1-8B) as our AlignVLM-8B. The increased parameters from 8B to 11B in Llama-3.2-11B are primarily due to deep fusion connectors (additional cross-attention/feed-forward layers). The results in Table 1 shows that our AlignVLM-8B model achieves superior performance despite using a smaller connector, clearly demonstrating its efficiency and effectiveness compared to deep fusion methods.\n\n\nThank you for your thoughtful review. We hope our response has addressed your concerns and kindly ask you to **consider raising your score** to support the dissemination of our work within the ML community.\n\n[1] Llava-Next: Improved reasoning, OCR, and world knowledge"}, {"review_id": "t8a3QIHS9k", "reviewer": "Reviewer_XUHv", "summary": "In order to tackle with the problem of misalignment caused by connectors in VLMs, this paper proposes a new vision-text alignment technique, ALIGNVLM. It injects the embedding layer of LLM into the connector to generate a weighted average of LLM text embeddings in order to align vision features with LLM embedding space. The experiment results reveal a significant improvement on general document benchmarks brought by ALIGNVLM. Further analyses show that ALIGNVLM can also improve model's performance in general vision-language tasks including MMMU, SeedBench, MMVet, POPE, and GQA. What's more, ALIGNVLM also shows strong robustness to noise.", "questions": "1. Is the embedding layer in ALGNVLM freezed during training?\n2. The injection of embedding layer will bring in more parameters. Can you show the comparison of training costs between ALIGNVLM and existing connnectors?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths\n1. The method is simple without huge change to the framework of VLMs but effective, and it provides a new insight in designing connectors for VLMs by injecting LLM embedding layer into them.\n2. It can be applied to different domains including document comprehension and general vision-language tasks.\n3. It shows stronger robustness to noises compared to models with different connectors.\n\nweaknesses\n1. ALIGNVLM is only applied to LLama in the experiments. Maybe more experiments with other VLMs should be conducted for a more solid conclusion.", "was_revised": true, "final_justification": "My questions and concerns have been adequatly addressed by the author rebuttal and I will keep my positive rating.", "rebuttal": "Thank you for acknowledging the effectiveness of the align connector and its robustness to noise. We address your concerns and questions below. \n\n\n# Concern 1: Experimentation with other LLMs\nWe understand your interest in evaluating the Align Connector using other LLMs. However, due to computational constraints, we primarily experimented with the open-source SOTA Llama 3 models with different sizes (1B, 3B, and 8B parameters) to highlight Align’s scalability and consistent performance. Nonetheless, our Align module is LLM-agnostic and applicable in principle to any LLM. We have launched our training experiment with Qwen2.5-3B and will report the results in the camera-ready revised manuscript once it’s completed. \n\n# Question 1: Do we keep the embedding layers frozen during training? \nNo, we train all weights of the Align connector in all stages, as detailed in Table 5 of the appendix. In initial experiments, freezing the LM head and the LLM embedding matrix in the Align connector resulted in unstable training. This is primarily due to the distribution shift in the LM head layer's inputs (in the Align Connector), from the previously encoded textual outputs of the LLM to the current visual features from our vision encoder.\n\n# Question 2: Runtime comparison between Connectors \n\nWe acknowledge that the Align connector includes an additional LM head layer, which slightly increases the parameter count. However, this addition has a negligible impact on runtime efficiency due to its straightforward design, involving simple matrix multiplication operations (as shown in Equations 1 and 2 in the paper), rather than stacking multiple complex layers requiring sequential processing like the deep fusion methods.\n\nTo empirically support this, we benchmarked the runtime and memory usage of models using different connectors (MLP, Align, Ovis, and Perceiver), corresponding to the experiments reported in Table 2 of our paper. The results below show minimal differences in inference speed and GPU memory usage between the connectors, despite the Align connector's notably superior performance as demonstrated previously in Table 2.\n\n\n| Model             | Samples | Avg Time (s) | Tokens/sec | GPU Memory (GB) |\n|------------------|---------|--------------|------------|-----------------|\n| mlp_module        | 2500    | 0.161        | 118.3      | 10.9            |\n| perceiver_module  | 2500    | 0.140        | 135.1      | 10.9            |\n| ovis_module       | 2500    | 0.155        | 122.5      | 10.8            |\n| align_module      | 2500    | 0.165        | 115.4      | 10.9            |\n\n\nThank you for your thoughtful review. We hope our response has addressed your concerns."}, {"review_id": "kQdlhmSJwY", "reviewer": "Reviewer_tqc6", "summary": "This paper proposes ALIGNVLM, a novel vision-language model (VLM) framework that enhances cross-modal alignment between visual and textual representations, specifically targeting multimodal document understanding. The core contribution is the ALIGN connector, which avoids direct projection of visual features into the LLM’s embedding space. Instead, it represents each visual feature as a weighted average over the LLM's text embedding matrix, ensuring that visual inputs reside within the semantic space the LLM is pretrained to understand. The authors evaluate ALIGNVLM on a range of document-related tasks and compare it against both open-source and closed-source VLMs. Experiments demonstrate state-of-the-art performance, especially under limited training regimes, and provide in-depth analysis on robustness, alignment quality, and generalization to non-document V+L tasks.", "questions": "Low-resource regime analysis: The model is praised for efficiency, but no experiments are shown under low-resource settings (e.g., <1M pairs). Could the authors provide such analysis or comment on whether ALIGN still works with sparse data?\nClaims of \"parameter efficiency\" (vs. deep fusion) lack quantifiable evidence (training/inference speed, memory).\nVisual semantics leakage: Since ALIGN uses text embeddings to encode vision features, do you observe any semantic leakage (e.g., mapping to semantically plausible but incorrect tokens)?  Figure 4 suggests some alignment issues on white space – could these degrade QA performance?\nNo comparison with document-specialized models (e.g., LayoutLMv3, Pix2Struct). Is AlignVLM truly superior for structured content (tables/forms)?\nScalability to higher-resolution images: Your tiling approach supports up to 9 tiles.  How does the model scale (in performance and memory) when fed with dense layouts or longer document chains?", "limitations": "Yes. Authors partially discussed limitations (Sec 5.4) but should address:\nHow ALIGN would perform with LLMs that lack good vocabulary priors,\nPossible misalignment in highly visual tasks (e.g., fashion, scene graph QA),\nSocietal risks such as hallucination when visual context is ambiguous.\nSuggestion: Add an analysis/discussion section on transferability to less structured multimodal tasks and limitations of the convex-hull assumption in non-document domains.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nQuality – Strong empirical performance:\nExtensive experiments across 9 document understanding benchmarks and several general V+L tasks.\nALIGNVLM outperforms both strong shallow fusion (e.g., MLP, Perceiver Resampler) and deep fusion models (e.g., LLaMA-3.2-11B), validating the method's effectiveness.\nNovel alignment mechanism:Rather than learning new embedding spaces or large connector modules, ALIGN uses the LLM’s own vocabulary embeddings as a semantic anchor, preserving compatibility and reducing OOD risk.\nStrong to noise:The connector shows significantly lower degradation under Gaussian noise than MLPs (−1.67% vs. −25.54%).\nParameter-efficient design:Outperforms larger models (e.g., ALIGN-3B vs. DocOwl-8B), highlighting efficient alignment without scaling up LLM parameters.\nWell-organized and clearly written:Figures, tables, and appendix provide thorough support for claims; methodology is detailed and reproducible.\n\nWeakness:\nLimited theoretical insight:While the convex combination idea is intuitive, the paper lacks a formal theoretical justification or analysis of why this form of projection is optimal.\nData limitation disclosure needs clarification:Though the authors avoid using undisclosed instruction data, the actual differences in label coverage or image diversity between BigDocs vs. proprietary datasets are not deeply analyzed.\nLLM dependency:The method’s reliance on the LLM's embedding space assumes high-quality pretrained embeddings; it's unclear how ALIGN performs with smaller or less capable LLMs.", "was_revised": true, "final_justification": "All of my concerns have been addressed, I have adjusted my score to borderline accept.", "rebuttal": "Thank you for your constructive feedback and acknowledging the novelty of our Align connector and its robustness to noise . We address your comments below.\n\n# Concern 1: Low-resource regime training\nWe agree that experimenting with a low-data regime can further demonstrate ALIGN's efficiency. To address this, we conducted additional experiments using SigLIP-400M and Llama3.2-3B under a low-data regime by finetuning on the LLava-Next [1] dataset (779K data points). Specifically, we followed the LLava-Next recommended training configurations for both pretraining (freezing the llm and vision encoder, learning rate: 1e-3, batch size: 64, epochs: 1) and instruction tuning (LLM and connector learning rate: 1e-5, vision encoder: 2e-6, batch size: 8, epochs: 1).\n\nWe evaluated models on both document understanding benchmarks (DocVQA, InfoVQA, ChartQA, TextVQA) and general vision-language tasks (MMMU-dev, SeedBench, MMVet, POPE, GQA). Our findings indicate that the Align connector significantly outperforms other connectors under the low-data regime, especially on document understanding tasks. This makes the Align connector very valuable in resource-constrained environments such as Academic labs.\n\n\nDocument Understanding \n\n| Model                 | DocVQA | InfoVQA | ChartQA | TextVQA |\n|----------------------|--------|---------|---------|----------|\n| LLama-3.2-3B-MLP      | 42.11  | 19.93   | 48.44   | 51.97    |\n| LLama-3.2-3B-Perceiver| 32.18  | 18.10   | 40.00   | 44.31    |\n| LLama-3.2-3B-Ovis     | 41.88  | 19.64   | 47.44   | 51.76    |\n| LLama-3.2-3B-Align    | 71.43  | 30.50   | 69.72   | 65.63    |\n\nGeneral VisionTasks\n\n| Model                 | MMMU  | SeedBench | MMVet | POPE  | GQA   |\n|----------------------|-------|-----------|-------|-------|-------|\n| LLama-3.2-3B-MLP      | 33.33 | 58.54     | 31.14 | 87.35 | 57.62 |\n| LLama-3.2-3B-Perceiver| 35.22 | 63.70     | 26.19 | 84.92 | 55.86 |\n| LLama-3.2-3B-Ovis     | 32.22 | 58.22     | 29.58 | 87.10 | 57.64 |\n| LLama-3.2-3B-Align    | 35.33 | 63.27     | 35.32 | 88.85 | 61.67 |\n\n\n# Concern 2: Runtime and Memory Comparisons with Deep Fusion Methods\nWe compared Align against the deep fusion mechanism used in LLaMA-3.2 Vision [2]. We use the same backbone vision encoder and llm (SigLIP-400M, LLaMA-3.2-8B) to isolate the impact of the connector itself on inference runtime and memory usage. As shown in the table below, our Align connector reduces runtime and memory consumption, confirming our claim regarding its computational efficiency compared to deep fusion methods. We will include these details in the revised manuscript.\n\n\n\n| Connector           | # Params | Avg. Inference Time (s) | Tokens/s | GPU Memory (GB) |\n|---------------------|----------|--------------------------|----------|-----------------|\n| AlignVLM Connector  | 9B       | 34.12                    | 30.3     | 17.4            |\n| Llama 3.2 Connector | 10.3B    | 35.08                    | 29.4     | 20.0            |\n\n\n\n# Concern 3: Visual Semantic Leakage and Mappings\n\nTo further understand the mapping process for Align, we conducted a deeper analysis of the token distributions described in Section 5.3. We observed that ALIGN does not directly map visual patches to individual semantic tokens in a one-to-one manner. Instead, it computes a dense convex combination of the LLM text embeddings for each patch as we illustrated in Section 5.3. In addition, we noticed that Align  consistently assigns high probabilities to approximately 3.4K tokens from the entire vocabulary, while assigning negligible probabilities (below 1e-6) to the remaining tokens. When we reduced the dimensionality using PCA and plotted all the llm text embeddings in a 2D graph, we noticed that these 3.4K tokens densely and comprehensively span the latent space of all the LLM's text embeddings.\n\nTo further validate this observation, we conducted additional evaluation experiments using only these 3.4K high-probability embeddings in the Align connector, removing the remaining embeddings entirely during evaluation. The results presented in the table below demonstrate negligible performance differences compared to using the complete set of embeddings (128K). This finding confirms that Align effectively leverages and combines these select embeddings to guide visual features into meaningful regions within the LLM's latent text space that the LLM can effectively understand and interpret. It also shows that the Align connector can benefit from further pruning to improve its efficiency significantly. \n\n| Model                         | DocVQA | InfoVQA | Deepform | KLC   | WTQ   | TabFact | ChartQA | TextVQA | TableVQA |\n|------------------------------|--------|---------|----------|-------|-------|---------|---------|----------|-----------|\n| AlignVLM-3B (3.4K tokens)     | 79.40  | 44.13   | 63.64    | 35.02 | 38.26 | 78.83   | 71.72   | 57.48    | 59.80     |\n| AlignVLM-3B (full)            | 79.63  | 44.53   | 63.49    | 35.25 | 38.59 | 78.51   | 71.88   | 57.38    | 60.10     |\n\n\n\nUnfortunately, we are not allowed to provide links to images in the rebuttal, but we will add these very interesting findings and figures in the revised manuscript to clarify the inner workings of the Align connector. \n\n# Concern 4: Comparison with LayoutLMv3 and Pix2Struct. \n\nWe believe that LayoutLMv3 and Pix2Struct are not directly comparable to our models due to fundamental architectural differences. LayoutLMv3 is an encoder-only model (similar to BERT), while Pix2Struct follows an encoder-decoder design. In contrast, AlignVLM is built on a vision-language architecture with an LLM decoder.\n\nMoreover, both LayoutLMv3 and Pix2Struct are relatively small in scale (typically 100M–300M parameters), whereas our models range from 1B to 8B parameters, making direct comparisons less meaningful and potentially misleading.\n\nWe understand the importance of comparing AlignVLM with strong document-specific baselines. For this reason, Table 1 includes results for DocOwl-1.5, a recent and powerful document understanding model with a similar architecture and size. This offers a more appropriate and fair comparison. As shown in Table 1, AlignVLM consistently outperforms DocOwl1.5 across various benchmarks when instruction tuned on the same datasets, highlighting its effectiveness in the document domain.\n\n# Concern 5: Differences between BigDocs and Proprietary datasets\nDetailed datasets used by models like Qwen2-VL and InternVL remain undisclosed, making a completely unbiased comparison difficult. However, our deliberate selection of the BigDocs-7.5M and DocDownstream datasets was guided by their focus on document understanding which is the main focus and scope of our paper. \n\nFurthermore, open models (qwen2-vl, Intern-vl2) are anticipated to have been trained on significantly larger datasets (trillions of tokens and billions of images) and benefit from far more extensive GPU resources. We expect that this already places AlignVLM at a disadvantage, as larger-scale training typically leads to stronger generalization and performance improvements. Despite these constraints, AlignVLM remains competitive within the scope of multimodal document understanding. Our paper focuses explicitly on architectural improvements rather than extensive dataset curation.\n\n# Suggestion on Adding an analysis section on non-document tasks. \n\n\nOur paper already includes results on several widely used general vision-language benchmarks (MMMU, MMVet, POPE, GQA, and SeedBench) as presented in Section 5.4, demonstrating the generalizability of the Align connector beyond document understanding.\n\nWe also agree with the reviewer that evaluating Align on additional diverse domains, such as fashion, scene understanding, or scene graph generation, would further strengthen the paper’s contribution. We appreciate this suggestion and will incorporate such analyses in the revised manuscript.\n\nThank you for your thoughtful review. We hope our response has addressed your concerns and kindly ask you to **consider raising your score** to support the dissemination of our work within the ML community.\n\n[1] Llava-Next: Improved reasoning, OCR, and world knowledge\n[2] The Llama 3 Herd of Models"}, {"review_id": "QcC9osfJ4x", "reviewer": "Reviewer_ZYJ3", "summary": "The paper introduces ALIGNVLM, a novel vision-language model architecture designed to improve the alignment between visual features and the latent space of Large Language Models (LLMs), particularly for document understanding. Instead of using traditional projection layers like MLPs, ALIGNVLM's connector maps visual features to a probability distribution over the LLM's entire vocabulary. It then generates visual embeddings as a weighted average of the LLM's existing text token embeddings. This approach constrains the visual inputs to lie within the convex hull of the text embedding space, which the authors argue prevents out-of-distribution features and enhances robustness. Through extensive experiments, the paper demonstrates that this method achieves state-of-the-art performance on various document-related benchmarks and shows superior resilience to noisy inputs compared to previous methods.", "questions": "Please, see weakness part.", "limitations": "The paper doesn't provide its societal impact. For the limitation, the paper only depicts that ALIGNVLM may be biased toward commonly used vocabs, in Appendix A.3.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "### Strength\n- The paper's most compelling and well-executed contribution is the noise robustness experiment (Section 5.5). By demonstrating a significantly smaller performance drop under noisy conditions compared to a standard MLP, the authors provide strong, quantifiable evidence for their core hypothesis. This finding clearly shows that constraining visual features to the LLM's latent space acts as an effective regularizer, which is a valuable and tangible result.\n\n\n### Weakness\n- Weak Justification for Scope and Generality\n    - The paper's framing suffers from a lack of clear justification. While it predominantly focuses on document understanding, it fails to articulate a compelling reason why the ALIGN mechanism is uniquely suited for this domain. The argument that it should excel on documents because their visual content maps well to text tokens is implied but never explicitly made or proven.\n    Furthermore, the paper's attempt to demonstrate broader applicability falls short. While results on general vision-language benchmarks are presented (Section 5.4), the comparison is limited to a basic MLP connector. This is an insufficient baseline. To make a convincing case for general-purpose utility, the ALIGN module needed to be benchmarked against more sophisticated and widely adopted general connectors, such as the OVIS, Perceiver Resampler or Q-Former. As it stands, the paper neither provides a strong rationale for its primary focus on documents nor does it rigorously validate its effectiveness as a general-purpose solution.\n- Lack of Rigorous and Fair Experimental Comparison\n    - The most significant weakness of this paper is its failure to properly situate ALIGNVLM within the competitive landscape of VLM connectors. The study's claims of superiority are unsubstantiated because it avoids the very comparisons needed to prove its merit. Specifically, two critical comparisons are missing:\n        - **Omission of Key Architectural Baselines**: The paper neglects to perform a head-to-head architectural comparison against several state-of-the-art connectors, such as HoneyBee [A] or, most critically, the H-Reader module [B]. This is a fundamental flaw. To prove its value, a new architecture must be benchmarked directly against its most relevant competitors under identical training conditions.\n        - **Invalid Justification via Unfair Benchmarking**: The paper attempts to sidestep a direct comparison with the H-Reader module by instead comparing its overall model performance to that of mplugDocOwl-1.5. This comparison is fundamentally unsound. There is a significant disparity in the training data used for ALIGNVLM versus what is typically used for SOTA models like mplugDocOwl-1.5, yet the paper makes no attempt to account for this crucial confounding variable. Therefore, claiming superiority based on this benchmark is misleading, as the performance difference cannot be confidently attributed to the ALIGNVLM architecture over the profound influence of the training data.\n- Unconvincing Analysis\n    - The qualitative analysis intended to offer insight into the model's inner workings is shallow and unconvincing. Figure 4, for instance, does not provide clear evidence that the model effectively distinguishes between text and background, nor does it show that text features are mapped to semantically meaningful embeddings. This superficial analysis fails to build a strong, evidence-backed narrative for how and why its method truly works, leaving its internal mechanisms poorly understood.\n\n[A] Cha et al., Honeybee: Locality-enhanced Projector for Multimodal LLM, CVPR 2024 \n\n[B] Hu et al., mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding, arXiv 2024", "was_revised": true, "final_justification": "The authors' comprehensive rebuttal, especially the new experiments against strong baselines, has addressed all my major criticisms. While I find the framing on the document domain slightly narrow given the method's general potential, the empirical results are now solid and convincing. The paper has been significantly improved, and I've raised my score.", "rebuttal": "Thank you for your constructive feedback and acknowledging the robustness to noise of our Align connector. Below we address your concerns. \n\n\n# Concern 1: Scope and Additional Results on General Vision-Language Tasks \n\n\nDocument understanding is challenging due to its inherent multimodal nature (integrating complex visual and textual information) and has numerous critical real-world industry applications. Thus, it serves as an ideal domain for evaluating our proposed AlignVLM approach. Our Align Connector maps visual features into convex combinations of the LLM’s text embeddings which are regions of the latent space that the LLM is inherently trained to interpret as text. As shown in Tables 2 and 3, the Align connector yields substantially larger improvements over existing connectors (e.g., MLP) on document benchmarks (Table 2) compared to general vision-language tasks (Table 3), supporting its design motivation.\n\n\nWe agree with the reviewer that testing additional baselines on general vision-language tasks would further validate the generalizability of our method. To that end, we conducted ablation experiments using SigLIP-400M and Llama-3.2-3B with different connector designs, all trained on the LLaVA-Next dataset [1]. These models were trained under identical configurations: for pretraining, we used a learning rate of 1e-3, batch size of 64, and 1 epoch; for instruction tuning, we set the LLM and connector learning rate to 1e-5, vision encoder learning rate to 2e-6, batch size of 8, and 1 epoch. We evaluated the resulting models on standard general vision-language benchmarks including MMMU-dev, SeedBench, MMVet, POPE, and GQA. The results, summarized in the table below, demonstrate that the ALIGN connector outperforms all other connector variants on these general tasks on average, proving its effectiveness beyond the document domain.\n\n| Model                        | MMMU  | SeedBench | MMVet | POPE  | GQA   | Avg. |\n|-----------------------------|-------|-----------|-------|-------|-------|----------|\n| LLama-3.2-3B-MLP            | 33.33 | 58.54     | 31.14 | 87.35 | 57.62 | 53.59 |\n| LLama-3.2-3B-Perceiver      | 35.22 | 63.70     | 26.19 | 84.92 | 55.86 | 53.17 |\n| LLama-3.2-3B-Ovis           | 32.22 | 58.22     | 29.58 | 87.10 | 57.64 | 52.95\n| LLama-3.2-3B-HReducer       | 32.78 | 59.50     | 32.78 | 85.60 | 39.10 | 49.95 |\n| LLama-3.2-3B-HoneyBee (C-Abs)| 35.11 | 58.90     | 36.19 | 87.93 | 55.88 | 54.80 |\n| LLama-3.2-3B-Align          | 35.33 | 63.27     | 35.32 | 88.85 | 61.67 | 56.86 |\n\n\n# Concern 2: Comparison with HReducer and HoneyBee\n\nWe would like to clarify that all models in the lower half of Table 1 (including mPLUG-DocOwl) were instruction-tuned on the same datasets (BigDocs-7.5M and DocDownstream). Table 1 is intended to provide a comparison between AlignVLM and state-of-the-art models at the full model level, under a consistent instruction-tuning regime.\n\nWe also agree with the reviewer that a connector-level comparison with HReducer from mPLUG-DocOwl and HoneyBee (C-Abs) is necessary for a fair and direct architectural evaluation against our proposed connector (Align), similar to our comparisons with MLP, Perceiver Resampler, and Ovis in Table 2.\n\n\nTo ensure a fair comparison under a compute-efficient yet controlled setup, we reused the same experimental configuration described in Concern 1 using SigLIP-400M and Llama-3.2-3B trained on the LLaVA-Next dataset with identical pretraining and instruction tuning settings. We evaluated all connectors, including HReducer and HoneyBee (C-Abs), on both document understanding benchmarks (DocVQA, InfoVQA, ChartQA, TextVQA) and general vision-language tasks (MMMU-dev, SeedBench, MMVet, POPE, GQA). As shown in the tables above (general vision tasks) and below (documents tasks), the Align connector consistently outperforms these strong baselines, highlighting its robustness across domains. Notably, HoneyBee is the second-best connector, likely due to its ability to preserve local visual context. We attribute ALIGN's superior performance to its unique design, which projects visual features into the convex hull of the LLM's text embeddings (a region in the latent space that the LLM is inherently pretrained to understand)\n\n| Model                         | DocVQA | InfoVQA | ChartQA | TextVQA | \n|------------------------------|--------|---------|---------|----------|\n| LLama-3.2-3B-MLP             | 42.11  | 19.93   | 48.44   | 51.97    |\n| LLama-3.2-3B-Perceiver       | 32.18  | 18.10   | 40.00   | 44.31    |\n| LLama-3.2-3B-Ovis            | 41.88  | 19.64   | 47.44   | 51.76    |\n| LLama-3.2-3B-HReducer        | 31.77  | 17.07   | 42.12   | 35.44    |\n| LLama-3.2-3B-HoneyBee (C-Abs)| 50.99  | 15.57   | 50.72   | 55.91    |\n| LLama-3.2-3B-Align           | 71.43  | 30.50   | 69.72   | 65.63    |\n\n\n# Concern 3: Additional Analysis of the Align Connector Internal Mechanism\n\nWe conducted a deeper analysis of the token distributions described in Section 5.3. We observed that Align does not directly map visual patches to individual semantic tokens in a one-to-one manner. Instead, it computes a dense convex combination of the LLM text embeddings for each patch as we illustrated in Section 5.3. In addition, we noticed that Align consistently assigns high probabilities to approximately 3.4K tokens from the entire vocabulary, while assigning negligible probabilities (below 1e-6) to the remaining tokens. When we reduced the dimensionality using PCA and plotted the llm’s text embeddings in a 2D graph, we noticed that these 3.4K tokens densely and comprehensively span the latent space of all the LLM's text embeddings.\n\nTo further validate this observation, we conducted additional evaluation experiments using only these 3.4K high-probability embeddings in the Align connector, removing the remaining embeddings entirely during evaluation. The results presented in the table below demonstrate negligible performance differences compared to using the complete set of embeddings (128K). This finding confirms that ALIGN effectively leverages these select embeddings to guide visual features into meaningful regions within the LLM's latent text space that the LLM can effectively understand. It also shows that the Align connector can benefit from further pruning to improve its efficiency significantly.\n\n\n| Model                         | DocVQA | InfoVQA | Deepform | KLC   | WTQ   | TabFact | ChartQA | TextVQA | TableVQA |\n|------------------------------|--------|---------|----------|-------|-------|---------|---------|----------|-----------|\n| AlignVLM-3B (3.4K tokens)     | 79.40  | 44.13   | 63.64    | 35.02 | 38.26 | 78.83   | 71.72   | 57.48    | 59.80     |\n| AlignVLM-3B (full)            | 79.63  | 44.53   | 63.49    | 35.25 | 38.59 | 78.51   | 71.88   | 57.38    | 60.10     |\n\nUnfortunately, we are not allowed to provide links to images in the rebuttal, but we will add these very interesting findings and figures in the revised manuscript to clarify the inner workings of the Align connector. \n\nThank you for your thoughtful review. We hope our response has addressed your concerns and kindly ask you to **consider raising your score** to support the dissemination of our work within the ML community.\n\n[1] Llava-NeXT: Improved reasoning, OCR, and world knowledge"}]

Author rebuttal:

Meta-review: The paper introduces a connector that maps vision features to convex combinations of LLM token embeddings. The goal is to keep visual inputs in-distribution within the LLM’s text latent space. Experiments on document understanding (and some general tasks) show improved accuracy and robustness in comparison to standard connectors. 

Reviewers highlight strong experimental gains using a simple and principled design, noise robustness  and low-data wins. The rebuttal adds controlled baselines and efficiency comparisons. 

Initial concerns included baseline omissions and fairness questions. Reviewers saw interpretability figures as unconvincing but the authors promise to revise these. Some mentions of the theory being light and cross-LLM validation being limited. 

During the discussion some reviewers raised their scores after new experiments and added low-data/efficiency analyses; one reviewer maintained caveats on analysis clarity. 

Overall, I recommend to **accept** this paper to NeurIPS. Its clear idea with solid experimental evidence make it very suitable for presentation in my opinion.
Decision: Accept (poster)