Title: AlignVLM: Bridging Vision and Language Latent Spaces for Multimodal Document Understanding
Venue: NeurIPS 2025
Accepted: yes
Keywords: vision language models

multimodal

large language models
Abstract: Aligning visual features with language embeddings is a key challenge in vision-language models (VLMs). The performance of such models hinges on having a good connector that maps visual features generated by a vision encoder to a shared embedding space with the LLM while preserving semantic similarity. Existing connectors, such as multilayer perceptrons (MLPs), lack inductive bias to constrain visual features within the linguistic structure of the LLM’s embedding space, making them data-hungry and prone to cross-modal misalignment. In this work, we propose a novel vision-text alignment method, AlignVLM, that maps visual features to a weighted average of LLM text embeddings. Our approach leverages the linguistic priors encoded by the LLM to ensure that visual features are mapped to regions of the space that the LLM can effectively interpret. AlignVLM is particularly effective for document understanding tasks, where visual and textual modalities are highly correlated. Our extensive experiments show that AlignVLM achieves state-of-the-art performance compared to prior alignment methods, with larger gains on document understanding and under low-resource setups. We provide further analysis demonstrating its efficiency and robustness to noise.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ALIGNVLM, a novel vision-language model architecture designed to improve the alignment between visual features and the latent space of Large Language Models (LLMs), particularly for document understanding. Instead of using traditional projection layers like MLPs, ALIGNVLM's connector maps visual features to a probability distribution over the LLM's entire vocabulary. It then generates visual embeddings as a weighted average of the LLM's existing text token embeddings. This approach constrains the visual inputs to lie within the convex hull of the text embedding space, which the authors argue prevents out-of-distribution features and enhances robustness. Through extensive experiments, the paper demonstrates that this method achieves state-of-the-art performance on various document-related benchmarks and shows superior resilience to noisy inputs compared to previous methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n- The paper's most compelling and well-executed contribution is the noise robustness experiment (Section 5.5). By demonstrating a significantly smaller performance drop under noisy conditions compared to a standard MLP, the authors provide strong, quantifiable evidence for their core hypothesis. This finding clearly shows that constraining visual features to the LLM's latent space acts as an effective regularizer, which is a valuable and tangible result.\n\n\n### Weakness\n- Weak Justification for Scope and Generality\n    - The paper's framing suffers from a lack of clear justification. While it predominantly focuses on document understanding, it fails to articulate a compelling reason why the ALIGN mechanism is uniquely suited for this domain. The argument that it should excel on documents because their visual content maps well to text tokens is implied but never explicitly made or proven.\n    Furthermore, the paper's attempt to demonstrate broader applicability falls short. While results on general vision-language benchmarks are presented (Section 5.4), the comparison is limited to a basic MLP connector. This is an insufficient baseline. To make a convincing case for general-purpose utility, the ALIGN module needed to be benchmarked against more sophisticated and widely adopted general connectors, such as the OVIS, Perceiver Resampler or Q-Former. As it stands, the paper neither provides a strong rationale for its primary focus on documents nor does it rigorously validate its effectiveness as a general-purpose solution.\n- Lack of Rigorous and Fair Experimental Comparison\n    - The most significant weakness of this paper is its failure to properly situate ALIGNVLM within the competitive landscape of VLM connectors. The study's claims of superiority are unsubstantiated because it avoids the very comparisons needed to prove its merit. Specifically, two critical comparisons are missing:\n        - **Omission of Key Architectural Baselines**: The paper neglects to perform a head-to-head architectural comparison against several state-of-the-art connectors, such as HoneyBee [A] or, most critically, the H-Reader module [B]. This is a fundamental flaw. To prove its value, a new architecture must be benchmarked directly against its most relevant competitors under identical training conditions.\n        - **Invalid Justification via Unfair Benchmarking**: The paper attempts to sidestep a direct comparison with the H-Reader module by instead comparing its overall model performance to that of mplugDocOwl-1.5. This comparison is fundamentally unsound. There is a significant disparity in the training data used for ALIGNVLM versus what is typically used for SOTA models like mplugDocOwl-1.5, yet the paper makes no attempt to account for this crucial confounding variable. Therefore, claiming superiority based on this benchmark is misleading, as the performance difference cannot be confidently attributed to the ALIGNVLM architecture over the profound influence of the training data.\n- Unconvincing Analysis\n    - The qualitative analysis intended to offer insight into the model's inner workings is shallow and unconvincing. Figure 4, for instance, does not provide clear evidence that the model effectively distinguishes between text and background, nor does it show that text features are mapped to semantically meaningful embeddings. This superficial analysis fails to build a strong, evidence-backed narrative for how and why its method truly works, leaving its internal mechanisms poorly understood.\n\n[A] Cha et al., Honeybee: Locality-enhanced Projector for Multimodal LLM, CVPR 2024 \n\n[B] Hu et al., mPLUG-DocOwl 1.5: Unified Structure Learning for OCR-free Document Understanding, arXiv 2024\n\nQUESTIONS:\nPlease, see weakness part.\n\nLIMITATIONS:\nThe paper doesn't provide its societal impact. For the limitation, the paper only depicts that ALIGNVLM may be biased toward commonly used vocabs, in Appendix A.3.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes ALIGNVLM, a novel vision-language model (VLM) framework that enhances cross-modal alignment between visual and textual representations, specifically targeting multimodal document understanding. The core contribution is the ALIGN connector, which avoids direct projection of visual features into the LLM’s embedding space. Instead, it represents each visual feature as a weighted average over the LLM's text embedding matrix, ensuring that visual inputs reside within the semantic space the LLM is pretrained to understand. The authors evaluate ALIGNVLM on a range of document-related tasks and compare it against both open-source and closed-source VLMs. Experiments demonstrate state-of-the-art performance, especially under limited training regimes, and provide in-depth analysis on robustness, alignment quality, and generalization to non-document V+L tasks.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nQuality – Strong empirical performance:\nExtensive experiments across 9 document understanding benchmarks and several general V+L tasks.\nALIGNVLM outperforms both strong shallow fusion (e.g., MLP, Perceiver Resampler) and deep fusion models (e.g., LLaMA-3.2-11B), validating the method's effectiveness.\nNovel alignment mechanism:Rather than learning new embedding spaces or large connector modules, ALIGN uses the LLM’s own vocabulary embeddings as a semantic anchor, preserving compatibility and reducing OOD risk.\nStrong to noise:The connector shows significantly lower degradation under Gaussian noise than MLPs (−1.67% vs. −25.54%).\nParameter-efficient design:Outperforms larger models (e.g., ALIGN-3B vs. DocOwl-8B), highlighting efficient alignment without scaling up LLM parameters.\nWell-organized and clearly written:Figures, tables, and appendix provide thorough support for claims; methodology is detailed and reproducible.\n\nWeakness:\nLimited theoretical insight:While the convex combination idea is intuitive, the paper lacks a formal theoretical justification or analysis of why this form of projection is optimal.\nData limitation disclosure needs clarification:Though the authors avoid using undisclosed instruction data, the actual differences in label coverage or image diversity between BigDocs vs. proprietary datasets are not deeply analyzed.\nLLM dependency:The method’s reliance on the LLM's embedding space assumes high-quality pretrained embeddings; it's unclear how ALIGN performs with smaller or less capable LLMs.\n\nQUESTIONS:\nLow-resource regime analysis: The model is praised for efficiency, but no experiments are shown under low-resource settings (e.g., <1M pairs). Could the authors provide such analysis or comment on whether ALIGN still works with sparse data?\nClaims of \"parameter efficiency\" (vs. deep fusion) lack quantifiable evidence (training/inference speed, memory).\nVisual semantics leakage: Since ALIGN uses text embeddings to encode vision features, do you observe any semantic leakage (e.g., mapping to semantically plausible but incorrect tokens)?  Figure 4 suggests some alignment issues on white space – could these degrade QA performance?\nNo comparison with document-specialized models (e.g., LayoutLMv3, Pix2Struct). Is AlignVLM truly superior for structured content (tables/forms)?\nScalability to higher-resolution images: Your tiling approach supports up to 9 tiles.  How does the model scale (in performance and memory) when fed with dense layouts or longer document chains?\n\nLIMITATIONS:\nYes. Authors partially discussed limitations (Sec 5.4) but should address:\nHow ALIGN would perform with LLMs that lack good vocabulary priors,\nPossible misalignment in highly visual tasks (e.g., fashion, scene graph QA),\nSocietal risks such as hallucination when visual context is ambiguous.\nSuggestion: Add an analysis/discussion section on transferability to less structured multimodal tasks and limitations of the convex-hull assumption in non-document domains.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-01"}

comments: SUMMARY:
In order to tackle with the problem of misalignment caused by connectors in VLMs, this paper proposes a new vision-text alignment technique, ALIGNVLM. It injects the embedding layer of LLM into the connector to generate a weighted average of LLM text embeddings in order to align vision features with LLM embedding space. The experiment results reveal a significant improvement on general document benchmarks brought by ALIGNVLM. Further analyses show that ALIGNVLM can also improve model's performance in general vision-language tasks including MMMU, SeedBench, MMVet, POPE, and GQA. What's more, ALIGNVLM also shows strong robustness to noise.

STRENGTHS AND WEAKNESSES:
strengths
1. The method is simple without huge change to the framework of VLMs but effective, and it provides a new insight in designing connectors for VLMs by injecting LLM embedding layer into them.
2. It can be applied to different domains including document comprehension and general vision-language tasks.
3. It shows stronger robustness to noises compared to models with different connectors.

weaknesses
1. ALIGNVLM is only applied to LLama in the experiments. Maybe more experiments with other VLMs should be conducted for a more solid conclusion.

QUESTIONS:
1. Is the embedding layer in ALGNVLM freezed during training?
2. The injection of embedding layer will bring in more parameters. Can you show the comparison of training costs between ALIGNVLM and existing connnectors?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper describes a new method to align visual and textual features in the context of multimodal vision-language models. The proposed method projects visual features as a weighted combination of textual embeddings of vocabulary tokens. This ensures that the projected visual features lie withing the convex hull of the textual embedding space. The proposed visual alignment is integrated into an state-of-the art Language Model and evaluated on Document Understanding tasks , showing improvement over other alignment methods, although the final performance is lower than current SoA.\n\nSTRENGTHS AND WEAKNESSES:\nSTRENGTHS\n- The paper proposes a novel method for visual-text alignment, that tries to leverage prior semantics of the textual embedding space. Results show that this alignment can perform better than other existing alignment techniques.\n- The paper is clear and well written,\n\nWEAKNESSES\n- I have some concerns about the design of the alignment technique. One of the problems of the existing methods (as claimed in the introduction) is that current methods can produce out-distribution and noisy representations. It is supposed that the proposed method would alleviate this problem. However, the first step in the proposed alignment method is a linear layer that projects the visual features into the textual space without any prior. Thus, this initial point that is used to get the final representation could also be noisy and out-of-distribution. \n- One of the claims of the paper is that the representation obtained after the alignment can leverage semantic linguistic priors. However, the visualization of the token probability distribution in figure 3 and the predicted tokens in figure 4 raises several concerns about this claim. The dense probability distribution in figure 3 suggests that there is not a semantic differentiation according to the semantics of the visual token, but all textual tokens contribute similarly to the representation of visual tokens. In the same way, figure 4 seems to show that relevant textual tokens tend to concentrate on the most common vocabulary words, without any distinction according to the semantics of the visual token. \n- Experimental results show a lower performance of the proposed alignment method with respect to SoA multimodal models, that also integrate some kind of alignment. When compared with models using the same data regime, all models decrease their performance with respect to the baseline version, which raises concerns about the data used for training the model. In table 1 I miss the comparison of the proposed method with exactly the same version of the baseline Llama model, to see exactly the effect of the alignment method.\n\nQUESTIONS:\nSee above in strengths and weaknesses\n\nLIMITATIONS:\nYes, limitations are addressed", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces ALIGNVLM, a novel vision-language model architecture designed to improve multimodal document understanding. At the core of ALIGNVLM is a new connector module, ALIGN, which aligns visual features with a large language model (LLM) by mapping them to weighted combinations of pretrained LLM text embeddings. This approach constrains visual features to remain within the LLM’s latent space, addressing issues of out-of-distribution and noisy inputs common in previous shallow fusion methods like MLPs. Extensive experiments across multiple document understanding benchmarks (e.g., DocVQA, TableVQA, ChartQA) show that ALIGNVLM achieves state-of-the-art performance, outperforming both parameter-heavy deep fusion models and strong open-source instruction-tuned baselines, even when trained under the same data regime. The paper also presents robustness and generalization analyses, demonstrating the method’s effectiveness beyond document tasks and under noisy input conditions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength**\n\n1. The proposed ALIGN connector innovatively maps visual features to a convex combination of LLM text embeddings, ensuring the features remain in-distribution with respect to the LLM. This idea is simple, elegant, and grounded in strong inductive bias, avoiding ad hoc projection layers like MLPs.\n\n2. ALIGNVLM consistently outperforms state-of-the-art shallow and deep fusion methods, including larger models like LLaMA 3.2-11B and DocOwl-8B, across a suite of diverse document understanding benchmarks (DocVQA, ChartQA, TableVQA, etc.).\n\n3. The paper includes carefully controlled ablations comparing ALIGN with MLP, Perceiver Resampler, and Ovis under the same training regimes.\n\n4. Although focused on document understanding, the method is tested on general V+L tasks (MMMU, POPE, GQA), showing ALIGN can generalize beyond the document domain.\n\n**Weakness**\n\n1. The paper uses SigLIP-400M, but does not analyze sensitivity to the vision encoder or explore whether ALIGN benefits from stronger backbones (e.g., CLIP ViT-G, SAM). This limits understanding of its scalability.\n\n2. While the paper mentions encoder-free VLMs (like Fuyu-8B, EVE), it does not empirically compare ALIGNVLM against them, nor analyze trade-offs in removing the vision encoder altogether. This limits its positioning within the current landscape of lightweight VLM architectures.\n\n3. The paper claims ALIGN is more efficient than deep fusion methods, but provides no concrete runtime, FLOPs, memory, or inference latency comparisons, which are critical in practical deployments.\n\nQUESTIONS:\n1. How sensitive is ALIGN to the choice of the vision encoder? Could the authors provide ablations or analysis on whether stronger or weaker vision backbones (e.g., CLIP-ViT or smaller SigLIP variants) affect ALIGN’s performance or alignment robustness?\n\n2. Can the authors offer a more detailed analysis or visualization of how ALIGN distributes visual features over text token embeddings? For instance, are the top tokens consistently interpretable or class-specific? How do these change across domains?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}