DAVE: A VLM Vision Encoder for Document Understanding and Web Agents
Authors: Brandon Huang
Hang Hua
Zhuoran Yu
Trevor Darrell
Rogerio Feris
Roei Herzig
Venue: ICLR 2026 Poster (2026)
Keywords: Visual Representation Learning
Vision Language Models
Document Understanding
Web Agents
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: Vision Encoder for Document and Web Understanding

Abstract: While Vision–language models (VLMs) have demonstrated remarkable performance across multi-modal tasks, their choice of vision encoders presents a fundamental weakness: their low-level features lack the robust structural and spatial
information essential for document understanding and web agents. To bridge this
gap, we introduce DAVE, a vision encoder purpose-built for VLMs and tailored
for these tasks. Our training pipeline is designed to leverage abundant unlabeled
data to bypass the need for costly large-scale annotations for document and web
images. We begin with a self-supervised pretraining stage on unlabeled images,
followed by a supervised autoregressive pretraining stage, where the model learns
tasks like parsing and localization from limited, high-quality data. Within the supervised stage, we adopt two strategies to improve our encoder’s alignment with
both general visual knowledge and diverse document and web agentic tasks: (i)
We introduce a novel model-merging scheme, combining encoders trained with
different text decoders to ensure broad compatibility with different web agentic
architectures. (ii) We use ensemble training to fuse features from pretrained generalist encoders (e.g., SigLIP2) with our own document and web-specific representations. Extensive experiments on classic document tasks, VQAs, web localization, and agent-based benchmarks validate the effectiveness of our approach, establishing DAVE as a strong vision encoder for document and web applications.

Published as a conference paper at ICLR 2026
DAVE: A VLM VISION ENCODER FOR DOCUMENT UNDERSTANDING AND WEB AGENTS
Brandon Huang !2, Hang Hua', Zhuoran Yu! Trevor Darrell',
Rogerio Feris and Roei Herzigh
MIT-IBM Watson Al Lab, UC Berkeley, University of Wisconsin-Madison
ABSTRACT
While Vision-language models (VLMs) have demonstrated remarkable perfor- mance across multi-modal tasks, their choice of vision encoders presents a fun- damental weakness: their low-level features lack the robust structural and spatial information essential for document understanding and web agents. To bridge this gap, introduce DAVE, vision encoder purpose-built for VLMs and tailored for these tasks. Our training pipeline is designed to leverage abundant unlabeled data bypass the need for costly large-scale annotations for document and web images. We begin with a self-supervised pretraining stage on unlabeled images, followed by a supervised autoregressive pretraining stage, where the model learns tasks like parsing and localization from limited, high-quality data. Within the su- pervised stage, we adopt two strategies to improve our encoder's alignment with both general visual knowledge and diverse document and web agentic tasks: (i) We introduce a novel model-merging scheme, combining encoders trained with different text decoders to ensure broad compatibility with different web agentic architectures, (ii) We use ensemble training to fuse features from pretrained gen- eralist encoders (e.g., SigLIP2) with our own document and web-specific repre- sentations. Extensive experiments on classic document tasks, VQAs, web local- ization, and agent- agent-based benchmarks validate the effectiveness of our approach, establishing DAVE as a strong vision encoder for document and web applications. Code available at wtlhtps:: http//gtttttuutttt thub. www.coom///c..e.. ccc.m//rrannnnn3///55....
INTRODUCTION
Vision-Language Models (VLMs) (Liu et al. 2024a Bai et al. 2025: Achiam et al. 2023) have
shown remarkable capabilities in multi-modal reasoning and understanding, enabling wide range of applications from image captioning to interactive web agents (Wu et al. 2024 Qin et al. 2025 Liu et al. 2023 Li et al. 20226). A central component of VLMs is the vision encoder (Yin al. 2024). which converts images into visual tokens for the language backbone to process (Li et al.
2023b
Liu et al. 2023). As such, the design of these encoders has become key focus in VLM
research TTong et al. 2024 Shi et al. 2024).
However, the vision encoders predominantly used by VLMs suffer from fundamental weakness: their low-level features lack the robust structural and spatial information essential for document un- derstanding and web agents (Tschannen et al. 2025 Radford et al. 202T). Conversely, DINO-style models (Oquab et al. (223) offer low-level features yet are tuned to natural images (Oquab et al.) 2023 Siméoni et al. 2025) and transfer poorly to documents, UIs, and diagrams (Tong et al. 2024). To bridge this gap, we introduce DAVE (Document and web Agents Vision Encoder), vision encoder purpose-built for VLMs and tailored for these tasks. See Figure ! for an overview.
A key challenge in training document and web images is the scarcity of high-quality annotated data, since current annotations rely on OCR models that bottleneck both scalability and quality (Kim) ct al. 1021 Lee et al. (202).. We address this with two-stage training process: self-supervised pre- training on large-scale unlabeled data, followed by autoregressive pretraining on limited annotated structural and grounding data. Yet, the supervised stage introduces further challenge: specialized
Equal advising. Correspondence zhaobi thaobin@bers n@berkeley beeeekeeey.eduu eddu, hang. hual 1 @ bbm com

---

Published as a conference paper at ICLR 2026
Stage Self- Supervisdd Pretraining
Stage 2: Supervised Pretraining
Grounding, Layout Prediction, OCR
Welcome
Pixel Reconstruction
LLM-1 (Qwen)
LLM-N (LLaMA)
Doc Web Features
High Level Semantic
Dave
Dave
Decoder
Encoder
SigLIP2
Encoder
SigLIP2
MAE architecture
Weight-space Merging
Encoder
Ensemble
DAVE
Training
DAVE
Welcome
Masked Image
DAVE
Representation Alignment
DAVE
DAVE
Figure I: DAVE Overview. Stage trains the vision encoder with decoder using MAE, learning strong structural and spatial priors from unlabeled data. Stage 2 performs autoregressive pretraining on diverse tasks with different text decoders and fuses the high-level semantic features from SigLIP 2. After that, different encoders are combined into a single one by learning a merge coefficient using unsupervised representation alignment, while keeping the encoders frozen.
vision encoders trained with a single text decoder tend to overfit to that decoder, leading to mis- alignment when paired with other decoders. We mitigate this issue via multi-decoder pretraining and weight-merging ("model soup") (Wortsman et al. 2022), which produces largely decoder-agnostic encoder. To preserve high-level semantics while leveraging fuse generalist encoder with our encoder during pretraining via ensembling. The resulting spe- cialized encoder integrates seamlessly with diverse decoders and generalist VLM stacks. Finally, our purpose-buil vision encoder serves as the core component of VLM architecture specifically designed for document and web agent tasks. We evaluate DAVE across diverse settings, including traditional and Lasion-language tasks. On traditional benchmarks, DAVE surpasses all SOTA mod- els in document recognition and segmentation, while achieving competitive results with SigL.IP evaluation, we test on document understanding,
structural and spatial features,
on screenshot classification. For uusion-languse web localization, and web agent. Notably, DAVE improves performance by an average 10.5% compared to SigLIP 2, highlighting its ability to provide specialized representations. Finally, we evaluate web agent settings on the Mind2Web benchmark (Deng et al.] 2023), where DAVE im- proves web agent performance by 5% over the strongest baseline vision encoder.
We summarize our main contributions as follows: (i) We introduce DAVE, a vision encoder purpose-built for VLMs and tailored for document and web agents tasks; (ii) We propose two- stage pretraining framework that combines a self-supervised pretraining stage on unlabeled images, followed by supervised autoregressive pretraining stage; (iii) We introduce model weight merging and ensemble training strategies to enhance the alignment of structural and spatial representation for documents and web images with diverse VLM and agentic frameworks; (iv) We perform extensive experiments to evaluate the impact of visual representations on document and web understanding, as well downstream web-agent tasks.
2 RELATED WORK
Pretrained Vision Encoders. Image-text contrastive learning (Radford et al. 202 Zhai et al.
2023 Jia et al. 202 Yu et al. 2022) and self-supervised representation learning (He et al.) 2022 Zhou et al. 2021 Oquab et al. 2023 Chen al., 2020; Caron et al.] 2021)
the two main
paradigms for large-scale visual pretraning. Recent work has sought to combine the strengths of
the two approaches through joint training or post-hoc alignment (T'chhannen al. 2025 Fini al.
2

---

Published as a conference paper at ICLR 2026
2023
Maninis al. 2024 Nacem et al. 2024), yielding strong encoders for both vision and mul- timodal tasks. The works most closely related to ours are Eagle (Shi et al. 2024) and Perception Encoder (Bolya et al. 2025), which align pretrained vision encoders a pretrained LLM with large- scale general datasets. In contrast, we pretrain the vision encoder from scratch with self-supervised
learning, use multiple pretrained L.MSs backbone, and use domain-specifie data create specialized foundation model.
Document Understanding with LLMs. Document processing and understanding have transitioned from OCR-based (Xu et al. 2020b a) methods to vision-language-
One line of work trains
encoder-decoder models with various objectives, like extracting structures (Feng al. 2025 Kim
it al. 2022, Lv et al.) [2033] Lee et al. (2023). and image reconstruction conditioned on text (Huang
et al., 2022 Tang et al. 2023). Recent advancements in VLMS led to another line of work that
uses document data to finetune a VLM (Nassar et al. 2025 Hu et al.] 2024] Liu et al. (024g) or
incorporate it as part of the VLM training (Team al. 2025 Bai al. 2025 Chen et al. 2024).
Our work seeks to bridge these two lines of work by introducing specialized vision encoder within a general VLM framework.
Vision-Language Models. Inspired by the success of recent large language models (LLMs) (Brown ct al. 2020 Dubey et al.] 2024). Vision-language models (Radford et al. 2021 Liu et al. 2024a
Sun et al. 2025 Hua et al. [2024b]]] Ye et al. 2023 Hu et al. [022]] Tang et al. 2024| Bai et al. 2025 Zuu et al. 2025 Hua et al. 2025a [Togg et al. 2024] Avogaro et al. (2026) aim to achieve
multimodal intel intelligence by jointly understanding and generating visual and language information.
Flamingo (Alayrac et al. 2022) and BLIP-2 (Li et al. (2023b) are two of the early works that explore integrating LLMs as part of VLM. Beginning with LLaVA (Liu et al. 2024a), researchers have used instruction-following chat data in VQA format for instruction tuning, achieving significantly
improved results (Liu et al. 2024e Li et al. 2023a Hua et al. 2024c Tang et al. 2025 Hua al. 2025b
Lu et al. 2023).
Multimodal Web Agents. Vision-based autonomous web agents have recently attracted attention for their simplicity and stronger generalization compared to LLM-based agents (Deng et al.) 2023 Kim et af., (023). Early efforts such as WebGUM (Furuta al., (023) and CogAgent (Hong al, 2024) pretrained vision-language models (VLMs) on web and GUI data to enhance agentic capabil- ities. Follow-up work [Wu et al. 2024 Xu et al. 2024 Liu et al. 2024b Qin et al.) 2025) expanded training with large-scale grounding and interaction datasets, further improving VLM performance on web-based tasks. While these works are promising steps toward a general-purpose visual web agent, the role of the vision encoder in these systems remains underexplored.
THE DAVE MODEL
3.1 PRELIMINARILIES
Masked Autoencoder. Masked Autoencoder (MAE) (He et al. 2022) adopts an asymmetric en- coder-decoder architecture. The vision encoder processes only subset of visible patches, while a lightweight transformer decoder reconstructs the full image from the encoded features and mask tokens. After pretraining, only the encoder is retained for downstream tasks.
By default, MAE applies per-patch normalization before computing the reconstruction loss to learn better representations. The reconstruction loss is then defined as
[MAE
Σ LEM
fo(#),
2; -: ((xx)
6= 10
M
(o²(x:)
where ş denotes the input image with masked patches, fe(ă), is the reconstructed output of patch i predicted by the decoder with parameters 0, 2; is the ground-truth pixel values of patch i, p(z,)) and (xx)) are the mean and variance of pixels within patch i, M is the set of masked patches.
Vision-Language Model. A vision- languag model (VLM) consists of vision encoder é, an MLP projector, and a text decoder @ The vision encoder takes an input image x E Mx x3 and produces sequence of patch-level features { s=1- These projected into the text embedding space via the MLP projector and used along with the text input.
In what follows, we describe the process get our vision encoder.
3

---

Published as a conference paper at ICLR 2026
3.2 STAGE 1: SSL ON DOCUMENT AND WEB IMAGES
In Stage I, we conduct self-supervised training with MAE to learn rich structural and spatial features from document and web images. While MAE demonstrates the strongest performance on OCR tasks compared to other methods, the training is prone to instability when trained at scale (Fan et al.) 2025j. We find that this instability particularly acute for document and web images. Our analysis (Section 5.4) attributes this to their characteristically low inter-patch variance, which destabilizes the target normalization in the standard MAE objective (Equation I objective to reconstruct raw pixel values directly:
To address this, we modify the
LMAE-pixel
Σ 1fa(z); (1: BE.M
(2)
|M
This change stabilizes training and enables scaling the training sample to 20 million images without additional hyperparameter tuning. With this robust self-supervised procedure, we then proceed to the second stage of our training process.
3.3 STAGE SUPERVISED MULTI-TASE PRETRAINING
To further enhance the encoder's structural and spatial understanding for document and web images, we perform a supervised, autoregressive pretraining stage. This stage utilizes a limited set of high- quality labeled data for tasks like OCR, layout extraction, and web localizations. We the VLM architecture as described in Section 3.1 and DAVE as the vision in the VLM architecture.
Weight-space Merging. A key limitation of the architecture discussed above is that the pretrained vision encoder becomes tightly coupled to its specific text decoder. This coupling significantly degrades performance when the encoder is integrated with a different decoder. To address this, we use model merging to create vision encoder that is agnostic to the choice of text decoder.
Formally. given a set of pretrained text decoders {O1..... Ond we train n corresponding in- stances of our vision encoder, denoted {Ø)..... 0. }. Each instance is aligned with a different text decoder but otherwise shares the same architecture, training data, and hyperparameters.
iiiillation-basse merging scheme that learns small set of coefficients combine pretrained weights while keeping the original parameters frozen. Considering each encoder o; as set of m weights {0(3) } we form each merged weight @ by learning set of coefficients {a, for each of the encoders:
To merge these different encoders, we propose
(5) } to compute the weighted sum of the corresponding weight
(G)
Σ
E [0.1].
merge
The merged encoder is thus composed of merged weights:
Oenenge { (01)) 1
The resulting encoder produces patch-level features Omerge (T). To ensure that Omerge preserves the features from all n teacher encoders, we define a distillation loss, Luisill- This objective mini- mizes the average Mean Squared Error between the merged features and the features from each teacher encoder z, (Z):
Cdistill
Σ
22
During the distillation process, all encoder parameters remain frozen; only the newly introduced combination coefficients are optimized. The final vision encoder given by the merged encoder:
@ Ómerge (fo; }
where {a: 12 are the optimized coefficients obtained from this training.
The next section details our method for fusing specialized document and web features with general visual representations.
Ensemble Training. Pretraining the encoder exclusively on document and web data provides strong structural and spatial features, but it also limits its grasp of general visual representation. This is
4

---

Published as a conference paper at ICLR 2026
erucial shortcoming, the high-level semantic features learned from diverse, scale datasets are equally important for robust performance.
To obtain both types of features, design an ensemble pretraining paradigm that combines frozen pretrained generalist encoder Ogen with our document and web specialist encoder @spec from the previous stage: (D)VE(r) = Concat (g)n( (x)ec(x)).
This design provides two main benefits: (i) it encourages (x) to focus on learning low-level structural and spatial representations, as high-level semantics are captured by @gon(<r;; and (ii) it enables early fusion structural and spatial features with high-level semantic features.
EVALUATION
4.1 IMPLEMENTATION DETAILS
Sule-supe Stage. For the SSL stage, we follow the original MAE implementation (He et al.) 2022], with ViT-L16-384 (Dosovitskiy et al. 2020) as our vision encoder. We train the MAE with batch size of 4096 for 120K steps. More detail can be found in Appendix
Supervised Training Stage. We adopt the VLM architecture discussed in Section 3.1 with an image tilting size of 384. During the ensemble training, employ frozen SigLIP-2 as the generalist vision encoder, while the encoder from the Self-supervised stage serves as the domain-specianl nent. This forms the full DAVE encoder. We experiment with multiple LLMs of varying scales and architectures, including S.Nen2.5-0.5B-Insee
compo-
(Bai et al. 2025), Phi-4-mini-Instruct (Abouelenin) al. 2025), and Granite-3. I-3IB-Instruct (Team et al. 2025). ). After a single epoch of full-parameter training, we retain only the DAVE encoder for weight-merging and downstream tasks. During the weight-merging distillation, we train the merge coefficient for 20 epochs on unlabeled documents and web images. Refer to Appendix B for implementation details.
4.2 EVALUATION SETTING
In this section, we first describe our encoder baselines, followed by two types of evaluations for the vision encoders: (i) finetuning vision encoders for classic document tasks, (ii) performing instruction tuning to build a VLM with the vision encoder for vision-language tasks and agentic tasks.
Baselines. We compare our vision encoder with both generalist and specialist encoders. For gener-
alist encoders, we utilize DinoV2 (Oquab et al., (023) and Web-SSI. MAE (Fan et al., (025), a MAE variant trained on 2 billion images. We also include SigLIP2 (Tschannen et al. 2025) and AIMv2 (Fini et al. 2025), which are SOTA encoders trained with both contrastive and reconstruction. For specialist vision encoders, we compare against DiT (Li et al. 2022a), Pix2Struct (Lee et al., 2023), and Dolphin (Feng al. (025). Since both Pix2struct and Dolphin are encoder-decoded models, use their encoders for comparison.
Classic Document Tasks. For each classic document task, we finetune the vision encoder and the suitable prediction heads. In DocBank, we use attention pooling to pool the visual feature, followed by an MLP head predict the bounding box. In Doclaynet, we train an MLP head to predict the semantic segmentation based on the feature patches. For RICO-SCA, we use attention pooling followed by an MLP to predict the class. Refer to Appendix for more details.
Vision-Language Model as Evaluator. To evaluate our vision encoders on more complicated and realistic tasks, such as VQA and instruction baasedd localization, we use VLMs as evaluators. Specif- ically, we follow the standard LLaVA architecture as discussed in Section 3.T We use a tilting size of 336 for ATMv2 and 384 for all other vision encoders. For vision encoders that produce a different number of visual tokens than the tilting resolution, we use bilinear interpolation as in LLaVAA-Oeviision (Li et al.) 2024) to interpolate the visual tokens. Note that we omit the projector alignment phase to improve training efficiency, as prior work has shown that this stage yields only limited gains Tong et al. (2024): Karamcheti et al. (2024). We use Llama- 3.2-3B--2---tttte- -B--stttt
(Dubey et al. 2024) and Swee22.57BB. Instruct (Bai al. 2025) the LLM backbone. We train for one
epoch with the vision encoder frozen. For Mind2web, we finetune the VLMs on the training set before performing offline evaluation the test set. More detail can be found in Appendix D
5

---

Published as a conference paper at ICLR 2026
Document
General VQA
Web
Screenspot-Vy V
Real
OCRBench
VisualWeb
Doc DocVQA
InfoVQA
ChariQA
MMMU
TextVQa
WebSRC
A12D
Model/ Variant
Llama-3.2-31 3.mma-33.-3BInsstruuc
MAE-Web
53.0
27.6
43.8
26.2
43.8
29.9
48.6
35.1
36.9
51.1
46,8
DinoV2
53.2
2.6
13.7
20.4
14.1
35.0
49.4
13.7
13.2
16.5
17.6
DiT
49.9
2.1
11.3
19.2
12.6
28.8
43.9
10.0
9.2
2.5
14.9
Pix2Struct
51.0
6.9
22.8
21.4
21.5
33.3
45.0
16.4
22.6
32.6
26.5
Dophine
50.8
44.7
74.8
37.4
60.3
33.3
44.4
42.8
48.8
56.3
76.0
AIMv2
56.1
48.3
56.6
35.6
48.7
36.3
51.4
58.7
46.7
32.0
49.4
SigLIP
58.0
51.5
72.1
40.6
51.8
36.9
53.5
64.4
54.7
40.7
67.8
DAVE (ours)
59.6
62.2
82.1
50.2
63.1
36.6
55.6
69.2
59.2
64.5
82.6
Dwen-22.5-BBImme Instruct
MAE-Web
63.8
33.2
55.1
28.6
50.3
38.4
45.4
40.6
59.5
70.1
60.7
Pix2struct
58.7
10.0
29.8
20.0
28.3
37.9
39.1
18.2
52.3
47.8
30.3
Dophine
63.3
50.9
83.3
42.0
66.3
40.2
44.6
49.4
60.7
75.2
85.6
AIMv2
69.7
69.3
89.3
57.8
77.4
41.3
56.2
73.1
64.9
79.3
86.1
SigLIP 2
73.6
67.4
90.2
55.2
75.9
43.9
58.7
74.3
65.4
81.4
88.3
DAVE (ours)
74.0
67.5
90.9
60.2
82.5
45.8
55.2
73.7
67.3
82.9
88.6
Table I: The DAVE's performance on Document understanding, general VQA, and Web under- standing benchmarks with VLM architectures using different LLMs. The best result per row is highlighted in bold and the second best with underline. Higher values represent better performance.
4.3 BENCHMARKS
In this section, we describe our evaluation datasets for several downstream benchmarks.
Classic Document Understanding Tasks. To directly evaluate the structural and spatial visual representation from the vision encoders, we consider several document parsing tasks that can be performed with only images. We employ DocBank (Li et al. 2020a) to comprehensively evaluate document recognition on different categories, including tables, charts, and paragraphs. We also use Doclaynet (Ptitzmann et al.) 2022) to test document segmentation. For web tasks, we use RICO- SCA LL et al. 2020b) to pertorm web UI classification.
Document and General VQA. For document understanding and question answering, evaluate on A12D (Kembhavi et al.) 2016). OCRBench (Liu et al.] 2024f).. Doc (Mathew et al. [021), InfoVQA (Mathew al. 2022), and ChartQA (Masry al. 2022). For general vasion-language
understanding. we report results on MMMU (Yue et al. 2024). RealWorldQA (xA] 2024). and
TextVQA (Singh al. 2019).
Web UI and Agent Tasks. For web UI localization, we use Screenspot-V2 V2 Wu et al. 2024). which consists of text instructions (e.g., "click the button in cooridnate (x, y)"), and the corresponding bounding boxes. We report the center accuracy: the label-box center lies within the predicted box.
We also evaluate on WebSRC (Chen et al., (02T) and VisualWebBench (Liu al. (024d) for web UI question answering. For the agentic benchmark, we evaluate on Mind2Web (Deng et al. (023)), where the model receives a user goal and webpage screenshot, and must predict an action like clicking.
4.4 DATASETS
In this section, we provide a description of our training datasets for the different stages.
Self-supervised Learning Data. We use 20 million images for self-supervised training. For document data, we sample 10 million PDF images from DocFM (Team et al. 2025). which contains 85 million document pages extracted from Common Crawl, Wikipedia, and ESG (Eavironmental,

---

Published as a conference paper at ICLR 2026
Mind2Web
Model
Cross- Task
Seoss-Website
Cross-Domain
Element Acc.
Step SR
Element
Step SR
Element Acc.
Step SR
Acc.
Mamaa-322-BB-naa
MAE
21.0
16.8
16.5
11.9
15.5
11.6
DinoV2
13.5
11.4
10.2
6.6
9.5
7.2
DIT
5,8
4.1
2.7
1.2
2.1
1.1
Pix2Struct
15.8
12.3
12.1
10.8
7.5
Dolphin
24.0
19.6
20.9
13.6
19.3
14.5
AIMv2
14.7
11.6
9.2
5.9
8.8
6.1
SigLIP 2
20.6
17.3
11.8
8.7
12.8
9.7
DAVE (ours)
30.8
26.1
24.2
18.0
23.9
19.1
Table 2: Results on Web Agent. Performance on Mind2Web with three splits (Cross-Taske Cross- Website, Cross-Domain). We report the stepwise accuracy (correct grounding) and the element accuracy (correct grounding and action) for each task.
Social, and Governance) reports. The PDFs are filtered to include English only. For web screenshot, we sample 10 million from Common Web (com 2023 without any language filtering.
Supervised Learning Data. For the autoregressive supervised training, leverage data from diverse sources, including PlotQA (Methani et al. 2020), ChartQA (Masry et al.] (022). fintab- net (Zheng et al., (021), Datikz (Belouadi et al. 2023), Pubtables (Smock al. 2022). and
DocFM (Team et al. (2055). Importantly, we use the version from Granite Vision (Taam al. 2025) where all the problems are reformulated into content extraction. The data for this stage is a mixture of curated benchmarks and a large-scale, self-processed dataset. The cu- rated benchmarks cover tasks such as Mhart-to-markooww ing with UGround Gou et al. (2024). To expand on this, we processed 500K PDF samples from arXiv with an OCR model (Cui al. 2025), formulating them into additional recognition and grounding tasks. Altogether, this training stage comprises approximately 2 million samples.
table-to-caption, and web UI ground-
Instruction Tuning Data. Cambrian-1 Tong et al., 2024).
Following use
Model
DocLayNet DocBank RICO-SCA
DinoV2
68.4
38.3
85.6
the LLaVA-1.3-mix 665K (Liu et al.) 2023),
MAE-Wa Web
64.6
44.5
88.3
DoA (Mathew et al. 2021). ChartQA (Masry al. (022). and AL2D (Kembhavi et al. 2016) as our instruction tuning data.
Dolphin
53.8
50.5
88,4
Pix2Struct
56.7
47.2
90.1
AIMv2
70.5
42.4
91.3
We also integrate Pixmo-Doc (Yang al. 2025) into the data mix to improve document understanding. Additionally, we add MultiUI (Liu et al. 2024c) to improve the web and agentic capability. This combined to a total of 2.5 million training images.
SipLIP 2
70.8
51.7
93.3
DAVE
74.1
56.9
92.8
Table Performance comparison on classic doc- ument tasks. DocLayNet and DocBank use mAP, while RICO-SCA uses classification accuracy.
RESULTS
5.1
VISIAN-LANGUAGE TASKS
Table shows the evaluation results for document, general, and web benchmarks. Overall, DAVE consistently outperforms the strongest baseline, SigL.P 2 in the Llama-3.2-3B-Innsee document and web benchmarks by an average of 10.5%. This is achieved without losing the gen- eral VQA capabilities, like MMMU and RealWorIdQA, which implies that the ensemble training successfully merges the structural and spatial features with the general visual features. In addition, when using Qwen-2.5-7B Instructt as the VLM decoder, show that the improvements in bench- marks hold similar trend. This suggests that our merging scheme effectively aligns with different text decoders. Finally, one interesting observation is that specialized models like Pix2Struct and
setup on 8

---

Published as a conference paper at ICLR 2026
Distribution of Inter-Patch std With =6
Stage Ablation
Benchmarks
Scratch Decoder
@ 52.2
Inter-Patee dt
10
Pretrained Decoder
Ensemble
55.6
Model Merge
50
10a3
Accuracy (%)
ImageNet-llke
Document-llks
Web
Figure 2: Each row indicates an additional mod- ification to the training strategy.
Figure Inter- Inter-patch standard deviation across different data sources.
Dolphin perform suboptimally compared to SigLIP and AIMv2, which shows that both general visual features and document/web--gg -eppeeiiiieessiiiiiiieeeeeeeiiee features are critical to the performance. This hypothesis is further supported by DAVE's strong results.
5.2 WEB AGENTIC TASKS
The results for weeed are outlined in Table DAVE outperforms the best baseline encoder, Dolphin, by an average of 5%. Surprisingly, self-supervised vision encoders like MAE and D2 perform competitively with SigLIP and AIMv2, while document-sppciialzee encoder Dolphine achieves the best accuracy among all baselines. This suggests that structural and spatial aspects may be more important in web navigation compared to broad general features.
5.3 CLASSIC DOCUMENT AND WEB TASKS
As shown in Table we evaluate DAVE on dense document tasks. DAVE outperforms both spe- cialized and generalist encoders on DocBank and Doylaynet. This demonstrates the strong structural and spatial visual features learned by our 2-stage pretraining. Furthermore, DAVE performs slightly worse than SigLIP 2 on screen classification, semantics-heavy task that is not the primary opti- mization target DAVE. In principle, DAVE should be able to match SigLIP 2 because it fuses SigLIP features, and additional ablations (See Appendix E..4) confirm that adding SigLIP 2 im- proves performance, supporting the claim that it contributes high-level semantic information. We attribute DAVE's suboptimal performance compared to SigLIP 2 to the increased hidden dimension- ality. Since DAVE's embedding dimension is twice as large as that SigLIP 2, it is more difficult to pool and process the information with the same lne-layer MLP prediction head.
5.4 ABLATIONS
In this section, we present detailed ablations on the design choice of our training method. In all the ablations, we report the average performance on document and web vision- languae benchmarks, which we denote as Doc and Web in the Table 4
Training Method. To assess the contribution of each design choice in the stage 2 supervised pre- training, we incrementally add each choice and plot the performance trajectory on document vision- language tasks tasks in Figure 2 Beginning with the scratch text decoder and subsequently incor- porating a pretrained LLM as the text decoder, ensemble training, and finally weight merging. The consistently improving trend highlights the positive impact of our proposed design choices.
Image Inter-Patch Variance. To show the instability of the MAE training on document and web image discussed in Section 3.3] plot the distribution density of inter- patch variance across dif- ferent data sources in Figure 3 Compared to ImageNet (Deng et al., (009), document and web
images exhibit much lower variance. This distributional gap underscores the need for specialized
8

---

Published as a conference paper at ICLR 2026
vision encoder capable of handling such data. Additional implementation details provided in Appendix
Method
Doc
Web
Method
Doc
Web
No merge
55.6
53.0
From Scratch
52.2
54.7
Average
62.8
67.7
Granite
55.6
53.0
Fisher Merge
60.3
67.0
Granite+Qwen
62.1
64.8
Learned Coef
63.4
68.2
Qrenit+Qwen++Phi
63.4
68.2
Comparison of Different Merge Methods.
(b) Comparison of Different Merging LLMs.
Method
Doc
Web
Method
Doc
Web
SIgLIP 2 DiT
50.3
47.6
SigLIP 2
49.1
54.4
SIgLIP 2 PS
48.9
42.2
Finetune SigLIP
58.2
65.2
SIgLIP 2+ DP
49.7
44.0
DAVE
63.4
DAVE
63.4
68.2
(d) Comparison with Finetuning.
(c) Comparison of Different Specialized Fi- coder Fusion. Dolphin
Key: PS-Pix2struct,, DP-
Table 4: Ablation study: (a) merge methods, (b) LLM merging, and additional analyses (c, d).
Different Merge Setting. As described in Section 3.3 we use weight merging to create a decoder- agnostic vision encoder. Table 4a examines the effect of different merge strategies. Our end-to- end approach for learning merge coefficients outperforms both simple averaging and the heuristic- based Fisher Merge (Matena Raffel (022) method, which weights parameters by their estimated importance using the Fisher information. Table 4b) evaluates merging DAVE from different LLM backbones, where performance improves consistently with the number merged encoders.
Multi-encoder Comparison. Because DAVE combines with both generalist and specialist features, we ask how DAVE performs compared to multi-encoder settings. Specifically, we concatenate dif- ferent document and web specialized models with SigLIP during the instruction tuning of building the VLM. In Table 4c we present the result of different multi-encoder settings. We highlight that without any feature fusion, all the specialized encoders fail to improve performance on document and web benchmarks.
Comparison with Finetuning. To test whether pretraining approach is more effective than directly finetuning an existing vision encoder on the supervised data used our pretraining, we finetune SigLIP with our supervised pretraining data. In Table 4d] we show that DAVE outper- forms finetuned SigLIP 2, demonstrating the effectiveness of our approach.
6 CONCLUSION
We introduced DAVE, foundational vision encoder for document and web understanding, Our approach combines self-supervised learning on large-scale unlabeled data with supervision from limited vaniun-l annotations, enabling data-efficient training in specialized domains. By leveraging pretrained generalist encoders, DAVE can focus on learning domain-specific features. Extensive experiments demonstrate that strong visual representations are critical for document, web, and agentic tasks, where our encoder achieves significant improvements over prior models. Ablation studies further confirm the method's 's flexibility on different VLM and agentic frameworks.
7 LIMITATIONS AND FUTURE WORK
Despite the significant progress achieved in document and web understanding by DAVE, several limitations remain. The model operates at fixed resolution, depends on an external pretrained gen- eralist encoder, and requires substantial compute for self-supervised training. In agentic settings,

---

Published as a conference paper at ICLR 2026
its vision encoder is also not conditioned prior actions. These limitations highlight opportunities for developing vision encoders that are more compute- and data-efficient, can process inputs of any resolution aspect ratio, and incorporate prior agentic actions to produce more tailored representa- tions. In addition, this paradigm offers a promising path for domains with scarce vision-language data, such as medical imaging. While we do not anticipate specific negative impacts, with any machine learning method, caution should be exercised during deployment.
REFERENCES
Common screens. http://////eeeee......aaaaaaaaaa
2023.
Abdelrahman Abouelenin, Atabak Ashfaq, Adam Atkinson, Hany Awadalla, Nguyen Bach, Jianmin Bao, Alon Benhaim, Martin Cai, Vishrav Chaudhary, Congcong Chen, et al. Phi-4-mini technical report: Compact yet powerful multimodal language models via mixture-offloraa arXiv preprint arXiv:2503.01743. 2025.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Ale- man, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXi::3333.0874,, 2023.
Lean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 2022.
Niccolo Avogaro, Nayanika Debnath, Li Mi, Thomas Frick, Junling Wang. Zexue He, Hang Hua, Konrad Schindler, and Mattia Rigotti. Spare: Separating perception and reasoning circuits for test-time scaling of vlms. arXiv preprini arXiv:2602.06566 2026.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint rrXiv:2502.13923, 2025.
Jonas Belouadi, Anne Lauscher, and Steffen Eger. Automatikz: Text-guided synthesis of scientific vector graphics with tikz. arXiv preprint arXiv:23/0.003:5 2023.
Daniel Bolya, Yao Huang, Peize Sun, Jang Hyun Cho, Andrea Madotto, Chen Wei, Tengyu Ma, Jiale Zhi, Jathushan Rajasegaran, Hanoona Rasheed, et al. Perception encoder: The best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504. 13181, 2025.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33: 13:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pp. 1597-1607. PmLR, 2020.
Xingyu Chen, Zihan Zhao, Lu Chen, Danyang Zhang, Jiabao Ji, Ao Luo, Yuxuan Xiong, and Kai Yu. Websre: A dataset for web-based structural reading comprehension. arXiv preprint arXiv:2101.09465. 2021.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shen- glong Yc, Hao Tian, Zhaoyang Liu, al. Expanding performance boundaries of soures multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271, 2024.
Cheng Cui, Ting Sun, Manhui Lin, Tingquan Gao, Yubo Zhang, Jiaxuan Liu, Xueqing Wang, Zelun Zhang, Changda Zhou, Hongen Liu, et al. Paddleocr 3.0 technical report. arXiv preprint arXiv:2507.05555, 2025.
10

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icce, 2009.
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36:28091-2811 14, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, al. The llama 3 herd of models. arXiv e-prints, pp. arXiv-2407, 2024.
David Fan, Shengbang Tong, Jiachen Zhu, Koustuy Sinha, Zhuang Liu, Xinlei Chen, Michael Rab- bat, Nicolas Ballas, Yann LeCun, Amir Bar, and Saining Xie. Scaling language-free visual repre- sentation learning. 2025.
Hao Feng, Shu Wei, Xiang Fei, Wei Shi, Yingdong Han, Lei Liao, Jinghui Lu, Binghong Wu, Qi Liu, Chunhui Lin, al. Dolphin: Document image parsing via heterogeneous anchor prompting. arXiv preprint arXiv:2505. 14059, 2025.
Enrico Fini, Mustafa Shukor, Xiujun Li, Philipp Dufter, Michal Klein, David Haldimann, Sai Aitharaju, Victor G Turrisi da Costa, Louis Béthune, Zhe Gan, et al. Multimodal autoregres- sive pre-training of large vision encoders. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 9641-9654, 2025.
Hiroki Furuta, Kuang- Huei Lee, Ofir Nachum, Yutaka Matsuo, Aleksandra Faust, Shixiang Shane Gu, and Izzeddin Gur. Multimodal web navigation with instruction arXiv preprint arXiv:2305.1 1854, 2023.
instruction-ineetued finetuned foundation models.
Boyu Gou, Ruohan Wang, Boyuan Zheng, Yanan Xie, Cheng Chang, Yiheng Shu, Huan Sun, and Yu Su. Navigating the digital world as humans do: Universal visual grounding for gui agents. arXiv preprint arXiv:2410.05244 2024.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000-16009, 2022.
Wenyi Hong, Weihan Wang, Qingsong Lx, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14281-14290, 2024.
Anwen Hu, Haiyang Xu, Jiabo Ye, Ming Yan. Liang Zhang, Bo Zhang, Chen Li. Ji Zhang, Qin Jin, Fei Huang, et al. mplug-docowl 1.5: Unified structure learning for ocr-free document understand- ing. arXiv preprint ar 2403 /2895, 2024.
Yushi Hu, Hang Hua, Zhengyuan Yang, Weijia Shi, Noah A Smith, and Jiebo Luo. Prompteap: Prompt-guide task- image captioning. arXiv preprint arXiv:2211.09699 2022.
Hang Hua, Jing Shi, Kushal Kafle, Simon Jenni, Daoan Zhang, John Collomosse, Scott Cohen, and Jiebo Luo. Finematch: Aspect- based fine-grained image and text mismatch detection and correction. In European Conference on Computer Vision, pp. 491. Springer, 2024a.
Hang Hua, Yunlong Tang, Chenliang Xu, and Jiebo Luo. V2xum-Ilm: Cross-modal video summa- rization with temporal prompt instruction tuning. arXiv preprint arXiv:2404.12355 2024b.
Hang Hua. Yunlong Tang, Ziyun Zeng, Liangliang Cao, Zhengyuan Yang, Hangfeng He, Chen- liang Xu, and Jiebo Luo. Mmcomposition: Revisiting the compositionality of are-trained vision- language models. arXiv preprint 100.922410.09733, 2024c.

---

Published as a conference paper at ICLR 2026
Hang Hua, Qing Liu, Lingzhi Zhang, Jing Shi, Soo Ye Kim, Zhifei Zhang, Yilin Wang, Jianming Zhang, Zhe Lin, and Jiebo Luo. Finecaption: Compositional image captioning focusing on wher- you want any granularity. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 24763-24773, 2025a
Hang Hua. Ziyun Zeng, Yizhi Song, Yunlong Tang, Liu He, Daniel Aliaga, Wei Xiong, and Jiebo Luo. Mmig-bench Towards comprehensive and explainable evaluation of multi-modal image generation models. arXiv preprint arXiv:2505. 19415, 2025b.
Yupan Huang, Tengchao Lv, Lei Cui, Yutong Lu, and Furu Wei. Layoutlmv3: Pre-training for document ai with unified text and image masking, Proceedings of the 30th ACM international conference multimedia, pp.
2022.
Peter Huber. Robust estimation of a location parameter. The Annals of Mathematical Statistics, 35 (1):73-101, 1964.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International conference machine learning, pp. 4904-4916. PMLR, 2021.
Siddharth Karamcheti, Suraj Nair, Ashwin Balakrishna, Percy Liang, Thomas Kollar, and Dorsa Sadigh. Prismatic vlms: Investigating the design space of visually-conditioned language models. In Forfy-first International Conference Machine Learning, 2024.
Aniruddha Kembhavi. Mike Salvato, Eric Kolve, Minjoon Sco, Hannaneh Hajishirzi, and Ali Farhadi. A diagram is worth a dozen images. In Computer Vision ECCV 2016, pp- 235-251, Cham, 2016. Springer. doi: 10. 000079978-33319444999
Geewook Kim, Teakgyu Hong. Moonbin Yim, Jinyoung Park, Jinyeong Yim, Wonscok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Donut: Document understanding trans- former without ocr. arXiv preprint arXiv:2111.1566 7(15).2, 2021.
Geewook Kim, Teakgyu Hong. Moonbin Yim, Jeong Nam, Jinyoung Park, Jinyeong Yim, Wonseok Hwang, Sangdoo Yun, Dongyoon Han, and Seunghyun Park. Ocr-free document un- derstanding transformer. In European Conference on Computer Vision, pp. 998-517. Springer, 2022.
Geunwoo Kim, Pierre Baldi, and Stephen McAleer. Language models can solve computer tasks. 2023.
Advances in Neural Information Processing Systems,
Kenton Lee, Mandar Joshi, Julia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wes Wei Chang, and Kristina Toutanova. Pix2struct: Screen- shot parsing as pretraining for visual language understanding. In International Conference on Machine Learning. pp. 18893-18912. PMLR, 2023.
Bo Li, Yuanhan Zhang. Dong Guo, Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Easy visual task transfer. arXiv preprint arXiv:2408.0 03326, 2024.
Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Bench- marking multimodal Iims with generative comprehension. arXiv preprint arXiv:2307.16125, 2023a.
Junlong Li, Yiheng Xu, Tengchao Lv, Lei Cui, Cha Zhang, and Furu Wei. Dit: Self-supervised pre-training for document image transformer. In Proceedings of the 30th ACM international conference on multimedia, pp. 3530-3539, 2022a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping maaguage-imagge pre- training for unified languael understanding and generation. In International conference on machine learning, pp. 12888-18 12900. PMLR, 2022b.
12

---

Published as a conference paper at ICLR 2026
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-imad pre-training with frozen image encoders and large language models. In International conference machine learning. pp. 19730-19742. PMLR, 2023b.
Minghao Li, Yiheng Xu, Lei Cui, Shaohan Huang, Furu Wei, Zhoujun Li, and Ming Zhou. Docbank: A benchmark dataset for document layout analysis. arXiv preprini arXiv:2006.01038 2020a.
Yang Li, Jiacong He, Xin Zhou, Yuan Zhang, and Jason Baldridge. Mapping natural language instructions to mobile ui action sequences. arXiv preprint arXiv:2005.03770 2020b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, Bo Li, Yuanhan Zhang, Sheng Shen, and Yong Jae Lee. Llava-next: Improved reasoning, ocr, and world knowledge, January 2024a. URL https:/
Ilava vl.githuk. httpoo///oo...ss/ 01 30- 1lava llava-next/
Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Gra- ham Neubig, and Xiang Yue. Harnessing webpage uis for text-rich visual understanding. arXiv preprint arXiv:2410. 1 13824, 2024b.
Junpeng Liu, Tianyue Ou, Yifan Song, Yuxiao Qu, Wai Lam, Chenyan Xiong, Wenhu Chen, Graham Neubig, and Xiang Yue. Harnessing webpage uis for text-rich visual understanding, 2024c. URL https: arxiv.iiivvv...ve org/ 2oos/2110.. 13824
Junpeng Liu, Yifan Song, Bill Yuchen Lin, Wai Lam, Graham Neubig, Yuanzhi Li, and Xiang Yue. Visualwebhench: How far have multimodal Iims evolved in web page understanding and grounding? arXiv preprint arXhv:240..05555, 2024d.
Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, et al. Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pp. 216-233. Springer, 2024c.
Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xucheng Yin, Cheng-lin Liu, Lianwen Jin, and Xiang Bai. Ocrbench: on the hidden mystery of ocr in large multimodal models. Science China Information Sciences, 67(12):22010,, December 2024f. ISSN 1869-1919. doi: 10.1007/111132--24-4333
URL http://dx dei org/10.../... 1007/
s11432- 024 4235-6
Yuliang Liu, Biao Yang, Qiang Liu, Zhang Li, Zhiyin Ma, Shuo Zhang, and Xiang Bai. Textmonkey: An ocr-free large multimodal model for understanding document. arXiv preprint 2411::2003004473, 2024g.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:171..01101, 2017.
Pan Lu, Hritik Bansal, Tony Xia, Jiacheng Liu, Chunyuan Li, Hannanch Hajishirzi, Hao Cheng, Kai- Wei Chang, Michel Galley, and Jianfeng Gao. Mathvista: Evaluating mathematical reasoning of foundation models in visual contexts. arXiv preprint arXiv:2310.0225.. 2023.
Tengchao Lv, Yupan Huang, Jingye Chen, Yuzhong Zhao, Yilin Jia, Lei Cui, Shuming Ma, Yaoyao Chang, Shaohan Huang, Wenhui Wang, al. Kosmos-2.5; A multimodal literate model. arXiv preprint arXiv:2309. 11419, 2023.
Kevis-Kokitsi Maninis, Kaifeng Chen, Soham Ghosh, Arjun Karpur, Koert Chen, Ye Xia, Bingyi Cao, Daniel Salz, Guangxing Han, Jan Dlabal, et al. Tips: Text-image pretraining with spatial awareness. arXiv preprint arXiv:2410.16511 2024.
Ahmed Masry, Do Xuan Long. Jia Qing Tan, Shafiq Joty, and Enamul Hoque. Chartqa: bench- mark for question answering about charts with visual and logical reasoning, arXiv preprint arXiv:2203. 10244, 2022.
Michael S Matena and Colin A Raffel. Merging models with fasher-weghhtd averaging. Advances in Neural Information Processing Systems, 35:17703-17716 2022.
13

---

Published as a conference paper at ICLR 2026
Minesh Mathew, Dimosthenis Karatzas, and C. V. Jawahar. Docvqa: A dataset for vqa document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pp. 2199-2208, 2021. doi:
Minesh Mathew, Viraj Bagal, Rubên Tito, Dimosthenis Karatzas, Ernest Valveny, and C.V. Jawahar. the Infographievga. In Proceedings the IEEE/CVF Winter Conference on Applications of Computer
Vision (WACV). pp. 1697-1706, January 2022.
Nitesh Methani, Pritha Ganguly, Mitesh M Khapra, and Pratyush Kumar. Plotqa: Reasoning scientific plots. In Proceedings of the ieee/cyf winter conference on applications of computer vision, pp. 1527-1536, 2020.
Muhammad Ferjad Naeem, Yongqin Xian, Xiaohua Zhai, Lukas Hoyer, Lue Van Gool, and Fed- erico Tombari. Sile: Improving vision language pretraining with self-distillation. European Conference on Computer Vision, pp. 38-55. Springer, 2024.
Ahmed Nassar, Andres Marafioti, Matteo Omenetti, Maksym I Lysak, Nikolaos Livathinos, Christoph Auer, Lucas Morin, Rafael Teixeira de Lima, Yusik Kim, A Said Gurbuz, et al. Smoldocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion. arXiv preprint arXiv:2503. 2025.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
Birgit Pfitzmann, Christoph Auer, Michele Dolfi, Ahmed Nassar, and Peter Staar. Doclaynet: A large annan-ad dataset for document-layou segmentation. In Proceedings of the 28th ACM SIGKDD conference on knowledge discovery and data mining, pp. 3743-3751, 2022.
Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, et al. Ui-tars: Pioneering automated gui interaction with native agents. arXiv preprint arXiv:2501.12322 2025.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PmLR, 2021.
Claude Shannon. mathematical theory of communication. Bell System Technical Journal, 27 (3):379-423, 1948.
Min Shi, Fuxiao Liu, Shihao Wang, Shijia Liao, Subhashree Radhakrishnan, Yilin Zhao, De-An Huang, Hongxu Yin, Karan Sapra, Yaser Yacoob, et al. Eagle: Exploring the design space for multimodal Iims with mixture of encoders. arXiv preprint arXiv:2408. 15998, 2024.
Oriane Siméoni, Huy V Vo. Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 2025.
Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Brandon Smock. Rohith Pesala, and Robin Abraham. Pubtables- Im: Towards comprehensive ta- ble extraction from unstructured documents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4634 4642, 2022.
Guohao Sun, Hang Hua, Jian Wang, Jiebo Luo, Sohail Dianat, Majid Rabbani, Raghuveer Rao, and Zhiqiang Tao. Latent chain-of- ofthought for visual reasoning. arXiv preprint arXiv:2510.23925, 2025.
Yunlong Tang, Daiki Shimada, Jing Bi. and Chenliang Xu. Avicuna: Audio-visual 1lm with interleaver and context-boundary alignment for temporal referential dialogue. arXiv preprint arXiv:2403.16276 2024.
14

---

Published as a conference paper at ICLR 2026
Yunlong Tang, Junjia Guo, Hang Hua, Susan Liang, Mingqian Feng, Xinyang Li, Rui Mao, Chao Huang. Jing Bi, Zeliang Zhang, et al. Vidcomposition: Can milms analyze compositions in compiled videos? In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 8490-8500, 2025.
Zineng Tang, Ziyi Yang, Guoxin Wang, Yuwei Fang, Yang Liu, Chenguang Zhu, Michael Zeng, Cha Zhang, and Mohit Bansal. Unifying vision, text, and layout for universal document processing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 19254-19222 19264, 2023.
Granite Vision Team, Leonid Karlinsky, Assaf Arbelle, Abraham Daniels, Ahmed Nassar, Amit Al- fassi, Bo Wu, Eli Schwartz, Dhiraj Joshi, Jovana Kondic, et al. Granite vision: a lightweight, open-source multimodal model for enterprise intelligence. arXiv preprint arXiv:2502.09227, 2025.
Shengbang Tong, Ellis Brown, Penghao Wu, Sanghyun Woo, Manoj Middepogu, Sai Charitha Akula, Jihan Yang, Shusheng Yang, Adithya Iyer, Xichen Pan, et al. Cambrian-1: A fully open, vision-centric exploration of multimodal 1lms. arXiv preprint arXiv:2406. 16860, 2024.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14788 2025.
Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roclofs, Raphael Gontijo-Lppes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In International conference on machine learning, pp. 23965-23998. PMLR, 2022.
Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, et al. Os-atlas: A foundation action model for gener- alist gui agents. arXiv preprint arXiv:2410.2.32.
2024.
XAI. Realworldga: A benchmark for evaluating real-world spatial understanding in multimodal ai models. hhttps:////t htw.iggingaacccco///oae/ cessed: 2025-09-17.
www.aaaoooooeeee.eeee...eo 2024. Ac-
Yang Xu, Yiheng Xu, Tengchao Lv, Lei Cui, Furu Wei, Guoxin Wang, Yijuan Lu, Dinei Floren- cio, Cha Zhang, Wanxiang Che, et al. Layoutlmy2: Multi-modal pre-training for visually-rich document understanding. arXiv preprint 20Xiv:2012.147400 2020a.
Yiheng Xu, Minghao Li, Lei Cui, Shaohan Huang, Furu Wei, and Ming Zhou. Layoutlm: Pre- training of text and layout for document image understanding. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1192-1200, 2020b.
Yiheng Xu, Zekun Wang, Junli Wang. Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, and Caiming Xiong. Aguvis: Unified pure vision agents for autonomous gui interaction. arXiv preprint ar 2xiiv2242..0445.0 2024.
Yue Yang, Ajay Patel, Matt Deitke, Tanmay Gupta, Luca Weihs, Andrew Head, Mark Yatskar, Chris Callison-Burch, Ranjay Krishna, Aniruddha Kembhavi, et al. Scaling text-rich image understand- ing via code-guided synthetic multimodal data generation. arXiv preprint arXiv:2502.1488 2025.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang. Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arX1v:2304.1417. 2023.
Shukang Yin. Chaoyou Fu, Sirui Zhao, Ke Li, Xing Sun, Tong Xu, and Enhong Chen. A survey on multimodal large language models. National Science Review, 11(12):nwae403 2024.
15

---

Published as a conference paper at ICLR 2026
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang, Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multi- modal understanding and reasoming benchmark for expert agi. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2024.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 11975-11986, 2023.
Xinyi Zheng, Douglas Burdick, Lucian Popa, Xu Zhong, and Nancy Xin Ru Wang. Global table extractor (gte): A framework for joint table identification and cell structure recognition using visual context. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 697-706, 2021.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:211 2021.
Jinguo Zhu, Weiyun Wang. Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Hao Tian, Yuchen Duan, Weijie Su, Jie Shao, et al. Internv13: Exploring advanced training and test-time recipes for open-source multimodal models. arXiv preprint arXiv:2504. 10479, 2025.
16

---

Published as a conference paper at ICLR 2026
Hyperparameter
Value
Batch Size
4096
Learning Rate
lc-5
Epochs
25
Warmup Epochs
Weight Decay
0.05
Mask Ratio
0.75
Scheduler
Cosine
Table 5: Training hyperparameters for MAE training.
A STAGE 1: SELF-SUPERVISED TRAINING
Here, we present the implementation detail and dataset used for the self-supervised training.
A.1 IMPLEMENTATION DETAILS
We use the official implementation of masked autoencoder (MAE) for our training, and we use 32 H200 GPUs for the training. Our vision encoder is ViT-L-384 initialized from scratch, whereas our decoder is a transformer model with depth of 4 and 16 attention heads. The detail training hyperparameter be found in Table
A.2 DATASETS
Our self-supervised pretraining dataset consists of 10 million document images from DocFM and 10 million web screenshots from Common Screen
D. DocFM is large scale document data collected by IBM consisting of 85 million document pages extracted from unique PDF documents sourced from Common Crawl, Wikipedia, and ESG (Environmental, Social, and Governance) reports. From this dataset, first filter and keep only the English pdfs, then we randomly sample 10 million pdfs and store them as images.
Common Screen. Common Screen is large scale web screenshot data consisting 70 million screenshot images based on the Common Crawl data. We randomly sample 10 million images as our training data, without any language filtering.
A.3 ADDITIONAL RESULTS ON MAE
Model
DocBank
Doclaynet
RICO-SCA
MAE-We Webb
64.6
44.5
MAF-Doc
72.8
52.5
90.7
Table 6: Evaluation results classic document tasks.
In this section, demonstrates the effectiveness of pretraining document and web images by comparing our stage- encoder, which we denote as MAE-Doc, with the the pretrained MAE-Web introduced in [4.2 classic document tasks. In Table the superior performance of MAE-Doc over MAE-Wa Web highlights importance of the domain pretraining for document and web tasks.
A.4 TRAINING DIVERGENCE
In Figure we show that training MAE on document and web screenshot images with the normal- ized pixel loss leads to training divergence.
17

---

Published as a conference paper at ICLR 2026
10K
Itarations
Irstr Joax
Figure 4: Training loss curve of MAE with normalized pixel as objective.
Hyperparameter
Value
Max sequence length
20000
Learning Rate
3e-5
Epochs
Warmup ratio Scheduler
0.03
Cosine
Table 7: Training hyperparameters for supervised pretraining.
B STAGE 2: SUPERVISED PRETRAINING
In this section, we present the detail implementation and the dataset composition of our supervised pretraining after the self-supervised pretraining.
B. IMPLEMENTATION DETAILS
We use the official LLaVA-Next (Li al. 2024) repo with some customization to support different LLM and encoder settings, and we use 32 H200 GPUs for the training. To handle high resolution images, we use AnyRes image tilting with size of 384x384. The hyperparameters listed in Table
B.2 DATASETS
Dataset
#Image
Description
FM4D
250k
Doc/Chart/Table Extraction
PlotQA
146k
Plot Extraction
Fintabnet
Table Extraction
ChartQA
17k
Chart Extraction
Datikz
94k
Image to Latex
Pubtables
480k
Table to HTMI.
UGround
756k
Web UI Grounding
Self-curated Data
Doc Grounding/straature Recognition
Table 8: Dataset description for supervised pretraining.
We use a wide range of publicly available data for our supervised pretraining. Notably, all the data excluding UGround reformulated into layout and information extraction. For more detail on the data curation, refer to Granite Vision (Team et al. 2025). For the Self-curated Data, we use PaddleOCR our OCR model. We first filter arxiv papers include only those published in venues
18

---

Published as a conference paper at ICLR 2026
Model
Abstract
Author
Caption
Equation
Figure
Footer
V2
86.4
9.0
45.2
14.9
90.5
3.6
MAE
84.1
17.9
20.7
33.4
90.8
23.8
Dolphin
73.8
21.6
48.1
25.7
91.5
32.9
Pix2Struct
80.6
26.3
21.6
15.3
83.1
27.7
AIMv2
85.8
5.5
27.7
15.5
90.4
19.4
SigLIP 2
85.9
23.1
52.6
34.7
91.6
31.6
DAVE
87.4
35.3
54.1
91.5
17.9
Model
List
Paragraph
Reference
Section
Table
Title
DinoV2
42.2
10.3
3.2
68.5
MAE
40.8
10.2
88.8
9.2
88.1
26.5
Dolphin
68.7
11.5
88.0
9.7
84.3
50.2
Pix2Struct
33,9
13.7
87.3
5.4
62.2
64.4
AIMv2
45.8
9.8
86.2
2.8
80.3
39.0
SigLIP 2
51.8
13.2
7.5
90.9
48.8
DAVE
66.4
20.6
91.6
27
87.9
49.7
Table 9: Evaluation results DocBank by category.
or journals, ensuring high-quality source corpus. During post-processing, remove all bounding boxes with confidence scores below 0.5 to ensure accurate OCR localization. We then randomly resize, crop, and add noise prevent the model from overfitting to specific PDF size. Next, normalize all bounding box coordinates to the range [0, 999] with respect to the new size. To control task difficulty, discard bounding boxes that contain either too few words (c.g., single tokens) or excessively long paragraphs. Finally, from each remaining (category, bounding box, text) tuple, we construct diverse training examples for bbox-to-text, text-to-bbos. and bbox-te to-category tasks. The data composition is shown in Table g
B.3 WEIGINT-MEGGNG
For the weight-merging, we randomly sample 10k document and web images from self-supervised pretraining as our training data. We use the AdamW Optimizer (Loshchilov Hutter] (007)), learning rate of le-4, and batch size of 32 to train for 20 epoches. The only trainable component the newly introduced merge coefficient, while all the vision encoders remain frozen.
C CLASSIC DOCUMENT AND WEB TASKS
C.I DOCBANK
DocBank is large-scale benchmark for document layout analysis, providing token-level annota- tions that capture both textual and structural information. It is widely used to evaluate models on tasks such as text detection, segmentation, and understanding of document formatting. In our work, we formulate DocBank as Document element recognition. where the task the predict the bounding box of specific element given a document image.
Implementation. DocBank consists of 13 categories. For each category, we finetune the vision encoder with an attention pooling network on the corresponding training set, and perform evaluation on the validation set. Before training, each image processed and resized according the default preprocessor for the specific vision encoder. For each category, train with 5 epoch, a learning rate of le-5, weight decay le-4, and a batch size of 32. We use Smooth LI objective (Huber, 1964) for the training. The per-category result is shown in Table
C.2 DOCLAYNET
DocLayNet is large-scale dataset for document layout analysis that contains richly annotated page images collected from diverse sources such as scientific articles, reports, and business documents. provides pixel-level segmentation masks for broad set of layout elements, enabling fin-ggraine
19

---

Published as a conference paper at ICLR 2026
evaluation of models tasks like detection, segmentation, and structural understanding. In work, we use Doclaynet to perform semantic segmentation.
Implementation. Doclaynet consists of 11 distinct classes, and in our implementation, introduce an additional class for the area with no class label. We finetune the vision encoder along with MLP projector project the visual feature patches into segmentation mask. Then crossentropy (Shannon 1948) between the prediction and the ground truth segmentation as objective and train on the training set. We train for 2 epoch, with learning rate of 1e-5, weight decay of le-4, and batch size of 32.
A blation the generalist encoder. In this section, we conduct ablation the improvement to RICO-SCA when we incorporate the generalist encoder feature. As shown in Table I0 is clear tha the generalist encoder provide high-level semnatic features that important for classification tasks.
Model
RICO-SCA
2igLIP
93.3
DAVE-spocialized
90.7
DAVE
92.3
Table 10: Performance on the RICO-SCA benchmark.
C.3 RICO-SCA
RICO-SCA is benchmark derived from the RICO dataset, which contains large-scale mobile appli- cation user interfaces. The SCA (Semantic Component Annotation) extension enriches each screen with detailed semantic labels for UI elements such as class type, buttons, images, text fields, and navigation components. In our work,
RICO-SCA to evaluate web UI classification.
Implementation. We finetune the vision encoder together with an attention pooling network to classify the class type of given web UI image, using the provided training set. We train for epoch, learning rate le-5, a weight decay of le-4, and batch size of 32. We then evaluate the finetuned model the validation set.
D VLM AS EVALUATOR
Dataset
#Image
Description
A12D
2k
Chart understanding
DocVQA
10k
Document understanding
InfoVQA
2k
Infographic reasoning
ChartQA
17k
Chart undcrstanding
Pixmo-Doc
250k
Document understanding
MultiUT
2m
Web localization and understanding
LLaVA 665K
340k
General visual instruction data
Table 11: Dataset description for VLM training.
Here we provide the detail implementation and data composition on using VLMs evaluator for vision encoders.
Data Composition. We use set of instruction tuning data with a focus document and web understanding for the VLM training. We then use the Mind2Web training set to finetune our VLM web agentic tasks. The full data composition can be found in Table
Implementation. The training is conducted with 32 H200 GPUs. We use the same architecture described in Section B For image tilting, use tilting to split the image into size of 336x336 for AIMv2, and 384x384 for other vision encoders. The instruction tuning hyperparameter and finetuning is the same in except that Mind2Web finetuned for epoch.
20

---

Published as a conference paper at ICLR 2026
D.1 EVALUATION STANDARD ERRORS
Model
AI2D
ChartQA
DocVQA
InfoVQA
RealWorIdQA TextVQA
Ilama-Sigiip 0.88
0.99
0.55
0.85
1.80
0.64
Hama-DAVE
0.88
0.97
0.47
0.87
1.77
0,62
qwen-Siglip
0.79
0.85
0.36
0.74
1.70
0.58
qwen-DAVE
0.78
0.76
0.35
0.76
1.80
0.59
Table 12: Standard errors in vision-language benchmarks.
In this section, we provide the standard errors for most lision-language benchmarks. As shown in Table [2] the standard errors remain are reasonable, indicating that the model's performance is stable and consistent across benchmarks.
D.2 EVALUATION ON CROSS-DOMAIN DOMAIN TASKS
Model
CMMMU
DTCBench
SigLIP DAVE
21.4
29.2
26.1
38.3
Table 13: Evaluation results on CMMMU and DTCBench.
Here, we provide results on DTCCBench for Korean document understanding and CMMMU for Chi- nese visual question answering. In Table [3] DAVE's consistent improvements on these benchmarks show that can generalist to cross domain
E ABLATIONS
In this section, we provide the implementation and evaluation details of our ablation studies, along with additional ablations DAVE.
E. INTCR- VARIANCE
We randomly sample 1000 document and web images from our pretraining data conduct the analysis. For each image, we divide it into non-overlapping 16 x 16 patches and compute the per- patch standard deviation of pixel intensities after normalization. This provides a distribution of inter-patch variance that reflects the inherent local variability in different domains. As discussed in Section 5.4 document-like images exhibit consistently lower variance across patches, while web- like and natural images demonstrate broader spread. These differences highlight the difference in low-level representation, particularly structural and spatial, between natural and document/web images.
E.2 MULTI-ENCODER SETUP
Given setup consisting of SigLIP 2 and specialized encoder, the images first processed sepa- rately by their respective preprocessor. The processed images are then passed into the corresponding vision encoders obtain visual patch features. Finally, the patch features concatenated channel- wise. In scenarios where the specialized vision encoder produces different number of patch fea- tures compared SigLIP 2, we apply linear interpolation the specialized features to match the patch count.
E.3 GENERALIZATIN TO UNSEEN GENERALIST ENCODER
We denote the specialized encoder within DAVE as DAVE-spec. evaluate whether D-VE-spec adopt to unseen generalist encoder, pair it with with AIMv2 during instruction tuning. The
21

---

Published as a conference paper at ICLR 2026
Model
Doc
Web
DAVE
63.4
68.2
DAVE-sper DAVE-sppc-- AlMv2
64.5
64.8
Table 14: Evaluation results on document and web understanding.
result show in Table 14 The comparable performance with DAVE highlights the generalizability of DAVE-spec to novel vision-language and agent frameworks.
E.4
EFFECTS OF FUSING SIGLIP 2 FEATURES
In Table 15 we ablate the effect of fusing SigLIP on high-level semantic tasks.
Model
RICO-SCA
2igLIP
93.3
DAVE-specialized
90.7
DAVE
92.8
Table 15: Evaluation results on RICO-SCA.
22


Reviews:
[{"review_id": "mgPacyP09P", "reviewer": "Reviewer_gMFC", "summary": "The main contribution is a vision encoder architecture for document and web understanding evaluated on a broad array of benchmarks and reports improvements over strong baselines. The use of PlotQA, ChartQA, FinTabNet is well motivated. Key contribution is enabling data efficient training in specialized domains.\nCompared to SigLIP2 and AIMv2 in particular, the submission shows that\n1) On Document Understanding Benchmarks (OCRBenchm DocVQA, InfoVQA) the gains are meaningful and suggests that the proposed architecture provides a non trivial improvement in the document domain\n2) On General VQA the results are a bit mixed which calls for more understanding of the regressions esp wrt to SigLiP2\n\nOverall in terms of novelty and significance, compared to known encoder types and training strategies (contrastive + self-supervised for SigLIP2l autoregressive multimodal for AIMv2) the core contribution lies in demonstrating broad generalized across heterogenous benchmarks.", "questions": "[1] We need to have confidence intervals for the results esp when comparing SigLIP2 to DAVE using Qwen-2.5-7B-Instruct model in Table 1. \n[2] A little more discussion of regression on RealWorldQA and TextVQA would be useful to understand if regressions on general VQA capabilities are real\n[3] we processed 500K PDF samples from arXiv with an OCR model (Cui et al., 2025), formulating them into additional recognition\nand grounding tasks. What kinds of post processing was done on the data to ensure high quality examples?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The breadth of evaluations and ablations is a key strength.  \n[1] Thorough benchmarking on Document understanding, general VQA, and Web understanding benchmarks: We see evaluation across a wide variety of benchmarks and baseline models. Using both Qwen-2.5-7B-Instruct and Llama-3.2-3B-Instruct decoders further solidify the alignment on different text decoders\n[2] Very strong improvements on Mind2Web (Table 2)\n[3] Clean ablation setups are useful to justify the training setup. Table 4 c,d are esp interesting contribution comparing different multi encoder settings", "weaknesses": "[1] average perfomance is used across all results which might under or over estimate the performance. Having some best of N results using some confidence or majority voting would further help understand the variance of the model performance better.\n[2] there might be potential domain bias, example overfitting to synthetic chart styles in PlotQA or financial domain layouts in FinTabNet. Might have been useful to detect and discuss such biases.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pqrXanxSNn", "reviewer": "Reviewer_rHNd", "summary": "DAVE presents a compelling approach to developing a specialized vision encoder for document and web understanding, demonstrating strong empirical results. The innovative MAE reconstruction and decoder-agnostic merging scheme are notable contributions. However, the paper suffers from several weaknesses, primarily in providing insufficient experimental evidence or logical clarity to fully support certain design choices and claims. Addressing these points, particularly through more rigorous ablation studies and transparent comparative methodologies, would significantly strengthen the paper's impact and credibility.", "questions": "see Weaknesses.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper proposes a modification to the standard MAE objective by reconstructing raw pixel values directly, rather than normalized pixel values. This aims to stabilize training, particularly for document and web images which exhibit low inter-patch variance. While further empirical evidence directly linking this change to stability would strengthen the claim, the approach itself is a thoughtful adaptation to domain-specific data characteristics.\n2.  A significant strength is the introduction of a distillation-based model merging scheme. This approach trains multiple encoder instances, each aligned with a different text decoder, and then learns optimal coefficients to combine their weights into a single, decoder-agnostic encoder. This method effectively addresses the challenge of encoder overfitting to specific decoders and enhances compatibility across diverse VLM architectures.\n3.  The authors leverage a vast amount of data for both self-supervised and supervised pretraining, including large-scale unlabeled document/web images and curated annotated datasets. This extensive data utilization, combined with the proposed training strategies, leads to DAVE achieving impressive performance across a wide range of document, general VQA, and web agent benchmarks, often outperforming strong baselines like SigLIP 2.", "weaknesses": "1.  The core hypoththesis of this paper is that current VLM vision encoders \"lack the robust structural and spatial information essential for document understanding and web agents.\" However, this assertion is made without references or experimental evidence. Given the existence of numerous VLMs works for document understanding, a more detailed explanation is needed to clarify why these models specifically lack the required powerful structural and spatial information. The observed performance gains of DAVE, while compelling, could also be attributed to its extensive and specialized training data rather than a fundamental architectural superiority in capturing these specific features, necessitating a more rigorous analysis.\n2.  For document and web images, the authors argue that low inter-patch variance leads to training instability, and therefore they reconstruct from raw pixel values rather than normalized pixel values. However, the authors only show a difference in inter-patch variance distribution between ImageNet-like, document-like, and web-like images, without providing experimental evidence which directly demonstrating that this variance difference leads to training instability with the standard MAE reconstruction method. \n3.  DAVE combines a specialized encoder ($\\phi_{spec}$) for low-level structural features with a generalist encoder ($\\phi_{gen}$) for high-level semantics. However, if we  fine-tuning a strong, generalist encoder like SigLIP 2 (with DAVE's training protocol and data) would yield comparable results? \n4.   When comparing DAVE with other models like SigLIP 2, it's unclear whether the original decoder in SigLIP2 was replaced by the same LLM  decoder used in DAVE? \n5. In Table 3, DAVE’s performance on RICO-SCA is slightly lower than that of SigLIP 2. The authors explain this by stating that the task emphasizes semantic understanding, which DAVE was not designed to tackle. However, this explanation directly contradicts the motivation claimed by the authors for integrating SigLIP 2 into DAVE, namely to provide high-level semantic capabilities for better document understanding. This inconsistency needs to be addressed.\n6. Figure 2, which illustrates the incremental impact of design choices, uses terms like \"Scratch Decoder\" and \"Pretrained Decoder\" without clear definitions. Furthermore, some data points in Figure 2 can not be found in the tables in the paper, making it difficult for readers to fully understand and verify the ablation results. \n7. In Table 4c, when comparing DAVE with multi-encoder setups (e.g., SigLIP 2 + DiT, PS, DP), it is not explicitly stated whether these models underwent the same extensive training pipeline as DAVE — including self-supervised pretraining, multi-decoder supervised training, and model merging. If these pre-trained models were merely concatenated and directly evaluated on the benchmarks without following the same training procedure, the comparison would be unfair. \n8. The comparison in Table 4d between DAVE and a fine-tuned SigLIP 2 is potentially unfair. The fine-tuned SigLIP 2 is described as being trained \"only on the supervised data,\" implying it did not undergo DAVE's full two-stage pipeline, including self-supervised pretraining and multi-decoder merging.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TzgZbf4858", "reviewer": "Reviewer_Q3WH", "summary": "The paper presents DAVE, a vision encoder tailored for document understanding and web agent tasks within vision-language models. It introduces a two-stage pretraining framework combining self-supervised masked autoencoding on large unlabeled datasets with a supervised autoregressive stage for tasks such as OCR, layout prediction, and web grounding. To enhance generalization across architectures, the authors employ a weight-space merging scheme that combines encoders trained with different text decoders, producing a decoder-agnostic encoder, and an ensemble training strategy that fuses general semantic features from SigLIP2 with DAVE’s specialized structural features.", "questions": "There are a few typos in the paper:\n* Line 240: \"Appendix X\"\n* Line 268: It should be \"Qwen2.5-7B-Instruct.\"", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The paper is clearly written and easy to follow.\n* The research question of vision encoder for document images is important.", "weaknesses": "* The contribution of the paper is limited. The training methods employed in the paper including self-supervised pre-training, model merging are well-established ideas such as MAE self-supervised learning [1], model soups [2].\n\n* Although the experiments encompass a wide range of benchmarks (e.g., DocVQA, ChartQA, Mind2Web), the paper provides limited analysis of cross-domain generalization—such as performance on non-English documents—as well as robustness and interpretability aspects.\n\n[1] Masked Autoencoders Are Scalable Vision Learners.\n\n[2] Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CEGJ1CZx7k", "reviewer": "Reviewer_zSTT", "summary": "This paper introduces DAVE, a specialized vision encoder for VLMs targeting document understanding and web agent tasks. The authors propose a two-stage training approach: (1) self-supervised pretraining using MAE on unlabeled document and web images, and (2) supervised autoregressive pretraining with limited annotated data. Key innovations include a weight-merging scheme to create decoder-agnostic encoders and ensemble training to combine general visual features with domain-specific representations. Experiments demonstrate improvements over existing encoders on document, web, and agent benchmarks.", "questions": "1. How sensitive is the approach to the choice of generalist encoder? Would similar improvements be seen with other encoders?\n\n2. More visual encoders should be compared, such as [1][2][3].\n[1] Web-DINO: Scaling Visual Representation Learning\n[2] A Token-level Text Image Foundation Model for Document Understanding\n[3] RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models\n\n3. A similar idea to Vary, which uses two encoders to represent the text scenes.", "limitations": "", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The focus on document and web understanding addresses a real limitation of current VLMs.\nTesting across multiple benchmarks (document tasks, VQA, web agent tasks) demonstrates broad applicability.", "weaknesses": "The paper lacks deeper analysis of why this specific combination of techniques works. Why is ensemble training with SigLIP2 optimal? What are the theoretical guarantees of the weight merging approach?\n\nWhile the combination is new, individual components are well-established, such as MIM pretraining and Visual-language alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)