UNIT: Unifying Image and Text Recognition in One Vision Encoder
Authors: Yi Zhu
Zhou Yanpeng
Chunwei Wang
Yang Cao
Jianhua Han
Lu Hou
Hang Xu
Venue: NeurIPS 2024 poster (2024)
Keywords: large-scale vision-language model
vision foundation model
Primary area: deep_learning_architectures

Abstract: Currently, vision encoder models like Vision Transformers (ViTs) typically excel at image recognition tasks but cannot simultaneously support text recognition like human visual recognition. To address this limitation, we propose UNIT, a novel training framework aimed at UNifying Image and Text recognition within a single model. Starting with a vision encoder pre-trained with image recognition tasks, UNIT introduces a lightweight language decoder for predicting text outputs and a lightweight vision decoder to prevent catastrophic forgetting of the original image encoding capabilities. The training process comprises two stages: intra-scale pretraining and inter-scale finetuning. During intra-scale pretraining, UNIT learns unified representations from multi-scale inputs, where images and documents are at their commonly used resolution, to enable fundamental recognition capability. In the inter-scale finetuning stage, the model introduces scale-exchanged data, featuring images and documents at resolutions different from the most commonly used ones, to enhance its scale robustness. Notably, UNIT retains the original vision encoder architecture, making it cost-free in terms of inference and deployment. Experiments across multiple benchmarks confirm that our method significantly outperforms existing methods on document-related tasks (e.g., OCR and DocQA) while maintaining the performances on natural images, demonstrating its ability to substantially enhance text recognition without compromising its core image recognition capabilities.

UNIT: Unifying Image and Text Recognition in One Vision Encoder
Yi Zhu', Yanpeng Zhou', Chunwei Wang', Yang Cao", Jianhua Han', Lu Hou', Hang Xu' HHawwii Noah's Ark Lab, "Hong Kong University of Science and Technology
com/yeezhu/UNIT
Abstract
Currently, vision encoder models like Vision Transformers (ViTs) typically excel image recognition tasks but cannot simultaneously support recognition like human visual recognition. To address this limitation, we propose UNIT, a novel training framework aimed at UNifying Image and Text recognition within a single model. Starting with a vision encoder pre-trained with image recognition tasks, UNIT introduces lightweight language decoder for predicting text outputs and a lightweight vision decoder to prevent catastrophic forgetting of the original image encoding capabilities. The training process comprises two stages: intra-scale pretraining and inter-scale finetuning. During intra-scale pretraining, UNIT learns unified representations from multi-scale inputs, where images and documents are at their commonly used resolution, enable fundamental recognition capability. the inter-scale finetuning stage, the model introduces ccale-exchanged data, featuring images and documents at resolutions different from the most commonly used ones, to enhance its scale robustness. Notably, UNIT retains the original vision encoder architecture, making it cost-free in terms of inference and deployment. Experiments across multiple benchmarks confirm that our method significantly outperforms existing methods document-related tasks (e.g., OCR and DocQA) while maintaining the performances on natural images, demonstrating its ability substantially enhance recognition without compromising its core image recognition capabilities.
Introduction
are crucial for extracting high-level level visual features. thereby enhancing performance across range of downstream tasks [9] 8 [44 They play vital role integrating visual information into intelligent applications, driving advancements both computer vision and artificial intelligence. In recent years, Vision Transformers (biTs) encoder models [6 66 27 58 71 have revolutionized the field of computer vision. VIT models pretrained on image classification tasks are extensively used as backbones for wide array of image recognition tasks and achieves state-of-the-ar performances. Another line of remarkable ViT models are trained via image-text cross-modal contrastive learning [43
Vision encoder models 6
These models are often used as plug-in vision encoders in Large-scale Vision-Language Models (LVLMs), which have emerged as versatile tools with the potential to revolutionize various domains, including healthcare diagnostics, autonomous driving, digital assistants, and advanced content analysis media and entertainment. Despite these advancements, existing vision encoder models, exhibit significant limitation: they have not yet demonstrated the capability to simultaneously support both image and text recognition like humans do. Such ability is essential for document analysis applications, for instance, model must accurately recognize and interpret textual information embedded within complex layouts, such tables, graphs, and mixed media.
64
52 67
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Text recognition [50 49 12
particularly in the context of dense documents and complex visual environments, presents unique challenges that are distinct from those encountered in pure image recognition. While image recognition typically focuses on global feature extraction and classification, text recognition requires precise local feature extraction and sequence prediction. ViT', though adept at handling global image features, often struggle with the detailed local features needed for accurate text recognition, particularly in high-resolution document images where fine details are crucial. A straightforward solution is to finetune pre-trained ViTs using high-resolution documents. However, this approach requires interpolating the pre-trained positional embeddings to handle longer sequences. Such interpolation with a large magnification will disrupt the alignment between the original positional embeddings and their corresponding spatial positions, thereby impacting the model's 's performance.
Existing methods 39 build doeument-specific models by retraining ViT's exclusively on the Optical Character Recognition (OCR) task, thereby discarding the original image encoding capability, In downstream applications, these models are often ensembled with other expert models, requiring users to specify the data type in advance, which is inadequate for scenarios requiring dynamic and simultaneous recognition both image and text without prior knowledge of the input content. Other use similar model architectures trained independently on image and text recognition tasks. Inputs are fed into both models, and the output features are concatenated to form unified representation. However, this approach will result in significant increase in computational cost. Additionally, some LVLM methods [62 63 enhance document analysis tasks (e.g., DocQA) in OCR-free manner by finetuning with document- inntrruttiontttoionnn data. Nevertheless, their capabilities are limited to handling images with prominent text and fail with dense documents, struggling generalize across different font sizes, typefaces, and backgrounds.
ensemble- based methods [ 54
In this paper, we propose UNIT, novel training framework for UNifying Image and Text recognition abilities within a single model. UNIT upgrades an existing Vision Transformer (ViT) model to effectively integrate text recognition. First, a lightweight language decoder (e.g., OPT-125M) is introduced to decode the learned visual features into text sequences in an auto-regressive manner, enabling the encoder model to capture fine-detailed shape and sequential information necessary for text recognition. Then, tiny vision decoder (c.g., two MLP layers) is introduced to reconstruct the visual features of the original vision encoder from the newly learned features, preventing catastrophic forgetting of the model's fundamental encoding ability for natural images. The training pipeline involves two stages. The first is the intra-scale pretraining stage, where the model takes images and documents at their commonly used resolutions inputs, specifically low-resolution images and high-resolution documents (x4 times the low-resolution). During this stage, the model is optimized with three objectives: OCR for document inputs, feature reconstruction of the original ViT encoder's features and image captioning for natural image inputs. The second stage is inter-scale finetuning stage, where the model is trained under a cealenexchanged setting with hisouresonution images and low-resolution documents, enhancing its scale robustness for both image and text recognition. It is important to note that UNIT retains the original vision encoder architecture, ensuring that there is no increase in inference cost.
Extensive experiments show that UNIT significantly outperforms document-specific models OCR tasks while maintaining core image recognition capabilities. When integrated into LVLMs, it also benefits downstream document analysis tasks without degrading image understanding. This demonstrates that UNIT effectively unifies image and text recognition abilities.
The main contributions of this paper are summarized as follows:
We propose UNIT, novel framework that unifies image and text recognition in a single vision encoder through joint training of multi-tasks. The model retains the original vision encoder architecture, ensuring cost-free deployment and no increase in inference cost.
The training paradigm comprises two key stages: an intra-scale pretraining stage, where images and documents are trained at their commonly used resolutions, and an inter-scale finetuning stage, where their resolutions are exchanged to enhance scale robustness.
Extensive experiments demonstrate that UNIT significantly enhances performance on vi- sion tasks requiring text recognition compared to their counterparts, while preserving the fundamental encoding ability on natural images.

---

2 Related Work
Vision Transformer for Text Recognition. In recent work, the application of Vision Transformers (ViT) to document reading has gained traction due to their success in image recognition [30 [0] 24
[1]. Nougat [Sleevereees ViT model for Optical Character Recognition (OCR) tasks, focusing on converting human-readable scientific documents into a maahine-rradaale markup language. Similarly, KOSMOS-2.5 [9 employs ViT-based vision encoder coupled with YYansformer-basel language decoder, aiming to serve as a versatile tool for diverse text-intensive image understanding tasks through supervised fine-tuning. The Donut model [23]. while introducing OCR-free transformer for visual document understanding, may encounter challenges in generalizing to unfamiliar document types. Despite their demonstrated efficacy in specialized OCR scenarios, these models may not fully harness the original image encoding capabilities present in pre-trained ViT architectures. This limitation can restrict their applicability text-only images.
LVLMs for Document Analysis. Several recent studies have explored OCR-free iisualsituaaed language understanding using Large Vision-Language Models (LVLMs) 47 55 59 where doccmmnntiinstrucee
2
77 40
28
data is used to fine-tune LLMs without adapting the vision encoder [62 60 61. These methods often rely a pretrained Vision Transformer (ViT) a fixed vision encoder, which may present several challenges: 1) The frozen ViT may limit text recognition capabilities, especially when documents feature varied fonts, styles, low image quality not seen during its pretraining. 2) The model's reliance on a constant image resolution can lead to inaccuracies in text recognition for high-resolution documents those with noise and distortion. Additionally, several recent works [53 544 have successfully enhanced LVLMs with OCR-like abilities, by retraining document-specific Vision Transformer (ViT) and then concatenating its output visual features with those from a pretrained ViT model. However, this method comes with trade-offs, notably an increase in computational expense and significant underutilization of the model's capacity, leading to inefficiencies. In contrast, our UNIT integrates both capabilities within a single model, offering greater efficiency, fost-free deployment, and no increase in inference time.
Multi-scale Vision Transformer. Vision Transformer models are commonly pre-trained to process images at fixed resolution, such as 224 or 336 pixels, which is commonly used for many image understanding tasks [17 20 33 26 25 However, these resolutions might be insufficient for precisely discerning tightly packed text in higher-resolution documents. Consequently, multi-scale training approach for Vision Transformers is crucial for effectively managing both image and text recognition challenges. The progression of Vision Transformer architectures has embraced a notable trend toward multi-scale modeling. This paradigm enables the models to capture and process visual information across different scales, enhancing their capability to understand complex scenes and texts. Some methods (29 51 56 integrate a hierarchical pyramid structure into the original Vision Transformer architectures, which continue to operate at constant resolution. In contrast to these methods, our proposed UNIT model breaks the limitations of fixed-resolution inputs. UNIT is designed to handle multi-scale training for multi-tasks, yielding a unified representation with scale robustness for both images and documents.
3 Methodology
3.1 Preliminaries
Backbone. We employ the Vision Transformer (ViT) [66 architecture as our backbone. Let fo(-) be the vision encoder with parameters 0, for an input image I E Hxxxx3 where H and W represent the image height and width, respectively. ViT model first partitions each image into fixed-size of patches in p P pixels, and then encode these patches into hidden features of dimension d. Subsequently. the features are added with their corresponding position embeddings and fed into multiple layers of stacked Transformer blocks, interacting with each other via attention mechanisms. Finally the model outputs visual tokens X E BB xd as:
X X= - {x
where N- N₂ + N₂ denotes the total number of visual tokens, comprising the spatial tokens x W and the number of [CLS] tokens N..
N, H. p
3

---

High resolution
Queries
Visual Tokens
Unified Vision Encoder
Input Embedding Layer
Instructions
Low resolution
Document Images
Vision Decoder
Language Decoder
Low resolution
Original +
Teacher Tokens Student Tokens
Ground-Truth Sentences
OCR results
Vision Encoder
8
Captions
Natural Images
Reconstruction Loss
Cross-Entropy Loss
Figure I: Overview of UNIT Architecture. The model processes high- reghluesotution documents and low- resolution images, generating a set of visual tokens. These tokens pass through an input embedding layer, with document tokens fed into the language decoder to predict text sequences, enhancing the model's text recognition capability. Simultaneously, preserve the model's original image encoding ability, the visual tokens from natural images are reconstructed via a lightweight vision decoder, mimicking the output of the teacher model. Additionally, an image captioning task is included alongside the OCR task to further enhance image understanding.
Architecture. UNIT aims to unify image and text recognition capabilities within a single ViT model, as illustrated in Fig. [ This is achieved through the introduction of lightweight language decoder (e.g., OPT-125M) for document-level OCR, enabling the ViT model to recognize text. Additionally, preserve the original image recognition capabilities of the vision encoder, UNIT incorporates vision decoder responsible for token-wise feature reconstruction from the original ViT model. To further enhance image understanding, an image captioning task is integrated alongside the OCR task. UNIT is trained in multi-scale setting, where images and texts are processed at their commonly used resolutions during pretraining and then finetuned at other resolutions.
3.2 Text Recognition Ability Enhancement
Multi-Scale Inputs. In contrast image recognition that focuses on global feature extraction and classification, text recognition requires fine local feature extraction and sequence prediction. ViTs are excellent global feature extraction but may not perform well in text recognition. The reason is that ViTs use a learned position embedding for each input patch in an image, which in turn forces that the model always operate constant resolution, typically chosen as 224, 256 or 336. At these resolutions, the generated visual features may lose critical information, leading to difficulties in recognizing characters from dense texts such as PDF documents, websites, or digital books. Naively cropping nigh-resolution document images into low-resolution inputs would break the sequential order of characters and may impair the recognition of letters at the cutting boundaries. To prevent these issues, we introduce the Cropped Position Embedding (CPE) [22] augmentation, which interpolates the original position embeddings to match the number of positions of the maximum input size. For www-resouttion images, the position embeddings are then randomly cropped and interpolated to match the number of input patches for the original model.
Language Decoder. Conventional methods often employ CLIP-style contrastive learning to align images with their language annotations. However, this approach presents notable drawbacks in our context. Firstly, contrastive learning relies on a CLIP pretrained text encoder to generate language embeddings. The maximum sequence length of 77 tokens insufficient for text recognition annotations, especially in dense documents. Secondly, such training paradigm is inadequate for the vision encoder accurately capture each word, leading to undesirable information loss during the encoding process. Taking the above factors into account, we introduce a lightweight Transformer decoder, e.g., OPT- 125M [69], efficiently predict the language sequences presented in the input documents. Similar to LVLMs, we employ an input embedding layer to project the visual features from the vision encoder into the embedding space of the language decoder. Here we initialize the input embedding layer with a two-layer Q-Former (25 with K learnable query tokens Q {qx _ qe E RRd The visual tokens = [ generated from the vision encoder are first fed into the input embedding layer and then the language decoder. We denote this process f-(-) and

---

denotes the parameters of the input embedding layer and the language decoder:
(z1.-.,, nN) = f<((x),..., XN): {qu,-,, 9K })-
(2)
The decoder uses the last hidden state z E I2 at time step to predict language sequence in the form of text tokens y = {ū1, 02...., ûr) via the language decoder in an auto-regressive manner. The cross-entropy loss for training the auto-regressive Transformer decoder is defined follows:
C 8)
E log Pge - y0lv
(3)
where y = {31, ,.,,.., 3 represents the target sequence, and the probability P(t yelyia-11 z obtained from the softmax output of the final layer of the decoder.
3.3 Image Recognition Ability Preservation
Vision Decoder. Finetuning the vision encoder exclusively on document datasets would result in severe catastrophic forgetting of its original image encoding capabilities. To mitigate this, we introduce an token-wise feature reconstruction task on natural image datasets, ensuring that the newly learned vision encoder retains its ability to encode visual concepts. We incorporate visual decoder (-) consisting of two fully connected layers with a GeLU activation function in the intermediate layer. The decoder processes each visual token independently, preserving the positional information inherent in each token more effectively. We utilize the original pretrained ViT model as teacher model to provide original features for each natural image, providing supervision signals for visual feature reconstruction. Denoting the teacher tokens as X { x E Rd we enforce the alignment of the new learned student tokens with the original ones using weighted sum of the cosine distance Leos and smooth LI loss L11- This approach ensures consistency in both vector direction and magnitude between the new and original features, as:
Vvi
Σ iEC
8.) + plln (Jx()),,
where C represents a subset of features randomly sampled from the X, to mitigate overfitting and enhance robustness during training. f denotes the loss weight sealar.
Language Decoder. Furthermore, we incorporate an image captioning task alongside the text recognition task to ensure that the learned visual tokens can be effectively projected into the language space, thereby enhancing image understanding ability. The forward process and loss formulation for image inputs are the same as those for document inputs.
3.4 Intra-Scale Pretraining
As described in Sec. 3.2 and Sec. 3.3 UNIT tackles images and documents at different input scales with different optimization objectives. In this section, we introduce the intra-scale pretraining stage (see Fig. 2 h) where image and text recognition are trained at fixed scale, namely, low resolution for images and high resolution for documents, without considering variations in scale.
Dataset Preparation. Let be curated dataset with 5M samples, where D D' U DJ Here, D/1 represents dataset of natural images annotated with coarse captions (less than words). The images are resized to x times the original scale r, primarily sourced from the Conceptual Caption dataset [46). comprising 3M samples. Mcanwhile, D represents a dataset of documents annotated with dense OCR data (more than 500 words). The document images are resized x4 times the original scale r, sourced from our synthetic dataset of English PDF documents, comprising 2M samples.
Instruction Prompts. The instruction prompts follow the format used in popular LVLMs 770 25 232. where image tokens are prefixed with text tokens. Specifically, we use two special tokens "<img>" and "</img>", to indicate the start and the end position of the image tokens, followed by instructions that indicate task requirements. For natural images, the prompt is set as "Give caption of this image:" and the LLM outputs language sequence summarizing the content of the image. For document images, the prompt is set as "Read the text in this image:" and the L.LM outputs sentences row by as they appear in the document in the order of reading.
5

---

Intra-scale Pretraining
(b) Inter-scale Finetuning
(c) Application LVLMS
Imagees
Imagee
Caprion
LakN
Image Captioning
Coarse Cuption
Detailed Caption
Visual Question
UNIT
OPT-TAI
895
UNTT
Vicuma-1
Answer
Documentts
Documents
Documents
Document
Long
Long
Analyais
Chart Understanding
Short Text
89%
896
Illustration of the UNIT training paradigm. The (a) intra-scale pretraining stage processes images and documents at their commonly used resolutions to integrate basic text recognition with existing image recognition capabilities. The (b) inter-scale finetuning stage processes scale-exchanged data and tasks to enhance scale robustness, benefiting downstream document analysis tasks when integrated into (c) LVLMs applications.
Figure 2:
Joint Optimization. We train all parameters, including the VIT backbone, vision decoder and language decoder, using images and documents annotated with language sentences. During training, the model is optimized by the cross-entropy loss Lian between language outputs and annotations, and also the feature reconstruction loss Lvis force the newly learned visual features to be similar with the original ones. The UNIT model is updated to minimize the total loss:
arg min Lian ((D' UD + ALvis((D
(5)
3.5 Inter-Scale Finetuning
After the intra-scale pretraining stage, UNIT is capable of processing both image and text recognition at their commonly used resolutions, namely, low-resolution images and high-resolution documents. However, observed that this model lacks scale robustness when handling texts with larger fonts and images with larger dimensions, significantly limiting its generalization ability in various vision tasks. Naively add the scale- ceale-exchanged datasets in the intra-scale pretraining process may slow down the convergence of the model and making it prone to local optima. To address this, we further conduct inter-scale finetuning (see Fig, 2)), using high-resolution images and low-resolution documents.
Dataset Preparation. Here we build another scale exchanged dataset D ' DJ with 1M samples for the inter-scale finetuning. We conduct dataset D comprising natural images annotated with detailed captions (each containing over 100 words). These images are resized to x4 times the original scale r, primarily drawn from the ShareGPT4V dataset [[], totaling 1M samples. Additionally, create dataset D comprising IM documents with an average font size x4 times larger than that of DTT The document images resized xr times, sourced from synthetic dataset which includes cropped pages from digital books, advertisements, websites, and other sources containing short yet large font texts. We adopt the instruction prompt setting same the intra-scale pretraining stage.
Random Feature Sampling. When dealing with natural images higher resolutions, the resulting increase in visual tokens compared to lower resolutions can lead to gradient overflow during the training process, primarily due to the token-wise feature reconstruction loss. To address this challenge, we propose to randomly sample a set of visual tokens equal to the number in the low-resolution inputs and discarding other tokens.
Joint Optimization. This stage aims to enhance the scale-robuss recognition ability for both image and text. We retain half of the data at the previous stage and then introduce the scale-exchanged data. The model is updated to minimize the total loss:
arg min Liau ((D) UDD, U D (.a))) ALriall, 2 DK Dkal))
(6)
2
4 Experiments
Implementation Details. We select OpenCLIP VIT-H 43 with 32 layers and a hidden size of 1280 our vision encoder. We choose OPT-125M 68 model with hidden size of 768 as the language

---

Fitaition Poley and Sporti Laker conroaag on wiker cmiiing comreetoo reppe segarme tacaste Roua sakker protd
TAL
wi ln Brodramelle déda aelitt
bonele conaleee of the de basares taagaaee maai and plass of tiaaeo topp poomeee Mortowems Voer same atern oepferee
J.515 PROMO
17500
hotwanng wir wie walke
capio deind, and the slari
TQTA AL C.
01000
Apcos trom
lover sonkis
undenien
lis finst lnow pal for the
ogloud budl atsider and milere prplecisiy
and segiyyenn hapcle conetaiigg the plac and seradig labes k dust radkuie Wickkke Schoo, the Schod in Sena do do plalllllyyy the beac Repurijaa a Woodhhp Marpoo PSO logle aub- 206 19c5--5
wikwing troe in regioes ailee
Dejpeb rapoovid the clils gool socnd smn-terniiumeee who wes rakking his season dehett and the chill ssccend mad. screel Miraee Plani, find god
Teae Woctshop
1ST PROMO 1750 4000) 37500
TOTAL CASH53000 90000
Hepurt
Figure 3: Visualization examples of text recognition. UNIT predicts accurate OCR results even across diverse scenarios, e.g., handwritten texts, receipts, and interleaved image-text documents. Please see clearly by zooming in. More promising examples are shown in the supplementary material.
Method
Backbone
#Params
Input
FUNSD
SROIE
CORD
SYN-L-val
MD-val
Donut 22
Swin-B
260M
1280 960
9.08
8.94
16.64
44.78
5.077
Nougat S
SIT-V
247M
896 x 672
55.35
33.64
1.57
66.76
86.71
Vary" 3
SAM-ViT-B
525M
1024x 1024
21.01
9.84
12.89
91.20
59.30
RADIO 44
VIT-H/14
632M
896x886
26.12
10.42
10.01
93.90
37.57
UNIT (ours)
VIT-H/14
632M
896x896
67.14
41,48
58.87
95.33
78.50
Table I: Comparison with ViT-based models for text recognition ability. The presented numbers are Fl values. An asterisk (") indicates reimplemented results on our document datasets.
decoder in the two training stages. Our optimization strategy involves Adam 32 with a weight decay 0.01. The initial learning rate to 5e-5, and changes with a cosine learning rate decay scheduler. The warmup ratio is set to 0.03, and the global batch size 256. We set loss weights = 0.2. These settings are shared for both two training stages. Our model can be trained with any resolution below the maximum limit. We chose two primary resolutions for training to ensure consistent tensor sizes, simplifying batch processing and optimizing resource use. UNIT is trained on these two resolutions and tested on different ones. To ensure efficient data loading and processing during training, preprocessed the document images by padding them to square shapes. During inference, we maintain the original aspect ratio of the input images.
2 and
4.1 Evaluation Benchmarks
We evaluate our model on three public OCR datasets: FUNSD (50 images). SROIE (347 images), and CORD (100 images). reporting the Fl score. Given the lack of large blocks of dense text in these datasets, we create two additional OCR evaluation datasets using English PDF files from arXiv. The first, SYN-L-val (200 images), consists of PDFs with small font sizes at approximately 1k resolution. The second, SYN-S-val (200 images), includes images cropped from these PDFs, resized to a lower resolution (e.g., 224), resulting in larger font sizes. 2) Markdown conversion: The task requires structured text output from images of documents, capturing the stylistic and structural elements of the text in markdown format. We add 1M markdown data (collected following Nougat (5) into the inter-scale finetuning stage, endowing the ability of markdown conversion of our model. Due to the lack of cleaned markdown conversion validation set, MD-val, we conduct dataset containing 82 images with markdown format annotations.
Text Recognition: 1) Document-level OCR:
Image Recognition: 1) Zero-shot classification: We calculate the feature similarity between the [CLS] token of the visual encoder and the text features extracted from the corresponding CLIP text encoder. The evaluation is performed on the ImageNet- IK dataset, and top-1 accuracy is reported. 2) k-NN classification: We first compute the [CLS] token of the visual encoder for the training images of ImageNet- IK, and then for each validation image, we utilize weighted sum of the k nearest training vectors to select a label. 3) Semantic Segmentation: we append a linear head onto the vision encoder and train it with the encoder frozen. The Adam W optimizer is used with a learning rate of 10-3, The segmentation mloU (%) is computed as in the MMSeg framework
[ on the
ADE20k datasct. For both training and evaluation, we use input size of 512 x 512.
7

---

Method
#Param.
7S cls.
KNN cls.
Segm.
OCR SYN-I.-val
Method
7S cls.
EfficientViT-L.. 7
38M
71.73
79.90
33.12
Prec.
Rec.
Fl
SwinV2-S 29
49M
74.70
81.12
35.57
w/o Image captioning
ConyNext-B
31
88M
75.43
81.73
38.95
N/A
93.03
90.63
91.81
MViTV2-B 27
51M
75.92
81.39
41.39
À =
74.20
92.18
89,43
90.78
NFNet-F3 66]
254M
76.93
80.50
38.31
A= =
76.24
94.13
91.72
92.91
MaxViT-B [88
119M
77.49
79.34
38.46
OpenCI.IP-H/14 43
632M
77.19
81.10
40.04
with Image captioning
RADIO-L/14 45
304M
77.25
84.03
48.70
A=
N/A
95.17
93.39
94.26
E-RADIO-L/14 44
265M
77.87
83.73
45.50
A =
75.16
92.56
89.26
90.88
RADIO-H/14
632M
78.62
84.17
49.01
=
78.54
95.45
93.72
94.57
UNIT (ours)
632M
78.76
84.18
50.19
Table 3: Ablation of the feature reconstruction
Table 2: Comparison of image recognition capabilitiess loss weight À with both the image captioning task during the intra-scale pretraining stage.
with existing vision encoders.
Commonly-used Resolution
Exchanged Resolution
Method
OCR Prec
OCR Rec.
OCR F1
7S cls.
OCR Prec
OCR Rec.
OCR
w/o inter-scale
78.54
95.45
93.72
94.57
0.40
51.79
28.53
34,99
w/o feat. sample
78.02
93.96
86.63
90.14
31.66
96.23
89.93
92.40
UNIT (ours) 78.76
96.67
94.03
95.33
80.49
96.35
90.61
92.68
Table 4: Ablation of the inter-scale finetuning stage and the random feature sampling strategy.
Downstream Vision Tasks:
We replace the vision encoder in the LLava- 1.5
28 setting with our
own encoder. We first pretrain the input embedding layers with the vision encoder and the LLM frozen, then conduct instruction tuning to finetune a Vicuna-7B model 442. The learning rate of stage 3 is set to 5e-4 and batch size of 512. We evaluate the trained models
general visual question answering (VQA) datasets including VQAv2 [17. GQA [20), and OK VKVQA [3], and also document
comprehension datasets including ChartQA 34 DocQA (36), and InfoVQA [35]
4.2 Comparison with Existing Approaches
Text Recognition. We compare UNIT with two ViT-based OCR expert model, including Donut 23 and Nougat [ and two open-source vision encoders that support high-resolution inputs, including Vary [53 and RADIO [44]. We use the open-source weights of Donut-Baad and Nougat for evaluation.
As for Vary and RADIO, we retrain them on our dataset for fair comparison. The models are evaluated on public OCR dataset like FUNSD (1I) SROIE 19), CORD [19] and our synthetic datasets SYN-L-val and MD-val. As shown in Tab, I UNIT consistently outperforms these methods on OCR and markdown conversion tasks, except for Nougat, exhaustively trained on markdown data. This shows UNIT's superior ability to encode fine-detailed sequential and spatial information in documents, resulting in highly accurate OCR outcomes, Fig. B] Based on the fundamental text recognition ability, UNIT can be further finetuned with a small amount of data to adapt to highly specialized tasks. such text recognition with grounding (in both natural images and pure texts), markdown conversion and Chinese OCR. Please refer to the supplementary for more details.
Image Recognition. To validate the image encoding ability of UNIT, we compare it with existing efficient vision encoders on zero-shot and kNN classification on ImageNet- 1k, as well linear prob semantic segmentation benchmark. In Tab. 2 for fair comparision, all models share the same teacher knowledge, derived from DINOv2 and OpenCLIP-H. The compared models directly use these two models for training feature reconstruction, and UNIT employs one of the best-performing students, RADIO-H. HH, as the teacher encoder. From the results shown in Tab. 2 the best performance among all these models, demonstrating its ability to effectively preserve the fundamental image encoding capabilities.
we observe that UNIT achieves
4.3 Ablation Studies
About intra-scale pretraining. As shown in Table 3 removing the additional image captioning task described in Sec. 3.3 leads to a significant drop in performance zero-shot classification (from 78.54% to 76.24%). This demonstrates that the image captioning task can further enhance image recognition capabilities. Additionally, ablate the feature reconstruction loss weight A, finding
8

---

GLOBAL MOBILE SEARCH SHARE
12 What is the "actial" value per 1000, during the year 19707 0.2.2
Nora AVEESICA
Minel
Q: What the number of procenggr movemunts
Hare
@ When 'sdhisory board meeting' scheculed! Oetbber
(in thousands) Dover ports in the UK?
Q: From which country in the Middle East did 25% of searches comn? A: saudi anbiia
A: 10901
Figure 4: Visualization examples of downstream document analysis tasks. UNIT accurately recog- nizes tiny words and digits, providing correct answers for recament-related questions from users.
Benchmark
224
336 x 336
448 x 448
672 672
896 x 896
Zero-shot classification
78.8
81.3
81.7
81.3
80.5
OCR Fl
92.7
93.7
93.2
93.9
95.3
Table 5: Ablation studies on different input resolutions during inference.
the best performance when À 2. When the feature reconstruction loss À 0, the zero-shot classification failed. This is reasonable because zero-shot classification evaluates the aligmment between the vision and text encoders. If X the vision encoder cannot ensure proper alignment with the text encoder of OpenCLIP-H, leading to the failure of zero-shot classification.
About inter-scale finetuning. As shown in Table 4 "w/o inter-scale" represents the model trained without inter-scale finetuning. Commonly-used Resolution indicates that the input sizes are at the common resolution of the evaluated tasks, e.g., 224 x 224 for zero-shot classification and 896 x 896 for document OCR (SYN-L-val). Exchanged Resolution indicates 896 x 896 for zero-shot classification and 224 x 224 for document OCR (SYN-S-vall We can that after inter-scale finetuning, UNIT demonstrates enhanced scale robustness on the exchanged resolution for both image and documents. "w/o feat. sample" represents the model trained without random feature sampling mentioned in Sec. 3.5 This setting causes a dramatic performance drop on zero-shot classification on at exchanged resolution, showing the effectiveness of random feature sampling.
Different resolutions. Our model supports arbitrary input sizes within the maximum limit. As shown in T our model, trained on two primary resolutions, generalizes well and maintains consistent performance in both zero-shot classification and OCR tasks.
4.4 Downstream Task Performance
Naive resolution. Vision encoders act key component in Large Vision-Language Models (LVLMs) to provide rich and detailed visual representations that enable effective cross-modal un- derstanding and reasoning. Following LL5 (28), we build LVLMs using UNIT or other vision encoders based on the Vicuna-7B LLM. The vision encoders are fixed during LVLM training. Images padded and resized one of two primary resolutions based the task requirements. A lower resolution (e.g., 224x224) is used for global understanding tasks (e.g., VQA), while a higher reso- lution (e.g., 896x896) used for tasks requiring fine-detail perception (c.g., DocQA). Each image is represented as 256 visual tokens through the input embedding layer. The LLM is finetuned with LoRA [8). The multi-modal pretraining process uses 4M image-caption data (randomly extracted from Conceptual Caption Ao] and the MAION-COCE 45) and 200k document OCR data (randomly sampled from the training set). In the Supervised Finetuning (SFT) stage, we use the LLaVA-80k or LLaVA-CC665k along with the train set of DocVQA (3) and ChartQA [34] as the fine-tuning dataset. In Tab. 6 UNIT significantly outperforms the teacher model RADIO and other compared models on the document analysis tasks, including ChartQA, DocQA and InfoVQA, while maintains comparable performance on image understanding tasks. From Fig. 3 we can see that UNIT can extract the texts in documents, charts or websites, showing its potential in real-world document analysis applications.

---

Document Analysis
Image Understanding
Method
#Param
DocQA
ChariQA
InfoVQA
VQAv2
GQA
OKVQA
CLIP-L 43
304M
22.3
23.6
25.0
72.8
61.2
55.8
DINOv2
632M
14.7
15.9
63.9
SAM-L [4
632M
13.9
15.0
51.0
Vary 53
525M
42.8
41.8
42.6
RADIO-H
44
632M
18.2
20.2
24.5
72.9
61.8
55.9
UNIT (ours)
632M
43.0
46.0
28.7
72.3
60.4
55.5
Table 6: Comparison of the performance of various vision encoders with naive resolution.
Method
CharQA
DocVQA
InfoVQA
OCRBench
GQA
OKVQA
MME
MathVist
CLIP-L
43
52,0
57.2
29.3
382
62.3
57.0
1503.6
42.7
SigLIP
56.5
62.0
29.7
429
63.0
61.1
1489.4
44.2
UNIT (ours)
61.0
65.5
31.9
480
63.9
61.5
1529.8
44.6
Table 7: Comparison of commonly used vision encoders in LVLMs with high-resolution grid slicing.
High resolution. In the naive resolution setting, each image is represented using 256 tokens. However, this may be insufficient for capturing the details of images with extensive content, such densely organized documents containing thousands of words. To address this, we evaluate UNIT Llava- Next with grid slicing strategy and compare it with This method divides each image into several grids based on their aspect ratio. For instance, with maximum of 4 grids, possible configurations include {2 2,
commonly used vision encoders in LVLMs.
x 3,2 1,3 x 1}, with each grid sized at M x M. Here, MM corresponds to the resolution used during model pretraining 336 for CLIP-L and 384 for SigLIP. And we set 448 for UNIT. After the vision encoder, we use C- Abstractor 37 as the input embedding layer. It generates 256 visual tokens for each grid, which are then concatenated create the final image representation. During the multi-modal pretraining stage, the input embedding layer is initially trained on Llava665k. Following this, the LLM and the last few layers of the vision encoder are trained using 1.5M image captioning data. the SFT stage, all parameters of the LVLM are fine-tuned using 1. 1.2M SFT data. As shown in Tab. 7 our method significantly outperforms the compared models on document-oriented QA tasks and demonstrates comparable performance other QA tasks. CLIP-L and SigLIP are initially pretrained only on natural images and subsequently fine-tuned with document images for downstream tasks. In contrast, UNIT is pretrained on both natural images and documents using fundamental recognition tasks such as image captioning and OCR. Our approach offers two key advantages: First, pretraining with fundamental recognition tasks enhances UNIT's versatility and suitability for downstream document tasks. Second, the early integration of text recognition capabilities in UNIT provides a robust initialization for LVLM training, leading more stable training and faster convergence.
5 Conclusion
We propose UNIT, simple yet effective framework that unifies image and text recognition in single vision encoder, without increasing deployment or inference cost. UNIT incorporates a lightweight language decoder for text recognition and lightweight vision decoder to preserve image recognition capabilities. Trained with multi-scale inputs, UNIT processes images and documents at their conven- tional resolutions during intra-scale retraining for basic recognition, and at exchanged resolutions during inter-scale finetuning to enhance scale robustness. Extensive experiments demonstrate that image encoding ability,
UNIT significantly outperforms document-specd speecic models, maintains and benefits downstream document analysis tasks when integrated into LVLMs, showing great potential in processing interleaved text and image information in real-world scenarios.
10

---

References
[1] J. Alayrac, Donahue, Luc, A. Micch, I. Barr, Y. Hasson, K. Lene, A. Mensch, K. Millican, M. Reynolds, R. Ring, E. Rutherford, S. Cabi. T. Han, Z. Gong. S. Samangooei. M. Monteiro, J. L. Menick, S. Borgeaud. A. Brock, A. Nematzadeh, S. Sharifzadeh, M. Binkowski. R. Barreira, O. Vinyals, A. Zisserman, and
K. Simonyan. Flamingo: visual language model for few-shot learning. In Advances in Neural Information Processing Systems (NeurlPS), 2022.
[2] Alibaba. Introducingg qwen-7b: Open foundation and buman-aligned models (of the state-of-the-arts). cot/e/oee///e//eeeee.
2023.
[3] Bai, S. Bai, Y. Chu, Z. Cui, K. Dang, X. Deng, Y. Fan, W. Ge, Y. Han, F. Huang, B. Hui, L.. Ji, M. Li,
J. Lin, R. Lin, D. Liu, G. Liu, C. Lu, K. Lu, J. Ma, R. Men, X. Ren, X. Ren, C. Tan, S. Tan, J. Tu, P. Wang, S. Wang. W. Wang. S. Wu, B. Xu, J. Xu, A. Yang, H. Yang. J. Yang. S. Yang, Y. Yao, B. Yu, H. Yuan,
Z. Yuan, J. Zhang, X. Zhang, Y. Zhang, Z. Zhang, C. Zhou, J. Zhou, X. Zhou, and T. Zhu. Qwen technical
report, arXiv preprint arXiv:2309. 16609, 2023.
[4] D. Bautista and R. Atienza. Scene text recognition with permuted autoregressive sequence models. arXiv preprint XXX1:2207.06660 2022. cs.
[5] L. Blecher, G. Cucurull, T. Scialom, and R. Stojnic. Nougat: Neural optical understanding for academic documents. arXiv preprini arXiv:2308. 1..300813111 13418, 2023.
[6] A. Brock, S. De, S. L. Smith, and Simonyan. high-performance large-scale image recognition without normalization. International Conference on Machine Learning (ICML), pages 1059-1071. PMLR, 2021.
[7] H. Cai, J. Li, M. Hu, C. Gan, and S. Han. Efficientvit: Multi-scale linear attention for high-resolution dense prediction, 2023.
[8] Y. Cao, Z. Yihan, H. Xu, and D. Xu. Coda: Collaborative novel box discovery and tross-modal alignment for open-vocabulary 3d object detection. Advances in Neural Information Processing Systems (NeurlPS), 36, 2024.
[9] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In European Conference on Computer Vision (ECCV), pages 213-229. Springer, 2020.
[10] K. Chen, J. Wang. J. Pang, Y. Cao, Y. Xiong, X. Li. S. Sun, W. Feng, Z. Liu, J. Xu, Z. Zhang, D. Cheng. C. Zhu, T. Cheng, Q. Zhao, B. Li, X. Lu, R. Zhu, Y. Wu, J. Dai, J. Wang. J. Shi, W. Ouyang. C. C. Loy, and D. Lin. MMDetection: Open mmlab detection toolbox and benehmark. arXiv preprint arvi (tw:/906.0700 07155, 2019.
[11] L. Chen, J. Li, X. Dong, P. Zhang, C. He, J. Wang, F. Zhao, and D. Lin. Sharegpt4v: Improving large multi-modal models with better captions. arXiv preprint arXiv:2311.12799 2023.
[12] X. Chen, L. Jin, Y. Zhu, C. Luo, and T. Wang. Text recognition in the wild: A survey, ACM Computing Surveys (CSUR), 54(2):133, 2021.
[13] M. Contributors, MMSegmentation: Openmmlaa semantic segmentation toolbox and benchmark. https //github. cotttuu..oo///pee.cc - www.mmmmaaaaaaemaaeeeemmmmmeeeee htt////mnngmeetttioonnnno 2020.
[14] x. Dai, Y. Chen, J. Yang, P. Zhang, L. Yuan, and I.. Zhang. Dynamic detr: End-to-end object detection with dynamic attention. In Proceedings of the IEFF/CVF International Conference Computer Vision (ICCV), pages 2988-2997, 2021.
[15] D. H. Diaz, Qin, R. Ingle, Y. Fujii, and A. Bissacco. Rethinking text line recognition models, arXiv preprint arxiv:2104.07770 2021. cs.
[16] A. Dosovitskiy, I.. Beyer, A. Kolesnikov, D. Weissenbom, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021.
[17] Y. Goyal, T. Khot, D. Summers-Stne Stay, D. Batra, and D. Parikh. Making the V in VQA matter: Elevating the role of image understanding in Visual Question Answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), 2017.
[18] E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang. and W. Chen. Lora: Low-rank adaptation of large language models. In International Conference on Machine Learning (ICLR), 2022.
11

---

[19] S. Huang, L. Dong, W. Wang, Y. Hao. S. Singhal, S. Ma. Lv, L. Cui, O. K. Mohammed. B. Patra, et al. Language is not all you need: Aligning perception with language models. Advances in Neural Information Processing Systems (NeurIPS), 36:72066-71109, 2023.
[20] D. A. Hudson and C. D. Manning. GQA: a new dataset for compositional question answering over real-world images. CoRR, 190/0999009506, 2019.
[21] G. Jaume, H. K. Ekenel, and I.-P. Thiran. Funsd: A dataset for form understanding in noisy scanned documents. In International Conference on Document Analysis and Recognition Workshops (ICDARW), volume 2. pages 1-6, IEEE, 2019.
[22] D. Kim, A. Angelova, and W. Kuo. Region- aware pretraining for open-vocabulary object detection with vision transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11 11144-11154, 2023.
[23] G. Kim, T. Hong, M. Yim, J. Nam, J. Park, J. Yim, W. Hwang, S. Yun, D. Han, and S. Park. Oer- free document understanding transformer. In European Conference on Computer Vision (ECCV), pages 498-517. Springer, 2022.
[24] A. Kirillov, Mintun, N. Ravi, H. Mao, C. Rolland, Gustaßson, Xiao, S. Whitthead, C. Berg. W.-Y. Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 4015-4026, 2023.
[25] J. Li, D. Li. S. Savarese, and S. Hoi. 22ip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International Conference Machine Learning (ICML), pages 19730- 19742. PMLR, 2023.
[26] Li, D. Li, C. Xiong, and C. Hoi. BLIP: bootstrapping language-ime image pre-training for unified Dision-languees understanding and generation. In International conference on machine learning (ICML), volume 162 of Proceedings of Machine Learning Research, pages 12888-12990 PMLR, 2022.
[27] Y. Li, C.-Y. Wu, H. Fan, K. Mangalam, B. Xiong. J. Malik, and C. Feichtenhofer. Mvitv2: Improved multiscale vision transformers for classification and detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4804-4814, 2022.
[28] H. Liu, C. Li, Q. Wu, and Y. Lee. Visual instruction tuning. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
[29] Z. Liu, H. Hu, Y. Lin, Z. Yao, Z. Xie, Y. Wei, Ning, Y. Cao, Z. Zhang, L. Dong, et al. Swin transformer v2: Scaling up capacity and resolution. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12009-12019, 2022.
[30] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang. S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 10012-10022, 2021.
[31] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xic. A convnet for the 2020s, 2022.
[32] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Machine Learning (ICLR), 2019.
[33] K. Marino, M. Rastegari, A. Farhadi, and R. Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3195-3204, 2019.
[34] A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque. Chartqa: A benchmark for question answering about charts with visual and logical reasoning. arXiv preprint arXiv:2203. 10244, 2022.
[35] M. Mathew, V. Bagal, R. Tito, D. Karatzas, F. Valveny, and C. Jawahar. Infographicvqa. In Proceedings of the JEEF/CVF Winter Conference on Applications of Computer Vision pages 1697-1706, 2022.
[36] M. Mathew, D. Karatzas, and C. Jawahar. Docvqa: A dataset for vqa on document images. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages
2021.
[37] B. McKinzie, Z. Gan, J.PP. Fauconnier, S. Dodge, B. Zhang, P. Dufter, D. Shah, X. Du, E. Peng. F. Weers, et al. Mml: Methods, analysis & insights from multimodal Ilm pre-training. arXiv preprint arXiv:2403.0%0 09611,, 2024.
12

---

[38] I. Misra, R. Girdhar, and A. Joulin. An end-to-end transformer model for 3d object detection. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV). pages 2906-2917. 2021.
[39] B. Moysset, C. Kermorvant, and C. Wolf. Full-page text recognition: Learning where to start and when to stop. In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, pages 771- IEEE, 2017.
[40] OpenAL. Chatgpt. https://openaa: cww com/blog/chattpai 2023.
[41] M. Oquab, T. Dareet, T. Moutakanni, H. Vo, M. Szafraniec, V. Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, Galuba, R. Howes, P.-Y. Huang, S.-W. Li, Misra, M. Rabbat, V. Sharma, G. Synnaeve, Xu, H. Jegou, J. Mairal, P. Labatut, Joulin, and Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[42] B. Peng, C. P. Hc, M. Galley, and J. Gao. Instruction tuning with gpt-4, arXiv preprint 2r.0::3300000020007777 2023.
[43] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), pages 8748 8763. PMLR, 2021.
[44] M. Ranzinger, G. Heinrich, J. Kautz, and P. Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12490-12233 12500, June 2024.
[45] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarezyk, C. Mullis, A. Katta, Coombes, J. Jitsev, and A. Komatsuzaki. Laion-400m: Open dataset of hil-fiitereed 400 million image-text pairs. arXiv preprint arXiv:211102214.222 2021.
[46] P. Sharma, N. Ding. S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Annual Meeting of the Association for Computational Linguisticy (ACL), pages 2556-2565, 2018
[47] Y. Shen, K. Song, X. Tan, Li, W. Lu, and Y. Zhuang. Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface. arXiv preprint arXiv:2303. 17580, 2023.
[48] Z. Tu, H. Talebi, H. Zhang, F. Yang, P. Milanfar, A. Bovik, and Y. Li. Maxvit: Multi-axis vision transformer, 2022.
[49] K. Wang, B. Babenko, and S. Belongie, End-to-end scene text recognition. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages
IEEE, 2011.
[50] T. Wang, D. Wu, A. Coates, and A. Y. Ng. End-to-end text recognition with convolutional neural networks. Proceedings of International Conference on Puttern Recognition (ICPR), pages 3304-3308. IEEE, 2012.
[51] W. Wang, E. Xie, X. Li, D.-P. Fan, K. Song, D. Liang, T. Lu, P. Luo, and L. Shao. Pyramid vision transformer: A versatile backbone for dense prediction without convolutions. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 568 578. 2021.
[52] S. T. Wasim, M. Naseer, S. Khan, F. S. Khan, and M. Shah. Vita-clip: Video and text adaptive clip via multimodal prompting. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23034-23044, 2023.
[53] H. Wei, L. Kong, J. Chen, L. Zhao, Z. Ge, J. Yang, J. Sun, C. Han, and X. Zhang, Vary: Scaling up the vision vocabulary for large lasion-language models. arXiv preprint 20X10:231..06109, 2023.
[54] H. Wei, I.. Kong, J. Chen, I.. Zhao, 7. Ge, F. Yu, J. Sun, C. Han, and X. Zhang. Small language model meets with reinforced vision vocabulary. arXiv preprint arXiv:2401. 12503, 2024.
[55] C. Wu, S. Yin, W. Qi, X. Wang, Tang, and N. Duan. Visual chatgpt: Talking, drawing and editing with visual foundation models. arXiv preprint ar Xr33::2303.0467,, 2023.
[56] H. Wu, B. Xiao, N. Codella, M. Liu, X. Dai, L. Yuan, and L. Zhang. Cvt: Introducing convolutions to vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 22 31, 2021.
13

---

[57] E. Xie. W. Wang. Z. Yu. A. Anandkumar, J. M. Alvarez, and P. Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems (NeurIPS), 14:12077-12090, 2021.
[58] J. Xu, S. De Mello, S. Liu, W. Byeon, I. Breuel, Kautz, and X. Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18144418144, 2022.
[59] R. Yang, L. Song. Li, Zhao, Y. Ge, X. Li, and Y. Shan. Gpt4tools: Teaching large language model to use tools via self- instruction. arXiv preprint arXiv:2305. 2023.
[60] Ye, A. Hu, Xu, Q. Ye, M. Yan, Y. Dan, C. Zhao. G. Xu, C. J. Tian, et al. mplug-docowl; Modular- ized multimodal large language model for document understanding, arXiv preprint urXiv:2307.02499 2023.
[61] J. Ye, A. Hu, H. Xu, Q. Ye, M. Yan, G. Xu, C. Li, J. Tian, Q. Qian. J. Zhang, et al. Ureader: Universal ocr-free iisuall--ituated language understanding with multimodal large language model. arXiv preprint 200.0 026000/2222222 2023.
[62] Q Ye, H. Xu, G. Xu, J. Ye, M. Yan, Y. Zhou, J. Wang, A. Hu, P. Shi, Y. Shi, et al. mplug-ow owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304.141 2023.
[63] Q. Ye, H. Xu, J. Ye, M. Yan, A. Hu, H. Liu, Q. Qian, J. Zhang, and F. Huang. owl2: Revolution- izing multi-modal large language model with modality collaboration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13000-13055 2024.
[64] Y. Zeng. C. Jiang, Mao, J. Han, C. Ye. Q. Huang, D.-Y. Yeung. Z. Yang, X. Liang, and H. Xu. Clip2: Contrastive language-image-poing pretraining from real-world point cloud data. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 15244-15253, 2023.
[65] X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pages 11975-11986, 2023.
[66] Q. Zhang, J. Zhang, Y. Xu, and D. Tao. Vision transformer with quadrangle attention. IEEE Transactions Pattern Analysis and Machine Intelligence (TPAMI), 2024.
[67] R. Zhang, 7. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. Qiao, P. Gao, and H. Li, Pointelip: Point cloud understanding by clip. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8552-8562, 2022.
[68] S. Zhang. S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, al. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01 01068, 2022.
[69] S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V. Lin, et al. Opt: Open pre-trained transformer language models. URL https://arxiv org/abs/2205.0 201.2220000006. 01068, 3:19-0, 2023.
[70] D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny. Minigpt-4: Enhancing vision languael understanding with advanced large language models. arXiv preprint arXiv:2304. 10592, 2023.
[71] L. Zhu, X. Wang, Z. Ke, W. Zhang, and R. W. Lau. Biformer: Vision transformer with livee routing attention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10323-10333, 2023.
14

---

A Appendix supplemental material
A.1 Computer Resources
We implement our method using PyTorch 2.1.0 with CUDA 1 11.7. The training process requires 128 Ascend 910B GPUs (each with 64 GB of memory), Each training stage runs for one epoch, totaling 150 hours of training time.
A.2 Limitations
Integrating text and image recognition in single model may entail trade-off', potentially yielding slightly inferior performance in highly specialized tasks compared to models exclusively dedicated to text or image recognition. For example, as illustrated in the Table I of our paper, UNIT's performance in the markdown conversion task (a highly specialized OCR task) falls short of Nougat, model exhaustively trained on markdown data and specifically optimized for markdown conversion.
A.3 Broader Impacts
Positive Societal Impacts: 1) Enhanced Accessibility: By unifying image and text recognition within single model, UNIT could potentially improve accessibility for individuals with visual impairments. The model's ss ability to accurately recognize and interpret both images and text could facilitate the development of assistive technologies such as sereen readers and optical character recognition (OCR) systems. 2) Efficiency in Document Analysis: UNIT's capability to seamlessly handle both image and text recognition tasks could streamline document analysis processes in various domains, including healthcare, education, and legal sectors. This could lead to increased efficiency and productivity in tasks such as document digitization, content extraction, and information retrieval. 3) Technological Advancement: The development of novel training frameworks like UNIT contributes to the advancement of computer vision and natural language processing technologies. Such advancements have the potential to drive innovation in diverse fields, including artificial intelligence, robotics, and human-computer interaction.
The improved accuracy and efficiency of text recognition enabled by models like UNIT may raise concerns regarding privacy and data security. Increased capability to extract and analyze text from images could potentially infringe upon individuals' privacy rights if not properly regulated safeguarded. 2) Job Displacement: The automation of document analysis tasks facilitated by UNIT and similar models could lead to job displacement in industries that rely heavily on manual data entry and document processing. While this could result in increased efficiency and cost savings for businesses, it may also contribute to unemployment and economic inequality.
Possible Negative Societal Impacts: 1) Privacy Concern::
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We believe that the main claims reflect the areas targeting: vision fundation models and large-scale vision-language models.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Please see the Appendix.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
16

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: See the Section 3 Methodology and Section 4 Experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
17

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Answer: [No]
Justification: According to the code and data authorization rule of our institution, we are not allowed directly attach the code and data in the anonymous submission. We will apply for a open-source license once our paper is accepted.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (kttps://nips.cr ec/
public/guidess htt////oeeeummsssoooooo
for more details.
While encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
7 7/nips. //nips.cc/pub.e httppss.ccpubllc/guuudeee wttp:ee///oooSSeeesseeieiiie
ssioiiooiiiiioiociicc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: See the Section 4 Experiments for implementation details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: Our experiments are quite stable with multiple runs. All results are observed via fixed seed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
18

---

The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative emrr rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix for more details.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research is conducted with the NeurlPS code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: See Appendix.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
19

---

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
We have cited the creators of all existing assets. All the used existing assets are open-source.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt paperswi thcode. cww/ datasets package should be provided. For popular datasets,
has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
20

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets,
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
21


Reviews:
[{"review_id": "Geo3aTjvf9", "reviewer": "Reviewer_qQPk", "summary": "The paper proposes to enable vision encoders to support general image recognition and text recognition at the same time, in which a lightweight language decoder is proposed for predicting text outputs and a lightweight vision decoder is proposed to prevent catastrophic forgetting of the original image encoding capabilities. Besides the intra-scale pertaining, an inter-scale fine-tuning stage is utilized to improve the robustness of both image recognition and text recognition. Experiments on multiple datasets demonstrate the effectiveness of the proposed method and its potential to process interleaved text and image information in real-world scenarios.", "questions": "Please refer to the weakness part.", "limitations": "Yes. Positive societal impacts and possible negative societal impacts are well described in the A.3 section.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1) The paper proposes to integrate strong text recognition and image recognition into one encoder, which provides good potential for complex document analysis and general vision recognition.\n2) The proposed techniques including inter-scale training and feature sampling help the model to deal with diversities for image recognition and text recognition.\n3) Experiments on several benchmarks demonstrate the superiority over general encoders (CLIP, DINO) and specialized document models (vary, nougat).", "weaknesses": "1) The ability to keep ability from multiple sources is mainly achieved by distillation from a teacher model, which follows existing works like RADIO and is not quite new. \n2) As stated in L300-303, the teachers used for other models are different from UNITS, which may lead to unfair comparison.\n3) How does the model effectively processing high-resolution document images with the transformer vision encoder?\n4) Missing comparison with existing document MLLM, e.g., Monkey. \n[1] Monkey: Image resolution and text label are important things for large multi-modal models, CVPR 2024.\n5) Missing reference for Figure 2. Incomplete information in the references part, e.g.,  reference [6] A. Brock, S. De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization, 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your comprehensive and encouraging review. We are pleased to hear that you appreciate the technical contributions and the significantly improved performance of our proposed visual encoder, UNIT, along with the intra/inter-scale training paradigm. In the following, we will respond to your concerns and questions:\n\n## **Q1. Compare with RADIO**\n\nThe main contribution of this paper is the successful integration of text recognition capabilities into a vision encoder that was pretrained solely on natural images, while still preserving its original image recognition abilities. In our method, the text recognition capability is integrated via a language decoding process, and the image recognition ability is preserved through model distillation. This approach differs from pure model distillation methods like RADIO in two key aspects:\n\n1. **Training tasks:** Our method employs a multi-task training framework that jointly optimizes distillation, image captioning, and OCR tasks. This necessitates careful model design and implementation to balance these different objectives, making the approach more challenging than optimizing solely with distillation tasks, as done in RADIO.\n\n2. **Capability upper bound:** In pure distillation methods, the teacher models represent the upper limit of the student's capabilities (e.g., RADIO's performance on zero-shot classification and segmentation is within 3% of that of their teacher models), which may limit the student's learning potential. In contrast, our method learns text recognition from millions of data samples, allowing it to surpass state-of-the-art text recognition models by significant margins, ranging from 2% to 42%.\n\n   \n\n## **Q2. Experimental settings about teacher models** \n\nThe compared models use DINOv2 and OpenCLIP-H as their teachers, while we use RADIO-H, which is also distilled from DINOv2 and OpenCLIP-H. Therefore, both the compared models and our model **share the same foundational teacher knowledge** derived from DINOv2 and OpenCLIP-H. We believe this makes the comparison reasonable.\n\n\n## **Q3. High-resolution document inputs** \n\nUNIT supports input images with a maximum size of 896x896. For images with resolutions higher than this, we employ a grid slicing strategy, where each image is divided into several grids according to their aspect ratio. For example, when the maximum grid number is set to 4, the grids could be {2x2, 1x{2,3,4}, {2,3,4}x1}, each of size 896x896. The encoder generates visual tokens for each grid, which are then concatenated as the final representation of the image. \n\nThis slicing strategy fragments the original input, disrupting contextual continuity and spatial geometry across patches. However, training the encoder directly with high-resolution images incurs extremely high computational costs. To balance these factors, we should minimize the number of splits. Our method extends the supported resolution range compared to conventional ViTs, with a maximum resolution of 896x896.\n\n\n## **Q4. Comparison with existing document MLLM** \n\nWe create an MLLM using UNIT as the vision encoder. During training, we utilize the CC3M dataset along with 2 million document images for pretraining, and SFT data from LLAVA-665K, including the training sets of DocQA and ChartQA. The document images are divided into grids of size 448x448 to ensure consistency with Monkey[1]. We compare our method with existing document MLLMs, including Monkey[1], Qwen-VL[2], and mPLUG-DocOwl[3], all of which use 7B LLM models. As shown in the table below, our method achieves superior performance on document-oriented VQA tasks.\n\n|  Method  | Vision Encoder  | DocVQA | ChartVQA | InfoVQA |\n|  ----  | ----  | ---- | ----  | ----  |\n| mPLUG-DocOwl | CLIP-L 0.3B | 62.2 | 57.4 | 38.2 |\n| Qwen-VL | ViT-BigG 1.9B | 65.1 | 65.7 | 35.4 |\n| Monkey  | ViT-BigG 1.9B |  66.5 | 65.1 | 36.1 |\n| **Ours** | UNIT 0.6B |  **68.3** | **66.2** | **39.1** | \n\nThe main difference between these MLLMs and our method is that their vision encoders are pretrained solely on natural images and are trained with LLMs directly on document QA tasks by adding LoRA [1] or through direct finetuning [2,3]. In contrast, UNIT, as a vision encoder, is pretrained on both natural images and documents with fundamental recognition tasks, i.e., image captioning and OCR. The advantage lies in two aspects: First, pretraining on fundamental recognition tasks makes UNIT more versatile and better suited for downstream document tasks. Second, UNIT's early integration of text recognition capabilities provides a robust initialization for document MLLM training, resulting in more stable training and faster convergence. \n\nTo address your concern, we will add these discussions in the revised manuscript.\n\n- [1] Monkey: Image resolution and text label are important things for large multi-modal models, CVPR 2024.\n- [2] Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond.\n- [3] mPLUG-DocOwl: Modularized Multimodal Large\nLanguage Model for Document Understanding\n\n\n\n## **Q5. Reference and citation details** \n\nThank you for your valuable suggestions. We will address the issues as follows:\n\n1. The reference for Figure 2 will be added in the appropriate sections, specifically in lines 204 and 232.\n2. We acknowledge the incomplete information in the references. The citation for the paper you mentioned, \"[6] A. Brock, S. De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization, 2021,\" will be updated with the complete details.\n\nWe appreciate your attention to detail and will ensure these corrections are made promptly.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VBuCDEkkw7", "reviewer": "Reviewer_W6eH", "summary": "This paper presents a new visual backbone training framework named UNIT, which integrate image and text recognition simultaneously. The UNIT framework leverages text recognition, visual feature construction and image captioning as the pre-training tasks. The training pipeline includes two stage: intra-scale pre-training and inter-scale pre-training with different solution for different types of images. Extensive experiments demonstrates UNIT significantly outperforms existing methods on document-related tasks while maintaining performance on natural images without additional inference cost.", "questions": "1. In Table 3, we observe that even when using image captioning during pre-training, the zero-shot classification performance on natural images remains poor without applying feature reconstruction as the pre-training task. I am curious as to why this approach fails, given that Cappa [1] also utilizes image captioning for pre-training and scales well in natural image classification. Could you provide a detailed discussion on this discrepancy?\n\n2. For the LLaVA experiments, it seems that two different resolutions are applied for different types of images. Why not considering using both 224 or 896 resolution? Is there any negative signals for using large image resolutions for natural image?\n\n3. For the language decoder, using Q-former would be not necessary and not trivial. It seems that using naive MLP layers can also work.\n\n[1] Tschannen, Michael, et al. \"Image captioners are scalable vision learners too.\" Advances in Neural Information Processing Systems 36 (2024).", "limitations": "The authors have addressed the limitation in the manuscripts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. UNIT enables the visual backbone to recognize both text and natural images without the need for additional visual experts.\n\n2. The proposed model retains the original Vision Transformer architecture, making it cost-effective in terms of inference and deployment for various types of images.\n\n3. The two pre-training stages are well-suited for document image pre-training, which typically requires high resolution.\n\n4. The experiments demonstrate that UNIT achieves state-of-the-art performance on various document-related tasks while maintaining strong performance on natural image tasks.", "weaknesses": "1. The experiments are insufficient for understanding how the training pipeline works. The paper only leverages resolutions of 1x and 4x and does not show results for training with different resolutions.\n\n2. Document images tend to be non-square. During the pre-training stage, using square images might hinder learning for documents with extreme aspect ratios, such as those found in InfoQA [1]. This issue is not discussed in the paper.\n\n3. The model/data/resolution scaling behaviors of the proposed UNIT is not discussed which is important for pre-training framework.\n\n[1] Mathew, Minesh, et al. \"Infographicvqa.\" Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the constructive comments. We understand your main concerns are related to resolution settings and scaling behaviors. We will address these issues one by one and hope you can recognize the significance of our work.\n\n## **Q1. Training with different resolutions** \n\nOur model can be trained with any resolution below the maximum limit. We chose two primary resolutions for training to ensure consistent tensor sizes, simplifying batch processing and optimizing resource use. UNIT is trained on these two resolutions and tested on different ones. The Table below shows zero-shot classification and OCR results at various resolutions, demonstrating that our model generalizes well and maintains consistent performance.\n\n|  Resolution | 224 | 336 | 448 | 672 | 896 |\n|  ----  | ----  | ---- | ----  | ----  | ----  |\n| Zero-shot cls. | 78.8 | 81.3 | 81.7 | 81.3 | 80.5 | \n| OCR F1 | 92.5 | 93.7 | 93.2 | 93.9 | 95.8 |     \n\n\n\n## **Q2. Different document input size** \n\nTo ensure efficient data loading and processing during training, we preprocessed the document images by padding them to square shapes. During inference, we maintain the original aspect ratio of the input images. \n\n\n## **Q3. Scaling behaviors** \n\n\n**1. Model scaling:** In our paper, we primarily present the UNIT model with 0.6B parameters. To scale up the model, we added 18 more transformer layers, creating a new version with 1B parameters. After training this UNIT-1B model using the same settings and data, we observed further improvements in almost all the benchmarks.  \n\n| Model | Zero-shot Cls. | OCR Prec. | OCR Rec. | OCR F1 |\n|  ----  | ----  | ---- | ----  | ---- | \n| UNIT-0.6B | 78.8 | 96.7 | 94.0 | 95.3 |\n| UNIT-1B | 79.2 | 98.7 | 95.6 | 98.9 |\n\n| Model | VQAv2 | GQA | OKVQA | DocQA | ChartQA | InfoVQA |\n|  ---- | ---- | ----  | ---- | ---- | ----  | ----  | \n| UNIT-0.6B | 72.3 | 60.4 | 55.5 | 43.0 | 46.0 | 28.7 |\n| UNIT-1B | 76.6 | 62.9 | 56.7 | 69.6 | 62.2 | 37.9 |\n\n\n**2. Data scaling:** \n\n- More Natural Image Data: We added 5M LAION-COCO data to $\\mathcal D^I_{\\times 1}$. Increasing the description data of natural images did not lead to significant performance gains, as the original CC3M dataset already covers most common visual concepts in downstream VQA tasks.\n\n- More Document Data: We added 1M document data (from Nougat) to $\\mathcal D^T_{\\times 4}$, resulting in a slight improvement in document-oriented QA tasks, as the new data provided more diverse samples for different fonts and sizes.\n\n- Finetuning with SFT Data: Beyond freezing the encoder model as done in Llava experiments, we further finetuned the encoder model with the LLM during SFT. This led to significant performance improvements, especially in document-oriented QA tasks, indicating the necessity of high-quality instruction data.\n  \n    | Setting | VQAv2 | GQA | OKVQA | DocQA | ChartQA | InfoVQA |\n    | ----  | ---- | ----  | ---- | ---- | ----  | ----  | \n    | UNIT(ours) | 72.3 | 60.4 | 55.5 | 43.0 | 46.0 | 28.7 |\n    | More natural image data | 72.5 | 60.5 | 54.9 | 42.4 | 46.3  | 28.3 |\n    | More document data | 71.9 | 59.7 | 54.3 | 44.5 | 47.7 | 29.8 |\n    | Finetune with SFT data | 76.7 | 63.4 | 56.9 | 67.6 | 63.4 | 36.2 |\n\n\n**3. Resolution scaling:** Scaling up the maximum resolution increases computational costs for training the encoder with higher-resolution images. To address this, we follow existing document MLLMs (mPLUG-DocOwl, Qwen-VL, and Monkey) by using a grid slicing strategy, dividing each image into grids. The encoder generates visual tokens for each grid, which are then concatenated to form the final representation. Despite using LLMs with similar parameters, our method achieves superior performance, as shown in the table:\n\n|  Method  | Vision Encoder  | DocVQA | ChartVQA | InfoVQA |\n|  ----  | ----  | ---- | ----  | ----  |\n| mPLUG-DocOwl | CLIP-L 0.3B  | 62.2 | 57.4 | 38.2 |\n| Qwen-VL | ViT-BigG 1.9B | 65.1 | 65.7 | 35.4 |\n| Monkey  | ViT-BigG 1.9B |  66.5 | 65.1 | 36.1 |\n| **Ours** | UNIT 0.6B |  **68.3** | **66.2** | **39.1** |  \n\n## **Q4. Zero-shot cls results in Table 3** \n\nThis result is reasonable because zero-shot classification evaluates the alignment between the vision and text encoders. When the feature reconstruction loss $\\lambda=0$, the vision encoder cannot ensure proper alignment with the text encoder of OpenCLIP-H, leading to the failure of zero-shot classification. CapPa does not use the CLIP models' text encoder for zero-shot classification. Instead, it uses LiT[1], an efficient method to equip any pretrained vision backbone with zero-shot classification, to learn a text embedding that matches the embedding of their pretrained vision encoders.\n\n- [1] LiT: Zero-Shot Transfer with Locked-image text Tuning.\n\n## **Q5. About LLaVA experiments** \n\nThe resolution is determined by task requirements, not image type. Low resolution is used for global understanding (e.g., VQA), while high resolution is used for fine-detail perception (e.g., DocQA). The token number is more important and effective than the resolution. Using a 896 resolution for natural images does not improve performance, but increasing the token number to 576 can enhance it.\n\n|  Resolution | Token num | VQAv2 | GQA | OKVQA |\n|  ----  | ----  | ---- | ----  | ----  | \n| 224 | 256 | 72.3 | 60.4 | 55.5 | \n| 896  | 256 | 71.0 | 60.0 |  54.3 | \n| 896  | 576 | 74.0 | 62.1 |  55.8 | \n\n\n## **Q6. Replace Q-former** \n\nQ-former reduces the number of visual tokens for images. An image of 896x896 generates 4096 tokens, which is too computationally intensive for a lightweight language decoder and slows convergence. In our training framework, Q-former should be replaced with modules like pooling layers that reduce token count. However, when integrated with a large-scale LLM like Llama-7B, which supports long input sequences, Q-former can be replaced with MLP layers.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "4qXO0Hj3zf", "reviewer": "Reviewer_rNyF", "summary": "In this paper, the authors propose a novel training framework aiming at unify image and text recognition within a single model. The framework contains both visual and language decoders. In addition, the authors adopt a two-stage training, where the pre-training stage uses intra-scale data while the finetuning stage uses inter-scale samples. The experimental results demonstrate the effectiveness of the framework.", "questions": "(1)\tIn line 162, the authors describe that “low-resolution images, the position embedding are then randomly cropped”. I wonder why low-resolution images will need cropped instead of high-resolution images?", "limitations": "Not addressed yet.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "(1)\tThe authors aim to unify image and text recognition tasks in one module, which is good for the community. \n\n(2)\tThe general pipeline of the paper is clear.\n\n(3)\tThe performances improvement in the experiments seems significant.", "weaknesses": "(1)\tAlthough the authors claim that they unify image and text recognition into one module, I find there are two different decoders. It seems the authors just combine two decoders in the framework, one for image recognition and another for text recognition. Hence, I think this unification is not very elegant.\n\n(2)\tThe general pipeline is clear, but some details are not given. For example, the data preparation of both intra-scale and inter-scale training is not clear. In addition, it is not clear why the random feature sampling is effective. Because it will discard tokens, if those discarded tokens contain important information, the recognition will be definitely failed. By the way, I am not sure how many tokes will be discarded commonly.\n\n(3)\tIn the paper description, the authors claim that the scale robustness is an important contribution (line 129), but I can not find sufficient experiments about this. In the experiments, I find the input scales are fixed, i.e., documents with 896\\*896, and images with 224\\*224 (I guess). Why not show more results of flexible inputs about the image and documents? Based on my understanding, the training data is about 896\\*896 and 224\\*224, so the test data will follow this resolution. If my understanding is correct, I suppose it can not be claimed about the scale robustness. \n\n(4)\tI can not understand the test stage. Does it mean the visual decoder is used for image recognition and the language decoder for text recognition? if it is correct, we need to specify the task for each image during the test stage, which will limit the usage. For example, if an image will be used for both image recognition and text recognition, how to deal with? In addition, this design is not flexible for users.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for recognizing the contribution of our model to the community. We will make it publicly available in the coming days. We understand that your main concerns and rating stem from some technical details and test settings. Below, we will address these issues one by one. We sincerely hope you can appreciate the significance of our work.\n\n## **Q1. Unification of image and text recognition** \n\nAs claimed in our title, we unify image and text recognition into **one single encoder** (not just \"one module\" as mentioned in your comment). This unification lies in the shared encoder, not the decoders. The two lightweight decoders primarily serve as auxiliary modules for training and are discarded during actual application.\n\nIn our experiment, to evaluate image and text recognition abilities independently, we retain the vision decoder for zero-shot classification and the language decoder for OCR. We recommend users pair our encoder model with a large-scale decoder model (e.g., Vicuna-7B) to enhance both natural image and document-related tasks.\n\n\n\n## **Q2. Some more details** \n\n\n**1. Data preparation**: We are unsure which specific part you find unclear. We have detailed the data sources, amounts, resolutions, and the ratio of natural images to documents in lines 207-212 and 233-241. I guess your confusion is about generating synthetic datasets of PDF documents, they are rendered with Latex/HTML codes from arXiv papers and websites, as in Nougat. If you have any other questions, please let us know.\n\n**2. Effectiveness of random feature sampling**: \n\n- **Ablation study**: In Table 4 of our paper, the results of \"w/o feat. sampling\" under \"Exchanged Resolution\", indicate a significant performance drop (31.66% vs. 80.49%) when the sampling strategy is disabled. \n\n- **Theoretical explaination**: Images have significant spatial redundancy; a missing patch can be recovered from neighboring patches through high-level understanding of parts, objects, and scenes. MAE[1] has demonstrated that even when 75% of image patch features are randomly dropped, the original image can still be reconstructed using a lightweight decoder.  \n\n- **Advantages**: Randomly dropping vectors acts as a regularization technique, reducing noise in gradient calculations. This helps prevent overfitting, improves generalization, and leads to more stable and faster convergence.\n\n  [1] Masked Autoencoders Are Scalable Vision Learners, CVPR2022.\n\n\n## **Q3. Scale Robustness** \n\n**1. Scale robustness**: This refers to the model's ability to maintain consistent perceptual capabilities across different input resolutions. Our model achieves scale robustness via inter-scale finetuning. The ablation study is presented in lines 312-316 and the 1st row in Table 4, which demonstrates a significant improvement in \"Exchanged Resolution\" (0.40% vs. 80.49%).\n\n**2. Input scales**: Our model can be trained with any resolution below the maximum limit. We chose two primary resolutions for training to ensure consistent tensor sizes, simplifying batch processing and optimizing resource use. UNIT is trained on these two resolutions and tested on different ones. The Table below shows zero-shot classification and OCR results at various resolutions, demonstrating that our model generalizes well and maintains consistent performance.\n\n|  Resolution | 224 | 336 | 448 | 672 | 896 |\n|  ----  | ----  | ---- | ----  | ----  | ----  |\n| Zero-shot cls. | 78.8 | 81.3 | 81.7 | 81.3 | 80.5 | \n| OCR F1 | 92.5 | 93.7 | 93.2 | 93.9 | 95.8 |  \n   \n  We also list some experiments in our paper that use different input size:\n   \n  - In Table 4, we not only evaluate our model on zero-shot classification and OCR on their \"Commonly used resolution\" (image 224 and document 896), but also show the results on \"Exchanged resolution\" (image 896 document 224). \n  - As in line 278, the input resolution is set to 512x512 for semantic segmentation.    \n\n\n\n**3. More experiments**: We also evaluated ViTs commonly used in MLLMs on zero-shot classification. These ViTs, pretrained on fixed resolutions, perform poorly as the difference between input size and pretraining resolution increases. In contrast, our model maintains consistent perceptual ability across different resolutions, with no sharp drop in performance. \n\n  |  Method  | 224 | 336 | 448 | 672 | 896 |\n  |  ----  | ----  | ---- | ----  | ----  | ----  |\n  | CLIP-L | 72.9 | 71.3 | 63.3 | 59.7 | 51.0 |\n  | OpenCLIP-H | 80.5 | 80.2 | 78.5 | 72.7 | 65.7 |\n  | EVACLIP-G  | 78.5 | 78.2 |  76.8 | 72.9 | 68.0 |\n  | UNIT | 78.8 | 81.3 | 81.7 | 81.3 | 80.5 | \n\n\n\n## **Q4. Test setting** \n\nThe two lightweight decoders primarily serve as auxiliary modules for training and are discarded during actual application. In our experiments, they are retained to independently compare image and text recognition abilities against state-of-the-art methods on zero-shot classification and OCR tasks.\n\nUNIT serves as a fundamental vision backbone, providing general visual embeddings for both natural images and documents. We recommend pairing our encoder with a large-scale decoder capable of handling both image and text information. For example, integrating the encoder with a LLM to form a MLLM supports a wide range of applications, including recognizing and understanding standalone images or documents, as well as scenarios where images and text appear together.\n        \n\n\n## **Q5. Position embedding**\n\nThe position embeddings are set based on the model's maximum resolution. For example, if initialized to 64x64xd, where d is the hidden size, this corresponds to a maximum resolution of 896x896 with a patch size of 14. For an image of 896x896, the required position embeddings are 64x64xd, so cropping is unnecessary. However, for an image of 224x224, we need to randomly crop 16x16xd from the position embeddings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time, insightful suggestions, and valuable comments. We respond to each reviewer's comments in details. These issues will be addressed in the revised manuscript, and we believe this makes our paper much stronger. 

We promise to release all the pre-trained models and their codes to promote the development of vision foundation models and MLLMs. We believe this will significantly contribute to the community by providing valuable resources for research and innovation.

Meta-review: The manuscript received originally mixed reviews. Nevertheless, the authors response clarified many aspects raised by the reviewers, leading to positive final reviews from all reviewers.

A lot of detailed information was presented by the authors during the rebutal. Although I consider the paper worth presenting in the conference, it is important for these clarifications provided during the reviewing process to be properly integrated in the revised version of this manuscript.
Decision: Accept (poster)