UNIT: Unifying Image and Text Recognition in One Vision Encoder

Authors: yi  zhu
yanpeng  zhou
chunwei  wang
yang  cao
jianhua  han
lu  hou
hang  xu
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 03:28:24

Abstract: Currently, vision encoder models like Vision Transformers (ViTs) typically excel at image recognition tasks but cannot simultaneously support text recognition like human visual recognition. To address this limitation, we propose UNIT, a novel training framework aimed at UNifying Image and Text recognition within a single model. Starting with a vision encoder pre-trained with image recognition tasks, UNIT introduces a lightweight language decoder for predicting text outputs and a lightweight vision decoder to prevent catastrophic forgetting of the original image encoding capabilities. The training process comprises two stages: intra-scale pretraining and inter-scale finetuning. During intra-scale pretraining, UNIT learns unified representations from multi-scale inputs, where images and documents are at their commonly used resolution, to enable fundamental recognition capability. In the inter-scale finetuning stage, the model introduces scale-exchanged data, featuring images and documents at resolutions different from the most commonly used ones, to enhance its scale robustness. Notably, UNIT retains the original vision encoder architecture, making it cost-free in terms of inference and deployment. Experiments across multiple benchmarks confirm that our method significantly outperforms existing methods on document-related tasks (e.g., OCR and DocQA) while maintaining the performances on natural images, demonstrating its ability to substantially enhance text recognition without compromising its core image recognition capabilities.

Main paper: 
 introduction: 
 Vision encoder models [6, 43, 44, 53, 41, 24] are crucial for extracting high-level visual features, thereby enhancing performance across a range of downstream tasks [9, 38, 8, 14, 57] . They play a vital role in integrating visual information into intelligent applications, driving advancements in both computer vision and artificial intelligence. In recent years, Vision Transformers (ViTs)based encoder models [16, 66, 27, 58, 71] have revolutionized the field of computer vision. ViT models pretrained on image classification tasks are extensively used as backbones for a wide array of image recognition tasks and achieves state-of-the-art performances. Another line of remarkable ViT models are trained via image-text cross-modal contrastive learning [43, 64, 52, 67] . These models are often used as plug-in vision encoders in Large-scale Vision-Language Models (LVLMs), which have emerged as versatile tools with the potential to revolutionize various domains, including healthcare diagnostics, autonomous driving, digital assistants, and advanced content analysis in media and entertainment. Despite these advancements, existing vision encoder models, exhibit a significant limitation: they have not yet demonstrated the capability to simultaneously support both image and text recognition like humans do. Such ability is essential for document analysis applications, for instance, a model must accurately recognize and interpret textual information embedded within complex layouts, such as tables, graphs, and mixed media.Text recognition [50, 49, 12] , particularly in the context of dense documents and complex visual environments, presents unique challenges that are distinct from those encountered in pure image recognition. While image recognition typically focuses on global feature extraction and classification, text recognition requires precise local feature extraction and sequence prediction. ViTs, though adept at handling global image features, often struggle with the detailed local features needed for accurate text recognition, particularly in high-resolution document images where fine details are crucial. A straightforward solution is to finetune pre-trained ViTs using high-resolution documents. However, this approach requires interpolating the pre-trained positional embeddings to handle longer sequences. Such interpolation with a large magnification will disrupt the alignment between the original positional embeddings and their corresponding spatial positions, thereby impacting the model's performance.Existing methods [39, 4, 5] build document-specific models by retraining ViTs exclusively on the Optical Character Recognition (OCR) task, thereby discarding the original image encoding capability. In downstream applications, these models are often ensembled with other expert models, requiring users to specify the data type in advance, which is inadequate for scenarios requiring dynamic and simultaneous recognition of both image and text without prior knowledge of the input content. Other ensemble-based methods [53, 54] use similar model architectures trained independently on image and text recognition tasks. Inputs are fed into both models, and the output features are concatenated to form a unified representation. However, this approach will result in a significant increase in computational cost. Additionally, some LVLM methods [62, 63] enhance document analysis tasks (e.g., DocQA) in an OCR-free manner by finetuning with document-instruction data. Nevertheless, their capabilities are limited to handling images with prominent text and fail with dense documents, struggling to generalize across different font sizes, typefaces, and backgrounds.In this paper, we propose UNIT, a novel training framework for UNifying Image and Text recognition abilities within a single model. UNIT upgrades an existing Vision Transformer (ViT) model to effectively integrate text recognition. First, a lightweight language decoder (e.g., OPT-125M) is introduced to decode the learned visual features into text sequences in an auto-regressive manner, enabling the encoder model to capture fine-detailed shape and sequential information necessary for text recognition. Then, a tiny vision decoder (e.g., two MLP layers) is introduced to reconstruct the visual features of the original vision encoder from the newly learned features, preventing catastrophic forgetting of the model's fundamental encoding ability for natural images. The training pipeline involves two stages. The first is the intra-scale pretraining stage, where the model takes images and documents at their commonly used resolutions as inputs, specifically low-resolution images and high-resolution documents (×4 times the low-resolution). During this stage, the model is optimized with three objectives: OCR for document inputs, feature reconstruction of the original ViT encoder's features and image captioning for natural image inputs. The second stage is inter-scale finetuning stage, where the model is trained under a scale-exchanged setting with high-resolution images and low-resolution documents, enhancing its scale robustness for both image and text recognition. It is important to note that UNIT retains the original vision encoder architecture, ensuring that there is no increase in inference cost.Extensive experiments show that UNIT significantly outperforms document-specific models on OCR tasks while maintaining core image recognition capabilities. When integrated into LVLMs, it also benefits downstream document analysis tasks without degrading image understanding. This demonstrates that UNIT effectively unifies image and text recognition abilities.The main contributions of this paper are summarized as follows:• We propose UNIT, a novel framework that unifies image and text recognition in a single vision encoder through joint training of multi-tasks. The model retains the original vision encoder architecture, ensuring cost-free deployment and no increase in inference cost.• The training paradigm comprises two key stages: an intra-scale pretraining stage, where images and documents are trained at their commonly used resolutions, and an inter-scale finetuning stage, where their resolutions are exchanged to enhance scale robustness.• Extensive experiments demonstrate that UNIT significantly enhances performance on vision tasks requiring text recognition compared to their counterparts, while preserving the fundamental encoding ability on natural images. 
related work: 
 Vision Transformer for Text Recognition. In recent work, the application of Vision Transformers (ViT) to document reading has gained traction due to their success in image recognition [30, 10, 24, 4, 15] . Nougat [5] leverages a ViT model for Optical Character Recognition (OCR) tasks, focusing on converting human-readable scientific documents into a machine-readable markup language. Similarly, KOSMOS-2.5 [19] employs a ViT-based vision encoder coupled with a Transformer-based language decoder, aiming to serve as a versatile tool for diverse text-intensive image understanding tasks through supervised fine-tuning. The Donut model [23] , while introducing an OCR-free transformer for visual document understanding, may encounter challenges in generalizing to unfamiliar document types. Despite their demonstrated efficacy in specialized OCR scenarios, these models may not fully harness the original image encoding capabilities present in pre-trained ViT architectures. This limitation can restrict their applicability to text-only images.LVLMs for Document Analysis. Several recent studies have explored OCR-free visual-situated language understanding using Large Vision-Language Models (LVLMs) [47, 55, 59, 1, 2, 70, 40, 28] , where document-instruction data is used to fine-tune LLMs without adapting the vision encoder [62, 60, 61] . These methods often rely on a pretrained Vision Transformer (ViT) as a fixed vision encoder, which may present several challenges: 1) The frozen ViT may limit text recognition capabilities, especially when documents feature varied fonts, styles, or low image quality not seen during its pretraining.2) The model's reliance on a constant image resolution can lead to inaccuracies in text recognition for high-resolution documents or those with noise and distortion. Additionally, several recent works [53, 54] have successfully enhanced LVLMs with OCR-like abilities, by retraining a document-specific Vision Transformer (ViT) and then concatenating its output visual features with those from a pretrained ViT model. However, this method comes with trade-offs, notably an increase in computational expense and a significant underutilization of the model's capacity, leading to inefficiencies. In contrast, our UNIT integrates both capabilities within a single model, offering greater efficiency, cost-free deployment, and no increase in inference time.Multi-scale Vision Transformer. Vision Transformer models are commonly pre-trained to process images at a fixed resolution, such as 224 or 336 pixels, which is commonly used for many image understanding tasks [17, 20, 33, 26, 25] . However, these resolutions might be insufficient for precisely discerning tightly packed text in higher-resolution documents. Consequently, a multi-scale training approach for Vision Transformers is crucial for effectively managing both image and text recognition challenges. The progression of Vision Transformer architectures has embraced a notable trend toward multi-scale modeling. This paradigm enables the models to capture and process visual information across different scales, enhancing their capability to understand complex scenes and texts. Some methods [29, 51, 56] integrate a hierarchical pyramid structure into the original Vision Transformer architectures, which continue to operate at a constant resolution. In contrast to these methods, our proposed UNIT model breaks the limitations of fixed-resolution inputs. UNIT is designed to handle multi-scale training for multi-tasks, yielding a unified representation with scale robustness for both images and documents.3 Methodology 
preliminaries: 
 Backbone. We employ the Vision Transformer (ViT) [16] architecture as our backbone. Let f θ (•) be the vision encoder with parameters θ, for an input image I ∈ R H×W ×3 , where H and W represent the image height and width, respectively. ViT model first partitions each image into fixed-size of patches in p × p pixels, and then encode these patches into hidden features of dimension d. Subsequently, the features are added with their corresponding position embeddings and fed into multiple layers of stacked Transformer blocks, interacting with each other via attention mechanisms. Finally the model outputs visual tokens X ∈ R N ×d , as:EQUATIONwhere Architecture. UNIT aims to unify image and text recognition capabilities within a single ViT model, as illustrated in Fig. 1 . This is achieved through the introduction of a lightweight language decoder (e.g., OPT-125M) for document-level OCR, enabling the ViT model to recognize text. Additionally, to preserve the original image recognition capabilities of the vision encoder, UNIT incorporates a vision decoder responsible for token-wise feature reconstruction from the original ViT model. To further enhance image understanding, an image captioning task is integrated alongside the OCR task. UNIT is trained in a multi-scale setting, where images and texts are processed at their commonly used resolutions during pretraining and then finetuned at other resolutions. ViTs are excellent at global feature extraction but may not perform well in text recognition. The reason is that ViTs use a learned position embedding for each input patch in an image, which in turn forces that the model always operate at a constant resolution, typically chosen as 224, 256 or 336. At these resolutions, the generated visual features may lose critical information, leading to difficulties in recognizing characters from dense texts such as PDF documents, websites, or digital books. Naively cropping high-resolution document images into low-resolution inputs would break the sequential order of characters and may impair the recognition of letters at the cutting boundaries. To prevent these issues, we introduce the Cropped Position Embedding (CPE) [22] augmentation, which interpolates the original position embeddings to match the number of positions of the maximum input size. For low-resolution images, the position embeddings are then randomly cropped and interpolated to match the number of input patches for the original model.N = N s + N c denotesLanguage Decoder. Conventional methods often employ CLIP-style contrastive learning to align images with their language annotations. However, this approach presents two notable drawbacks in our context. Firstly, contrastive learning relies on a CLIP pretrained text encoder to generate language embeddings. The maximum sequence length of 77 tokens is insufficient for text recognition annotations, especially in dense documents. Secondly, such training paradigm is inadequate for the vision encoder to accurately capture each word, leading to undesirable information loss during the encoding process. Taking the above factors into account, we introduce a lightweight Transformer decoder, e.g., OPT-125M [69] , to efficiently predict the language sequences presented in the input documents. Similar to LVLMs, we employ an input embedding layer to project the visual features from the vision encoder into the embedding space of the language decoder. Here we initialize the input embedding layer with a two-layer Q-Former [25] with K learnable query tokensQ = {q k } K k=1 , q k ∈ R d .The visual tokens X = {x i } N i=1 generated from the vision encoder are first fed into the input embedding layer and then the language decoder. We denote this process as f ϕ (•) and ϕ denotes the parameters of the input embedding layer and the language decoder:{z 1 , ..., z N } = f ϕ ({x 1 , ..., x N }, {q 1 , ..., q K }).(2)The decoder uses the last hidden state z L t ∈ R d at time step t to predict language sequence in the form of text tokens ŷ = {ŷ 1 , ŷ2 , ..., ŷT } via the language decoder in an auto-regressive manner. The cross-entropy loss for training the auto-regressive Transformer decoder is defined as follows:EQUATIONwhere y = {y 1 , y 2 , ..., y T } represents the target sequence, and the probability P (ŷ t = y t |y 1:t-1 , z L t ) is obtained from the softmax output of the final layer of the decoder.Vision Decoder. Finetuning the vision encoder exclusively on document datasets would result in severe catastrophic forgetting of its original image encoding capabilities. To mitigate this, we introduce an token-wise feature reconstruction task on natural image datasets, ensuring that the newly learned vision encoder retains its ability to encode visual concepts. We incorporate a visual decoder f π (•) consisting of two fully connected layers with a GeLU activation function in the intermediate layer. The decoder processes each visual token independently, preserving the positional information inherent in each token more effectively. We utilize the original pretrained ViT model as a teacher model to provide original features for each natural image, providing supervision signals for visual feature reconstruction. Denoting the teacher tokens as X = {x i } N i=1 , xi ∈ R d , we enforce the alignment of the new learned student tokens with the original ones using a weighted sum of the cosine distance L cos and smooth L1 loss L l1 . This approach ensures consistency in both vector direction and magnitude between the new and original features, as:EQUATIONwhere C represents a subset of features randomly sampled from the X, to mitigate overfitting and enhance robustness during training. µ denotes the loss weight scalar.Language Decoder. Furthermore, we incorporate an image captioning task alongside the text recognition task to ensure that the learned visual tokens can be effectively projected into the language space, thereby enhancing image understanding ability. The forward process and loss formulation for image inputs are the same as those for document inputs.As described in Sec. 3.2 and Sec. 3.3, UNIT tackles images and documents at different input scales with different optimization objectives. In this section, we introduce the intra-scale pretraining stage (see Fig. 2a ) where image and text recognition are trained at a fixed scale, namely, low resolution for images and high resolution for documents, without considering variations in scale.Dataset Preparation. Let D be a curated dataset with 5M samples, whereD = D I ×1 ∪ D T ×4 .Here, D I ×1 represents a dataset of natural images annotated with coarse captions (less than 30 words). The images are resized to ×1 times the original scale r, primarily sourced from the Conceptual Caption dataset [46] , comprising 3M samples. Meanwhile, D T ×4 represents a dataset of documents annotated with dense OCR data (more than 500 words). The document images are resized ×4 times the original scale r, sourced from our synthetic dataset of English PDF documents, comprising 2M samples.Instruction Prompts. The instruction prompts follow the format used in popular LVLMs [70, 25, 1, 3, 2] , where image tokens are prefixed with text tokens. Specifically, we use two special tokens "<img>" and "</img>", to indicate the start and the end position of the image tokens, followed by instructions that indicate task requirements. For natural images, the prompt is set as "Give a caption of this image:" and the LLM outputs a language sequence summarizing the content of the image. For document images, the prompt is set as "Read the text in this image:" and the LLM outputs sentences row by row as they appear in the document in the order of reading. Joint Optimization. We train all parameters, including the ViT backbone, vision decoder and language decoder, using images and documents annotated with language sentences. During training, the model is optimized by the cross-entropy loss L lan between language outputs and annotations, and also the feature reconstruction loss L vis to force the newly learned visual features to be similar with the original ones. The UNIT model is updated to minimize the total loss:θ = arg min θ L lan ({D I ×1 ∪ D T ×4 }) + λL vis ({D I ×1 }).(5)After the intra-scale pretraining stage, UNIT is capable of processing both image and text recognition at their commonly used resolutions, namely, low-resolution images and high-resolution documents. However, we observed that this model lacks scale robustness when handling texts with larger fonts and images with larger dimensions, significantly limiting its generalization ability in various vision tasks. Naively add the scale-exchanged datasets in the intra-scale pretraining process may slow down the convergence of the model and making it prone to local optima. To address this, we further conduct inter-scale finetuning (see Fig. 2b ), using high-resolution images and low-resolution documents.Dataset Preparation. Here we build another scale exchanged dataset D * = D I ×4 ∪ D T ×1 with 1M samples for the inter-scale finetuning. We conduct a dataset D I ×4 comprising natural images annotated with detailed captions (each containing over 100 words). These images are resized to ×4 times the original scale r, primarily drawn from the ShareGPT4V dataset [11] , totaling 1M samples. Additionally, we create a dataset D T ×1 comprising 1M documents with an average font size ×4 times larger than that of D T ×4 . The document images are resized ×r times, sourced from our synthetic dataset which includes cropped pages from digital books, advertisements, websites, and other sources containing short yet large font texts. We adopt the instruction prompt setting same to the intra-scale pretraining stage.Random Feature Sampling. When dealing with natural images at higher resolutions, the resulting increase in visual tokens compared to lower resolutions can lead to gradient overflow during the training process, primarily due to the token-wise feature reconstruction loss. To address this challenge, we propose to randomly sample a set of visual tokens equal to the number in the low-resolution inputs and discarding other tokens.Joint Optimization. This stage aims to enhance the scale-robust recognition ability for both image and text. We retain half of the data at the previous stage and then introduce the scale-exchanged data. The model is updated to minimize the total loss:EQUATION4 ExperimentsImplementation Details. We select OpenCLIP ViT-H [43] with 32 layers and a hidden size of 1280 as our vision encoder. We choose OPT-125M [68] model with a hidden size of 768 as the language decoder in the two training stages. Our optimization strategy involves AdamW [32] with a weight decay 0.01. The initial learning rate to 5e-5, and changes with a cosine learning rate decay scheduler. The warmup ratio is set to 0.03, and the global batch size is 256. We set loss weights λ = 2 and µ = 0.2. These settings are shared for both two training stages. Our model can be trained with any resolution below the maximum limit. We chose two primary resolutions for training to ensure consistent tensor sizes, simplifying batch processing and optimizing resource use. UNIT is trained on these two resolutions and tested on different ones. To ensure efficient data loading and processing during training, we preprocessed the document images by padding them to square shapes. During inference, we maintain the original aspect ratio of the input images. 
evaluation benchmarks: 
 Text Recognition: 1) Document-level OCR: We evaluate our model on three public OCR datasets: FUNSD (50 images), SROIE (347 images), and CORD (100 images), reporting the F1 score. Given the lack of large blocks of dense text in these datasets, we create two additional OCR evaluation datasets using English PDF files from arXiv. The first, SYN-L-val (200 images), consists of PDFs with small font sizes at approximately 1k resolution. The second, SYN-S-val (200 images), includes images cropped from these PDFs, resized to a lower resolution (e.g., 224), resulting in larger font sizes. 2) Markdown conversion: The task requires structured text output from images of documents, capturing the stylistic and structural elements of the text in a markdown format. We add 1M markdown data (collected following Nougat [5] ) into the inter-scale finetuning stage, endowing the ability of markdown conversion of our model. Due to the lack of cleaned markdown conversion validation set, MD-val, we conduct a dataset containing 82 images with markdown format annotations.Image Recognition: 1) Zero-shot classification: We calculate the feature similarity between the [CLS] token of the visual encoder and the text features extracted from the corresponding CLIP text encoder. The evaluation is performed on the ImageNet-1K dataset, and top-1 accuracy is reported. We replace the vision encoder in the LLava-1.5 [28] setting with our own encoder. We first pretrain the input embedding layers with the vision encoder and the LLM frozen, then conduct instruction tuning to finetune a Vicuna-7B model [42] . The learning rate of stage 3 is set to 5e-4 and a batch size of 512. We evaluate the trained models on general visual question answering (VQA) datasets including VQAv2 [17] , GQA [20] , and OKVQA [33] , and also document comprehension datasets including ChartQA [34] , DocQA [36] , and InfoVQA [35] .Text Recognition.We compare UNIT with two ViT-based OCR expert model, including Donut [23] and Nougat [5] , and two open-source vision encoders that support high-resolution inputs, including Vary [53] and RADIO [44] . We use the open-source weights of Donut-Base and Nougat for evaluation.As for Vary and RADIO, we retrain them on our dataset for a fair comparison. The models are evaluated on public OCR dataset like FUNSD [21] , SROIE [19] , CORD [19] and our synthetic datasets SYN-L-val and MD-val. As shown in Tab. 1, UNIT consistently outperforms these methods on OCR and markdown conversion tasks, except for Nougat, exhaustively trained on markdown data. This shows UNIT's superior ability to encode fine-detailed sequential and spatial information in documents, resulting in highly accurate OCR outcomes, see Fig. 3 . Based on the fundamental text recognition ability, UNIT can be further finetuned with a small amount of data to adapt to highly specialized tasks, such as text recognition with grounding (in both natural images and pure texts), markdown conversion and Chinese OCR. Please refer to the supplementary for more details.Image Recognition. To validate the image encoding ability of UNIT, we compare it with existing efficient vision encoders on zero-shot and kNN classification on ImageNet-1k, as well as a linear prob semantic segmentation benchmark. In Tab. 2, for fair comparision, all models share the same teacher knowledge, derived from DINOv2 and OpenCLIP-H. The compared models directly use these two models for training feature reconstruction, and UNIT employs one of the best-performing students, RADIO-H, as the teacher encoder. From the results shown in Tab. 2, we observe that UNIT achieves the best performance among all these models, demonstrating its ability to effectively preserve the fundamental image encoding capabilities.About intra-scale pretraining. As shown in Different resolutions. Our model supports arbitrary input sizes within the maximum limit. As shown in Table 5 , our model, trained on two primary resolutions, generalizes well and maintains consistent performance in both zero-shot classification and OCR tasks.Naive resolution. Vision encoders act as a key component in Large Vision-Language Models (LVLMs) to provide rich and detailed visual representations that enable effective cross-modal understanding and reasoning. Following LLava-1.5 [28] , we build LVLMs using UNIT or other vision encoders based on the Vicuna-7B LLM. The vision encoders are fixed during LVLM training. Images are padded and resized to one of two primary resolutions based on the task requirements. A lower resolution (e.g., 224x224) is used for global understanding tasks (e.g., VQA), while a higher resolution (e.g., 896x896) is used for tasks requiring fine-detail perception (e.g., DocQA). Each image is represented as 256 visual tokens through the input embedding layer. The LLM is finetuned with LoRA [18] . The multi-modal pretraining process uses 4M image-caption data (randomly extracted from Conceptual Caption [46] and the LAION-COCO [45] ) and 200k document OCR data (randomly sampled from the training set). In the Supervised Finetuning (SFT) stage, we use the LLaVA-80k or LLaVA-CC665k along with the train set of DocVQA [36] and ChartQA [34] as the fine-tuning dataset. In Tab. 6, UNIT significantly outperforms the teacher model RADIO and other compared models on the document analysis tasks, including ChartQA, DocQA and InfoVQA, while maintains comparable performance on image understanding tasks. From Fig. 4 we can see that UNIT can extract the texts in documents, charts or websites, showing its potential in real-world document analysis applications. High resolution. In the naive resolution setting, each image is represented using 256 tokens. However, this may be insufficient for capturing the details of images with extensive content, such as densely organized documents containing thousands of words. To address this, we evaluate UNIT in Llava-Next with a grid slicing strategy and compare it with two commonly used vision encoders in LVLMs. This method divides each image into several grids based on their aspect ratio. For instance, with a maximum of 4 grids, possible configurations include {2 × 2, 1 × 2, 1 × 3, 2 × 1, 3 × 1}, with each grid sized at M × M . Here, M corresponds to the resolution used during model pretraining-336 for CLIP-L and 384 for SigLIP. And we set 448 for UNIT. After the vision encoder, we use C-Abstractor [37] as the input embedding layer. It generates 256 visual tokens for each grid, which are then concatenated to create the final image representation. During the multi-modal pretraining stage, the input embedding layer is initially trained on Llava665k. Following this, the LLM and the last few layers of the vision encoder are trained using 1.5M image captioning data. In the SFT stage, all parameters of the LVLM are fine-tuned using 1.2M SFT data. As shown in Tab. 7, our method significantly outperforms the compared models on document-oriented QA tasks and demonstrates comparable performance on other QA tasks. CLIP-L and SigLIP are initially pretrained only on natural images and subsequently fine-tuned with document images for downstream tasks. In contrast, UNIT is pretrained on both natural images and documents using fundamental recognition tasks such as image captioning and OCR. Our approach offers two key advantages: First, pretraining with fundamental recognition tasks enhances UNIT's versatility and suitability for downstream document tasks. Second, the early integration of text recognition capabilities in UNIT provides a robust initialization for LVLM training, leading to more stable training and faster convergence. 
conclusion: 
 We propose UNIT, a simple yet effective framework that unifies image and text recognition in a single vision encoder, without increasing deployment or inference cost. UNIT incorporates a lightweight language decoder for text recognition and a lightweight vision decoder to preserve image recognition capabilities. Trained with multi-scale inputs, UNIT processes images and documents at their conventional resolutions during intra-scale retraining for basic recognition, and at exchanged resolutions during inter-scale finetuning to enhance scale robustness. Extensive experiments demonstrate that UNIT significantly outperforms document-specific models, maintains core image encoding ability, and benefits downstream document analysis tasks when integrated into LVLMs, showing great potential in processing interleaved text and image information in real-world scenarios.A Appendix / supplemental material Justification: The research is conducted with the NeurIPS code of Ethics in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: See Appendix.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We have cited the creators of all existing assets. All the used existing assets are open-source.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: 38th Conference on Neural Information Processing Systems (NeurIPS 2024). 


Decision: The manuscript received originally mixed reviews. Nevertheless, the authors response clarified many aspects raised by the reviewers, leading to positive final reviews from all reviewers.

A lot of detailed information was presented by the authors during the rebutal. Although I consider the paper worth presenting in the conference, it is important for these clarifications provided during the reviewing process to be properly integrated in the revised version of this manuscript.

Reviews: review: paper_summary: paper_summary: The paper proposes to enable vision encoders to support general image recognition and text recognition at the same time, in which a lightweight language decoder is proposed for predicting text outputs and a lightweight vision decoder is proposed to prevent catastrophic forgetting of the original image encoding capabilities. Besides the intra-scale pertaining, an inter-scale fine-tuning stage is utilized to improve the robustness of both image recognition and text recognition. Experiments on multiple datasets demonstrate the effectiveness of the proposed method and its potential to process interleaved text and image information in real-world scenarios.
strength_weakness: strength_weakness: 1) The ability to keep ability from multiple sources is mainly achieved by distillation from a teacher model, which follows existing works like RADIO and is not quite new. 
2) As stated in L300-303, the teachers used for other models are different from UNITS, which may lead to unfair comparison.
3) How does the model effectively processing high-resolution document images with the transformer vision encoder?
4) Missing comparison with existing document MLLM, e.g., Monkey. 
[1] Monkey: Image resolution and text label are important things for large multi-modal models, CVPR 2024.
5) Missing reference for Figure 2. Incomplete information in the references part, e.g.,  reference [6] A. Brock, S. De, S. L. Smith, and K. Simonyan. High-performance large-scale image recognition without normalization, 2021.
questions: questions: Please refer to the weakness part.
limitations: limitations: Yes. Positive societal impacts and possible negative societal impacts are well described in the A.3 section.
paper_summary: paper_summary: This paper presents a new visual backbone training framework named UNIT, which integrate image and text recognition simultaneously. The UNIT framework leverages text recognition, visual feature construction and image captioning as the pre-training tasks. The training pipeline includes two stage: intra-scale pre-training and inter-scale pre-training with different solution for different types of images. Extensive experiments demonstrates UNIT significantly outperforms existing methods on document-related tasks while maintaining performance on natural images without additional inference cost.
strength_weakness: strength_weakness: 1. The experiments are insufficient for understanding how the training pipeline works. The paper only leverages resolutions of 1x and 4x and does not show results for training with different resolutions.

2. Document images tend to be non-square. During the pre-training stage, using square images might hinder learning for documents with extreme aspect ratios, such as those found in InfoQA [1]. This issue is not discussed in the paper.

3. The model/data/resolution scaling behaviors of the proposed UNIT is not discussed which is important for pre-training framework.

[1] Mathew, Minesh, et al. "Infographicvqa." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.
questions: questions: 1. In Table 3, we observe that even when using image captioning during pre-training, the zero-shot classification performance on natural images remains poor without applying feature reconstruction as the pre-training task. I am curious as to why this approach fails, given that Cappa [1] also utilizes image captioning for pre-training and scales well in natural image classification. Could you provide a detailed discussion on this discrepancy?

2. For the LLaVA experiments, it seems that two different resolutions are applied for different types of images. Why not considering using both 224 or 896 resolution? Is there any negative signals for using large image resolutions for natural image?

3. For the language decoder, using Q-former would be not necessary and not trivial. It seems that using naive MLP layers can also work.

[1] Tschannen, Michael, et al. "Image captioners are scalable vision learners too." Advances in Neural Information Processing Systems 36 (2024).
limitations: limitations: The authors have addressed the limitation in the manuscripts.
paper_summary: paper_summary: In this paper, the authors propose a novel training framework aiming at unify image and text recognition within a single model. The framework contains both visual and language decoders. In addition, the authors adopt a two-stage training, where the pre-training stage uses intra-scale data while the finetuning stage uses inter-scale samples. The experimental results demonstrate the effectiveness of the framework.
strength_weakness: strength_weakness: (1)	Although the authors claim that they unify image and text recognition into one module, I find there are two different decoders. It seems the authors just combine two decoders in the framework, one for image recognition and another for text recognition. Hence, I think this unification is not very elegant.

(2)	The general pipeline is clear, but some details are not given. For example, the data preparation of both intra-scale and inter-scale training is not clear. In addition, it is not clear why the random feature sampling is effective. Because it will discard tokens, if those discarded tokens contain important information, the recognition will be definitely failed. By the way, I am not sure how many tokes will be discarded commonly.

(3)	In the paper description, the authors claim that the scale robustness is an important contribution (line 129), but I can not find sufficient experiments about this. In the experiments, I find the input scales are fixed, i.e., documents with 896\*896, and images with 224\*224 (I guess). Why not show more results of flexible inputs about the image and documents? Based on my understanding, the training data is about 896\*896 and 224\*224, so the test data will follow this resolution. If my understanding is correct, I suppose it can not be claimed about the scale robustness. 

(4)	I can not understand the test stage. Does it mean the visual decoder is used for image recognition and the language decoder for text recognition? if it is correct, we need to specify the task for each image during the test stage, which will limit the usage. For example, if an image will be used for both image recognition and text recognition, how to deal with? In addition, this design is not flexible for users.
questions: questions: (1)	In line 162, the authors describe that “low-resolution images, the position embedding are then randomly cropped”. I wonder why low-resolution images will need cropped instead of high-resolution images?
limitations: limitations: Not addressed yet.

Comments: title: Official Comment by Reviewer qQPk
comment: Dear reviewer,

Thank you for your prompt and thoughtful feedback. We appreciate your understanding regarding the random feature sampling. Following your suggestion, we will make sure to highlight that the sampling is specifically for natural images.

We are also committed to revising the paper to ensure that our contributions are clearly presented. Your input has been invaluable, and we're grateful for your increased score.

Thank you once again for your time and support!
thanks for your quick response. Then It make sense for random feature sampling. Since the Table 4 shows the text recognition ability, it very easy to make readers think this is also for document images. I suggest the authors can highlight the sampling for natural images. I plan to increase the rating, but I strongly suggest the authors to make the paper more clear and highlight the contributions.
Dear reviewer,

Thank you for your thoughtful comments and for engaging in a detailed discussion. 

**Q2. About random feature sampling**

We would like to clarify that the random feature sampling approach is indeed applied **only to natural images**, not document images. As mentioned in lines 242-244 of our paper, this technique is specifically designed to enhance training stability and efficiency when computing the token-wise feature reconstruction loss for high-resolution natural images. The purpose of this feature reconstruction loss is to maintain the original image recognition capability, and it is solely utilized for natural images. Therefore, the concern you have about random sampling causing a loss of text information will not occur.

**Q5. About position embedding**

Our position embedding is 64x64xd, but for input sizes smaller than 896x896, such as 224x224, a 16x16xd position embedding is required. There are generally three possible solutions: cropping, interpolation, and training another 16x16xd position embedding. We chose cropping for the following reasons:

1. Cropping is a more straightforward approach compared to other methods, as it avoids additional computational costs and memory usage.
2. As mentioned in lines 238-239 of our paper, the low-resolution document dataset consists of images cropped from high-resolution documents. Therefore, we use cropped position embeddings for low-resolution inputs to preserve local structural information, ensuring consistent spatial feature representation across resolutions. For natural images, both interpolation and cropping are viable, and we selected cropping for its simplicity.

We hope this explanation addresses your concerns. If our response has clarified the issues and alleviated your concerns, we would sincerely appreciate it if you could kindly consider updating your score.
I am sorry that it is a little late for the discussion. Thanks authors for the responses. These responses really address some of my concerns, like the usage of encoder and decoder. However, I still can not understand some of your answers, like Q5, why crop 16*16*d for 224*224, but never mind, I can read the original paper CPE for more details.  

Another question is your answer in Q2 about the random feature sampling. Although you give an example of the MAE paper, it is about the natural image, which is very different with the document image. I do not believe that dropping 75% of patches can be predicted for the document images, because the text patterns are much more complicated than the general objects, like cat, dog.
Dear reviewer,

Thank you very much for reconsidering our paper and for raising your score. We appreciate your recognition of our efforts to address your concerns.

Your feedback has been instrumental in enhancing the quality of our submission, and we are grateful for the time and thoughtfulness you have dedicated to your review. We have carefully considered your suggestions and are committed to revising the paper accordingly. We believe these revisions will further strengthen our submission.
After reading the authors' feedback and other reviewers' opinions, I would like to thank the authors for their rebuttal.

The rebuttal addresses most of my concerns, and thus, I raise my score. I would like to note that even though the authors addressed my concerns, I believe that the paper needs some sort of revision, as it appears that many basic concerns were needed to be clarified in the rebuttal. I strongly encourage the authors to fix the main comments mentioned by the reviewers.
Dear Reviewer,

Thank you for your thorough review and constructive feedback throughout the process. We appreciate your recognition of our efforts to address your concerns.

We have carefully considered your suggestions and are committed to revising the paper as you recommended. These revisions will further strengthen our submission, and we are grateful for your insights that have guided us in this direction.

Thank you again for your support!
Thank the authors for their response.

Regarding W1, in fact, the proposed method integrates both distillation in works like RADIO and supervised learning widely used in OCR methods, e.g., vary, which is not originally designed.

Regarding W2-W5, I appreciate the response and believe the details serve as a good supplement to the original manuscript.

I'd like to maintain the rating in the current stage.
Dear reviewer,

We have provided detailed clarifications on all your questions point to point. If you have any additional questions or suggestions, we would be very happy to discuss them with you. We sincerely hope you can appreciate the significance of our work and consider updating your rating. Thank you!
Dear reviewer, 

Thank you for your valuable comments. As the deadline is approaching, we kindly inquire whether our discussions have addressed your concerns. If you have any further questions, we would be happy to continue our conversation. If our response has resolved your concerns, we would greatly appreciate it if you could consider updating your score and providing feedback. Thank you again.