A Token-level Text Image Foundation Model for Document Understanding
Authors: name: Tongkun Guan; name: Zining Wang; name: Pei Fu; name: Zhengtao Guo; name: Wei Shen; name: Kai Zhou; name: Tiezhu Yue; name: Chen Duan; name: Hao Sun; name: Qianyi Jiang; name: Junfeng Luo; name: Xiaokang Yang
Published: 2025-03-04 06:05:33

Abstract: In recent years, general visual foundation models (VFMs) have witnessed
increasing adoption, particularly as image encoders for popular multi-modal
large language models (MLLMs). However, without semantically fine-grained
supervision, these models still encounter fundamental prediction errors in the
context of downstream text-image-related tasks, i.e., perception, understanding
and reasoning with images containing small and dense texts. To bridge this gap,
we develop TokenOCR, the first token-level visual foundation model specifically
tailored for text-image-related tasks, designed to support a variety of
traditional downstream applications. To facilitate the pretraining of TokenOCR,
we also devise a high-quality data production pipeline that constructs the
first token-level image text dataset, TokenIT, comprising 20 million images and
1.8 billion token-mask pairs. Furthermore, leveraging this foundation with
exceptional image-as-text capability, we seamlessly replace previous VFMs with
TokenOCR to construct a document-level MLLM, TokenVL, for VQA-based document
understanding tasks. Finally, extensive experiments demonstrate the
effectiveness of TokenOCR and TokenVL. Code, datasets, and weights will be
available at https://token-family.github.io/TokenOCR_project.