Kosmos-2.5: A Multimodal Literate Model
Authors: name: Tengchao Lv; name: Yupan Huang; name: Jingye Chen; name: Lei Cui; name: Shuming Ma; name: Yaoyao Chang; name: Shaohan Huang; name: Wenhui Wang; name: Li Dong; name: Weiyao Luo; name: Shaoxiang Wu; name: Guoxin Wang; name: Cha Zhang; name: Furu Wei
Published: 2023-09-20 15:50:08

Abstract: We present Kosmos-2.5, a multimodal literate model for machine reading of
text-intensive images. Pre-trained on large-scale text-intensive images,
Kosmos-2.5 excels in two distinct yet cooperative transcription tasks: (1)
generating spatially-aware text blocks, where each block of text is assigned
its spatial coordinates within the image, and (2) producing structured text
output that captures styles and structures into the markdown format. This
unified multimodal literate capability is achieved through a shared Transformer
architecture, task-specific prompts, and flexible text representations. We
evaluate Kosmos-2.5 on end-to-end document-level text recognition and
image-to-markdown text generation. Furthermore, the model can be readily
adapted for any text-intensive image understanding task with different prompts
through supervised fine-tuning, making it a general-purpose tool for real-world
applications involving text-rich images. This work also paves the way for the
future scaling of multimodal large language models.