PIN: A Knowledge-Intensive Dataset for Paired and Interleaved Multimodal
  Documents
Authors: name: Junjie Wang, name: Yin Zhang, name: Yatai Ji, name: Yuxiang Zhang, name: Chunyang Jiang, name: Yubo Wang, name: Kang Zhu, name: Zekun Wang, name: Tiezhen Wang, name: Wenhao Huang, name: Jie Fu, name: Bei Chen, name: Qunshu Lin, name: Minghao Liu, name: Ge Zhang, name: Wenhu Chen

Recent advancements in Large Multimodal Models (LMMs) have leveraged
extensive multimodal datasets to enhance capabilities in complex
knowledge-driven tasks. However, persistent challenges in perceptual and
reasoning errors limit their efficacy, particularly in interpreting intricate
visual data and deducing multimodal relationships. Addressing these issues, we
introduce a novel dataset format, PIN (Paired and INterleaved multimodal
documents), designed to significantly improve both the depth and breadth of
multimodal training. The PIN format is built on three foundational principles:
knowledge intensity, scalability, and support for diverse training modalities.
This innovative format combines markdown files and comprehensive images to
enrich training data with a dense knowledge structure and versatile training
strategies. We present PIN-14M, an open-source dataset comprising 14 million
samples derived from a diverse range of Chinese and English sources, tailored
to include complex web and scientific content. This dataset is constructed
meticulously to ensure data quality and ethical integrity, aiming to facilitate
advanced training strategies and improve model robustness against common
multimodal training pitfalls. Our initial results, forming the basis of this
technical report, suggest significant potential for the PIN format in refining
LMM performance, with plans for future expansions and detailed evaluations of
its impact on model capabilities.