BigDocs: An Open and Permissively-Licensed Dataset for Training
  Multimodal Models on Document and Code Tasks
Authors: name: Juan Rodriguez; name: Xiangru Jian; name: Siba Smarak Panigrahi; name: Tianyu Zhang; name: Aarash Feizi; name: Abhay Puri; name: Akshay Kalkunte; name: François Savard; name: Ahmed Masry; name: Shravan Nayak; name: Rabiul Awal; name: Mahsa Massoud; name: Amirhossein Abaskohi; name: Zichao Li; name: Suyuchen Wang; name: Pierre-André Noël; name: Mats Leon Richter; name: Saverio Vadacchino; name: Shubbam Agarwal; name: Sanket Biswas; name: Sara Shanian; name: Ying Zhang; name: Noah Bolger; name: Kurt MacDonald; name: Simon Fauvel; name: Sathwik Tejaswi; name: Srinivas Sunkara; name: Joao Monteiro; name: Krishnamurthy DJ Dvijotham; name: Torsten Scholak; name: Nicolas Chapados; name: Sepideh Kharagani; name: Sean Hughes; name: M. Özsu; name: Siva Reddy; name: Marco Pedersoli; name: Yoshua Bengio; name: Christopher Pal; name: Issam Laradji; name: Spandanna Gella; name: Perouz Taslakian; name: David Vazquez; name: Sai Rajeswar
Published: 2024-12-05 21:41:20

Abstract: Multimodal AI has the potential to significantly enhance
document-understanding tasks, such as processing receipts, understanding
workflows, extracting data from documents, and summarizing reports. Code
generation tasks that require long-structured outputs can also be enhanced by
multimodality. Despite this, their use in commercial applications is often
limited due to limited access to training data and restrictive licensing, which
hinders open access. To address these limitations, we introduce BigDocs-7.5M, a
high-quality, open-access dataset comprising 7.5 million multimodal documents
across 30 tasks. We use an efficient data curation process to ensure our data
is high-quality and license-permissive. Our process emphasizes accountability,
responsibility, and transparency through filtering rules, traceable metadata,
and careful content analysis. Additionally, we introduce BigDocs-Bench, a
benchmark suite with 10 novel tasks where we create datasets that reflect
real-world use cases involving reasoning over Graphical User Interfaces (GUI)
and code generation from images. Our experiments show that training with
BigDocs-Bench improves average performance up to 25.8% over closed-source
GPT-4o in document reasoning and structured output tasks such as
Screenshot2HTML or Image2Latex generation. Finally, human evaluations showed a
preference for outputs from models trained on BigDocs over GPT-4o. This
suggests that BigDocs can help both academics and the open-source community
utilize and improve AI tools to enhance multimodal capabilities and document
reasoning. The project is hosted at https://bigdocs.github.io .