Data-Juicer Sandbox: A Comprehensive Suite for Multimodal Data-Model
  Co-development
Authors: name: Daoyuan Chen; name: Haibin Wang; name: Yilun Huang; name: Ce Ge; name: Yaliang Li; name: Bolin Ding; name: Jingren Zhou
Published: 2024-07-16 14:40:07

Abstract: The emergence of large-scale multi-modal generative models has drastically
advanced artificial intelligence, introducing unprecedented levels of
performance and functionality. However, optimizing these models remains
challenging due to historically isolated paths of model-centric and
data-centric developments, leading to suboptimal outcomes and inefficient
resource utilization. In response, we present a novel sandbox suite tailored
for integrated data-model co-development. This sandbox provides a comprehensive
experimental platform, enabling rapid iteration and insight-driven refinement
of both data and models. Our proposed "Probe-Analyze-Refine" workflow,
validated through applications on state-of-the-art LLaVA-like and DiT based
models, yields significant performance boosts, such as topping the VBench
leaderboard. We also uncover fruitful insights gleaned from exhaustive
benchmarks, shedding light on the critical interplay between data quality,
diversity, and model behavior. With the hope of fostering deeper understanding
and future progress in multi-modal data and generative modeling, our codes,
datasets, and models are maintained and accessible at
https://github.com/modelscope/data-juicer/blob/main/docs/Sandbox.md.