MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality
  Instruction and Reasoning Data Built from Permissive-First Text Sources
Authors: name: Huu Nguyen; name: Victor May; name: Harsh Raj; name: Marianna Nezhurina; name: Yishan Wang; name: Yanqi Luo; name: Minh Chien Vu; name: Taishi Nakamura; name: Ken Tsui; name: Van Khue Nguyen; name: David Salinas; name: Aleksandra Krasnodębska; name: Christoph Schuhmann; name: Mats Leon Richter; name: Xuan-Son; name: Vu; name: Jenia Jitsev
Published: 2025-09-29 21:40:10

Abstract: We present MixtureVitae, an open-access pretraining corpus built to minimize
legal risk while providing strong model performance. MixtureVitae follows a
risk-mitigated sourcing strategy that combines public-domain and permissively
licensed text (e.g., CC-BY/Apache) with carefully justified low-risk additions
(e.g., government works and EU TDM-eligible sources), alongside targeted
instruction, reasoning and synthetic data with documented provenance. We detail
a transparent, multi-stage pipeline for license-aware filtering, safety and
quality screening, and domain-aware mixing, and we release the dataset and
curation recipes to support reproducible research. In controlled experiments
using the open-sci-ref training protocol (fixed architectures at
130M/400M/1.3B/1.7B parameters; training budgets of 50B and 300B tokens),
models trained on MixtureVitae consistently outperform other permissive
datasets across a suite of standard benchmarks, and at the 1.7B/300B setting
they surpass FineWeb-Edu and approach DCLM in the later stages of training.
Performance is particularly strong on math/code and competitive on QA tasks.
These results demonstrate that permissive-first, risk-mitigated data provides a
practical and legally mitigated foundation for training capable LLMs, reducing
reliance on indiscriminate web scraping without sacrificing competitiveness.
Code: https://github.com/ontocord/mixturevitae