MixtureVitae: Open Web-Scale Pretraining Dataset With High Quality
  Instruction and Reasoning Data Built from Permissive-First Text Sources
Authors: name: Huu Nguyen, name: Victor May, name: Harsh Raj, name: Marianna Nezhurina, name: Yishan Wang, name: Yanqi Luo, name: Minh Chien Vu, name: Taishi Nakamura, name: Ken Tsui, name: Van Khue Nguyen, name: David Salinas, name: Aleksandra Krasnodębska, name: Christoph Schuhmann, name: Mats Leon Richter, name: Xuan-Son, name: Vu, name: Jenia Jitsev
Organization: Ontocord.AI

We present MixtureVitae, an open-access pretraining corpus built to minimize
legal risk while providing strong model performance. MixtureVitae follows a
risk-mitigated sourcing strategy that combines public-domain and permissively
licensed text (e.g., CC-BY/Apache) with carefully justified low-risk additions
(e.g., government works and EU TDM-eligible sources), alongside targeted
instruction, reasoning and synthetic data with documented provenance. We detail
a transparent, multi-stage pipeline for license-aware filtering, safety and
quality screening, and domain-aware mixing, and we release the dataset and
curation recipes to support reproducible research. In controlled experiments
using the open-sci-ref training protocol (fixed architectures at
130M/400M/1.3B/1.7B parameters; training budgets of 50B and 300B tokens),
models trained on MixtureVitae consistently outperform other permissive
datasets across a suite of standard benchmarks, and at the 1.7B/300B setting
they surpass FineWeb-Edu and approach DCLM in the later stages of training.
Performance is particularly strong on math/code and competitive on QA tasks.
These results demonstrate that permissive-first, risk-mitigated data provides a
practical and legally mitigated foundation for training capable LLMs, reducing
reliance on indiscriminate web scraping without sacrificing competitiveness.
Code: https://github.com/ontocord/mixturevitae