ICML 2025

CHAMELEON: A Flexible Data-mixing Framework
for Language Model Pretraining and Finetuning
Wanyun Xie 1 Francesco Tonin 1 Volkan Cevher 1
Abstract
Training data mixtures greatly impact the gen-
eralization performance of large language mod-
els. Existing domain reweighting methods often
rely on costly weight computations and require
retraining when new data is introduced. To this
end, we introduce a flexible and efficient data
mixing framework, CHAMELEON, that employs
leverage scores to quantify domain importance
within a learned embedding space. We first con-
struct a domain affinity matrix over domain em-
beddings. The induced leverage scores determine
a mixture that upweights domains sharing com-
mon representations in embedding space. This
formulation allows direct transfer to new data by
computing the new domain embeddings. In ex-
periments, we demonstrate improvements over
three key scenarios: (i) our computed weights im-
prove performance on pretraining domains with
a fraction of the compute of existing methods;
(ii) CHAMELEON can adapt to data changes with-
out proxy retraining, boosting few-shot reason-
ing accuracies when transferred to new data; (iii)
our method enables efficient domain reweight-
ing in finetuning, consistently improving test per-
plexity on all finetuning domains over uniform
mixture. Our code is available at https://
github.com/LIONS-EPFL/Chameleon.
1. Introduction
Pretraining large language models (LLMs) relies heavily
on vast and diverse datasets, encompassing sources such as
academic papers, books, and code repositories (Brown et al.,
2020; Dubey et al., 2024). The composition of these datasets
1Laboratory for Information and Inference Systems, École
Polytechnique Fédérale de Lausanne (EPFL), Switzerland. Corre-
spondence to: Wanyun Xie <wanyun.xie@epfl.ch>.
Proceedings of the 42 nd International Conference on Machine
Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025
by the author(s).
significantly influences the generalizat...