GRAPE: Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
Authors: Simin Fan
Maria Ios Glarou
Martin Jaggi
Venue: NeurIPS 2025 poster ()
Keywords: LLM pretrain
data selection
Primary area: deep_learning
TL;DR: This paper introduces GRAPE, a novel multi-source-multi-target domain reweighting framework designed to calibrate pretraining data mixtures for robust performance across multiple target tasks simultaneously.

Abstract: The performance of large language models (LLMs) across diverse downstream applications is fundamentally governed by the quality and composition of their pretraining corpora.
Existing domain reweighting algorithms primarily optimize data mixtures for a single target task, thereby resulting in models that overfit to specialized objectives while exhibiting substantial performance degradation on other benchmarks.
This paper introduces $\textbf{G}$roup $\textbf{R}$obust Multi-target $\textbf{A}$daptive $\textbf{P}$r$\textbf{E}$training (GRAPE), a novel multi-source-multi-target domain reweighting framework designed to calibrate pretraining data mixtures for robust performance across multiple target tasks simultaneously.
GRAPE dynamically adjusts sampling weights across source domains ($\textit{domain weights}$) while concurrently modulating $\textit{task weights}$ that quantify the relative importance of each individual target task.
This adaptive process prioritizes tasks based on their learning difficulty throughout training. 
We formulate this interleaved reweighting mechanism as a minimax optimization problem: 
The inner maximization adjusts task weights leveraging group distributed-robust-optimization (DRO), where those tasks demonstrating the least improvement under the current data mixture are prioritized with higher weights; 
The outer minimization then optimizes domain weights to maximize loss reduction on the prioritized tasks.
Experiments on $\texttt{ClimbLab}$ and $\texttt{SlimPajama}$ datasets demonstrate that GRAPE consistently outperforms baseline methods in terms of reasoning accuracies across 6 benchmarks. 
Furthermore, when applied to multilingual targets, GRAPE effectively identifies optimal training mixtures from mainstream languages, achieving superior language modeling capabilities across 8 low-resource target languages.

GRAPE Optimize Data Mixture for Group Robust Multi-target Adaptive Pretraining
Simin Fan, Maria Ios Glarou, Martin Jaggi
MI.0, EPFL
firstname.lastnameeee wastsmeeeeetmmn..ccccmmmmm ch
Abstract
The performance of large language models (LLMs) across diverse downstream applications is fundamentally governed by the quality and composition of their pretraining corpora. Existing domain reweighting algorithms primarily optimize data mixtures for single target task, thereby resulting in models that overfit specialized objectives while exhibiting substantial performance degradation on other benchmarks. This paper introduces Group Robust Multi-target Adaptive domain reweighting
PrEtraining (GRAPE), a novel framework designed calibrate pretraining data mixtures for robust performance across multiple target tasks simultancously. GRAPE dynamically adjusts sampling weights across source domains (domain weights) while concurrently modulating task weights that quantify the relative importance of each individual target task. This adaptive process prioritizes tasks based on their learning difficulty throughout training. We formulate this interleaved reweighting mechanism as minimax optimization problem: The inner maximization adjusts task weights leveraging group distributed- wewrbbuuued-rbbuttoo-oo cobbut-optimizztionn (DRO), where those tasks demonstrating the least improvement under the current data mixture are prioritized with higher weights; The outer minimization then optimizes domain weights to maximize loss reduction on the prioritized tasks. Experiments on ClimbLab and S1 jlimpajjan ama datasets demonstrate that GRAPE consistently outperforms baseline methods in terms of reasoning performance across 6 benchmarks. Furthermore, when applied multilingual targets, GRAPE effectively identifies optimal training mixtures from mainstream languages, achieving superior language modeling capabilities across 8 low-resource target languages.
1 Introduction
The composition of pretraining data, typically encompassing diverse sources, topics, and languages [Gao et al., 2020, Together Computer, 2023], is a critical determinant of large language model (LLM) capabilities. While various data sources are conventionally combined using fixed, heuristically determined mixing ratios, recent research indicates that adaptively adjusting these ratios (domain weights) via domain reweighting during pretraining can substantially enhance downstream task performance [Xie et al., 2023, Liu et al., 2025, Fan et al., 2024a]. However, prevailing domain reweighting methods are largely confined to the multi-sourree-ee single-targe paradigm, focusing exclusively on source-target relevance while failing address complex interactions across multiple target tasks. In particular, task-adaptive domain reweighting approaches [Grangier et al., 2025, Fan al., 2024b] curate data mixtures based performance one specific task, while often leading overfitting with compromised generalization ability. General domain reweighting techniques such Liu et al. [2025] and Fan al. [2024a] adjust training data mixtures for average performance across various target tasks, assessed via mean stochastic task losses or gradients. Yet, due to the heterogeneous dynamics of multi-task learning, average assessment proves suboptimal when confronted with dominant gradients where loss or gradient magnitude on certain tasks overshadow others, or conflicting gradients where improving one task hinders another. Conversely, traditional multi-task learning (MTL) strategies like gradient surgery [Yu et al., 2020, Liu et al., 2023, Bohn et al.,
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

2024] require substantial data from each target task and often involve complex gradient manipulations that scale poorly to foundation models, rendering them infeasible for LLM pretraining.
To facilitate effective multi-task adaptation during LLM pretraining, we propose Group Robust Multi- target Adaptive PrEtraining (GRA (GRAPE), the first domain reweighting algorithm explicitly designed for the munlti-sourc-mmuii--ee target setting. Instead of optimizing for average task performance, GRAPE calibrates domain weights to maximize improvement on the most challenging tasks each stage of training, guided by the key principle:
Tasks that are improving slowly deserve more attention.
(DRO) [Duchi and Namkoong, 2020, Sagawa et al., 2020] to adaptively assign higher task weights to target tasks that are currently learning slowest. We introduce the Rate-of-Improvennee (Rol, Equation 1), a normalized, step-wise metric assessing task-specific learning progress. Unlike prior metrics like excess loss [Xie et al., 2023], Rol avoids reliance on auxiliary models and inherently accounts for varying task difficulties by normalizing improvement against current performance.
Specifically, GRAPE employs group
We formulate this bifold reweighting process as a minimax optimization problem (Equation 2), jointly optimizing: (1) domain weights (at,, the sampling distribution over source domains; and (2) task weights (z). a distribution over target tasks reflecting their priority during the evolution of training. The inner maximization phase uses DRO to identify the task distribution = that emphasizes the least-improved tasks (minimal Rol). Tasks demonstrating minimal Rol under the current domain mixture are thus prioritized in subsequent optimization steps; The outer minimization phase then optimizes the domain weights a to maximize the expected improvement according to this dynamically determined worst-case task distribution. This interleaved mechanism enables GRAPE to balance domain utilization against multi-task performance adaptively via a dynamic curriculum, preventing dominance by any single task and fostering robust generalization.
Contributions. Our primary contributions are threefold:
We introduce APAPE, the first principled multi- target domain reweighting algorithm for LLM pretraining. We provide a rigorous derivation grounded in optimization theory, including a theoretical analysis of its properties ($ 2).
We empirically validate GRAPE's efficacy, demonstrating superior performance over baseline methods on reasoning benchmarks (S 3.1) and low-resource multilingual language modeling ($ 3.2).
Through comprehensive ablation studies and comparisons of different progress assessment metries used in DRO, we offer insights on the training dynamics of multi-target domain reweighting, shed light on the future research in data-centric LLM pretraining and curriculum learning ($ 4).
2 Methodology: Multi-Targed Domain Reweighting via GRAPE
This section formalizes the problem of domain reweighting for multi-target adaptation and presents GRAPE, our proposed algorithm that optimizes pretraining data composition for robust performance across multiple target tasks.
Problem Formulation. Let Dirain (D1, ..... DR) represent a pretraining corpus partitioned into k source domains based on meta-atrributes (e.g., source, topic, etc.). During training, data is sampled from Dusin according to domain weights a A where A is the probability simplex in R* Let Der A [Ti...., TN) } be a of N target tasks, which can be presented as different benchmarks. Our objective is to find the optimal domain weights & that yield strong performance across all tasks Tn E Digr- We introduce lask weighıs z E A C R-V represent the relative importance assigned to each target task during training. The core idea is to dynamically update z to reflect task learning progress and subsequently adjust a to optimize the training trajectory based on z.
Let O, denote the model parameters at step t, and let I, (0.) be the stochastic loss evaluated on target task T, for n E [N]. We assume optimization proceeds via gradient descent: 0.1 0, de where 7 is the learning rate and d, is the update direction. Let 1 ( ). gle (0:) be the stochastic loss and gradient computed data from training domain D2 for i E [K]. The overall parameter update direction is weighted combination of stochastic domain gradients: d, == Σk=1 aggg: where the domain weights a' are used as the dynamic importance sampling weights at step t.

---

2.1 GRAPE: Prioritizing the Lagged Tasks by Group DRO
Algorithm. To achieve balanced improvement across target tasks, GRAPE employs Group Dis- tributed Robust Optimization (Group DRO) to jointly adapt task weights (z) and domain weights (ax). We assess task improvement using the Imprrvemmmm (Rof), inspired by Liu al. [2023]:
L, (O,) 1, (0++1)
(0.)
Here, (o,) - L, (0++1) approximates the one-step improvement in loss for task Tw Normalizing by L,, (0,) provides scale-invarian measure of relative progress, preventing bias towards tasks with intrinsically higher loss magnitudes. For sufficiently small learning rates %t a first-order Taylor expansion yields:
1 (oln((.)) md) + l1?)
K
(V log (o di) % a (Vo log I, (. (V)
Ln (0,)
where Vo log In (0.) = (0... Following the principles of Group DRO [Sagawa et al., 2020, Qi
(.,
et al., 2021], we formulate a minimax objective to optimize the domain weights (a) for the worst-case (minimum) Rol score across tasks:
N Σ
max min E|r[0] x max min aEAR ME[N]
aEAk ZEAN 2n
E
-
-i(a) + h(z)
(2)
n=1
N
max min 7 aE #EA
Okk
Σ n=1
(o (0:). M(a)) + h(2).
k=1
Here, the inner minimization finds the task weights z that concentrate on the tasks with the lowest expected Rol: The outer maximization finds the domain weights a that maximize this minimum expected Rol. Following Qi et al. [2021], Namkoong and Duchi [2016], we introduce Bregman divergence regularization terms l((x) and lz(z) relative to the previous weights a (t-1) and ((t-1) to stabilize the updates and manage stochastic variance:
T((a) == Pa Dv D (t)laa h(2) De
where (bbb E, by log(b:) is the negative entropy. Solving this regularized minimax problem (details in Appendix B) yields multiplicative update rules resembling online mirror descent:
21
with 2t + 2/ exx (-
Itt E log (O.), Vo((
eepp
Ene(N 2 a' with af at1 exp ΣkE[K] ag
a'
Tt [ Ve log (0.
Pl
In practice, the expectations are replaced by stochastic estimates using mini-batches. The inner product (Vo log ( (0:)) captures the alignment between the gradient direction of task e [N] (normalized by loss) and the gradient direction of domain k E [K]. Vol(O,; a)) denotes the (training loss) gradient from a training batch æ sampled according to "-1 and Vo log ((0,;; y) denotes the tesk-weighted normalized validation gradient from a validation batch y sampled according to zz-1 The pscudocode is presented in Algorithm
Efficiency analysis. Updating a and z at every step (Equation 3, Equation 4) requires N target gradients and K domain gradients computations, which computationally expensive. We mitigate this by performing updates periodically, where domain weights (ar and task weights (z) are updated every AT; and ATa steps. For our experiments on SlimPaj ama with K=7 and N=6, using AT, 100, AT a = 100, the computational overhead is approximately (N (N+11 +T + (K + AT; -15% terms of gradient computations relative to standard training. Compared to prior domain reweighting algorithm DoGE [Fan et al., 2024a], our task reweighting step only introduce (N -1AAT-1=777 overhead computations. The memory overhead stems primarily from storing gradients for each domain/task during the update step. If model parameters require storage, Adami optimizer stores
3

---

Algorithm 1 Group Robust Multi-target Adaptive PrEtraining (GRAPE)
1: Input: Training domains Duain - (Di...., D target task validation sets Dir - (Ti.....TAy ]w optimizer Optimizer, loss function I(+), initial parameters 0º, learning rate schedule "o. initial weights a², =", update frequencies AT,, AT,, regularization coefficients Ha. total steps T.
2: for E (0... do
# Standard training: update model parameters 0
3: Sample a batch from Deain: x ~ mix((x')
4: Update model parameters 0'+1 +- Optimizer Vot(O',
5: if t%AT= 0 then
# Task Reweighting: update task weights
6:
Sample one batch from each target task: yn ~ Tn Deg g for E [M] Sample one batch from Denin: x ~ mix(a")
7:
Compute gradient alignments: a', f (Vo log E(0 :y,) Vol(0++); ;x))
with 2 = z' @ exp Tt a')
9:
Update task weights: x"
Σn= 2p
10:
else
t-
11:
end if
12:
if
A%AT, 0 then
# Domain Reweighting: update domain weights
13:
Sample one batch from each domain: 26 ~ D2 E Dusin for ls 6 [K]
14:
Sample one batch from Pigi y mix(=')
15:
Compute gradient alignments: a + Vol(0't);=a)) Vo log E(0 4)
Update domain weights: a (+1 á
with á = a' 3 exp Te a')
16:
-==1 de
pa
17:
else
a1+11 ax
18:
end if
19:
end for
20:
Return Model parameters 0
x m (model parameters, gradients, and two EMA states). The peak memory usage at reweighting steps stores one additional gradient (m)) leading sa 25% memory overhead.
Interpreting the Reweighting Mechanism. The update rules (Equation 3, Equation 4) reveal an elegant negative feedback regulation at the heart of GR A APE's reweighting mechanism: Task weights on increase for tasks where the normalized gradient V log E(O, y) has low alignment with the overall update direction d,, indicating slow progress under the current regime (Equation 3). Subsequently, domain weights a, increase for domains whose gradients gi align well with the normalized gradients of these prioritized, struggling tasks (represented by V log E(O, y) where y ~ mix(2)) (Equation 4). This mechanism continuously directs training focus towards underperforming tasks by upweighting relevant source domains. The full derivation is provided in Appendix B.
2.2 Theoretical Properties of GRAPE
The GRAPE algorithm, formulated as a regularized minimax optimization problem solved via multiplicative updates, possesses desirable theoretical properties under standard assumptions. The update rules (Equation 3. Equation 4) are instances of online mirror descent applied to the variables (a, x) of the minimax game. Theorem 2.1 suggests that under strong convexity and smoothness assumptions, the algorithm converges towards the set of Pareto optimal solutions at an OLLTT) rate, where T is the number of training iterations. However, the strong convexity assumption. In practical non-convex scenarios of L.M pretraining, convergence guarantees are typically weaker. The proof of Theorem 2.1 is presented in Appendix C.I.
acknowledge the limitation imposed by
Theorem 2.1 (Convergence of GRAPE). Let the loss functions I (0) be L-smooth for all n [N] and the norm of stochastic gradients be upper- bounddd by G. If the learning rate satisfies Y S and the regularization parameters Paa Pz are chosen such that faa algorithm with update rules given by the above equations converges to a neighborhood of the Pareto optimal solution rate of O(1/T) where T the number of training iterations. Specifically, for any >> O, there exists To such that for all T > To:
and μs > then the GRAPE
min (E
max [O,) min max (n(()) SE NE[N]
0 NE[N]

---

Furthermore, Theorem 2.2 posits that under similar assumptions of smoothness and bounded gradients, the variance of task performances, of Varne([) (0))). tends to be monotonically decrease after an initial phase 0:+1 I of for t 2 To. It indicates that the algorithm actively counteracts the divergence of the task performances, which promotes more uniform progress across the task suite compared to static weighting or simple averaging. The proof is presented in Appendix C.2.
Theorem 2.2 (Monotonic Variance Reduction of Task Performance). Let of = VVannein] (I,(()))) (0))
denote the variance of task performances at iteration I. Let the loss functions I, (0) be I-smooth for all [N] and the norm of stochastic gradients be upper-bounder by G and assuming the task losses are L-smooth and y-strongly convex, the variance decreases monotonically until reaching a minimal basin, Le., 0441 S of for all t To for some finite To-
3 Experiments
To evaluate the efficacy of GRAPE, we conduct experiments in two distinct multi-target pretraining scenarios: (1) optimizing a generic English language model for diverse reasoning tasks, and (2) optimizing the data mixture from mainstream languages for low-resource language modeling, For all experiments, we train various scales of decoder-only transformers following Vaswani et al. [2023] from scratch. More details on architecture and hyperparameters are provided in Appendix D.1.
3.1 Domain Reweighting for Multi-task Reasoning
Setup. We consider two pretraining corpora, Cl ClimbLab [Diao et al., 2025] with K=20 source domains clustered by topics: and SlimPaj ama with K=7 domains classified by collection sources. We first apply GRAPE to standard English language model pretraining, where the objective is to dynamically adapt the data mixture to maximize performance across suite of reasoning bench- marks simultaneously. We select N=6 diverse reasoning tasks spanning scientific, logical, physical, and commonsense reasoning: ARC-Easy (ARC-E), ARC-Challenge (ARC-C) [Clark et al., 2018], SeiQ [Welbl et al., 2017], PIQA [Bisk et al., 2019]. LogiQA [Liu et al., 2020], and HellaSwag [Zellers et al.,
S-shot Accuracy
2019]. For each target task T.. use its standard vali- dation set to compute the task loss I, (6.) and the Rate-of-
Improvement (t) needed for GRAPE's updates during training. We update task weights & every AT= = 100
Valform
CRISP
steps and domain weights every ATa 100 steps. Ini- tial weights a and 2 are set to uniform distributions. We use the AdamW optimizer with standard hyperparameters for LLM pretraining. Experiments were conducted using 4 H100 80GB GPUs. Implementation details and hy- perparameter settings are provided in Appendix D. We report the results on ClimbLab in the following sections and present the results on SlimPajama in subsection D.3.
RogMix
<limbmix
0.32
0.61
0.96
1.28
1.6
Training Tokens
(B)
(a) 125M-Climblaab
S-bbot Accuracy
0.48
Baselines. On experiments with 0.7 B scale models, we compare GRAPE to two baseline methods: (1) Uni- form: training data sampled uniformly from each source domains; (2) DoGE [Fan et al., 2024a]: domain weights are dynamically adjusted by gradient alignment with uniform task weights across N target tasks. On small- scale (125M) models, we employ comprehensive suite of baseline approaches: (3) DoGF-PCGrad: a multi-task learning extension of DOGE, where the target gradient are dynamically updated with a gradient surgery algo- rithm PCGrad [Yu et al., 2020]: (4) RegMix [Liu et al., 2025]: predict the domain weights using regression to- wards the optimal average loss across all target tasks; (5) CRISP [Grangier et al., 2025]: determine the domain weight distribution by importance sampling, i.e. clustering all data samples from target tasks onto source domains; (6) ClimbMix [Diao et al., 2025]: the optimized domain weights curated by itera- tively regression and bootstrapping: (7) GRAPE CIIMMMIX: apply GR A PE with domain weights initialized from ClimbMix weights
0.48
0.44
Uniform
DoGE
0.42
GRAPE
1.3 2.6
3.9
5.2
6.5 7.8 9.1
10.411.7 13
Training Tokens (3)
(b) 17B--limbLab
Figure 1: GRAPE facilitates multi-task reasoning, For 125M models, GRAPE and GRAPE-climbni greatly outperform five baselines: For larger 0.7B models, GRAPE achieves comparable scores as uniform hase- line with 40% fewer tokens.
5

---

GRAPE improves multi-task reasoning capability. According to Table and Figure 1, GRAPE demonstrates consistent improvements on multi-task reasoning capabilities across different model scales, achieving superior average 5-shot reasoning accuracy and learning efficiency compared to various baseline methods. According to Figure 3.1, GRAPE and GRAPF-climbbix, significantly outperform a suite of baselines including DoGE, DoGE-PCGR.N
ClimbMix, RegMix, CRISP, and uniform domain sampling when training small-scale (125M) models. On 0.7B models (Figure 3.1), both DoGE and GRAPE substantially outperform the uniform sampling baseline, demonstrating the general efficacy of adaptive domain reweighting approaches. Notably, GRAPE achieves comparable average 5-shot accuracies to the uniform baseline while utilizing approximately 40% fewer training tokens. Furthermore, compared to DOGE, GRAPE not only exhibits an approximate 25% accel- eration in reaching similar accuracy thresholds, but also obtains more well-rounded improvements across all tasks above uniform baseline, as benefit from its distinctive task reweighting mechanism.
Table 1: 5-shot exact match accuracies( %) on target reasoning tasks. The scores of 125M (resp. 0.7B) models trained on 1.6B (resp. 7.8B) tokens are reported. The best-performed scores are marked as Bold, and the second-bes scores are Underlined. GRAPE outperforms other baselines on most of target reasoning tasks.
125M-ClimbLab
ARC-C
ARC-E
LogiQA
PIQA
SciQ
Hellaswag
Averuge
# tusk uniform buse.
Uniform
23.12
40.74
27. 27.96
58.98
67.50
27.02
40.89
DOGE
24.57
45.33
25.35
59.52
72.20
27.77
42.45
5
REGMIX
24.66
42.85
26.11
59,30
65.90
27.30
41.02
4
CRISP
22.70
44.32
29.65
57.34
70.40
27.40
41.97
CLIMBMIX
24.83
42.85
25.19
60.55
70.80
27.67
41.98
5
DOGE-PCGRAD
24.66
42.63
27.19
58.71
70.50
27.50)
41.86
4
GRAPE
26.11
47.14
27.65
61.10
74.40
28.56
44.16
26.02
46.72
27.19
62.24
73.80
28.01
43.99
0.7B- LBB-ClibbLab
ARC-C
ARC-E
LogiQA
PIQA
SciQ
Hellaswag
Averuge
task > uniform
Uniform
28.07
52.99
28.57
62.73
84.00
32.33
48.11
DOGE
29.61
57.58
25.96
61.70
86.40
31.05
48.72
GRAPE
28.92
55.60
28.58
65.56
84.50
34.19
49.56
Task weights evolution reflects learning curriculum. The evolution of task weights (z,) on 0.7B model, depicted in Figure 2, reveals the dynamic learning curriculum adopted by GRAPE. In the early stage, the reading compre- hension tasks like ARC-E and ARC-C are mostly up-weighted. As training pro- gresses, in the late stage, physical and com- monsense reasoning tasks like PIQA and Hellaswag are steadily prioritized, demon- strating the emergence of a skill-wise learn- ing curriculum that moves from founda- tional to more complex reasoning abilities. Concurrently, tasks like SciQ and LogiQA consistently receive lower weights. This suggest they may either benefit sufficiently from cross cross-task generalization driven by the prioritized tasks or can be learnt more efficiently relative to other tasks. This adaptive curriculum, by directing resources to evolving bottlenecks, ensures that difficult tasks are not neglected, fostering robust and balanced multi-task reasoning capabilities.
W
Weights
ARC-E
04
ARC-C
ScQQ
PIQA
02
LogitA
Hellaswag
0.0
25%
500
75k
100k
Oplimization Steps
Figure 2: Task weight evolution of GRAPE.
0.02
0:36
8.01
5.30
8.02
0.50
0.00
0.
0.
2.03
0.03
0.02
0.00
2.0L
2.05
3.03
0.03
0.00
topls
6.0z
n0t
0:
0.
5:01
5.85
5.02
0.42
0.21
0.00
5:00
5.02
003
0.83
0.22
2.01
9.02
0.63
0.00
4:24
CAARR Dhppt ctter
0:03
002
0.00
0.66
6.02
2.15
5.03
0.63
0.24
0.62
3/3
2.04
009
0.02
0.62
2/00
2.01
0.27
6.1
6.02
0.02
1.20
8.18
0.62
0.01
0.67
1.2
1.23
082
0.82
0.00
20
2.00
0.02
0.11
1:00
0.50
3:00
5.33
001
0.83
0.66
2:00
2/00
200
67
c12
cl:
432
033
Cb
cia
C17
c20
Claxter
Figure 3: Domain weights attributions across 20 clusters the ClimbLab dataset.
Which data domains and topics are critical for general reasoning? Figure presents the domain weight distributions across different methods across 20 data clusters within the ClimbLab aa corpus, whose main topics are detailed in Table GRAPE notably assigns high importance on Cluster 1,4, 10, which primarily focus broad science, mathematics and education contents. It also

---

explains the constant low-priority on SciQ task indicated in Figure 2 since the training data mixture contains substaintial scientific-QA related contents. In contrast, other methods exhibit different topical biases: DoGE shows a strong preference on Cluster 6, featuring healthcare and genetic contents; REGMIX heavily utilizes Cluster 17 which with topics on health and medical research; CRISP mostly favors Cluster 11, indicating great emphasis on software and programming. Notably, CRISP achieves significant improvement on LogiQA task, which indicates training on code data is critical to improve the logical reasoning ability of language models.
uk
9.5
9.50
9.25
9.0
99.00
7,5
8.75
1.55
0.30
0.52
0.31
6.59
3.52
6.26
0.31
6.45
2113
6.5
016
m1
9.0
8.0
Uniform
8.5
9.0
DoGE
7.0
GRAPE
8.0
7.0
8.5
7.5
6.5
8.0
335
126
0.39
0.52
0.55
0:13
0.25
030
2.52
2.00
0.33
0.26
0.23
0.52
0.55
0.13
2.25
6:39
052
0.45
Truining
Figure 4: Low-resource language learning progress by Log-Perplexity. GRAPE significantly outperforms DoGE and Uniform sampling across all target languages.
3.2 Language Mixture for Multi-lingual Learning
Setup. In this scenario, we investigate GRAPE's ability to optimize the language composition of the pretraining corpus from mainstream languages to enhance language modeling capabilities across multiple low-resource target languages. The source corpus consists of data from K = 6 languages selected from the wiki40b dataset [Guo et al., 2020], including high-resource lan- guages English (en), French (fr), German (de). Spanish (es), Russian (ru) and Italian (it). Each language constitutes a domain D,, and GRAPE adapts the sampling weights a E A over these source languages. We select N=8 low-resource languages target tasks, distinet from the source languages: Turkish (tr). Danish (da), Catalan (ca), Polish (po), Romanian (ro), Portuguese (pt), Dutch (nl), and Ukrian (uk). Per- formance is measured by the language modeling loss, i.e. the log-perplexity (log-PPL.), held-out test sets for each target language. Lower log-PPL. indicates better performance.
0.15
GRAPE Enhances Multilingual Adaptation. Figure 4 illustrates the test log-perplexity each target low-resource language. GRAPE consis- tently outperforms both uniform sampling and DOGE basclines across all languages, effectively accelerating the low-resource language modeling. Specifically, GRAPE accelerates the low-resource language modeling by less than 60% in terms of log- perppeeeexity scores, achieving significantly lower final perplexity within the same training budget. While Fan et al. [2024a] showed DOGE greatly improve single-target language learning, its efficacy diminishes when confronted with mul- tiple target languages simultaneously, where it struggles to deliver competitive performance with uniform task weights. This outcome underscores the sub-optimalit of average task weighting strate- gies for multi-task learning, especially when tasks exhibit conflicting characteristics. Notably, accord- ing to Figure 20, the offline reweighting algorithm such as RegMix and CRISP exhibit a strongly biased performance across various target languages: while RegMix slightly facilitate the learning on most of targets languages including Catalan, Portuguese, Ukrainian and Polish, it sacrifices the method CRISP only accelerates the
21h Staps
(a) Train Domain Weights
0.200
0.155
0.190
0.100
pt
0.955
4x
1.kk
20.6
Stops
(b) Task Weights
Figure 5: Weights evolution on multilingual pre-
training for low-resource language modeling.
performance on Dutch and Danish; the purely embedding- learning Turkish while sabotaging all the other languages.
7

---

Weight trajectory reveal linguistic relations. The dynamic interplay between task weights (2:) for target languages (Figure 5, a) and domain weights (a,) for source languages (Figure b) offers valuable insights into GRAPE's adaptive learning strategy and its implicit discovery of inter-lingual relationships. For instance, the initial prioritization of target languages Ukrainian (uk; East Slavic) and Romanian (ro; Romance) corresponds with an increased weighting of linguistically related source languages, namely Russian (ru: East Slavic) and Italian (it; Romance). Subsequently, the sustained high task weight for Polish (pl: Balto-Slavic) aligns with the continued high domain weight of Russian, from the same broad linguistic family. For Turkish (tr), a Turkie language without close phylogenetic relatives within the predominantly Indo-European source domains, its persistent learning challenge is evident from its high task weight. Nevertheless, GRAPE facilitates its learning by strategically up- ap-weighting source languages like German (de) and French (fr), presumably due to their rich linguistic feature sets or similarity in syntax structures.
4 Discussion and Limitations
Progress measurements for Group DRO. We evaluate the impact of different step-wise progress metrics on the performance of Group DRO within the GRAPE framework. In addition to the primary Pate-of-Improveneet Improvement (GOI) and EMA--atteoffmmmrroemeen (ROI-ema). The step-wise improvmenet of task Tn at step are assessed as:
(ROI) metric (Equation 1), investigate alternatives: Gap-of-
GO = (0.) I. (0.+1), ROI-ema (0.) (0c+1)
Vn E [N]
(5)
16 mm
The exponential moving average loss lema. is updated as: l'ma,n -- (1-8)) where the hyperparameter /3 is set to 0.7 in our experiments. Substituting these alternative progress metrics for rn in the minimax objective Equation 2 yields modified update rules according to Equation 8 and 9. The core optimization principle for adjusting task weights (z) and domain weights (ax) remains, but the specific gradient terms within the exponents change. More details are provided in Appendix 1.
are easy
are challenge
sciq
Results on the six reasoning tasks (Figure 6) indicate that us- ing GOI (GRAPE-gai) prior- itizes intrinsically easier tasks. This is attributed the fact that tasks with larger loss values can exhibit larger absolute improve- ments (0,) - Ln (O.+1)) while their relative progress is slow. The ROI metric, by normaliz- ing improvement with the cur- rent loss 4(0)). effectively mit- igates this bias, offering a fair as- sessment of task learning progress less biased to easy ones. In comparison, the ROI-ema metric, which normalizes the absolute improvement by an exponential moving average of past losses, introduces a historical dependency. While this can smooth out noisy step-wise improvements, our experiments suggest may also lead to slower adaptation of task weights since the smoothed historical loss might not accurately reflect the current learning state. Comprehensive results across all 12 evaluated task combinations are provided in Appendix E, further substantiating these observations.
Weughts
0.75
0.4
0.50
6.2
0.25
0.0
0.0
4k
Lin
20%
46
66
1ak
146
24k
46
86
124
30c Steps
Steys
Steps
pigs
logigs
hellaswag
0.6
0.3
0.2
0.4
0.2
N6008"
GBATE
0.2
0.1
0.1
L22
16x.
20%
4
12
164
28k
124
35c 23k
Stops
Steps
Steps
Figure 6: Task weights evolution on reasoning tasks.
GRAPE outperforms baselines on diverse sets of target tasks. We further assess the robustness and adaptability of GRAPE on 12 diverse sets of target tasks with varying degrees of diversity and complexity. The task configurations and detailed results are presented in Appendix E. According to Figure 27, to support both code comprehension (Kodcode) and mathematical reasoning (GSM8K) alongside commonsense reasoning task (Hellaswag), GRAPE effectively balance code-specine sources (e.g. GitHub, Stackexchange) and technical web text (e.g. CommonCrawl and C4) for mathematical reasoning with broader corpora. With high task diversity on 8 tasks Figure 28 where commonsense reasoning tasks are predominant, GRAPE largely upweights of general knowledge sources like the Book domain to cater to the majority need, while maintaining smaller yet significant allocation from domain-specific sources ge.g., GitHub, StackExchange). The comprehensive results in Table 2 and Appendix E demonstrate that GRAPE maintains strong average performance
8

---

diverse task combinations with various complexities, outperforming baselines that struggle to cater to disparate learning objectives with fixed data mixtures and uniform task priorities.
Table 2: Average test log-perplexity (1) on 12 task combinations. GRAPE and the variants outperform uniform and DoGE across all task sets; on 7 out of 12 tasks, GRAPE outperforms other two DRO variants.
Method
T1
T2
T3
T4
T5
T6
T7
T8
T9
T10
T11
T12
Uniform
4.30
4.01
4.11
3.75
3.85
3.62
3.20
3.90
4.33
4.62
4.81
4.53
DOGE
4.18
4.94
4.00
3.66
3.70
3.63
3.12
3.83
4.13
4.44
4.57
4.33
GRAPE
3.53
3.61
3.70
3.39
3.36
3.03
2.93
3.45
3.85
3.94
3.93
3.70
GRAPE-ema
3.68
3.69
3.73
3.37
3.37
3.09
2.96
3.53
3.88
3.91
3.90
3.73
GRAPE-gap
3.58
3.68
3.68
3.42
3.36
3.05
3.11
3.51
3.88
3.86
3.85
3.59
Hyperparameter tuning. The performance and efficiency of GRAPE is dependent on its hyper- parameters, including the regularization coefficients M4 and Maz, well the update frequencies ATa and A7,- Due to the high resource demands associated with LLM pretraining, an exhaustive hyperparameter sweep was beyond the scope of the current work. We posit that the performance of GRAPE can be further improved through extensive hyperparameter tuning, tailored to specific datasets and task configurations. How to determine theoretical near-optimal step-sizes of domain weights and task weights updates is a direction for future work.
Impact of training data quality and domain granularity. Our investigations reveal that the quality of the pretraining corpus and the granularity of its domain partitions critically influence the efficacy of domain reweighting algorithms, including GRAPE, DoGE, and offline methods such as REGMIX and CRISP. On ClimbLab with higher-qualin data and fine- grained semantic-basee based clustering, the domain weights exhibit more precise and impactful adjustments in response to the task curriculum compared to the coarser, source-defined domains of SlimPajama, where the benefits of reweighting can be less distinct. Notably, the offline reweighting algorithm such as RegMix and CRISP prone to over-connentrated on some particular domains, which leads to biased performance in multi-task learning. These findings underscore the importance of meticulously partitioned data domains for the benefits of adaptive data mixing strategies, suggesting potential direction for future research.
Towards better understandings of scaling effects on learning curriculum. The efficacy of domain reweighting are observed to vary significantly with model scale, besides the domain granularity of the training corpus. The 0. 0.7B model tended to develop highly concentrated focus single challenging task in later training stages, whereas the 125M model maintained a more distributed prioritization across several tasks (10). This suggests that the larger-scale model might be more sensitive to the conflicts between targets. Understanding these scaling effects, how the ideal curriculum changes with model size, and how its efficacy is bounded by data quality and partitioning, is crucial to developing more robust and universally effective adaptive pretraining strategies.
Towards a sample-level DRO framework for fine-grrinne reasoning. The current formulation of GRAPE applies Group Distributed Robust Optimization (DRO) at the task level, prioritizing data from entire target tasks that exhibit slower learning progress. A promising avenue for future research is the extension of this framework to sample-level DRO, where individual data instances (c.g.. specific questions examples) would be dynamically identified and assigned higher importance. By directing attention to the hardest samples, it allows the model to focus intensely on correcting specific errors in the mid- post-training stages, during interactions with evolving environment.
5 Conclusion
This paper introduced GRAPE, a novel multi-target domain reweighting algorithm designed to calibrate pretraining data mixture for robust performance across diverse target tasks. By dynamically adjusting domain and task weights through a Group DRO minimax framework, GRAPE effectively prioritizes tasks demonstrating the slowest improvement. Empirical results on challenging reasoning tasks and multilingual modeling showease GR AP'S significant advantages over existing baselines, highlighting the efficacy of its adaptive curriculum for multi-target learning.

---

References
Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, Shawn Presser, and Connor Leahy. The pile: An 800gb dataset of diverse text for language modeling, 2020.
Together Computer. Redpajama: An open source recipe to reproduce Ilama training dataset, 2023.
URL https://gi thtb. con/toggtthercomputeee jedPaj ama- ama-Data.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoe Le, Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining, 2023. URL https::///// http://aaavi...aa/////////e
10429.
Qian Liu, Xiaosen Zheng, Niklas Muennighoff, Guangtao Zeng, Longxu Dou, Tianyu Pang, Jing Jiang, and Min Lin. Regmix: Data mixture as regression for language model pre-training, 2025. URL.
hwwgg//bb/400/// 01492.
Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation, 2024a. URL pesttss:/arree www.oogg///.//./...o...ooo...o/o https//331.11111 15393.
David Grangier, Simin Fan, Skyler Seto, and Pierre Ablin. Task-adaptive pretrained language models via clustered-importance sampling, 2025. URL https /tt::///aaiiiii org/ abs//410......///// http///41.033555
Simin Fan, David Grangier, and Pierre Ablin. Dynamic gradient alignment for online data mixing, 2024b. URL https: btaps://arxxt org.
02498.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, 2020. URL https://arxiv org/abs//00/t 06782.
Bo Liu, Yihao Feng, Peter Stone, and Qiang Liu. Famo: Fast adaptive multitask optimization, 2023.
URL https: //arxi///xxiv org/ harg/abs/23000 03792.
Christian Bohn, Ido Freeman, Hasan Tercan, and Tobias Meisen. Task weighting through gradient projection for multitask learning, 2024. URL
http:///axvvvvvo
www.aasg////eooooooooo..ooeeeee/ ab////////// 017993
John Duchi and Hongseok Namkoong. Learning models with uniform performance via distributionally robust optimization, 2020. URL https: http///axvvvvv. http:/awiiiigaaeeaeee/eee orgooog////a ht/aaaas//111 habbb//8000085000
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization, 2020. URL https: / arxiv. http://awwtw.ooaaa////////aaaaa/ /19119911111 08731.
Qi Qi. Zhishuai Guo, Yi Xu, Rong Jin, and Tianbao Yang. An online method for a class of distributionally robust optimization with non-convex objectives, 2021. URL https:///aax org/abs/2006. 10138.
Hongseok Namkoong and John C Duchi. Stochastic gradient methods for distributionally robust optimization with f-divergences. D. Lee. M. Sugiyama, U. Luxburg, Guyon, and Gar- nett, editors, Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. URL. https: https://prooceeiings wew.eeeeeddiiieeeeeeeeeeeeeeeeee cccppaperfilleeppeeeeeeeeeeeeeee fitpsss/pppppp///01///////////// #111/55588744B011111887777744433 http://444cc.133dddddddddda----- pdf.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. URL https: //tp::////xxvii arxiv. org/org./a....ooao... abs/ 1706. 1706.03762. 03762.
Shizhe Diao, Yu Yang, Yonggan Fu, Xin Dong, Dan Su, Markus Kliegl, Zijia Chen, Peter Belcak, Yoshi Suhara, Hongxu Yin, Mostofa Patwary, Yingyan, Lin, Jan Kautz, and Pavlo Molchanov. Climb: Clustering-based iterative data mixture bootstrapping for language model pre-training, 2025. URL httph: wtwpv. org/ 2504/ 13161.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try are, the ai2 reasoning challenge, 2018. URL https: // arriiiiv org/ abs////s/.1../ 05457.
10

---

Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsourcing multiple choice science questions, 2017. URL https:://aat https://aare //tr///axvvvv//////////e///aaa org/ abs 110./100770000..
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about (rrg/bbs/1911 11641.
physical commonsense in natural language, 2019. URL https:
Jian Liu, Leyang Cui, Hanmeng Liu, Dandan Huang, Yile Wang, and Yue Zhang. Logiqa: A challenge dataset for machine reading comprehension with logical reasoning, 2020. URL https: // //arxiv... org/ /baaabbs/000/ 08124.
Rowan Zellers, Ari Holtzaan, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence?, 2019. URL https: ptattpps///riivv arxiv. org/abs/ /905 .78300.
Mandy Guo, Zihang Dai, Denny Vrandečié, and Rami Al-Rfou. Wiki-40B: Multilingual lan- guage model dataset. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, Christopher Cieri, Thierry Declerek, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mar- iani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors, Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 2440-2452, Marseille, France, May 2020, European Language Resources Association. ISBN 979-10-95546-34-4 URL org/2020. wrec- 1.297/
http:://acaatthhloo
Mayee F. Chen, Nicholas Roberts, Kush Bhatia, Jue Wang, Ce Zhang, Frederic Sala, and Christopher Ré. Skill-it! a data-driven skills framework for understanding and training language models, 2023. URL https::tttaareeeeee
org/ / /b3/ 14430.
Hang Wang, Minghao Xu, Bingbing Ni, and Wenjun Zhang. Learning combine: Knowledge aggregation for multi-source domain adaptation, 2020. URL https:: /tatp:://arivv. wwwriiii..g//aieeeeeee 08801.
org/ 220g
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training, 2023. URL http:::://ariii..rrr/////////// 02406.
Bo Liu, Xingchao Liu, Xiaojie Jin, Peter Stone, and Qiang Liu. Conflict-aversse gradient descent for multi-task learning, 2024. URL attpps//aarie
org/ abs//aall.../////////a//aa/aaa/a 14048.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks, 2018. URL https://
arxiv. arxiv.org/ain abs/ / 1111..1111 .22557.
Aviv Navon, Aviv Shamsian, Idan Achituve, Haggai Maron, Kenji Kawaguchi, Gal Chechik, and Ethan Fetaya. Multi-task learning as a bargaining game, 2022. URL 2202. 01017.
Jean-Antoine Désidéri. Multiple-gradient descent algoritt (MGDA) for multiobjective optimization. Comptes Rendus Mathematique, 350(5-6):313-318, 2012.
Shengding Hu, Yuge Tu, Xu Han, Chaoqun He, Ganqu Cui, Xiang Long, Zhi Zheng, Yewei Fang, Yuxiang Huang, Weilin Zhao, Xinrong Zhang, Zheng Leng Thai, Kaihuo Zhang, Chongyi Wang, Yuan Yao, Chenyang Zhao, Jie Zhou, Jie Cai, Zhongwu Zhai, Ning Ding, Chao Jia, Guoyang Zeng. Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm: Unveiling the potential of small language models with scalable training strategies, 2024. URL https: //arxiv/// org//////aa abs/////////
abs/2404.. 2404. 06395.

---

Contents
1
Introduction
2 Methodology: Multi-Target Domain Reweighting via GRAPE
2.1 GRAPE: Prioritizing the Lagged Tasks by Group DRO
2.2
Theoretical Properties of GRAPE
3
Experiments
3.1
Domain Reweighting for Multi-task Reasoning
3.2
Language Mixture for Multi-lingual Learning
Discussion and Limitations
Conclusion
Related Work
14
B Derivation of GRAPE
14
C Theoretic Properties of GRAPE
17
C.I Proof of the Convergence Theorem
17
C.2 Proof of the Variance Reduction Theorem
18
Domain Reweighting for Multi-task Reasoning
20
D.I
Implementation Details
20
D.2 Results ClimbLab
22
D.2.1
Description of Dataset
22
D.2.2
Learning Curriculum from Various Reweighting Approaches
22
D.3 Results SlimPajama
24
D.3.1
Learning Curriculum from Various Reweighting Approaches
24
D.3.2 Results on Various Task Configurations
25
D.4 Results on Wiki-40b
27
D.4.1 Language Mixture from Various Reweighting Algorithms
27
D.4.2 Results on Various Task Configurations
29
E Ablation Study
31
E.1
Progress Assessment Metrics for Group DRO
................................
31
E.2
Task Combination GSM8K, ARC-C, ARC-E
................................
33
E.3
Task Combination 2: GSM8K, Hellaswag
34
E.4
Task Combination 3: GSM8K, PIQA
35
E.5
Task Combination 4: GSM8K, LogiQA
36
E,6
Task Combination 5: GSM8K, SeiQ
37
12

---

E.7
Task Combination 6: GSM8K, ARC-E, ARC-C, Kodcode
38
E.8
Task Combination 7: GSM8K, Kodcode, Hellaswag.
39
E,9
Task Combination 8: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, Kodcode, GSM8K
40
E.10
Task Combination 9: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA
E.11
Task Combination 10: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, MathQA, MedQA
42
E.12 Task Combination 11: ARC-E, ARC-C, MathQA, MedQA
43
E,13 Task Combination 12: LogiQA, Hellaswag, MathQA, MedQA
44
F
Hyperparameter Tuning
44
13

---

A Related Work
Data Reweighting for LLM Pretraining Pretraining aims to create general-purpose models, requiring training on massive datasets. Data selection methods can be used to determine the optimal dataset composition according various objective functions. They operate at different granularities with domain-level data selection methods allowing for less fine- saue-grrined control than token-level and sample-level methods but offering more tractable optimization that scales better to the massive datasets typical for LLMs. Learned approaches for domain reweighting either determine fixed domain weights [Xie et al., 2023, Fan et al., 2024a] or dynamically adjust the weights during training of the final model [Fan et al., 2024b, Chen et al., 2023, Wang et al., 2020, Albalak et al., 2023]. Fixed- weight approaches rely on proxy models which require significant upfront compute and assume transferability of domain weights across model and training data scales. Notably, RegMix [Liu et al., 2025] trains hundreds of proxy models under tight computational budgets fit a regressor predicting the optimal mixture for held-out validation domain, then employs the top-ranked mixture during large-scale training; it relies on a single static surrogate objective (Pile-CC validation loss) that may not generalize to varied downstream tasks. CLIMB [Diao et al., 2025] similarly uses regression but first partitions the entire pretraining corpus into 20 semantically coherent "domains" via unsupervised clustering of document embeddings and then runs more efficient iterative mixture search to optimize their weights. DoGE [Fan et al., 2024a] uses a small proxy model to learn static domain weights based on bilevel optimization and gradient alignment, upweighting domains whose gradients most closely align with the chosen target domain. DGA [Fan et al., 2024b] extends this approach to the online scenario by periodically updating weights based on gradient alignment on the large model.
Domain reweighting is in general two-step process, where we first define objective that captures model performance, and secondly optimize domain weights according to that objective. Existing methods consider only single-objective scenarios that optimize validation loss of single target as a proxy for downstream task performance. They lack mechanisms to balance improvements across multiple target tasks with competing objectives. Our approach provides method to balance multiple tasks during pretraining, building on the gradient alignment score idea and using validation loss of downstream tasks as surrogate to downstream task performance.
Gradient Surgery for Multi-task Learning. The multi-task learning (MTL) problem has been intensively explored in literature [Yu et al., 2020, Liu et al., 2024, Chen et al., 2018, Navon et al., 2022], while most of the proposed methods requires sufficient amount of training data directly from the target tasks. Given N 2 2 different tasks associated with loss function (0), the goal is to optimize model parameters 0 that perform well across all target tasks. A standard objective for MTL minimizing the average loss over all tasks. However, optimizing this averaged loss objective often lead to suboptimal and biased multi-tasking performance because of tasks with confficting gradients or dominant gradients with large magnitudes [Yu et al., 2020, Liu et al., 2024]. Existing methods aim avoid gradient conflict by gradient surgery, where the target gradients are linearly combined with adaptive weights Chen et al. [2018], Yu et al. [2020]. For instance, PCGrad [Yu et al., 2020] heuristically removes inter-task gradient conflicts via projections at cost ( 22). CAGrad [Liu et al., 2024] strikes a balance between worst-task improvement and average loss reduction by solving a constrained optimization problem that requires ( operations. MGDA [Désidéri. 2012] frames MTL as multi- cojtti-ojecctvve optimization and solves quadratic programming problem, guarantecing Pareto-optimal updates. NASH-MTL [Navon et al., 2022] offers equivalent Pareto guarantees, by treating gradient conflicts as a bargaining game that maximizes the product of task improvements (sum of log utility functions). All these methods share common drawback substantial memory requirements to store task gradients, ( N), and at least ((0)) 2 for the optimization step. These computational and memory costs become prohibitive for large-scale models and numerous tasks. FAMO [Liu et al., 2023] attempts to address these efficiency concerns. It dynamically weights task losses using (1) space and time per iteration by trying to enforce equal loss decrease rate among tasks. We apply this idea in the domain reweighting setting to develop a multi-task multi-task-adaptiii adaptive domain reweighting method.
B Derivation of GRAPE
We start with the minimax problem from Equation (2):
14

---

max min 7 aEA* WEAM
jjj
24 Va (o)) h(xx) + +r(2)
With the regularization terms defined as:
h(a): == Ha 111 r(z) He (z(1(()()
Where Dy is the Bregman divergence with (bbb = Σi b, log(b;):
at log a; by Da(allb) Σ
Σas a; + Σ b, Σ
a, log a2 by
where the last simplification follows because E; a; Σ, b, for distributions. For the inner minimization over z, we form the Lagrangian:
k
N ([ + 1z
N
[(z, A) = 7/t
24
23 log
2i (t-1)
-=-1)
1)
j=1
2
Taking the partial derivative with respect to 2, and setting to zero:
Oc
7 Σ a (1
(Vo (O)) + Px
1 log
24 (1-1)
+ x
0z,
Zi
Solving for z,
23 (1-1)
exp
Yt
(0))
+ Hz
fz
Let d, = Ef= a (O,) be the update direction.
Then:
2, 11-1)
exp (- Tt
(Vo log ((0,)
X + F
f2
P1
Using the constraint
EN = 1 to solve for A:
Σ
(1 1)
exp
(-
Tt
(Vo log),
X 12
Pz
Hh
Let 7,2 = ΣN 21
11
exp
(- (V (vo log (0,) (0.).d))) then exp
(- AtH
He
%
Substituting back, get the update rule for task weights:
(t-1) exp
( I4 (Vo log ((0.).
Zz
15

---

Now, for the outer maximization over we substitute the optimal ((() back into the original objective function. Let fi (Va log g,
The resulting optimization problem becomes:
N
max
faz
Σ a log (1-1)
DEA4
a
We form the Lagrangian for this maximization problem:
k Mo
k
v(a, = 7
fij
ag log
+
Σ ag a
(t-1)
Taking the partial derivative with respect to aj and setting zero:
ac
N
7/-
fes
1 log (t-1) aj
+ -セ=
Jcg i=1
Solving for ay:
aj
(t-1)
exp
Ttt
fis
v + Ma
Na
N
i=1
Using the constraint ΣØ= 0j to solve for
Σ (1-1)
Yt
(1) Sig
v Max
exp
Na
1iae
i=1
Let Za = Eド 0j
(t-1)
exp
Pa
E
(1) fiz)
then exp
(-utae)
utta na
7a
Substituting back, we get the update rule for gradient weights:
((-1) exp
2e ΣN
2, S4j
Za
Expanding fiz back to its full form:
aj
(1-1)
exp
2t ΣN
24
(1) (Vo
(Vo
log (())
(
Ha
Za
16

---

C Theoretic Properties of GRAPE
C.1 Proof of the Convergence Theorem
Theorem C.1 (Convergence of GRAPE), Let the loss functions (0) be L-smooth for all E [N] and the norm of stochastic gradients be upper-bounded by G. If the learning rate 7 satisfies % S T.
and the regularization parameters Paa μaz chosen such that Max algorithm with update rules given by the above equations converges to a neighborhood of the Pareto
and μı > then the GRAPE
optimal solution rate of (1/T)) where T is the number of training iterations. Specifically, for any > 0, there exists To such that for all T To:
}
max [L,, min max Ln (0) S nE(N]
min LE12)
nEIN]
Proof. We establish the convergence of GRAPE by analyzing the dynamics of the minimax optimiza- tion problem and demonstrating that the algorithm makes consistent progress toward Pareto optimal solutions while balancing performance across tasks. First, analyze the task weight updates. Given that 2 is updated according to:
(-72
[h (di Vo log!n(0:)) Pxz
2/1 = 2/ 1-1 exp
(6)
The task weights adjust to increase focus on tasks with slower improvement rates. For any task i where (d,, Vo log (0.)) < 0 (indicating poor alignment between the current update direction and task gradient), 2/ increases proportionally. This ensures
attention tasks making less progress.
For each iteration t, let L(O.) max (0.) represent our worst-case objective. Let n1 arg [N] (0,) be the index of the task with highest loss at iteration t. Due the exponential update rule, increases, 2,, approaches while other weights approach 0, directing optimization toward the current worst-case task. Considering the domain weight updates:
N
a az t-1 eex
(g) (0)), Vo log 4, (O.))
Y4
Na
These updates increase weights for domains whose gradients align well with the task gradients weighted by z. Since z concentrates on the porst-performing tasks, a increasingly favors domains beneficial these tasks. Given the L-smoothness of the loss functions, progress on the worst-case objective:
upper-bound the
C(0++1) C((.) S (O.), -n) + L |de
Let O. = = (0.), d.). Due to our update rules for z and a, as / increases, Q. becomes increasingly positive (as domain weights shift toward domains beneficial for the worst-performing task). With the condition ?/ S 1. we have:
C(0,+1) [(0.) - |d.11
Since the domain weights probability distributions and gradients are bounded (due to L-- smoothness), |d is bounded by some constant G. Therefore:
C(0 ((,,) - 7.
As optimization progresses and Q, increases due our reweighting strategy, we eventually reach a point where Q. > G² ensuring consistent progress on the worst-case objective. Summing over iterations and applying the j-strong convexity, we obtain:
2
G622
Σ
S
17

---

This implies:
T
-Q + "²)
-(0r+1) - -
Σ
2
(=1
Given our reweighting strategy ensures 2 > G negative, establishing convergence.
after sufficient iterations, this difference becomes
2
For the convergence rate, with constant step size ?t 7. we can show:
min (C(0,) [ L 0 1- EE(T]
G
T Q 3 t=1
2T
2L
Where L ming maxxE[N] (0) and 0* is the corresponding optimal parameter. As T - oo, this
G² bound approaches 27 L. where Q is the average alignment. Therefore, GRAPE converges to neighborhood of the Pareto optimal solution a rate of O117T))
C.2 Proof of the Variance Reduction Theorem
Theorem C.2 (Monotonic Variance Reduction of Task Performance). Let o7 VaIneiN] (a denote the variance of task performances at iteration t. Let the loss functions (0) be L-smooth for all E [N] and the norm of stochastic gradients be upper-bounded by G and assuming the task losses are L-smooth and p-strongly convex, the variance decreases monotonically until reaching minimal basin, ie., o441 S ot for all t To for some finite To-
Proof. According to the definition of Rate-of- lipiooof--mmooementtt (Rol) for task at iteration
(,,0,) (((0++1)
(0,)
For any task i, by the L-smoothness assumption, bound the change in loss:
4(0.+1) - (0.)) < - (Vo4 d:) L |de]
Let (t(() 100+. - (,,) represent the change in loss for task i.
First, we establish direct relationship between task weights and improvement rates. At each iteration, GRAPE updates task weights according to:
2. exp (-7h 7t μ1z (d,, Vo Volog
2 x
reflects the principle that tasks with lower Rol receive higher weights in the next iteration.
When domain weights are subsequently updated, domains that yield better improvements for these higghr-weighted (struggling) tasks are upweighted:
a x
a 1-1 exp
(g) (0.), Vo log
Pax
This creates feedback mechanism that specifically targets and improves tasks with lower RoI values. This indicates that tasks with lower Rol experience proportionally greater increases in their improvement rates. We can formalize this in Lemma 1:
Lemma 1: There exists constant / > such that if r < r for tasks and j. then after GRAPE's
reweighting mechanism, the expected improvement in the next iteration satisfies: -((r)"))))
1 3(r ((())
We further define the normalized task losses:
4(0,) miny (0.)
4,(1) max, 1,(0) min, (0,)
18

---

We show that the variance of normalized loss (t) is proportional to the variance of losses: Lemma 2 (Variance of Normalized Losses): The variance of normalized losses o7 (4) is directly proportional the variance of original losses of (1.(())
Proof. We begin with the definition of normalized losses:
[.(1) (,(0,) min, 4, (0.)
max, [, min, L, (0.)
Let's denote a₂ min, L, (0.) and b, max, 1, (0.) for brevity. The normalized loss can be written as:
((t) = 1 ( dz b,
This an affine transformation of the original losses. By the properties of variance for affine transformations, for any random variable X and constants and d, we have:
Var(eX d) Var(()
In our case, viewing the task losses samples from a distribution, have bi-ai and b Thus, we have,
(4(0,) de
of (4((0)))(((00))) (b: oz)
o (OrreN]((((())) VarieN
b at
(be ax)²
Therefore:
0, (b. oz)²
Which establishes direct proportionality between of and of, with the proportionality constant
(2
Given the relationship between task weights and Rol established in Lemma 1, tasks with higher normalized losses tend to have lower Rol values and thus receive higher weights in GRAPE. This leads correlation between (t) and the subsequent change in normalized loss A4(t) 1
Specifically, for any pair of tasks i and j where (4(t) (t), GRAPE's reweighting mechanism ensures:
E[A [t)
Therefore, express the change in variance:
N
0.41
Σ (((t+1)-in
1
Σ 1=1
(i((t) -i)
N
N
Σ
((+(t)) Ai((t) - - 10:0~11. -i,((t -i)t
N
1=1
02 141
03
Σ ((((())) i)(i(( -i - AĀ.) + (Ai(()) 2
Given that the tasks with higher normalized loss will experience greater improvements on normalized loss under GRAPE, this correlation term is negative:
(L((1) (((0)-i(aa is(a(())) AĀ:) < -KO?
for some constant x > 0, representing the strength GRAPE's balancing effect.
The second term, 1
N Σ> (Ai,(t) =)))², represents the variance of the changes in normalized
losses, which is bounded by 0 under the smoothness assumption. Thus:
0 of < 2kő + +0??)
For sufficiently small 7, this difference becomes negative, establishing that 02,1 < o?. Following Lemma 2, it indicates that oi < of
19

---

D Domain Reweighting for Multi-task Reasoning
Table 3: Architecture hyperparameters for various model scales used in the paper. All models are vanilla Transformer decoder-only models.
Layers
Attention heads
Embedd dim
Hidden dim
Context Ien.
learning rate (t)
3//aa
7/Pss
batch size
1M
2
256
512
512
1x-
125M
12
768
3072
512
1.5 x 10-4
1.5
10
64
0.78
36
20
1280
5120
512
x 10-1
1.5
10
128
D.1
Implementation Details
On small-scale experiments with 125M models. we employ cosine learning-rate scheduler with maximum Ir = 1.5e On large-scale runs with 0.7B models, we adopt the WSD scheduler [Hu et al., 2024] where the learning rate remains constant (Ir = 1.5e=4) during training while linearly decaying to min 1.5e at the last 20% of total iterations. The model architecture and hyperparameter settings are detailed Table The full implementation of GRAPE open-sourced in hhttts://ggtt/ut
GRAPE. For GRePE-specific parameters, we generally set the regularization coefficients /aa = 1e and /zz 1.5e-5, corresponding to the step-size of 7/Pa 1.5, = 10. Both domain weights (ar) and task weights (z) were periodically updated every AT = AT= 100 steps. Initial domain weights (a") and task weights (z") were set to uniform distribution, unless otherwise specified. Gradients required for the GRAPE weight updates (e.g., V log ((,)) on target tasks, g; (0,) on source domains) were estimated stochastically using dedicated mini-batches processed within the update step.
DoGE. We implement DoGE following the out-of-domain generalization setting in [Fan et al., 2024a]. Domain weights (a) are updated every ATo = 100 steps during training according to the alignment of gradients between each training data domain and the average gradient across all target tasks, which is equivalent to applying a uniform task weights (z) constantly. We apply the same regularization hyperparameters GRAPE.
CLIMBMIX. We set the domain weights following Diao et al. [2025]. Specifically, we utilize the final iteration weights derived by iterative regression and bootstrapping, which is claimed as the optimal mixture in the original paper.
GRAPE-CLIMENIE While the GRAPE algorithm initializes domain weights (a*) ) as the uniform distribution, we implement a variant of GRAPE with a initialized with the optimized CLIMBMIX weights [Diao et al., 2025]. The other hyperparameters for reweighting and optimization are kept consistent as GRAPE.
DOGE-PCGRAD.
We tailor the DoGE [Fan et al., 2024a] method for multi-task learning by dynamically tune the target gradient with a gradient surgery method PCGrad [Yu et al., 2020], where the gradients from each target task combined avoid gradient conflicts. PCGrad eliminates conflicting gradient components by projecting each task gradient onto the normal plane of any conflicting task gradient. Conflict is indicated by negative dot product between target gradients. We process gradients iteratively, taking each task gradient in a randomized sequence and adjusting it when conflicts are detected, continuing until all gradients have been properly modified. The random ordering of gradient processing helps prevent systematic bias toward specific tasks. The final target gradient, grCGred. calculated the average of the processed target gradients. At each reweighting step to calculate conflict- free target gradient, gPCGrad Domain weights (a:) are updated every AT 100 steps during training according to the alignment of gradients between each training data domain and gPCGrad- We apply the same regularization hyperparameters as GRAPE. With N target tasks, each reweighting step requires storing N gradients into memory, which incures x (0) storage costs. Thus, the traditional gradient surgery algorithm for multi-task learning can lead to OOM issue due to the significant GPU memory load, which prevent it from scaling up to large models and datasets.
20

---

REGMIX. We implement REGMIX [Liu et al., 2025] to predict the optimal domain weights (a) by fitting regression model. Following Liu et [2025], we train a total of 768 TinyLlama- 1M proxy models with with various data mixture configurations: 512 of them are used to fit the regression model, while the other 256 of them are saved for evaluating the performance of the regressor. For each experiment run, we record the average validation loss across all target tasks; we then train the LightGBM regressor to predict the input domain weights which achieves the lowest average loss. The other hyperparameters for proxy model training set to the default values in [Liu et al., 2025].
CRISP compute embedding- based domain weights using CRISP [Grangier et al., 2025]. For each train and target domain calculate the sequence embeddings using SBERT MiniLM-L6-v2: To avoid overfitting, we select representative centroids for each training domains Dx. & E [K] respectively using k-means. We then obtain the domain weights by applying nearest-neighbor (k=5) classification on each piece of the samples from all target sets, i.c. mapping every individual target embedding to centroids within each training domains. We finally compute the distribution of target samples located in each training domain to get the final domain weights.
21

---

D.2
Results on ClimbLab
D.2.1
Description of Dataset
ClimbLab is a dataset introduced by Diao et al. [2025], which is partitioned into 20 clusters according to semantic-based sequence embeddings, The related topic each cluster (i.e. train domain) are detailed Table 4.
Table 4: Main topies of 20 clusters in ClimbLab corpus...
Cluster ID)
Topics
Mathematics. Algorithms, Programming, Software Development, Data Analysis Books, Education, Writing. Literature, Al Ethics, History, Philosophy
2
3
Environmental Education. History, Architecture, Engineering. Classical Music Education, Teaching, Science, Engineering, Psychology. Special Education International Trade, Business, Economics, Al Consulting, Ethical Decision Making
5
Genetics, Biotechnology. AL. Robotics, Aging, Healthcare, Industrial Automation
Chemistry, Insects, Taxonomy, Agriculture. Gardening. Veterinary Science
8
Gaming, Role- Role-Playing, Board Games, Video Games, Stralegy, Faniasy, Virtual Reality
9
Astronomy, Cosmology, Astrophysies. Space Exploration, Urhan Planning
10
Health, Sleep. Clinical Technology. Healthcare. Fitness, Addiction, Early Childhood Education
11
Software Development, Programming, Web Development, JavaScript. Databases
12
Technology, Mathematics, Legal Content, Human Rights, Energy Efficiency, Industrial Equipment
13
Sports, Cricket, Soccor, Tennis, Baskethall, Cultural Heritage, Competition Music, Instrumental Practice. Guitar, Jazz, Singing. Composition. Music Theory
15
Film. Cinema, Horror, Sci-Fi, Comics, Literature, Criticism, Philosophy Sustainability. Climate Change, Renewable Energy, Environmental Conservation
16
17
Cardiovascular Health, Medical Research, Immunology. Cancer Prevention, Drug Therapy Technology, Cybersecurity, Social Media, Privacy, Artificial Intelligence. Cloud Computing
18
19
Social Media. Digital Communication, Internet Culture, Misinformation, Psychology Public Safety, Law Enforvement, Political History, Social Justice, Government
20
D.2.2 Learning Curriculum from Various Reweighting Approaches
Domain Weights from RegMix. We present the domain weights from RegMix in Figure 7 and Figure& 8, with different sets of target tasks. The RegMix regressor ends up concentrating almost all of its weight to just two clusters. In both settings, Cluster 17 (advanced biomedical and clinical research content) receives the highest weight. Cluster 15 (film and cultural-arts) gets picked up for the general reasoning benchmarks showing that its narrative and conceptual text aligns with the story-and-cone and-commonsense inference style those tasks need. Including MedQA/MathQQ,, increases the weight of Cluster 9 (space science, urban planning), showing that its physics- heavy, formula-rich science content align well with the formal quantitative reasoning that MathQA requires.
Ditsurt Pisee
Priptal
6.30
Corn: 0.06
6.00
5.75
02
5.25
3.50
5.75
LO55
(a) Original vs. RegMix weights
(b) Spearman conrelation between predicted v.s. true loss
Figure 7: RegMix results on ClimbLab with LightGBM regression on 6 reasoning target tasks [ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA]
Domain Weights from CRISP. We present the domain weights from CRISP in Figure 9, with different sets of target tasks. In both settings CRISP assigns similar weights to the dataset clusters. Main focus is given on code data (Cluster 11), while Cluster with technical life-science and automation content also receives significant attention. Including MedQA and MathQA, slightly
22

---

Cor 0.50
Original vs. RegMix weights
(b) Correlation between Predicted
v.s. True loss
Figure 8: RegMix results on ClimbLab with LightGBM regression on 8 target tasks [ARC-E, ARC-C, Hellaswag, SciQ. PIQA, LogiQA, MathQA, MedQAJ.
boosts the weight of Cluster 10 (health and wellness content), reflecting MedQA's embeddings lie closest to clinical-style text.
0.1
Budder dedeheaiigy meal scare
Elister 2., meniaiiggn segiticiiend
(a) Target
(b) Target Tasks
Figure 9: CRISP Domain weights across data domains in ClimbLab.
Domain Weights Evolution from GRAPE and Scaling Effects. We observed that the task priority differs notably between model sizes. According to Figure 10, 0. parameter model tended to develop a highly concentrated focus single challenging task in later training stages, whereas the 125M model maintained more distributed and stable prioritization across several tasks. This suggests that the larger-scale model might be more sensitive to the conflicts between targets. Additionally, in both cases, the task weight trajectories demonstrate clear, stage-wise curriculum, which indicates a dynamic, adaptive currieulum is crucial to yield better multi-task learning performance.
Weight
ARC-E
0.1
ARC-C
SeQ
PIOA
0.3
Helhaway
Opdimizztonn Steps
Oprimizztion Steps
(a) 125M- Lbb
(b) 0.7B -BB-ClimbLab
Figure 10: Task weights trajectory from GRAPE on ClimbLab from 125M/0.7P models.
23

---

D.3 Results on SlimPajama
We present the experimental results on SlimPajama in this section. The hyperparameter and baseline settings are constant with the experiments on ClimbLab.
D.3.1 Learning Curriculum from Various Reweighting Approaches
Domain Weights from REGMIX. We present the domain weights derived from RegMix in Figure 11 and Figure 12, on various sets of target tasks. In both cases, the RegMix regressor learns a very spiky training mixture, where large percentage (~~ 80%) are allocated web-crawled dataset such as C4, while nearly eliminates the domains other than C4, CC, and Book. Compared to 6 common-sense and logical related reasoning tasks, including domain-specific QA tasks MathQA and MedQA further increases the weights on C4, which can be attributed to its broad coverage on various knowledge sources.
Dornest Fricr
Reghlix
Corr: 0.98
6.75
0.4
0.2
6.00
5.75
Medectleng
9.55
Donai
Prodicted Locs
(a) Original vs. RegMix weights
(b) Predicted true loss
Figure 11: RegMix results on SlimPajama with 6 target tasks [ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA]
Dornest Price
Toghlis
7:23
Corr: 0.98
7.00
Woddh
0.5
0.4
6.00
vioondo
Docaat
Prodicted Losa
(a) Original vs. RegMix weights
(b) Predicted vs. true loss
Figure 12: RegMix results on Slim SlimPajama with target tasks [ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, MathQA, MedQA]
Domain Weights from CRISP. We present the domain weights derived from CRISP in Figure 13 on various sets of target tasks. CRISP assigns most of the weight Github, followed by StackExchange. All other domains contribute minimally. A plausible rationale behind the CRISP weight distribution is that short Q&A tasks (ARC-E/C, SciQ. PIQA) embed closest to the concise question question-answer answeer format of StackExchange, while multi-step reasoning puzzles (LogiQA) and formula-heavy problems (MathQA) align more with GitHub's 's structured code snippets, and MedQA's longer clinical passages slightly with Book and arXiv. Including MedQA and MathQA brings clinical and formulaic embeddings that shift bit of weight back toward Book and arXiv.
24

---

Top demeins by weight
0.5
domeirs weight
Other domains
Other dollaiss
0.4
Weight
Weights
0.3
0.3
0.2
0.1
0.0
github
stackercangg
wikipedia
0.0
ardtv
book
gittub
stackorchargee
wikipedia
Domains
Domains
(a) 6 Target Tasks
(b) 8 Target Tasks
Figure 13: CRISP Domain weights across 7 data domains in SlimPajama.
D.3.2 Results on Various Task Configurations
Reweighting with 6 Target Tasks: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA We present the full results on multi-task reasoning experiment, where the data mixture from SlimPajama dataset are adapted towards the optimal performance on 6 target tasks: ARC-E, ARC-C, Hellaswag, SciQ. PIQA, LogiQA. The Log-perplexity and 5-shot accuracy scores on each target reasoning tasks are presented in Figure 14, Table 5 and Table 6, where GRAPE consistently achieves better (lower) perplexity compared to the other baselines. In contrast, CRISP, as an purely embedding- reweighting algorithm, leads to significantly worse results all benchmarks, in terms of perplexity and 5-shot accuracy scores.
aic arc_challlee challenge
aFC, easy
belllewway
Regliss
4.8
ITRAP
4.6
12A
18
321
11A
2/.
48
14
12.
30% Stops
Sucpe
Stayo
logiga
piga
sciq
Perjloity
1.75
36-
3.25
Figure 14: Log-Perplexities on target reasoning tasks
Table 5: Log perplexity scores ()). Models (125M) are trained on 0.66B training tokens. The best-performi scores are marked as Bold, and the second-best scores are Underlined.
Method
ARC-C
ARC-E
HellaSwag
LogiQA
PIQA
SciQ
Average
Worst
Uniform
4.83
4.92
4.30
4.77
4.08
4.56
4.58
4.92
CRISP
4.94
5.09
4.41
4.95
4.10
4.60
4.68
5,09
REGMIX
4.29
4.36
4.05
3.68
4.16
3.81
4.06
4.36
DOGE
4.35
4.51
3.83
4.34
3.66
4.09
4.13
4.51
DOGE+PCGRAD
4.38
4.54
3.88
4.35
3.71
4.18
4.17
4.54
GRAPE
4.05
4.08
3.23
4.19
3.54
4.04
3.86
4.19
25

---

Table 6: 5-shot accuracies( %) target reasoning tasks(1). 125M (resp. 0.7B) models trained on 0.66B
(resp. 6.5B) tokens. The best-performed scores are marked Bold, and the second- best scores are Underlined.
125M
ARC-C
ARC-E
HellaSwag
LogiQA
PIQA
SciQ
Average
Uniform
21.50
32.45
26.44
26,42
55.50
52.40
35.78
CRISP
21.08
30.89
26.58
27.80
54.62
53.80
35.80
REGMIX
20.56
32.79
26.83
24.42
58.92
52.50
36.00
DOGE
21.67
32.66
26.05
30.11
57.18
37.38
21.33
32.20
25.96
27.19
58.60
53.50
36.46
GRAPE
21.76
34.09
26.69
26.27
58.60
58.50
37.65
0.78
ARC-C
ARC-E
Hellaswag
PIQA
SciQ
LogiQA
Average
Uniform
21.84
37.84
29.04
62.13
69.60
29.34
41.63
DOGE
22.87
40.61
31.16
63.82
69.80
24.88
42.19
GRAPE 23.46
42.17
31.90
64.42
68.70
27.50
42.92
Reweighting with 8 Target Tasks: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, MathQA, MedQA. We present the full results on multi-tat task reasoning experiment, where the data mixture from SlimPajama dataset are adapted towards the optimal performance on 8 target tasks: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, MathQA, MedQA, covering board topics and knowledge domains. The Log-perplexity and 5-shot accuracy scores each target reasoning tasks are presented Figure 15, Table and Table where GRAPE consistently outperforms the other 5 baselines on most of benchmarks (6 out of 8), while achieving comparable performance as RegMix and DoGE- PCGrad on Hellaswag and PIQA. Notably, GRAPE improves the perplexity scores on MedQA by a large margin above all the other baselines, indicating its distinct adaptability to unique target tasks with Nomain-speciife skill and knowledge, In contrast, CRISP demonstrates marginal improvements compared to uniform sampling baseline, which demonstrates its weakness in the multi-task learning setting.
are. challesge
aZC. easy
hullaswag
logige
5.4
5.2
4.-
4.5
DuG2 (NCC
4.5
1.5
44-
Stegs
motheq
medga
plca
sclc
3.0
3.8
4.5
2.5
4.0
4.0
0.5
3.5
150
350
Stree
Strys
200CE
Figure 15: Log-Perplexities on 8 target datasets
26

---

Table 7: 5-shot accuracies( %) target tasks. The best-performed marked Bold, and the
second
best best scores are Underlined.
Method
ARC-C
ARC-E
HellaSwag
LogiQA
PIQA
SciQ
MathQA
MedQA
|| Average
Uniform CRISP
21.50
32.45
26.44
26.42
55.50
20.97
32.53
22.01
30.56
26.24
27.19
55.88
51.10
19.97
21.21
29.27
REGMIX
22.00
32.40
26.80
25.50
59.70
56.00
20.70
21.40
33.10
DOGE
21.59
31.57
26.32
25.96
56.31
48.60
19.50
27.34
32.14
DOGE+PCGRAN
21.16
32.28
26.17
26.11
57.62
52.2
20.60
21.52
32.21
GRAPE
21.76
33.08
26.20
27.34
57.70
57.94
20.74
27.18
33.99
Table 8: Log perplexity scores on 8 target datasets (L). Models (125M) are trained on 0.66B training tokens. The best-performed scores marked as Bold, and the second-best scores are Underlined.
Method
ARC-C
ARC-E
HellaSwag
LogiQA
PIQA
SciQ
MathQA
McdQA
Average
Worst
Uniform
4.83
4.92
4.56
4,08
4.77
4,30
4.88
4.60)
4.62
4.92
CRISP
4.89
4.04
4.79
4.69
4.63
4,95
REGMIX
4.3
4.4
4,06
3.71
4.15
3.81
4.55
3.93
4.11
4.55
DOGE+PCGRAD
4.35
4.47
4.17
3.71
4.35
3.84
4.65
4.07
4.20
4,65
4.63
4.72
3,90
4.60
4.36
4.44
4.82
GRAPE
4.26
4.37
4.15
3.58
4.35
3.76
4.52
2.58
3.95
4.52
D.4 Results Wiki-40b
We present detailed results on the multilingual pretraining experiments on Wiki-40B dataset in the following sections. Specifically, experiment with two various target configurations: (1) target at low-resource languages: Catalan, Danish, Romanian, Ukrainian; (2) target at low-resource languages: Catalan, Danish, Romanian, Ukrainian, Polish, Portuguese, Turkish, Dutch, i.e. the setting we present in the main paper ($ 3.2).
D.4.1 Language Mixture from Various Reweighting Algorithms
Domain Weights from RegMix. We present the optimized domain weights distributions from RegMix in Figure 16 and 17. Specifically, with 4 target languages, RegMix upweights German
(de) and Spanish (es) while reducing the weights assigned on French (fr) and Italian (it). With target languages, RegMix significantly increases the domain weights Russian (ru), while further decreases the proportion of English (en).
RegMix
Corr: 0.96
0.4
0.3
Thus
0.
8.0
8.5
0.50
8:75
9.00
10.25 Predicted Losa
0.25
0.50
0.75
10.00
Doceetn
(a) Original vs. RegMix weights
(b) Spearman correlation between predicted true loss
Figure 16: Mig results on wiki-40b with LightGBM regression on 4 target languages (Catalan, Danish, Romanian, Ukrainian]
27

---

SigMit
Corr: 0.95
0.4
101420
0.2
0.2
0.1
9.0
Doradi
Predicted Lows
(a) Original RegMix weights
(b) Spearman correlation between predicted
v.S. true loss
Figure 17: Mig results on wiki-40b with LightGBM regression on 8 target languages (Catalan, Danish, Romanian, Ukrainian, Polish, Portuguese, Turkish, Dutch]
Domain Weights from CRISP. We present the optimized domain weights distributions from CRISP in Figure 18, with various task configurations, Notably, CRISP yields very similar domain weights across languages, where English (en) and Italian (it) are mostly upweighted, while they are proved not helpful for multi-language learning according to Figure 19 and 20.
0.30
Hight Other dorains
0.25
0.20
0.00
0.00
Dorruita
Domruna
(a) target languages
(b) larget languages
Figure 18: CRISP Domain Weigths across 6 high-resource languages wiki-408
28

---

D.4.2 Results on Various Task Configurations
Reweighting with 4 target languages: Catalan, Danish, Romanian, Ukrainian Figure 19 present the full results on the multilingual pretraining experiments, where the training langauge mixture are adapted to 4 target low-resource languages: Catalan, Danish, Romanian, Ukrainian. GRAPE significantly outperforms all the other baseline methods across target languages. Notably, none of the other reweighting approaches can effectively facilitate the learning on Romanian, while only GRA APE unleashes the learning capability on it.
Catalan
Donish
6.8
64-
0.2
6.4
0.0
6.2
7.5
Romenian
Ukraimian
0.4
Unitorm
0.2
BEGT
RogMix
Log
0.0
CRISP
7.6
CHHAPE
7.6
48
124
201
24
12k
10k
20k
Stage
Stags
Figure 19: Log-perplexities on target languages (Catalan, Danish, Romanian, Ukrainian)
29

---

Reweighting with 8 target languages: Catalan, Danish, Romanian, Ukrainian, Polish, Por- fuguese, Turkish, Dutch. Figure 20 present the full results on the multilingual pretraining ex- periments, where the training langauge mixture are adapted to 8 target low-resource languages: Catalan, Danish, Romanian, Ukrainian, Polish, Portuguese, Turkish, Dutch. GRAPE significantly outperforms all the other baseline methods across all 8 target languages. In contrast, the offline reweighting algorithm such as Meg and CRISP exhibit very biased performance across various larget languages: RegMix accelerate the learning Catalan, Portuguese, Ukrainian and Polish, while sacrificing the performance on Dutch and Danish; CRISP only accelerates the learning on Turkish while sabotaging all the other languages.
Polish
Eag Parpletry
16.
Uaidorn
0.2
Beetiicc
862
2.0
1.8
3.0
8:0
12k
200 Stage
20%
32A
116
20k
Stuga
Steps
btaga
Portuhese
Romanian
Turkish
Ulerainian
Log Bergueery
7.
11.3
16.2
7.8
Figure 20: Log-perplexities on 8 target languages (Catalan, Danish, Romanian, Ukrainian, Polish, Portuguese, Turkish, Dutch]
30

---

E
Ablation Study
In this section, we present the full results on our ablation experiments, with three various progress measurement metrics for Group DRO, and 12 different target task configurations.
E.1 Progress Assessment Metries for Group DRO
We evaluate the impact of different step-wise progress metrics the performance of Group DRO within the GRAPE framework. In addition to the primary Rate-of- In--of---mooveeentt (ROI) metric (Equa- tion 1), investigate two alternatives: Gap-of-Imin lfppo--f--rooement (GOI) and EMA-Rate Reate-of-Imrrvvemeet (ROI-ema). The step-wise improvmenet of task Tn at step t are assessed as:
G = (O.) - In (0.41), ROT-ema = L. (..( (0++1)
Vn E [N]
(7)
1t I'maa
The exponential moving average loss I mmma, updated lema. --1 - (1 where the hyperparameter / is set to 0.7 in our experiments. Substituting these alternative progress metrics for r, in the minimax objective Equation yields modified update rules according to Equation 8 and 9. The core optimization principle for adjusting task weights (z) and domain weights (a) remains, but the specific gradient terms within the exponents change.
GOI:
with 2t <- 2/-1 exp
2t (e-mii(at->>((Vollal Vol(Or, Az
En 2
a
with o + ag 1-1 exp
1t Ha
LeelK] of
(8)
ROI-ema:
gt with E 2/- .ex
It xxmmx((" 1)l (0.)
Vol(O,, x])]]
Ene(y R. á' with of <- 0% 1-1
Pz
a²
Σtetg op
exp
2+ yomix((*--) (1g Vo log ((0,, y))
Task Configurations.
We evaluate the efficacy of GRAPE on 12 various target task combinations
as follows:
T1: GSM8K, ARC-C, ARC-E
T2: GSM8K, Hellaswag
T3: GSM8K, PIQA
T4: GSM8K, LogiQA
TS: GSM8K, SciQ
T6: GSMSK, ARC-E, ARC-C, Kodcode
T7: GSM8K, Kodcode, Hellaswag
T8: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, Kodcode, GSM8K
T9: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA
T10: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, MathQA, MedQA
TII: ARC-E, ARC-C, MathQA, MedQA
T12: LogiQA, Hellaswag, MathQA, MedQA
31

---

Results. We present the average and worst log- perplexxy scores on each of 12 groups of ablation experiments in Table Specifically, GRAPE outperforms uniform and DoGE across all task sets; on 7 out of 12 tasks, GRAPE with ROT metric outperforms the other two DRO variants with GOT and ROT-ema, respectively. We present the og-peeplexxty scores associated with task weights evolution trajectories for all 12 groups of ablations in Figure 21-32.
Table 9: Average and worst-case test (erpleeeex))ty() on 12 task combinations (trained on 0.66 SlimPaj ama) tokens. GRAPE outperforms uniform and DoGE across all task sets; on out of 12 tasks, GRAPE with ROI metric outperforms the other two DRO variants.
Method (avg. wst.)
Uniform
DoGE
GRAPE
GRAPE-eme ema
GRAPE- gap
T1
4.30 I 4.66
4.18
4.53
3,53 3.55
3.68 3..33
3.58 3.59
T2
4.01 4.35
3.941 4.24
3.61 3370
3.70 3.81
3.68 3.70
T3
4.11 4.55
3.7013.88
3.73 13.89
3.69 3.80
T4
3.67 13.69
3.40 33.47
3.38 13.46
3.41 13.4
T5
3.85 14.02
3.70
3.37 13.43
3.37 13.47
3.35 3.48
T6
3.6214.66
3.63
3.04 3.58
3.10 13.65
3.05 3.52
T7
3.20 14.34
3.1214.24
2.93 3.800
2.96 13.81
3.11 3.72
T8
3.90 14.66
3.46 4.30
3.41
3.51 4.32
T9
4.33 14.66
4.1314.51
3,86 4.19
3.88 14.21
3.88 4.21
TIO
4.62 4.92
4.32
4.82
3.95 4.52
3.66 4.51
3.86 4.55
TII
4.81 14.92
4.57
4.74
3.94 4.52
3.90
44.33
3.85 4.53
T12
4.53 4.88
4.33 4.76
3.70 4.51
3.73 4.56
3.59
4.50
32

---

E.2 Task Combination 1: GSM8K, ARC-C, ARC-E
gsmmk
easy
challange
3.9
6.0
Log Perplexity
5.5
5.5
Uniform
DoGE
3.7
5.0
CRAPE-ema
4.3
4.0
3.5
3.5
3.5
20k
06
20k
20c
Strps
Strrg
Steps
(a) Log- Log-Perplexity Scores
gamm8k
anc easy
arc challenge
0.8
0.5
0.4
Weighas
DOGE
0.2
0.5
0.3
0.1
0.4
0.2
GRAPE
0.3
0.1
GRAPE-gaP
2.6c
20kc
Steips
Steps
(b) Task Weights
0.25
Weights
book
github
0.10
stackeethange
wikipwdii
(c) Domain Weights
Figure 21: Ablation on task combination [GSM8K, ARC-C, ARC-E).
33

---

E.3
Task Combination 2: GSM8K, Hellaswag
gsmök
hellaswag
4.75
Uniform
4.0
DoGF
Perplexity
4.50
CRAPE-AMA
4.25
4,00
3.6
3.75
80
20%
20k
Steps
Steps
(a) Log- PerpPeelyy Scores
gsm8k
hellaswag
1.0
0.4
GRAPE
Weights
CRAPE-ema
0.8
GRAPE-g4p
0.2
DoGE
0.0
4k
8k
12k
16k
20k
4k
8k
12k
16k
20k
Steps
Steps
(b) Task Weights
0.4
Weights
book
hih
stackeschenge
wildpedia
(c) Domain Weights
Figure 22: Ablation on task combination [GSM8K, Hellaswag).
34

---

E,4 Task Combination 3: GSM8K, PIQA
gsmök
piqa
5.00
Unitorm
3.9
4.75
DoGE
Perplexity
GRAPE
3.8
4.50
GRAPE-40 44-1
GRAPE-Sap
3.7
4.25
3.6
4.00
Steps
Steps
(a) Log-Perplexity Scores
gsm8k
piqa
0.6
1.0
GRAPE
CRAPE-40R
Weights
0.4
0.8
GRAPE- GRP
0.2
0.6
Do
0.0
4k
8k
12k
16k
20k
4k
8k
12k
16k
20k
Steps
Steps
(b) Task Weights
0.4
0.3
Welghts
LESSY
Dook
Stockenchange
wildpedia
(c) Domain Weights
Figure 23: Ablation on task combination [GSM8K, PIQA].
35

---

E,5 Task Combination 4: GSM8K, LogiQA
gsm8k
logiqa
5.0
Unilomm
4.0
Perplexity
4.5
GRAPE
3.8
GRAPE E ema
GRAPE-g5D
3.6
3.5
12k
16k
20%
12%
16k
20k
Steps
Steps
(a) Log- Log-Peerllextty Scores
gsm8k
logiqa
1.0
Welghts
0.4
0.6
GRAPE
0.2
GRAFE-ema
0.4
CRAPE-gaP
0.0
16t
Stops
Stops
(b) Task Weights
0.4
Weights
0.1
stackeechangg
wikipwdda
Steps
(c) Domain Weights
Figure 24: Ablation on task combination [GSM8K, LogiQA].
36

---

E.6 Task Combination 5: GSM8K, SciQ
gsm8k
sciq
4.0
4.5
Unitora
DoGE
Perplexity
3.8
GRAPE
CRAPE-AMA
GRAPE-gap
3.6
3.5
12%
166
20k
4
122
10k
Steps
Steps
(a) Log-Perplexity Scores
gsm8k
sciq
1.0
0.6
DOGE
Weights
0.8
0.4
0.6
0.2
GRAPE
GRAPE-ema
04
GRAPE-gap
0.4
4k
8K
12k
16k
20k
4k
8k
12k
16k
208
Steps
Steps
(b) Task Weights
0.30
0.25
Welphts
book
0.10
stackexchange
0.05
wilipedia
(c) Domain Weights
Figure 25: Ablation on task combination [GSM8K, SciQ).
37

---

E.7 Task Combination 6: GSM8K, ARC-E, ARC-C, Kodcode
3FC c8sy
LEC challcage
kodoode
3.4
3.2
1.0
150
1.6
Exips
2xpps
Sriga
Sig.
(a) Log- Log-Perplyxity Scores
lodoade
3.6
3.5
3.6
0.13
3.02
GRAPE-
02
12
(b) Task Weights
0.30
Weugats
0.25
0.20
book
0.15
Klackonchange
wilipedia
12k
Stops
(c) Domain Weights
Figure 26: Ablation on task combination [GSM8K, ARC-E, ARC-C, Kodcode].
38

---

E,8 Task Combination 7: GSM8K, Kodcode, Hellaswag
gsmök
kodcode
hellaswag
Unitorm
4.0-
2.25
4.50
DoQE
Perplexity
GRAPE
2.00
4.25
CRAPE- qup
4.00
1.50
12k
20k
Steps
Steps
Stops
(a) Log- Log-Perplexity Scores
gsm8k
kodcode
hellaswag
0.3
GRAPE
GRAPE-4M9
Weights
0.4
0.8
GRAPE-gAp
0.2
0.6
0.1
0.4
0.0
0.0
Steps
Steps
Steps
(b) Task Weights
Weights
book
0.15
github
0.10
Mleckwcchange
0.05
wikipedia
Stapa
(c) Domain Weights
Figure 27: Ablation on task combination [GSM8K, Kodcode, Hellaswag].
39

---

E.9 Task Combination 8: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, Kodcode, GSM8K
guadde
20 4317
2r1, Chateeg
Iovlonde
4/-
Dtnge
Exgs
6011
1911
hellvway
4.-
4.1-
114
(a) Log- Jog-Perplexity Scores
11.30
1.0
2.35
0.000
134TP9 GIATE.OO
1.100
Pags
20gs
(b) Task Weights
0.5
Woights
stockenchange
wildpedia
(c) Domain Weights
Figure 28: Ablation on task combination [ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, Kodcode, GSM8K).
40

---

E.10 Task Combination 9: ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA
arc easy
arc arc_ challenge
SCig
Perplecity
4.75
4.0
4.50
1.25
4.25
3.5
4.00
22k
Stops
Steps
Steps
piga
logiqa
hellaswag
4.8
DuGE
4.6
4.2
4.2
Stops
Steps
Steps
(a) Log- Log-Perplexity Scores
easy
arc challenge
sciq
Weights
0.4
0.50
0.2
0.25
Steps
Stegs
Steps
piça
logiqa
hellaswag
0.6
GRAPE c003
GRAPH
0.2
GHAPP:-20 pup
0.4
DoGB
0.1
0.2
0.1
Steps
Stegs
Steps
(b) Task Weights
0.30
Weusht
0.25
book
0.15
0.10
stackonchange
0.05
wildpedia
13k
14k
Stops
(c) Domain Weights
Figure 29; Ablation on task combination IARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA].
41

---

E.11 Task Combination 10: ARC-E, ARC-C, Hellaswag, SciQ. PIQA, LogiQA, MathQA,
MedQA
STC .easy
are challenge
sciq
piga
5.5
3.5
Perplenty
4.5
4.5
4A
12k
208
48
06
12k
30a
206
86
126
106
200
46
66
12k
10L
20c
Stepp
Stopp
Stepa
log igg
heelaswag
meddg
3.0
DvdE
GRAZS
5.0
4.0
Stops
Steps
Stops
Steps
(a) Log-Perplexity Scores
casy
challenge
0.2
0.3
Weights
0.2
0.3
0.1
0.3
0.2
logiqa
helleswagg
ma thqs
medga
0.15
0.2
0.10
0.50
0.1
0.00
0.0
42
12h
10%
20%
CK.
Tae
15s 236
18t 28s
12A
18A 20k
Stapo
Stepo
Stops
Stops
(b) Task Weights
0.50
0.25
Weights
0.20
CC
0.15
cd
0.10
starkwcrbenge
wikipedis
124.
1.5k
201 Steps
(c) Domain Weights
Figure 30: Ablation on task combination [ARC-E, ARC-C, Hellaswag, SciQ, PIQA, LogiQA, MathQA, MedQA].
42

---

E.12 Task Combination 11: ARC-E, ARC-C, MathQA, MedQA
challenge
mathqa
medga
5.5
3.5
DaGE
Perpleetty
GRAZS
(a) Log- Log-Perplexity Scores
630 easy
mergge
Welehos
0.8
0.4
0.2
GIRAPE
6.1
0.1
GRAPE-WIN
022
Stoga
Stago
Stope
(b) Task Weights
0.25
0.20
Weigbis
book
0.15
stackouchange
wildipp dia
12k
1CK
Steps
(c) Domain Weights
Figure 31: Ablation on task combination [ARC-E, ARC-C, MathQA, MedQA].
43

---

E.13 Task Combination 12: LogiQA, Hellaswag, MathQA, MedQA
Logiga
heilaswag
mathga
medqq
DuGE
Pergleot
4.25
GRAPS
GRAIT--IT
4.5
5.0
3.775
3.30
4.5
(a) Log-Perplexity Scores
hellaswag
1.0
0.8
0.6
0.6
GRAIE-4ZA
0.2
Step4
Strps
Stops
Staps
(b) Task Weights
0.20
Weighte
book
github
slankwerhenge
wikipedis
Staps
(c) Domain Weights
Figure 32: Ablation on task combination [LogiQA, Hellaswag, MathQA, MedQA).
Hyperparameter Tuning
We perform hyperparameter search the reweighting intervals AT.. AT, 50, 100,200 200 for both GRAPE and DoGE. We train 125M model for 50k steps and report the final task accuracies and run time as Table 10. These results show that GRAPE consistently outperforms DoGE across all tested update frequencies, We also observe that the most frequent updates (AT,=50) do not yield the best performance. This likely because the Cate-of-Improvente after the model has trained for a sufficient number of steps on the current data mixture, making overly frequent weight adjustments suboptimal.
(Rol) metric becomes more reliable
Table 10: Hyperparameter Tuning Reweighting Frequency and Efficiency Comparison GRAPE and DoGE. GRAPE with AT, -T,=-A1.1100 outperforms 50 and 200.
AT., AT,
Method
ARC Challenee
ARC Easy
Hellaswag
Logiqu
PIQA
SciQ
Average
Runtime (I1100)
50
GRAPE
25.00%
45.83%
27.86%
25. 25.50%
60. 12%
73.30%
42.95%
23.9%
DoGE
24.40%
43.64%
27.53%
26.88%
58.98%
70.10%
41.02%
22.36
100
GRAPE
26.11%
47.14%
28.66%
27.65%
61.10%
74.40%
44.16%
15.80
DOGE
24.57%
45.33%
27.77%
25.35%
50.52%
72.20%
42.45%
15.06
200
GRAPE
23.8.1%
44.40%
27.77%
29.03%
58%
72. 70%
42.88%
10.3%
DOGE
21.32%
27.07%
12%
70.10%
44

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: [NA]
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of the work is discussed section
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
45

---

The proofs of Theorem 2.1 and Theorem 2.2 are provided in subsection C.1 and subsection C.2.
Justification:
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide the implementation details for GRAPE and baselines Ap- pendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
46

---

The authors should provide scripts to reproduce all experimental results for the new reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide the anonymous link to the codebase. All data used in the experi- ments are open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.c cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ httpssccccppbccccceceeceeecccl guggeuiesss.e. CooeeSubmis ssi iesionPooccyy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All implementation details included in Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: error bars are not reported because the pretraining experiments would be too computationally expensive
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
47

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We used 4 x H100 80GB GPUs for large-scale experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://neure: httppp//////u/s...////// httppublii/Ethicsuuiiiiie
Answer:
[Yes]
Justification:
We confirm that we follow the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- cration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [No]
Justification: The work does not have significant societal impact since mostly focuses on the data selection aspect for LLM pretraining.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
48

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer[[No)
Justification:
The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we properly cite the original paper that produced the code package or dataset.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswi twwode.cooo..... com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
49

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: the paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasettcoodemmodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
50

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: the LLM is used for editing and formatting purpose for this paper. The method development in this work does not involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy (https : /tt////eerppp//eeeee/eee/ee /tterrppp.cccoooeeeeeeeee.eeeeee /LLM) for what should or should not be described.
51


Reviews:
[{"review_id": "BfXyoZDNWB", "reviewer": "Reviewer_yAw7", "summary": "This paper considers the problem of choosing the best pretraining data mixture that result in good accuracy on a set of downstream target tasks. Specifically, the authors propose optimizing over the relative proportions/weights of a set of pretraining domain to minimize the worst case task (GRAPE). Group distributionally robust optimization (DRO) is applied to choose domain weights that result in as much (normalized) loss reduction as possible. Experiments on training language models for reasoning tasks and multilingual capabilities show improvements over baselines.", "questions": "- line 100: why is taking a small learning rate justified (in order to take the Taylor expansion)?\n - line 288: typo \"improvmenet\"\n - Theorem 2.1: this theorem seems to prove convergence to a global optimum, but this shouldn't be true for general (non-convex) loss functions?\n - Theorem 2.2: assuming that the task losses are strongly convex is a pretty strong/unrealistic assumption, but is this necessary, given that monotonicity if more of a local notion?\n - Given that the method is about minimizing the worst case loss, why is average accuracy what is mainly reported?\n - From 125M to 0.7B parameters, it seems like the improvement of GRAPE over the baseline is reduced; what are the prospects of GRAPE still outperforming the baseline when scaled up to 7B or beyond?\n - Can you comment on the runtime required of the different methods?\n - How well does GRAPE do compared to the oracle of selecting the best data for a given task? I'd like to understand how much of the value is being able to select data period versus being able to juggle multiple tasks.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- Overall, the problem (of finding good pretraining data for downstream tasks) is well-motivated. The paper proposes a new solution that leverages principles from robust optimization (building off of FAMO, which updates parameters to reduce task loss) and appears to work quite well.\n - The paper tries to upweight tasks that are improving more slowly, but I wonder if this is the right principle; if you have a task that has high error and cannot be improved, it seems like this will be maximally upweighted, which seems not ideal. Group DRO methods sometimes use a reference model that tries to take this into account; it would be nice to discuss this more.\n - Appendix D.1 says that DoGE used the same hyperparameters as GRAPE, but is there any reason to believe these are optimal? In general, for fair comparison, the hyperparameters need to be tuned carefully and separately for each method.", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the effort to read through our work and the constructive feedback! These questions have helped us clarify key aspects of our work. We provide our responses as follows:\n## Q1: Hyperparameter tuning & Runtime: \nWe perform a hyperparameter search on the reweighting intervals $\\Delta T_{\\alpha}, \\Delta T_{z} = 50, 100, 200$ for both GRAPE and DoGE. We train 125M model for 50k steps and report the final task accuracies and run time as follows:\n\n$\\Delta Τ_{\\alpha}, \\Delta Τ_{z}$ | Method | arc_challenge | arc_easy | hellaswag | logiqa | piqa | sciq | Average | Runtime (H100)\n----|--------|---------------|----------|-----------|--------|-------|-------|---------|--------\n50 | GRAPE | 25.09% | 45.83% | 27.86% | 25.50%| 60.12%| 73.30%| 42.95% | 23.9h \n 50 | DoGE | 24.40% | 43.64% | 27.53% | 26.88%| 58.98%| 70.10%| 41.92% | 22.3h\n100 | GRAPE | **26.11%** | **47.14%** | **28.56%** | 27.65%| **61.10%** | **74.40%** | **44.16%**  | 15.8 h\n100 | DoGE | 24.57% | 45.33% | 27.77% | 25.35%| 59.52%| 72.20%| 42.45% | 15.0 h\n200 | GRAPE | 23.81% | 44.40% | 27.77% | **29.03%** | 59.58%| 72.70%| 42.88% | 10.3h\n 200| DoGE | 24.32% | 43.31% | 27.07% | 28.42%| 59.14%| 70.10%| 42.06% | 10.8h\n\n\nThese results show that GRAPE consistently outperforms DoGE across all tested update frequencies. We also observe that the most frequent updates ($\\Delta T$=50) do not yield the best performance. This is likely because the Rate-of-Improvement (RoI) metric becomes more reliable after the model has trained for a sufficient number of steps on the current data mixture, making overly frequent weight adjustments sub-optimal.\n\nA side note on efficiency improvement: Regarding efficiency, while GRAPE has a theoretical FLOP overhead of ~ 28%, we note that the practical runtime is also largely influenced by the extra GPU memory occupation for storing extra gradients. A promising direction for future work is to incorporate a recently proposed \"ghost gradient computation\" technique [1], which could further improve the wall-clock efficiency for both GRAPE and DoGE. \n\nWe will add a discussion on the hyperparameter tuning in the updated version of our paper.\n\n## Q2: Average versus worst-case accuracy\nWe did not report worst-case accuracy as it can be volatile and skewed by a single, exceptionally difficult task, which may not reflect the model's broader capabilities across a diverse task suite. \n\nHowever, to directly evaluate the worst-case performance against our optimization objective (cross-entropy loss), we provide extensive results on worst-case log-perplexity in Table 5, Table 8, and Table 9. The results demonstrate that GRAPE consistently achieves superior (lower) worst-case perplexity compared to baselines across numerous task configurations.\n\n## Q3: Justification on the small learning rate assumption\nThe assumption of a small learning rate is a standard condition for applying the first-order Taylor expansion to approximate the one-step loss improvement, which is a common practice in the theoretical analysis of gradient-based algorithms. Empirically, it can be justified as the learning rates used in large-scale language model pretraining are typically considered small relative to the optimization landscape.\n\n## Q4: Assumptions on Theorems\n- **On Theorem 2.1**:  we agree with the reviewer that the general LLM training is a highly non-convex scenario. However, the convex assumption is commonly used for theoretical analysis for optimization to provide theoretical grounding and intuition for the algorithm's behavior in an idealized setting, such as the very first work on influence function [2]. Our empirical results show that even in the non-convex LLM pretraining practice, GRAPE with the underlying group-DRO algorithm can still provide valuable insight on the data mixture selection and brings significant empirical improvements. \n- **On Theorem 2.2**: We thank the reviewer for this sharp observation. We agree that the μ-strongly convex condition is not necessary to prove monotonic variance reduction. We will remove this assumption in the revised version of our paper.\n\n## Q5: Scaling behaviors of GRAPE\nWe thank the reviewer for this insightful question! While GRAPE outperforms all baselines at both the 125M and 0.7B scales, the relative improvement over the uniform baseline is less pronounced for the larger model. We hypothesize a few reasons:\n- (1) **Limited Training Tokens**: due to the limited computation budget, The 0.7B model was trained on 13B tokens, which may be insufficient to show the full benefits of a dynamic data curriculum at that scale; \n- (2) **Sub-optimal Hyperparameters**: The optimal hyperparameters may differ across model scales, and our settings may perform better on 125M models than 0.7B models;\n- (3) **Shifting Task Dynamics**: while the model is scaling up, the conflict pattern among various target tasks can be shifted. For example, larger models can be more sensitive to task conflicts or easier to generalize across various domains. Understanding the scaling effect of the multi-task learning behavior is an intriguing direction to explore. We will leave it to our future work.\n\n## Q6: Comparison to the oracle of data selection\nWhile the true oracle of data mixture is hard to define in the context of LLM pretraining, we apply CRISP as a proxy for a static oracle baseline. CRISP identifies the domain sampling weights as the closest training data distribution to the target data distribution according to the sequence embedding similarity. Therefore, the CRISP mixture is determined using an auxiliary embedding model (e.g. sentence-bert) before the training begins.  \nOur results show that GRAPE's dynamic approach significantly outperforms this static oracle. In Table 1,5,6,7,8, GRAPE largely improves perplexity and reasoning accuracies compared to CRISP; according to Figure 19,20, in the multilingual setting, the static mixtures from CRISP can sabotage performance on most languages, whereas GRAPE delivers consistent gains on all the target languages, leveraging the dynamic data curriculum.\n\n[1] Data Shapley in One Training Run. Wang et. al, 2025.\n\n[2] Understanding Black-box Predictions via Influence Functions. Koh and Liang, 2018."}, {"review_id": "52StWCbfb7", "reviewer": "Reviewer_P62V", "summary": "This paper docuses on optimizing data mixtures in pretraining language models with the goal of improving performance of the models on multiple target tasks at the same time. This is an extension of past research which set this up mostly with the goal of optimizing perormance on a single target task. \n\nThe paper introduces a method to adjust dynamically both sample weights from source domains, as well as task weights. Adjusting task weights dynamically allows the model to prioritize different tasks at different times in training. The learning problem is formalzied as a minimax optimization problem, where the task weights are selected using group distributed-robest-optimization and the domain weights are maximizing loss reduction on the weighted tasks.\n\nThe experiments are conducted on two setups:\n1. learning from ClimbLab and SlimPajama and testing on a set of 6 reasoning dataset using 5-shot in-context learning\n2. learning from Wikipedia data from 6 languages and evaluating using perplexity on 8 other languages", "questions": "-  baselines would require a bit more description for example it is not clear what the domain weights are trained and their frequency in regmix\n\n- line 244:  Ukrian > Ukrainian", "limitations": "Highlighting these limitations of the research, but I acknowledge the space and scope limitations of the current paper:\n\n- There is still to be studies what is happening after running SFT/RL on top of these trained models\n- Models trained are only up to 0.75 on limited training steps\n- Comprehensive baselines, but only for the first experiment; this is probably fine given that DoGE is the best non-GRAPE method in that experiment. But for completeness, it would be good to have more baselines for the other experiments", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strenghts:\n- new problem/experimental setup (multi-target testing)\n- theoretical justification of the algorithm \n- comparison with many baselines for the first experiment, including all natural ones from past work\n- experimentation on 2 different setups\n- feature analysis and interpretation of results\n\nWeaknesses:\n- evaluation for the multi-lingual setup is only done using perplexity, which may not be related to actual task performance. One could have evaluated instead on downstream tasks with ICL (e.g. NER or classification) or machine translation. These would make these results stronger", "was_revised": true, "final_justification": "Thank you for the additional clarifications. These were relatively minor points, and the response will improve the revision. I remain positive overall on the paper.", "rebuttal": "We thank the reviewer for the thoughtful questions and constructive suggestions! We provide our response as follows:\n## Q1: Clarification on the baselines\nWe provide detailed implementation descriptions for all baseline methods in **Appendix D.1** (line 620-674). We also provided the distribution of regmix-optimized domain weights in Figure 7,8,11,12,16,17. The actual domain weights from RegMix on the CLIMBLAB dataset with 6 target tasks is presented as follows:\n\n| C1   | C2   | C3  | C4  | C5  | C6 | C7 | C8 | C9 | C10 |\n|-------|-----|------|-------|------|-----|-----|-----|----|------|\n|0.00 | 0.01 |0.00 |0.19 |0.01 |0.03 |0.02| 0.02 | 0.01 | 0.00 |\n|C11 | C12  | C13 |C14 | C15 |C16 | C17  | C18  | C19 | C20 |\n|0.01 | 0.02 | 0.00 |0.00 |0.02 |0.01| 0.62 | 0.02| 0.00 |0.00|\n\nWe will add the tables with the actual domain weights from all baseline methods in our revised version for better readability and clarity!\n\n\n## Q2: Performance after SFT\nTo investigate if GRAPE's pretraining advantages remains after post-training, we conducted supervised fine-tuning experiments on the pretrained 125M models. \nWe present the 5-shot accuracy scores before and after supervised fine-tuning on PIQA, SciQ, ARC-Easy, ARC-Challenge as follows:\n\n| Method | Base | Base + SFT | DOGE | DOGE + SFT | GRAPE | GRAPE + SFT |\n|--------|------|-------------------|------|-------------------|-------|-------------------|\n| PIQA | 58.98| 63.06 | 59.52| 62.62 | 61.10 | **64.25** |\n| SciQ | 67.50| 75.3 | 72.20| 79.0 | 74.40 | **79.70** |\n|ARC-E| 40.74| 44.23 | 45.33 | 48.44|47.14| **48.86**|\n|ARC-C | 23.12 |  **27.22** | 24.57 | 26.54 | 26.11 | 27.13|\n\nThe results show that GRAPE keeps the advantage after the SFT stage in most of the tasks. On ARC-Challenge, GRAPE outperforms DoGE and performs comparably as the uniform baseline.\nWe did not conduct the RL here since we did not apply a relevant task as our target, but how to select the optimal data for RL is definitely a critical and challenging direction to explore.  \n\n## Q3: Limited training length and baselines in large-scale experiments\nDue to the limitation of computation budget, we did not manage to pretrain models larger than 0.7B on sufficient numbers of tokens. \nAs the reviewer notes, we prioritized comparing against DoGE as it was the strongest baseline under the limited resources but did not include RegMix and CRISP baselines in 0.7B experiments. We will be sure to add the results for RegMix and CRISP to the 0.7B experiments in the final version of the paper to ensure a complete comparison."}, {"review_id": "p5WBgQ7v36", "reviewer": "Reviewer_aZMB", "summary": "This paper presents a novel method for dynamically rebalancing task weights in a bilevel optimization for data weighting where multiple tasks are simultaneously being optimized for. This approach significantly outperforms previous methods and baselines on a variety of experiments. The authors present interesting experiment setups which allow for deeper analysis of the functioning of the method. The paper concludes by presenting many compelling directions for extending this work.", "questions": "The authors present an approach where a first-order taylor expansion is used to approximate RoI in order to provide the setup for applying GroupDRO. Did the authors impelment the simpler version of just calculating RoI directly per-step and updating the task-weights via a simple heuristic (ie using the softmax of the inverse RoIs)? This seems a simple and efficient baseline that would be useful to compare against before getting more complicated.\n\nIn section 4, much space is spent on comparing GRAPE to GRAPE-ema and GRAPE-gap (l 285-314, fig 6, table 2). This is fine to establish that GRAPE is the better of the three methods but not terribly interesting beyond that. Relatively little space is spent on the next section (l 315-326) discussing the relative performance of GRAPE under different task distribution settings, where the results are referred to in the appendix. Is this not critical information to understand both how the method is functioning and in what settings it might be useful to be applied? Consider showing less of the GRAPE-ema comparison and more of the GRAPE-in-different-task-settings. \n\nDid the authors run any experiments where the relative improvements between tasks are not uniformly weighted? In practical terms, practitioners may prioritize some tasks over others in their downstream evaluations, so it would be very useful and interesting to know if the strong performance of GRAPE extended to these settings. Do the authors have any comments / intuitions on this?\n\nNits:\nline 288:  typo on \"improvement\"", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\nThe paper is clearly written, the algorithm presented is a novel approach to the problem. The results are very compelling. The presented analyses are very thorough and provide fascinating insight into the function of the method. The result is very likely to be used by the research community in future work.\n\nWeaknesses:\n\nmagical hyperparams:\n(This is bit of a nit, and does not detract terribly from the overall high quality of this work)\nIt makes sense that ∆T_z and ∆T_alpha should not be 1, for both computational cost and diminishing retuns on frequency of update. It would be very useful however to have some sense of what this tradeoff looks like in practice rather than simply punting the hyper-param tuning (lines 328-334). It does not need to be exhaustive, but it would be nice to include *some* analysis of this tradeoff. Did the authors ever run any exp with ∆T == 50? == 200? A simple sweep on a tiny model would suffice.", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful to the reviewer for their careful assessment of our work and the insightful feedback. We agree with the suggested points and provide our responses below:\n\n## Q1: Hyperparameter tuning & Runtime: \nWe perform a hyperparameter sweep on the reweighting intervals of $\\Delta T_{\\alpha}, \\Delta T_{z} = 50, 100, 200$ for both GRAPE and DoGE. We train 125M model for 50k steps and report the final task accuracies and run time as follows:\n\n$\\Delta Τ_{\\alpha}, \\Delta Τ_{z}$ | Method | ARC-C | ARC-E | Hellaswag | LogiQA | PIQA | SciQ | Average | Runtime (H100)\n----|--------|---------------|----------|-----------|--------|-------|-------|---------|--------\n50 | GRAPE | 25.09% | 45.83% | 27.86% | 25.50%| 60.12%| 73.30%| 42.95% | 23.9h \n50 | DoGE | 24.40% | 43.64% | 27.53% | 26.88%| 58.98%| 70.10%| 41.92% | 22.3h\n100 | GRAPE | **26.11%** | **47.14%** | **28.56%** | 27.65%| **61.10%** | **74.40%** | **44.16%**  | 15.8 h\n100 | DoGE | 24.57% | 45.33% | 27.77% | 25.35%| 59.52%| 72.20%| 42.45% | 15.0h\n200 | GRAPE | 23.81% | 44.40% | 27.77% | **29.03%** | 59.58%| 72.70%| 42.88% | 10.3h\n200 | DoGE | 24.32% | 43.31% | 27.07% | 28.42%| 59.14%| 70.10%| 42.06% | 10.8h\n\nThese results show that GRAPE consistently outperforms DoGE across all tested update frequencies. We also observe that the most frequent updates ($\\Delta T=50$) do not yield the best performance. This is likely because the Rate-of-Improvement (RoI) metric becomes more reliable after the model has trained for a sufficient number of steps on the current data mixture, making overly frequent weight adjustments sub-optimal.\n\n**A side note on efficiency improvement:** Regarding efficiency, while GRAPE has a theoretical FLOP overhead of ~ 28%, we note that the practical runtime is also largely influenced by the extra GPU memory occupation for storing extra gradients. A promising direction for future work is to incorporate a recently proposed \"ghost gradient computation\" technique [1], which could further improve the wall-clock efficiency for both GRAPE and DoGE. \n\nWe will add a discussion on the hyperparameter tuning in the updated version of our paper.\n\n\n## Q2: GRAPE with exact ROI scores\nWe intended to predict the Ratio-of-Improvements at the next step using a look-ahead strategy, where the first-order approximation stands. Also, the gradient alignment between tasks and training set can provide more information about the optimization directions on the landscape than the exact ROI score. \n\nTo validate this, we ran an experiment with the exact ROI scores, where we calculated ROI directly to update task weights at every reweighting step and then update domain weights according to the task weights. The 5-shot accuracy on the target tasks are presented as follows:\n\nMethod | ARC-C | ARC-E | Hellaswag | LogiQA | PIQA | SciQ | Avg. \n----|--------|---------------|----------|-----------|--------|-------|-------\nUniform | 23.12% | 40.74% | 27.02% | 27.96%| 58.98%| 67.50%| 40.89% \nGRAPE | 26.11% | 47.14% | 28.56% | 27.65%| 61.10%|74.40%| 44.16%\nDoGE | 24.57% | 45.33% | 27.77% | 25.35%| 59.52%| 72.20%| 42.45% \nGRAPE+direct ROI | 23.29% | 42.76% | 27.16% | 27.50% | 56.86%| 72.80% | 41.73%\n\nThe results above show that the underlying GroupDRO framework in GRAPE is more effective than the directly computed ROI scores. We will add more discussions and analysis on this ablation in the revised version of the paper.\n \n## Q3: Apply on tasks with priority preference\nGRAPE is primarily designed for robustness, where the goal is to achieve balanced performance by automatically identifying and lifting the lagged tasks. \nIn practice, we can run GRAPE with task-weights initialized non-uniformly according to the pre-defined task priority. However, the principle of GRAPE that continuously optimizes the worst-improved task could conflict with the predefined priority-order. For instance, GRAPE might upweight a low-priority task if it falls significantly behind, which would counter the user's explicit preference.\n\nTherefore, if a practitioner has pre-defined, static priorities for tasks, applying DoGE with a static non-uniform task-weights would be more appropriate. \n\n[1] Data Shapley in One Training Run. Wang et. al, 2025."}, {"review_id": "ofiWTbQYTj", "reviewer": "Reviewer_qSR5", "summary": "This paper points out that pretraining has two goals: to reduce the loss on domains in the pretraining corpus, and to improve on downstream tasks. The authors introduce GRAPE, an algorithm that balances these two goals. GRAPE works similar to DOGE, using the DOGE update rule for both pretraining domain weights and downstream task weights.", "questions": "1. Do you have feedback for the reader on tuning hyperparameters? Hyperparameter tuning seems critical here because there are many of them.\n2. What do you think of other data mixing methods like Data Mixing Laws, Aioli, and Adaptive Data Optimization? These methods aren't derived from the DRO framework. I'd be interested in seeing a little commentary what the benefit of the minimax DRO framework is, over just a minimization framework seen in the methods above and RegMix.\n3. Some information about wall-clock time would be helpful. Although GRAPE is an online method, it will take longer than normal pretraining because you're taking lots of gradients on validation sets.\n4. Relatedly, how would GRAPE interact with pretraining corpora that have many partitions, like DCLM-Baseline or WebOrganizer? In these cases it seems like you would have many validation gradients, in which case you would be decreasing your throughput because you have to allocate more of each batch to computing validation gradients.", "limitations": "Yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "Strengths: The core points of the paper---that pretraining aims to improve performance on many downstream tasks, and that existing algorithms don't explicitly do this---are well made. The authors also make thoughtful evaluations on the curriculum and multilingual training. The results are strong and GRAPE seems to work. I think GRAPE is a well-motivated and natural evolution of DoReMi and DoGE.\n\nWeaknesses: In using DoGE update rule twice, the GRAPE algorithm naturally more hypeparameters than DoGE. This can potentially be a challenge for hyperparameter turning.\n\nQuality: The paper is impressively thorough.\nClarity: I found the paper easy to follow. I clearly understood both GRAPE's connection to and clear separation from prior work.\nSignificance: To my knowledge, this is the first algorithm that adjusts both domain and downstream task weights.\nOriginality: Ideas build on prior work but are applied to a new setting.", "was_revised": true, "final_justification": "Technically solid work. The authors adequately addressed my questions and provided creative and thought-provoking solutions. I have published recently in this area and am familiar with the existing literature, and this paper is an original departure from previous work. Thus, I maintain my score of accept and confidence of 5.", "rebuttal": "We thank the reviewer for the thoughtful questions and constructive suggestions! We provide our response as follows:\n\n## Q1: Hyperparameter tuning & Runtime: \nWe perform a hyperparameter search on the reweighting intervals $\\Delta T_{\\alpha}, \\Delta T_{z}$ = 50, 100, 200 for both GRAPE and DoGE. We train 125M model for 50k steps and report the final task accuracies and run time as follows:\n\n$\\Delta Τ_{\\alpha}, \\Delta Τ_{z}$ | Method | arc_challenge | arc_easy | hellaswag | logiqa | piqa | sciq | Average | Runtime (H100)\n----|--------|---------------|----------|-----------|--------|-------|-------|---------|--------\n50 | GRAPE | 25.09% | 45.83% | 27.86% | 25.50%| 60.12%| 73.30%| 42.95% | 23.9h \n50| DoGE | 24.40% | 43.64% | 27.53% | 26.88%| 58.98%| 70.10%| 41.92% | 22.3h\n100 | GRAPE | **26.11%** | **47.14%** | **28.56%** | 27.65%| **61.10%** | **74.40%** | **44.16%**  | 15.8 h\n100| DoGE | 24.57% | 45.33% | 27.77% | 25.35%| 59.52%| 72.20%| 42.45% | 15.0h\n200 | GRAPE | 23.81% | 44.40% | 27.77% | **29.03%** | 59.58%| 72.70%| 42.88% | 10.3h\n200| DoGE | 24.32% | 43.31% | 27.07% | 28.42%| 59.14%| 70.10%| 42.06% | 10.8h\n\nThese results show that GRAPE consistently outperforms DoGE across all tested update frequencies. We also observe that the most frequent updates ($\\Delta T=50$) do not yield the best performance. This is likely because the Rate-of-Improvement (RoI) metric becomes more reliable after the model has trained for a sufficient number of steps on the current data mixture, making overly frequent weight adjustments suboptimal.\n\n**A side note on efficiency improvement:** Regarding efficiency, while GRAPE has a theoretical FLOP overhead of ~ 28%, we note that the practical runtime is also largely influenced by the extra GPU memory occupation for storing extra gradients. A promising direction for future work is to incorporate a recently proposed \"ghost gradient computation\" technique [1], which could further improve the wall-clock efficiency for both GRAPE and DoGE. \n\nWe will add a more comprehensive discussion on the hyperparameter tuning in the updated version of our paper.\n\n## Q2: Benefits of the Minimax DRO Framework vs. Minimization-Only Frameworks \nThe primary benefit of GRAPE's minimax (min-max) framework is that it explicitly optimizes for robustness of multi-task learning performance by adaptively tuning the task weights, while the minimization-only frameworks (like RegMix, DoGE, or Aioli) only optimizes the domain weights according to a static, pre-defined target distribution. \nTherefore, GRAPE is able to adaptively adjust the guidance distribution (i.e. task weights) for balanced and robust improvements across a diverse suite of benchmarks.\n\n## Q3: Scalability to a large number of target tasks\nWe thank the reviewer for the insightful question! The scalability of gradient-based approaches is indeed a critical issue since computing gradients for all $N$ target tasks can become a bottleneck. While our experiments used a manageable number of tasks, the GRAPE framework is flexible and can be adapted for scalability using two primary strategies: \n\n#### Method 1: Task Reweighting with Partial Feedback \nInstead of computing gradients for all $N$ tasks at every update, we can sample a small subset of $m << N$ tasks. This can be done in a principled manner using techniques from the multi-armed bandit literature (e.g., EXP3.M [2]), which balance exploiting known slow-learning tasks with exploring others. This would reduce the per-update cost from $O(N)$ to $O(m)$ without losing the core benefit of the algorithm.\n\nDenote task weights as $z(t)$ as in the paper, define a sampling distribution:\n$$p_n^{(t)} = (1-\\epsilon) \\cdot z_n^{(t)} + \\frac{\\epsilon}{N}$$\n\nto balance exploitation (focusing on tasks currently believed to be important) with exploration (ensuring all tasks remain observable).\nFor each sampled task $n \\in S_t$, we compute the alignment score to approximate ROI scores:\n\n$A^{(t)} = \\[a^{(t)}_{1}, \\hdots, a^{(t)}_{N}\\]$, where $a^{(t)}_{n} = \\langle \\Delta_{\\theta} l_n^{val}, d_t \\rangle$\n\nwhere $\\Delta_\\theta l_n^{val}$ is the normalized validation gradient for task $n$, and $d_t$ is the current model update (a weighted combination of training domain gradients). \nTo convert this alignment signal into a bandit-style reward in [0,1], we rescale it as follows:\n$$r_{n}^{(t)} = \\frac{1}{2} (1 - a_{n}^{(t)}/C)$$\nwhere $C > 0$ is a scaling constant that upper-bounds the magnitude of alignment scores (e.g., a norm bound). This ensures that $r_{t,n} \\in \\[0,1\\]$, with larger rewards assigned to more misaligned tasks—consistent with GRAPE's principle of prioritizing underperforming tasks.\nWe then update the task weights using the standard multiplicative-weights rule (as in EXP3.M [2]), with importance-weighted feedback:\n$$z_{n}^{(t+1)} \\leftarrow z_{n}^{(t)} · exp(\\eta_{z} · r_{n}^{(t)}/p_{n}^{(t)} · 1_{n \\in S_t})$$\n\nThis procedure allows GRAPE to focus on the most corrective tasks **using only a small number of gradient evaluations per step**.\n\n#### Hierarchical Task Reweighting\nFor corpora with thousands of fine-grained partitions, we can first group similar domains or tasks into semantically coherent clusters based on metadata or embeddings similarity. GRAPE would then operate on these high-level clusters, reweighting groups of tasks rather than individual ones. This dramatically reduces the number of entities to reweight, making the process highly scalable.\n\n[1] Data Shapley in One Training Run. Wang et. al, 2025.\n\n[2] Algorithms for adversarial bandit problems with multiple plays. T. Uchiya, A. Nakamura, and M. Kudo, 2010."}, {"review_id": "CZ0cWGU1su", "reviewer": "Reviewer_qr9c", "summary": "This paper addresses the challenge of balancing training across multiple source and target domains in language model pretraining. It optimizes the Distributed Robust Optimization (DRO) algorithm by using the rate of improvement on test sets as a metric, rather than just training set loss. This allows the method to dynamically adjust task weights to prioritize more challenging tasks, and subsequently adjust domain weights to better achieve this learning objectives. The efficacy of this approach is demonstrated through experiments in both monolingual and multilingual environments.", "questions": "1. Generalization to Unseen Tasks: The primary objective of pretraining LLMs extends beyond achieving superior performance on a predefined set of tasks (six in the main experiments of this paper). LLMs are also expected to exhibit strong generalization capabilities. How does this training method perform on tasks not included in the training process, particularly those requiring different types of abilities or knowledge?\n2. Applicability to Broader Scenarios: Can this method be effectively extended to broader practical applications, such as instruction tuning or continued pretraining?", "limitations": "Yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Simplicity and Theoretical Soundness: The proposed method is straightforward and direct, backed by theoretical guarantees that support its overall effectiveness.\n2. Clarity of Writing: The paper is well-written, making the methodology and results easy to understand.\n\nWeaknesses:\n1. Code Availability: The authors do not provide code for their implementation.\n2. Limited Test Set Scope: The experiments utilize a relatively small number of test sets.", "was_revised": true, "final_justification": "The authors' paper is a complete piece of work and makes a meaningful contribution. Their response has mostly addressed my concerns. Regarding my second question, it is understandable that the authors could not complete the additional experiments within the limited rebuttal period. Therefore, considering the relatively positive score I have already given, I will maintain my rating.", "rebuttal": "We thank the reviewer for the thoughtful questions and constructive suggestions! We provide our response as follows:\n## Q1: Generalizability to unseen tasks\nWe evaluate the model optimized using the 6 reasoning tasks from our main experiment on 6 unseen tasks and report the 5-shot reasoning accuracy as follows:\nMethod      \t| RACE  | Winograd | OpenbookQA | MMLU  | CommonsenseQA | COPA\n----------------|-------|----------|--------|-------|--------|-------\nUniform     \t| 25.65 | 49.80\t| 15.40  | 25.07 | **20.15**  | 57.00\nDoGE        \t| 26.12 | 49.80\t| 26.40  | **26.43** | 19.82  | 60.00\n**Ours**    \t| **27.37** | **52.49** | **28.60** | 25.53 | 18.92 | **61.00**\n\nOur method outperforms both the Uniform and the strong DoGE baseline on a majority (4 out of 6) of these unseen benchmarks, which suggests that GRAPE's advantages generalize well beyond the seen tasks. \n\n\n\n\n## Q2: Broader applications to Post-training \nGRAPE is highly applicable as a general data mixture optimization method to broader scenarios, including continued pretraining and instruction tuning. \nDuring the post-training stage like instruction tuning, a more fine-grained data selection method can be preferable like a sample-level selection to get a better control of the final model behavior. Using sample-level DRO to upweight the most difficult individual instructions is a powerful concept for the SFT stage and a promising direction for future research, as we discussed in the Discussion section (line 353-359).\n\nWe will add the above discussions to the revised version of the paper. For code availability, according to the rebuttal guideline, we cannot put external links here. We promise to open-source our codebase in the updated version."}]

Author rebuttal:

Meta-review: The paper introduces a new method called GRAPE for data mixing for pretraining large language models. The main problem it addresses is that existing techniques often optimize the pretraining data for a single goal/metric, which can hurt the model's performance on other tasks. GRAPE is designed to improve a model's performance across multiple different tasks at the same time. It works by dynamically adjusting two things during training: the mix of the source data and the importance given to each target task. The system identifies tasks that the model is struggling to learn and prioritizes them by giving them higher weight. Then, it adjusts the data mixture to specifically help improve performance on these prioritized, harder tasks. Reviewers agreed that this was a well-motivated and novel extension of previous work, with strong experimental results.

The reviewers' feedback was mostly positive, leading to three "Accept" and two "Borderline Accept". Initially, several issues raised including the practicality of the method, such as its runtime cost, the difficulty of tuning its many hyperparameters, and whether its benefits would diminish as models get larger. Reviewers also questioned aspects of the evaluation, such as the fairness of hyperparameter settings for baseline methods and the scope of the test sets. Despite these initial concerns/questions, the authors' responses addressed most of these concerns and overall, the reviewers concluded that the paper made a meaningful contribution to the field. The authors should include the additional experiments and discussions in the revised paper.
Decision: Accept (poster)