HBO: Hierarchical Balancing Optimization for Fine-Tuning Large Language Models
Authors: Weixuan Wang
Minghao Wu
Barry Haddow
Alexandra Birch
Venue: ICLR 2026 Poster (2026)
Keywords: Large Language Models
Fine-Tuning
Multilingual
Multitask
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Fine-tuning large language models (LLMs) on a mixture of diverse datasets poses challenges due to data imbalance and heterogeneity. Existing methods often address these issues across datasets (globally) but overlook the imbalance and heterogeneity within individual datasets (locally), which limits their effectiveness. We introduce Hierarchical Balancing Optimization (HBO), a novel method that enables LLMs to autonomously adjust data allocation during fine-tuning both across datasets (globally) and within each individual dataset (locally). HBO employs a bilevel optimization strategy with two types of actors: a Global Actor, which balances data sampling across different subsets of the training mixture, and several Local Actors, which optimizes data usage within each subset based on difficulty levels. These actors are guided by reward functions derived from the LLM's training state, which measure learning progress and relative performance improvement. We evaluate HBO on three LLM backbones across nine diverse tasks in multilingual and multitask setups. Results show that HBO consistently outperforms existing baselines, achieving significant accuracy gains. Our in-depth analysis further demonstrates that both the global actor and local actors of HBO effectively adjust data usage during fine-tuning. HBO provides a comprehensive solution to the challenges of data imbalance and heterogeneity in LLM fine-tuning, enabling more effective training across diverse datasets.

HBO: HIERARCHICAL BALANCING OPTIMIZATION FOR FINE-TUNING LARGE LANGUAGE MODELS
Weixuan Wanglt
Minghao Wu
Barry Haddow 1.3
Alexandra Birch 1.3
School of Informatics, University of Edinburgh
Toogyyi Lab, Alibaba Group
3 Aveni
(weixuan. wang, bhaddow, a. birch ac.uk minghao. iweellbbba-inn.com
ABSTRACT
Fine-tuning large language models (LLMs) mixture of diverse datasets poses challenges due to data imbalance and heterogeneity. Existing methods often address these issues across datasets (globally) but overlook the imbalance and heterogeneity within individual datasets (locally), which limits their effectiveness. We introduce Hierarchical Balancing Optimization (HBO), a novel method that enables LLMs to autonomously adjust data allocation during fine-tuning both across datasets (globally) and within each individual dataset (locally). HBO employs bilevel optimization strategy with two types of actors: a Global Actor, which balances data sampling across different subsets of the training mixture, and several Local Actors, which optimizes data usage within each subset based on difficulty levels. These actors are guided by reward functions derived from the LLM's training state, which measure learning progress and relative performance improvement. We evaluate HBO on three I.I.M backbones across nine diverse tasks in multilingual and multitask setups. Results show that HBO consistently outperforms existing baselines, achieving significant accuracy gains. Our in-depth analysis further demonstrates that both the global actor and local actors of HBO effectively adjust data usage during fine-tuning, HBO provides a comprehensive solution to the challenges of data imbalance and heterogeneity in LLM fine-tuning, enabling more effective training across diverse datasets. In addition, we release the code to foster research along this line
INTRODUCTION
Large language models (LLMs) have demonstrated remarkable capabilities in understanding, rea- soning, and generating answers across diverse tasks (Reid et al.) 2024 DeepSeek-AI et al. 2025
OpenAl 2024a|c) c). One of the key factors contributing to the success of LLMs is the dataset mixture
used during their fine-tuning process (Taori et al. 2023 Yang al. 2024b Muennighoff et al.) 2023
Chung et al. 2024a). These dataset mixtures typically encompass a diverse range of tasks, domains, and languages, ensuring that the models can perform well across wide variety of applications (Yang
ct al. 2024a Dubey et al. 2024 Team et al. 2025 Martins et al. 2024 Aryabumi et al.) 2024).
critical challenge in fine-tuning LLMs is managing data imbalance and heterogeneity, which arise from the diverse nature of tasks and datasets used for fine-tuning Data imbalance refers to the uneven distribution of examples across different tasks, domains, or languages (Liu al.) 20200 Kamalov & Denisov 2020) PPuyaanar et al., 20188) Wang et al.) (2019). while data heterogeneity encompasses variations in the characteristics of the data, such as quality and difficulty (Liu et al. 20224] Hendrycks
al. 202 Li et al, 2024b) Albalak et al. 2023). Addressing these factors is crucial for achieving optimal performance, as imbalanced or heterogeneous data can lead to overfitting on certain tasks and underperformance others (Zhao al. 2023 Li et al. 2024b). Recent research highlights the
Equal contribution.
hnttps://giie wwtpgitthu..cc/.iiiiiiuuue http:///eixxuaaaaag122233//////

---

importance of strategically balancing data from various sources to mitigate these issues (Wei et al. 2021; Iyer et al. 2022; Wu et al., 2024b). However, existing methods often assume that datasets are internally balanced and homogeneous, which may not hold true in practice, as the examples from the same sources may also exhibit different characteristics (Schwartz & Stanovsky [022 He & Garcia 2009). This limitation underscores the need for more effective strategies that manage imbalance and heterogeneity both globally (across datasets) and locally (within datasets), Developing such strategies is challenging, as finding the optimal allocation of data usage across these dimensions requires substantial efforts and resources. This raises a natural research question:
Can LLMss determine optimal data usage strategies by themselves to address imbalance and heterogeneity?
To address this research question, we propose novel framework, Hierarchical Balancing Opti- mization (HBO), that enables LLMs to autonomously adjust their data allocation both globally and locally based on their current training state. Our method employs bilevel optimization framework (Colson et al. 2007). where the outer problem minimizes the objective function of training the LLM over a mixture of training datasets, and the inner problem adjusts the sampling probabilities both globally (across datasets) and locally (within datasets). To achieve this optimization, we introduce two types of actors: Global Actor and Local Actor. The global actor responsible for balancing data allocation across the subsets of the training data mixture, while the local actor for each individual subset optimizes data usage within the subset. Specifically, we categorize the examples in each subset into four groups based on their difficulty levels. To guide the global and local actors, we define two reward functions based on the training state of the LLM. The global reward is computed as the L2 norm of the gradients, which reflects the current learning progress of the model. The local reward is defined as the ratio of the perplexities given by the tine-tuned LLM and the original LLM, capturing the relative improvement in model performance on specific groups. By integrating these components, our HBO framework effectively optimizes the training process of L.MMs.
To validate the effectiveness of HBO, we conduct extensive experiments using three model backbones: Llama-3. 1-BB, Qwen2.5-7B, and EuroLLM-9B. These experiments span two setups, covering a total of nine tasks: Multilingual setting (MMMLU, XCOPA, XStoryCloze, XNLI, and MGSM) and Multitask setting (MMLU, MultiFin-EN, GSMBK, and MedMCQA). Our results demonstrate that HBO consistently outperforms existing sampling strategies, achieving substantial improvements in model performance.
Our contributions in this work can be summarized as follows:
We propose HBO, a novel hierarchical dynamic data sampling method that enables LLMs to autonomously address data imbalance and heterogeneity during fine-tuning. By leveraging a bilevel optimization framework with two types of actors, HBO dynamically adjusts sampling probabilities both globally and locally (see (see|Section 3).
We demonstrate the broad applicability and effectiveness of HBO across multiple LLM backbones and tasks. Through extensive experiments with three model backbones over nine diverse tasks, HBO consistently outperforms existing sampling strategies, achieving substantial accuracy improvements. Visualizations of the sampling probabilities reveal that HBO dynamically adjusts these probabilities in a fascinating cyclical pattern, highlighting its ability to adaptively focus areas that enhance model learning (see [Section 4.2).
We conduct extensive analyses to investigate the contributions of the global and local actors, the robustness of HBO varying sampling priors, the impact of data volume, and more. Additionally, we highlight the critical role of incorporating easy examples, which are often discarded during fine-tuning. in boosting model performance (see Section 5).
2 RELATED WORK
Fine-Tuning with Multi-Datasets Recent advances in LLMS have shown that utilizing diverse datasets for both pre-training and fine-tuning is crucial for developing robust, generalized models (Team et al. 2024| Anthropic 2024 OpenAl 2024b ch. Multi-task learning, which fine-tunes models on multiple tasks or languages simultancously. leverages shared knowledge and improves overall performance (Crawshaw) 2020: Zhang et al., [022] Le Scao et al. 2023 Aryabumi et al. 2024). Despite these successes, critical challenges remain, particularly in balancing diverse task objectives
2

---

updale for Environment
Environment
updale Actor
LLM
Train Data
sampling
Global Reward
probabilty
Global
Actor
Local Reward
Local Reward 2
Local Reward
Local
Local
Local
Actor1
Actor2
Actor3
The bilevel optimization framework of HBO. Global Actor and Local Actors jointly adjust data sampling probabilities both globally (across datasets) and locally (within datasets) to optimize the LLM parameters. Based on the LLM's s training state, the global reward and the local reward are computed to guide the optimization of the global and local actors, respectively.
Figure 1:
(Chen et al. 2018 Kendall et al. 2018 et al. 2020; Wang et al. 2020c enhancing cross- lingual transferability (Kew al. 2024 Chen al. 2024 Shaham al.
Khalifa et al. 2024) and
2024)
Data Balancing Recent studies emphasize strategies ensure that training datasets not only diverse but also reflective of various domain-specific characteristics (Wei et al. 2021 Iyer et al. 2022
Chung et al./ [2024b] Liu al. 2020; Kamalov & Denisov 22020). Prior static balancing approaches
apply fixed sampling probabilities throughout training (Arivazhagan et al.) 2019 Conneau et al.
2020). In contrast, dynamic balancing methods adapt these probabilities over time (Wang al.
2020b; Wu et al., 2021] Zhu et al. [024]] Wang et al. 2019),, often by incorporating another scorer network, to more effectively manage the evolving learning process (Wang al. 2020bb a Wu et al. 2024a). However, most of these methods primarily address differences between datasets (global balaneing) while often neglecting the internal diversity found within a single dataset (local balancing).
Ours Our work introduces HBO that addresses data imbalances and heterogeneity at both global and local levels. Unlike prior methods that focus solely on global balancing (Wang et al. [0200b] Wu ctt al., 2024a), our approach simultaneously optimizes sampling across datasets and within individual datasets, ensuring effective utilization of heterogeneous training data.
METHODOLOGY
3.1 PRELIMINARIES
Supervised Fine- Tine-Tuning Supervised fine-tuning (SFT) is process that enables LLMs to follow and respond to human instructions. During SFT, LLM with parameters 0 is trained on instruction- response pairs to improve its ability to generate helpful, accurate responses to user queries. For single training dataset with M1 examples D' = { (Ixx.)))) ) D6 is the instruction (input prompt), and yk is the desired response, the SFT objective minimizes the negative likeelooooood:
Mi
LD(D):))
= log plyx (x:;0)
In practical applications, LLMs are often fine-tuned on multiple diverse datasets D = {Di) N where each dataset D {(=]. vi) contains M, examples. this multi-datased setting, the naive approach combines the losses across all datasets:
N ((D:0) Σ (D(:)
(2)
3

---

Algorithm 1: Hierarchical Balancing Optimization
Input :D ([[(<]])) y Rm [
3 M4 ) training set organized into N subsets, where each subset contains M. groups, and each group holds Q., pairs consisting of instruction 2 and their response " Fegnhal and "akeal, update frequency and learning for gebeal Fleaal and *Koeds update frequency and learning rate for Wiveat; T, total training steps; a. learning rate 0:
Output: The converged model 0:
Initialize pe (NV) and Pennall (M.) using Equation 3 with 7 1
for = to I do
i gea (N);
j Pui (M;);
Sample batch (x. y) Di.;; 0 - 0 - a VoL zz 0); i %6 F Palabai
0 then for i to N do
xx' v ~ Dr:
Compute reward (1) for D' as Section 3.3
11
end
12
gllballl <- glotall + Ei=1 Talobul R Rehbba()). V Maliia log Pe) purea (t)
13
end
14
i % Fiocol == 0 then for i 1 to N do
16
for = to M; (x')8') Di-::
Compute reward Rical (i, for Disd as in Section 3.3 end
18
Vuuall Plucal EM,, Mocal Riocal (1, V shoo log Pennal (1,5);
end
22
end
end
Static Balancing underrepresent ones. Static balancing addresses this issue by adjusting each dataset's sampling probability with a temperature parameter (Arivazhagan et al. 2019 Conneau et al.) 2020). The base sampling probability of the i-th dataset is q(i) Em M.
When fine-tuning on multiple datasets of varying sizes, simply merging them
then adjusted using temperature 7 as:
(+) q(1)1/T
33)
ΣN (n))/T
The temperature parameter 7 provides flexible control over dataset representation: 7 = yields proportional sampling based dataset sizes (equivalent to the naive approach in Eqqution 2). while as approaches co, sampling becomes increasingly uniform across datasets regardless of their original sizes. With this temperature-adjustedd sampling, the training objective becomes:
L(D;0, Ei-g, [D::
3.2 HIERARCHICAL BALANCING OPTIMIZATION
We introduce Hierarchical Balancing Optimization (HBO), hierarchical dynamic data sampling framework designed to address both global (across datasets) and local (within datasets) heterogeneity. Our approach leverages bilevel optimization jointly optimize the LLM parameters 0 and two types of actors: the Global Actor glgiobal and the Local Actor 1/ 2] As shown in Figure 1. the L.M and the training datasets D form the environment, while the actors &b globall and Vlicaal act agents that dynamically adjust sampling probabilities. The global actor glolalll balances the contributions of
2The a/ represents the collection of the local actors for each individual dataset and the ahicall indicates the local actor for the i-th dataset.
4

---

different datasets, ensuring fair representation across datasets of varying sizes. Meanwhile, the local actors Vlocal adjusts sampling probabilities within each dataset, accounting for internal heterogeneity.
Our framework is formulated as bilevel optimization problem (Colson et al.) 2007), where the solution to the inner problem constrains the outer problem. In our case, the outer optimization minimizes the objective OD; which evaluates the LLM's performance The inner optimization adjusts the sampling probabilities (N) and Pei..! (M,) based on the LLM's performance. This results in the following bilevel optimization formulation:
the training datasets.
global, Vlocal) = arg min J (D; O (V) theca))), where
(5)
0 globbt: (hocal) arg min Pe (N)
boal [By-pen (M
(M))L(((>>)) yy") 0:0]
This hierarchical optimization framework ensures that both global and local sampling probabilities are dynamically adjusted to maximize the L.L''s performance, enabling more effective utilization of heterogeneous training data.
We present the detailed implementation of HBO in Algorithm 1 We organize the training data into hierarchical structure with N subsets, where each subset contains M; groups. The algorithm begins by initializing both global and local sampling distributions with temperature = 1. During each training iteration, we first sample a subset according to the global distribution Poraao (N), then select group based the local distribution Pvi.. (M;). After updating the model parameters 9 using the sampled batch, we periodically update both actors. The global actor giaabal is updated every Felobal steps by evaluating rewards Rylohal (i) for each subset, while the local actors v/tocal are updated every Flocat steps by computing rewards Riscal (i, for each group. These updates follow a policy gradient approach, where the gradient of the log probability is scaled by the corresponding reward. This optimization strategy ensures that sampling probabilities at both hierarchical levels are continuously refined to maximize model performance.
A critical issue in Algorithm iss that Equation 5 is not directly differentiable with respect to the actors giobal and Plocal- We address this using the Reinforce algorithm (Williams) 1992). a policy gradient method that enables gradient-base based optimization of non-differentiable rewards. In HBO, we compute rewards based on the model's 's performance on sampled training data, then update the policy parameters to maximize expected rewards. The updates for ggghhal and Th take the unified form:
w + + R V log (.).)
where &p represents either vb gglotall or Vhocal, Y is the learning rate (Yglotbul or "focat). R is the computed reward (Rehohua (i) or Riccal (t. ). and p (-) is the probability of selecting particular group subset.
Implementation Details In HBO, both the global actor Ligibbal and local actors &lucal are imple- mented as 2-layer fully connected network. Each actor takes input feature vector representing the corresponding sampling unit (subset or group). This lightweight architecture is sufficient because the actors only need model relatively simple distributions over the hierarchical training data structure. It is important to note that these actors are used exclusively for optimizing data sampling during the fine-tuning process, incurring about 15% additional training overhead in total training runtime compared with static balanced sampling. They are entirely separate from any reward models typically employed in Reinforcement I. Learning with Human Feedback (RIHFF). Furthermore, we employ the SuperFiltering metric (Li et al., 2024a) to evenly divide each subset into four groups based on task difficulty. Group contains the easiest examples, while Group contains the hardest examples.
3.3 TALE OF TWo REWARDS
In HBO, we introduce two distinct reward functions to guide the optimization of the global and local actors. The global reward Riloba((1) evaluates the performance of the LLM on a specific subset i, while the local reward Ricat (i,j) assesses the performance
specific group j within subset i. These rewards are designed to capture different aspects of model performance and are crucial for effective sampling in heterogeneous datasets.
Global Reward Recent work demonstrates that the L2 norm of the gradients decreases as the model gradually learns (Chen et al. 2018), suggesting that the Lo2 norm of the gradients is ideal
5

---

signal of learning dynamics of the LLM across various datasets. Based this insight, given random
batch B; = {(x' y')} uniformly sampled from subset i, we define our global reward Rglobal (t) as the L2 norm of the gradients computed on subset
Rglobal (t) [B,
This reward mechanism prioritizes datasets with larger gradient norms, effectively allocating more training resources subsets where the model has more to leamn As the model becomes more proficient on particular subset, the corresponding gradient norm naturally decreases, causing the global actor global to gradually shift focus to other subsets where improvement still needed.
Local Reward The local heterogeneity typically stems from the varying difficulty of examples within each subset. To address this, we design local reward mechanism that effectively prioritizes examples based on their learning progress. Inspired by Wu al. (2024a), define the local reward Riocal (i,j) as the ratio between the current perplexity and the initial perplexity of the model on examples from group j within subset i. This ratio serves as an indicator of learning progress, where a higher value suggests the model has made less progress on these examples relative to its starting point. Specifically, given random batch B,, = { y } sampled from group j of subset i, Raoeal (i,j) is defined as
1
PVL Fk. 0)
(1,))
where
K
PPL y Tx. 0o)
(8)
(y1: exp
log Dk,<1;
yk
Here, 0 denotes the current model parameters, 9, represents the initial model parameters, |yk the length of the response yu. and K is the batch size. For examples where the model improves quickly, Reocal (i, decreases, reducing their sampling probability. Conversely, examples where improvement is lower maintain higher rewards.
4 EXPERIMENT
4.1
EXPERIMENTAL SETUP
Multitask Setup We construct our training mixture from four distinct domains: general, math, financial, and medical. The General dataset is collected from the English part of the WildChat (Zhao et al. 2024) and LMSYS-Chat-11M (Zheng et al., 2024), resulting in total of 1,196K examples. The Math dataset is derived from the Met MetaMathQA Yu al. 2024), containing 393K examples. The Finance dataset is obtained from the AQ22
3l with 256K examples. Finally, the Medical dataset is 2024), comprising 409K examples. Accordingly, we conduct zero-shot evaluations on the following testsets: MMLU (Hendrycks et al. 222 for general, GSM8K (Cobbe al., (02T)) for math, MultiFin-EN (Jørgensen et al.32023) for finance, and MedMCQA (Pal et al. 2022)) for medical. Due to computational constraints, we sample 10% of total examples from each training dataset. Task performance is evaluated using the accuracy metric given by (Gao)
sourced from the UltraMedical (Zhang
et al. 2024). and the overall performance is reported as the macro-average across all tasks (μMM))
Multilingual Setup
We fine-tune multilingual LLMs using a combination of Aya Dataset (Singh al. 2024) and Wilchhat (Zhao al., 2024), covering eight languages: English (273K), Arabic
(12K), German (6K), Spanish (15K), Hindi (IK), Russian (49K), Swahili (578), and Chinese (102K).
Zero-shot evaluations are conducted on diverse downstream tasks, including MMMLU (Hendrycks et al.
2021a), XCOPA (Ponti et al. (020). XStoryCloze (Lin et al. 22221). XNLI (Conneau et al. 2018). and MGSM (Shi et al. (022). To address computational constraints, we sample 20% of the examples from each training dataset. Task performance is measured as macro-average accuracy across languages
(Gao et al. 2024). The overall performance reported as the macro-average across all tasks (JMM)).
WW omit the superseript j in the notation for brevity.
https::/huggiinffacce httt:///wgggnnnte......aaaaaaaas htt.a//attt/eettaaaaaaaaaaaaaaaa

---

Table I: Main results given by EuroLLM-9B, (aama-3.1-BB and Qwen2.5-7B on the multilingual and multitask settings. XSC and are the XStoryCloze and MultiFin--B datasets. The best results and second-best results are highlighted. f indicates statistical significance against the best baseline at p < 0.05 terms of PMI. and μMT. following (Koehn 2004).
Multilingual
Multitask
JMI
MMMLU
MGSM XCOPA
XSC
XNLI
JAMT
MMLU
Fin GSM8K MedMCQA
M.r9LLM--9
Heuristic
Prop.
48.50
45.29
21.27
65.00
66.32
44.59
49.10
55.13
53.48
46.70
41.09
Temp.
47.32
43.38
19.07
65.40
66.09
42.67
48.43
55.36
53.11
43.14
42.12
Uni
47.49
42.64
21.13
65.20
66.25
42.23
48.07
55.38
53.30
41.09
42.53
Dynamic
MultiDDS
47.34
41.76
21.87
63.90
66.81
42.36
44.02
47.66
52.38
39.42
36.62
MMltiUAT
47.35
36.47
23.73
65.60
67.08
43.85
47.20
49.86
52.95
44.73
41.26
MoS
47.79
44.05
19.40
65.90
66.73
42.86
48.37
53.29
53.66
44.20
42.31
MoS+
48.03
44.22
20.47
64.70
66.45
44.32
49.04
53.98
54.13
47.95
40.11
HBO
49.37
45.58
24.07
65.90
66.82
44.48
50.16'
55.56
53.83
48.36
42.89
Llama-3.1-8B
Heuristic
-Prop.
46.25
40.78
18.00
62.70
64.63
45.11
46.74
51.84
51.10
43.14
40.88
Temp.
44.78
41.97
10.33
63.50
63.95
44.16
47.68
54.34
52.93
42,91
40.55
Uni
44.38
40.18
11.27
63.90
64.39
42.17
46.75
52.98
52.24
42.23
39.54
Dynamic
MultiDDS
46.86
41.80
18.00
64.90
65.10
44.49
47.94
54.15
52.97
48.22
36.43
MultiUAT
46.80
41.31
19.73
62.90
65.48
44.55
50.51
55.41
51.21
54.28
41.12
MoS
46.44
42.60
17.87
64.30
65.58
41.86
49.41
52.96
50.55
56.77
37.37
-MoS+
46.94
41.25
17.73
64.10
64.96
46.64
50.94
55.55
52.69
53.07
42.43
HBO
48.07
44.28
20.40
63.00
65.98
46.67
52.28
56.87
52.56
56.94
42.74
Qwen2.5-7B
Heuristic
Prop.
53.50
53.20
41.60
64.90
64.57
43.22
58.13
60.49
59.52
67.10
45.40
Temp.
54.20
51.10
46.20
65.50
64.85
43.32
54.97
62.48
49.63
62.47
45.28
Uni.
53.90
50.46
42.93
66.30
64.92
44.90
57.42
61.52
53.66
68.31
46.19
Dynamic
MultiDDS
53.82
51.84
41.80
66.70
65.42
43.36
59.27
64.81
59.16
64.90
48.22
MMltiUAA
53.79
53.13
40.40
66.40
65.10
43.91
58.90
63.66
57.33
67.70
46.93
.MMS
53.99
49.80
44.87
66.10
65.72
43.48
58.84
64.79
52.75
70.28
47.55
MMS++
54.26
51.53
44.80
66.20
65.42
43.36
58.11
63.72
53.37
70.25
45.10
HBO
55.21
53.74
48.07
66.80
64.63
42.83
60.371
65.25
59.93
70.35
45.94
Model Backbones and Baselines We evaluate HBO on: Qwen2.5-7B (Yang et al. 2024a), Llama-
3.B (Dubey et al. 2024), and EuroLLM-9B (Martins et al., 2024). For baselines, we compare against: Heuristic Methods: Proportional sampling (Prop., 1), temperature sampling (Temp.,
7 = 10), and uniform sampling (Uni., T 20), based on [Equuttion 3 Dynamic Methods: MoS (Wu) al. (2024a), MultiDDS (Wang al. 2020b), and MultiUAT (Wu 2021), which adjust dataset distributions globally.
4.2 MAIN RESULTS
In this section, we present the performance HBO compared to heuristic and dynamic methods across three L.MSs multilingual and multitask setups, shown in Table 1, We demonstrate that HBO not only significantly outperforms a number of established baselines but also effectively balances the data allocation during the fine-tuning process.
HBO consistently outperforms all the baselines in both multilingual and multitask setups. Results Table demonstrate that HBO consistently outperforms both heuristic and dynamic baselines across various models and evaluation settings. In the multilingual evaluation (PML.), HBO
7

---

Prebak 111r
Math
Ficania
Brow
(a) ML-Global
(b) MT-Global
(c) English-Local
(d) Math-Local
Figure 2: The variation sampling probabilities given by (a) global actor in the multilingual setup, (b) global actor in the multitask setup, (c) local actor in the English subset, and (d) local actor in the Math subset. The model backbone is Llama-3.14 1--8B.
Table 2: Ablation study of global actor and local actors given by Llama-3. 1-8m--3.1-8B in the multilingual setting. // indicates the actor is used, while X indicates the actor is ablated. The values in red and green indicate the gap with the results given by the full HBO.
1-88
1/ globul
Loocal
PML
MMMLU
MGSM
XCOPA
XStoryCloze
XNLI
/
48.07
44.28
20.40
63.00
65.98
46.67
47.22 L0:85
43.15
11.13
19.47 10.93
63.30 10.30
65. 04 10.94
45.14
U1.53
47.46
L0.61
43.43
10.85
19.13 11.27
63.10 10.10
65.22 J0.76
46.44
L0.23
x
46.25
L1.82
40.78
13.50
18.00 12:40
62.70 10.30
64.63 11.35
45.11 $1.56
surpasses the best baselines by margins of +0.87, +1.13, and +0.95 on Eur9LLM-9B, Llama-3. and Qwen2.5-7B, respectively. Similarly, in the multitask setting (UMT), HBO delivers superior performance, with gains of +1.06, +1.34, and +1.10 over the best baselines. Beyond average scores, HBO achieves even larger performance gains on specific tasks. For example, HBO outperforms competing baselines by substantial margins ranging from +1.68 to +4.10 in MMMLU task with the Llama-3. 1-8B backbone. These results show HBO global-local balancing mechanism crucial for consistently achieving optimal performance across diverse tasks and languages.
HBO can effectively balance the data allocation globally and locally. We visualize the evolution of sampling probabilities for global actor and local actor in FFiuure 2] At the global level, global actor adaptively balances emphasis between languages or tasks, gradually shifting focus from high-resource (e.g., English General tasks) less-represented languages and tasks throughout training (see Figure 2(a) and Figure 2(b)). This dynamic rebalancing prevents overfitting to large datasets while ensuring comprehensive capability development across all languages or tasks. At the local level, local actor reveals a cyclical pattern in sampling distributions by example difficulty. As illustrated in Figure 2(c) the English subset cycles approximately every 800 steps, where harder examples (groups 3 and 4) and easier examples (groups and 2) alternately receive increased sampling probability. This emergent curriculum learning behavior shows HBO's ability to autonomously adapt for effective training dynamics. The complementary global-local optimization enables HBO to simultaneously address dataset imbalance and heterogeneity at multiple levels, resulting in superior performance.
ANALYSIS
In this section, we evaluate HBO in a multilingual setting, examining actor ablation, different difficulty groupings, various sampling priors, and training data sizes. Further analyses on model sizes, reward functions and updating frequency are provided in Appendix B
Both global actor and local actors effectively improve performance. We conduct an ablation study to assess the individual contributions of global actor and local actors. The results in Table demonstrate that ablating local actors reduces JMM by 0.85 and XNLI I by 1.53, while removing global
8

---

Table Effect of different sampling priors (7) on performance using L 1-BB backbone across multilingual benchmarks. The best results are high- lighted in bold.
0.2
1.82
Data Percentage
0.4
1.08
0.69
2.18
AML MMMLU MGSM XCOPA XSC XNLI
1.64
Prop.
46.25
40.78 18.00
62.70 64.63 45.11
1.75
0.8
0.25
MoS
+T 46.44
42.60
17.87
64.30
65.58 41,86
0.87
MOS HBO
+T 10 47.06
42.80
18.00
64.90
65.10 44.49
0.09
46.40 41.31
17.73
62.90
65.48 44.55
HBO
Figure 3: Improvements of HBO and MoS compared to the Prop. with Llama-3.1-8B backbone the MMMLU testset.
+T 48.07 44.28
20.40
63.00 65.98 46.67
+T = 10 47.98 44.30
20.80
63.60
65.92 45.30
+ = Do 48.06 44.64 19.73 63.30 66.05 46.59
actor lowers JML by 0.61 and MGSM by 1.27. Removing both actors causes the largest drop (1.82 on PML and 3.50 MMMLU), confirming that each type of actor offers distinct advantages and that their combination optimally balances global and local distributions during training.
The optimal number of difficulty levels balances granularity and effective learning. We examine the impact of difficulty level granularity on HBO by partitioning each dataset into 1, 2, 4, 8, and 16 groups. As shown in Table 3 four-level grouping consistently yields the best overall performance, achieving JML score of 48.07. The pattern suggests that moderate granularity provides levels fail to capture meaningful distinctions between examples, while excessive divisions may fragment the learning signal and increase the complexity.
Table 3: Difficulty level granularity with Llama-3.1-8B. Best results are in bold.
#Groups MML MMMLU MGSM XCOPA XSC XNLI
1
47.22
43.15 19.47 63.30 65.04 45.14
optimal balance, too few
2
47.09
44.09 18.07 63.60 65.52
44.16 44.166
48.07
44,28 20.40 63.00 65.98 46.67
47.45
44.16 18.53 63.80 65.06 45.68
16
47.87
43.92 20.33 64.00 65.60 45.48
HBO shows robust improvements across sampling priors. Dynamic methods often sensitive to the choice of prior sampling distribution (Wu et al. [202]). We evaluate HBO at various temperature settings in Table and find that HBO consistently outperforms the best heuristic baseline Prop. across all tasks, with gains of to + +1.22 in PML- While the optimal prior may vary by task, HBO remains stable and shows significantly less variance than MoS, demonstrating its robustness against changes in the prior sampling distribution and consistent performance gains across diverse evaluation scenarios.
HBO consistently outperforms the baselines even in resturce-constrained environments. previously only 20% of the available training data test method under constrained conditions, and now we increase data usage to 40%, 60%, 80%, and 100%. As shown in Fig ure 3 HBO consistently achieves larger performance gains compared to MoS in every setting. Notably, when using only 20% of the train- ing data, HBO delivers performance boost of 3.50 over Prop..
Table 5: Performance comparison when progressively discarding the casiest examples from the training dataset with 1.ama-3.1-8B The "pet." is the percent- age of easiest examples discarded from training set.
Easy examples matter for model perfor- mance. To investigate the importance of easy examples in training, we progressively discard increasing percentages of the easi- est training examples while maintaining the total training compute (e.g., doubling the number of training steps when 50% of exam- ples were discarded). Our findings in Table 5 show consistent performance degradation as more easy examples are discarded. Removing 25% of the easiest examples causes noticeable drop
pet.
MMMLU
MGSM
XCOPA
XSC
XNLI
Prop.
0%
46.25
40.78
18.00
62.70
64.63
45.11
0%
48.07
44.28
20.40
63.00
65.98
46.67
25%
47.41
43.08
21.00
62.30
65.94
44.75
HBO
50%
47.08
41.32
22.33
62.60
65.72
43.45
75%
46.55
41.09
18.80
66.10
63.20
43.58

---

in average performance (-0.66 (ML), particularly XNLI (-1.92). As removal reaches 75%, performance nears that of the bascline. Although easy examples are often considered less informative (Xu et al. 2024h, they prove crucial by diversifying the training mixture.
CONCLUSION
In this paper, we present Hierarchical Balancing Optimization (HBO), novel hierarchical dynamic data sampling method designed to tackle the critical challenges data imbalance and heterogeneity in fine-tuning LLMs. Leveraging bilevel optimization framework with Global Actor and several Local Actors, HBO enables LLMs to autonomously adjust data sampling both across datasets (globally) and within datasets (locally) based their current training state. Through extensive experiments across three LLMs and nine tasks in multilingual and multitask setups, we demonstrate the effectiveness of HBO, achieving significant performance improvements. By autonomously adapting LLMs' learning strategies, HBO represents significant ad udvancment in addressing the complexities of dataset mixture balancing, contributing more effective fine-tuning of LLMs.
ETHICS STATEMENT
This work introduces HBO, method for fine-tuning large language models (LLMs) using hierarchical dynamic data sampling. All experiments are conducted on publicly available datasets and open-source model backbones, strictly adhering their respective licenses and terms of use. No human subjects or private data are involved. While HBO aims to improve fairesss and generalization by addressing data imbalance and heterogeneity, we acknowledge that biases present in the underlying datasets or models may persist. We encourage responsible use of HBO, with attention to fairness, transparency, and accountability in downstream applications.
REPRODUCIBILITY STATEMENT
We are committed to ensuring the reproducibility of our findings. Detailed descriptions of the HBO methodology, including the bilevel optimization framework, actor architectures, reward functions, and training procedures, are provided in [Sectionn 3 Experimental setups, including model backbones, dataset statistics, sampling strategies, and evaluation metrics, are thoroughly described in [Section 4 and [pppendix A All datasets and models used are publicly available, with references and links included. To further support reproducibility, we will release our code and scripts for data preparation and experiments upon publication, enabling other researchers to replicate and build upon our results.
THE USE OF LARGE LANGUAGE MODELS (LLMS)
In preparing this work, we utilize large language models (LLMs) as general-purpose tools to assist with writing polish and grammar correction. The LLMs are not involved in research ideation, experimental design, or substantive content generation. Their role limited to improving the elarity and readability of the text, ensuring grammatical accuracy, and refining the presentation of our findings. All scientific contributions, analyses, and conclusions are solely the work of the authors.
ACKNOWLEDGEMEN
This project has received funding from UK Research and Innovation (UKRI) under the UK govern- ment's Horizon Europe funding guarantee [grant numbers 10039436).
The computations described this research were performed using the Baskerville Tier 2 HPC funded by the EPSRC and UKRI through
service httss///ww..asseervii .ckuk//. Baskerville the World Class Labs scheme (EP/TO22221/I) and the Digital Research Infrastructure programme (FP/W032244/1) and is operated by Advanced Research Computing the University of Birmingham.
10

---

REFERENCES
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. CoRR, abs/2312.024066 2023. doi: 1044550/ARXIV2312.02400
URL httpss//d// org/10. 40550//rrXi.2211..0000
Anthropic. Claude 3.5
sonnet.
2024.
URL https://ww.aarttrpppcc.co///////
claude- 3- 3-5-een
Naveen Arivazhagan, Ankur Bapna, Orhan Firat, Dmitry Lepikhin, Mclvin Johnson, Maxim Krikun, Mia Xu Chen, Yuan Cao, George F. Foster, Colin Cherry, Wolfgang Macherey, Zhifeng Chen, and Yonghui Wu. Massively multilingual neural machine translation in the wild: Findings and challenges. CoRR, abs/1907.05015 2019.
05019
Viraat Aryabumi, John Dang, Dwarak Talupuru, Saurabh Dash, David Cairuz, Hangyu Lin, Bharat Venkitesh, Madeline Smith, Jon Ander Campos, Yi Chern Tan, Kelly Marchisio, Max Bartolo, Sebastian Ruder, Acyr Locatelli, Julia Kreutzer, Nick Frosst, Aidan N. Gomez, Phil Blunsom, Marzich Fadace, Ahmet Üstün, and Sara Hooker. Aya 23: Open weight releases to further multilingual progress. CoRR, abs/2405. 15032, 2024, doi: 10.48550/RXIV.2405..10 https://doi.org// 0001//00..555///// 18550/arXiv 2405. 15032
URL
Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Andrey Kutuzov, Barry Haddow, and Kenneth Hcafield. Monolingual or multilingual instruction tuning: Which makes a better alpaca. In Yvette Graham and Matthew Purver (eds.), Findings of the Association for Computational Linguisties: EACL22024 St. Julian's 's Malta, March 17-22, 2024, pp. 1347-1356. Association for Computational org/ 2024. findings eacl.
Linguistics, 2024. URL URL httpsss////caaattoooogg
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normalization for adaptive loss balancing in deep multitask networks. In International conference on machine learning, pp. 794 803. PMLR, 2018
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang. Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. Scaling instruction-finctuned language models. /etppppae///23--2000p22 J. Mach. Learn. Res., 25:70:1-70:55 2024a. URL http::tts/////// j/mmr org/ papers/ 2ttp//////222222/22 html
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. Scaling instruction-finete language models. Journal of Machine Learning Rexearch, 25(70):1-53, 2024b.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers solve math word problems. arXiv preprint 1r.iv:2110.14166 2021.
Benoît Colson, Patrice Marcotte, and Gilles Savard. An overview of bilevel optimization. Ann. Oper. Res., 153(1):235-22) 2566 2007. doi: 10.10077/10479-00700777 URL htt.:tttttsssoooooet 10. 1100///1449--00-001166
Alexis Conneau, Ruty Rinott, Guillaume Lample, Adina Williams, Samuel R. Bowman, Holger Schwenk, and Veselin Stoyanov. Xnli: Evaluating cross-lingual sentence representations. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, 2018.
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Fran- cisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault (eds.). Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 8451, Online, July 2020. Association for Computational Linguistics. doi: 10. 1.118653/11/2220..em 747. URL URR.httpss//aallanttoooogg
ael-main. 747

---

Michael Crawshaw. Multi-task learning with deep neural networks: A survey. arXiv preprint arXiv:2009.09796, 2020.
DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng. Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang. Han Bao, Hanwei Xu, Haocheng Wang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Qu, Hui Li, Jianzhong Guo, Jiashi Li, Jiawei Wang. Jingchang Chen, Jingyang Yuan, Junjie Qiu, Junlong Li, J. L. Cai, Jiaqi Ni, Jian Liang, Jin Chen, Kai Dong, Kai Hu, Kaige Gao, Kang Guan, Kexin Huang. Kuai Yu, Lean Wang. Lecong Zhang. Liang Zhao, Litong Wang. Liyue Zhang, Lei Xu, Leyi Xia, Mingchuan Zhang, Minghua Zhang, Minghui Tang, Meng Li, Miaojun Wang. Mingming Li. Ning Tian, Panpan Huang, Peng Zhang, Qiancheng Wang, Qinyu Chen, Qiushi Du, Ruiqi Ge, Ruisong Zhang, Ruizhe Pan, Runji Wang, R. J. Chen, R. I.. Jin, Ruyi Chen, Shanghao Lu, Shangyan Zhou, Shanhuang Chen, Shengfeng Ye, Shiyu Wang, Shuiping Yu, Shunfeng Zhou, Shuting Pan, and S. S. Li. Deepseek-rl: Incentivizing reasoning capability in Ilms via reinforcement learning. CoRR, abs/2501. J 12948, 2025. doi: 10.4850/ARXXV.2501. URL httpt:t//diiooorr org/ 10. 48550/ att/v. 2501. 12948.
Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang. Zheng Yuan, Chang Zhou, and Jingren Zhou, How abilities in large language models are affected by supervised fine-tuning data composition. In Lun-Wei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), ACL 2024, Bangkok, Thailand, August 11-16. 2024, pp. 177-198. Association for Computational Linguistics, 2024. doi: 1211666NN/2024AAALLLLLO 12. URL https://doi org/10/ /00 18653/v1/2022/ acl-long. 12
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang. Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurélien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozière, Bethany Biron, Binh Tang. Bobbie Chern, Charlotte Caucheteux, Chaya Nayak, Chloe Bi, Chris Marra, Chris McConnell, Christian Keller, Christophe Touret, Chunyang Wu, Corinne Wong, Cristian Canton Ferrer, Cyrus Nikolaidis, Damien Allonsius, Daniel Song, Danielle Pintz, Danny Livshits, David Esiobu, Dhruv Choudhary, Dhruv Mahajan, Diego Garcia-Olano, Diego Perino, Dieuwke Hupkes, Egor Lakomkin, Ehab AlBadawy, Elina Lobanova, Emily Dinan, Eric Michael Smith, Filip Radenovic, Frank Zhang, Gabriel Synnaeve, Gabrielle Lee, Georgia Lewis Anderson, Graeme Nail, Grégoire Mialon, Guan Pang, Guillem Cucurell, Hailey Nguyen, Hannah Korevaar, Hu Xu, Hugo Touvron, Iliyan Zarov, Imanol Arrieta Ibarra, Isabel M. Kloumann, Ishan Misra, Ivan Evtimov, Jade Copet, Jaewon Lee, Jan Geffert, Jana Vranes, Jason Park, Jay Mahadeokar, Jeet Shah, Jelmer van der Linde, Jennifer Billock, Jenny Hong, Jenya Lee. Jeremy Fu, Jianfeng Chi, Jianyu Huang, Jiawen Liu, Jie Wang, Jiecao Yu, Joanna Bitton, Joe Spisak, Jongsoo Park, Joseph Rocca, Joshua Johnstun, Joshua Saxe, Junteng Jia, Kalyan Vasuden Alwala, Kartikeya Upasani, Kate Plawiak, Ke Li, Kenneth Heafield Kevin Stone, and et al. The llama 3 herd of models. CORR, abs/2407 22.224477.27788 21733, 2024. doi: URL httph:tt///dlt org/ 10. 48550/arxiv 2407. 21783
Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot or wwtg////co//////// 12608602
language model evaluation, 07 2024. URL StLLhttts://eeee
Haibo He and Edwardo A Garcia. Learning from imbalanced data. IEEE Transactions on knowledge and data engineering, 21(9): 1264-1284 2009.
Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding. In 9th International Conference on Learning Representattons, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021a. WtRL:hhtps:///opot wttpee/wwwwweeeeeeooor///ddd///a
12

---

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the MATH dataset. In Joaquin Vanschoren and Sai-Kit Yeung (eds.). Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks 1, NewrlPS Datasets and Benchmarks 2021, Decem- ber 2021, virtual, 2021b. URL http::htttt:ttttttttttht hewpre/seeeeeeeeenneeeeeeeeeeeee neureiinnnnneeei........m.. www.irs.rcs..... cc/ html
Srinivasan Iyer, Xi Victoria Lin, Ramakanth Pasumuru, Todor Mihaylov, Daniel Simig, Ping Yu, Kurt Shuster, Tianlu Wang, Qing Liu, Punit Singh Koura, al. Opt-iml: Scaling language model instruction meta learning through the lens of generalization. arXiv preprint arXiv:22 22. 12017, 2022.
Rasmus Jørgensen, Oliver Brandt, Mareike Hartmann, Xiang Dai, Christian Igel, and Desmond Elliott. Multifin: A dataset for multilingual financial nlp. In Findings of the Association for Computational Linguistics: EACI 2023, pp. 894-909, 2023.
Firuz Kamalov and Dmitry Denisov. Gamma distribution-based sampling for imbalanced data. Knowl. Based Syst. 107:10 2020. doi: (OO06/J..NOSYS.2000 106368. URL https
//doi.org/// 10. 1016 wtt.///..//.........oe///ooooo.. 2020. 106368
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7482-7491, 2018.
Tannon Kew, Florian Schottmann, and Rico Sennrich. Turning english-centric Ilms into polyglots: How much multilinguality is needed? In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, Miami, Florida, USA, November 12-16, 2024, pp. 13097-13124, Association for Computational Linguistics, 2024. grg/ 2824. findings-emi emn1p. 766
URL https::/aclaanthoooor
Muhammad Khalifa, Yi Chern Tan, Arash Ahmadian, Tom Hosking, Honglak Lee, Lu Wang, Ahmet Üstün, Tom Sherborne, and Matthias Gallé. If you can't use them, recycle them: Optimizing merging at scale mitigates performance tradeoffs. CoRR, 0tbs/2412.04144,, 2024. doi: 10.48550/ http::::/oo..org////// 411.0.4855//88 (88850/aaxiv..202 241.. 04144
ARX1V.2412.04144,
Philipp Koehn. Statistical significance tests for machine translation evaluation. In Dekang Lin and Dekai Wu (eds.), Proceedings of the 2004 Conference Empirical Methods in Natural Language Processing, pp. 388-395, Barcelona, Spain, July 2004. Association for Computational Linguistics.
UBt_ 3250/
URL https::/aclantholllle wwwp:/oooooooo//---130
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b- parameter open-access multilingual language model. 2023.
Ming Li, Yong Zhang, Shwai He, Zhitao Li, Hongyu Zhao, Jianzong Wang, Ning Cheng, and Tianyi Zhou. Superfilterigg Weak-to-stie took-nn-sttong data filtering for fast instruction-tunie -uning. In Lun-Wat Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers). pp. 14255-14273, Bangkok, Thailand, August 2024a. Association for Computational Linguisties. doi: 10.18653/v1/202.. acl-long. 769. URL
w0gg/222...c// 2ttgg/222..cllloog..77... 769/
Ming Li, Yong Zhang, Zhitao Li, Jiuhai Chen, Lichang Chen, Ning Cheng, Jianzong Wang. Tianyi Zhou, and Jing Xiao. From quantity to quality: Boosting LLM performance with self-guided data selection for instruction tuning. In Kevin Duh, Helena Gómez-Adorno, and Steven Bethard (eds.), Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies ( (Volume 1: Long Papers), NAACL 2024, Mexico City, Mexico, June 16-21, 2024, pp. 7602-7635. Association for Computational Linguistics, 2024b. doi: 10.
URL https: org/10
18653/v1 / /2024. naacl-lone 421
Xize Liang, Lin Yang, Jie Wang, Yiyang Lu, Runyu Wu, Hanzhu Chen, and Jianye Hao. Boosting multi-domain fine-tuning of large language models through evolving interactions between samples. In Forty-second International Conference on Machine Learning, 2025.
13

---

0Tt.:///CDDEE330222200000000 Xi Victoria Lin, Todor Mihaylov, Mikel Artetxe, Tianlu Wang. Shuohui Chen, Daniel Simig, Myle Ott, Naman Goyal, Shruti Bhosale, Jingfei Du, Ramakanth Pasumuru, Sam Shleifer, Punit Singh Koura, Vishrav Chaudhary, Brian O' O'Horo, Jeff Wang. Luke Zettlemoyer, Zornitsa Kozareva, Mona T. Diab, Veselin Stoyanov, and Xian Li. Few-shot learning with multilingual language models. CoRR, abs/2112 12. 10668, 2021. URL httphttss:://///
10668
Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. What makes good data for alignment? A comprehensive study of automatic data selection in instruction tuning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL htnttps:///ppereeieeess net hett//orrummi--BTAALLLL
Zhining Liu, Wei Cao, Zhifeng Gao, Jiang Bian, Hechang Chen, Yi Chang, and Tie-Yan Liu. Self- paced ensemble for highly imbalanced massive data classification. In 36th IEEE International Conference on Data Engineering, ICDE 2020, Dallas, TX, USA, April 20-24, 2020, pp. 841- 852. IEEE, 2020. doi: ICDE48307 2020. 00078
URL https:://oi..... http///ii..rg/1/11110/0 1109/
Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno Miguel Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, M. Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. Eurollm: Multilingual language models for europe. CoRR, abs/2409. 16235, 2024. doi: 10.48550/ARXIV. 48550/arXiv wtt.://........a........ 2409. 16235
2409.16235. URL https:////ooi org/10
Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang. Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. Crosslingual generalization through multitask finetuning. In Anna Rogers, Jordan L. Boyd- Graderr, and Naoaki Okazaki (eds.), Proceedings of the 6l Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pp. 15991-1611 11. Association for Computational LinguaatiionlLgLniiiiitiitti Linguistics, URL
2023. doi: 10. acl-long. 891
18613/112235333
OpenAI. Introducing gpt-4.1 in the api, 2024a. Utthhhttpp///ppppa...co/////////
OpenAl. Hello gpt-4o. 2024b. URL.ttps:///geaa..cmm// http///////////eeeloottt----
OpenAl. Learning to reason with 1lms. 2024c.
URL https://oppeai. cotpineexccom//iex////
Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu. Medmeqa: A large-scale multi-subject multi-choice dataset for medical domain question answering. In Gerardo Flores, George H. Chen, Tom J. Pollard, Joyce C. Ho, and Tristan Naumann (eds.), Conference on Health, Inference, and Learning, CHIL 2022, 7-8 April 2022, Virtual Event, volume 174 of Proceedings of Machine Learning Research, pp. 248-24422202 260. PMLR, 2022. /tLL/htps:///p/t pttpss////.ooceeigg....m mlr press/v1744pal122. html
Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Ko- rhonen. Xcopa: A multilingual dataset for causal commonsense reasoning. arXiv preprint NXiv:2005500333. 2020.
Samira Pouyanfar, Yudong Tao, Anup Mohan, Haiman Tian, Ahmed S. Kaseb, Kent Gauen, Ryan Dailey, Sarah Aghajanzadeh, Yung-Hsiang Lu, Shu-Ching Chen, and Mei-Ling Shyu. Dy- namic sampling in convolutional neural networks for imbalanced data classification. In IEEE 1st Conference on Multimedia Information Processing and Retrieval, MIPR 2018, Miami, FL, USA, April 10-12. 2018, pp. 112-117. IEEE, 2018 doi: 10. 10.1009//PP.200 201..2011.0000.22 URL https://dot: http://l....eecommmmttrrrrooeee. wrw.//r.g11oo. 1T09/MIPR...... 2018. 60027
Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry Lepikhin, Timothy P. Lillicrap, Jean- Baptiste Alayrac. Radu Sorieut, Angeliki Lazaridou. Orhan Firat, Julian Schrittwieser, loannis Antonoglou, Rohan Anil. Sebastian Borgeaud, Andrew M. Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu,
14

---

James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross Mcllroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thortoon, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, and et al. Gemini 1.5: Unlocking multimodal understanding across millions of tokens doi: 10445550ARRXV.22433.05
of context. CoRR, abs/2403.055 055300, 2024.
URL https:
7/doi. wrg/r0.g1110 /0/ 48550/ Xiv. 2403. 05530
Roy Schwartz and Gabriel Stanovsky. On the limitations of dataset balancing: The lost battle against spurious correlations. In Marine Carpuat, Marie-Cathering de Marnelfe, and Iván Vladimir Meza Ruíz (eds.), Findings of the Association for Computational Linguistics: NAACL 2022, Seattle, WA, United States, July 10-15, 2022, pp. 2182-2194, Association for Computational Linguistics, 2022. doi:
URL https:////do 18633/111661333222222
findings- 168
Uri Shaham, Jonathan Herzig, Roee Aharoni, Idan Szpektor, Reut Tsarfaty, and Matan Eyal. Multi- lingual instruction tuning with just a pinch of multilinguality. arXiv preprint arXiv:2401.0185% 2024.
Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung. Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. Language models are multilingual chain-of-though, reasoners, 2022.
Shivalika Singh, Freddie Vargus, Daniel D'souza, Börje Karlsson, Abinaya Mahendiran, Wei- Yin Ko, Herumb Shandilya, Jay Patel, Deividas Mataciunas, Laura O'Mahony, Mike Zhang, Ramith Het- tiarachchi, Joseph Wilson, Marina Machado, Luisa Souza Moura, Dominik Krzeminski, Hakimeh Fadaei, Irem Ergün. Ifcoma Okoh, Aisha Alaagib, Oshan Mudannayake, Zaid Alyafeai, Minh Vu Chien, Sebastian Ruder, Surya Guthikonda, Emad A. Alghamdi, Sebastian Gehrmann, Niklas Muennighoff, Max Bartolo, Julia Kreutzer, Ahmet Ustün, Marzieh Fadaee, and Sara Hooker. Aya dataset: An open-access collection for multilingual instruction tuning. In Lei Ku, Andre Martins, and Vivek Srikumar (eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pp. 11521-11567. Association for Computational Linguistics, 2024. doi: 10.18653) V1/2024.ACL-LONG..00 URL h orgo 10. 00.865331122225 //////0224..22 acl-llcllln 620
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An cnatucttio-folloinng Ilama model, 2023.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Julictte Love, et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295 2024.
Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. Gemma 3 technical report. arXiv preprint arXiv:2503. 19786, 2025.
Xinyi Wang. Hieu Pham, Paul Michel, Antonios Anastasopoulos, Jaime Carbonell, and Graham Neubig. Optimizing data usage via differentiable rewards. In International Conference on Machine Learning, pp. 9983-9995. PMLR, 2020a.
Xinyi Wang, Yulia Tsvetkov, and Graham Neubig. Balancing training for multilingual neural machine translation. arXiv preprint mmXiv:2004.06745 2020b.
Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In 2019 JEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoud. Korea (South), October 27 November 2019, pp. 5016-5025. IEEE, 2019. doi: 10. (09/CC..22099.000///
http:: //ooi. 11g/ 1109/ICCV 025122
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improv- ing multi-task optimization in massively multilingual models. arXiv preprint a0x0022010.05874. 2020c.
15

---

Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint 200.0:2009.0162,, 2021.
Ronald J. Williams. Simple statistical gralient-following algorithms for connectionist reinforcement learning. Mach. Learn., 8:229-256, 256, 1992. doi: 10.1007/BF00999666 URL htttps://oii.orr org/ 10. 807/8F88992266
Minghao Wu, Yitong Li, Meng Zhang, Liangyou Li, Gholamreza Haffari, and Qun Liu. Uncertainty- aware balancing for multilingual and multi-domain neural machine translation training. arXiv preprint
2021.
Minghao Wu, Thuy-Trang Vu, Lizhen Qu, and Reza Haf. Mixture-of-kkill:: Learning to optimize data usage for fine-tuning large language models. Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.). Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, EMNLP 2024, Miami, FI., USA, November 12-16, 2024, pp. 14226- 117226-12240 14240. Association for Computational Linguistics, 2024a. URL. htt.https//// httttp::/aallanthologgg.g org 2ww.2.mm2......... emnlp-mainn ma-main
787
Minghao Wu, Thuy-Trang Vu, Lizhen Qu, and Gholamreza Haffari. The best of both worlds: Bridging quality and diversity in data selection with bipartite graph. CoRR, abs/2410. 12458, 2024b. doi: 10.48550/ARXIV.241000 12458. URL httpt:h/tp/ott org/ 48550/arXiv 2410. 12458
Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, Qingwei Lin, and Daxin Jiang. Wizardlm: Empowering large pre-trained language models to follow complex instructions. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:://pperrevie..r net/ Forum??d=CFXh993
An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang. Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. CoRR, abs/2412. 15115, 2024a. doi: 10.48550/ARXIV.44111 15115, httpsttss: //doi...:///d.... orgoorgg///// /00 48550/ 2412. 15115
Jingfeng Yang, Hongye Jin, Ruixiang Tang, Xiaotian Han, Qizhang Feng. Haoming Jiang. Shaochen Zhong, Bing Yin, and Xia Ben Hu. Harnessing the power of Ilms in practice: A survey on chatgpt and beyond. ACM Trans. Knowl. Discov. Data, 18(6):160:1-1600) 2024b. doi: 10.1 10.1145/3649506. URL https://doi org/10 100 1145/3644 3649506
Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu. Metamath: Bootstrap your own mathematical questions for large language models. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024. OpenReview.net, 2024. URL https://openreviie wwt.:enttttoooomiinnnanneneeeee
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. Advances in Neural Information Processing Systems, 33: 5824-5836, 2020.
Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Jinfang Hu, Zhiyuan Liu, and Bowen Zhou. Ultramedical: Building specialized generalists in biomedicine. CoRR, abs/2406.033 03949, 2024. doi: htrg//0..kk 410...855//rra/ ar.50//rxxiv..m 2406 03949
URL httpss:///// orwoorg////// htt
Zhihan Zhang, Wenhao Yu, Mengxia Yu, Zhichun Guo, and Meng Jiang. A survey of multi-task learning in natural language processing: Regarding task relatedness and training methods. arXiv preprint arXiv:2204. 2Xi5:2204.03508, 2022.
16

---

Wenting Zhao, Xiang Ren, Jack Hessel, Claire Cardie, Yejin Choi, and Yuntian Deng. Wildchat: Im chatgpt interaction logs in the wild. In The Tweifth International Conference on Learning Representations, ICLR 2024, Vienna. Austria, May 7-11, 2024. OpenReview.net, 2024. URL https:://openreview. het/forrum1ddBB1BB7RRRD0
Yingxiu Zhao, Bowen Yu, Binyuan Hui, Haiyang Yu, Fei Huang, Yongbin Li, and Nevin L. Zhang. preliminarry study of the intrinsic relationship between complexity and alignment. CoRR. abs/2308.05666, 2023, doi: 1044850/ARXX1.23000056666 URL https://dooo htgg10..48550 arXiv. 0..388.05999
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Tianle Li, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zhuohan Li, Zi Lin, Eric P. Xing, Joseph E. Gonzalez, Ion Stoica, and Hao Zhang. Lmsys-chat- A large-scale real-world LLM conversation dataset. In The Twelfth Interna- fional Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.
OpenReview.net, 2024. URLhhtsss//ppperreiiee...ee http:///eeeeeeeeeeet///////oo/ta
Tong Zhu, Daize Dong, Xiaoye Qu, Jiacheng Ruan, Wenliang Chen, and Yu Cheng. Dynamic data mixing maximizes instruction tuning for of CoRR, abs/2406. 1256, 2024. doi: 1256. URL http::/t::///d org/ 48550/arXins 2tt.:/......2...112...
17

---

A TRAINING DETAILS
We fine-tune all parameters of LLMs using the AdamW optimizer with a learning rate of 1 x 10-S and a batch size of 16. This process is conducted over three epochs 8 NVIDIA A A00 GPUs (80GB). During training, we use a linear learning rate schedule with a warm-up phase that constitutes 10% of the total training steps. For HBO, 1 gloohal and thrncal are updated for every 200 steps with the learning rate of 1 10-4 and the batch size of 64. gglobal and Local are initialized by
B ADDITIONAL ANALYSIS
Performance gains scale with param- eter count. As shown in TTble 6 we observe that HBO consistently outper- forms heuristic methods across all model sizes. Notably, the performance advan- tage scales with the model's parame- ter count, suggesting that larger mod- els are better able to leverage the opti- mized sampling strategies. For average performance (JMI.). HBO demonstrates significant improvement over Prop.. achieving a +1.83 gain with Llama-3.1- 8B, compared to smaller gains of +1.25 with the 3B model and +0.40 with the IB model. While the extent of these im- provements may vary depending the specific tasks, the consistent advantage demonstrated by HBO underscores the robustness of this approach. These re- sults indicate that our balancing method effective across different model sizes, with larger models benefiting more sig- nificantly. This is likely because larger models often possess a greater capacity to leverage advanced optimization techniques.
Table 6: Comparisons between HBO and heuristic meth- ods with various model sizes in the multilingual setup.
PML
MMMLU
MGSM
XCOPA
XSC
XNLI
I.lma-3.2-1B
Prop.
37.60
27.43
3.53
58.30
58.00
40.73
Temp.
37.59
27.60
3.67
58.40
57.58
40.70
Uni.
37.62
27.68
3.20
58.60
57.51
41.13
HBO
38.00
28.27
3.40
58.60
58.22
41.53
Llama-3,2-3B
Prop.
41.90
37.26
7.47
60.40
61.45
42.91
Temp.
42.21
37.21
7.67
61.50
62.13
42.55
Uni.
42.32
38,26
8.20
60.80
61.79
42.57
HBO
43.15
38.23
10.93
61.30
62.08
43.18
Llama-3.1-8B
Prop.
46.24
40.78
18.00
62.70
64.63
45.11
Temp.
44.78
41.97
10.33
63.50
63.95
44.16
Uni. 44.38
40.18
11.27
63.90
64.39
42.17
HBO
48.07
44.28
20.40
63.00
65.98
46.67
HBO is compatible with various re- ward functions. The choice of reward function significantly affect the model performance, so we investigate the im- pact of reward functions and present the results in Table 7 The L2 norm and PPL Ratio the default global and local ward function as defined in Equation
Table 7: Various combinations of reward functions in HBO using Llama-3.1-88 under multilinguao setup.
Rgiobal
Riocal
PML.
MMMLU
MGSM XCOPA
XSC
XNLI
norm
PPL Ratio
48.07
44.28
20.40
63.00
65.98
46.67
L2 norm
PPL
46.94
43.18
17.81
63.30
65.08
45.31
and Eqqution 8 respectively. Following
L2 norm
Loss
47.22
44.11
18.17
62.70
64.77 46.34
CosSim PPL Ratio
47.69
43.36
19.33
64.00
65.33
46.44
Wu et al. (2024a), we introduce CosSim as additional global reward and PPL and Loss as additional local rewards. The CosSim is defined as the cosine similar- ity between the hidden states from two batches, the PPL and Loss are defined in Equation 8 and
CosSim PPI.
47.05
42.89
18.50
62.80
65.08
45.97
CosSim Loss
47.15
43.45
18.27
63. 64.63
46.32
Equation respectively. We observe that all the combinations of reward function achieve perfor- mance gains compared to the best heuristic baseline Prop. (46.24) in PMI., and the combination of I norm and PPL Ratio achieves the best performance among all these combination. These findings validate the effectiveness of our design choice.
The updating frequency of global actor and local actor should be carefully determined. We conduct experiment investigate the impact of the updating frequency the runtime and model performance, and present the results in Figure 4 As shown in Figure 4(a)) we observe that global actor and local actor consistently improve the model performance across all the updating frequency
18

---

1.8
500
1. 19
08
1. 18
07
0. 96
500
1.09
1. 07
1. 05
1. 03
1. 03
1.
300
1. 08
1. 67
1. 63
1. 43
Local Frequency
300
12
06
1. 09
1. 07
1. 06
200
7.
1.
200
1. 14
1.2
100
36
34
1. 63
1. 48
1. 34
100
1.
1.
2.
1. 21
1.
1.1
50
1.
29
19
4.
0.
1. 1.3
50
1.46
1.
1.
1. 3.
1. 39
1000
200
200
500
Global
Frequency
Global Frequency
(a) Absolute Performance Gains
(b) Relative Runtime Overhead
Figure 4: (a) The absolute performance gains HBO compared Prop. with different settings of updating frequency for global actor and local actor. (b) The relative runtime overhead introduced by HBO compared to Prop. with different settings of updating frequency for global actor and local actor.
settings and achieve the best performance when setting the updating frequency of both global actor and local actor 200. Furthermore, more frequent updating results more computational overhead. Figure 4(b) demonstrates that frequency of 200 for both global actor and local actor provides the best balance between performance gains and computational efficiency.
C ADDITIONAL BASELINES COMPARISON
There are several additional basclines that we compare against to further validate the effectiveness of HBO. As the code of these works is not open-sourced, re-implement the baselines DMT (Dong et al. 2024) and EVIC (Liang et al. 2025) and conduct additional experiments using Llama-3.1 backbone in multilingual setting. As shown in Table 8 we observe that HBO outperforms both DMT and EVIC across all the benchmarks.
Table 8: Comparison with additional baselines using Llama-3.1 131-8 backbone in multilingual setting.
Methods
PML.
MMMLU
MGSM
XCOPA
XNLI XStoryCloze
DMT
45.43
40.26
17.73
62.10
64.85
42.23
EVIC
46.10
41.76
19.73
62.70
63.95
42.36
HBO
48.07
44.28
20.40
63.00
65.98
46.67
D METRICS TO DIVIDE DATASETS
we use SuperFiltering metric to evenly divide each subset into four groups based on task difficulty. This metric essentially is Instruction Following Difficulty (IFD) score (L) al. 2024b) but computed using a smaller language model, which is Qwen2.5-0.5B in this work. SuperFiltering demonstrates that small language model is equally effective in computing IFD scores for data selection compared to using a large language model. IFD score is a pure statistical metric, that compares the losses or perplexities when the model generates response Y4 with and without instructional context 2;, measuring how much help the instruction provides to the generation of the corresponding response. For given instruction-followii
In Section 3.2
data pair, the IFD score is calculated as: IFD(y, (z.) = PPL(0: =x) where PPL(y. (x:) and PPL(y.) denote the perplexities the given model in PPL[x.)
fitting response y; with and without the instruction 2;, respectively. A higher IFD score, indicating less instructional help, suggests greater diffieulty. On the contrary, the low IFD score represents that
19

---

the given instruction directly benefit the language model largely without further training, representing the easiness and necessity of the instruction. For instance, if z; contains strong clues or direct answers, the model can easily generate y; with 24, leading low IFD score.
Regarding robustness, we conduct additional experiments with using PPL and loss as metrics. The results with 1Jama-3.1-8b backhone in the multilingual setup shown in Table 9 We observe that both of HBO-PPL and HBO-Loss achieve the performance gains compared to the best heuristic baseline Prop. It demonstrates that long as the metric provides meaningful signal of difficulty, the local actors will learn to balance them effectively.
Table 9: Comparison with different metrics for dataset division.
Metrics
μML.
MMMLU
MGSM
XCOPA
XStoryCloze
XNLI
Prop.
46.25
40.78
18.00
62.70
64.63
45.11
Suo--upeFFittriin
48.07
44.28
20.40
63.00
65.98
46.67
HBO-PPL
47.42
44.71
19.33
62.60
64.65
45.81
HBO-Loss
47.23
44.88
19.04
62.50
64.39
45.34
20


Reviews:
[{"review_id": "j9SZt2qVOP", "reviewer": "Reviewer_dqCa", "summary": "This paper introduces Hierarchical Balancing Optimization for fine-tuning large language models (LLMs) on diverse datasets by addressing both global and local data imbalance and heterogeneity. HBO uses a bilevel optimization framework with two types of actors: a Global Actor, which adjusts data allocation across different datasets, and Local Actors, which optimize data usage within individual datasets based on difficulty levels. Extensive experiments show that HBO consistently outperforms existing methods, offering significant improvements in model performance. The proposed method provides a comprehensive solution to the challenges of data imbalance and heterogeneity during fine-tuning.", "questions": "See weaknesses.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) HBO effectively addresses both global and local data imbalances, providing a more comprehensive solution to the challenges of fine-tuning LLMs on diverse datasets.\n\n2) The bilevel optimization framework with Global and Local Actors allows for fine-grained control over data sampling, leading to improved model performance across various tasks.\n\n3) Extensive experiments demonstrate HBO's strong applicability across multiple LLM backbones and tasks, consistently outperforming existing baselines and offering superior accuracy gains.", "weaknesses": "1) My main concern is the proposed method adds more computations based on MoS. The reinforcement learning framework, as well as some reward, is similar to the MoS method. This work adds more actors and the grad norm reward, more insights of this field could be added.\n\n2) This paper primarily compares three sampling balancing methods: MoS, MultiUAT, and MultiDDS. However, many of the results are similar to uniform sampling. What is the next step of this field could be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OkcjTJQNwE", "reviewer": "Reviewer_9MNG", "summary": "This paper introduces Hierarchical Balancing Optimization (HBO), a novel framework designed to address data imbalance and heterogeneity during the fine-tuning of Large Language Models (LLMs). The authors argue that existing methods typically focus on balancing data across different datasets (globally) but neglect imbalances and variations within individual datasets (locally). HBO aims to solve this by enabling the LLM to autonomously adjust data sampling probabilities at both levels. It employs a bilevel optimization strategy featuring a Global Actor, responsible for balancing sampling across datasets, and multiple Local Actors, each optimizing sampling within a specific dataset based on predefined difficulty groups. These actors are trained using reward signals derived from the LLM's training state: the global reward is based on the L2 norm of gradients (indicating learning progress), and the local reward uses the ratio of current to initial perplexity (indicating relative improvement on difficulty groups). The framework is evaluated on several LLM backbones across multilingual and multitask settings, reportedly outperforming various baseline sampling strategies.", "questions": "1. Could the authors provide more specifics about the SuperFiltering metric  used to partition data into difficulty groups? How is difficulty defined and measured by this metric, and how robust is the local balancing performance if a different difficulty metric were used?\n\n2. The cyclical patterns observed in the local sampling probabilities (e.g., Figure 2c) are interesting. Is this behavior consistently observed across different datasets and models? What mechanism within the HBO framework drives this apparent emergent curriculum?\n\n3. Please review the formatting of Equation 8; the \"where\" clause defining PPL seems awkwardly placed on the same line and might be clearer if moved below or reformatted.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper tackles a significant and nuanced challenge in LLM fine-tuning by explicitly addressing hierarchical data imbalance and heterogeneity (both global, across datasets, and local, within datasets), which is often overlooked by simpler methods.\n\n2. The proposed HBO mechanism, utilizing a bilevel optimization framework with distinct global and local actors guided by rewards derived from the model's own training state, is a novel and sophisticated approach to achieve autonomous, dynamic data balancing.\n\n3. The empirical results are strong and comprehensive, showing consistent improvements over heuristic and existing dynamic sampling baselines across multiple LLM backbones (Llama, Qwen, EuroLLM), diverse tasks (multilingual, multitask), and including insightful analyses like the dynamic evolution of sampling probabilities.", "weaknesses": "1. The framework introduces substantial complexity compared to standard fine-tuning or simpler dynamic sampling. Implementing and tuning the bilevel optimization setup, managing multiple actors (one global, potentially many local), and ensuring stable training with the Reinforce algorithm likely requires significant expertise and effort.\n\n\n2. The reported computational overhead, while quantified (~15%), is non-negligible and could be a barrier to practical adoption. This additional runtime cost for actor updates and reward computations might be prohibitive, especially for resource-intensive LLMs or scenarios requiring frequent retraining.\n\n\n3. The method's performance relies heavily on the specific choices for reward functions and the difficulty grouping metric. While the chosen rewards (L2 gradient norm, perplexity ratio) and the SuperFiltering metric  show good results in the experiments, their universal optimality or robustness across different datasets, tasks, or model architectures is not guaranteed. The framework might be sensitive to these design choices.\n\n4. The paper would be improved by comparing this RL-based actor approach to other recent dynamic strategies for multi-domain data, including those based on sample interactions[1] or alternative optimization objectives [2]\n\n[1] Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples. ICML 2025.\n\n[2] Mixture-of-Skills: Learning to Optimize Data Usage for Fine-Tuning Large Language Models. EMNLP 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T8U4n3x9T6", "reviewer": "Reviewer_9Qbd", "summary": "This paper presents Hierarchical Balancing Optimization (HBO), a bilevel optimization framework designed to mitigate data imbalance and heterogeneity when fine-tuning LLMs. HBO introduces two interacting components, i.e., a Global Actor that adjusts sampling probabilities across datasets and several Local Actors that manage sampling within datasets based on difficulty levels. Both actors are guided by rewards derived from the LLM's training state: gradient norms for global balancing and perplexity ratios for local balancing. The authors evaluate HBO on three model backbones (Llama-3.1-8B, Qwen2.5-7B, EuroLLM-9B) and nine tasks across multilingual and multitask setups, reporting consistent performance improvements over both heuristic and dynamic baselines.", "questions": "Please see Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper targets an important problem, i.e., data imbalance and heterogeneity in LLM fine-tuning,which is relevant to current multi-task and multilingual training paradigms.\n2. The hierarchical bilevel optimization formulation is conceptually interesting and provides a unified framework for global and local data balancing.\n3. The paper is well-written and easy to follow.", "weaknesses": "I have the following concerns. *If the authors could properly address them during the rebuttal phase, I am willing to raise my score.*\n1. The technical novelty is somewhat limited. While the hierarchical structure and bilevel setup are well-motivated, they mainly combine known techniques such as policy gradients and dynamic sampling into a straightforward framework, without introducing fundamentally new optimization principles.\n2. This paper lacks strong theoretical or analytical justification for why the proposed reward formulations (gradient norm and perplexity ratio) should lead to optimal or stable data allocation.\n3. Although the experiments are comprehensive, the reported improvements are modest (often within 1–2 points), raising questions about whether the additional implementation overhead (15% extra training cost) is justified.\n4. Some important recent baselines [1,2] are missing. The paper would benefit from either a comparison with these methods or a discussion of their relevance.\n5. The authors may want to provide deeper analysis of failure cases or scenarios where HBO underperforms, as well as qualitative insights into how the hierarchical balancing actually affects learning dynamics beyond probability curves.\n\n[1] How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition. ACL 2024.\n\n[2] Boosting Multi-Domain Fine-Tuning of Large Language Models through Evolving Interactions between Samples. ICML 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AVcPjZbwoi", "reviewer": "Reviewer_9Boj", "summary": "This paper introduces Hierarchical Balancing Optimization (HBO), a novel framework designed to address data imbalance and heterogeneity challenges in fine-tuning LLMs. HBO employs a bilevel optimization strategy with two key components: a Global Actor for balancing data sampling across datasets and Local Actors for optimizing sampling within individual datasets based on difficulty levels. The paper demonstrates HBO’s effectiveness across three LLM architectures on multilingual and multitask setups, showing consistent performance improvements over existing baselines.", "questions": "Have the authors considered task-specific reward functions or alternative optimization techniques for the global and local actors? If so, how do these compare to the current design?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The hierarchical optimization framework with global and local balancing is novel and addresses a critical gap in LLM fine-tuning.\nThe experiments are extensive and well-designed, covering multiple models, tasks, and evaluation metrics. The ablation studies and sensitivity analyses provide deep insights into the effectiveness of HBO.\nThe paper is clearly written, with detailed explanations of the methodology and results. Figures and tables are well-designed and enhance understanding.\nThe proposed method is broadly applicable and demonstrates consistent performance gains, making it a valuable contribution to the field.", "weaknesses": "While HBO demonstrates strong performance improvements, the additional 15% training overhead may limit its applicability in resource-constrained environments. The authors could provide more discussion on strategies to mitigate this overhead.\n\nAlthough the paper evaluates different reward functions, the focus is primarily on L2 norm and perplexity ratio. Exploring alternative reward mechanisms (e.g., task-specific metrics) could further enhance HBO’s applicability.\n\nThe experiments are conducted on sampled datasets (e.g., 20% of total data). It would be helpful to discuss how HBO scales to larger datasets and whether the performance gains remain consistent.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)