DUET: Optimizing LLM Training Data Mixtures via Noisy Feedback from Unseen, Downstream Evaluation Tasks
Authors: Zhiliang Chen
Gregory Kang Ruey Lau
Chuan-Sheng Foo
Bryan Kian Hsiang Low
Venue: ICLR 2026 Poster (2026)
Keywords: Bayesian Optimization
Data Mixture Optimization
Optimization from feedback
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
TL;DR: We interleave data selection and Bayesian Optimization to optimize data mixture based on coarse and noisy feedback from unseen evaluation tasks, which existing methods cannot.

Abstract: The performance of an LLM depends heavily on how well the training data matches the downstream evaluation task. However, in many practical settings, we typically do not know the data in the evaluation task (e.g., conversations between a chatbot and users are end-to-end encrypted). We refer to such tasks as unseen evaluation tasks. We can only deploy the LLM on these unseen evaluation tasks to gather multiple rounds of feedback on how well the model performs (e.g., gathering user ratings from a chatbot). In addition, this feedback can be noisy. How can we exploit such noisy feedback efficiently to optimize the LLM training data-mixture? Our paper presents DUET, a novel global-to-local algorithm that optimizes training data mixtures by interleaving data selection with Bayesian optimization to exploit coarse and noisy feedback from a downstream evaluation task. DUET is flexible enough to incorporate different data selection methods, each with different performance-compute tradeoffs. By analyzing DUET's cumulative regret, we theoretically show that DUET converges to the optimal training data mixture even without any fine-grained data information from an unseen task. Finally, our experiments across a variety of language tasks demonstrate that DUET attains substantial performance improvements over existing data selection and mixing methods in the unseen-task setting. Our library, which is flexible enough to optimize different LLM training ingredients, can be found at  https://github.com/chenzhiliang94/BO-for-LLM.

Published as a conference paper at ICLR 2026
DUET: OPTIMIZING LLM TRAINING DATA MIXTURES VIA NOISY FEEDBACK FROM UNSEEN EVALUATION TASKS
Zhiliang Chen Gregory Kang Ruey Lau 1,3,* Chuan-Sheng Foo" Bryan Kian Hsiang Low
1 Department of Computer Science, National University of Singapore
2 Agency for Research, Science, Technology and Research (A (STR)), Singapore
& *CNRS@ @ CREATE, Create Way, #08-01 Create Tower, Singapore 138602
ichenzhiliang, gregorylau @u. w.w.nun.ecll
lowkhê comp. nus edu.sq
ABSTRACT
The performance of an LLM depends heavily on how well the training data matches the downstream evaluation task. However, in many practical settings, we typically do not know the data in the evaluation task (e.g., conversations between a chatbot and users are end-to-end encrypted). We refer to such tasks as unseen evaluation tasks. We can only deploy the LLM on these unseen evaluation tasks to gather multiple rounds of feedback how well the model performs (e.g., gathering user ratings from a chatbot). In addition, this feedback can be noisy. How can we exploit such noisy feedback efficiently to optimize the LLM training data-mixture" Our paper presents DUET, a novel global-to-loca, algorithm that optimizes training data mixtures by interleaving data selection with Bayesian optimization exploit coarse and noisy feedback from downstream evaluation task. DUET is flexible enough incorporate different data selection methods, each with different performance- compute tradeoffs. By analyzing DUET's s cumulative regret, we theoretically show that DUET converges to the optimal training data mixture even without any fine- grained data information from an unseen task. Finally, our experiments across a variety of language tasks demonstrate that DUET attains substantial performance improvements over existing data selection and mixing methods in the unseen-task setting. Our library, which is flexible enough to optimize different LLM training ingredients, can be found at https: //githugt com. http:/hhhzzhiiingg... 30- for-LLMS
INTRODUCTION
The performance of an LLM depends heavily on how well the training data domain (Chen et al. 2024a: Xie al. 2023a) matches the downstream evaluation task (Hoffmann et al. 2022 Long et al. 2017). For instance, we knew that LLM users are interested in asking layman science questions, then training fine-tuning the LLM with more Wikipedia data allows it converse better with these users. Hence, knowing the evaluation task is important for curating a more relevant training data mixture, producing an LLM with better performance over the specific task of interest.
Unfortunately, in practice, the data (e.g., its domain, distribution, or labels) involved in an unseen evaluation task are often unknown. Thus, it is not obvious what data is relevant for training or fine-tuning the model. Consider the following problem setting: An LLM owner is interested in fine-tuning their L.M to converse better with users but due to privacy concerns (Li et al.) 2024). conversations between the deployed LLM and users are end-to-end encrypted (opena i com/ enterprise privacy). Hence, the LLM owner does not know the actual evaluation data seen during test-time. Rather, they only receive coarse, noisy feedback on how well the LLM has performed in the conversation (e.g., user ratings or duration spent on the application). Of course, a naive idea is to simply iterate through all possible data mixtures and observe the resulting LLM performance. better question is to ask: How can we exploit the noisy feedback loop efficiently to improve and optimize the training data mixture?

---

Published as a conference paper at ICLR 2026
Wiki
Data selection select data points (Sec. 3.2)
Unknown conversation content.
Med
Train/fine- tune LLM
Math
LLM
LLM
Feedback from unseen evaluation task.
Trivia
Optimized data mixture and LLM for the unszen
News
After Y foodback rounds.
Proposed data mixing ratio r
Bayesian Optimization on mixing ratio (Sec. 3.3)
Figure I: DUET exploits a feedback loop to optimize the data mixture for an unseen evaluation task. In contrast, conventional data mixing and selection works require fine-grained data information of the task, which is not available here.
This paper presents DUET (Fig.[) an efficient algorithm that exploits noisy feedback loop to optimize the training Data mixture for an Unseen Evaluation Task. DUET is a global-to-local algorithm that interleaves data selection (Albalak et al., 2024 [ing & Brochu 2017 Koh & Liang 2017) with Bayesian optimization (BO) (Snoek et al. 2012, Srinivas et al. 20T0) to optimize the training data mixture. Globally, BO in DUET uses coarse, noisy feedback from the unseen evaluation task to automatically refine the mixing ratio of data domains the training data mixture iteratively. Locally, DUET uses data selection retrieve high-quality data points from each data domain until the proposed mixing ratio reached. This results in an algorithm that can optimize training data iteratively even without having access fine fine-nrained data information from the evaluation task.
Related works. In our problem setting, (a) there no direct access to the data (c.g.. its domain, distribution, or labels) involved in the unseen evaluation task but (b) we gather multiple rounds of feedback (details covered in Sec. 2.2 from the task using an LLM. App- A......pppp provides few more practical examples of this setting. This setting different from those considered in conventional domain adaptation (DA) and domain generalization (DG) works. Prior DA works assume fine-grained knowledge of data (e.g., labeled/unlabeled data (Zhang et al., 2022) or data distribution (Ganin & Lempitsky 2015 Zhang et al. (2021)) from the evaluation task for selecting relevant training data that match the evaluation data. On the other hand, DG considers rigid setting with no knowledge
(not even feedback) of the evaluation task (Muandet et al. 2013 Shin et al. 2024
Wang et al. 2022).
Similarly, data mixing works such as DoReMi (Xie et 2023a). BiMix (Ge et al.
2025) and more
(Chen et al. 2024a; Fan et al. (2044) Xie et al. [025]] 2023b) introduced methods to optimize data mixtures, and data selection works (Albalak et al. [024 Xia al. 2024 Pruthi et al. 2020 Xie et al.) 2023b) explored ways to find high-quality data to improve an LLM's performance. However, these methods assume some availability of fine-grained evaluation data information, such as evaluation gradients, labels, distribution naively assuming the training data shares the same distribution the task. In practice (like in our setting), these are not always available. In fact, when we applied existing data mixing and selection methods directly our setting, they perform worse than DUET (Sec. (). We provide more discussion of the shortfalls of these prior works in App. A.2
To the best of our knowledge, DUET is the first work that interleaves data selection with BO to iteratively optimize training data mixture based on feedback from an unseen evaluation task. At first glance, eliciting multiple rounds of feedback with BO seems expensive. However, BO sample- efficient and is the only way we can exploit such coarse and noisy feedback iteratively, unlike prior methods that require much more tnn-grained data information (see above). In fact. subjecting models to multiple rounds of training or fine-tuning in feedback loop is natural part of the deployment life-cycle to improve L.Mss. Specifically, our contributions are:
We introduce a novel and realistic problem setting where the data involved in unseen evaluation task is unknown but we can deploy our LLM to gather multiple rounds of coarse and noisy feedback. Then, we introduce DUET, novel algorithm that exploits the feedback loop optimize training Data mixture for the Unseen Evaluation Task. To achieve this, DUET interleaves data selection (Sec. 3.2) with Bayesian optimization (Sec. (.3) to iteratively optimize the training data mixture. DUET is flexible enough to incorporate any data selection choice in its inner loop, and qualitatively and quantitatively analyzed different choices in our paper.
2

---

Published as a conference paper at ICLR 2026
We provide theoretical analysis of DUET's convergence to the optimal training data mixture by analyzing DUET's attained cumulative regret (Chen et al. 2022bb] Chowdhury Gopalan 2017) under the BO framework (Sec. 4
We demonstrate the effectivvness of DUET on LLM fine-tuning for language tasks comprising both in-domain and out-of-domain unseen tasks spanning different domains. Compared to conventional data selection and mixing methods (e.g.. DoReMi, LESS, Aioli (Chen et al. 2024a) and more), DUET produces more optimal training data mixtures (Sec. 6.2))
2
PRELIMINARIES
2.1
BAYESIAN OPTIMIZATION
We first provide outline of how BO can be used to optimize generic black-box objective function before explaining how BO is used in DUET (Sec. (.3). We consider a black- box objective function R". As we show later (Sec. 2.2 we will use the data mixing ratio in setting. The goal is to find r" - arg min, (r) which minimizes the objective function. BO a query-efficient active algorithm that strategically selects input points to query the black-box objective function, conditioned on previous function observations. At each iteration t= 2, T of BO, we query the black-box function with a selected input Tr to obtain a noisy observation ÿt - f(re) + with a Gaub-Gasssian noise e (e.g.. Gaussian or bounded noise) to form sample (re, ). Consistent with (Chowdhury Gopalan 2017), we model the unknown function as realization of a Gaussian process (GP) (Williams & Rasmussen 2006) that is fully specified by its prior mean p(r) and covariance <(r, r') for all r, E R" where is is a kernel function chosen to characterize the correlation of the observations between any two inputs r and r': a common choice is the squared exponential (SE) kernel k(r,r' å exp(-IIr - r' 221m2mmm²) (2m²)) with length-scale hyperparameter vn that can be learned via maximum likelihood estimation. Given a column vector y: _ [U+]T=I,.... of noisy observations at previous inputs ri,.... rt, the posterior belief of f at any new input r' is a Gaussian distribution with the following posterior mean and variance:
f R R over the space of inputs
pe(r') & KT (r')(K+ + ' y
oe(r') A
where (e(rr) A [s(rr [s(r",rr)]] TT=),_ is column vector, K, A [k(rr; rrrrrrr is matrix, and C > is viewed free hyperparameter that depends on the problem setting (Chowdhury & Gopalan 2017). Using equation | the BO algorithm selects the next input query T++11 by optimizing an acquisition function, such minimizing the lower confidence bound (LCB) acquisition function (Srinivas et al. 2010): r-+1 = arg min, (r) - 32+10r(r) with an exploration parameter Bt+1- In addition, BO can also handle constraints on inputs r (Gardner et al. (014). The cumulative regret (for T BO iterations w.r.t. a minimization problem) Rr _ El= (rr) - f(r")] is used to assess the performance of BO algorithm (Tay et al. 2023) given that f(r") is the true function minimum. lower cumulative regret indicates a faster convergence rate. We provide a theoretical analysis of DUET's cumulative regret in Sec. 4
covariance
2.2 PROBLEM SETTING: OPTIMIZING DATA MIXTURES FOR AN UNSEEN TASK
Now, we formally describe our problem setting. Suppose that we have n training datasets D A (D,, D Dr) from different domains (e.g., Wikipedia, ArXiv), where D is the union of these training datasets. Let (f) ) be the unseen evaluation task loss w.r.t. an LLM parameterized by This "loss" represents feedback from the unseen evaluation task and does not have a closed, mathematical form. Our goal is to find an optimal data mixture X" E D (a set of training data points) and learn model parameters 0x-- such that the unseen evaluation task loss Loval is minimized:
min NED
(Ox)
X M,
where 0x - arg miny Crain (X, 8) is the model parameters learned in standard supervised learning manner (e.g., gradient descent) from chosen data mixture X' and Lunin is standard model training
3

---

Published as a conference paper at ICLR 2026
loss (e.g., cross-entropy loss for LLM prediction). To make theoretical formulation and expository simpler, we consider the feedback Leval deterministic. However, DUET works equally well for in noisy feedback setting, which demonstrate empirically (Sec. 6 and elaborate in App. A.3 M is practical, pre-decided constraint (Mirzasoleiman et al. 2020) to ensure the selected data mixture is not too large. In practice, evaluation task loss Leval is just feedhack that indicates how well the LLM is performing and does not contain any evaluation data information. It can also be interchanged with other measures be maximized (e.g., accuracy, user ratings) with slight mathematical adjustment to later statements.
OPTIMIZING TRAINING DATA MIXTURES USING DUET
Unfortunately, solving problem is challenging because the unseen evalu- ation task loss Leval does not have a closed, mathematical form and finding the optimal data mixture X directly is a high high-dimensional discrete op- timization problem. To address this, DUET adopts a global-to-local approach optimize the training data mixture. Globally, DUET uses BO to adjust the mixing ratio in the data mixture adaptively based on the task feedback. Locally, we interleave data selection method of choice (depending on the practitioner's compute budget) to refine the data mixture every iteration. Fig. illlsttrats, in simple setting, how DUET progressively finds better data mixtures close to the optimal (green star). We also discuss several extensions of DUET in App. A.3
Tuthfu104
X
x
x
TIMaOA
gonds
Figure 2: DUET finds the optimal data mixture iter- atively and strategically.
3.1 REPARAMETERIZATION OF THE OPTIMIZATION PROBLEM
We first reparameterize the objective function problem 2 into a bilevel optimization problem that, at the outer level, depends the mixing ratio E TR= of training data domains (such reparameterizatioo has been considered in AutoML works (Chen et al. 2024b)). This reparameterized problem has unique structure that aligns with DUET's global-to-local nature (Sec. 3.2 3.3).
Theorem 3.1. X", the optimal set of data points from DD is the solution of the original problem 2 iff
r" = ratio(x*) ) is the optimal mixing ratio solution of the reparameterized problem:
min min PER" XES,
(0v))(0x))
(3)
X E D, =r, M) and ratio means that the data points in where S, (X: 1X:Xe
X satisfies the mixing ratio r E R from n data domains and ||]| = 1.
The proof can be found in App. B.I where we show that X', the optimal data mixture of original problem 2 satisfies mixing ratio that is also the solution of reparameterized problem DUET aims to solve problem 3 in an iterative manner. At the outer optimization level (global), DUET uses BO exploit feedback from the evaluation task propose a promising mixing ratio r, each iteration l. At the inner optimization level (local), we introduce sampling strategy that uses local domain data selection retrieve high-quality data subset that satisfies mixing ratio Tz.
3.2 USING DATA SELECTION METHODS FOR INNER PROBLEM
In this section, show how data selection methods can be used to solve the inner problem in DUET. For ease of expository and illustration, we use Influence Function (IF) as the choice of data selection method to explain our method. DUET flexible enough to incorporate different data selection choice and we analyzed different data selection methods in our experiments (Sec. (). Our inner optimization problem aims to find the best-performing data mixture that satisfies:
x
A arg min (ev)((O),, X'S..
where S, A {X' ra)i = = = M). In other words, we need to find a subset data X; that yields the lowest evaluation task loss y (0x)((x;) while constrained mixing ratio
4

---

Published as a conference paper at ICLR 2026
First, let's consider simple approach, based prior works estimating distribution extrema (de Haam 1981 Lee & Miller 2022). We randomly sample it different data mixtures from S,-. This yields k data mixture samples {di.....,. (each satisfying the mixing ratio r). A uniform random estimator for y, is obtained by evaluating the unseen task performance of an LLM trained on each data mixture sample and taking the minimum: y² = minx; (0xx)
Leval (xxx) )
with X = arg minx, x, (Leva((x) ..... Leval (0x;) } as the solution estimate of inner problem 4 The estimator J, is the Ist-order statistic (Arnold et al., 2008)) and a random variable. While consistent (i.e., as we increase the sampling size k, we can estimate the solution of Eq. 4 more accurately), uniform random estimator V, has high variance (we provide empirical evidence in Fig. { because from k uniformly random data mixture samples, it is unlikely we select the optimal data mixture.
How can data selection help? We aim to improve the quality of estimator y by incorporating data selection methods (Sim ct al. 2022 Wang et al., (224a) into our sampling process. Specifically, we want to increase the chance of sampling high-quality data points (conversely, reduce the chance of sampling low-quality data points) from each data domain, before using it to train an LLM. To do so, let consider the of influence function (Koh & Liang 2017, Saunshi al. 2023) (IF) as data selection method into estimator yı to estimate the inner problem solution more accurately. In App. A.5 we discuss the tradeoffs between different data selection methods, such as coresets (Mirzasoleiman et al.) 2020), diversity-driven measures (Wang et al. 2024b) and LESS (Xia et al. 2024) when used in DUET. Our experimental results (Fig. () also analyzed the performance of DUET paired with different data selection methods.
IF-iiveen estimator. We construct an IF-driven estimator in the following manner: first, for each dataset D, DD from the training domains, we fine-tune separate, potentially smaller, LLM on that dataset. Second, we derive the IF score of each training data point w.r.t. the trained LLM for its respective domain (this can be computed and stored beforehand; more details in App. A.4). Lastly, given a mixing ratio r proposed at each iteration, we perform weighted sampling from each domain based on each data point's IF score within the domain dataset (instead of uniform sampling as mentioned previously) until we satisfy the mixing ratio From hereon, process as IF-weighted sampling. For each data domain, there is a higher chance to sample a data point with higher IF score. This yields a data mixture sample XTF By performing IF- we-ghthhted sampling k times, we obtain it samples of weighted data mixtures {x ,.... N/ F producing a new Frivenn estimator:
refer to this sampling
min Lova (0 X,
Leval (0va(0xa21 )}
(5)
which estimates the solution of inner optimization problem 4 Unlike the uniform random estimator mentioned earlier, IF-driven estimator emphasizes selecting data with high IF scores, and prior works (Saunshi al. (023) have regarded data points with higher IF scores of higher quality. Next, we will discuss the empirical distribution of the IF-driven estimator.
Empirical distribution. In Fig. 3 we mixed data from two training domains to train an LLM to maximize an unseen task
empirical distribution of different extimators
IF-driven estimator
accuracy (while Eq. 4 & 5 consider the minimization case,
1250
uniform random estimator
true inner problem solution
we can use max instead of min for the maximization case).
We used a fixed mixing ratio r = [0.5, 0.5). The optimal data mixture satisfying this ratio attains task accuracy indicated by the red line (obtained by iterating through all possible data mixtures in a brute-force manner). Ideally, we want our esti- mator to be as close to the red line as possible. Next, we plot the empirical distribution of the uniform random estimator and IF-driven estimator. Empirically, the IF-driven estimator (green histogram) has lower variance and bias than the uni- form random estimator (gray histogram), producing a closer estimate to the true solution (red line).
250
accuracy
Figure 3: Empirical distribution of the uniform random and IF- estimator y/. Red line is the true inner problem solution.
This suggests that the IF-vrinen estimator y estimates the solution of problem 4 more accurately.
Theoretical distribution. Exactly how well does the IF-driven estimator y/ estimate the optimal unseen evaluation task loss y, w.r.t. a given data ratio r? To answer this, we theoretically analyze this estimator's iss empirical distribution. Empirically (App.(..) (. the negative of the sampling distribution of the unseen task accuracy (we consider the negative because we are looking to maximize the accuracy,
5

---

Published as a conference paper at ICLR 2026
instead of minimizing the loss) of each sample (evr(( resembles truncated exponential distribution. Based on this, we characterize how well the IF-driven estimator yf estimates y,:
Theorem 3.2. Let { xx.., be k data mixture samples drawn from S, using IF-weighted sampling. Furthermore, assume each independent sample L (0x)²) follows the shifted truncated exponential distribution y + (x, c), for i 1.2.....,k k where exp, (A, c) is a truncated exponen- tial distribution governed by rate parameter À and truncated at e >> O. Then, the IF-driven estimator yr, defined in Eq. is a random variable: yf + E, where Vfr is the true inner problem solution of Eq. and is random noise variable with probability density function (PDF):
Ake Au
(e- -Au - dr R-1
PDF
on u [0,c).
1 -
1- 1
The proof is shown in App. B.2 and computes the probability distribution of the 1st order statistic (in which our estimator uses) of a truncated exponential distribution. Theorem 3.2 is used in DUET's convergence analysis in Sec. f In App. [B.4] we also provide details to help readers extend our analysis to other empirical sampling distributions. This also indicates that estimation error of the IF-driven estimator reduces to as the sampling size k increases asymptotically. Surprisingly. our experiments (Sec. 6 show that using k- = i is enough to select good data mixtures, underscoring the effectiveness of using data selection as opposed to random sampling. We also found that given sufficient budget, using varying &o gives us granular control of DUET's performance (Sec. 6.3
3.3 USING BAYESIAN OPTIMIZATION FOR OUTER PROBLEM
With the IF-driven estimator introduced to estimate the inner optimization problem solution (or any estimator using a desired data selection method of choice), we shift our focus to solving the outer optimization problem of problem which aims find the optimal data mixing ratio r' for the unseen evaluation task. Since the solution of the inner problem y = minxes, (0x) depends only on the mixing ratio r, we can define a function f(r) - yf = minxes. (fxe) where for given mixing ratio r, we use the IF-driven estimator estimate solution for the inner problem. producing f(r). As such, the outer optimization problem of problem can be rewritten into min, f(r) where r E T8 is a probability simplex representing the mixing ratio over the n training domains. DUET uses BO constrained to ||r = 1 (See. 1 to find the optimal mixing ratio r* for the outer problem.
BO is suitable for solving this problem for few reasons. First, evaluating f requires us to use the [F-driven estimator to estimate the inner optimization problem solution and thus is a black- box function with no closed, mathematical form: BO is a principled and popular framework to optimize such black-box functions (Garnett 2023 Knapp/ 2018). Second, are estimating the inner problem solution (Theorem. 3.2 using data selection. This implics we can only obtain noisy observations (r) + 6, where is random noise variable with the same distribution as that in Theorem 3.2 fortunately, BO handles noisy function observations gracefully (Srinivas et al.) 2010 Chowdhury & Gopalan 2017) during the optimization process, allowing us to find the optimal mixing ratio eventually (theoretical results shown in Sec.
3.4 INTERLEAVING THE IF-DRIVEN ESTIMATOR AND BO
DUET uses BO at the outer level and IF-driven estimator at the inner level to iteratively optimize the data mixture, solving problem We formally describe DUET in Algorithm !
At iteration 6, DUET uses the LCB acquisition function (Srinivas et al. 2010) on the GP posterior to propose a candidate mixing ratio r; for our data domains (Line 3). Using the proposed mixing ratio rr. we use IF scores of each data point compute the IF-driven estimator y², and fine-tune an LLM with the selected data points and observe the feedback from the downstream unseen task based on the fine-tuned L.M.. We keep track of the best performing data mixture sample X²' at every iteration (Line 4, and 6). Next, we include (re+1- into our historical observations De+1 (Line 7) and update our GP posterior (Line 8). After which, we repeat the entire feedback process to select the next LLM fine-tuning data mixture, until the budget of BO iterations is exhausted. In the end, we recover the best performing data mixture 2 for the unseen evaluation task (Line 10).

---

Published as a conference paper at ICLR 2026
Algorithm DUET:
Optimizing training Data Mixtures for an Unseen Evaluation Task
1: Input: n training datasets from domains (Di...., D. Computed IF scores of each data point (App. A.44) w.r.t. its domain dataset and locally trained model. Initial observation of data mixing ratio and evaluation task performance: Do _ {(ro: fo SE kernel Ki sampling size k, parameter B3 for acquisition step and total number of BO iterations T.
2: for 1,..., do
3: Tt arg min, (e(rr) Be (r) (BO acquisition step)
4: IF-weighted sampling to obtain k samples of data mixtures (X1 ..... XE (Sec. 3.2
5: IF-driven estimator at iteration
Mr. minx, { (eva (xx;)) Leval (6x!= (e.a))
6: Keep track of best performing data mixture X² arg minx, (0 ).. ..... (O
1 Uf ( (r......)
7: D. = D
8: Update the GP posterior and with updated observations Di+1 (Sec. 2.1
9:
end for
10: X" arg min x ((t Leval (0x;)
THEORETICAL ANALYSIS
4.1
CONVERGENCE ANALYSIS OF DUET USING CUMULATIVE REGRET
We analyze the convergence rate of DUET using the growth of attained cumulative regret (Chen et al. 2024b) Rt Ein Iy, [(xx) Σ'= (f(")) + 6g f(r:)] for T BO iterations. The attained cumulative regret consists of two terms, where proposed at each iteration while e, indicates how well we can estimate the inner problem solution at every iteration. By analyzing the attained average regret RT/TT with T + oc, the following Theorem helps understand how close our algorithm converges (Berkenkamp et al. 2019
indicates the quality of mixing ratio re
Theorem 4.1. Let be the outer problem objective defined in Sec. 3.3 with bounded RKHS norm: llflk Vf, fix. Also, let our IF-driven estimator for the inner problem solution be governed by the
error distribution introduced in Theorem 3.2 with constant c and X 1. Let Ac, e1(1- 5)
[1-e (1-e-<ht
where k a fixed predecided sampling size. Then, running DUET over J using the LCB acquisition function found in (Chowdhury & Gopalan 2017) each BO iteration
1...., T yields the
following attained average regret (Chen et al. 2024b) upper bound with probability at least 1 8:
RT
6( 6(V8 + Vk)
22Ac, 48
lim T 700
+ +
T
V8k
f(r:) and c, independently using BO regret analysis (Chen et al.) 2024b Chowdhury & Gopalan 2017) and the error distribution defined in Theorem 3.2 Our Theorem's average regret indicates how close our algorithm converges to the optimal evaluation task loss with increasing BO iteration T and different choices of sampling size k. Notice that because characterizes the error of our estimator Theorem 3.2 a larger e would decrease Ac, and our average regret. In addition, larger sampling size k reduces the estimation error of the inner problem (Theorem. 3..2), decreasing Az,k and reducing our regret bound, although our experiments (Sec. 6.2) show that setting &o = is sufficient to achieve good performance.
The proof is provided in App.
B.3 and bounds (f()))
5 PRACTICAL CONSIDERATIONS
are free to use any data selection methods in DUET's inner loop. We specifically highlighted as a data selection method because in our experiments, IF worked slightly better when paired with BO (see Fig. for detailed ablation) as compared to other selection methods. It also has some interpretable advantages (Sec. A.6). Even though computing IF scores could be budget-intensive, practical tricks, such as parallel computation, Hessian approximation (Agarwal et al. 2017), pre-computation, or a smaller surrogate model can speed up computation.
7

---

Published as a conference paper at ICLR 2026
If scaling large-scale datasets is too compute-intensive methods in DUET, such as LESS (Xia et al., 2024) or TracIn (Pruthi et al.] 2020) with some performance-tradeoff In the extreme case, one introduced in Sec. S.c.3.2 which does not perform any data selection. We experimented with DUET paired with different data selection methods in Sec. [6.3] and discussed their actual compute-time in App. [.I In addition, DUET's iterative optimization process is feature: subjecting LLMs to multiple rounds of training in a feedback loop a natural part of its deployment life-cycle.
could also use cheaper data selection
resort to the uniform random estimator
EXPERIMENTS AND DISCUSSION
We conduct extensive experiments showcase the effectiveness of DUET compared other baselines. We optimize data mixtures with different methods based on multiple rounds of evaluation task performance feedback. Then, we fine-tung munetuneee an LLM with the optimized data mixture. Lastly, we deploy the L.M on the evaluation task to evaluate how well the model has performed. We provide more details of our experimental setup and our algorithm computational cost in App. C.T
6.1 EXPERIMENTAL SETUP
Our experiments are carried out by performing PEFT (Hu et 2021 of L.1 ama 3- 8b- Inst ruct (Touvron et al. 2023 across different L.M knowledge domains. We also ran our experiments with Qwen2. 5-:---77-- 7B Inst ruct (Qwen et al.] 2025) and present the results in App. C.3 Our findings were similar even for different LLMs. The training data domains for LLM evaluation consists of
topics: Wikitext (Merity et al. 2016). gsm8k (Cobbe et al. (021). PubmedQA (Tin al. (019)). HeadQA (Vilares& & Gómez-Rodrignt Rodriguez 2019). SciQ (Welbl et al. 2017). TriviaQA (Joshi et al.
2017). TruthfulQA (Lin et al.) 2022). Hellaswag (Zellers et al. 2019), and Commonsense(A
TTlmorr al.) 2019), We also varied the difficulty of the unseen task by making them out-of-domain (see captions of Fig. 4. Our LLM performance might have slight differences from existing papers, most likely due to evaluation setup differences, which we elaborate in App. C.I For DUET, "warm-started" the BO processes by evaluating 50 different random data mixtures and updated the GP with the performance observations.
We ran several baselines: DoReMi (Xie et al. 2023a) is a data- lata-mixing approach that optimizes the data mixture in distributionally robust manner. LESS (Xia et al. 2024) is data-selection method based on data gradient similarities. The Uniform weights baseline uses data mixture of uniform ratio across different domains. We ran our baselines for the same number of iterations DUET and take the best performing result to ensure similar compute comparison. We also used DUET with few different data selection methods: DUET-IF uses our IF-driven estimator (Eq. 5 to select data mixtures at each BO iteration; DUET-UR, introduced in Sec. 3.2 uses the uniform random estimator and randomly selects data mixtures that satisfy the proposed mixing ratio; DUET-RH (Remove Harmful) removes 20% of data points with the lowest IF scores from each data domain, before performing sampling. DUET-LESS (Xia et al., 2024) and DUET-logdet (Wang et al.) 2024b), which incorporate different data selection methods into DUET, were also used in our ablation studies (Fig. (). We used a sampling size of = and BO iterations T = 10. We also constrained the total number of selected data points to M 10000 with temperature of 0.75 in our LLMs. This makes the "feedback" (performance) of all valuation tasks noisy, similar to real-world tasks.
We also compared DUET with other baselines, such Aioli (Chen et al.) 2024a), Multi-fidelity BO (Yen et af. 2025), online data-mixing (Albalak et al. 2023a). alongside naive approaches: e.g., using more training tokens, random search or only data selection. Due to space constraints, we show these results in Table, 2 In general, DUET still finds better data mixtures than these baselines.
6.2 MAIN RESULT
DUET finds more optimal data mixtures. Our result (Fig. 4 shows that DUET finds better data mixtures within a few iterations of feedback loops. The first column in Fig. consists of a relatively easier task where the evaluation domain is part of the training task domains. In this case, DUET (green plot) uses feedback from the evaluation task to find the optimal data mixture with more weights on the relevant training data domain, TruthfulQA. On the other hand, we observe the weakness of conventional methods which cannot exploit coarse feedback: DoReMi (orange dotted line) and LESS
8

---

Published as a conference paper at ICLR 2026
ELETLA
(a) TruthfulQA
(b) gsm8k
(c) PubMedQA, HeadQA (d) Commonsense, Trivia
Figure Results on unseen LLM evaluation task domains over 10 iterations (higher is better) for - Instrrtt Experiments were repeated with Qwen2. .
ruct in App. The caption shows the evaluation task. Underlined evaluation tasks are harder because the evaluation task domains are removed from the training data (i.e., oft-off-omain) Results for more baselines are presented in Table. [
(orange dotted line) both cannot specifically adapt to the evaluation task and hence do not perform as well. In the 2nd, 3rd and 4th columns, we increased the difficulty of our evaluation task by removing the evaluation task domain from our training domains (the evaluation task is now out-of-domain). Surprisingly, DUET still can use feedback from the unseen task to automatically optimize the data mixture, achieving better LLM performance than other baselines. This suggests data from another training domain is still useful for the out-of-domain evaluation task (e.g.. Wikitext data can still be helpful for mathematical questions in gsmök). Hence, DUET effective in both in-domain and out- out-of-dont -ot-of--ommin tasks. In App. C.4 we qualitatively discuss the optimal mixing ratios found by DUET.
6.3 ABLATION EXPERIMENTS
While we have shown that DUET outperforms existing baselines, we also ran several ablations (using Fig. 4d) setting) tease apart several components in DUET.
Ablation of different components in DUET. Fig. shows the impor- tance of both BO and data selection techniques in DUET. If we used a uniform data mixture to train an LLM, we can only achieve a base- line performance given by the red dotted line. With just BO, DUET automatically reconfigures the mixing ratio and attains performance gain (A). Next, by incorporating data selection methods, such as using IF in DUET-IF, we attain further performance gains (B) indicated by the green plot. Different data selection methods used in DUET also improves the LLM's performance to a different extent (C). Therefore, this affirms the importance of interleaving data selection and BO.
Lut petter to
Figure Ablation of differ- ent components of DUET.
Ablation of using different data selection methods in DUET. How do different data selection methods fare when used in DUET's inner loop? In Fig. 6 we found that IF outperforms other data selection methods (LESS, RH, log-det (Wang al.) (024bb) when used in DUET's inner loop. This suggests that IF retrieves higher-quality (or remove lower-quality) data points at each iteration better than other methods. This aligns with our discussion in App. A.5 where we explained how IF, being able to remove low-quality data, yields better training data mixture in our unseen task setting. All in all, we are free to use different data selection techniques (each with different computational cost, performance) in DUET's 's inner loop.
CUETH launi
CURTLESS
DIETRE
cO
Figure 6: Ablation of using different data selection meth- ods in DUET.
Allaion of varying sampling size Lastly, we also found that increasing sampling size & in DUET's inner loop (Fig. 7 helps DUET find more optimal training data mixtures. This aligns with our theoret- ical findings from Theorem 4.1 which shows that larger k improves DUET's s convergence. In practical settings, if budget permits, LLM owners can fine-tune multiple copies of LLMS (i.e., increase k) to
DuETi de
0. 0-31
cO
Figure 7: Ablation of sam- pling size k in DUET.

---

Published as a conference paper at ICLR 2026
improve DUET's performance. However, results in Fig. 4 showed that even with = 1, DUET outperforms other baselines.
7 CONCLUSION AND LIMITATIONS
Our paper proposes DUET, novel algorithm that exploits multiple rounds of coarse, noisy feedback from a downstream unseen evaluation task automatically optimize training data mixture for LLMs. Our approach offers effective solution to address the unseen task setting, where fine-grained data information unavailable (and conventional approaches fail). is also quite flexible, allowing us to choose amongst different data selection methods in its inner loop. We provide theoretical guarantees of DUET and empirically show that optimizes data mixtures in a variety of LLM evaluation tasks better than other baselines, One limitation is that our paper focused LLM fine-tuning, but broadly speaking, we believe that DUET can be adapted to and would work equally well for pre- training. This leaves room for fruitful future research.
ACKNOWLEDGMENTS
This research is supported by the National Research Foundation, Singapore under its Al Singapore Programme (AISG Award No: DSSG2-PDD/202--01-033 DesCartes and is supported by the National Research Foundation, Prime Minister's s Office, Singapore under its Campus for Research Excellence and Technological Enterprise (CREATE) programme. Zhiliang Chen is supported by the Agency for Science, Technology and Research (A STAR), Singa- pore.
This research part the programme
8 ETHICS STATEMENT
Our work strives to improve the performance of LLMs for the greater good. We do not foresee any ethical concerns related our work. From our theoretical findings and experiments, our work can also handle noisy real-world feedbacks robustly.
REFERENCES
Naman Agarwal, Brian Bullins, and Elad Hazan. Second-order stochastic optimization for machine learning in linear time. 160.00339, 2017.
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training. arXiv:23 / 2,020 2023a.
Alon Albalak, Liangming Pan, Colin Raffel, and William Yang Wang. Efficient online data mixing for language model pre-training, arXiv:23/2.022400 2023b.
Alon Albalak, Yanai Elazar, Sang Michael Xie, Shayne Longpre, Nathan Lambert, Xinyi Wang, Niklas Muennighoff, Bairu Hou, Liangming Pan, Haewon Jeong, Colin Raffel, Shiyu Chang, Tatsunori Hashimoto, and William Yang Wang. A survey data selection for language models. arXiv:2402. 16827, 2024.
Julyan Arbel, Olivier Marchal, and Hien D. Nguyen. On strict uubb-aussianity. optimal proxy variance and symmetry for bounded random variables, 2019.
Barry C Arnold, Narayanaswamy Balakrishnan, and Haikady Navada Nagaraja. A first course in order statistics. SIAM, 2008.
Felix Berkenkamp, Angela P. Schoellig, and Andreas Krause. No-regret bayesian optimization with unknown hyperparameters. In Proc. ICML, 2019.
Mayee F. Chen, Michael Y. Hu, Nicholas Lourie, Kyunghyun Cho, and Christopher Ré. Aioli: A unified optimization framework for language model data mixing. arXiv:2411.055.5 2024a.
Zhiliang Chen, Chuan Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Towards AutoAI: Optimizing a
machine learning system with black-box and differentiable components. In Proc. ICMI,, 2024b.
10

---

Published as a conference paper at ICLR 2026
Sayak Ray Chowdhury and Aditya Gopalan. On kernelized multi- bandits. In Proc. ICML, 2017.
Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems. arXiv:2110.14168, 2021.
Laurens Haan. Estimation of the minimum of a function using order statistics. Journal of the American Statistical Association, 76(374):467-469, 1981.
Simin Fan, Matteo Pagliardini, and Martin Jaggi. Doge: Domain reweighting with generalization estimation. arXiv:2310. 15393, 2024.
Peter Frazier. A tutorial Bayesian optimization. arXiv:1807.02555 0200010,,11, 2018.
Yaroslay Ganin and Victor Lempitsky. Unsupervised domain adaptation by backpropagation. In Proc. ICML, 2015.
Lco Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding. Jeffrey Hsu, Alain Le Noac'h, h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. framework for few-shot language model evaluation, 07 2024. /ttp::t::
org/ trecordss126055
Jacob Gardner, Matt Kusner, Xu Zhixiang, Kilian Weinberger, and John Cunningham. Bayesian optimization with inequality constraints. In Proc. JCML, 2014.
Roman Garnett. Bayesian Optimization. Cambridge Univ. Press, 2023.
Ce Ge, Zhijian Ma, Daoyuan Chen, Yaliang Li, and Bolin Ding. Bimix: bivariate data mixing law for language model pretraining. arXiv:2405. 14908, 2025.
Stewart Greenhill, Santu Rana, Sunil Gupta, Pratibha Vellanki, and Svetha Venkatesh. Bayesian optimization for adaptive experimental design: A review. IEEE Access, 8:13937-13945 2020. doi: 10.1109/ACCESS. 2966222.
Ziyao Guo, Kai Wang, George Cazenavette, Hui Li, Kaipeng Zhang, and Yang You. Towards lossless dataset distillation via difficulty-aligned trajectory matching. arXiv:2310.05773, 2024.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Ruther- ford, Diego de las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katherine Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Oriol Vinyals, Jack William Rae, and Laurent Sifre. An empirical analysis of compute-optimal large language model training. In Proc. NeurlPS, 2022.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. (0ti::2006006885 2021.
Neil Jethani, Mukund Sudarshan, Ian Covert, Su-In Lee, and Rajesh Ranganath. Fastshap: Real-time shapley value estimation. arXiv:2107.07436 2022.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William W. Cohen, and Xinghua Lu. Pubmedqa: A dataset for biomedical research question answering. arXiv:1909.06146, 2019.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. Triviaqa: A large scale distantly supervised challenge dataset for reading comprehension. arXiv:1705.03551, 2017.
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization, 2017.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In Proc. ICML, 2017.
Anthony Lee and Steven Miller. Generalizing the german tank problem. arXiv:2210. /5339, 2022.
11

---

Published as a conference paper at ICLR 2026
Tianshi Li, Sauvik Das, Hao-Ping Lee, Dakuo Wang, Bingsheng Yao, and Zhiping Zhang. Human- centered privacy research in the age of large language models. PrXiv:2402.0199. 2024.
Stephanie Lin, Jacob Hilton, and Owain Evans. Truthfulqa: Measuring how models mimic human falsehoods. arXiv:2109.07990 2022.
Mingsheng Long. Han Zhu, Jianmin Wang, and Michael I. Jordan. Deep transfer learning with joint adaptation networks. In Proc. ICML, 2017.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. models. arXiv:1609.07843 2016.
Pointer sentinel mixture
Baharan Mirzasoleiman, Jeff Bilmes, and Jure Leskovec. Coresets for data-efficient training of machine learning models. arXiv:1906.0182. 2020.
Krikamol Muandet, David Balduzzi, and Bernhard Schölkopf. Domain generalization via invariant feature representation. arXiv:1301.2113 2013.
Garima Pruthi, Frederick Liu, Mukund Sundararajan, and Satyen Kale. Estimating training data influence by tracing gradient descent. arXiv:2002.08444, 2020.
Edward O Research and Development, 62(6):2-1, 2018.
Bayesian optimization for accelerated drug discovery. IBM Journal of
Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang. Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang. Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, and Zihan Qiu. Qwen2.5 technical report. 2411v:2412.151155 2025.
Nikunj Saunshi, Arushi Gupta, Mark Braverman, and Sanjeev Arora. Understanding influence functions and datamodels via harmonic analysis. In Proc. ICLR, 2023.
Seungjue Shin, HeeSun Bac, Byeonghu Na, Yoon-Yeong Kim, and Il chul Moon. Unknown domain inconsistency minimization for domain generalization. In Proc. ICLR, 2024.
Rachael Hwee Ling Sim, Xinyi Xu, and Bryan Kian Hsiang Low. Data valuation in machine learning: "ingredients", " strategies, and open challenges. In Proc. IJCAI, 2022.
Jasper Snock, Hugo Larochelle, and Ryan P Adams. Practical bayesian optimization of machine learning algorithms. In Proc. NeurIPS, 2012.
Niranjan Srinivas, Andreas Krause, Sham Kakade, and Matthias Seeger. Gaussian process optimiza- tion in the bandit setting: No regret and experimental design. Proc. ICML, 2010.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: question answering challenge targeting commonsense knowledge. arXiv:1811000331 2019.
Sebastian Shenghong Tay, Chuan-Sheng Foo, Daisuke Urano, Richalynn Leong, and Bryan Kian Hsiang Low. Bayesian optimization with cost-varying variable subsets. Proc. NeurIPS, 2023.
Daniel Ting and Eric Brochu. Optimal sub-sampling with influence functions. arXiv:1709.01716. 2017.
Kushal Tirumala, Aram H. Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. arXiv:2205.10770. 2022.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie- Anne Lachaux, Timothée Lacroix. Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation language models. arXiv:2302.1399 /3971, 2023.
12

---

Published as a conference paper at ICLR 2026
David Vilares and Carlos Gómez-Rodriguezz HEAD-QA: A healthcare dataset for complex reasoning. In Proc. ACL, 2019.
Jindong Wang, Cuiling Lan, Chang Liu, Yidong Ouyang, Tao Qin, Wang Lu, Yiqiang Chen, Wenjun Zeng, and Philip S. Yu. Generalizing to unseen domains: A survey on domain generalization. arXiv:2/03.03097, 2022.
Jingtan Wang. Xiaoqiang Lin, Rui Qiao, Chuan-Sheng Foo, and Bryan Kian Hsiang Low. Helpful or harmful data? fine-tuning-fiee shapley attribution for explaining language model predictions. In Proc. ICML, 2024a.
Peiqi Wang, Yikang Shen, Zhen Guo, Matthew Stallone, Yoon Kim, Polina Golland, and Rameswar Panda. Diversity measurement and subset selection for instruction tuning datasets. arXiv:2402.02318, 2024b.
Johannes Welbl, Nelson F. Liu, and Matt Gardner. Crowdsoureing multiple choice science questions. In Workshop on Noisy see--geneateed Text, 2017.
Christopher KI Williams and Carl Edward Rasmussen. Gaussian processes for machine learning. volume 2. MIT press Cambridge, MA, 2006.
Mengzhou Xia, Sadhika Malladi, Suchin Gururangan, Sanjeev Arora, and Danqi Chen. Less: Selecting influential data for targeted instruction tuning. arXiv:2402.04333. 2024.
Sang Michael Xie, Hieu Pham, Xuanyi Dong, Nan Du, Hanxiao Liu, Yifeng Lu, Percy Liang, Quoc Le. Tengyu Ma, and Adams Wei Yu. Doremi: Optimizing data mixtures speeds up language model pretraining. In Proc. NeurIPS, 2023a.
Sang Michael Xie. Shibani Santurkar, Tengyu Ma, and Percy Liang. Data selection for language models via importance resampling. In Proc. NeurIPS. Curran Associates, Inc., 2023b.
Wanyun Xie, Francesco Tonin, and Volkan Cevher. Chameleon: A flexible data-mixing framework for language model pretraining and finetuning. arXiv::5055 24844, 2025.
Thomson Yen, Andrew Wei Tung Siah, Haozhe Chen, Tianyi Peng, Daniel Guetta, and Hongseok Namkoong. Data mixture optimization: A multi-fidelity multi-scale bayesian framework, 2025.
URL https: htttttt/s///xiiiii http/axxvvvvoe/ abs/ 21023
Rowan Zellers, Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence? arXiv: 1905.07830, 2019.
Bo Zhang, Xiaoming Zhang, Yun Liu, Lei Cheng, and Zhoujun Li. Matching distributions between model and data: Cross-domain knowledge distillation for unsupervised domain adaptation. In Proc. ACL, 2021.
Wenyu Zhang. Li Shen, Wanyue Zhang, and Chuan-Sheng Foo. Few-shot adaptation of pre-trained networks for domain shift. 2rXiv:2205.152:: 2022.
Xiaoyu Zhang, Juan Zhai, Shiqing Ma, Chao Shen, Tianlin Li, Weipeng Jiang, and Yang Liu. Speculative coreset selection for task-specific fine-tuning. arXiv:2410.01226, 2024.
A SUPPLEMENTARY MATERIAL
A.1 REAL-WORLD EXAMPLES OF OUR PROBLEM SETTING
In our problem setting. (a) there is no direct access to the data (e.g., its domain, distribution, or labels) involved the unseen evaluation task but (b) multiple rounds coarse feedback (details covered in Sec. 2.2) can be gathered from the task using trained LLM. Here, we provide several real-world examples in which such a setting occurs.
End-to-end encrypted conversations between LLM and users. This setting is specific to the conversational setting between a trained LLM and human users. LLM owners are interested in fine- tuning an LLM to converse well with some human-user demographies but due to real-world privacy
13

---

Published as a conference paper at ICLR 2026
2024). conversations between deployed LLM and users are end-to-end encrypted So, an LLM owner does not have any knowledge of the conversation domain the (unlabeled or labeled) data seen during test-time. Instead, they only receive a feedback on how well the LLM has performed in the conversation (e.g.. ratings from the human user, how long each user stays on the applicaton). The LLM owner can collect multiple rounds of feedback over period of time. Hence, they can exploit this feedback to iteratively refine the training data mixture. Many chat-driven applications (e.g. whatsapp. telegram) nowadays use and-to-end encrypted chats, so our problem setting is relevant here.
concerns et. al.
during test-time (openai com/ wwweeerrreeerrrriiieeeee
Model marketplace. In addition, there are other scenarios in which a model owner needs to improve an ML model without having access to the data involved in the unseen evaluation task. For instance, an MI. model owner might rent or sell an MI. model in a model marketplace (e.g., (8.httss However, the consumer might give feedback (e.g., how often the model makes mistakes) to the ML model owner in hope that the ML model owner can improve the model's performance on its own evaluation task. Furthermore, the data used by the consumer in its evaluation task are considered sensitive data, so the ML model owner does not know any data involved in the unseen evaluation task. Hence, the ML owner can only rely on feedback from the consumer to improve the training data mixture.
//aws. ama zon com ma wketplace. so ut ons/ / ma c learniarnnng
A.2 MORE RELATED WORKS ON DATA MIXING AND SELECTION
Recently, large class of data selection methods utilizing coresets (Zhang et al. 2024), diversity measures (Wang et al., 2024b), gradient information (Xia et al.) 2024) or influence function (Koh & Liang 2017) has been introduced to retrieve smaller subset of data from an existing dataset. These data selection methods have become popular because they reduce training dataset size (which an attractive feature when traning LLMs) and prior work (Xia et al. 2024) showed that training a model with strategically selected data points allows it to perform better. In addition, data mixing works (Xie et al., 2023a Ge et al., 2025 Albalak et al., 2023b) have studied how to reweigh different data domains to produce optimal data mixtures, using distributionally robust optimization or entropy-basee signals. However, these works, when used in isolation, do not work well in our setting because they do not exploit feedback from an unseen evaluation task. For example, even if we can retrieve a high-quality data subset from the training data domain, this domain might not even be relevant the unseen evaluation task. Hence, data mixing and selection methods on their own are not applicable to our setting because they have no way discern how relevant the training data domain is to the unseen task. Instead, our paper's is algorithm interleaves BO and data selection method together to exploit feedback from the unseen evaluation task to optimize our training data mixture. Indeed, our experimental results in Sec. show that DUET performs better than other data mixing and selection works.
A.3 EXTENSIONS AND DISCUSSION OF DUET IN OTHER SPECIAL SETTINGS
Here, we discuss some extensions of DUET to other settings that fall beyond the scope of our paper. However, find them insightful and useful when implementing DUET in practice.
Should we -e-fine-tune/re-rrain the LLM from scratch each time in DUET or continue training the model from the previous iteration? Our problem formulation in Sec. 2222 theoretical findings (Sec. 4 assumes DUET re-train the LLM from the same initial checkpoint at every iteration. This is necessary to ensure our surrogate function landscape in GP remains consistent throughout the BO process, allowing DUET to converge. From a practical perspective, we speculate that DUET will be less effective if we continue training an LLM from the previous iteration. This is because training data mixtures from earlier iterations might not be useful for the unseen evaluation task and the model might memorize (Tirumala et al. 2022) irrelevant information that are difficult to be overwritten in later BO iterations.
DUET for extremely large datasets used pre-training. We can amortize the computational cost of IF computation by pre-computing and storing them beforehand (App. A.4) in our paper's fine-tuning setting. However, the size of datasets used in pre-training could be extremely large, which might still lead to large computational cost when computing IF scores of every data point in such datasets. To make computation faster, we can adopt methods in (Koh & Liang 2017) to approximate hessian inversions when computing IF scores. We can also sample a smaller subset of data compute
14

---

Published as a conference paper at ICLR 2026
IF scores, before training a neural network (Jethani al. 2022 predict the IF scores of other data points.
Noisy feedback setting. In some practical settings, the feedback from the unseen task is noisy. For instance, user ratings have a variance even within the same user demographics. How does DUET fare when the feedback from the unseen evaluation task is noisy? Fortunately, DUET is equally effective even when feedback is noisy. Feedback noise becomes part of the observation noise (Sec. (3.3) under the BO framework in DUET. In our experiments (Sec. 6.2). the evaluation task feedback is inherently noisy since LLM responses are probabilistic in nature, but DUET still performs well empirically.
A.4 INFLUENCE FUNCTION AND ITS CALCULATIONS
Influence function (IF) Koh & Liang 2017) has been developed to study the influence of single data point ML model's s predictions. In this section we provide a summary of IF and its derivation. The influence of a data point 2 on the loss of a test data point (or a set of test data points) Sea for an ML model parameterized by 0 is given by the closed-form expression:
Fi V
VoL(z,))
(6)
where L is the loss function of the ML model and H is the hessian of the ML model w.r.t. parameters 0. In short, a data point is deemed more "influential" in reducing the model loss on a test data point if it has a higher IF score. As such, IF scores have also become a popular method in selecting data points which are more helpful in training an ML model.
In our work, we segregated a validation dataset from each data domain's dataset, in which we use to derive the IF score of every training data point in that domain w.r.t. the validation dataset (after fine-tuning an L.M over the training data till convergence). Then, we normalize these IF scores (for data points in each data domain), allowing us to perform weighted random sampling at every BO iteration of our algorithm, obtaining data subset of size n for given data domain. This weighttt sampling is repeated for every data domain until we sample dataset fulfilling the proposed mixing ratio at every BO iteration. Hence, the resulting data mixture contains more proportion of high-quality data points (based on IF scores). A summary of the Ie-ihtttted sampling process for one data domain is given in Alg. 2 In our algorithm, we repeat this procedure for every data domain.
Algorithm 2 weighted sampling for one data domain containing dataset D
1: Input: number of data points n required for the given data domain (taken from the mixing ratio proposed at current iteration). Dataset D = {21, Ø2, .... 2Ip b. Influence value of each data point in data domain dataset D: I A [41, 12. .2.... pil small constant e avoid degenerate-case normalization.
2: Normalize the IF mi (Z)+e Σ1 3: Perform weighed sampling from dataset D according to weights given by Inormalized times.
scores
into (I)+e)
probabilities:
I Inormalizdd
4i+min (Z)+e
Ippi
Σl
IF scores can be pre-computed and stored. In addition, we just need to pre-compute the IF scores of every data point once before reusing them repeatedly at every BO iteration to perform IF-weighted sampling. This greatly improves our algorithm's efficiency and runtime, compared to other methods (see next section) which requires us to perform expensive computation every iteration. We provide the computation runtime of calculating IF scores in App. C.1
A.S DISCUSSION OF USING OTHER DATA SELECTION METHODS TO SOLVE INNER OPTIMIZATION PROBLEM IN DUET
Data selection methods (Albalak et al. 2024; Guo et al. 2024 Wang et al., 2024b) have been used to retrieve a representative subset of data from larger datasets. We note that in our work different data selection methods can be interchanged to produce different estimators for the inner problem solution in line 4 and 5 of Algorithm [ For example, instead of using the IF-driven estimator which performs weighted sampling based on each data point's 's IF scores, one could use LESS (Xia et al. 2024) to
retrieve data subsets for the inner optimization problem. However, experiments (Fig. 66 have
15

---

Published as a conference paper at ICLR 2026
shown that other data selection methods perform slightly worse than IF when used in DUET's inner loop. We speculate that this occurs for a few reasons.
Specifically, IF (Koh & Liang 2017) is effective identifying non-useful data (e.g., nonsensical text, text with lots of spelling mistakes, blur images) and so IF will down-weigh low-quality datapoints when sample from that data domain. Doing so is effective in DUET's setting because these nonsensical training data are unlikely to be useful for any tasks, so their removal can boost the performance of the selected data mixture on the unseen task. While LESS contains similar formulation as IF, it merely consider the gradient dot-products and ignores the hessian of the loss function (Koh& & Liang 2017) during computation. Hence, it does not contain as much information as IF.
In addition, diversity-driven methods (Wang et al. 2024b; Zhang et al. 2024) tend to select training data subsets that are "most representative" of the training data domain. However, from observation, they tend to keep nonsensical data points in the final data mixture, which is not as effective as IF, which down-samples these points. Also, representative data of training domain might not be useful for an unseen task if the task is not related. Lastly, when calculating the data log-determinant, we need to project data into embedding space with an embedding model, and hence the effectiveness of the embedding model also affects the selection process. Effectiveness aside, diversity-din also dependent the mixing ratio chosen at each iteration. Therefore, need to recompute the log-determinant (Wang et al. 2024b) or coreset (Zhang et al.) 2024) at every iteration. On the contrary, IF scores can be pre-computed and stored prior running DUET.
methods
Therefore, different selection methods have different properties (above), but conceptually and empiri- cally, we found IF to work better in our unseen task setting. Our ablation studies (Fig. (6) affirms our claim: using IF in DUET attains the highest performance as compared to selection methods such as LESS and -ogedetermnnant. We hope DUET can serve as a testbed for more advanced selection methods in the future.
A.6 MORE DISCUSSION ON THE IF-DRIVEN ESTIMATOR
Here, we provide more justification behind our choice of the IF-driven estimator.
Why not take the data with the top-N IF scores instead of sampling? One obvious alternative is to pick the data subset with the top IF scores (satisfying the given data ratio) or remove the datasubset with the lowest IF scores. We did not find this selection method effective in practice. Because IF-values are pre-computed and independent (App.AA (A.4), we end up selecting the same few datapoints with top IF scores every BO iteration in DUET. With the IF-weighted sampling, we select more diverse data points, yielding better data mixtures. This becomes more apparent when many data points have high IF scores and sampling provides us access to these data points every iteration. Empirically, we also found that the deterministic approach performs worse (See DUET-RH Sec. 6.2) than IF-weighted sampling.
Performing weighted-sampling with IF-scores not only retains the benefits of using IF-scores (we upweigh higher quality data while still having access to data points with moderate IF-scores), but also allows us exploit additional computational resources reduce the estimation error by increasing the sampling size. For instance, Theorem 3.2) shows that higher sampling size reduces the inner problem estimation variance and bias. Intuitively, LLM owner could exploit more compute to increase sampling size & and sample more data mixtures and reduce the estimation error at every BO iteration. These estimation error variances also handled gracefully in the BO framework.
In our experiments, we demonstrate that even with limited resource to sample and train an LLM once (k 1), DUET still outperforms other baselines in our setting. In fact, Our ablation (Fig. shows that increasing i results in a performance boost, showcasing the benefits of sampling in world settings (where computational resources are available to make multiple queries each BO step).
A.7 EMPIRICAL DISTRIBUTIONS OF ESTIMATORS FROM DIFFERENT DATA SELECTION METHODS
We have introduced the IF-driven estimator in Sec. 3.2 to estimate the solution of the inner problem. The IF-vri estimator performs WF-weighted sampling on data points from each data domain to
16

---

Published as a conference paper at ICLR 2026
emprical sampling ovaluat performance
emstrical extimators
sumple candanty
andon setimaoo
sumple drewn randomiy aber minssnng hamrfu deta
rereest Jumrta then aritim
frue inner problem sdutce
one problem solition
3000
500
(a) Empirical sampling distribution of (b) Empirical estimator distribution Leval (0x)
Figure 8: (a): Empirical distribution of evaluation task accuracy Leval (0x) from each data mixture sample X' (b): empirical distribution of the estimators introduced in Sec. [3.2 The green histogram our method of performing IF-weighted sampling to obtain data mixtures. The gray histogram is simply randomly sampling data mixtures with no data selection methods. The purple histogram is the method of removing 20% of the data points with the lowest IF scores.
produce data mixture samples (Fq. 5) constrained to a data mixing ratio r. Each data mixture sample LLM before obtaining a feedback on how well has performed
then used train/fine-tune on the unseen evaluation task. Hence, this feedback based on each data mixture sample is also sampling distribution that we can empirically observe. Fig. &a shows the sampling distribution the evaluation task performance obtained from each data mixture. Empirically, we see that the negative of this distribution similar truncate exponential distribution mentioned in Theorem 3.2 (We consider the negative of this random variable because our paper considers the evaluation task loss, empirically we maximize the evaluation task accuracy). In addition, the truncated exponential distribution is appropriate because it implies the unseen evaluation task loss upper bounded at y for non-nee negative constant c; this is a reasonable assumption because many real- world feedbacks are bounded (e.g. user ratings).
We also plot the empirical distribution the IF-driven estimator introduced in Eq. in Fig. 8b The distribution coincides with the estimator's 's distribution (formally, y + c) introduced in Theorem 3.2 From the estimator's 's distribution, we see that the IF-driven estimator (green histogram) has the lower bias and variance compared to other estimators.
B PROOFS
B.1 PROOF OF THEOREM 3.1
Theorem 3.1. X", the optimal set of data points from D, the solution of the original problem iff
r" =ratio(X" X*) the optimal mixing ratio solution of the reparameterized problem:
min min PER" XES.
(0x
where S. & {X: X' E D, ratio (X) =r = MM} and ratio(X)) means that the data points in X' satisfies the mixing ratio r E N from n data domains and ||||] =
Proof. Theorem. 3.1 can be proven in two steps. First, we restate the theoretical results from (Chen et al. 2024b) in Lemma B.1 This Lemma reparameterizes any optimization problem (ming f()))
(while retaining the solution set exactly) under some regular assumptions:
Lemma B.1. Let E T&d and R Also, consider well-defined functions f over Rdd - T2 and over Rdd - Raa Then x" is solution of arg min, f(x)) if and only if g(x*) is solution of the
second optimization problem over domain {y 3x, (xx y)
min min y
f(x)
s.t.
g(x) =
17

---

Published as a conference paper at ICLR 2026
The proof of Lemma B.1 can be found in Lemma C.I of (Chen et al. 2024b). Next, we show that the objective function of problem 3 introoduced in our optimization problem satisfies these assumptions, allowing us to apply the Lemma B.1 directly.
In our setting, set X, f(x) - Leval (0x) and g(x)) _ (at)o(())). We can see that both functions are well-defined, where for any chosen input X' there certainly exists an observed evaluation task loss Leval (x)x and mixing ratio ratio(X²). Lastly, by setting _ r. our optimization problem in problem is of the identical form of the optimization problem shown in Lemma B.1 Therefore, reparameterization process is valid.
B.2 PROOF OF THEORRM
F be k data mixture samples drawn from S, using IF-weighted sampling. Furthermore, assume each independent sample Levat(022 ) follows the shifted truncated exponential distribution yi. + exp, for 1.2..... where exp, (A, c) is a truncated exponen- tial distribution governed by rate parameter À and truncated at >> Then, the IF-driven estimator yr defined in Eq. is random variable: yr where yr the true inner problem solution of Eq. and e is a random noise variable with probability density function (PDF):
Theorem 3.2. Let {X(F.. x
Ake Au
e - -
PDF, (u)
--x
1
cO,c).
Proof. Let X2,X....... X5 be k samples randomly drawn from a sampling distribution and Xmin min{ X1, X2,..., Xx)- This scenario mirrors the setting Theorem 3.2 Our goal to derive the distribution of min and show that it is exactly the same as the distribution y shown in the Theorem 3.2
If each random sample X; (X,, first use commonly known result (Chen et al. 2024b)
that the CDF of any truncated distribution on [0,c] is ([w)-F(0) where the CDF the original distribution. Also, we note that for the untruncated exponential distribution, F(u) = Av
F(c)-F(0)
Hence, The CDF of Xmin is
cdf (Xai)) (u) = - P(Xmin 2 u)
= 1 - P(X, 2 u, 22 2 u. Xp 2
1 1- 1-e k
0 u S
-
and the PDF of Xmin can be computed
②
PDF PDF((..)) (u)
F (1)
Ou
Ake Au (c- -c-du-e-dey k-1 k-1
-- (1-6 1- 1
0susc.
In the original Theorem, each sample X, follows the shifted truncated exponential distribution 3r2 ex, c) where y is constant. Hence, see that estimator has the distribution of 0"" Xmii where Xmin has the PDF above. Hence, the Theorem proven by setting the random variable X
B.3 PROOF OF THEOREM 4.1
Theorem 4.1. Let be the outer problem objective defined in Sec. 3.3 with bounded RKHS norm:
|| VJ, f' Also, let our IF-vveeen estimator for the inner problem solution be governed by the
18

---

Published as a conference paper at ICLR 2026
2
error distribution introduced in Theorem 3.2 with constant e. and X = 1. Let Ac.k
(1-e-0)
Then, running DUET over using the LCB acquisition function found in (Chowdhury Gopalan 2017) at each BO iteration
where k is fixed predecided sampling size.
1,...y : T yields the following attained average regret (Chn aL. 2024b) upper bound with probability at least 1 8:
Rs
6 6(V7 + Vk)
V2Ack
lim T
2Ac
Vök
48
Proof. We provide the proof of the sub-linear Ry growth of DUET in Theorem 4.1 by establishing upper bounds (x) - f(x)] and 6 separately at each BO iteration and use the independence rule to bound their sum. To do so, we introduce the following two Lemmas.
Our first Lemma taken from from known literature on Kemeliied Bandits (Chowdhury & Gopalan 2017 and provides the upper bound on difference between f f(xx) and pz(z)) at each BO iteration I.
Lemma B.2. Let ||]| VL,fi& S B. Also, assume that the observation noise associated with each BO iteration is R-sub- Gaussian with R > 0. Then with probability at least 1 - 8, the following holds for BO iteration T:
(m((x) - f(x)] & B (B+ R/2(( +1+ (n(11))) (x)
where "/t is the maximum information gain after t observations and [r(x)), (z))) are mean and variance of posteror distribution GP defined in Equation | with X 1 2/T.
Our second Lemma attempts to bound the expectation and variance of ç, the non-negative observation noise (in our case, it corresponds to the estimation error involved in solving the inner problem) at each BO iteration t. These expectation and variance will be used later to bound our cumulative regret.
Lemma B.3. Let each observation noise E, of BO iteration follow the same probability dis- tribution defined in Theorem 3.2 with sampling size k probability density function fc, (u)
Ake. Xu Au Ae
k-1
with 0 < c 1 and E 10, c). then EE(e)) S 20 201( -)))
1-6-
A0
(1-e-0y
and Var(c) < E(e.).
Proof. For = 1, we have that fc, (u) Then, the expectation:
ke-u
( (e-
k-1
with < c
and [0.c]
T
T-c
19

---

Published as a conference paper at ICLR 2026
B(a)) J ufe, (u) du
uhe
e - k-1
k-l
du
1 E
1 1- c
ue "= (e 00 k-1
du
k
(e e -- k-1 du
1-
k
((1 (112)) 2 - - 8-1
du
9)
(3)
k
(u-2(1 ())((1 - 12 -
(2(1-
(k 1)u + u)
k(k + 1)
u==
(4)
1
(c- - -
(2
+ 4(1 g
(1---9e
++
(8)
4(1 - -e g3tt1
2kc 2kc (
(11- - g)
2((1 - e - 1) -
(k + 1)(1 c-
(k 1) e c
(k 1)(1 - -
(0) 6 20 -6 -9)a-1
$
+
k
(1 - e ec
(8)
(1)
(2)
where S makes of the fact that ~Au for [0,c] with e 0, I uses the inequality
11-33 for c]. and e < 1, uses the fact that e> 1,
(3)
(4) derived by solving
the definite integral by parts and substitution and simplifies the upper bound with algebraic manipulation.
Next, the upper bound of the variance of can be derived by
Var(e:)
u² Sa, Se,(u) du
J0
ufe, (u) du
(9)
(2)
/ (u) du
B(ee)
(2) where S makes use of the fact that e, lies in [0, cj and v makes use of the fact that < c.S. S 1. This
(1)
completes the proof the bounds on E(G,) and Var(c).
Next, observe that 2, at each BO iteration t is chosen via the IGP-LCB acquisition function (i.e.. 21, arg min, (x) 8,or-1 (a)) and B, B R / + 1 + In(1/61) where the observation noise associated with each BO iteration is R-sub Gaussian). Thus, we can see that at each iteration i. 1, we have (x;) + B.o (xx) > (x*) 3. (x*). It then follows
20

---

Published as a conference paper at ICLR 2026
that for all t 2 1 and with probability at least 1 - ò1.
(1)
(5(x)) - [(x"))
- (xx) 14-1(2²) - 1x0")((x"))
1 f(x)) (x)) + 3,00-1
(10)
1 -10 (xr) + 14 - f(x))
- 23,00-1(11)
Therefore, by setting 81 = 62 = Vo, it follows that with probability 1 - 6 (this follows by rule of independence applied the upper bound of events Σ fxx,) f((")] and > G)) that our attained cumulative regret can be bounded as
Rr Σ
[i f(x))1
Σ
(1xt) -s(x)) + +
T
T
(s(z)) - s(x))1 +
Σ C7
1=1
T
T
2,T
Tt O0-1(11)
Σ
E1
(=1
t=1
B + BV 2 + 1++
(2:)
(1I)
(4)
T
Var(e,)
2 2
B + RV 2(7T + 1+
0 (1:1 +
B(e))
+
82
1=1
T
T
Var(e:)
(6)
B + RV 2(7T + 1 In(1v/8)
O OlV TTT)
Σ
Blee)
82
0 (000( (VT(B + +
+
Σ
B(ee)
Var(e))
52
1=1
- (TT(BVTT
T
Var(c)
E(et)
4
82
where we have followed the attained cumulative regret proof in (Chen et al. 2024b) closely and used the following facts:
uses the fact that e, is non-negative in our problem setting (Theorem. 3.2
is derived from Eq. equation 10
uses the definition of Br in IGP-LCB acquisition function (Chowdhury & Gopalan 2017
w.r.t. 61
V8
uses Chebyshev's 's inequality over 6, with probability at least 1 - 82.
T -=) - (Ix) < O(v (Trr as shown in Lemma by Chowdhury & Gopalan
uses
Chowdhury & Gopalan 2017).
21

---

Published as a conference paper at ICLR 2026
(6) uses the fact that is bounded on 0..c and all bounded random variables are R-sub-
Gaussian with Arbel al. 2019
Var(e Next, we need to derive the upper bound of Σx= B(ee) + + by using the upper bound of the expectation and variance of E, proven Lemma B.3
w.r.t. T. This can be done
d2
T
Var(er)
(1)
T
6 +
2c 2c (1----)- 5)h-1
6
+ 2c e- 2)
B(ex)
82
k
(1 - -
Özk
82(1 --
67
2T ((1 - - - 2)(=11
6 Ózk
2c ((1 - -6- - 2 2)--1
TT
82(1 00
(12)
where S uses Lemma B.3 directly.
Then, it follows from Eq. 11 and 12 that with probability 1 1-d and 82 V5, the attained cumulative
regret Ry at iteration T is upper bounded by:
( (VI(BNTT+ 2xz) )
6T (1 g)--1
6
2c (1--)--) 5)*
Rr so
+T
+
c)
8zk
82(1 c-ch
(13)
Finally we set Ac,k = -1-- 8) g)k-1 (1-e-c)k
As + co, with probability 1- -
and 82 = Vo, the attained
average regret converges to:
Rr
(1)
2((1 - -6-9 - -)k-11
6
2((1 - -e- 5)
lim T++00
+
T
(1---ojk
8zk
d2(1 c-
(2)
2Ack
+ 2A. +
(14)
h
V ôzh
82
66 + Vk)
/2Ac,
+ 2Ac
Vok
82
(1) where & divides Eq. 13 by T throughout, eliminating the O expression and uses the subsitition of A and triangle inequality. This completes proof for the attained average regret in Theorem 4.1
B.4 EXTENDING THEORETICAL ANALYSIS BASED ON DIFFERENT DATA SELECTION METHODS
Readers might be interested in how different data selection methods used to create different estimators affect our theoretical analysis. Here, we provide details on how one could replicate our paper's theoretical analysis to different estimators.
Step 1. Establish the sampling distribution of Levai (fx)) Using a particular data selection method, one obtains & data mixture samples (XI.... } (in our paper, these samples are obtained via weighted sampling based on each data point's IF scores). Then, one trains an I.LM for each data mix- ture and obtaín the evaluation task loss for each resulting I.I.,, yielding/ [Ceval(0x ).... Leval Ox, )- From this set, one empirically derive the sampling distribution of cach sample Leval (0x,) ). In Theorem 3.2] we assumed that each sample (oxx)(xx)) follows the truncated exponential distribution. However, different data selection methods would certainly lead to different empirical sampling distributions.
Step 2. Derive an estimator's empirical distribution. Next, we need to theoretically derive the 1st-order statistic (Amold et al.) 2008) of the empirical sampling distribution from Step 1, since we use the Ist-order statistic as estimator. The procedure to do is shown in App. B.2 and uses
22

---

Published as a conference paper at ICLR 2026
fairly standard procedure to derive the distribution of order statisties. For subsequent analysis to be tractable, the PDF of the Ist-order statistic should have a closed form (hence, a simpler sampling distribution in Step 1 is preferred). More importantly, the estimator's 's empirical distribution should be R-sub-gaussian for a fixed R >> O. This is because for the regret-analysis proof in Eq. [ to hold true, the observation noise in the BO process should be R-sub-Gaaussial Fortunately, large family of random distributions, including our IF-driven estimator introduced in this paper, are all (e.g., exponential family, all bounded random variables).
Step 3. Derive the upper bound of estimator's expectation and variance. Next, we derive the upper bound of the Ist-order statistic's 's expectation and variance as shown in Lemma. B.3
Step 4. Derive attainable cumulative regret. Lastly, we analyze the convergence rate of our algorithm using the growth of attained cumulative regret (Chen et al. 2024b) it = - IM7, f(r=)) = Σ [f((") + Et -(r)) for T BO iterations. Since the error term Et has the same expectation and variance of our estimator, we can use the results from Step 3 to derive our regret bound (as shown in Eq. 11.
C ADDITIONAL EXPERIMENTAL RESULTS AND DISCUSSIONS
C.I ADDITIONAL. DETAILS ON EXPERIMENTAL. SETUP
In this section, provide additional details in our experiments for ease of reproduceability. Through- out our experiments, we used the SE kernel with lengthscale parameters learned from historical observations via maximum -ikel-iiiolooo (Williams & Rasmussen 2006). In our LCB acquisition function (Greenhill et al. 2020), we set B, = 0.5 (see Alg. throughout our experiments. Fur- thermore, we need to perform constrained BO (Gardner al. 2014) in our experiments because the inputs to our optimization problem is a data mixing ratio r whose sum of entries is constrained to 1. BoTorch allows us to implement such constraints (bot orch. wor.chorg//eee.....oo docs/ constraint s) casily. All evaluation for language tasks is done on llm-harness (Gao et al. 2024) with default 3-shot settings with no tfain-of-thougee or special prompting techniques. Hence, it is possible some of our paper's s results differ from those reported in other papers (due to different prompting and inference settings). However, our paper's emphasis on improving the LLM's 's performance with a few rounds refinement on the training data mixture. Hence, we expect DUET to work well even in other inference settings. We treat the evaluation results on llm-harness as the feedback observed in our problem setting. For methods (e.g.. uniform mixture, LESS, DoReMi) which do not use feedback, we repeat them 10 times to ensure fairness in comparison with DUET and show the best LLM performance in results.
To train the LLM, we used a LoRA (Hu et al. 2021) rank of 128 and the Adam optimizer (Kingma & Ba 2017) with initial learning rate of le5. The specific hyperparameters surrounding our optimizers can be found in our code, which we have released. Each iteration of model fine-tuning is done in 1 epoch L40 Nvidia GPU and takes approximately an hour. So, performing 10 BO iterations take 10 hours to run. In reality, the optimization process (all 10 iterations) will be carried out over a long span of time (e.g.. weeks, or even months) as part of the LLM deployment life-cycle. So this is a reasonable amount of compute time.
IF computation. To derive the IF scores of our training data, we remove 10% of the training data from each data domain and treat it as the validation set. Then, we fine-tune a separate LLM for each data domain (using the same setting as above and the same model type as that in our experiments). before deriving the IF score of every data point from each data domain based on the converged LLM and the validation dataset. Using 4 Nvidia I.40 GPUS, we were able to compute the IF scores of TriviaQA (containing around 170k data points) in around 2-3 hours with the torch- influence library (https: : h/gitthui com/ alst
orrch- i in infune))). Smaller datasets required even shorter computation time. Certainly, these runtimes are reasonable in practical settings, since we only need to compute the IF scores once and store them before running DUET.
Other data selection methods. We can also other data selection methods in DUET's inner loop. For instance, LESS (Xia et al. 2024) and TracIn (Pruthi et al., (020) uses around 4-5 hours to build a data-gradient store. On the other hand, diversity-driven selection techniques (Wang al. 2024h)
23

---

Published as a conference paper at ICLR 2026
usually computationally expensive, taking more than 30 hours to select the top 10000 data points.
C.2 COMPARISON WITH OTHER BASELINES
One alternative baseline is simply fine-tune the LLM training dataset for multiple more training tokens (and epoch) and compare with DUET. In [ we fine-tuned Llama: ---Insttttt ruct for more epochs and training tokens on each training domain and evaluated on our evaluation task. The results show that DUET-IF attains better results because it can exploit the feedback from the task.
Table 1: Performance of models trained on different datasets across evaluation tasks.
Train / Eval. +
TruthfulQA
gsm8k
PubmedQA-HcadQA Commonsense+ TriviaQ
Wikitext
42.8
70.4
40.6
59.9
gsm8k
47.2
86.1
43.3
64.1
Pubmed
43.3
71.5
49.3
58.4
HeadQA
45.0
75.2
50.2
60.0
SciQ
45.6
75.6
44.6
63.4
TruthfulQA
59.0
74.0
43,8
61.0
Hellaswag
46.1
72.1
43.2
60.4
CommonsenseQA
50.1
73.3
47.2
65.8
TriviaQA
51.2
70.1
48.1
66.5
DUET-IF (ours)
59.8
84.2
52.4
69.6
Next, compared DUET (paired with different data selection methods) with a variety of naive baselines. Aioli (Chen et al. 2024a), Multi-Fid (Yen et al. 2025) are two baselines that use domain reweighting and multi-fidelity BO to optimize data mixtures. IF just picks the top M 20000 datapoints with the highest influence scores. Random just selects a random subset data, but we subjected it to more training epochs and .M 50000 data points to ensure equal compute comparison.
Table 2: Performance of other baselines across evaluation tasks.
Other baselines
TruthfulQA
gsm8k
PQA+HQA
Commonsense, TriviaQA
Aioli (Chen al. 2024a
51.1 10.7
76.5
48.8
63.7
Multi-Fid Yen et al. 2025
52.8+0.9
73.9
47.2+0.6
65.2+0.8
ODM (Albalak et 2023a IF only
46.1+1.1
77.3
45.8+1.2
60.1+0.7
50.8
76.9
47.7
57.840.6
Random
49.3
64.3=1.4
41.210.7
57.341.0
Uniform + more training tokens
51.640.8
64.4-1.3
44.5+1.2
59.2
DUET-IF (ours)
59.8
84.2
52.4
69.6
DUET-LESS (ours)
58.7
80.5
50.8
67.641.3
24

---

Published as a conference paper at ICLR 2026
C.3 ADDITIONAL EXPERIMENTAL RESULTS WITH
We repeated our experiments with Qwen2. can optimize data mixtures better than other baselines. This indicates that the effectiveness of DUET is independent of the model choice. Hence, we expect DUET to work well for other models as well.
in Fig. and observe that DUET still
DATUR
(a) TruthfulQA
(b) gsm8kk
(c) PubMedQA, HeadQA (d) Commonsense, Trivia
Figure 9: Results on unseen LLM evaluation task domains over 10 iterations (higher is better) for Qwen2. 7--5---b1e 1oo-Instrucc Underlined evaluation tasks are more difficult because the evaluation task domains are removed from the training data (i.e., OOD).
The subcaption indicates the evaluation domain.
C.4 MIXING RATIO FOUND BY DUET
We also present the mixing ratio found by DUET for our experiments after 10 BO iterations. The column title denotes the evaluation task. When the unseen task is in-domain (TruthfulQA), DUET automatically finds that TruthfulQA is relevant training domain and places more weights on it. For OOD cases, DUET automatically finds relevant training domains as well. For example, even though we do not have gsmök data for training, DUET automatically finds data from wikitext and seiq more relevant in improving the performance of the trained LLM.
Table Mixing ratio found by DUET in our I.1 ama ---8------- 8----b-- Teb-Insttuuce experiments after BO iterations. The column indicates the unseen task domain, in the same setting and order those found in our main experiments (Fig NA indicates the respective domain was not included in the training data.
Domains
TruthfulQA
gsm8k
PubMedQA, HeadQA
CommonsenseQA, TriviaQA
Commonsense
NA
gsm8k
0
NA
0
10
headqa
0
NA
hellaswag
0
28
pubmedqa
0
NA
sciq
2
38
34
22
triviaqa
3
12
19
NA
wikitext
8
41
30
22
TruthfulQA
84
6
18
D EXTENSION AND ADDITIONAL RESULTS ON HILL-PARMMETER FINETUNING OF LLMS
While it is too computationally expensive perform large-scale pre-training from scratch for now, we have performed additional experiments on continual, full-parameter training on 8B and 14B models (in contrast to LoRA fine-tuning used in our paper). We performed these experiments on the same setting (but we had to increase the training epoch to 3 for full- nall-parmmeter training to converge better) as in F(a.4(a) and 4(b) and show the best LLM performance achieved in 10 BO iterations.
25

---

Published as a conference paper at ICLR 2026
Model Method
DUET-IF 64.3 + 0.8
DUET w/o
LESS
Llama-3-8B-Inste Instruct
60.1 1.1
51.7 = 0.7
Qwen3-14B
73.5 - 0.6
70.1 - 0.9
63.2 = 1.0
Table 4: Performance gain on TruthfulQA.
Model Method
DUET-IF
DUET w/o IF
LESS
Llama-3-8B 1amaa------Inntuuue
85.1 0.4
81.0 0.7
74.4 0.4
Qwen3-14B
88.6 + 0.6
84.1 + 0.9
80.2 = 0.8
Table 5: Performance gain on gsm8k (OOD).
The results generally consistent with our paper's finding on LLM LoRA fine-tuning, and DUET with IF performs better than its baselines. We will include this into the revised manuscript. We hope these additional results suggest that our approach equally feasible for full-parameter fine-tuning.
One noteworthy point is that while computing IF scores for larger models is more expensive (i.e., computational cost scales with number of model parameters [1]). practical approach is use smaller surrogate model to compute the IF scores, which reduces the computational time. We used the same set of IF-scores computed from the LoRA parameters here. If computational budget is concern, we can also free to use less expensive data selection methods in DUET's inner loop (as elaborated in Sec. 3.2 of our paper) with some performance-coss tradeoff.
E DISCUSSION ON COMPUTATION AND MEMORY OVERHEAD OF DUET
Here, provide discussion of DUET-IF's computation and memory overhead.
F COMPUTATION OVERHEAD OF BO IN DUET
Let T denote the number of Bayesian Optimization (BO) iterations and n denote the dimension of the data mixture (i.e., the number of training domains). In practice, the dominant cost DUET comes from fine-tuning the LLM T times, since BO requires multiple function evaluations (Frazier 2018). This allows BO to exploit feedback from the unseen evaluation task and avoid brute-force enumeration of all possible mixture ratios.
Beyond LLM fine-tuning, BO incurs additional computational overhead. When using a Gaussian Process (GP) surrogate, the primary cost arises from inverting the T x T kernel matrix when re- estimating GP hyperparameters using maximum likelihood (see Eq. (5.8) in (Williams & Rasmussen 2.06).. This is typically performed using a Cholesky decomposition, which costs ((T)).
Next, optimizing the acquisition function at each iteration typically requires gradient-based opti- mization. For the UCB acquisition function used in our work, computing gradients of the GP mean and standard deviation incurs O(nT + nT operations, where OnT'' comes from differentiat- ing μu( (( O from differentiating a (Fcandidane) and is the number of restarts used in acquisition optimization (see botorch acquisition optimization documentation).
(A) Putting these together, the total BO compute at iteration 7 is:
( O(nT nTT2))) O(T3).
since typically t. Summing from the first iteration yields the O(T3) complexity because Cholesky updates allow reuse of factorizations, avoiding full recomputation at each iteration.
G COMPUTATION OVERHEAD OF IF SCORES IN DUET-IF
When DUET'S is combined with data selection methods such as Influence Functions (IF), additional compute costs arise.
Influence Scores. Given N datapoints and p trainable model parameters (in other paper, p is the number of parameters in the model LoRA), direct computation of influence scores requires
26

---

Published as a conference paper at ICLR 2026
O(N2² p²) in (Koh & Liang 2017)) reduce this to O[Np). Importantly, IF scores only need to be computed once and can be reused across BO iterations.
operations (Koh & Liang
2017). However, stochastic estimation techniques (Sec.
Other Data Selection Methods. Different data selection strategies incur different costs, but these are typically amortized since they are computed once. For example, LESS incurs ( operations, where b is the number of checkpoints used (Xia et al. (024).
Putting (A) and (B) together, the joint compute cost of DUET-IF is:
O + ND).
H MEMORY OVERHEAD
BO alone requires O memory to store the T x kernel matrix. During IF computation, we require (pp) memory to store parameter gradients; the Hessian H need not be stored explicitly, as Hessian-vector products can be computed efficiently using conjugate gradient or stochastic methods (Koh & Liang 2017). After computation, storing the IF scores requires ((N) memory.
Thus, DUET-IF has a total memory overhead of:
O +)-
Additionally, because DUET fine-tunes LLM using different data mixtures across BO iterations, we maintain copy of the best best-performing LoRA adaptor throughout the optimization.
In summary, the computation overhead of DUET-IF is:
O + O(Np),
where (NNp) can be precomputed before optimization and the memory overhead is:
O +p)
MORE DETAILS ON MIXING RATIO FOUND
Domains + Iterations
8
10
DoReMi
commonsense(A
11
11
28
11
80
3
14
gsm8k
11
0
0
10
16
0
90
4
headQA
11
0
6
hellaswag
11
13
0
2
0
0
0
0
3
pubmedqa
11
0
0
8
0
0
6
4
0
0
9
sciq
16
10
14
triviaQA
11
60
17
0
0
3
0
20
wikitext
11
38
11
10
0
0
8
0
22
truthfulQA
11
100
62
12
50
84
18
LLM Performance
47
52
53
54
48
57
51
58
60(*)
40
51
In the table below (for clarity reasons, the numbers are rounded), we show the data mixing ratio found by DUET-IF at each iteration as compared that found by DoReMi for the in-domain TruthfulQA task in Fig. 4(a) (to bridge the discussion point we raised above). We also show the data mixing ratio found by DoReMi.
We used uniform data mixture (~ 11% of data to each data domain) the initial data mixture for BO. By chance (since BO with confidence-basee acquisition function tends to explore boundary inputs initially), it finds that placing more emphasis on the TruthfulQA data domain yields better LLM performance. After some adjustments, in the 9th iteration, the best performing data mixture was found. In fact, in some of our exploratory process, found that if increased the number of iterations beyond 10, we can find even better training data mixtures, but the gain in performance typically plateaus (as in many BO applications).
27

---

Published as a conference paper at ICLR 2026
Comparison to DoReMi: We can that DoReMi adopts distributionally robust approach and allocates mixture weights more uniformly across different domains (since cannot exploit the task feedback to infer that truthfulQA data is more relevant). This is clearly suboptimal because it is not optimized specifically towards the evaluation task. and hence its data mixture does not perform as well as DUET-IF, as shown in our experiments.
28


Reviews:
[{"review_id": "Drlr8FiJyv", "reviewer": "Reviewer_9jKt", "summary": "The paper proposes DUET, a data selection method for optimizing the data mixture used in language model fine-tuning. DUET leverages feedback from downstream evaluation tasks to guide its selection process. The method combines Bayesian optimization with influence function to identify the optimal data composition. The authors validate DUET's effectiveness through comprehensive empirical evidence and theoretical analysis.", "questions": "Please see weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. Optimizing the data mixture with \"evaluation aware\"  for language model fine-tuning is a practical and important field, primarily because the ideal data distribution is rarely known and language models are intrinsically multitask learners.\n2. The authors propose a novel method to find an optimal data mixture, using Bayesian optimization for the outer optimization loop and influence functions for the inner optimization.\n3. The authors support their approach with sound theoretical analysis and comprehensive experimental results.", "weaknesses": "1. The method is computationally intensive, requiring multiple separate LLM fine-tuning runs to gather feedback for the Bayesian Optimization loop.\n2. The reliance on IF is questionable, as some of works point out IF for LLM is not very reliable [1]. \n2. The experiments are confined to fine-tuning. The approach's feasibility for large-scale pre-training, especially for computationally heavy estimators like IF, is not demonstrated. \n\n[1] Basu, Samyadeep, Philip Pope, and Soheil Feizi. \"Influence functions in deep learning are fragile.\" arXiv preprint arXiv:2006.14651 (2020).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fcqNlvnNqa", "reviewer": "Reviewer_tBBS", "summary": "This paper proposes DUET, a novel global-to-local data mixture optimization algorithm for LLM finetuning, especially when the target evaluation task is unseen and with noisy performance feedback. Theoretically, the authors applies Bayesian Optimization (BO) for global (inter-domain) mixture rate search, combined with a local (inner-domain) sample-level search by influence function based methods. Empirically, the proposed method demonstrates an improvements on the performance on in-domain and out-of-domain tasks, compared to several existing baselines.", "questions": "1. can you provide an analysis on computation and memory overhead caused by the BO and BO+IF methods?\n\n2. How does the BO method scales by the number of data domain $k$? \n\n3. To compute the influence scores (IF), do you require the validation set from the target distribution? if so, how do you ensure the IF scores are reliable under the unseen and noisy evaluation environment? Also, the data may not be available if the evaluation system is a black-box.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper targets at a critical and challenging problem of data mixture optimization, under a practical and real-world scenario, when the target task is unseen and with noisy feedbacks.\n2. The paper proposes a bayesian optimization based method for data mixture search, which serves as a reliable estimation with coarse feedbacks from the target, black-box-like tasks.\n3. The proposed DUET method obtains a significant performance gain, outperforming several existing baselines.", "weaknesses": "1. The computational cost and overhead have not been discussed. Both the retraining involved in the BO process and the influence function computation can be computational intensive, especially when the model gets scaled-up (e.g. 70B+). can you provide an analysis on computation and memory overhead caused by the BO and BO+IF methods?\n\n2. To compute the influence scores (IF), do you require the validation set from the target distribution? if so, how do you ensure the IF scores are reliable under the unseen and noisy evaluation environment? Also, the data may not be available if the evaluation system is a black-box.\n\n3. The effectiveness of BO depends on how well the Gaussian Process models the objective function.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gBD4QAh2XC", "reviewer": "Reviewer_6k1g", "summary": "This paper presents DUET, a method that combines data mixture optimization and data selection, for LLM fine tuning. The authors propose a two stage optimization, in which the inner problem takes a mixture ratio as input and samples data satisfying the mixture ratio via data selection, and the outer problem solves the mixture ratio taking the optimization of the inner problem as a blackbox. In particular, the outer problem is solved by Bayesian optimization.\n\nThe authors conduct extensive experiments to demonstrate the effectiveness of the proposed method.", "questions": "1. The authors highlight that the experiments are done with sampling size of k = 1 in each BO iteration. However, the results seem too good to be true, given that the optimization is done over a degree-8 simplex. Note that almost all experiments in Figure 4 show >10-point improvements within 10 total samples. Moreover, in 4(a), there are >10-point improvements even with only 1 sample, and if the reviewer understands correctly how Bayesian optimization works, with only 1 sample, Bayesian optimization cannot provide useful suggestions. The reviewer may be missing something here. Could the authors help clarify?\n\n2. Related to 1 above, could the authors provide the mixture found by the proposed methods for each iteration?\n\n3. It seems that this paper mainly applies Bayesian optimization to the data mixture optimization problem (along with data selection), which limits the technical contributions from this paper. Are there technical difficulties the authors could highlight? In particular, are there modifications needed for Bayesian optimization in order to address challenges from \"noisy feedback from unseen evaluation tasks\"?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-written, clear, and easy to follow. The proposed method is elegant and easy to implement.\n\n2. The empirical results strongly support the effectiveness of the proposed method.", "weaknesses": "1. The proposed method is a simple combination of data selection and Bayesian optimization.\n\n2. The reviewers have some questions about the empirical results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zWydVZ7ZQL", "reviewer": "Reviewer_QVFn", "summary": "Optimizing the deployment of the model without an instance-wise loss function but only at a model-wise loss function is an important problem. The paper formalizes the setting, focuses on optimizing only data selection, and shows that this problem can be divided into two steps: optimizing a data mixture ratio and optimizing a dataset meeting this ratio. The author introduces DUET to use BO for the first optimization problem and IF-driven estimators for the second problem. Cumulative regret results for DUET is provided. DUET performs better than other baselines on LLM benchmarks with small LMs (<=8B).", "questions": "- Can we extend the set of optimization variables to be more than just the data mixing ratio (e.g., optimizing a small set of parameters of the LLM) or is the setting too noisy to reliably do so? I don't see an explicit dependence on $n$, the dimension of $r$ in Theorem 4.1.\n- Can we get Theorem 4.1 without an IF-driven estimator? If so, having that somewhere would help showing how IF helps DUET theoretically?\n- Does the unseen evaluation task setting mean that you train on all datasets except for that task? Also, is it true the evaluation is just one feedback (good/bad model) at the model level without access to the exact evaluation loss? These details should be clearer. You should not point to LLM-harness and be done with it.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The problem formulation is clear and the writing is good.\n- The introduction of the two components of DUET to solve the problem is very natural.\n- Theoretical results on cumulative regrets are provided.", "weaknesses": "- The theoretical results don't seem very novel and more like applying previous results in the literature in a new setting.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)