Constrained Best Arm Identification
Authors: Tyron Darnell Lardy
Christina Katsimerou
Wouter M Koolen
Venue: NeurIPS 2025 poster ()
Keywords: best arm identification
cost constraints
track and stop
Primary area: reinforcement_learning
TL;DR: We identify the arm of maximum expected reward subject to expected cost constraints.

Abstract: In real-world decision-making problems, one needs to pick among multiple policies the one that performs best while respecting economic constraints. This motivates the problem of constrained best-arm identification for bandit problems where every arm is a joint distribution of reward and cost. We investigate the general case where reward and cost are dependent. The goal is to accurately identify the arm with the highest mean reward among all arms whose mean cost is below a given threshold. We prove information-theoretic lower bounds on the sample complexity for three models: Gaussian with fixed covariance, Gaussian with unknown covariance, and non-parametric distributions of rectangular support. We propose a combination of a sampling and a stopping rule that correctly identifies the constrained best arm and matches the optimal sample complexities for each of the three models. Simulations demonstrate the performance of our algorithms.

Constrained Best Arm Identification
Tyron Lardy CWT and Leiden University
Christina Katsimerou
Booking.com
christing inastinaaaae htt.ria....taiiemmooooooooioigo. com
Wouter M. Koolen
CWI and University of Twente
wmkoolen@cwi.co nl
Abstract
In real-world maaiiinnnmmainggggg problems, one needs to pick among multiple policies the one that performs best while respecting economic constraints. This motivates the problem of constrained best-arm identification for bandit problems where every arm is a joint distribution of reward and cost. We investigate the general case where reward and cost are dependent. The goal is to accurately identify the arm with the highest mean reward among all arms whose mean cost is below a given threshold. We prove
cenmmatiioonthoretti lower bounds the sample complexity for three models: Gaussian with fixed covariance, Gaussian with unknown covariance, and non con-paraettric distributions of rectangular support. We propose combination of a sampling and stopping rule that correctly identifies the constrained best arm and matches the optimal sample complexities for each of the three models. Simulations demonstrate the performance of our algorithms.
Introduction
In real-world decision-making systems, identifying the best policy is rarely a matter of optimizing single performance metric isolation. Effective policies often involve inherent trade-offss: they achieve desirable outcomes but incur costs. For instance, online platforms routinely deploy pro- motions drive customer engagement [Zhao and Harinen] 2019 Zhang al., 2024). To ensure sustainability, such incentives must not only be effective but also economically viable -ttoicallll captured through feasibility constraints on budget or return on investment (ROI) [Goldenberg et al.) 2020]. Similarly, bidding policies in online advertising need to drive traffic or purchases at an acceptable incremental ROI [Chen and Au 2022]. healtheare, a treatment needs to improve health outcomes under safety constraints. In all these examples, the goal is not simply to maximize the average benefit, but do so subject feasibility mosstrints-ecooome
operational, or ethical.
This motivates a constrained exploration problem: given finite set of policies (arms), confidently identify the best policy with respect to quality metric, among those that satisfy feasibility constraint on a cost metric, such as minimum return on investment (ROI), a risk threshold, or fairness requirement. That is, each arm associated with an unknown, joint distribution on reward and cost, which may be arbitrarily dependent. During the exploration. the decision- an arm in each round and observes real-valued reward and cost, sampled from that arm. Once there is enough evidence that one arm has the highest quality amongst all feasible arms, or that no arm feasible, the exploration stops and returns the best feasible arm or no arm respectively. This setting generalizes the classical best arm identification (BAI) problem in multi-armed bandits, where arms are judged only by their quality [Garivier and Kaufmann| 2016), and introduces an additional challenge not captured by standard BAI frameworks: coupled reward and cost metrics: business KPIs are rarely independent-more aggressive policies often yield higher rewards but also incur
chooses
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

greater costs. This interdependence violates the independence assumptions common in prior work on constrained bandits. In addition, reward and cost distributions are typically not parametric, especially in monetary applications.
In this paper, we provide general strategy for the constrained BAI problem in the fixed-confidence setting. That is, we consider algorithms that are d-correct, i.e., that output the correct answer (either the best feasible arm or no arm) with probability least 1 ô for some fixed confidence level 6. The goal is to minimize the expected number of samples needed by the algorithm, while guaranteeing 5-correctness.
Many variants of constrained BAI have previously been studied. Faizal and Nair [2022 and Yang et al.
|025 studied the same problem in the fixed budget setting, and Kone et al. [025]] the
Pareto set identification problem. An alternative way to incorporate costs in the BAT problem is the multi-fidelity formulation, in which known costs are tied to the desired accuracy level (Poiani
ett al. [2024], or by minimising the overall cost while still maximising a single reward dimension
[Kanarios et al. 2024]. [Wang et al. [2022] consider BAI with safety constraints with separate quality
and feasibility dimensions. However, they assume independence and a linear or monotonic relation
between the cost and reward. Furthermore, the constraints are required be satisfied throughout the
exploration, whereas we focus on pure exploration. David el al. 20188) and Hou et al. [022] optimise a single performance metric, but with a constraint on some risk measure of the returned arm, such as its variance or a given quantile. Hu and Hu (2024) study problem that is close to our setting. Each arm is associated with multiple performance metrics and their goal is to find the arm with the highest mean for a given metric, while certain quantile of the others remains below threshold. They assume only one of the metrics is sampled per rounds and that the different metries are independent. Finally. Katz-Samuels and Scott [2019) propose a modification of the LUCB algorithm to solve the constrained BAI problem with multidimensional constraints. While they are the only one among the above to allow dependence, they focus purely on sub-Gaussian distributions, which is not always realistic for business purposes. Overall, most prior works ignore the dependence between reward and cost and/or focus on Gab-sass settings. To the best of our knowledge, none consider both dependence and arbitrary models, as we do here.
1.1 Contributions
Building on techniques from standard BAI [Garivier and Kaufmann 2016 Degenne and Koolen 2019). we derive instance-dependent lower bounds on the sample complexity for generic bandit models. The main difficulty therein lies in the fact that case distinctions arise from possible tradeoffs between cost and reward that do not occur when only considering the reward dimension. As is well-known in the BAI literature, the sample complexity lower bound also gives rise the proportion of samples that an optimal algorithm should allocate to each arm. We show that these weights and the lower bound can be computed whenever we have numerical access to two transportation functions. In contrast to existing frameworks for BAI, which assume either exponential families (Garivier and Kaufmann 2016) or nonparametric distributions [Agrawal et al.. 2020). this allows us to treat all models in unified framework.
We show that the transportation functions can be efficiently computed for three bivariate arm models: Gaussian with fixed 2 x 2 covariance matrix, Gaussian with unknown 2 x 2 covariance matrix, and non- non-parametric distributions of rectangular support. Our proposed algorithm then uses a plugin strategy to track these weights. As the stopping rule, we use a similar to c.g. Garivier and Kaufmann [2016), Degenne and Koolen [2019]. For proving d-correctness in the case of Gaussian with fixed covariance and non con-paraeetric distributions of rectangular support. we import known results on the concentration for weighted sums of such statistics (Agrawal et al., 2021, Kaufmann and Koolen| 2021]. For the case of Gaussian with unknown covariance matrix, we prove a concentration result as proof of concept.
statistic,
2 Sample complexity lower bounds
To set the stage, let M be a set of bivariate distributions on RR... For any v E M, we denote its mean reward and cost by m(v)= = (ml (V), mz)))). We consider K-armed bandit v (VI,.... uK) E M At every time n = 1, 2,.... drawn from PI,, where R., represents the obtained reward and Cn the incurred cost. Given
.. one arm I, E [K] is chosen and a pair X,, = (R,, C,)) is

---

threshold E R. the objective is to identify the best feasible arm ("()) = 1" ((m(v)),, where (m(m) = arg max m,,1- Here, the arg max over the empty set is defined to be None, and we assume that the bandit v has unique best feasible arm i" (2) A = [K] U (None). To avoid confusion, will crucial to distinguish between arms [K] and answers A, especially because for our problem these nearly coincide. Algorithm design must resolve the type conversion: which arm
must be pulled increase evidence in favor of a given answer?
Applying the generic lower bound of Garivier and Kaufmann (2016) results in the following lower bound on the sample complexity of any ò-correct algorithm for constrained BAI (CBAI).
Theorem 2.1 (Garivier and Kaufmann (2166). Let ò (0, 1). For any è-correct strategy with stopping time T5 and any bandit model V E M
( Ey[Tal T (2) KL KL
where KL denotes the Kailbacc--eiibler divergence and
("))- = max WEAK
min V'EM (1))
Σ wk KL (1x1l
(I)
Their proof reveals that any strategy that matches the lower bound will also match w" (v) as proportion of arm draws, where w" (w) are the weights that achieve (I) To find strategy with optimal sample complexity, we will compute the characteristic time (v)) and corresponding oracle weights. To this end, first present abstraction, show how it still allows efficient computation, and then implement the abstraction for the following three models:
1. Gaussian with fixed covariance = 0: Ma,. (N, 2) 4 E RR).
2. Gaussian with unknown covariance: Ma = (W, E R,, 0)
3. Non-parametric distributions on the unit square: MB { P [0, [J.
Other models can be worthwhile, for example modeling cost and reward as independent, each drawn from some single-parameter exponential family member. We focus on the above three models to highlight the role of dependent rewards and costs.
2.1 Solving I generically in terms of transportation cost interface to the model
We analyze the CBAI characteristic time T"(v) from (I), and provide efficient algorithm for computing w" (v) and hence T (v). To start, we introduce a shorthand for the KL projection of an arm E M onto the set of distributions {v' E M: m((') = [4) with given mean μa R2:
KLinf 1t) = min
(Tll)).
VeM
We suppress the dependence on M from the notation. The characteristic time I can be rewritten as
T
max wEAK
min AERRER (A)
Σ wy KLinf(rk. Ar).
At this point we see that we need to know about the mean vectors À such that 1" (A) + i*. We have Proposition 2.1. For each answer E A, let -i = E (*) # #i). Then
U 2 Ai,1 and XÚ.2 S 2 (2|1,, 2 + None,
cl(-i) =
U (11A32 Sa) jElK]
= None.
This shows particular that the Pure Exploration Rank [Kaufmann and Koolen 202T Definition 20] of constrained BAI is two, -i is union of parts in which most two constrained. This

---

will be useful in obtaining deviation thresholds below. For now, we use the partition above to simplify the characteristic time. Namely, if (1) * None, the problem of finding the characteristic time reduces to
max min wEAK
min J7
min A (E(i"))
Σ
we (Linf( 2e)
wi- KLinf(ui, Ao-1
(5a)
ER. 2=",227
X SA A
and for (1)) None, it reduces to
max min min wy KLinf A,). wEAx X A3.25
(5b)
We will argue that numerical access to the following two transportation cost functions, ci and cz. suffices to implement this characteristic time T and the corresponding oracle weights w".
Interface 2.1. The following two functions need to be implemented efficiently:
1. the weighted cost for making arm Vj beat arm (here i can be assumed feasible)
ei (V, w) =
min ARR
KL,nf(u;;, >a) + w (Linf Az).
(6)
A S Ag.1 and A3,2 S
We need separate access to both terms of the sum at the minimum, that is, to KLinf A;) and (L, Az). We will denote these by c1,((%,, Vj, w) and ( Vi, w) We will also assume computational access to the limit ei (1s, Vjj 00) = 1om Ci (Vi, Uj, w)
2. the cost for changing the fcasibility status of an arm
min xR 22
KLinf(v, X) (22(v) S
c2(2)
min A KLinf(u, A) if m2(2 > Y Sa
In terms of Interface 2.1 our problem (5) simplifies to
min min wi-c1 pi--, Py, wi: w. 02
if * None,
('))-- = max W
minje[K] ())2(1;)
if = None.
To interpret the revealed structure, note that both cases minimize over precisely K terms; one for each alternative answer different from the correct answer i". For the None case, we find ourselves in 1d thresholding problem Garivier al. [2017), where the cost, c2. is that of discriminating an arm from the threshold y. For the jf None case, the inner minj sub-expression matches that the transport cost for the best arm identification problem Garivier and Kaufmann [016). where the cost to reverse the quality of two arms there is replaced by our 01 (which in addition ensures feasibility of the second arm). The outer binary minimum adds extra case to the range of possibilities to be considered, namely where the best looking arm is rendered infeasible.
We can solve the lower bound problem generically in terms of Interface 2.1
Theorem 2.2. Let v be K-armed bivariate bandit. Let i" 1 (v). For all [K], we have
EK EK (C*)
C.((C))
if + None
Σ (0())))
('(v) =
and wv(v)
E -2
(3(v))
if None
(a(w))
where Wi- (Ò) == 1, and for each sub-optimal j i (y(C) ) is the unique solution to w in C1(Us,, V/. w) Č
(8)
and Ō" is the unique solution for Ö
Σ C1,r (Vi+ wú (C)) jfi
CLSs (Vi. ,, u úT)(
it is attained the interval [o, Emax) and & Čmax otherwise, where we abbreviate Cinax
min(cz(vi-) ), ming43 ex Ci(Ve, Vgj }.

---

Efficient Computation Note that this theorem is not only characterisation, it unlocks generic computational recipe given oracle access oj and c2. To see why, we first observe that the left-hand side of (8) is increasing in starting at 0 when w 0, and reaching cı (Ve, : Vj, 00) for w - co. Hence wy (C' can be computed by binary search. Moreover, the proof reveals that the left-hand-side of (9) is increasing in C. So again, we can solve (9) for C by binary search. All in all, we can compute I and w" using two nested binary searches. This is the same computational cost as the algorithm of [Garivier and Kaufmann 2016, below Theorem 5] for the oracle weights in BAL efficient implementation for constrained BAT is the basis for the Track-and-Stop algorithm template. therefore remains to implement C1 and 0g for each of our three arm models of interest.
2.2 Efficient Implementation of Interface 2.1 for our Three Models for Arms
We implement the interface functions e1 and 02 efficiently for our three arm models of interest. We also implement KTinf and discuss the effect of dependence.
(a)
(b)
(c)
(d)
(e)
Figure 1: (a) shows the bivariate means of example six-armed bandit v, with its feasible best arm i" (w) circled green. The vertical axis is reward, while the horizontal axis is cost, with the vertical dashed line indicating the feasibility threshold y. The four possible types of transports underlying (5a) are illustrated in (b)-(e) We can make the best arm infeasible (b). We can render an arm feasible that was already high (c). We can make feasible arm exceed the best arm (d). And we can make an arm both feasible and better than the best arm (c). In these diagrams the reward-cos. dependence
within each arm manifests by the cheapest transports (indicated by red arrows) not being axis aligned.
2.2.1 Gaussian fixed covariance: KLinf, e₁ and C2
Proposition 2.2. Let v = NNe, Σ) and consider M = -M... Then
1
c2(v) (7 17222
KLinf((,, A) = 2 11 2113-1 and
(10)
2202
Moreover, let = N(4, E) and V = Nu,,, E) with 1" ({pi;H1]) then =(H 22n2 (Mi.1 M
if M1.1
Ena (Mij 7)+ 2 Pi,1
En
if M4.2 + E.1++ (Mi,1 (5,1) S
E12
21+((++]
ei (1,,,,
M1.1 p4+,1
WEn () +
19,2 7
else.
2( E( +2=+
Here, the case distinction in 01 arises by first solving the infimum in (6) while forgetting about one of the constraints at a time. If the resulting minimizer happens to satisfy both constraints, then it must be the solution to the original problem, since its value is at least low that of the original problem. If this does not happen for either constraint, both of them must be active.
The Impact of Dependence on Transportation Cost We are interested in the effect of dependence between reward and cost in all three models. The explicit formulas above allow us to highlight
its effect explicitly. Here, dependence manifests as nonzero covariance Σ12 * O. To see its effect, we observe that the minimum cost to move an arm from mean 4 to a new location À such 2
that d2 y (N Σ)) minxeR" 1 1 21 (7-P2)² which is attained at
Eia (y p2), 7). So we that Eι * 0 causes the arm move diagonally, even x (M1 + Eez
5

---

though the objective was to move in dimension two. This is illustrated in Figure [T. On the other hand, counter intuitively, the cost of that move does not depend on Σ12. These diagonal motions make it subtle to determine the active constraints for the ei motion (where we ask for certain arm to be made feasible and better than another arm). E.g. an arm that starts feasible may rendered infeasible by making it better. As visualized in Figure | the active constraints at the optimal solution can either be feasibility only (c), flipped mean reward only (d) or both (c).
2.2.2 Gaussian unknown covariance: KLinf, 01 and 02
For unknown covariance subtlety arises: of x. As a result, C1 as specified by (6) is not a convex optimization problem. Fortunately, 01 can still be computed efficiently.
below, (Linf A) is not convex function
Proposition 2.3. Let v- NN, Σ) and consider M = =Ma- Then
1 c2(1) - In 1+ (7 n22
KLinf(v, A) 2 In (1 |1 --11--1)) and
(11)
Σ22
Moreover, let vi - Nuss Ex) and Vj NNus, Eg), then, abbreviating ((x) == *In(1 + x),
1f S 7 aand 11j11 2 0
(14,1 0}2 Σ,,11
e
(1122-112
if 3,,0 (1g2 - -7++ 2 0
E,12
E9,22
min DER
+
(0)
if + E,n (Py1 0) S 7
E,S2
Σg.11
KLinf(P,, (0,7))
else.
Notice that the 22 cost is fully determined by characteristics of the second dimension. In particular, it is independent of the dependence Σ12 between the cost and reward dimension. This happens because the optimal move will take the covariance structure into account, which cancels its effect. Furthermore, the variable / that appears in e₁ is introduced as a parameter such that X,, With this extra parameter, the searches over Aa. and λa.2 are straightforward. For the remaining search over 0 E R, it is possible to identify the points at which the active case in the second term switches; this will be instance dependent. For example, if fr,2 S 7 and Σi, < O, the second term will always be active, while for 1x > O, case 3 takes over whenever 0 2 p4, + + E1 1 The optimal value on each segment can be found by setting the derivative of the objective to zero, which is a matter of finding the roots of a cubic. The global minimizer can then efficiently be found by comparing the minimizers on each segment. So computing C1 or C2 takes a constant amount of work.
Xx,2-
2.2.3 Non-parametric supported on [0, 1J:: KLinf, ej and C2
Proposition 2.4. Let v be a distribution on [0, 1]2 and consider M = -M.. Furthermore, let Rx = {(01, a2) E RR2 Vx E 10. a1 A1) + az(22= X2) 2 0}. Then
KLinf(v, A) an X~w
E [ln (1 + ai (Xi A1) + a2 A2))1
(12)
(2(1)= = max E
[ln (1 + a(X2 - 7))]
X
Finally, let vs. vy be distributions on [0, 113. With R2 = (b E R²3 by 2 be, Vx E 10,1]3
1 w(b + b221) 2 and 1 + + b2 by (2² - ?) 2 0}. we have
(1 w)
max BER
E [In(1 - w(bi + b2X1))
[ln(1 + bb b2 b2 7))]
Note that when E (0, 1)2, the region Rx is a compact convex set in R2. In fact, being the intersection of four half-spaces, it is quadrilateral with its four vertices on the axes. Moreover, the objective concave in A. This means that for v a distribution of finite support (e.g. an empirical distribution) we can compute [Linf(y, X) using off-the-shet convex optimisation methods e.g. the ellipsoid method. Similarly, R' for w > 0 is compact convex subset of 3 being an intersection of six half spaces.

---

K The case distinction that we saw for in the Gaussian fixed covariance case did not disappear. In fact, manifests in the region for by the Lagrange multiplier for enforcing feasibility of arm j. and b is the Lagrange multiplier for enforcing the correct order of mean rewards. Hither (but not both) can be zero at optimality if the respective constraint satisfied already.
3 Asymptotically Optimal Algorithm
We now develop an asymptotically optimal algorithm. As this is classic, we defer details to the appendix. Throughout, we denote by NN.n) = =>.=1 1,,=i the number of samples taken from arm i in the first n rounds.
estimate (n) of the bandit after samples. For the fixed covariance case, we let ii(n) == (n), E), where (i,(n) = Ni(n) Σ.= ,,, is the empirical mean of arm i after n bivariate outcomes X, = (R., C.). For the unknown covariance case, we let 2, (n) == N (n,()), (n)), where E, is the empirical covariance of the samples from arm i. Finally, for the non-pate con-aaametric case, we let êg (n) be the empirical distribution of the samples {X,, n, L }} from arm i.
Estimates Our approach is based
Stopping and recommendation rule for = (B(n). by
The Generalised Likelihood Ratio (GLR) statistic is defined,
min { mmnnss Nı(n)a, (I)(n),,g()), M(() (N(()))2(((()))) minje[[] (()))))((,(n))
if / None, if None.
An
(13)
We stop at the first time == in E N: An 2 3(),)) n)} the GLR crosses the exploration threshold given below, and that point we will recommend the empirical best feasible arm i == i (P(n)). We show in Appendix that
Theorem 3.1. The following exploration thresholds result in a 5-correct recommendation
K
In
K 8 +8
once In 1 8
(c, = In
+
4ln
6,
5
K
+ 2lnn 4mn 2ln In K + 2 Inn 4ln
B) In
n)( = In 5 + 2 42n nl))
These thresholds all account for confidence (In })), a union bound across incorrect answers (In K), and a courser (In n) or finer (In In union bound across time. These bounds are conservative in practice; in the experiments we will use In In n instead (and verify that the rate of incorrect recommendations remains below d).
TaS Finally, our sampling rule ensures the asymptotic optimality, We compute a plug-in estimate of the oracle weights w, := w" (i)),, and pick I, = arg min, N2(n- - 1) Σr= Ws,i (C-Tracking). We add forced exploration to keep (nnn) Z Vn. All all, these ingredients guarantee
Theorem 3.2. TaS is asymptotically optimal, i.e. limg
Eo(rol (v).
In
4 Simulations
In this section, we put our algorithm, TaS, to the test on the four bandits depicted in Figure 2 We treat both the
Gaussian model and the bounded model. For the latter, clip the Gaussian arms from Figure to [o. 1 Since there are no off-the-shelf algorithms to compare to, adjust a number of sampling strategies used in the BAI literature to the constrained BAI setting.
For the Gaussian model with unknown covariance, the EV-TaS sampling rule uses the empirical covariance to track the weights as if we were in the fixed covariance model. A similar rule was
7

---

075
reward
noward
90096
0.00
04-
09
0.
-0.2
025
0.50
0.75
1.00
125
525
1.50
0.73 COsT
1.00
122
02
06
32
50
03
0.6
03
Cost
CosT
cost
Easy 10.5, 48.3
Hard 410.4, 1890.0
All feasible 26.5, 122.0
None feasible
29.3, 134.9
36, 24, 21, 10, 9
18, 39, 39,
40, 39, 14, 4,
3, 4. 10, 18, 65
Figure 2: Each diagram illustrates a 5-arm bandit v with Gaussian arms. The dots and ellipses give the mean and one standard deviation ring around it in covariance matrix = [0.1 0.05; 0.05 0.09]. The two numbers and vector are 7 (v), T (v) In & for å = 0.01, and w" (v) in percentages.
previously considered for regular BAI by Pourdan et al. 2023 Note that there is reason believe that this should work well (and it does not), as the sampling proportions will be sub-optimal. However, it is reasonable consider For the bounded case, we introduce the GA-TaS [Ménard 2019), instead of the original TaS, due to the high cost of computing the optimal sampling ratios per round. Instead we use gradient ascent to solve the optimization problem online, and thus more efficiently.
does not know how properly handle the unknown covariance.
TopTwo-TCI sampling rule is based Top Two algorithms in regular BAI, where the arm to sample at each time is randomly chosen between a leader and challenger (Jourdan et al. 2022 We thus need to define what the leader and challenger mean in 8, = i" (#)) will serve as the leader. For the challenger, note that the GLR (A,, as in 13 is defined as minimum of K terms, each corresponding an answer different from i, Let denote each of these terms by An.j. As challenger, we take the answer that minimizes An.j + ( (n)), where we let Nnona (n) N;,, (n). If either the challenger the leader is None, we sample the other deterministically. This setup resembles the best challenger implementation Hu and Hu [2024), with the substantial difference that our implementation regards a constrained mean rather than a quantile, and our selection criterion for the challenger accounts for the dependence between reward and cost dimensions. Oracle samples all arms with the optimal weights for the true model. Racing repeatedly samples uniformly all arms and eliminates an answer (and the corresponding arm if j 3 None) once can be rejected. That is, if None, we eliminate when An. 2 B(n, 8). i.e. feasibility of arm j is implausible. If * None, we eliminate answer when min {An.j- An.Kone) 2 3(n, 8). The second term ensures that it is implausible that infeasible. If j 3f None, the first term addition ensures that it is implausible that j is better than i. We keep sampling until one answer remains. Uniform samples all arms round robin fashion. TaS-Id solves the unconstrained BAI problem, ignoring the cost dimension.
The current best answer
All algorithms use the same GLR rule and the stylized stopping threshold log(1/8) + log log(t). originally used by Garivier and Kaulmann (2016) and heavily adopted in the literature for allowing shorter runtimes while keeping the errors lower than As initialization, we start by pulling each arm 3 times, which is the minimum required for the covariance matrix estimation. We work in the moderate confidence regime of 0.01. All instances were repeated 1000 times, except the hard one, which we ran 500 times. The results are shown in Table for the unknown covariance Gaussian case and Table 2 for the bounded case. All empirical error rates remain below
Table 1: Gaussian unknown covariance: average runtimes with standard errors
Instance
TaS-EV
TaS
Oracle
Uniform
TopTwo-TCI
Racing
TaS-Id
Fasy
79.4+ 0.7
76.3+
0.7
89.6+
0.4
136.4+
0.6
68.5+
0.6
136.74 1.5
96.6+
0.5
Hard
3291.1.1
70.0
3218.7.1
68.6
4225.41
59.4
$498.9.1299.
2850.9.1
54.9
2864.6.1
51.4
4815.91101.6
All feasible
199
2.5
190.4+
2.4
228.74
2.3
354.0+
4.8
174.6+
2.4
271.04
2.6
186.4+
2.3
None Fensible
234.2.1
2.7
222.6.1
2.6
270.1.1
3.1
576.0.1
13.4
241.9.1
5.4
174.6.1
2.2
3293.41
84.4

---

Table 2: Bounded (non-parametric on [0. 1J²): average runtimes with standard errros
Instance
GA-TaS
Oracle
Uniform
TopTwo-TCI
Racing
Easy
98.8. 0.9
104.74 0.9
120.7 1.2
96.44 0.8
114.54 0.9
Hard
539.74 7.6
669.3 6.6
1088.4+13.0
457.3 4.7
933.4+17.8
All feasible
241.0# 3.5
256.7
2.8
409.81
6.1
191.1
2.9
221.7H 2.8
None feasible
69.6
0.7
68.24
0.7
116.9# 1.8
47.5# 0.5
49.0 0.5
30
0.8
25
0.6
20
0.
-1.0
-0.5
0.0
0.5
1.0
-1.0
-0.5
0.0
0.5
1.0
P
Figure 3: Sample complexity and optimal weights as function of dependence p.
5 The Impact of Dependence on the Sample Complexity
In this section we study the impact of dependence on the sample complexity. We study the following
two-arm problem V, in the fixed covariance Gaussian model function of correlation p E [-1,1]] the feasibility threshold is y reward each have variance Σ1 3 Σn = 1. and the correlation between them is Σ p. These parameters were selected to illustrate the three possible regimes we discuss below. Note that
2 the arm means are μl (0, 0) and P2 (-1 1), cost and
arm 1 is feasible while arm 2 is not, so the correct answer is always arm 1. Figure 3 shows the characteristic time (v)) as function of p. It is contrasted with the number of samples needed when sampling uniformly, T Twamt (V,), and when using the optimal sample weights ignoring the dependence, wis (w,) (by assuming p = 0). That is, the sample complexity for when we sample according to one of these rules, but still stop with the 'correct' GLR rule. Any other d-correct stopping rule would be slower.
By inspecting Z we see that the optimal unnormalized weight on the second arm will be chosen to maximize the cost of making arm two better than arm one, as long as that is cheaper than making arm one infeasible. This corresponds to the case that Č* is attained in the interval [o, Cmax] in Theorem 2.2 Notice that the cost of changing the feasibility status is independent of as can be seen in the expression for 02 in Proposition 2.2 The mean reward of arm two moved to - 1o.11-((2,,2 7), which does depend on p. This will result in arm two becoming better than arm one for # Therefore, the cost of making arm two feasible and better equals the cost of H2.3
just making it feasible; see also case of C1 in Proposition 2.2 This results in the flat regime on the left of the dashed line in Figure 3 As p further increases, making arm two feasible and better will involve moving both arms, the cost which does depend on p. as can be seen the third case of ci in Proposition 2.2 At some point, the maximum of this cost over the unnormalized weight becomes larger than the cost of making arm one infcasible. The optimum unnormalized weight will then make the transportation cost equal to the cost of making one infeasible, corresponding the case that Č" -0,-0(1;,) in Theorem
This causes the change in behavior the dotted line.
Finally, it is noteworthy that, in this case, uniform sampling sometimes outperforms the strategy assuming independence. This occurs because, for large values of p. uniform weights more closely resemble the optimal weights than those derived under the assumption of independence.
Discussion
We introduced the constrained best arm problem (CBAI), where each arm hides joint distribution of reward and cost. The goal identify from observations the arm highest mean reward among all arms with mean cost below given threshold, or to report None if all arms are infeasible. This

---

model in particular allows us to study the impact of dependence between cost and reward. We characterized optimal sample complexity, and implemented the resulting optimal algorithms for three classes of arm distributions. We analyzed our algorithms theoretically, and showed that they are asymptotically optimal. Finally, we experimentally investigated the performance of variety of algorithmic templates, including Track-And-Stop Top-Two and Racing, and show that they perform well. We now discuss questions and future directions.
Can one handle multiple constraints? Let us imagine multivariate problem with all-but-one dimensions being costs needing to be below respective thresholds. What would change? We could still have a ej and 02 decomposition, where ej is the cost to make a designated arm feasible and better, while 02 is the cost of toggling the feasibility status of an arm. For all our three models, changing the feasibility status of an arm a solvable problem, even with multiple constraints. We either enforce all of them, or enumerate all constraints to violate, and optimize KLinf. What gets tricky making an arm feasible and better than another one. In the known E and bounded cases, this still results in a convex problem. Yet in the unknown E case, KLinf (1/ A) is not convex in À (but it is quasi-convex) and ej is not even quasi-convex problem in (as it is a sum of quasi-convex objectives). We currently optimize in Proposition 2.3 by locating the optimal 0 by finding roots of a small number of cubics. With multiple constraints, the number of cases the right-most function of f in Proposition 2.3 may equal the (exponential) number of subsets of active constraints.
we propose GLR thresholds that are of the form In & with added correction factor of In In n for the known covariance model and In for the unknown covariance and bounded models. In the simulations, we used a stylized threshold of In In for all models. For known covariance, In In is proven valid using mixture martingale techniques Kaufmann and Koolen [202I]. For Gaussians with unknown covariance, it is also possible to relate the GLR statistic mixture martingale, as noted in the one-dimensional case by Wang and Ramdas [2025]. However, using this to achieve a In In threshold would require a sophisticated argument about the exact nature of this relation, which we leave for future work.
Is the stylized threshold valid? In Theorem 3.1
References
S. Agrawal, S. Juneja, and P. Glynn. Optimal 8-correct best-arm selection for heavy-tailed distri- butions. In Proceedings of the 31st International Conference Algorithmic Learning Theory. volume 117, pages 61-110, Feb. 2020. Corrected version available https: // //tr///axxvvv/// attpaviiiii/aaaaaa//.//../...lat abs/ 1908. 09094
S. Agrawal, W. M. Koolen, and S. Juneja. Optimal best-arm identification methods for tail-risk measures. In Advances in Neural Information Processing Systems (NeurIPS) 34, Dec. 2021.
A. Chen and T. C. Au. Robust causal inference for incremental return on ad spend with randomized paired geo experiments. The Annals of Applied Statistics, 16(1):1-20, 2022.
Y. David, B. Szörényi, M. Ghavamzadeh, S. Mannor, and N. Shimkin. PAC bandits with risk constraints. In ISAIM, 2018
R. Degenne and W. M. Koolen. Pure exploration with multiple correct answers. In Advances in Neural Information Processing Systems (NeurIPS) 32, pages 14591-14600, Dec. 2019.
F. Z. Faizal and J. Nair. Constrained pure exploration multi-armed bandits with a fixed budget. arXiv preprint arXiv:2211.14777 2022.
A. Garivier and E. Kaufmann. Optimal best arm identification with fixed confidence. In Conference on Learning Theory, pages 998-1027. PMIR, 2016
A. Garivier, P. Ménard, and L. Rossi. Thresholding bandit for dose-ranging: The impact of mono- tonicity. arXiv preprint arXiv: 1.04544, 2017.
D. Goldenberg. J. Albert, L. Bernardi, and P. Estevez. Free lunch! Retrospective uplift modeling for dynamic promotions recommendation within ROI constraints. Proceedings of the 14th ACM Conference on Recommender Systems, pages 486-491, 2020.
10

---

D. G. Hartig. The Riesz representation theorem revisited. The American Mathematical Monthly, 90 (4):277-280, 1983.
Y. Hou, V. Y. Tan, and Z. Zhong. Almost optimal variance--constrained best arm identification. IEEE Transactions Information Theory, 69(4))2603-2634 2022.
M. Hu and J. Hu. Multi-task best arm identification with risk constraints. Available at SSRN 52 2024.
M. Jourdan, R. Degenne, D. Baudry, R. de Heide, and E. Kaufmann. Top two algorithms revisited. Advances in Neural Information Processing Systems, 35:26791-26803, 2022.
M. Jourdan, R. Degenne, and Kaufmann. Dealing with unknown variances in best-arm identi- fication. In International Conference on Algorithmic Learning Theory, pages 779.899. PMLR, 2023.
K. Kanarios, Q. Zhang, and L. Ying. Cost aware best arm identification. arXiv preprint arXiv:2402.16710, 2024.
J. Katz-Samueh and C. Scott. Top feasible arm identification. In The 22nd international conference on artificial intelligence and statistics, pages 1593-1601. PMLR, 2019.
E. Kaufmann and W. M. Koolen. Mixture martingales revisited with applications to sequential tests and confidence intervals. Journal of Machine Learning Research, 22(246))1-44, Nov. 2021.
C. Kone, E. Kaufmann, and L. Richert. Constrained Pareto set identification with bandit feedback. arXiv preprint 250.::2500008/2277. 2025.
D. G. Luenberger. Optimization by vector spuce methods. John Wiley & Sons, 1997.
P. Ménard. Gradient ascent for active exploration in bandit problems. 190..06905..8155 2019.
arXiv preprint
M. Naaman. On the tight constant in the multivariate D Wolfowtt inequality. Statistics & Probability Letters, 173:10908; 2021. ISSN 0167-7152.
R. Poiani, R. Degenne, Kaufmann, A. M. Metelli, and M. Restelli. Optimal multi-fidelh best-arm identification. Advances in Neural Information Processing Systems, 37:121882-12192, 2024.
H. Wang and A. Ramdas. Anytime-valid t-tests and confidence sequences for Gaussian means with unknown variance. Sequential Analysis, 44(1):56-110, 2025.
Z. Wang, A. Wagenmaker, and K. Jamieson. Best arm identification with safety constraints. In International Conference Artificial Intelligence and Statistics, pages 91 14-9146. PMLR, 2022.
L. Yang, S. Gao, C. Li, and Y. Wang. Stochastically constrained best arm identification with Thompson sampling. Automatica, 176: 172:3, 2025. ISSN 0005-1098.
J. S. Zhang, B. Howson, P. Savva, and E. Loh. DISCO: An end-to-end bandit framework for person alised discount allocation. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, pages 33-49. Springer, 2024.
Z. Zhao and T. Harinen. Uplift modeling for multiple treatments with cost optimization. In 2019 IEEE International Conference on Data Science and Advanced Analytics (DSAA), pages 422-431, 2019.

---

This can be restructured to A Proofs for Section 2
A.1 Proof of Proposition 2.1
Technically, below we characterize the closure of -i. For the value of optimization problems constrained -i this difference is immaterial, yet it ensures the optimizers are attained.
First, suppose that = None. Any A with (A) # i must have an arm j such that 2 S y. Conversely, any À that has arm j with メ322 S Y has (2) # i. So we find that -i UJE[K] (A)2,22 I 7 for = None.
For + None, any À with (A) * must either have an arm j that is both feasible and better than i, or arm must not be feasible. That is. either there exists j such that A3.1 2 A1.1 and X32 S y or 2 y. Conversely, any À for which either Xj,1 2 Ax, and 2322 S 7 or X1 2 Y must necessarily
have 8" (A) # i. Therefore, -i = Ujfi {AAAg1 2 A1,1 and 2322 S 7) U (a) X 2 7
A.2 Proof of Theorem 2.2
We first handle the case that i" (m) o None. We interested, following (5a) in
max min
min Wi-C1 Vi-; Výj
Wj wi
(2
WEAR
Wi-
max w. E(O.1]
min [g(w.)) wi 02
(14)
where
Wij
g(we.) =
max w_. E(1 w1- w:- AAx-
min C1+01 1/1
Vi+, Vjj We
Now each wy for j + 1* only appears in one term of the minimum. This means that the optimal solution for w-s- is balance all contributions. We hence need to solve the system
Wj
C
Wt C1
Vo
(15a)
1 - W/s
Σ jitt"
Wj
(15b)
for w--- and C, and then the value is C. Note that C is concave function of wr. Resuming from (14), and introducing C' = C/w,-- and Wy(C, w.+ w. wy (C/w;,, w, ū, (C), where (C)
is the solution for úū in
Č = ei (Vr My)
we are left with
minfC, (2
max min (C, (, max S.L. 15
1 Σjjr (C)
1+E (C( and wi I+ 2 1,
where in particular we solved for w.-
()) The objective is
decreasing for Ö > c2(ux=), ). the maximum is between 0 and that. To find it, need cancel (or, for bisection, compute the sign of) the derivative, i.c.
1 Σeer w ú)(() ČE (0(
(1 Ejjr i))2
and that is equivalent to
1 1 Σ (Ow,((0)-15 - ,i)(
371
Differentiating the definition ū iū,(C), we find
1 = & (vr, Un ūú (i,(()) 10((0)
12

---

and it hence remains to solve for
C1 (Ve, Vy, wy) - ú (C)c Mo,, (Mr, My, (3,
KLinf(us, A.- (C))
=
& uc, Mj 1i (C))
KLinl(yj, A₃((1)
This is the same equating F(C)= where F(C) == Ejjr Cui - ú,₁(). For Õ1 > Cz, we see
((C))-F(C = Σ - + üh (i2(())
jp
E -((((()) -2 ((1) (Ōz - -
2
Σ
- ((,( > 0,
j7
where we use a tangent bound on úi together with the fact that w/ is increasing, since új is the inverse of a concave function. It follows that F(C) is increasing, that F(C) can be found through bisection.
If it (m) None, then, following (5).. interested in computing
max min uyC2(Hz). wEAK J
By reasoning similar to the above, the optimal will balance all contributions. It follows that for every arm
c2(v1)-
w = C2(21))
A.3 Proofs for the Gaussian transportation functions
In this section, we provide the proofs of Propositions 2.2 and 2.3 For both, we will need the KI. between two Gaussians. That is, for = (u, Σ) and // N [X E'), we have
Σ
KL (vlle/)
log
+ trEEE-'E) ( (1-
(16)
Σ
furthermore helps to know that, for fixed '₁ the minimum 2-dimensional KL is in fact the 1 1-dimensional KL (and this insight can symmetrically be applied with dimension 1 and 2 exchanged):
(M1 A1
2/ (EI - -1). (17)
min (μ D' (μ A) Aa
achieved at A2 = 42
E1
Σn
We now proceed with the proofs.
A.3.1 Proof of proposition 2.2
For -N E) and M = -Ma we have, by (16).
KLinf(p,A) (wll)))) 1 1110 2112-11
where v/ = =N(A, E). Using 17). we find that this means that
c2(v) (7- 22)²
2Σn2
Furthermore, for v4 NN Σ) and vj NNuj E) with i (1 14;3) i, we have
(1,,, Vy, w)
min ARR
||,, 1.1
2 il 2
N S Aga and Ay, 4
The solution for A, Ay falls in three cases, depending on which of the two constraints are active at the solution.
13

---

1. A1,1 S A3L1 active and 232 S inactive. Using 17 we need to find
(Ni,1 )² (1511 )5,1)²
min A. A,,1
2
2
where we have used
Ei
E1
X12 Mi. En
(14,1 Xx,1)
and
A3.2 Mj.
(1511 -1.1).
En
We then make the means in the first coordinate equal, and get
Ar, A1.1 = s Enp4a + Enpaa
En + En
with value
1 (M4.1 13.1
2 E1 + -111
and optimal second coordinates
E12
LE
2 = 14. Σ1
(4.1-1j,1)
and
Az.2 = Mj2 +
(Mi,1 11,11)
+ En
En + E
2. Ar,1 S A5,1 inactive and 2322 S active. Using
17 we find that then A; - pe; and
Ee (1y2 -?) 122 7
M1j1
A =
and the cost is
w (pg2 7)
2
Σx
3. both di,1 S Aj,1 and A322 S y active. Here need to optimize
(M,,1 101-00 0)2
E22 0)2 - 22 0)( -7) Eni 7)
min dER
2Σ
En1 -E2
Cancelling the 9 derivative gives
EE E2 Et2)uu11 wE1 (E22f1j)1 E12(11 - 7))
(En1 w2i1 297
With that, the value and optimizers become
122 11j,1)²
2(E 1 1222 - 52/2)
Σ - M4j.1 (11:2 7)
2w
2((En1 + WEE1 |E122 -x(2)
(w EE1)(11;2 7)
2(E + +wEnn)e - E22)
and
0
(14,1 0,)
( X,
(1)
E12
and
U1.2
En
A.3.2
Proof of proposition 2.3
Let v -N(,,)) Σ), = (A, Σ') and M = Ma. To derive the KLinf, we will use the following well-known facts about matrix derivatives:
d dA
log|A| (A-1)" and
tr(A-1B) -AA--BA-1
aA
14

---

Together with the fact that Σ and E' are symmetric (and so are their inverses), it follows that
i KL = ---
-1)=E-
O
Setting to zero and multiplying by 52' from both left and right, gives
E' E (f () A)7
By the matrix determinant lemma, we have |Σ'| = >> (n 2) 2)) Substituting everything back in gives
KLinf(V,, 2
in 1 + l| 211--1)
Using 17 it immediately follows that
02(v) = In 1+ (J12 - 7)2
2
22
Next, let vs = (ps Ex) and Uj NM1,, Ez). Then
min A,AyERe Ay
KLinf(V;, A,) + w KLinf(y,, 2U)
min BER
min KLinf(u,, X;) +
min A, ER
w KLinf(us;
(18)
A.ER² A,1100
We have essentially already computed the first term inside the parantheses above: equals (=1
In 1
For the second term, see
21.11
if M1, 7 and pjj1 2
b log
E5.22 1 (
if M5.1 En,2 (M3,2 - -7)+ 2
E,,2
min Ey 20,
KLinn((v,,)p)
w log 1 B4,11
(0)
if f3,2 + Eaa1 (15,1 -)_ S ?
E,,2
2 w KLinf(uj, (0,7))
else.
The values in the second and third case are the result of ignoring one of the two constraints. If the optimizer of this less constrained problem problem has been found. If this does not happen for either of them, the optimum must be in the point X, (0, (i.e. both constraints must be active). We'll refer to these cases as case 0-3 respectively.
feasible solution, the optimizer of the entire
To get a further handle on this quantity, we will proceed by case distinctions.
1. Let's first consider 14,2 S 7. If Ej,1 O, then 242 En (14,1 0)_ S for all 0, so that the conditions case 2 will always be satisfied (case 2 coincides with case 0 for 0 < 1),1). The optimum value of will then be in [ug, 11.1). since the first term in (18) is decreasing and the second term is on (-o0, 1. and reversed for [M4,1, oo). It follows that the optimum 0 can be found by solving
1 d0 2
1 + (mi,1 - 0)2
+ w log
11 (Hg.1 - 0)3
log
=0
21.11
2
Σ5,11
(Hr.1 8)
0 13,1
+ w
Ej11 + (Mj.1 0)2 = O,
(19)
11 + (Mi,1 0)2
so it is matter of finding the roots of cubic (and pruning to [M5,1, m4.1).
If Ej,12 > 0, then for 0 < M4j11 + E,,2 (y (1,,2), we will still be in case 2, but we end up in case
Eg.1
3 for 0 larger than that. This does not affect the analysis if p4.1 < 1 +
Ean
E (y (5,2). If u4,1 is
15

---

larger than that, we can separately find the minimizer for 0 E [1y1 + can be done by solving
(r - -1,2), Eau
M6,1). This
(11.1 - 0)2
E,,1 ) + (Liinf( (0,
log
1+
2
(P411 8)
w
E,,2(1151 - 7) 13,22111111 - -8) 0,
(20)
21,11 (Ni,1 8)²
12
(Mj - (0, ")) (m: - (0, 7))
which again comes down solving a cubic (and clipping to the right interval). We can then minimize over the two minima to find the global minimizer.
2. Now we will consider the case 24, > 7. Then case 0 can never hold and we will be in case
1 for all 0 M311
5,,22 (112 7). If Σ2,1 > 0, case 2 can also never be satisfied, so
Σ,
E, will be in case 3 for all P1.1 22,22 (Mz2 - 7). If this bound is larger than M4,1, then each
0 E [Ps.1. M311 E,,2 (1y.2 - -)] a minimizer. If the bound is smaller than M1,1. we can use (20)
E,
to find the optimal value.
Σj,1 < 0, then we will only be case 3 until 0 2 Mj,1 Eg.12
(/y,2 - 7), at which point E.s
we enter case 2. As sanity check, notice that E, > E,,2 since Σ,.1 Ej,22 > Σ3,1 by E,
5,,22
positive semi-definiteness (so case 2 happens after 3). So we can (19) to find the minimum
over all large 0, and again find the global minimizer by minimizing over the two cases.
A.4 Proof of Proposition 2.4
First, let v E MB. then
KLinf(u, = min KL(vIQQ). Qu miQ)=²
(21)
Introducing Lagrange multipliers d1, d2, ds, the constraints can be included the objective as
min max Q20 d2.d5
KL(wIQQ) + di 11 d2 d2 1)
X1) + da da( X2)
(22)
Here and in the following, Eo is meant to be read as {x, -- This becomes more tractable by (as is usual for Lagrange multipliers) swapping the max and min, that is,
max min KI(uIIQ) + di (B - 1) d2 A1) + da - A2)- da d2.dd Q20
We show that this swap does not change the value of the problem after further simplifying. The inner minimum has optimizer
dQ
(x,
(23)
du
di + d2 + day
so the dual problem becomes
max dg dy +dzz+dy220 for (0.
E In (di + d2X + d2 + - d1 d2di d2d22-
(24)
At this point, respect to di to zero. Then, reparameterizing once more to 01 I+d5Ai+a5a and a2 = I+2 gives the desired form
reparameterize by d' = d2 and d'g ds/d; d1 and set the derivative with d;
max 222
In +1+a An) + +az - Xz))
1+aa A1) +2z A2)20
for (0.1)
It remains to show that the min-max swap was allowed. To this end, work backwards from 24 (the max over the dual function) and first use Lagrange Duality Theorem [Luenberger 1997 Theorem 1, Section 8.6] to relate the max of the dual function to a minimum of the original function. In doing so, the domain of optimization changes to the dual of the constraint space, that is, the dual of the set of bounded linear functionals the compact unit square [0, 1 By Riesz' Representation
16

---

Theorem [see e.g. Hartig 1983), this dual space is equal to the set of finite signed measures on [0, 1 so that we recover (2
Next, fix input arm distributions Vi E MM and positive weight w. The quantity of interest
min Q Q m01 mh m1(Q)) m²(Q))5T
KI((s]](Q:) w (L))))Q)))
(25)
Introducing Lagrange multipliers d1, d2, d3, d4, can write this max
min KI(u; + w (I,)) + di [1
(2
11,d2.d:20:42200
Q00
Q.
Q
+ (b - a, [X] - d4 ( ( 7),
where the implicit min-max swap is allowed by the same argument as before. We then find optimizers
dQ:
dQi (x.y) dug
(x.y)
and
(26)
du,
di dgx
d2 dg + day
and dual problem
d2 dgX + d
max di 200420 2 In args 2
E [n (d1 + dgX)] E
In
1 +
d1
d2 d1%
where the constraint is that both of the arguments in the logarithm positive the unit square, i.c., d1 + dzæ 2 and d2 dxI + d4y 2 0 for all (z, y) E [0, 1J2. The constraint is homogeneous the (vector of) Lagrange multipliers. At the optimum, unconstrained optimality (i.e. zero derivative) in the (d1, d2, da, d4) direction requires
di d2 + day 1 w. We solve this for d2 and end up with
1 1 w d1 dzX + (Y - 7)
E In (d1 + dg X]]
+ w E vs
In
20,22,
In args
If we reparameterize by bj = -d1 1)/u, b2 ds/ ds/w and by d1/w, we get
max b1 S0, In args 2
E [In (1 w(bi + d2x))) +
E [In (1 + bi + b2X balY 7))]
For completeness, let us remark that the inner minimizers 23 and 26 are defined as densities w.r.t. the original arm distributions v,, 14, Vy. We now discuss how to recover the outer optimal Q, Q1, Q2 for the primal problem (21) 255 The above densities (when plugging in the optimal values of the Lagrange multipliers a or d) are part of the answer. However, these densities themselves may not yet sum one. The reason is that the primal solutions sometimes put mass outside of the support of their corresponding arm distribution. in other cases it may be driven by optimality. Solving for satisfaction of the primal constraints (i.e. normalization and means) then resolves how the missing mass must be allocated to recover the primal feasible solutions. is always possible to do adding mass in single point on the boundary of the unit square.
some cases this is the only way to satisfy the constraints,
B Proofs for Section 3
B.1 Proof of Theorem 3.1 ( Thresholds)
Let v be a bandit with answer i* = -v) and true means m, = m(1;). Following the proof of Kaufmann and Koolen 2021 Proposition 21]. we have
Po (r < oo and + {"}
S P. (3n: (e(()) * i and An 2 3(0,n))
< Pr {3n,j A it No. (n) KLinf((d, (n), m + (Lin((())))) 2 3(8,n))
Pr {3n.j: (n) KLinf(D,(m), my) 2 3(,,))}
17

---

where the first case is for i" * None and the second is for i" = None. Furthermore, the last step uses that -i" is covered by motions of at most two arms, one being i whenever i" * None, by Proposition 2.1 Note that for i * None the set -v" has a motion of just arm i* as well, but the cost of that is subsumed by that of any two-arm deviation. To find / that ensures d-correctness, we need an anytime deviation inequality for a sum of two statistics of the form (n) (L)
For the fixed covarance Gaussian case, we can leverage KKuffaann and Koolen 2021 Theorem 9]. Even though that theorem is formulated for arbitrarily many I Gaussian arms, we note that for i.i.d. X; ~ (p, Σ), our n KLinf, _ (1) / 1. is a sum of two independent 1d Gaussian contri- butions. To be more precise, let Y, E E-1/Xx,, where E 1/2 is the inverse of the positive definite and symmetric square root of E. Then Y, ~ W/E-1/222
1/2 μu (()). so that Y1, >N((E-1/2)) 1/2 (((E-)/2p)), 1) for j E 21 independently from one another. Furthermore, define Y,
Di=1 Y,. We then see that
KL 1, (E 1/2 (2-1/22))) (1)) KL(L,, (E 2)))
2 3 1 - E 1/2 1/2 ull 1 [l(nn) -
The concentration result by Kaufmann and Koolen [2021] Theorem 9] is stated for sums of univariate KLs, as in the left-hand side; this equality allows us also use in the fixed covariance setting. We thus find that taking 3(6, n) In A + +4 4. + In 4 + In n n/2) suffices once In 2 6. We chose that latter threshold for readability, see Kaufmann and Koolen [021] for a more involved threshold that works for any E (0, 1).
For the unknown covariance, we exploit the expression for KLinf from IT.. Recall that for the unknown covariance case we use as our estimate (nn) -V((())) E(n)), that is, a Gaussian with the maximum likelihood mean and covariance. Then for i..dd X² (H,,
12 In (1 + (m) 1 (()) .):
n (T)) 14) =
2
Under v, the statistic (n 1) [(( - 1 (m)) has a Hotelling 12 distribution with n - 1 degrees of freedom in 2 dimensions, and therefore (n (n-2) KLinf(D(n). 14) 1/2 X33 To show this, we will (1) rely on known results on the relations between different distributions and (2) slightly abuse notation to de- note distributions in equations instead of random variable. Up scaling, the statistic we are concerned
with is In 11 - T2, ((2,n=2) -2)) (F-distribution). This, in turn, is equivalent to In (1 + 8 (1, ") (beta-prime), This can be written as -nn (1 8r (1, ()))))) -In
It can be shown that this is the same as In (1 2 F(2, 2)
In (8' (2 1) (B' ( + +)) which equals
-In (3 (==2,1) (beta), which is known to be exp ("=²² (exponential). The desired result follows by reintroducing the scaling factor n-2 2
Hence, for fixed sample size n and X E [0. 1] the MGF evaluates to Ee, (Li)nf(((m)))))] = i-x (alternatively, the above reasoning can be sidestepped by computing this integral directly). We then find that for fixed sample sizes n, and ny, and threshold C' > 2, we get, by a Chernoff bound,
1 1
PA ((n.- - 2) (Linf(D, p4) + (ng - 2) (D, m1) 2 C} s
4
We have, with W-1 denoting the negative branch of the Lambert function.
1 + 2 In 1 1 c = iff C = -2 (-e (-e-1+) In ~ In
5
weighted union bound over all possible values of n, and ny with each prior a over N gives
} S 8.
Σ (N&(n) - (Tinf(Dx 2 -2 (-e-t- (-=11--666 kE(
P
3n
Picking (m)) x w(in motivates the choice ((6 = In ( 422nn+44m In In K 2n
4 In ln). Notice that there both factors (n) are now replaced by Nz(n) 2. Technically, this could be compensated for by adjusting the threshold. However, this compensation would disappea
as the sample sizes grow large, so that it does not matter for any asymptotic arguments. Furthermore,
18

---

this is the same as simply using the statistic with factor (m())) 2 and noting that, in the limit, it is indistuingishable from our original statistic. In our experiments, we choose do the latter.
For the bounded case we can use our dual expression from (12 as a maximum over parameters in the compact set R S R Using the technique of (Agrawal et al. 2021 Lemma F.1] based on worst-case regret bounds for online learning with exp-concave losses, we find that for each arm i there is a M4 martingale (22 (of mixture form) satisfying Mi,,, 2 e (n,,n) KLinf(P,, (n)) (m)s)) 1- (NN))+ Taking the product over two arms, applying Ville's inequality, and using concavity of the In and Na(n) + (n) S n yields d-correctness for the choice B(ō, n) = In K
+ 2 + 41n(1 + n/2).
8
B.2 Proof of Theorem 3.2 (Asymptotic Optimality)
We show that our algorithm, which consists of (a) the sampling rule (Track-and-Stoe with C-tracking) combined with (b) the GLR stopping rule and (c) the empirical- recommendation rule ensures asymptotic optimality, in the sense that for every bandit v with unique best answer i" (z)), our algorithm is 8-correct on v and ensures
EP [T8]
lim 8-+
= (v).
In
This argument was pioneered by Garivier and Kaufmann 2016 for BAI in exponential families, and extended to general answers by Kaufmann and Koolen 2021 Theorem 17]. [Degenne and Koolen
2019 Theorem 7] proved the (upper-hemi) continuity assumption which was assumed before, and Agrawal et al. 2020 Section 6] generalized to non-parametrie arms.
To apply the argument for constrained BAI in the three models, we need to check two things. (1) the estimates (t) concentrate sufficiently fast around the true bandit v. (2) the oracle weights
# -ー "-=" (w) are a continuous function of the bandit.
Sufficiently fast concentration of empirical mean and variance in sup norm is argued by (Jourdan et al., 2023, Section H.1.1] for one dimension, and it generalizes to our Gaussian cases in two dimensions. For the bounded model, sufficiently fast concentration of the empirical distribution in Lévy metric is given by the multivariate DKW inequality] Naaman [201]]
Continuity of the oracle weights (²) function of the bandit follows from two nested applications of Berge's Theorem, which bottom out in continuity of KLinf. For the Gaussian cases (10) and IT this holds by inspection. The bounded case (2) requires small argument.
Proposition B.1. Let M = [0, 1,2 be the unit square, and let M = (0, 1)2 be its interior: Let L be the set of all probability distributions on M. The function KLinf from 2 is jointly continuous on L. x M where we equip f. with the Lévy metric.
Proof. For joint lower semi continint we can directly apply Agrawal et al. 2020 Lemma C.2] after observing that our L is compact. For joint upper semi-continuity exploit the dual representation (1) and go through (Agrawal et al. 2020 Lemma C.3], noting that Skorokhod's theorem applies to the metric space M.
With those details supplied, the remainder of the classic proof applies. See Kaufmann and Koolen Appendix DJ.
2021
Let us sketch the template here for completeness. First,
Forced exploration ensures that the estimates converge the true bandit, i.c. it() + v. Note that convergence is measured in the appropriate metric, which is Euclidean distance between parameters for the Gaussian cases and Lèvy for the bounded non- poramaraetrric case.
Continuity of the oracle weight map v + w (v) ensures that the sampling weights converge to the oracle weights, i.c. w; = w (D()) + w" (v). Here continuity is in the upper- hemicontinuous sense.
C-Tracking ensures that the sample counts converge to the sampling weights, i.e. N( Eiz w. Since the latter average converges to the oracle weights, so do the sampling w (v)..
proportions themselves N(O)
19

---

The stopping rule involves the empirical sampling proportion NO and the estimates (t).. ? If these are close to w (v) and v respectively, then the stopping rule kicks in around time T (P) In
Without quantifying all these convergences, crux of the in-expectation argument is to invoke sufficiently fast concentration of the estimates, to ensure that the contribution the expected sample complexity due failures of any of the above convergences is of lower-order in In 5.
proving IP, (oims-o in = The =
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract are contextualized in the introduction. All of them correspond to either propositions or theorems.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We provide discussion with two main limitations of our work, namely the stylized threshold used experiments and the single constraint.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: Correct proofs are given in the supplementary material. Wherever possible, reference existing literature that we build upon.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We provide a detailed discussion of the set-up our experiments in Section well as a figure to further clarify.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We aim to make the code publicly available at a later date.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips //ipss.cc/pill wttp:ccppppcccceeeeeeeeee..c.ett httaa//gggiiee//eeuuabiisisssss /Cc////oddiiisssseeeeeee ssiiissiiciiiyyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Question:
Answer: [Yes]
Justification: Similar to the reproducibility. all details on the implementation are given in Section[]
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We provide a standard error for the runtime estimates.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [No]
Justification: This is not included, because the experiments were run on personal laptop. We therefore do not expect anyone to run into problems in this regard.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: There were questionable research practices.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal relevance and impact of the research are discussed at the start of the introduction.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does use existing assets. The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcone com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
25

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
26

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification:
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to L.LM policy hate:p///eerrippc.cc///ofeeerers for what should or should not be described.
27


Reviews:
[{"review_id": "LXVr5PPBuS", "reviewer": "Reviewer_zJnu", "summary": "The authors consider the task of best arm identification in multi-armed-bandits in the constrained setting. Here by constrained setting the authors mean that each arm corresponds to a bi-variate random variable where the second entry of its mean must be below a threshold $\\gamma$. The authors provide generic instance-specific lower bounds revealing the structure that emerges from having constraints (see Theorem 2.2.). They further instantiate their lower bounds under three arm distribution models, namely gaussian distributions with known and unknown covariance and non-parametric distributions supported on $[0,1]^2$.  Next, they propose a best arm identification algorithm that follows the track-and-stop design principle and prove it's asymptotic optimality. Finally, they complement their theoretical findings with numerical experiments.", "questions": "- The authors use the notion of $\\delta$-correct strategy by they never defined what that is. Can you provide a definition? shouldn't this be given in the paper? \n- In the case when the variance is known, is exact knowledge actually needed for the algorithm? can the authors relax their guarantees to only knowledge of say upper bounds and lower bounds of certain variance parameters?", "limitations": "-  The results appear to require a specific model for arms distributions. What if this information is not known? \n-  The provided guarantees for the given algorithm are only asymptotic. How good are the provided algorithms when $\\delta$ is not too small?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n- The paper is very well written making the ideas easy to grasp.\n- The authors derive instance-dependent lower bounds for best arm identification in constrained multi-armed bandits. Despite the simplicity of the problem, the resulting lower bound and its proves to be different and novel in contrast with the classical unconstrained setting. \n- The authors provide an algorithm whose asymptotic sample complexity matches that of the lower bound. This demonstrates the asymptotic optimality of their algorithm and tightness of their lower bound. \n- The authors also consider the case where the arms are gaussian but with unknown variance which is a pleasant setting.\n- Comparison with an exhaustive list of algorithms.  \n\n\n**Weaknesses**\n-  The authors are only able to show asymptotic optimality while recent progress have been able to tackle best arm identification problems in the fixed confidence setting in the moderate regimes. \n- The authors use ideas and tools for best arm identification which are by now well studied, namely, the change-of-of-measure argument for deriving lower bounds, and the TaS design principle. How challenging is it to deploy these ides in the constrained bandit setting?\n- The experimental results rely on stylized thresholds for the definition of the stopping rules. This is a bit unsatisfactory, especially when the covariance is unknown.", "was_revised": true, "final_justification": "I would like to maintain my score. But I consider this work to be borderline as I don't find the extension to the constrained best arm identification setting extremely challenging. There is also the fact that most of the used tools and algorithm design ideas are not very innovative and well established in prior work. For instance, I don't find Theorem 3.1. particularly new. The authors also acknowledge some limitations of their work that I believe solving would be more noteworthy, for example, going beyond known covariances or obtaining non-asymptotic guarantees in $\\delta$ would be interesting.", "rebuttal": "We thank the reviewer for the careful reading and acknowledging the theoretical and experimental contributions of our work. Below, we address the questions and concerns raised. For clarity, reviewer comments are repeated in **_bold italics_**, followed by our responses in regular font.\n\n## Weaknesses ##\n\n**_The authors are only able to show asymptotic optimality while recent progress have been able to tackle best arm identification problems in the fixed confidence setting in the moderate regimes._**\n\nIt is true that our current focus is on the high-confidence regime, which aligns with our motivation of modeling high-risk business decisions. Extending our approach to the moderate-confidence regime is an interesting and important direction for future work. That said, our goal here was to handle the added complexity introduced by constraints and two-dimensional arms in a principled way for various models. We view support for the moderate-confidence regime as complementary, and potentially addressable with adaptations of our framework.\n\n**_The authors use ideas and tools for best arm identification which are by now well studied, namely, the change-of-of-measure argument for deriving lower bounds, and the TaS design principle. How challenging is it to deploy these ideas in the constrained bandit setting?_**\n\nWe would like to highlight four main difficulties that, in our view, make this problem a challenging direction of study: \n1. The introduction of multi-dimensional bandits with dependencies changes the structure of the lower-bound optimisation problem, which is nontrivial even for the 2-dimensional case. In particular, there are now two ways for one arm to outperform another: by making the competing arm infeasible, or by making the original arm genuinely better. This makes it important to do a careful case-based analysis. \n2. Beyond characterizing the instance-based complexity, we provide a generic computational recipe and an efficient algorithm given oracle access to the functions $c\\_1$ and $c\\_2$. \n3. In the constrained setting, it is possible that none of the arms are feasible. This differs from the standard setting and this case requires a different analysis. Handling this case with care is important, because even when a feasible arm exists in the true model, the maximum likelihood estimates may not be feasible, so we must be prepared to handle instances where no arms are feasible. \n4. Concentration results in two dimensions are less well-developed and extending techniques that work in one dimension to the two-dimensional setting is nontrivial. \n\n**_The experimental results rely on stylized thresholds for the definition of the stopping rules. This is a bit unsatisfactory, especially when the covariance is unknown._**\n\nWe acknowledge that the stylized threshold is not (yet) theoretically justified generally, especially for the unknown covariance model. In practice, our simulations indicate that the stylized threshold is conservative as the error rates are below the confidence level (Section 4). Furthermore, this practice is common in the existing literature and often adopted as a pragmatic choice (Garivier and Kaufmann 2016, Optimal Best Arm Identification with Fixed Confidence, Degenne, Koolen, and Ménard 2019, Non-Asymptotic Pure Exploration by Solving Games, 2019, etc.).\n\n## Questions ##\n\n**_The authors use the notion of δ-correct strategy by they never defined what that is. Can you provide a definition? shouldn't this be given in the paper?_**\n\nThe definition is given on l.40-41, but we agree that it might deserve a more central spot.\n\n**_In the case when the variance is known, is exact knowledge actually needed for the algorithm? can the authors relax their guarantees to only knowledge of say upper bounds and lower bounds of certain variance parameters?_**\n\nIn the current formulation of our known-variance model, knowledge of the full covariance matrix is indeed required. However, if domain knowledge provides partial information (such as upper or lower bounds on variance components, or known functional relationships) this could be incorporated into an alternative model. For instance, we have considered a variant in which the variances along each dimension are fixed, but the covariance between them is allowed to vary freely. In such cases, one would simply need to re-compute the relevant KLinf, $c\\_1$ and $c\\_2$ functions for the new model, analogous to Propositions 2.2-2.4. We conjecture that leveraging accurate domain knowledge in this way gives improved performance compared to assuming a fully unknown covariance matrix.\n\n\n## Limitations ##\n\n**_The results appear to require a specific model for arms distributions. What if this information is not known?_**\n\nWhen no specific model is known for the arms, the natural approach is to assume the most general model possible. The most general setting our framework currently supports is that of distributions with a common rectangular support. This bounded model is sufficient to cover all the motivating practical cases of this paper.\n\n**_The provided guarantees for the given algorithm are only asymptotic. How good are the provided algorithms when δ is not too small?_**\n\nAs noted above, our primary focus is on the high-confidence regime, which aligns with our motivation to model high-risk business decisions. Investigating the performance of the proposed algorithms in the moderate-confidence regime remains an interesting direction for future work."}, {"review_id": "p00fuTxgM9", "reviewer": "Reviewer_F1rN", "summary": "The authors proposed a variant of the standard K-armed best arm identification problem, where we also have costs for the arms. The idea is to find the best arm subject to a cost constraint. Given that, every arm has 2 distributions, one for the reward and the other for the cost. The authors consider 3 scenarios: Gaussian with fixed covariance, Gaussian with unknown covariance, and non-parametric distributions with rectangular support. The authors provided lower bounds and an algorithm, which they tested compared to baselines.", "questions": "- Could the authors comment on the originality of the work?\n- Could the authors highlight the technical challenges of the work?", "limitations": "Yes. The limitations of the work are properly discussed.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "### Strengths \n- The work is solid from a technical perspective, and it is built on well-established BAI literature.\n- The theoretical results are in line with the BAI literature, even if I haven't checked the proofs in the appendix.\n- To the best of the reviewer's knowledge, related works are properly cited.\n- The authors also provided an experimental campaign to validate the results.\n\n### Weaknesses\n- The main weakness of the paper is the presentation: sections 2 and 3 are hard to follow and should be revised to make them more readable to non-experts in the BAI literature. Related to the presentation, some symbols in section 2 are not formally introduced (even if it is often easy to infer them, e.g., $\\gamma$).\n- Another weakness of the paper is the originality: the paper presents a standard implementation of constraints in a well-known setting, and there are no characteristic/unexpected findings that the authors highlighted in the work.", "was_revised": true, "final_justification": "The rebuttal solved my main concerns on the technical novelty. I still have some concerns on the originality. Given that, I increased my score from 3 to 4.", "rebuttal": "We thank the reviewer for their evaluation and for acknowledging the technical strength of the paper. Below, we address the questions and concerns raised. For clarity, reviewer comments are repeated in **_bold italics_**, followed by our responses in regular font.\n\n## Weaknesses ## \n\n**_The main weakness of the paper is the presentation: sections 2 and 3 are hard to follow and should be revised to make them more readable to non-experts in the BAI literature. Related to the presentation, some symbols in section 2 are not formally introduced (even if it is often easy to infer them, e.g., γ)_**\n\nWe agree that the presentation in Sections 2 and 3 may be somewhat terse for readers less familiar with the BAI literature. To address this, we will consider adding a brief overview of the standard approach commonly used in this setting, which our work builds upon. Additionally, we will carefully review the manuscript to ensure that all symbols and notations are clearly and properly introduced.\n\n**_Another weakness of the paper is the originality: the paper presents a standard implementation of constraints in a well-known setting, and there are no characteristic/unexpected findings that the authors highlighted in the work._**\n\nWe respectfully disagree with the claim that the work lacks originality or surprising insights. We believe the contributions of the paper are novel in several ways:\n1. The treatment of multi-dimensional rewards with arbitrary dependence remains largely unexplored in the literature (particularly in the nonparametric setting) despite being highly relevant in real-world applications. To our knowledge, we are the first to explicitly investigate this dependence structure and to demonstrate that ignoring it can lead to suboptimal performance.\n2. We provide valid GLRT thresholds for the two-dimensional setting with dependence, which align with the lower bound on expected sample complexity. This is important because ignoring dependence leads to a different GLRT that may no longer be valid, potentially compromising the correctness of the procedure.\n3. In terms of formulation, we incorporate feasibility constraints in a clean and general way by introducing a second cost function. This allows for a modular, plug-and-play approach to constrained best-arm identification problems, as long as cost functions are specified—something that we believe has practical and methodological value.\n4. Beyond characterizing the instance-based complexity, we provide a generic computational recipe and an efficient algorithm given oracle access to $c\\_1$ and $c\\_2$. \n\nLastly, we present (analytically for the known variance case) the effect of dependence on transportation costs, optimal sampling proportions and expected sample complexity.The interesting findings of our work is that a naive, dependence-unaware algorithm can actually perform worse than uniform allocation (Section 2.2.1 & 5). This is a counterintuitive result that highlights the need to account for dependence in multi-dimensional settings (see also our response to a question by reviewer MhZm). Moreover, we show that the oracle allocations themselves depend strongly on the correlation structure, even when all other parameters are held constant.\nWe will make sure to emphasize these points more clearly in the next version.\n\n## Questions ##\n\n**_Could the authors comment on the originality of the work?_**\n\nOn top of our response to the previous point, we have attempted to position our contributions within the existing literature at the end of the introduction and to clarify our specific contributions in Section 1.1. That said, we agree that this could be made clearer. To address this, we will expand the discussion of related work in a dedicated section and more prominently highlight our own contributions.\n\n**_Could the authors highlight the technical challenges of the work?_**\n\nAs noted in our response to reviewer zJnu, we believe the main challenges that make this problem worth studying are as follows:\n1. The introduction of multi-dimensional bandits with dependencies changes the structure of the lower-bound optimisation problem, which is nontrivial even for the 2-dimensional case. In particular, there are now two ways for one arm to outperform another: by making the competing arm infeasible, or by making the original arm genuinely better. This makes it important to do a careful case-based analysis. \n2. Beyond characterizing the instance-based complexity, we provide a generic computational recipe and an efficient algorithm given oracle access to the functions c₁ and c₂. \n3.  In the constrained setting, it is possible that none of the arms are feasible. This differs from the standard setting and this case requires a different analysis. Handling this case with care is important, because even when a feasible arm exists in the true model, the maximum likelihood estimates may not be feasible, so we must be prepared to handle instances where no arms are feasible. \n4. Concentration results in two dimensions are less well-developed and extending techniques that work in one dimension to the two-dimensional setting is nontrivial."}, {"review_id": "bCYkpHmHXp", "reviewer": "Reviewer_MhZm", "summary": "This paper studies best-arm identification (BAI) under an additional constraint in the fixed-confidence setting. Specifically, the learner observes bivariate outcomes and aims to identify, with high probability, the arm that maximizes the first criterion, subject to the second criterion being below a specified threshold. The authors consider several distributional assumptions over the arms: bivariate normal distributions with known covariance, bivariate normal distributions with unknown covariance, and general bivariate distributions supported on $[0,1]^2$. An information-theoretic lower bound on the sample complexity is derived, and an adaptation of the Track-and-Stop algorithm (Garivier and Kaufmann 2016) is proposed to match this lower bound in the small error regime.", "questions": "**Minor comments**: \n* l.91 $KL(\\delta\\\\| 1- \\delta)$ is conflicting with earlier notation\n* The standard errors reported in Section 4 are surprisingly small given the  confidence level and the sample complexities reported. It would help if the authors reported the (co)variances used in these experiments\n**Questions** : \n* Why does the \"Oracle\" strategy perform significantly worse than the other algorithms in some experiments? \n* As the authors assume common covariance in the Gaussian setting. How is the MLE of covariance computed ? \n* How were the oracle allocation weights computed?\n* In Figure 3, negative correlation does not seem to influence the sample complexity. Can the authors provide some intuition for this?\n* Does Figure 3 suggest a regime of lower sample complexity for uniform sampling (which by nature ignores correlation) than the optimal algorithm that ignores correlation ?", "limitations": "yes, this work is mainly theoretical", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**: \nThe paper tackles a well-motivated and practically relevant problem with compelling applications. \nAn information-theoretic lower bound is stated and the authors proposed an algorithm to match this bound in the small error regime. We  appreciate the experimental results provided to support the theoretical claims and the ablation studies on the effect of correlation. The computational cost of the proposed algorithm is discussed and enhances the practical perspective of the work. However the papers contains major weaknesses and some soundness issues as stated below. \n\n**Weaknesses**: \n* The related work section overlooks several contributions on constrained pure exploration, including works that studied the same problem as the authors. For example  Katz-Samuels and Scott, 2019, Top Feasible Arm Identification) studied the identification of the best arm under linear feasibility constraints (which generalizes  the constrained BAI  problem to dimension $d$); Faizal and Nair, 2022 (Constrained Pure Exploration Multi-Armed Bandits with a Fixed Budget) studied the same problem in the fixed-budget setting;, Kone et al 2025 (Constrained Pareto Set Identification with Bandit Feedback) or Katz-Samuels and Scott (Feasible Arm Identification) \n* The novelty is limited as it is another direct application of track-and-stop methodology. The approach relies heavily on a relatively straightforward adaptation of the Track-and-Stop methodology. As such, the novelty appears incremental. The computational cost is also prohibitive compared to simpler approaches like in  Katz-Samuels and Scott, 2019,  or even compared to Racing approaches (as reported in Section 4)\n* The proof of theorem 3.1 is not correct :  for the setting with known covariance, statement from l.500 to l.502 is not correct. The authors incorrectly references Kaufmann and Koolen (2021), which addresses exponential families with real-valued parameters. The result does not directly apply to the bivariate case with dependent marginals. Appropriate references for this setting include section 4.1.3\nof Degenne 2019 (Impact de la structure sur la conception et l’analyse d’algorithmes de bandits)\n or Lemma 20 in Kone et al 2025 (Pareto Set Identification With Posterior Sampling). For the setting with unknown  covariance, statement on line l.510 is not correct, $\\frac{n-2}{2} \\log(1+ \\\\| \\hat \\mu_n - \\mu \\\\|_{\\hat \\Sigma_n^{-1}}^{2})$ is not a  $\\chi_2^2/2$, hence the derived concentration result is not correct. \n* The paper refers to prior work (Garivier and Kaufmann, 2016; Jourdan et al., 2023) for the analysis of Track-and-Stop. However, those works deal with univariate settings and do not address the concentration properties of empirical mean vectors and covariances required here. Since this is a central contribution of the paper, the analysis should be made self-contained.\n* The authors propose using D-tracking for arm allocation, but as pointed out by Degenne and Koolen 2019, D-tracking may fail to converge when the optimal allocation is not unique. The paper does not justify the uniqueness of the optimal allocation in the settings studied.", "was_revised": true, "final_justification": "My initial review raised some technical concerns that I felt required more rigorous justification. The authors have since provided convincing arguments addressing these points. The remaining issues noted in my rebuttal appear to be addressable in a revised version without requiring substantial changes to the structure of the manuscript. Overall, the paper presents several interesting contributions ; notably, the concentration inequalities (Theorem 3.1) and the analysis of how correlation affects sample complexity. While the algorithm itself is another variant of the Track-and-Stop framework, I commend the authors for their efforts in designing computationally efficient methods for characterizing and computing the oracle weights (see Theorem 2.2 and Propositions 2.2–2.4).", "rebuttal": "We thank the reviewer for acknowledging the theoretical and practical impact of our work. We appreciate the careful reading, the concrete pointers to missed references and the insightful questions. Below, we address the questions and concerns raised in order (we do not repeat the reviewers comments due to a lack of space).\n\n## Weaknesses ##\n\n- All of the mentioned articles are indeed closely related to our setting, and we will be sure to include appropriate references. The most relevant among them is perhaps Katz-Samuels and Scott, 2019, Top Feasible Arm Identification. We would like to highlight a few key differences: (1) their work focuses exclusively on the sub-Gaussian setting, while we introduce a framework capable of handling arbitrary models. This includes, as we demonstrate, non-parametric settings such as distributions with a common rectangular support; and (2) our framework allows us to explicitly investigate the effect of dependence between the cost and reward on algorithm performance.\n\n- 1. While we can use the transportation lemma, finding the optimal sampling proportions is not straightforward. So the fact that it is formalized in the TaS framework, does not mean it is ‘direct’. The TaS framework allows us to use the notion of characteristic time and the transportation lemma, but every new pure exploration problem (pep), or even a different model for the same pep (gaussian with unknown variance, bounded support, etc) would yield different transportation cost function and require a potentially different solution (if it even exists) for the oracle weights. In the constraint BAI, the feasibility constraint(s) introduces two transportation cost functions, for which finding the infimum requires a sub-casing, a new answer “none”, new GLRT thresholds, and an efficient algorithm to estimate the oracle weights. \n    2. We wanted to provide a common computational interface for all models considered in this paper, mainly motivated by the bounded setting, which can be well described in information theoretic terms. In this regard, we were especially inspired by previous works such as by Agrawal, Juneja, and Glynn (Optimal δ-correct best-arm selection for heavy-tailed distributions) and Agrawal, Koolen, and Juneja (Optimal Best Arm Identification Methods for Tail-Risk Measures), which illustrate the effectiveness of information-theoretic tools in non-parametric settings.\n    3. The algorithm is not computationally prohibitive, as we show in Theorem 2.1 it can be solved by a nested binary search. Especially the Gaussian case (unknown variance) requires just solving a cubic equation, which is very fast. The non parametric case is slower than the gaussian models, but the bottleneck is dealing with the empirical data distribution, which is enhanced by the nested search (a few milliseconds per instance). To avoid the nested search, we propose a gradient-ascent technique, which is extremely fast. (It is hard to benchmark our computational performance, because related work on constrained BAI does not handle non parametric cases)\n\n- We respond to the two points that are raised separately:\n\n   * l. 500 to l.502: While we agree that the setting studied by Kaufmann and Koolen (2021) does not directly apply to our setting, we do not agree that this means that their result is not applicable at all. The way in which we implicitly use it is as follows: define $Y_n = \\\\Sigma^{-1/2}X_n$ where $\\\\Sigma^{-1/2}$ is the inverse of the positive definite and symmetric square root of $\\Sigma$. We then have $Y_n \\\\sim N(\\\\Sigma^{-1/2}\\\\mu, I_2)$, so that $Y_{n,j}\\\\sim N((\\\\Sigma^{-1/2}\\mu)_j, 1)$ for $j=1,2$  independently from one another. Let us also denote $\\\\hat Y_n = \\\\frac{1}{n} \\\\sum\\_{i=1}^n Y_i$. Theorem 9 by Kaufmann and Koolen (2021) can then be used to gain control over sums of terms of the form $nd(\\\\hat Y\\_{n,1}, (\\\\Sigma^{-1/2}\\\\mu)\\_1) +nd(\\\\hat Y\\_{n,2}, (\\\\Sigma^{-1/2}\\\\mu)\\_2)$. That is, each arm gives us two univariate Gaussian contributions. These terms can be rewritten to $n/2 \\\\| \\hat Y_n - \\\\Sigma^{-1/2}\\\\mu\\\\|^2 =n/2 \\\\| \\\\hat Y_n - \\\\Sigma^{-1/2}\\\\mu\\\\|^2= n/2 \\\\|\\\\hat \\\\mu - \\\\mu \\\\|\\_{\\\\Sigma^{-1}}$, i.e., precisely the $n$KLinf that we consider here. However, we now see that this reasoning was very implicit and that it should have been made more explicit.\n\n   * Unknown covariance: we politely disagree, but admit that the argument could have been made more concrete. To show the validity, we will (1) rely on known results on the relations between different distributions and (2) slightly abuse notation to denote distributions in equations instead of random variables. The statistic we consider is $\\\\ln (1+\\\\frac{T^2(2,n-2)}{n-1})$ (Hotelling's T-squared) scaled by $\\\\frac{n-2}{2}$, but we drop the scaling for now. The former is equivalent to $\\\\ln (1+\\\\frac{2}{n-2} F(2,n-2))$ (F-distribution), which in turn is the same as $\\\\ln (1+\\\\beta'(1, \\\\frac{n-2}{2}))$ (Beta prime). Now, write this as $-\\\\ln (1/(1+\\\\beta'(1,\\\\frac{n-2}{2})))= -\\\\ln ( \\\\beta'(\\\\frac{n-2}{2},1)/(\\\\beta'(\\\\frac{n-2}{2},1)+1))$ which equals $-\\\\ln (\\\\beta(\\\\frac{n-2}{2},1))$ (Beta). This is known to be equivalent to $\\\\exp(\\\\frac{n-2}{2})$. After reintroducing the scaling factor of $\\\\frac{n-2}{2}$, we get $\\\\frac{1}{2} \\\\exp(1/2)=\\\\frac{1}{2} \\\\chi^2(2)$, as was to be shown.\n\n- We understand the concern and acknowledge that we were somewhat non-verbose in invoking results from prior work without fully explaining why they apply in our setting. We will revise the text to clarify these connections and elaborate on the intermediate steps where needed to make the argument more self-contained and accessible.\n\n- This is a valid concern. While it is true that D-tracking may face convergence issues in cases where the optimal weights are non-unique, we note that this does not affect the $\\\\delta$-correctness of the algorithm. Moreover, in all our simulations, D-tracking consistently converged and performed as expected, suggesting that this issue does not arise in practice for the settings we consider. \n\n## Minor Comments ##\n\n- To the best of our knowledge, this is the first mention of KL in the paper, so we are unsure about the conflict being referenced. That said, we agree that the notation is never formally introduced, and we will make sure to define it clearly.\n\n- All the bandits used in the simulations are shown in Figure 2, along with the corresponding covariance matrix. The error bars represent standard errors of the average runtimes, which we will clarify in the text. As expected, the standard errors decrease as the number of simulations increases.\n\n## Questions ## \n\n- This is an intriguing question that we unfortunately do not have a more satisfactory answer to than “it happens in all pure exploration experiments”, see e.g. Degenne, Koolen, and Ménard (Non-Asymptotic Pure Exploration by Solving Games, 2019) and Jourdan, Degenne, and Kaufmann (Dealing with Unknown Variances in Best-Arm Identification, 2023). However, it is definitely worth mentioning in the discussion of the results.\n\n- In the Gaussian setting, we consider two models: (1) the covariance matrix is fixed and shared across all arms, and (2) each arm has its own unknown covariance matrix, which may differ across arms. In the first case, we use the true covariance directly and do not estimate it. In the second case, the covariance is estimated via the standard MLE (that is, a scaled version of the sample covariance) for each arm separately without imposing any additional assumptions. To avoid singularity issues, we always start with at least three data points. \n\n- The oracle allocation weights are computed under the assumption of oracle knowledge of which bandit we are in. Specifically, the optimal weights from Theorem 2.2 are calculated using the true bandit means, and sampling is then performed using these fixed weights throughout.\n\n- The flat regime to the left of the dashes line in Figure 3  appears because at this point, making arm two feasible automatically makes it better than arm one. As argued in the text, the cost of the latter is \\rho independent. However, the exact correlation value at which this occurs, i.e. the location of the dashed line, is instance dependent and can range from  -1 to 1. The extreme values of -1 and 1 occur when the mean reward of arm two is moved below -1/3 or above 1/3. In these cases, making arm two just feasible will always make it worse or better than arm 1. In the former case, there will not be a flat regime anywhere and in the latter case, the entire plot will be flat and neither negative nor positive correlation will seem to influence the sample complexity. In that sense, there is nothing special about negative correlations in general. \nWe chose this specific instance because it clearly highlights the three regimes that might happen in terms of $ρ$: there will be a flat line, followed by two different curves, where the behaviour of the plot is determined by the different cases outlined  in Theorem 2.2 and Proposition 2.2. However, we agree that it would be useful to stress the instance dependence and to mention how the figure might change for other instances.\n\n- That is correct, and intuitively this also makes sense: the optimal algorithm that ignores correlation samples arm two a lot more frequently than arm one (as can be seen for rho=0 in the right plot of Figure 3). However, for large values of rho, the optimal algorithm (that **does** take into account correlation) will sample arm one more frequently. Therefore, for large values of rho, uniform sampling more closely resembles the optimal algorithm. This is an interesting observation that deserves to be explicitly highlighted."}, {"review_id": "ym6nkHO94p", "reviewer": "Reviewer_gFFA", "summary": "The authors study a fixed confidence finite-arm pure exploration problem where each arm is related to a bivariate distribution over rewards and costs. The goal of the learner is to identify the arm with the largest expected reward subject to satisfy an hard constraint on its cost.\n\nThey assume that such an optimal arm is unique and they derive, as usual in the literature, a lower bound on the sample complexity for achieving the aforementioned objective. Then, they propose a track-and-stop (GK16) procedure to solve the problem. Specifically, this procedure can be implemented in the three following scenarios:\n- Gaussian distributions with fixed covariance\n- Gaussian distributions with unknown covariance\n- Non-parametric distributions on the unit square.\n\nThe authors then run some numerical experiments and conclude the paper by studing the impact of the dependence on the sample complexity.", "questions": "As a suggestion, I invite the authors to improve the clarity of the manuscripts (see problems mentioned above). To gain eventual space, it might also be useful to condense the known and unknown covariance setting.", "limitations": "See weaknesses above.\n\nThis paper is mainly of a theoretical nature. I do not see a direct path to negative societal concerns.", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 1, "significance": 2, "originality": 2, "strengths_and_weaknesses": "# Strengths\n\nThe problem studied by the authors looks interesting as in some applications there is indeed this notion of costs to be taken into account.\nTo improve the state-of-the-art, I also suggest the authors to include the following references:\n- \"Optimal Multi-Fidelity Best-Arm Identification\", Poiani et al., NeurIPS 2024\n- \"Cost Aware Best Arm Identification\", Kanarios et al., 2024\nas they are related to this problem of dealing with costs in real-world scenarios.\n\n# Weaknesses\n- The paper has some clarity issues that make the read hard in some points. For instance: **(1)**  Lines 66-68 are not clear to me. **(2)** Furthermore, it is probably missing something in lines 119-123. This is complicating the read (also lambda^* not defined in lines 120-122). **(3)** In Figure 3 I fail to see T^* anywhere. **(4)**\n\n- Large part of this paper deals with the computation of T^* which the authors claims to be mandatory (lines 93-97) but which is not that necessary (see also my comment below). The arguments behind this points adopts the reasoning of GK16 with the additional linear constraint on the costs. This part of the paper seems to be technically a mild contribution to the pure exploration literature.\n\n- The algorithmic part does not present innovation in the sampling rule. Here, novel arguments only arises in proving the correctness of the method given that the feedback is different wrt standard works. However, the proof of Theorem 1 is really poorly detailed and informal and I cannot ensure its correctness, \n \n- Other (minor) concern. The discussion in lines 93-97 is inaccurate as there are several works that do not explicitly compute T^* nor w^* in practice. Examples include \"Non-asymptotic pure exploration by solving games\", Degenne et al., NeurIPS 2019 and \"Fast Pure Exploration via Frank Wolfe\", Wang et al., NeurIPS 2021.  It is furthermore not clear why the authors are referring to the regret minimization method of Degenne et al., 2019 as GA-TaS, since the two algorithms are not really working under the same principles.", "was_revised": true, "final_justification": "I am confirming my score for the reasons highlighted during the discussion phase", "rebuttal": "We thank the reviewer for the careful reading and the concrete pointers to literature. Below, we address the questions and concerns raised. For clarity, reviewer comments are repeated in **_bold italics_**, followed by our responses in regular font.\n\n## Strengths ##\n\n**_The problem studied by the authors looks interesting as in some applications there is indeed this notion of costs to be taken into account. To improve the state-of-the-art, I also suggest the authors to include the following references:[...]_**\n\nThese are indeed interesting references that are worth including. To clarify how they differ from our setting: in the first work, the cost is known in advance and tied to the fidelity level, rather than being an intrinsic property of the arm itself. Similarly, in the second, the cost is independent of the reward, and the objective remains to identify the (unconstrained) arm with the highest reward, while minimizing the overall sampling cost. In contrast, our setting imposes a constraint on the arm selection itself, rather than on the sampling process. Finally, both papers focus exclusively on exponential families, whereas our framework does not make this restriction.\n\n## Weaknesses ##\n\n**_The paper has some clarity issues that make the read hard in some points. For instance: (1) Lines 66-68 are not clear to me. (2) Furthermore, it is probably missing something in lines 119-123. This is complicating the read (also $\\\\lambda^\\\\star$ not defined in lines 120-122). (3) In Figure 3 I fail to see $T^\\\\star$ anywhere.(4)_**\n\nWe agree that certain parts of the paper could be clearer, and we appreciate the specific pointers. We will revise these parts to improve readability. As for point (3): the caption and legend are currently lacking in clarity. All of the lines in the left plot are expected stopping times computed with different weights, as indicated by the color of the line. The blue line is the expected stopping time with optimal weights, which corresponds to the characteristic time. \n\n**_Large part of this paper deals with the computation of $T^\\\\star$ which the authors claims to be mandatory (lines 93-97) but which is not that necessary (see also my comment below). The arguments behind this points adopts the reasoning of GK16 with the additional linear constraint on the costs. This part of the paper seems to be technically a mild contribution to the pure exploration literature._**\n\nAs for the first part: we will rephrase those sentences, we had not meant to phrase it as the only way to go about the problem. However, one of the reasons we are interested in computing $T^\\\\star$ is to characterize the difficulty of the problem, which allows us to e.g. investigate the effect of dependence on said difficulty. We agree that it is, strictly speaking, not needed for the algorithm itself, even though that is the approach we take here.\n\nFurthermore, we disagree that it is a mild contribution to the pure exploration literature. We solve a practical and intricate problem. In particular, considering the feasibility constraint introduces two transportation cost functions, for which finding the infimum requires sub-casing, a new answer “none”, a different GLRT, and new thresholds. \n\n**_The algorithmic part does not present innovation in the sampling rule. Here, novel arguments only arises in proving the correctness of the method given that the feedback is different wrt standard works. However, the proof of Theorem 1 is really poorly detailed and informal and I cannot ensure its correctness_**\n\nAs we also noted in our response to reviewer MhZm, we understand the concern and acknowledge that we were not explicit enough  in justifying the applicability of results from prior work to our setting. We will revise the proofs to and elaborate on the intermediate steps where appropriate, to make the argument more self-contained and accessible.\n\n**_Other (minor) concern. The discussion in lines 93-97 is inaccurate as there are several works that do not explicitly compute $T^\\\\star$ nor $w^\\\\star$ in practice. Examples include \"Non-asymptotic pure exploration by solving games\", Degenne et al., NeurIPS 2019 and \"Fast Pure Exploration via Frank Wolfe\", Wang et al., NeurIPS 2021. It is furthermore not clear why the authors are referring to the regret minimization method of Degenne et al., 2019 as GA-TaS, since the two algorithms are not really working under the same principles._**\n\nFor the first part, we refer to our response to the point above. Regarding the citation, we agree that its placement was suboptimal. Our intention was to acknowledge that our gradient ascent approach, used to address the high computational cost in the bounded case, was inspired by a method used by Degenne et al. (2019). We will revise the text to make this connection clearer.\n\n## Questions ##\n\n**_As a suggestion, I invite the authors to improve the clarity of the manuscripts (see problems mentioned above). To gain eventual space, it might also be useful to condense the known and unknown covariance setting._**\n\nWe agree with the concerns about clarity and will address them through the following improvements:\n1. more clearly highlighting the key challenges of the problem;\n2. making our contributions more prominent and easier to identify; and\n3. expanding on the less obvious steps in the proofs to improve readability and self-containment."}]

Author rebuttal:

Meta-review: The paper considers a novel setting of best-arm identification in multi-armed bandits with vector observations (in this case a reward and a cost), where the goal is to identify the arm with the highest mean reward subject to its mean cost being below a specified threshold. The authors study the fundamental sample complexity of this problem using information-theoretic tools in detail, under the general case where the cost and reward variables can have arbitrary (correlated) joint distributions. Using a newly developed characterization of sample complexity, they then proceed to derive a workable sampling and stopping algorithm based on the standard Track-and-Stop framework.

The reviews of this paper commend the clarity of the technical exposition, the novelty in the form of the instance-dependent sample complexity characterization, and the generality of the cost-reward joint distribution modeling. On the flip side, concerns were raised during the review process about the limited novelty of the track-and-stop principle being applied, the lack of sample complexity results in the moderate confidence regime, lack of adequate comparisons to other existing related work, technical considerations surrounding concentration arguments, and clarity of certain technical details. Most of the concerns were admittedly resolved to the referees' satisfaction in the ensuing engagement with the authors, and I thank everyone for the active discussion of the paper's claims and techniques. 

While the referees were unable to reach a consensus on the final recommendation, with one of them still remaining negative, I believe that the paper makes a valuable contribution to active sequential hypothesis testing in multi-armed bandits with an interesting and relevant reward-cost trade-off structure, and exposes the key considerations in learning with a clear exposition. For this reason, I recommend that the paper be accepted.
Decision: Accept (poster)