Title: Balancing Performance and Costs in Best Arm Identification
Venue: NeurIPS 2025
Accepted: yes
Keywords: Bandit Algorithms

Exploration
Abstract: We consider the problem of identifying the best arm in a multi-armed bandit model. Despite a wealth of literature in the traditional fixed budget and fixed confidence regimes of the best arm identification problem, it still remains a mystery to most practitioners as to how to choose an approach and corresponding budget or confidence parameter. We propose a new formalism to avoid this dilemma altogether by minimizing a risk functional which explicitly balances the performance of the recommended arm and the cost incurred by learning this arm. In this framework, a cost is incurred for each observation during the sampling phase, and upon recommending an arm, a performance penalty is incurred for identifying a suboptimal arm. The learner's goal is to minimize the sum of the penalty and cost. This new regime mirrors the priorities of many practitioners, e.g. maximizing profit in an A/B testing framework, better than classical fixed budget or confidence settings. We derive theoretical lower bounds for the risk of each of two choices for the performance penalty, the probability of misidentification and the simple regret, and propose an algorithm called DBCARE to match these lower bounds up to polylog factors on nearly all problem instances. We then demonstrate the performance of DBCARE on a number of simulated models, comparing to fixed budget and confidence algorithms to show the shortfalls of existing BAI paradigms on this problem.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study how to balance performance and costs in bandit problems.\nSpecifically, they propose two new settings that interpolate the fixed confidence and fixed budget settings. Specifically, they study:\n- A \"missidentification setting\", where the goal of the agent is minimizing the probability of missidentification + the expected stopping time of sampling multiplied by some costs $c$ which has to be interpreted as the cost of sampling\n- A \"simple regret\" setting, where the goal of the agent is minimizing the distance in mean of the recommended arm wrt to the optimal one + the expected stopping time of sampling multiplied by some costs $c$ which, again, has to be interpreted as the cost of sampling\n\nThe authors provide first a detailed description of both problems for the case of $K=2$ arms. Their findings highlight that there exists two regimes: when the cost is large wrt the gap, then, the agent should not sample the arms, but just recommends an arm at random, while, when gap are larges wrt the costs, the agent should explore the arms in order to minimize the two risk functionals that I mentioned above. \nThen, the authors extend the results to the K-arm setting and perform some numerical experiments.\n\nOverall, the authors provided lower bounds, and almost matching upper bounds (both in a worst-case sense on the possible problem instances).\n\nSTRENGTHS AND WEAKNESSES:\n# Strenghts\n- Interesting setting that nicely interpolates between fixed budget and fixed confidence setting. It removes the need of specifying an horizon and a confidence level, by just introducing a cost parameter $c$. To the best of my knowledge, this is novel and deserves attention from the community\n- The clarity of the exposition is good and the paper is easy to follow.\n- The paper highlights nicely the presence of two regimes that depend on costs and gaps. This provides intuition on the challenges of this problem compared to more canonical BAI settings.  \n\n# Weaknesses\n- All results are not fully problem dependent, but only problem dependent on the worst-case on the possible instance. This raises questions on the tightness of the results (see also my question below). However, I do not believe that this is a sufficient reason to stand for rejection. \n\n# Minor comments\n- Proof ideas on the algorithm is rather clear. However, proof ideas for the LB (lines 88-90) is not that informative. I invite the authors to either expand it or remove it in the future\n- Additional work on multi-fidelity BAI \"Optimal multi-fidelity best-arm identification\", Poiani et al., NeurIPS 2024\n- As a future direction, I think it would also be interesting to model the cost of pulling an arm in an arm-dependent fashion. Do the authors have any intuition on how (and if) their results extend in this direction?\n\nQUESTIONS:\nFor the K-arm setting I was expecting a smoother transitions in the two regimes according to the gaps of the bandits. For instance, suppose that the top-K/2 arms are very close while the remaining ones are highly sub-optimal. In this case, I was expecting the LB to show that (e.g., for the MI setting) we are paying a fixed error term of proportional to K + a gap dependent quantity that encodes the cost of pulling arms to discriminate the K/2 arms with large gaps.  This is also how the algorithm is somehow working, but e.g., Theorem 2 does not show this sort of problem-dependent behavior.I suspect that this might be related to the weaknesses that I was mentioning above, i.e., the fact that all results are not problem dependent but only problem dependent on the worst-case on the possible instance. Can the authors comment on this point? It seems that this point is also related to the future directions that the authors are mentioning at the end of the work\n\nLIMITATIONS:\nThis work is of theoretical work. I do not foresee a direct path to negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-15"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel framework for Best Arm Identification (BAI) in multi-armed bandits that directly balances the trade-off between sampling cost and arm performance. Unlike classical fixed-budget or fixed-confidence setups, the proposed setting minimizes a risk functional that penalizes both incorrect arm identification and the cost of pulling arms. It considers two measures: the sum of misidentification probability and sampling cost, and the sum of simple regret and sampling cost. The authors establish instance-dependent lower bounds for both settings, showing a phase transition depending on problem complexity and cost parameters, which does not appear in fixed-confidence/budget setups. A new algorithm, DBCARE, is proposed and shown to nearly match the lower bounds up to polylogarithmic factors and constants across different regimes. Theoretical analysis is complemented with empirical evaluations on synthetic and real-world datasets.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n- The introduced cost-aware BAI formulation seems to be new from the canonical formulations.\n- The paper presents theoretical results, including lower bounds with clear phase transition behavior and matching upper bounds (up to polylogarithmic factors and constants) for the proposed algorithm across regimes.\n\n**Weaknesses**\n- The choice of the confidence parameter $\\delta$ plays a crucial role in DBCARE’s performance but lacks thorough discussion. A more detailed analysis of its sensitivity or adaptability to instance-specific features would be valuable.\n- DBCARE operates in a round-robin fashion and eliminates arms based on confidence intervals, which may limit its adaptivity. It would be beneficial to explore more adaptive sampling strategies that better focus on promising arms.\n\nQUESTIONS:\n- Could the authors elaborate on the role of $\\delta$ beyond worst-case guarantees? Is there potential to select $\\delta$ in a more data-dependent or adaptive way that improves empirical performance?\n- Can the authors provide a practical scenario where this problem formulation can be used? It has mentioned clinical trials. But I suppose in these safety-critical fields, identifying a good arm is quite essential and the cost is not the primary concern.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-26"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new performance metric on the topic Best Arm Identification, which combines the total expected pulling times and the failure probability(or simple regret) into the loss function. Targeted on minimizing this new metric, this paper discusses an oracular policy and its further extension DBCARE. This paper also provides theoretical analysis for the algorithm DBCARE. The comparison between the upper and lower bounds suggests that the excellency of the algorithm.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths: \n1. This paper proposes a new metric formulation on Best Arm Identification, different with the classical fixed-confidence setting and the fixed-budget setting. This new problem formulation is a great supplement for the academic community.\n2. This paper provides theoretical analysis for both upper and lower bounds. The comparison suggests the efficiency of the algorithm design, as it can handle both performance metric MI and SR, with modification on some hyperparameters.\n\nWeakness:\n1. As discussed by the authors, the upper and lower performance bounds differ in the case of $H\\rightarrow \\infty$, under the metric $\\mathcal{R}_{SR}$.\n2. In the numeric experiment, the proposed algorithm DBCARE cannot outperform others significantly. But it is still reasonable, as it remains reasonably good across all the instances.\n3. The authors should also compare their results with this preprint: https://arxiv.org/abs/2409.18909, which is closely related to the authors' models.\n\nQUESTIONS:\nI wonder whether there exist some conclusions for the hardness of achieving best of both. For example, if an algorithm can achieve relatively good performance in the case of large H, does it mean the algorithm will perform worse in the case of smaller H? This is only an optional question, as I think the minimax bound is well acknowledged in the current community.\n\nBased on my understanding, I believe $c$ can be considered the weights of stopping times, while setting the weight of failing to identify an arm is 1. Please elaborate more on the case when $c$ is very small or $c$ is very large. \n\nMeanwhile, given the known parameter $c$, can we derive performance guarantees on $\\Pr(\\mu_1\\neq \\mu_{\\hat{I}})$ or $\\mathbb{E}[\\tau]$? I know the current Theorem 2 already provides a common upper bound for these two terms. Can we do better?\n\nRegarding the numerical experiments, I wonder whether you can compare your results with an anytime algorithm. For example, “Anytime Exploration for Multi-armed Bandits using Confidence Information”, by Jun & Nowak, proposes algorithm ATLUCB. We can take the termination time of DBSARE as the termination time of ATLUCB and compare their failure probability. Do you think this comparison is reasonable? If so, can we expect DBCARE achieve similar or even better numeric performance compared to the ATLUCB?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

comments: SUMMARY:
The paper propses a novel formulation of the objective in a multi-arm bandit setting: rather than the common fixed budget or fixed confience settings, a natural objective is proposed that offers a natural tradeoff between the performance and the cost of the arm pulls. For this objective, the paper proves lower bounds and upper bounds that match upto logarithmic factors. The techniques used are the standard machinery of change of measure arguments with a clever adaptive scheme for the algorithm.

STRENGTHS AND WEAKNESSES:
Strengths:
- nice formulation which avoids the dilemma of what confidence or budget to set, and instead offers an automatic tradeoff between performance and number of arm pulls.
- nice explanation of the intuition by considering 2 arm case and policyy with known $\Delta$ before doing the general case.
- elegant analysis showing an interesting phase transition between regimes of low and high complexity

Weaknesses:
- very limited experimental evaluation.

QUESTIONS:
- The baselines are unclear -- Which specific fixed-budget and fixed-confidence algorithms served as baselines in your experiments?
- The end of the experiments section mentions experiments on a drug-discovery dataset. Could you highlight the main takeaways from these results and explain how they demonstrate DBCARE’s superiority over traditional BAI approaches in such a real-world context?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-01