Title: Best Arm Identification for Stochastic Rising Bandits
Venue: NeurIPS 2023
Accepted: no
Keywords: Best Arm Identification

Rising

Rested

Stochastic

Bandits
Abstract: Stochastic Rising Bandits (SRBs) model sequential decision-making problems in which the expected rewards of the available options increase every time they are selected. This setting captures a wide range of scenarios in which the available options are learning entities whose performance improves (in expectation) over time. While previous works addressed the regret minimization problem, this paper, focuses on the fixed-budget Best Arm Identification (BAI) problem for SRBs. In this scenario, given a fixed budget of rounds, we are asked to provide a recommendation about the best option at the end of the identification process. We propose two algorithms to tackle the above-mentioned setting, namely R-UCBE, which resorts to a UCB-like approach, and R-SR, which employs a successive reject procedure. Then, we prove that, with a sufficiently large budget, they provide guarantees on the probability of properly identifying the optimal option at the end of the learning process. Furthermore, we derive a lower bound on the error probability, matched by our R-SR (up to logarithmic factors), and illustrate how the need for a sufficiently large budget is unavoidable in the SRB setting.
Finally, we numerically validate the proposed algorithms in both synthetic and real-world environments and compare them with the currently available BAI strategies.

Abstract

Stochastic Rising Bandits (SRBs) model sequential decision-making problems in which the expected reward of the available options increases every time they are selected. This setting captures a wide range of scenarios in which the available options are learning entities whose performance improves (in expectation) over time. While previous works addressed the regret minimization problem, this paper focuses on the fixed-budget Best Arm Identification (BAI) problem for SRBs. In this scenario, given a fixed budget of rounds, we are asked to provide a recommendation about the best option at the end of the identification process. We propose two algorithms to tackle the above-mentioned setting, namely R-UCBE , which resorts to a UCB-like approach, and R-SR , which employs a successive reject procedure. Then, we prove that, with a sufficiently large budget, they provide guarantees on the probability of properly identifying the optimal option at the end of the learning process. Furthermore, we derive a lower bound on the error probability, matched by our R-SR (up to logarithmic factors), and illustrate how the need for a sufficiently large budget is unavoidable in the SRB setting. Finally, we numerically validate the proposed algorithms in synthetic and real-world environments and compare them with the currently available BAI strategies.

1 Introduction 18

Multi-Armed Bandits (MAB, Lattimore and Szepesvári, 2020) are a well-known framework that 19 effectively solves learning problems requiring sequential decisions. Given a time horizon, the learner 20 chooses, at each round, a single option (a.k.a. arm) and observes the corresponding noisy reward, 21 which is a realization of an unknown distribution. The MAB problem is commonly studied in two 22 flavours: regret minimization (Auer et al., 2002) and best arm identification (Bubeck et al., 2009). 23 In regret minimization, the goal is to control the cumulative loss w.r.t. the optimal arm over a time 24 horizon. Conversely, in best arm identification, the goal is to provide a recommendation about the 25 best arm at the end of the time horizon. Specifically, we are interested in the fixed-budget scenario, 26 where we seek to minimize the error probability of recommending the wrong arm at the end of the 27 time budget, no matter the loss incurred during learning. 28

This work focuses on the Stochastic Rising Bandits (SRB), a specific instance of the rested ban29

dit (Tekin and Liu, 2012) setting in which the expected reward of an arm increases according to the 30

number of times it has been pulled. Online learning in such a scenario has been recently addressed 31

from a regret minimization perspective by Metelli et al. (2022), in which the authors provide no32

regret algorithms for the SRB setting in both the rested and restless cases. The SRB setting models 33

several real-world scenarios where arms improve their performance over time. A classic example is 34

the so-called Combined Algorithm Selection and Hyperparameter optimization (CASH, Thornton 35

et al., 2013; Kotthoff et al., 2017; Erickson et al., 2020; Li et al., 2020; Zöller and Huber, 2021), a 36

problem of paramount importance in Automated Machine Learning (AutoML, Feurer et al., 2015; 37

Yao et al., 2018; Hutter et al., 2019; Mussi et al., 2023). In CASH, the goal is to identify the best 38

learning algorithm together with the best hyperparameter configuration for a given ML task (e.g., 39

classification or regression). In this problem, every arm represents a hyperparameter tuner acting 40 on a specific learning algorithm. A pull corresponds to a unit of time/computation in which we 41 improve (on average) the hyperparameter configuration (via the tuner) for the corresponding learning 42 algorithm. CASH was handled in a bandit Best Arm Identification (BAI) fashion in Li et al. (2020) 43 and Cella et al. (2021). The former handles the problem by considering rising rested bandits with 44 deterministic rewards, failing to represent the intrinsic uncertain nature of such processes. Instead, 45 the latter, while allowing stochastic rewards, assumes that the expected rewards evolve according to a 46 known parametric functional class, whose parameters have to be learned. 1 47

48

49

50

51

52

Original Contributions In this paper, we address the design of algorithms to solve the BAI task in the rested SRB setting when a fixed budget is provided. 2 More specifically, we are interested in algorithms guaranteeing a sufficiently large probability of recommending the arm with the largest expected reward at the end of the time budget (as if only this arm were pulled from the beginning). The main contributions of the paper are summarized as follows: 3

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

We propose two algorithms to solve the BAI problem in the SRB setting: R-UCBE (an optimistic approach, Section 4) and R-SR (a phases-based rejection algorithm, Section 5). First, we introduce specifically designed estimators required by the algorithms (Section 3). Then, we provide guarantees on the error probability of the misidentification of the best arm.

We derive the first error probability lower bound for the SRB setting, matched by our R-SR algorithm up to logarithmic factors, which highlights the complexity of the problem and the need for a sufficiently large time budget (Section 6).

Finally, we conduct numerical simulations on synthetically generated data and a real-world online best model selection problem. We compare the proposed algorithms with the ones available in the bandit literature to tackle the SRB problem (Section 7).

2 Problem Formulation

In this section, we revise the Stochastic Rising Bandits (SRB) setting (Heidari et al., 2016; Metelli et al., 2022). Then, we formulate our best arm identification problem, introduce the definition of error probability, and provide a preliminary characterization of the problem.

Setting We consider a rested Multi-Armed Bandit problem ν ' p ν i q i P /llbracket K /rrbracket with a finite number of arms K . 4 Let T P N be the time budget of the learning process. At every round t P /llbracket T /rrbracket , the agent selects an arm I t P /llbracket K /rrbracket , plays it, and observes a reward x t ' ν I t p N I t ,t q , where ν I t p N I t ,t q is the reward distribution of the chosen arm I t at round t and depends on the number of pulls performed so far N i,t : ' ř t τ ' 1 1 t I τ ' i u (i.e., rested). The rewards are stochastic, formally x t : ' µ I t p N I t ,t q ` η t , where µ I t p¨q is the expected reward of arm I t and η t is a zero-mean σ 2 -subgaussian noise, conditioned to the past. 5 As customary in the bandit literature, we assume that the rewards are bounded in expectation, formally µ i p n q P r 0 , 1 s , @ i P /llbracket K /rrbracket , n P /llbracket T /rrbracket . As in (Metelli et al., 2022), we focus on a particular family of rested bandits in which the expected rewards are monotonically non-decreasing and concave in expectation.

Assumption 2.1 (Non-decreasing and concave expected rewards) . Let ν be a rested MAB, defining γ i p n q : ' µ i p n ` 1 q ´ µ i p n q , for every n P N and every arm i P /llbracket K /rrbracket the rewards are non-decreasing and concave, formally:

Non-decreasing: γ i p n q ě 0 ,

Concave: γ i p n ` 1 q ď γ i p n q .

Intuitively, the γ i p n q represents the increment of the real process µ i p¨q evaluated at the n th pull. Notice that concavity emerges in several settings, such as the best model selection and economics, representing the decreasing marginal returns (Lehmann et al., 2001; Heidari et al., 2016).

3

The proofs of all the statements in this work are provided in Appendix D.

2

2

Learning Problem The goal of BAI in the SRB setting is to select the arm providing the largest 83 expected reward with a large enough probability given a fixed budget T P N . Unlike the stationary 84 BAI problem (Audibert et al., 2010), in which the optimal arm is not changing, in this setting, we 85 need to decide when to evaluate the optimality of an arm. We define optimality by considering the 86 largest expected reward at time T . Formally, given a time budget T , the optimal arm i ˚ p T q P /llbracket K /rrbracket , 87 which we assume unique, satisfies: 88

/llbracket /rrbracket where we highlighted the dependence on T as, with different values of the budget, i ˚ p T q may change. Let i P /llbracket K /rrbracket zt i ˚ p T qu be a suboptimal arm, we define the suboptimality gap as ∆ i p T q : ' µ i ˚ p T q p T q´ µ i p T q . We employ the notation p i q P /llbracket K /rrbracket to denote the i th best arm at time T (arbitrarily breaking ties), i.e., we have ∆ p 2 q p T q ď ¨ ¨ ¨ ď ∆ p K q p T q . Given an algorithm A that recommends ˆ I ˚ p T q P /llbracket K /rrbracket at the end of the learning process, we measure its performance with the error probability , i.e., the probability of recommending a suboptimal arm at the end of the time budget T :

89

90

91

92

93

94

95

96

97

Problem Characterization We now provide a characterization of a specific class of polynomial functions to upper bound the increments γ i p n q .

/llbracket /rrbracket /llbracket /rrbracket We anticipate that, even if our algorithms will not require such an assumption, it will be used for deriving the lower bound and for providing more human-readable error probability guarantees. Furthermore, we observe that our Assumption 2.2 is fulfilled by a strict superset of the functions employed in Cella et al. (2021).

Assumption 2.2 (Bounded γ i p n q ) . Let ν be a rested MAB, there exist c ą 0 and β ą 1 such that for every arm i P K and number of pulls n P 0 , T it holds that γ i p n q ď cn ´ β .

3 Estimators

In this section, we introduce the estimators of the arm expected reward employed by the proposed 98 algorithms. 6 A visual representation of such estimators is provided in Figure 1. 99

Let ε P p 0 , 1 { 2 q be the fraction of samples collected up to the current time t we use to build estimators of the expected reward. We employ an adaptive arm-dependent window size h p N i,t ´ 1 q : ' t ε N i,t ´ 1 u to include the most recent samples collected only, avoiding the use of samples that are no longer representative. We define the set of the last h p N i,t ´ 1 q rounds in which the i th arm was pulled as:

100

101

T i,t : ' t τ P /llbracket T /rrbracket : I τ ' i ^ N i, τ ' N i,t ´ 1 ´ l, l P /llbracket 0 , h p N i,t ´ 1 q ´ 1 /rrbracket u . Furthermore, the set of the pairs of rounds τ and τ 1 belonging to the sets of the last and second-last h p N i,t ´ 1 q -wide windows of the i th arm is defined as:

102

103

104

105

106

107

Pessimistic Estimator The pessimistic estimator ˆ µ i p N i,t ´ 1 q is a negatively biased estimate of µ i p T q obtained assuming that the function µ i p¨q remains constant up to time T . This corresponds to the minimum admissible value under Assumption 2.1 (due to the Non-decreasing constraint). This estimator is an average of the last h p N i,t ´ 1 q observed rewards collected from the i th arm, formally:

The estimator enjoys the following concentration property. 108

/llbracket /rrbracket In the following, we design a pessimistic estimator and an optimistic estimator of the expected reward of each arm at the end of the budget time T , i.e., µ i p T q . 7

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

Lemma 3.1 (Concentration of ˆ µ i ) . Under Assumption 2.1, for every a ą 0 , simultaneously for every arm i P K and number of pulls n P 0 , T , with probability at least 1 ´ 2 TKe ´ a { 2 it holds that:

As supported by intuition, we observe that the estimator is affected by a negative bias that is represented by ˆ ζ i p n q that vanishes as n Ñ8 under Assumption 2.1 with a rate that depends on the increment functions γ i p¨q . Considering also the term ˆ β i p n q and recalling that h p n q ' O p n q , under Assumption 2.2, the overall concentration rate is O p n ´ 1 { 2 ` cTn ´ β q .

Optimistic Estimator The optimistic estimator ˇ µ T i p N i,t ´ 1 q is a positively biased estimation of µ i p T q obtained assuming that function µ i p¨q linearly increases up to time T . This corresponds to the maximum value admissible under Assumption 2.1 (due to the Concavity constraint). The estimator is constructed by adding to the pessimistic estimator ˆ µ i p N i,t ´ 1 q an estimate of the increment occurring in the next step up to T . The latter uses the last 2 h p N i,t ´ 1 q samples to obtain an upper bound of such growth thanks to the concavity assumption, formally:

t

T

τ

τ

1

ˇ

µ

T

i

p

N

i,t

´

1

q

ˆ

µ

i

p

N

i,t

´

1

q

N

i,t

´

1

ˇ

β

T

i

p

N

i,t

´

1

q

B

T

i

N

i,t

1

h

p

N

i,t

´

1

q

q

´

p

The estimator displays the following concentration guarantee.

Lemma 3.2 (Concentration of ˇ µ T i ) . Under Assumption 2.1, for every a ą 0 , simultaneously for every arm i P /llbracket K /rrbracket and number of pulls n P /llbracket 0 , T /rrbracket , with probability at least 1 ´ 2 TKe ´ a { 10 it holds that: ˇ β T i p n q ď ˇ µ T i p n q ´ µ i p n q ď ˇ β T i p n q ` ˇ ζ T i p n q , (4)

Differently from the pessimistic estimation, the optimistic one displays a positive vanishing bias ˇ ζ T i p n q . Under Assumption 2.2, we observe that the overall concentration rate is O p Tn ´ 3 { 2 ` cTn ´ β q .

4 Optimistic Algorithm: Rising Upper Confidence Bound Exploration

In this section, we introduce and analyze Rising Upper Confidence Bound Exploration ( R-UCBE ) an optimistic error probability minimization algorithm for the SRB setting with a fixed budget. The algorithm explores by means of a UCB-like approach and, for this reason, makes use of the optimistic estimator ˇ µ T i plus a bound to account for the uncertainty of the estimation. 8

Algorithm The algorithm, whose pseudo-code is reported in Algorithm 1, requires as input an exploration parameter a ě 0 , the window size ε P p 0 , 1 { 2 q , the time budget T , and the number of arms K . At first, it initializes to zero the counters N i, 0 , and sets to `8 the upper bounds B T i p N i, 0 q of all the arms (Line 2). Subsequently, at each time t P /llbracket T /rrbracket , the algorithm selects the arm I t with the largest upper confidence bound (Line 4):

146

147

148

149

150

151

152

153

154

155

156

where ˇ β T i p N i,t ´ 1 q represents the exploration bonus (a graphical representation is reported in Figure 1). Once the arm is chosen, the algorithm plays it and observes the feedback x t (Line 5). Then, the optimistic estimate ˇ µ T I t p N I t ,t q and the exploration bonus ˇ β T I t p N I t ,t q of the selected arm I t are updated (Lines 8-9). The procedure is repeated until the algorithm reaches the time budget T . The final recommendation of the best arm is performed using the last computed values of the bounds B T i p N i,T q , returning the arm ˆ I ˚ p T q corresponding to the largest upper confidence bound (Line 12).

Bound on the Error Probability of R-UCBE We now provide bounds on the error probability for R-UCBE . We start with a general analysis that makes no assumption on the increments γ i p¨q and, then, we provide a more explicit result under Assumption 2.2. The general result is formalized as follows.

Theorem 4.1. Under Assumption 2.1, let a ˚ be the largest positive value of a satisfying:

where for every i P /llbracket K /rrbracket , y i p a q is the largest integer for which it holds:

If a ˚ exists, then for every a P r 0 , a ˚ s the error probability of R-UCBE is bounded by: 157

158

159

160

161

162

163

164

165

166

167

168

Some comments are in order. First, a ˚ is defined implicitly, depending on the constants σ , T , the increments γ i p¨q , and the suboptimality gaps ∆ i p T q . In principle, there might exist no a ˚ ą 0 fulfilling condition in Equation (7) (this can happen, for instance, when the budget T is not large enough), and, in such a case, we are unable to provide theoretical guarantees on the error probability of R-UCBE . Second, the result presented in Theorem 4.1 holds for generic increasing and concave expected reward functions. This result shows that, as expected, the error probability decreases when the exploration parameter a increases. However, this behavior stops when we reach the threshold a ˚ . Intuitively, the value of a ˚ sets the maximum amount of exploration we should use for learning.

Under Assumption 2.2, i.e., using the knowledge on the increment γ i p¨q upper bound, we derive a result providing conditions on the time budget T under which a ˚ exists and an explicit value for a ˚ .

Corollary 4.2. Under Assumptions 2.1 and 2.2, if the time budget T satisfies:

there exists a ˚ ą 0 defined as: 169

170

171

where H 1 , η p T q : ' ř i ‰ i ˚ p T q 1 ∆ η i p T q for η ą 0 . Then, for every a P r 0 , a ˚ s , the error probability of R-UCBE is bounded by:

First of all, we notice that the error probability e T p R-UCBE q presented in Theorem 4.2 holds under 172 the condition that the time budget T fulfills Equation (10). We defer a more detailed discussion 173 on this condition to Remark 5.1, where we show that the existence of a finite value of T fulfilling 174 Equation (10) is ensured under mild conditions. 175

Let us remark that term H 1 , η p T q characterizes the complexity of the SRB setting, corresponding to 176 term H 1 of Audibert et al. (2010) for the classical BAI problem when η ' 2 . As expected, in the 177 smallβ regime (i.e., β P p 1 , 3 { 2 s ), looking at the dependence of H 1 , 1 { β p T q on β , we realize that 178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

Algorithm 1: R-UCBE .

the complexity of a problem decreases as the parameter β increases. Indeed, the larger β , the faster the expected reward reaches a stationary behavior. Nevertheless, even in the largeβ regime (i.e., β ą 3 { 2 ), the complexity of the problem is governed by H 1 , 2 { 3 p T q , leading to an error probability larger than the corresponding one for BAI in standard bandits (Audibert et al., 2010). This can be explained by the fact that R-UCBE uses the optimistic estimator that, as shown in Section 3, enjoys a slower concentration rate compared to the standard sample mean, even for stationary bandits.

This two-regime behavior has an interesting interpretation when comparing Corollary 4.2 with Theorem 4.1. Indeed, β ' 3 { 2 is the break-even threshold in which the two terms of the l.h.s. of Equation (8) have the same convergence rate. Specifically, the term p A q takes into account the expected rewards growth (i.e., the bias in the estimators), while p B q considers the uncertainty in the estimations of the R-UCBE algorithm (i.e., the variance). Intuitively, when the expected reward function displays a slow growth (i.e., γ i p n q ď cn ´ β with β ă 3 { 2 ), the bias term p A q dominates the variance term p B q and the value of a ˚ changes accordingly. Conversely, when the variance term p B q is the dominant one (i.e., γ i p n q ď cn ´ β with β ą 3 { 2 ), the threshold a ˚ is governed by the estimation uncertainty, being the bias negligible.

As common in optimistic algorithms for BAI (Audibert et al., 2010), setting a theoretically sound value of exploration parameter a (i.e., computing a ˚ ), requires additional knowledge of the setting, namely the complexity index H 1 , η p T q . 9 In the next section, we propose an algorithm that relaxes this requirement.

5 Phase-Based Algorithm: Rising Successive Rejects

In this section, we introduce the Rising Successive Rejects ( R-SR ), a phase-based solution inspired by the one proposed by Audibert et al. (2010), which overcomes the drawback of R-UCBE of requiring knowledge of H 1 , η p T q .

Algorithm R-SR , whose pseudo-code is reported in Algorithm 2, takes as input the time budget T and the number of arms K . At first, it initializes the set of the active arms X 0 with all the available arms (Line 1). This set will contain the arms that are still eligible candidates to be recommended. The entire process proceeds through K ´ 1 phases. More specifically, during the j th phase, the arms still remaining in the active arms set X j ´ 1 are played (Line 5) for N j ´ N j ´ 1 times each, where:

and log p K q : ' 1 2 ` ř K i ' 2 1 i . At the end of each phase, the arm with the smallest value of the pessimistic estimator ˆ µ i p N j q is discarded from the set of active arms (Line 11). At the end of the p K ´ 1 q th phase, the algorithm recommends the (unique) arm left in X K ´ 1 (Line 13).

It is worth noting that R-SR makes use of the pessimistic estimator ˆ µ i p n q . Even if both estimators 210 defined in Section 3 are viable for R-SR , the choice of using the pessimistic estimator is justified 211 by its better concentration rate O p n ´ 1 { 2 q compared to that of the optimistic estimator O p Tn ´ 3 { 2 q , 212 being n ď T (see Section 3). 213

214

215

216

217

Note that the phase lengths are the ones adopted by Audibert et al. (2010). This choice allows us to provide theoretical results without requiring domain knowledge (still under a large enough budget). An optimized version of N j may be derived assuming full knowledge of the gaps ∆ i p T q , but, unfortunately, such a hypothetical approach would have similar drawbacks as R-UCBE .

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

Bound on the Error Probability of R-SR The following theorem provides the guarantee on the error probability for the R-SR algorithm.

Theorem 5.1. Under Assumptions 2.1 and 2.2, if the time budget T satisfies:

then, the error probability of R-SR is bounded by:

Similar to the R-UCBE , the complexity of the problem is characterized by term H 2 p T q that, for the standard MAB setting, reduces to the H 2 term of Audibert et al. (2010). Furthermore, when the condition of Equation (12) on the time budget T is satisfied, the error probability coincides with that of the SR algorithm for standard MABs (apart for constant terms). The following remark elaborates on the conditions of Equations (10) and (12) about the minimum requested time budget.

Remark 5.1 (About the minimum time budget T ) . To satisfy the e T bounds presented in Corollary 4.2 and Theorem 5.1, R-UCBE and R-SR require the conditions provided by Equations (10) and (12) about the time budget T , respectively. First, let us notice that if the suboptimal arms converge to an expected reward different from that of the optimal arm as T Ñ `8 , it is always possible to find a finite value of T ă `8 such that these conditions are fulfilled. Formally, assume that there exists T 0 ă `8 and that for every T ě T 0 we have that for all suboptimal arms i ‰ i ˚ p T q it holds that ∆ i p T q ě ∆ 8 ą 0 . In such a case, the l.h.s. of Equations (10) and (12) are upper bounded by a function of ∆ 8 and are independent on T . Instead, if a suboptimal arm converges to the same expected reward as the optimal arm when T Ñ`8 , the identification problem is more challenging and, depending on the speed at which the two arms converge as a function of T , might slow down the learning process arbitrarily. This should not surprise as the BAI problem becomes non-learnable even in standard (stationary) MABs when multiple optimal arms are present (Heide et al., 2021).

6 Lower Bound

In this section, we investigate the complexity of the BAI problem for SRBs with a fixed budget.

Minimum time budget T We show that, under Assumptions 2.1 and 2.2, any algorithm requires a minimum time budget T to be guaranteed to identify the optimal arm, even in a deterministic setting.

Theorem 6.1. For every algorithm A , there exists a deterministic SRB satisfying Assumptions 2.1 and 2.2 such that the optimal arm i ˚ p T q cannot be identified for some time budgets T unless:

Theorem 6.1 formalizes the intuition that any of the suboptimal arms must be pulled a sufficient number of times to ensure that, if pulled further, it cannot become the optimal arm. It is worth comparing this bound on the time budget with the corresponding conditions on the minimum time budget requested by Equations (10) and (12) for R-UCBE and R-SR , respectively. Regarding R-UCBE , we notice that the minimum admissible time budget in the smallβ regime is of order H 1 , 1 { β p T q β {p β ´ 1 q which is larger than term H 1 , 1 {p β ´ 1 q p T q of Equation (13). 10 Similarly, in the

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

largeβ regime (i.e., β ą 3 { 2 ), the R-UCBE requirement is of order H 1 , 2 { 3 p T q 3 ě H 1 , 2 p T q which is larger than the term of Theorem 6.1 since 1 {p β ´ 1 q ă 2 . Concerning R-SR , it is easy to show that H 1 , 1 {p β ´ 1 q p T q « max i P /llbracket 2 ,K /rrbracket i ∆ p i q p T q ´ 1 {p β ´ 1 q , apart from logarithmic terms, by means of the argument provided by (Audibert et al., 2010, Section 6.1). Thus, up to logarithmic terms, Equation (12) provides a tight condition on the minimum budget.

Error Probability Lower Bound We now present a lower bound on the error probability.

Theorem 6.2. For every algorithm A run with a time budget T fulfilling Equation (13) , there exists a SRB satisfying Assumptions 2.1 and 2.2 such that the error probability is lower bounded by:

Some comments are in order. First, we stated the lower bound for the case in which the minimum time budget satisfies the inequality of Theorem 6.1, which is a necessary condition for identifying the optimal arm. Second, the lower bound on the error probability matches, up to logarithmic factors, that of our R-SR , suggesting the superiority of this algorithm compared to R-UCBE . Finally, provided that the identifiability condition of Equation (13), such a result corresponds to that of the standard (stationary) MABs (Audibert et al., 2010; Kaufmann et al., 2016). A summary of all the bounds provided in the paper is presented in Table 1.

7 Numerical Validation

In this section, we provide a numerical validation of R-UCBE and R-SR . We compare them with state-of-the-art bandit baselines designed for stationary and non-stationary BAI in a synthetic setting, and we evaluate the sensitivity of R-UCBE to its exploration parameter a . Additional details about the experiments presented in this section are available in Appendix G. Additional experimental results on both synthetic settings and in a real-world experiment are available in Appendix H. 11

Baselines We compare our algorithms against a wide range of solutions for BAI:

RR : uniformly pulls all the arms until the budget ends in a round-robin fashion and, in the end, makes a recommendation based on the empirical mean of their reward over the collected samples;

RR-SW : makes use of the same exploration strategy as RR to pull arms but makes a recommendation based on the empirical mean over the last ε T K collected samples from an arm. 12

UCB-E and SR (Audibert et al., 2010): algorithms for the stationary BAI problem;

Prob-1 (Abbasi-Yadkori et al., 2018): an algorithm dealing with the adversarial BAI setting;

ETC and Rest-Sure (Cella et al., 2021): algorithms developed for the decreasing loss BAI setting. 13

The hyperparameters required by the above methods have been set as prescribed in the original papers. For both our algorithms and RR-SW , we set ε ' 0 . 25 .

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

0

50

100

150

200

250

300

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Number of pulls

n

Expected reward

µ

i

p

n

q

1

2

3

4

5

10

2

10

2

.

5

10

3

10

3

.

5

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Time Budgets

T

Empirical Error

e

T

R-UCBE

R-SR

RR

RR-SW

UCB-E

SR

Prob-1

ETC

Rest-Sure

100

120

140

160

180

200

220

240

0

0

.

2

0

.

4

0

.

6

0

.

8

1

Time Budgets

T

Empirical Error

e

T

a

'

a

˚

{

50

a

'

a

˚

{

10

a

'

a

˚

a

'

10

a

˚

a

'

50

a

˚

Setting To assess the quality of the recommendation ˆ I ˚ p T q provided by our algorithms, we consider a synthetic SRB setting with K ' 5 and σ ' 0 . 01 . Figure 2 shows the evolution of the expected values of the arms w.r.t. the number of pulls. In this setting, the optimal arm changes depending on whether T P r 1 , 185 s or T P p 185 , `8q . Thus, when the time budget is close to that value, the problem is more challenging since the optimal and second-best arms expected rewards are close to each other. For this reason, the BAI algorithms are less likely to provide a correct recommendation than for time budgets for which the two expected rewards are well separated. We compare the analyzed algorithms A in terms of empirical error e T p A q (the smaller, the better), i.e., the empirical counterpart of e T p A q averaged over 100 runs, considering time budgets T P r 100 , 3200 s .

Results The empirical error probability provided by the analyzed algorithms in the synthetically generated setting is presented in Figure 3. We report with a dashed vertical blue line at T ' 185 , i.e., the budgets after which the optimal arm no longer changes. Before such a budget, all the algorithms provide large errors (i.e., ¯ e T p A q ą 0 . 2 ). However, R-UCBE outperforms the others by a large margin, suggesting that an optimistic estimator might be advantageous when the time budget is small. Shortly after T ' 185 , R-UCBE starts providing the correct suggestion consistently. R-SR begins to identify the optimal arm (i.e., with ¯ e T p R-SR q ă 0 . 05 ) for time budgets T ą 1000 . Nonetheless, both algorithms perform significantly better than the baseline algorithms used for comparison.

Sensitivity Analysis for the Exploration Parameter of R-UCBE We perform a sensitivity analysis on the exploration parameter a of R-UCBE . Such a parameter should be set to a value less or equal to a ˚ , and the computation of the latter is challenging. We tested the sensitivity of R-UCBE to this hyperparameter by looking at the error probability for a P t a ˚ { 50 , a ˚ { 10 , a ˚ , 10 a ˚ , 50 a ˚ u . Figure 4 shows the empirical errors of R-UCBE with different parameters a , where the blue dashed vertical line denotes the last time the optimal arm changes over the time budget. It is worth noting how, even in this case, we have two significantly different behaviors before and after such a time. Indeed, if T ď 185 , we have that a misspecification with larger values than a ˚ does not significantly impact the performance of R-UCBE , while smaller values slightly decrease the performance. Conversely, for T ą 185 learning with different values of a seems not to impact the algorithm performance significantly. This corroborates the previous results about the competitive performance of R-UCBE .

8 Discussion and Conclusions

This paper introduces the BAI problem with a fixed budget for the Stochastic Rising Bandits setting. Notably, such setting models many real-world scenarios in which the reward of the available options increases over time, and the interest is on the recommendation of the one having the largest expected rewards after the time budget has elapsed. In this setting, we presented two algorithms, namely R-UCBE and R-SR providing theoretical guarantees on the error probability. R-UCBE is an optimistic algorithm requiring an exploration parameter whose optimal value requires prior information on the setting. Conversely, R-SR is a phase-based solution that only requires the time budget to run. We established lower bounds for the error probability an algorithm suffers in such a setting, which is matched by our R-SR , up to logarithmic factors. Furthermore, we showed how a requirement on the minimum time budget is unavoidable to ensure the identifiability of the optimal arm. Finally, we validate the performance of the two algorithms in both synthetically generated and real-world settings. A possible future line of research is to derive an algorithm balancing the tradeoff between theoretical guarantees on the e T and the chance of providing such guarantees with lower time budgets.

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

References

Tor Lattimore and Csaba Szepesvári. Bandit algorithms . Cambridge University Press, 2020.

Peter Auer, Nicolo Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning , 47(2):235-256, 2002.

Sébastien Bubeck, Rémi Munos, and Gilles Stoltz. Pure exploration in multi-armed bandits problems. In Proceedings of the Algorithmic Learning Theory (ALT) , volume 5809, pages 23-37, 2009.

Cem Tekin and Mingyan Liu. Online learning of rested and restless bandits. IEEE Transaction on Information Theory , 58(8):5588-5611, 2012.

Alberto Maria Metelli, Francesco Trovò, Matteo Pirola, and Marcello Restelli. Stochastic rising bandits. In Proceedings of the International Conference on Machine Learning (ICML) , pages 15421-15457, 2022.

Chris Thornton, Frank Hutter, Holger H Hoos, and Kevin Leyton-Brown. Auto-weka: Combined selection and hyperparameter optimization of classification algorithms. In Proceedings of the ACM (SIGKDD) , pages 847-855, 2013.

Lars Kotthoff, Chris Thornton, Holger H Hoos, Frank Hutter, and Kevin Leyton-Brown. Auto-weka 2.0: Automatic model selection and hyperparameter optimization in weka. Journal of Machine Learning Research , 18:1-5, 2017.

Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola. Autogluon-tabular: Robust and accurate automl for structured data. arXiv preprint arXiv:2003.06505 , 2020.

Yang Li, Jiawei Jiang, Jinyang Gao, Yingxia Shao, Ce Zhang, and Bin Cui. Efficient automatic CASH via rising bandits. In Proceedings of the Conference on Artificial Intelligence (AAAI) , pages 4763-4771, 2020.

Marc-André Zöller and Marco F Huber. Benchmark and survey of automated machine learning frameworks. Journal of Artificial Intelligence Research , 70:409-472, 2021.

Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, and Frank Hutter. Efficient and robust automated machine learning. In Advances in Neural Information Processing Systems (NeurIPS) , pages 2962-2970, 2015.

Quanming Yao, Mengshuo Wang, Yuqiang Chen, Wenyuan Dai, Yu-Feng Li, Wei-Wei Tu, Qiang Yang, and Yang Yu. Taking human out of learning applications: A survey on automated machine learning. arXiv preprint arXiv:1810.13306 , 2018.

Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. Automated machine learning: methods, systems, challenges . Springer Nature, 2019.

Marco Mussi, Davide Lombarda, Alberto Maria Metelli, Francesco Trovó, and Marcello Restelli. Arlo: A framework for automated reinforcement learning. Expert Systems with Applications , 224: 119883, 2023.

Leonardo Cella, Massimiliano Pontil, and Claudio Gentile. Best model identification: A rested bandit formulation. In Proceedings of the International Conference on Machine Learning (ICML) , volume 139, pages 1362-1372, 2021.

Hoda Heidari, Michael J. Kearns, and Aaron Roth. Tight policy regret bounds for improving and decaying bandits. In Proceeding of the International Joint Conference on Artificial Intelligence (AISTATS) , pages 1562-1570, 2016.

Benny Lehmann, Daniel Lehmann, and Noam Nisan. Combinatorial auctions with decreasing marginal utilities. In ACM Proceedings of the Conference on Electronic Commerce (EC) , pages 18-28, 2001.

Jean-Yves Audibert, Sébastien Bubeck, and Rémi Munos. Best arm identification in multi-armed bandits. In Proceedings of the Conference on Learning Theory (COLT) , pages 41-53, 2010.

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

Rianne De Heide, James Cheshire, Pierre Ménard, and Alexandra Carpentier. Bandits with many optimal arms. In Advances in Neural Information Processing Systems (NeurIPS) , pages 2245722469, 2021.

Emilie Kaufmann, Olivier Cappé, and Aurélien Garivier. On the complexity of best-arm identification in multi-armed bandit models. Journal of Machine Learning Research , 17:1:1-1:42, 2016.

Yasin Abbasi-Yadkori, Peter L. Bartlett, Victor Gabillon, Alan Malek, and Michal Valko. Best of both worlds: Stochastic & adversarial best-arm identification. In Proceedings of the Conference on Learning Theory (COLT) , volume 75, pages 918-949, 2018.

Victor Gabillon, Mohammad Ghavamzadeh, and Alessandro Lazaric. Best arm identification: A unified approach to fixed budget and fixed confidence. In Advances in Neural Information Processing Systems (NeurIPS) , pages 3221-3229, 2012.

Aurélien Garivier and Emilie Kaufmann. Optimal best arm identification with fixed confidence. In Proceedings of the Conference on Learning Theory (COLT) , volume 49, pages 998-1027, 2016.

Alexandra Carpentier and Andrea Locatelli. Tight (lower) bounds for the fixed budget best arm identification bandit problem. In Proceedings of the 29th Conference on Learning Theory , volume 49, pages 590-604, 2016.

Yonatan Mintz, Anil Aswani, Philip Kaminsky, Elena Flowers, and Yoshimi Fukuoka. Nonstationary bandits with habituation and recovery dynamics. Operations Research , 68(5):1493-1516, 2020.

Julien Seznec, Pierre Ménard, Alessandro Lazaric, and Michal Valko. A single algorithm for both restless and rested rotting bandits. In Proceedings of the International Conference on Artificial Intelligence and Statistics (AISTATS) , volume 108, pages 3784-3794, 2020.

Nir Levine, Koby Crammer, and Shie Mannor. Rotting bandits. In Advances in Neural Information Processing Systems (NeurIPS) , pages 3074-3083, 2017.

Julien Seznec, Andrea Locatelli, Alexandra Carpentier, Alessandro Lazaric, and Michal Valko. Rotting bandits are no harder than stochastic ones. In Proceedings of the International Conference on Artificial Intelligence and Statistics , volume 89, pages 2564-2572, 2019.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nStochastic Rising Bandits (SRBs) model sequential decision-making problems in which the expected reward of the available options increases after every time they are selected. \nWhile previous works addressed the regret minimization problem, this paper studied the fixed-budget Best Arm Identification (BAI) problem for SRBs. \nThis work proposed the R-UCBE and R-SR algorithms and showed that these two algorithms with classical designs achieve a small failure probability when the time horizon is sufficiently large. \nWith a lower bound, the author(s) also showed that the R-SR algorithm is near-optimal and a sufficiently large horizon is unavoidable for any algorithm to perform well.\nLastly, experiments are provided to validate the empirical performance of BAI algorithms.\n\n\n===========\n\nThe score is increased after I read the response from authors.\n\nSTRENGTHS:\n1. This work provides clearly formulates the rising bandit problem in Section 2.\n2. Before presenting the R-UCBE and R-SR algorithms, the author(s) describe how to estimate the expected rewards of arms in Section 3, which actually imply the intuition of algorithm designs. This can inspire readers to propose efficient bandit algorithms even beyond this rising bandit setting.\n3. In Section 7, experiments are provided to validate the empirical performance of BAI algorithms.\n\nWEAKNESSES:\n1. In Theorem 6.1, the lower bound on the time horizon $T$ depends on $\\Delta_i(T)$. The quantity, $\\Delta_i(T)$, depends on the instance, horizon $T$ and also the algorithm that we apply. Hence, $\\Delta_i(T)$ seems to be a random variable, and I don't think a random variable should appear in the lower bound. Moreover, we usually expect a lower bound holds true for many algorithms, and the term $\\Delta_i(T)$ seems to be different for different algorithms. or even different for the same algorithm in different trials.\n2. I have a similar concern for the upper bounds on the failure probabilities of the R-UCBE and R-SR algorithms.\n\nI think the contribution of this paper is much clearer, if my concerns above can be resolved.\n\nQUESTIONS:\nMy major concerns as listed in the *Weakness* section. Some other suggestions are as below:\n1. Line 45: 'failing to represent' may be revised to be 'but failed to represent'\n2. At the bottom of Page 2, it states that 'A complete discussion of the related works is available in Appendix A. Additional motivating examples are discussed in Appendix B.' I think these are important components of the paper, and at least a brief version should be included in the main paper.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper studies the fixed-budget best arm identification (BAI) under the stochastic rising bandit (SRB) problem. The stochastic rising bandit is to assume that the mean reward will increase as one plays the arm more. By assuming a concave increasing reward, the paper provides upper bounds for two algorithms: R-UCBE (which is UCB-type) and R-SR (which is elimination-based). It further provides lower bound proof, and such a bound is matched by R-SR up to logarithmic factors. Numerical experiments are listed.

STRENGTHS:
- The paper is well-written and carefully organized

- Matching lower and upper bounds (up to logarithmic factors)

WEAKNESSES:
1) Deterministic growth function $\gamma$, meaning that the randomness of $\gamma$ does not accumulate; Not the practical case (consider SGD, former parameters will affect the consecutive)

2) What if $c \rightarrow 0$? The requirement for $T$ (10) seems to vanish. Can you make further explanations? 

Can you recover the bound of Audibert et al. (2010)?

QUESTIONS:
See above.

LIMITATIONS:
Na.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper is about bandit best arm identification with fixed budget, in a non-stationary setting. This is a rested bandit problem: the mean reward of an arm changes each time it is pulled, but does not change when it is not pulled. The main assumption is that the mean reward is a non-decreasing, concave function of the number of pulls. Some results also use another assumption: an upper bound on the increments.\nThe authors introduce estimators of the mean rewards that are adapted to that setting and use them in two algorithms R-UCBE and R-SR, which are inspired by the UCBE and SR fixed budget algorithms. The paper contains upper bounds on the error probability of these algorithms as well as lower bounds on the error probability of any algorithm and a discussion of the minimal budget necessary to identify the best arm.\n\nSTRENGTHS:\nThe rising bandit problem is important: it corresponds to allocating resources to different learning algorithms, in order to identify the one with best performance once fully trained.\n\nThe estimators and algorithms are well explained and motivated. The graphical representation of figure 1 is very helpful.\n\nThe lower and upper bounds show that the methods are close to optimal for the problem.\n\nThe discussion of the minimal budget necessary to identify the best arm is interesting and highlights a feature of rising bandits which is not present in standard BAI.\n\nThe experimental evaluation is convincing.\n\nWEAKNESSES:\nR-UCBE depends on a parameter that needs to be tuned using unavailable information, but that theoretical weakness is directly inherited from UCBE and the practical performance of the algorithm is very good. Hence this is a very mild weakness.\n\nThe lower bound of theorem 6.2 is of order $\\exp(-T/H)$, while a famous feature of fixed budget BAI (in the stationary setting) is that this is not achievable. Indeed, it is shown in [Carpentier and Locatelli, Tight (lower) bounds for the fixed budget best arm identification bandit problem, Colt 2016] that there is a lower bound of order $\\exp(-T/(H \\log K))$, matching the upper bound of SR. In light of that lower bound, we would expect a similar result for rising bandits, stronger than Theorem 6.2.\n\nQUESTIONS:\nIs it possible to strengthen the lower bound?\n\nWhy nondecreasing and concave mean rewards? It should be easy to get large lower bounds if assumption 2.1 is not satisfied. Could you point to such results, in order to justify the assumption?\n\nLIMITATIONS:\nThe limitations are adequately discussed. No concern about a negative societal impact.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study explores the stochastic rising bandits (SRB) in fixed-budget best arm identification (BAI). The authors initially formulate this novel problem setting and then introduce two types of estimators. For these estimators, they demonstrate upper bounds that match their lower bounds. Lastly, they validate the reliability of their approach through various experiments.\n\nSTRENGTHS:\nFirstly, this is my initial exposure to a paper on the SRB setting. Consequently, I'm not sure if the setting is truly novel. However, I am persuaded that the setting is both crucial and intriguing, particularly given the practical importance of the CASH problem. As I'm unable to gauge the novelty of the setting, my remarks are primarily oriented towards the technical aspects of BAI.\n\nFirstly, the concepts of pessimistic and optimistic estimators are persuasive and well-founded. The authors deliver thorough and robust theoretical results for the estimators, consistent with established practices in this field. Even though the outcomes are not surprising, I see no strong grounds to reject this paper. On the whole, this study presents a typical analysis within an interesting and novel setting.\n\nPlease note, this is a preliminary review. I am currently delving into further details, including the proof. I may revise this review later.\n\nWEAKNESSES:\nIt appears that a weakness resides in the need for a large budget. Moreover, verifying whether the condition is met could be challenging due to the somewhat complex form of the time budget constraint.\n\nQUESTIONS:\nCould we derive more practically applicable results for realistic settings by assuming specific models for rewards? For instance, in economics, we often presume particular models for a utility function that grows as the quantity of certain variables (e.g., consumption) increases. That is, if we define a certain mechanism (e.g., linear models) for the increasing rewards, could we achieve tighter results? If the authors' claims are correct, this study serves as a pioneering effort in this field. Therefore, incorporating such constraints could be seen as future work, and isn't necessarily a task the authors need to undertake at present. However, I am interested in understanding the potential for such an extension.\n\nLIMITATIONS:\nSee weakness.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}